Introduction
The Department of Defense (DOD) Cybersecurity Service Provider (CSSP) Program is one of the largest and most complex Federal programs. Currently, the Program relies on 27 authorized DOD CSSPs to meet its mission of provisioning 24x7x365 cybersecurity services (i.e., identify, protect, detect, respond, and recover) to safeguard the DOD portion of the cyberspace domain known as the Department of Defense Information Network (DODIN). The DODIN is a federated technological environment of over 15,000 unclassified and classified networked and cloud environments around the world managed by 45 combatant commands, Services, and DOD agencies and field activities. As these Defense Department organizations rely on the DODIN to carry out their missions, each DOD CSSP is responsible for defending a small subset [of the DODIN] ensuring that military organizations, individually and collectively, are capable to sustain their missions in, through, and from the cyberspace domain.
To accomplish their missions, each individual DOD CSSP integrates operators, technologies, processes, services, and subscriber elements to provision 24x7x365 Defensive Cyberspace Operations (DCO) services and functions. These DCO actions and activities are designed to secure DOD components’ information systems, networks, data, and information used to organize, plan, direct, manage, and monitor their mission/business operations.
As it relates to technologies, DOD CSSPs rely on a variety of mechanisms and modern software development approaches such as Agile, DevOps (Development and Operations), and DevSecOps (Development, Security, and Operations) to collaborate in small cross–functional teams that include users, testers, software developers, and cybersecurity experts. Together, these teams deliver technical DCO solutions that meet global subscriber’s needs at relevant cyber speeds. The importance of cyber speed goes beyond being able to protect, monitor, detect, analyze, and respond to cyber incidents and events. Velocity, in the daily function of DOD CSSPs has become one of the key parameters by which success is defined for both Offensive Cyberspace Operations and DCO missions.
Simply put; by integrating Dev and Ops teams, DevOps has streamlined processes, accelerated delivery, and improved collaboration.
As DOD CSSPs continue to accelerate migration of services into approved DOD cloud service offerings, these 24x7x365 cyber providers have chosen software DevOps Continuous Integration/Continuous Delivery (CI/CD) environments as the ecosystem of choice to develop their DCO technologies. Why DevOps? DevOps describes an Agile software–development approach that does not end when the software is released, but extends to the operational phase, which supports the delivery and implementation of software to subscribers. Simply put; by integrating Dev and Ops teams, DevOps has streamlined processes, accelerated delivery, and improved collaboration. That stated, as DOD CSSPs continue to embrace DevOps practices, which are, by most CSSP developers, better suited for a faster response to subscribers without sacrificing quality, they are now confronted with after–the–fact cybersecurity concerns identified separately and much later in the application lifecycle through siloed security testing.
What is the sweet spot? When velocity is not a requirement, developers have the time to bake cybersecurity into the development process through DevSecOps. DevSecOps automates, monitors, and applies Security (Sec) at all phases of the DevOps software lifecycle (i.e., plan, develop, build, test, release, deliver, deploy, operate, and monitor). However, what can be done to enhance an already established DevOps working environment in order to maintain the speed of cyber and subscriber requirements, while minimizing unnecessary delays? As DevOps CI/CD environments continue to become an attractive target for malicious cyber actors, this article explores the integration of cybersecurity best practices into software DevOps CI/CD environments to secure and monitor CI/CD deployments after the fact.
...adversaries of all levels of sophistication are shifting their attention to CI/CD environments and adapting their TTPs to the new realities of CI/CD.
CI/CD Environment
CI/CD is a software development approach where all developers collaborate on a central co–repository. In this environment, developers build, test, and maintain a consistent code base for their applications while dynamically and quickly integrating code changes. CI/CD pipelines are mostly implemented in cloud environments because of the cloud’s current role in the Department’s IT modernization efforts. As the DOD continues its transition to the cloud, CI/CD environments have become an attractive target for malicious cyber actors whose goals are to compromise information by introducing malicious code into CI/CD applications, gaining access to intellectual property/trade secrets through code theft, and/or causing denial of service effects against applications. Realizing CI/CD services provides an efficient path to reaching an organization’s crown jewels, adversaries of all levels of sophistication are shifting their attention to CI/CD environments and adapting their TTPs to the new realities of CI/CD. Meanwhile, cyber defenders are still early on in their efforts to find the right ways to detect, understand, and manage the risks associated with these environments. As we continue to witness a significant rise in the amount, frequency, and magnitude of incidents and attack vectors focusing on abusing flaws in the CI/CD ecosystem, it is imperative that we find the right balance between optimal cybersecurity and engineering velocity to remain agile without compromising security.
Understanding DevOps CI/CD Security Risks
There are clear indications, a new class of CI/CD attacks targeting GitHub Actions is compromising the PyTorch supply chain. For example, researchers have identified exploited self–hosted runners in repositories compromised through unauthorized access by means of forked pull requests. With this access level, an attacker could theoretically execute arbitrary codes, access secrets, and potentially upload malicious PyTorch releases. This attack, demonstrated on PyTorch, highlighted significant security risks in CI/CD pipelines that have been recently mitigated through integrated isolation strategies, ephemeral runners, and mandatory approvals for all pull requests.
Another illustration is the AnyDesk breach. Recently discovered in mid–January 2024, this breach compromised production systems and exposed the importance of code signing security. The breach exposed AnyDesk customers’ license keys, active connections, duration of sessions, customer IDs and contact information, email associated with the account, and the total number of hosts that have remote access management software activated. The attack began in December 2023 and led AnyDesk to revoke security certificates and force password resets for all users after it found thousands of user credentials for sale on the dark web. The breach highlighted the risks of compromised code signing certificates, which could allow attackers to distribute malicious software appearing legitimate. This incident underscored the necessity for secure code signing practices in software development to prevent supply chain attacks.
Lastly, researchers have also found that proof–of–concept exploits for Jenkins vulnerabilities CVE–2024–23897 and CVE–2024–23898 are being actively targeted in the wild. The critical flaw CVE–2024–23897 allows remote code execution by exploiting the expandAtFiles functionality, while CVE–2024–23898 involves cross–site WebSocket hijacking. Both vulnerabilities have been newly patched in Jenkins versions 2.442 and LTS 2.426.3 and require administrators to update their Jenkins instances and/or disable CLI access to mitigate these risks.
Best Practices for Securing DevOps Environments
To address these security risks and ensure the security of DevOps environments, organizations should adopt a multi–faceted approach that encompasses people, processes, and technology. Here are some best practices to consider:
- Embrace a Security–First Culture. Security should be ingrained into every aspect of the DevOps lifecycle, from planning and development to deployment and operations. Foster a culture where security is everyone’s responsibility and provide regular training and awareness programs to educate developers and operations teams about security best practices.
- Implement Secure Development Practices. Adopt secure coding practices such as input validation, output encoding, and proper error handling to mitigate common vulnerabilities like injection attacks (e.g., SQL injection, XSS). Use static code analysis tools to identify security flaws early in the development process and conduct regular security code reviews to catch any issues before they reach production.
- Harden Infrastructure and Configuration. Ensure that all infrastructure components, including servers, containers, and cloud services, are properly hardened, and configured according to security best practices. Use configuration management tools to enforce security policies and automate the provisioning and configuration of infrastructure resources. Implement least privilege principles to restrict access and minimize the impact of potential security breaches.
- Secure CI/CD Pipelines. Integrate security checks into CI/CD pipelines to scan code for vulnerabilities, malware, and compliance violations before deployment. Use automated testing tools to perform static and dynamic application security testing (SAST and DAST) and container vulnerability scanning. Implement approval gates and manual review processes to validate changes and prevent unauthorized deployments.
- Monitor and Detect Anomalies. Deploy robust logging, monitoring, and intrusion detection systems to track activity across the DevOps environment and detect suspicious behavior or security incidents in real time. Use security information and event management (SIEM) solutions to aggregate and analyze logs from various sources and generate alerts for potential security threats. Implement anomaly detection algorithms to identify deviations from normal patterns of behavior.
- Enforce Access Control and Authentication. Implement strong authentication mechanisms, such as multi–factor authentication (MFA) and single sign–on (SSO), to control access to DevOps tools and resources. Use role–based access control (RBAC) to enforce the principle of least privilege and ensure that users have only the permissions necessary to perform their tasks. Regularly review and audit user permissions to identify and remediate any unnecessary or excessive access rights.
- Continuously Monitor and Improve Security Posture. Security is not a one–time effort but an ongoing process that requires continuous monitoring, assessment, and improvement. Conduct regular security assessments, penetration tests, and vulnerability scans to identify and remediate security weaknesses proactively. Stay informed about emerging threats and security trends, and update security policies and controls accordingly to adapt to evolving risks.
- Foster Collaboration and Communication. Promote collaboration and communication between development, operations, and security teams to ensure that security requirements are integrated into the DevOps workflow effectively. Establish clear channels of communication for reporting security incidents and coordinating incident response efforts. Encourage cross–functional teams to work together to address security challenges and implement proactive security measures.
Additional Considerations and Future Trends
As technology continues to evolve, so too will the security challenges facing DevOps environments. Organizations must remain vigilant and adapt their security strategies to address emerging threats and trends. Some additional considerations and future trends to keep in mind include:
- Zero Trust Architecture. The concept of zero trust architecture, which assumes that every user, device, and network resource is untrusted until proven otherwise, is gaining traction in the cybersecurity community. Implementing zero trust principles can help organizations better protect their DevOps environments by enforcing strict access controls and segmenting network traffic.
- DevSecOps Integration. DevSecOps, which integrates security into the DevOps workflow from the outset, is becoming increasingly important as organizations seek to build more secure software faster. By embedding security practices and tools into every stage of the development lifecycle, DevSecOps aims to shift security left and make security everyone’s responsibility.
- Artificial Intelligence and Machine Learning. Artificial intelligence (AI) and machine learning (ML) technologies have the potential to revolutionize cybersecurity by automating threat detection, response, and prediction. By analyzing large volumes of data and identifying patterns and anomalies, AI and ML algorithms can help organizations better protect their DevOps environments against evolving threats.
- Container Security. With the widespread adoption of containerization technologies, such as Docker and Kubernetes, container security has become a top priority for DevOps teams. Future trends in container security may include the development of new tools and techniques for securing containerized applications, as well as increased focus on runtime protection and container orchestration security.
Conclusion
Securing DevOps environments requires a proactive, holistic, and multi–faceted approach that encompasses people, processes, and technology. By embracing a security–first culture and implementing secure development practices―such as hardening infrastructure and configuration, securing CI/CD pipelines, monitoring, and detecting anomalies, enforcing access control and authentication, and continuously monitoring and improving security posture―organizations can mitigate risks and protect their DevOps workflows from potential security threats. As technology continues to evolve, organizations must remain vigilant and adaptable, continuously refining their security strategies to stay one step ahead of cyber threats. 
Travis Tyrer
Leave a Comment