You Build It, You Secure It

Leading to DevOps

In 2006 Werner Vogels, Amazon’s CTO, said in an interview with the Association for Computing Machinery (ACM) that developers at Amazon don’t throw software over the wall. His famous quote was “You build it, you run it”. This simple phrase became a battle cry for the DevOps movement.¹ In 2017, the process battle in which DevOps has been engaged in is going well. Unfortunately, we find ourselves in another, potentially more critical fight. Despite the fact that breaches and destructive attacks dominate the headlines (Equifax, anyone?), security remains an afterthought, relegated to the end of the development pipeline. And, while software development has become a team sport, security engineers often aren’t part of the team. However, it’s all part of an evolutionary process. We didn’t always have a team, and DevOps as we know it today didn’t exist a dozen years ago.

Many movements influenced DevOps, including Agile, Lean, Open Source, and Web Scale. Agile is a rapid method of delivering software incrementally with short lead times (called sprints), as opposed to the traditional “waterfall” sequential software development life cycle. The increments are based on user stories provided and prioritized by business owners. The Agile manifesto lays out twelve principles that define what processes developers should follow.² The Agile movement was successful and had a major impact on software development; however, one of its limitations was that it failed to include IT operations in the process.

Lean adapted processes employed by manufacturing then applied them to IT development and operations. Many of the principles that are essential to the DevOps philosophy are borrowed from manufacturing best practices, particularly those employed by Toyota.

Open Source’s contribution gave the community free operating systems and tools, which in turn economically facilitated the modern web. The myriad of other software that we use through open source projects in turn created the environment that made DevOps possible.

Finally, Web Scale IT is an architectural pattern that takes advantage of Agile, Lean, and Open Source, as well as Continuous Delivery, which is a fundamental DevOps methodology. Because of the capability this provides, executives are now willing to support the DevOps methodology, which would have previously been seen as a risk factor.

DevOps History

The first DevOps Days Conference was held in Ghent, Belgium in 2006. John Willis was the only American in attendance. A few months later another conference was held in Mountain View, California. At that conference, John, along with Damon Edwards, coined the acronym CAMS (Culture, Automation, Measurement, and Share) which defined DevOps in a nutshell. They viewed culture as including improvement, automation as being a better way of delivery, measurement as an opportunity for learning, and sharing as another way of saying collaboration.

Many of the principles that are essential to the DevOps philosophy are borrowed from manufacturing best practices…

CAMS can be related to the Three Ways defined in Gene Kim’s book, The Phoenix Project³, which has become a foundation text for DevOps philosophy. The First Way requires rapid and smooth workflow from development to operations in order to deliver customer value. The Second Way describes the enablement of fast and constant feedback from right to left at every stage of the value stream. The Third Way instills the creation of a culture of continual learning and experimentation. These three principles result in a steady creation of individual knowledge, which becomes team and organizational knowledge. In fact, we can think of Automation as the First Way, Measurement as the Second Way, and Sharing as the Third Way.

Of CAM’s four elements, culture is the most complicated for any organization trying to embrace DevOps. Due to the cultural and implementation issues that were impeding the DevOps movement, some of the people who had achieved considerable success decided to get together and document a ”how to” book. Their work, The DevOps Handbook⁴, provides a clear and concise roadmap for DevOps practices and patterns. These practices and patterns can be divided into two broad categories: Continuous Delivery and Culture.

Even after the DevOps movement came about, security was not part of the picture.

Continuous Delivery includes:

• Everything in version control
• Small batch principle
• Trunk based deployment
• Manage flow (work in process)
• Automate everything

Culture includes:

• Everyone is responsible
• Done means released
• Stop the line when it breaks
• Remove silos

High Performing Organizations

All companies and agencies want to be high performing organizations. High performing organizations accelerate technology delivery, which in turn accelerates market opportunities. These organizations do two things well: they create automated pipelines for software delivery (known as Continuous Delivery) and they create collaborative high trust cultures. High performing organizations do not think in terms of products or projects but they consider what they produce as services owned by the entire team. No one would ever throw code “over the wall” because they think done means software is released into production. Delivery does not mean that anything has ended; it is cyclical and therefore is a service to be continued. Traditional thinking emphasizes a choice between faster, cheaper and more resilient (better). High performing organizations break this paradigm because they have a generative culture. They are able to deliver faster with more resilience. Being faster means they fail fast. If they fail fast, they find defects early, and then defects get fixed rapidly before they have a major cost impact, so the message becomes – go faster to be cheaper and more resilient.

DevSecOps—DevOps with Security

It is interesting that we are now making the same mistakes with security that we made with operations prior to DevOps. Agile accelerated software delivery but operations were not included. This drove the software industry to create DevOps.

Even after the DevOps movement came about, security was not part of the picture. We no longer have a wall between development and operations but the wall between DevOps and security remains. After we are done with the delivery process we still throw it over the wall – this time to security. Now, security has become the bottle neck. After everything else has been completed, activities such as penetration testing, vulnerability assessment and testing, and vulnerability scanning happen. However, sometimes this does not happen at all.

It’s not just the bottleneck factor, or that security is an afterthought. Security failures have real consequences. For example, an average data breach incident costs “…$3.5 million dollars and 15 percent more than what it cost last year.”⁵

If this was not enough motivation, then there are the global security breaches that have made the news, such as the “WannaCry” ransomware or the Equifax breach. It’s become painfully clear that we must prioritize building security into our software. Some early leaders in the DevOps Security movement (aka DevSecOps), including Josh Corman and Gene Kim came up with a new battlecry, which demands that we all build security into software from the requirements phase forward: “Rugged Software: Security is Dead. Long Live Rugged DevOps.”⁶ The Rugged Software Manifesto also emerged from this movement, espousing the following principles:

• I am rugged – and more importantly, my code is rugged.
• I recognize that software has become a foundation of our modern world.
• I recognize the awesome responsibility that comes with this foundational role.
• I recognize that my code will be used in ways I cannot anticipate, in ways it was not designed, and for longer than it was ever intended.
• I recognize that my code will be attacked by talented and persistent adversaries who threaten our physical, economic, and national security.
• I recognize these things – and I choose to be rugged.
• I am rugged because I refuse to be a source of vulnerability or weakness.
• I am rugged because I assure my code will support its mission.
• I am rugged because my code can face these challenges and persist in spite of them.⁷

Since Continuous Integration, Continuous Delivery, and Continuous Deployment are critically important in a DevOps environment, it is equally important to have security built into every stage of the pipeline. This means that we must add security practices and tools into the process. We need a holistic model. Such a model begins with threat modeling during design and the addition of security engineers to the design team where they can lead the early discussion of security requirements definition. Since they would be an integral part of the team, they would be involved when users are building their use cases. Security personnel should also be present at the architecture reviews.

We can follow through in development during the build phase. On the Continuous Integration (CI) side, static analysis would be conducted in the integrated development environment (IDE), in the CI build server, and we will be monitoring and scanning for secure libraries. If infrastructure is now code, then security should be code. Security engineers can participate when the cloud templates and Chef cookbooks are being built. There is, frankly, no reason they cannot be taught to build templates themselves. Security engineers should be making developers aware of known issues and vulnerabilities. Eventually, security engineers will create plugins and code examples for developers that demonstrate “how to” fix known vulnerabilities.

Security engineers should be making developers aware of known issues and vulnerabilities.

We will want to drive this to the automated testing phase and include Behavior Driven Testing (BDT) and penetration testing. We can also set security coding standards and train developers about security. We want to be open minded about Open Source and therefore we need to apply governance policies. A surprisingly high percentage of security breaches can be traced to human factors and policy deficits around system administration, patching and updates.

Just as DevOps decrees, that responsibility does not end when releasing software into production, so does DevSecOps. Continuous monitoring has never been more important because distributed applications, mobile computing and the Cloud have caused the attack surface to expand exponentially. There were enough threats when an organization’s entire environment was contained within four walls! Given the cloud, it’s difficult to articulate the degree to which the environment has expanded. There are, literally, no boundaries. It becomes imperative to understand vulnerabilities in terms of on-going configuration management analysis. The software development process has become a mature delivery process with tools such as GitHub and Jenkins for every phase of development. Now we need to extend that model to DevSecOps. It is true that we have the beginnings of this maturity with tools like Sonatype for build and Fortify for test, but we have quite a way to go to match the toolset that exists for DevOps. In addition, just as we had cultural challenges in implementing DevOps, they also exist for DevSecOps. Once we are all adhering to the Rugged Software Manifesto, then we can finally say: I built it, I secured it.

Sources
1. https://queue.acm.org/detail.cfm?id=1142065
2. http://agilemanifesto.org/
3. https://itrevolution.com/book/the-phoenix-project/
4. http://itrevolution.com/devops-handbook
5. https://www.ponemon.org/blog/ponemon-institute-releases-2014-cost-of-databreach-global-analysis
6. https://www.ruggedsoftware.org/
7. https://www.ruggedsoftware.org/
John Willis, Barbara Bouldin