Fixing Vulnerabilities at Speed—How Engineering and Security Can Achieve True DevSecOps

What happens when two distinct teams with varied technological expertise, different incentive structures and contrasting priorities converge?—the answer is usually tension. During a recent discussion with colleagues, a completely different result was experienced. ‘Camaraderie’ is the word that immediately comes to mind when thinking about the veteran DevSecOps/engineering and security leaders. Sure there are still big hurdles to fixing vulnerabilities at speed without slowing the pipeline, but this conversation was honest, respectful and above all, realistic.

At most enterprise software companies, a healthy/unhealthy tension is especially palpable between the engineering team and the security team. Finding the right balance between features and functionality is a long-standing software quest, and it can be argued that security underpins both. But if so, why is it still so difficult for security and engineering teams to work cooperatively toward a common goal—the timely delivery of a secure, bug-free release of the highest quality? From what different colleagues in the industry had to say, there are proactive measures we can take, including:

• Clearly define goals between security and engineering.
• Set metrics, track them and align incentives across security and engineering.
• Learn to speak the same language, don’t “cry wolf,” and hold the culture accountable.

From personal observations and experience, security remains only an influencer in the development cycle. Security doesn’t actually remediate issues. That happens in engineering or in other teams, and those teams are often carrying the weight of the business. One colleague remarked, “If I’m a developer and I want to be promoted at the end of the year, my scorecard is based on what features I helped develop and ship this year that grew our audience and our revenue.” Slowing that delivery, even for security purposes, can be a tough sell. One of the first key takeaways is the need to clearly communicate goals and priorities across both groups.

Clearly define goals and priorities between security and engineering to ease tension and improve cooperation.

From an enterprise software standpoint, unless something is on fire, it may not receive priority. Developers focus on designing new features and adding more value to customers. How do you prove to customers that enhancing security also delivers value? Many times in a product roadmap discussion, security takes a hit unless it receives top priority from the outset.

Success depends on clearly defining the goal or desired outcome. In the ideal culture, the DevSecOps leaders have tremendous empathy and respect for the security leaders and vice versa. Ultimately it’s about driving clarity and helping people understand the holistic picture—that a high quality product is a secure product. Process and documentation can help define and implement this philosophy.

Set metrics and track them to align incentives across security and engineering.

Your metrics and incentives should reflect what you care about. For example, a system health strategy program is basically a framework to measure and assess the health of a system very consistently. It combines the maturity model with certain metrics so that system contributors can closely gauge the health of the system based on common workflows. By setting a benchmark for all systems internally, you can clearly communicate to the C-suite and other departments what issues you care about and why they’re important using neutral criteria.

Product won’t ship unless system health meets certain standards and QA signs off. Security should be embedded into the same model, using release criteria with metrics that mirror other QA standards. Just as Jira charts and burndown charts track the quality bugs in a release, we should see the same dashboard in terms of security bugs. As a result, a product manager not normally attuned to the small details of security vulnerability will realize that despite overall quality, the product should not ship with security flaws. Setting metrics and tracking them helps both engineering and security work from the same reality. It also aligns incentives.

Learn to speak the language, don’t “cry wolf,” and hold the culture accountable.

Communication is only helpful when you’re speaking the same language. The security colleagues agreed that framing security flaws as quality defects is important. But don’t assume that engineers understand or have the same information. The engineers advised, “Assume we don’t know, explain the security concerns, then listen to our perspective.”

Admittedly, some security professionals fall into the “boy who cried wolf” predicament. If they label everything a potential risk, they lose credibility with a development team fatigued by chronic vulnerability fire drills. Recognize that organizations can do two things—immediately remediate vital security flaws and simultaneously and incrementally improve processes and metrics focused on secure application development and prevention.

What if everyone buys in and agrees that security is important, but no one actually behaves any differently? That sounds like a culture problem, and if you zoom in, a behavior problem. Leaders should set the tone and model how to systematically integrate processes between security and engineering and align incentives. It turns out, we can and we must do two things at once if we’re going to address vulnerabilities at speed.