Introduction
In the fast–paced world of cybersecurity, staying ahead of emerging cyber threats is not simply a necessity but an essential, constant challenge. Since its inception, the Department of Defense (DOD) Cybersecurity Service Provider (CSSP) Program has been at the forefront of the Department’s cyberwarfare battleground. Over the years, the Program has morphed significantly and constantly, assisting and guiding authorized DOD CSSPs to protect the DOD portion of the cyberspace domain. Through periodic evaluations that have become more complete, comprehensive, and sound, the program has managed to withstand ever changing technical and operational threat environments, attack methodologies, emerging technologies, governance, as well as needed administrative and programmatic changes.
This article delves into the transformative journey of the DOD CSSP Program and its formal evaluations. It introduces the reader to the Evaluator Scoring Metrics (ESM), as well as adaptations from its initial version to the current ESM v11 evaluation criteria. This article also explores the approach some DOD CSSPs are now taking to stay ahead of new ESM requirements, while improving and optimizing 24/7 operators, the technologies they use, the processes they follow, the Defensive Cyberspace Operations (DCO) services they provision, and the subscriber’s level of service satisfaction they protect and defend.
Throughout its evolution, the program has relied on the ESM as a tool to gage the progress, compliance, and performance of DOD CSSPs triennially.
DOD CSSP Program
Established in 2003 with the release of DOD O-8530.1-M, the DOD CSSP Assessment and Authorization Program, formally known as the Computer Network Defense Service Provider (CNDSP) Certification and Accreditation Program, emerged as a major component of the Defense Department’s cyber risk management strategy. Today, the program is supported by 27 authorized service providers responsible for provisioning 24x7x365 cybersecurity services to protect the DOD portion of the cyberspace domain, which extends globally across 145 countries, 15,000 classified and unclassified networks, and over 7.5 million computers and connected devices worldwide.
Throughout its evolution, the program has relied on the ESM as a tool to gage the progress, compliance, and performance of DOD CSSPs triennially. The ESM contains the criteria by which DOD CSSPs are evaluated; it is also leveraged to drive contractual requirements and conduct self–assessments. The ESM is built from cybersecurity functions outlined in Federal, DOD, and Joint Staff cyber doctrine; and while it applies universally, advanced DOD CSSPs are known for tailoring metrics for applicability towards their respective unique operational environments. The following sections help illustrate the key characteristics and evolution of the ESM.
DOD ESM
ESM v1–v6 (2003 – 2007)
The formative years of the DOD CSSP Program were critical but neither effective nor decisive to the design and initial implementation of the ESM. For the first 4 years, almost annually, a new version and subversion of the ESM was negotiated and released for comments and feedback to revolving stakeholders that arguably and inadvertently delayed the operationalization of the ESM. During this stage, the program was somewhat informal, functioning mostly on a voluntary basis with cyber and non–cyber principals willing and available to participate. During this period, the resulting ESM was semi-structured and difficult to use as it focused on individual components rather than the whole. Some of its elements were not necessarily applicable to all service providers, yet they allowed for the basic forming of the program’s evaluation process.
ESM v7 (2007 – 2011)
The Program’s formal evaluation journey began with the release of ESM v7 in 2007. While earlier versions were geared towards basic incremental enhancements rather than a comprehensive framework, ESM v7 marked a vital step forward by introducing a systematic and methodical structure that addressed technical and operational ambiguities as well as inconsistencies within DOD policy. ESM v7 was organized into 4 functional areas: protect, detect, respond, and sustain and 18 Computer Network Defense (CND) services. ESM v7 metrics were also prioritized in 4 levels of criticality (i.e., PI–PIV) with PI being the most and PIV the least critical. After each individual assessment, the service provider was issued an Authorization to Operate (ATO) based on 3 levels (i.e., I–III) with a level III representing the highest and level I the lowest score. These levels were determined by the overall accumulation and percentage of complaint metrics weighted by priority. ESM v7 was replaced by ESM v8 in 2011.
ESM v8 (2011–2015)
Released in June 2011, ESM v8 was considered by many within the CNDSP Community as one of the most successful ESM versions, if not the most successful. While ESM v8 was organizational–wide focused (vs. services focused), it set the stage for a systematic evaluation process that incorporated a stringent document review and has proven valid throughout the years. In essence, ESM v8 aligned 117 metrics into 4 functional areas and 18 CND services derived primarily from DODD 8530.1 and DODI 8530.2. These metrics were categorized into 4 priorities I–IV with resulting evaluations falling under 3 ATO Levels I–III (defined by a compliance rate). ESM v8 was in effect for 4 years and was replaced in 2015 by ESM v9.
ESM v9.2 (2015–2019)
Released in December 2015, this version consolidated incremental enhancements observed and piloted through ESM v9. In essence, ESM v9.2 incorporated policy updates and most notably, a change in cyber lexicon that shifted the program from the administration of CNDSPs to CSSPs. Widely regarded as the apex of maturity within the CSSP community, ESM v9.2 introduced a maturity model and enhanced metrics to monitor progress and continuous improvement. The Department of Homeland Security (DHS) CSP Program, akin to the DOD CSSP Program, embraced ESM 9.2 in 2019, showcasing its adaptability and effectiveness. With 13 cybersecurity activities augmented by 38 performance metrics, ESM v9.2 laid the groundwork for evaluating CSSPs based on maturity levels of services that ranged from incomplete to innovative. By defining maturity levels, ESM v9.2 facilitated a more nuanced assessment of CSSPs which allowed for a thorough analysis of their capabilities and practices. This approach went beyond a basic evaluation and provided more granular insight into the extent of complexity and innovation integrated into the CSSP service offerings. Simply put, it assisted CSSPs in identifying current compliance levels, deficiencies, and strengths, as well as recognizing the areas in need of immediate improvement.
ESM v10 (2019–2023)
Driven by Executive Order 13800, ESM v10 was released in June 2019. This version mandated the adoption of the National Institute of Standards and Technology (NIST) Cybersecurity Framework (CSF), a comprehensive and widely recognized framework designed to help organizations manage, reduce, and communicate cybersecurity risks. Despite its positive intent to move the Program forward, ESM v10 faced immediate criticism within the CSSP community due to a limited understanding and inability to properly tailor and align the CSF to the DOD CSSP model and take advantage of an already robust ESM 9.2 framework, as DHS had previously done. Organized into NIST CSF core functions and 20 cybersecurity service categories, ESM v10 introduced 50 metric indicators, further divided into Measures of Performance (MOPs) and Measures of Effectiveness (MOEs). While the NIST CSF is a framework designed to be tailored to specific needs, ESM v10 failed to effectively customize it to meet the unique and diverse requirements of the Program. ESM v10 was considered by most CSSP stakeholders a setback for the Department.
ESM v11 (2023 – Present)
Released in July 2023, ESM v11 represents a significant expansion of DOD CSSP evaluation metrics. Going beyond Defensive Cyber Operations (DCO), ESM v11 now includes responsibilities for DOD Information Network Operations (DODIN Ops). This expansion ushers in a new era of proactive obligations, measuring collaboration, cooperation, and coordination to secure, operate, and sustain assigned portions of the DOD cyberspace. In essence, DOD CSSPs, executing reactive DCO missions in response to specific threats, now face additional proactive duties outlined in ESM v11. These metrics focus on both vertical and horizontal collaboration, emphasizing the need for a comprehensive approach to maintain the confidentiality, availability, and integrity of the DODIN.
CI–O Management Systems
ESM v11 triggered a massive redesign and re–implementation effort for most DOD CSSPs striving to integrate, manage, and optimize DCO and DODIN Ops measures. While some CSSPs have achieved foundational maturity levels, others are actively seeking ways to revamp their cyber operators’ roles, technologies, processes, services, and levels of subscriber’s satisfaction. In response to the challenges presented by ESM v10 and v11 requirements, next generation DOD CSSPs, such as the U.S. Army Futures Command’s C5ISR Center and the U.S. Transportation Command (USTRANSCOM), have successfully piloted customized Continuous Improvement and Optimization (CI–O) Management Systems (MS).
Beyond providing tailored solutions to address their unique operational needs, these CI–O MSs integrate applicable U.S. Executive, National, Federal, DOD, and Command–specific requirements, as well as proven and globally recognized methods such as ISO and CMMI, to manage, monitor baseline and target profiles, and continually improve and optimize all aspects of the service provider. These custom tailored and made to measure CI–O MSs integrate self–evaluations, spot assessments, inspections, artifact libraries, training, and several other benefits to meet the unique mission and operational environments of CSSPs both now and in the future.
Following successful implementations, the U.S. Army C5ISR Center CSSP achieved ISO 9001 Quality Management System and ISO 22301 Business Continuity Management System certifications. Similarly, the USTC CSSP obtained ISO 9001 Quality Management System certification status. These high levels of service management delivery have not been attained by any other DOD or Federal CSSP.
Way Forward
The continuous evolution of the CSSP Program ensures its position at the forefront of defending DOD data and information networks in an ever–changing cyber threat landscape. Since its inception, the DOD ESM has been at the heart of the DOD CSSP Program. Its continuous advancement and progression, whether positive or negative, underscores the dynamic nature of cybersecurity. Each ESM iteration has responded to changes in cybersecurity operators, technology, processes, services, subscribers, and unique organizational operational requirements. As DOD CSSPs navigate the demands of ESM v11 and future versions, initiatives like the CI–O MS offer a pathway not only to meet regulatory benchmarks but also to excel in service delivery and cybersecurity management. Maximizing the range of todays and tomorrow’s DCO and DODIN Ops capabilities and investments, the CI–O MS ensures that cybersecurity technologies and practices are fully adopted from the outset. Successful implementation of CI–O MS has empowered selected DOD CSSPs to not only meet ESM v11 requirements but also to excel in the management of operators, technologies, processes, services, and defended subscribers. 
About Mr. Tyson Gee
A veteran of the U.S Air Force, Mr. Tyson Gee serves as CSIOS Corporation’s Chief Compliance Officer (COO). Mr. Gee holds a Master of Professional Studies in Cybersecurity Policy and Risk Analysis from Utica College and a Master of Arts in Public Policy from Liberty University. Mr. Gee also holds multiple advanced industry certifications including Certified in the Governance of Enterprise IT (CGEIT), Certified Information Security Manager (CISM), Certified Information Systems Security Professional (CISSP), Certified Cloud Security Professional (CCSP), Global Information Assurance Certification (GIAC) Strategic Planning, Policy, and Leadership (GSTRT), and GIAC Law of Data Security & Investigations (GLEG).
About Mr. Roston Fyle
A recognized Subject Matter Expert (SME) within the DOD CSSP Community. Mr. Fyle serves as a Senior Cyber Strategist for CSIOS supporting corporate strategic initiatives and CSSP customers. He previously served as the Lead Technical SME on the Defense Intelligence Agency’s (DIA) Cybersecurity Defense Assessment Team where he assessed over 60 U.S. Intelligence Agency, DOD, and U.S. Federal Partner CSSPs. He received numerous letters of accommodation from CSSPs for outstanding services provided and was regarded as one of the most influential assessors in the areas of Detect and Respond. He was a significant contributor to the development of ESM v8 and v.9. Prior, he provided strategic direction and oversight leading the DIA’s Computer Network Defense Center’s (DCNDC) Incident Response Team.
Leave a Comment