Case Study 2: CREST Accreditation and the Evolution of the OWASP Testing Guide

Background

The quality of penetration testing varies enormously. A 2019 study by Bishop Fox found that the median penetration test report contained fewer than 10 findings, and many testers relied primarily on automated tools without significant manual testing. In response, the industry has developed accreditation bodies and testing guides that attempt to standardize quality. Two of the most influential are CREST (the Council of Registered Ethical Security Testers) and the OWASP Testing Guide.

This case study examines how CREST accreditation has shaped the penetration testing industry and how the OWASP Testing Guide has evolved from a niche reference into the definitive standard for web application security testing.

Part 1: The CREST Accreditation Journey

Origins and Purpose

CREST was founded in 2006 in the UK, emerging from the recognition that the penetration testing market lacked quality assurance. Anyone could call themselves a penetration tester, and clients had no reliable way to distinguish between competent testers and those who simply ran automated scans and repackaged the output.

The founding organizations --- major UK security consultancies and government bodies --- recognized three needs: 1. A way to certify individual tester competence through practical examinations 2. A way to accredit companies that demonstrate robust testing processes 3. A standard that regulators and procurement teams could reference

The Certification Hierarchy

CREST developed a hierarchy of certifications that maps to career progression:

CPSA (Practitioner Security Analyst): The entry-level certification tests foundational knowledge of networking, operating systems, web technologies, and security concepts. The exam is written (not practical), but it establishes baseline knowledge. CPSA is a prerequisite for CRT.

CRT (Registered Penetration Tester): The first practical certification. Candidates must demonstrate the ability to: - Enumerate and scan network targets - Identify and exploit vulnerabilities in common services - Test web applications for standard vulnerability classes - Document findings professionally

The CRT exam is a practical assessment conducted in a controlled environment over several hours. Candidates must compromise systems and write findings. The pass rate is estimated at 40-60%, reflecting the genuine difficulty of the assessment.

CCT (Certified Tester) --- Infrastructure and Application: The advanced certifications that demonstrate senior-level competence. CCT exams are significantly harder than CRT and are divided into two specializations:

• CCT Inf (Infrastructure): Advanced network and infrastructure exploitation, including Active Directory attacks, pivoting, privilege escalation chains, and post-exploitation
• CCT App (Application): Advanced web application testing, including complex injection attacks, authentication bypass, business logic flaws, and modern framework vulnerabilities

CCT exams take a full day and require both technical exploitation and professional report writing. The pass rate is estimated at 20-40%. Holding CCT qualifies individuals for CHECK Team Member status (with additional NCSC assessment for CHECK Team Leader).

CCSAS/CCSAM (Simulated Attack Specialist/Manager): The highest-level CREST certifications for red team operations. These qualify individuals for CBEST and TIBER-EU threat-led testing engagements.

A Tester's CREST Journey: Sarah's Path

To illustrate the CREST accreditation process, consider the journey of a fictional tester, Sarah, who aims to become a CHECK Team Leader:

Year 1-2: Building Foundations Sarah works as a junior security analyst at a CREST-accredited company. She studies for and passes the CPSA exam, demonstrating foundational knowledge. She begins assisting on penetration tests under supervision.

Year 2-3: CRT Preparation and Exam With two years of experience and her CPSA certification, Sarah begins preparing for the CRT exam. She practices on HTB machines, takes the OffSec PEN-200 course, and works through CREST's published exam preparation materials. After three months of intensive preparation, she attempts the CRT exam.

The exam environment presents her with a network of systems to compromise within a time limit. She must identify vulnerabilities, exploit them, and write professional findings. She passes on her second attempt (having failed the first due to spending too much time on one machine and not completing enough findings --- the "rabbit hole" pitfall from real engagements).

Year 3-5: Building CCT-Level Skills As a CRT-certified tester, Sarah leads standard engagements and begins developing advanced skills. She focuses on Active Directory attacks and web application security, working toward CCT certification in both infrastructure and application tracks.

Year 5-6: CCT Exams Sarah attempts the CCT Inf exam. The exam is significantly harder than CRT: the systems are more complex, the vulnerabilities require chaining, and the report expectations are higher. She must demonstrate the ability to compromise an enterprise-like environment and produce findings that would meet CREST reporting standards.

She passes CCT Inf after one attempt and CCT App after two attempts (the application exam includes business logic testing scenarios that required more experience than she initially had).

Year 6-7: CHECK Accreditation With both CCT certifications, Sarah applies for CHECK Team Leader status. This requires an additional assessment by the NCSC, including review of her professional experience and a vetting process for security clearance. Once approved, she can lead penetration tests of UK government systems.

Impact on Industry Quality

CREST accreditation has measurably improved the quality of penetration testing in markets where it is adopted:

Procurement Standards: UK government, financial institutions, and many large enterprises now require CREST-accredited firms for penetration testing. This has raised the barrier to entry and ensured that testing is conducted by demonstrated professionals.

Report Quality: CREST's reporting standards require specific finding components (description, impact, evidence, remediation) and are evaluated during company accreditation. This has driven consistency across the industry.

International Recognition: CREST has expanded beyond the UK to Australia, Hong Kong, Singapore, and other markets. The CREST STAR accreditation for adversary simulation services has become the standard for CBEST and TIBER-EU testing.

Challenges and Criticism: CREST is not without critics: - Cost barrier: Exam fees and the time investment required for preparation can be prohibitive, particularly for testers in developing markets - Update pace: Some argue that CREST exam syllabi do not keep up with the pace of new attack techniques - Market concentration: CREST accreditation creates a barrier to entry that can concentrate the market among a few large firms - Geographic limitation: Despite international expansion, CREST is not widely required in the US market, where OSCP and GPEN are more common

Part 2: The OWASP Testing Guide Evolution

Version 1-2: Establishing the Foundation (2004-2007)

The OWASP Testing Guide began as a community effort to standardize web application security testing. Version 1 (2004) was relatively brief, covering basic injection testing, authentication attacks, and session management vulnerabilities. It was one of the first attempts to create a systematic testing methodology for web applications.

Version 2 (2007) expanded significantly, introducing the test case structure that would become the guide's hallmark: each test case with a summary, test objectives, how to test, and remediation guidance. This version aligned with the OWASP Top 10 (2007 edition) and established OWASP testing as a recognized methodology.

Version 3: Maturity (2008-2013)

Version 3 represented a major leap in comprehensiveness. It introduced: - A formal testing framework with phases (passive testing, active testing) - Detailed test cases organized by category - Specific tool recommendations for each test - Integration with the OWASP Application Security Verification Standard (ASVS)

Version 3 became the reference standard for web application penetration testing and was cited in numerous compliance frameworks, including PCI DSS supplemental guidance.

Version 4.0-4.2: The Modern Guide (2014-2020)

Version 4.0 (published 2014, updated to 4.2) brought the guide into the modern web application era:

New Test Cases: - Client-side testing (DOM-based XSS, HTML5 security) - Business logic testing (critical addition --- automated tools cannot test business logic) - Web services testing (SOAP, REST) - API-specific test cases

Improved Structure: The guide organized 91 test cases into 11 categories:

Tool-Agnostic Approach: Version 4 deliberately described techniques rather than prescribing specific tools, ensuring the guide remained relevant as tools evolved. This was a significant improvement over earlier versions that referenced specific tool versions.

Version 5: The Future (In Development)

OWASP Testing Guide version 5, currently in development, addresses emerging testing needs:

• API-first testing: Expanded coverage of REST, GraphQL, and gRPC security testing
• Modern authentication: OAuth 2.0, OpenID Connect, FIDO2/WebAuthn testing
• Cloud-native applications: Serverless, container, and microservice-specific test cases
• Single-page applications: Testing frameworks (React, Angular, Vue) and their security implications
• CI/CD integration: How to integrate testing guide test cases into automated security pipelines

Impact on the Industry

The OWASP Testing Guide has had a profound impact on penetration testing practice:

Standardization: Before the Testing Guide, web application testing was ad hoc. Each tester had their own approach, and test coverage was inconsistent. The guide provided a common language and a shared understanding of what "thorough web application testing" means.

Compliance Recognition: PCI DSS, SOC 2, and ISO 27001 auditors routinely accept OWASP Testing Guide coverage as evidence of thorough application testing. The guide's structured approach maps cleanly to compliance evidence requirements.

Training and Education: The Testing Guide became the de facto curriculum for web application security training. SANS, OSCP, and CREST exam syllabi all reference OWASP test cases. This created a common knowledge base across the industry.

Quality Benchmark: Clients can now request that penetration testers demonstrate OWASP Testing Guide coverage, providing a measurable standard for test completeness. Reports that reference specific OWASP test case IDs (e.g., "OTG-INPVAL-005: SQL Injection") communicate precisely what was tested.

Using the Testing Guide in Practice

For practical application of the OWASP Testing Guide on an engagement:

Pre-Testing: 1. Review the application's technology stack and identify relevant test categories 2. Create a testing checklist from the applicable test cases 3. Estimate time for each test case based on application complexity 4. Identify test cases that require specific access levels or test data

During Testing: 1. Work through test cases systematically, category by category 2. Document each test case as: Tested (Pass), Tested (Fail/Finding), Not Applicable, Not Tested 3. For failed test cases, document findings using the standard finding template 4. Cross-reference between categories (e.g., an authentication bypass may enable authorization tests that were previously impossible)

Post-Testing: 1. Create a coverage matrix showing which test cases were executed 2. Document any test cases that were not executed and the reason why 3. Map findings to OWASP Top 10 categories for executive reporting 4. Reference specific test case IDs in the report for traceability

Connecting CREST and OWASP

CREST and the OWASP Testing Guide are complementary rather than competing:

• CREST certifies who can test and ensures they meet quality standards
• OWASP defines what to test and provides a systematic methodology
• CREST CCT App exam content aligns closely with OWASP Testing Guide test cases
• CREST-accredited companies typically reference the OWASP Testing Guide in their methodologies
• Both are evolving: CREST expanding internationally and OWASP expanding to cover modern application architectures

Together, they represent the industry's best answer to the question of penetration testing quality: qualified testers (CREST) following a comprehensive methodology (OWASP) to deliver consistent, thorough, and defensible results.

Discussion Questions

1. Should CREST accreditation be mandatory for all commercial penetration testing, or does this create an unnecessary barrier to entry? Argue both sides.

2. The OWASP Testing Guide contains over 90 test cases. A typical web application pentest might be scoped for 5-10 days. How would you prioritize which test cases to execute when time does not allow complete coverage?

3. Compare the CREST certification path to the OSCP/OffSec certification path. Which better prepares testers for real-world engagements? What does each path emphasize that the other does not?

4. How should the OWASP Testing Guide version 5 address the rise of API-first architectures and single-page applications? What new test cases are needed?

5. CREST company accreditation requires documented methodologies, QA processes, and reporting standards. If you were starting a penetration testing firm, how would you design your internal methodology to meet CREST accreditation requirements while remaining practical and efficient?