Don’t Be Fooled! There’s No Such Thing as an Automated Penetration Test.

October 22, 2012 • Published Categories PCI 101 Tags , , ,

Looking for a Penetration Test Vendor?

Many small merchants, having been told they need a “network penetration test,” will seek out the quickest and cheapest way possible to comply with this Payment Card Industry Data Security Standard (PCI DSS) requirement.  This is certainly understandable, given most small businesses’ tight operating budgets and the growing number of companies offering inexpensive testing services—some originating from off-shore companies.

In many cases, however, the service small merchants think they are paying for is not what they’re actually getting. The PCI DSS is explicit in its requirement that a penetration test be performed, but it is somewhat vague when it comes to the methods employed to perform testing.  This can leave a merchant wondering why they would ever pay for a manual penetration test when a vendor is telling them they can get an automated one for much less.

Vulnerability Scans and Penetration Tests are Not Synonymous

In an effort to help merchants spot security vulnerabilities within their business network and applications, the PCI Security Standards Council (SSC) has put forth two requirements within the PCI DSS: one that deals with vulnerability scans (Requirement 11.2) and one that deals with penetration tests (Requirement 11.3).

If you are looking for a penetration test vendor, be sure they are offering you a true penetration test and not a vulnerability scan. The following chart outlines the important differences between a penetration test and a vulnerability scan:

Penetration Test
Running Tests:
Manually conducted by a Certified Security Professional (“Ethical Hacker”)
Automatically administered via a computer program and an Internet connection
Testing Depth:
Performs exploitation against target systems
Looks for common vulnerabilities
Interpreting Results:
Results are professionally interpreted
Results may include “inferred threats”
Creating Reports:
Reports are tailored to the organization being tested
Reports are programmatically generated
Complying with PCI:
Satisfies PCI DSS 11.3
Satisfies PCI DSS 11.2

Penetration Tests are Always Manual

A penetration test is characterized by a person at a computer behaving as a hacker would, running a series of manual, simulated attacks against your information systems. Sure, there are automated elements to penetration testing (after all, hackers are smart; they leverage automated scripts and tools to quickly and efficiently gather data), but the test is orchestrated and driven by a real human trying to break into your network and its applications. This is important because information discovered during the various phases of testing must be intelligently fed back into the testing methodology – something that computers aren’t very good at doing.

Penetration Tests Leverage Professional Experience

While automated scripts and scanners are great at efficiently identifying “low-hanging fruit,” one noticeable and important trait they lack is experience-led logic. An experienced penetration tester can quickly identify the systems, services and configurations that present possible vectors for attacks, while automated vulnerability scanners rely on a pre-compiled list of signatures, or fingerprints, in order to detect vulnerabilities and vectors of attack. In addition, an automated scan is looking for very specific things, while a human being is free to apply creativity, look at the big picture and consider past experiences and findings that may lead to the detection of issues that a scanner won’t find.

Penetration Tests Have Unique Methodology

Penetration tests simulate the very real dangers of an intelligent human being actively attacking your systems and trying to bypass your countermeasures. A vulnerability scan or other automated “attack” against a network is not the same as a penetration test because it cannot adhere to the same methodology. Therefore, many PCI Qualified Security Assessors (QSAs) will not recognize automated penetration tests (a.k.a. vulnerability scans in disguise) as valid for compliance with PCI DSS Requirement 11.3.

Penetration Tests Yield Specialized Reports

The report resulting from your test should be written by a human being, not auto-generated through a computer program. The professional penetration tester’s goal is to give your organization the detailed information it needs to successfully secure all in-scope business systems. Therefore, the written report is tailored according to your organization’s unique risks and includes the testing methodologies used as well as any issues discovered, assessment of each issue’s level of risk and recommendations for addressing those issues.

“Amateurs Have Automated Tools; Professionals Have People”

The above quote came from a security professional speaking at a recent European risk conference. This person was reinforcing the widely acknowledged understanding that true penetration tests are characterized by manual intervention and guidance, defined goals (in the case of PCI, to identify and disclose unauthorized cardholder data), structured methodology, and creativity – not clicking a button and allowing a program to run, then generating a report from the results.

Carefully review penetration test proposals to ensure that vendors trying to pass off an automated scan as a penetration test are removed from consideration. Another good rule of thumb is to request the credentials of those performing the penetration test. Be sure to look for industry-respected certifications associated with the engineers performing your test, such as CISSP (Certified Information Systems Security Professional), GWAPT (GIAC Web Application Penetration Tester), GPEN (GIAC Penetration Tester) or even C|EH (Certified Ethical Hacker).

These accomplishments denote experience and achievements in penetration testing. Also, don’t be afraid to ask questions regarding test procedures and methods employed, as well as expect updates as the testing progresses. Remaining in contact with the testing team will help provide assurances that there are real people working on your project and not just an automated scan tool.

Ready to learn more?

Want to learn more about how the PCI DSS applies to your business or small business data security in general? Give us a call at 1-800-825-3301 x 2. We’d be happy to help.

Subscribe to this blog for additional tips and webinar announcements.

Leave a Comment