Crowdsourced security programs have grown in popularity to the point where some enterprises have dispensed with traditional pentesting, using the crowdsourced model exclusively for auditing the security of their applications and infrastructure.
What is Crowdsourced Security?
Crowdsourced security methodologies invite a group of people (a crowd) to test an asset for vulnerabilities. The number of people can range from less than a dozen to several hundred testing concurrently. ‘Bug bounties’, ‘vulnerability disclosure programs’, and ‘responsible disclosure programs’ all fall under the umbrella of crowdsourced security.
Penetration Testing – the Weaknesses
‘Pentesting’ as it’s more commonly known, has been part of the security landscape for some time now, and companies use it to audit their assets from an attacker’s point of view. Unfortunately, it has many inherent weaknesses that aren’t immediately obvious.
The Speed of Development Today
Many companies conduct pentesting annually. The pentester will assess a website for vulnerabilities, hand in a report, and draw up a remediation plan to fix any issues. The following year, the process is repeated. This would be acceptable if the website was updated annually, but today, many websites and applications are updated as often as several times per day. A pentest provides a snapshot of a security posture at a particular point in time, nothing more. As soon as a new version is deployed, the pentest findings are obsolete and new vulnerabilities are quite possibly in play.
They Are Time Limited
Even when leveraging automatic tools, pentesters don’t have the luxury of time. Taking a mid-size e-commerce site as an example, a pentester might have four to five days to test with an additional one or two days to draft the report. During the test, if a potential vulnerability is found, there is limited time to prove its existence. This means ‘low-hanging fruit’ will be discovered while vulnerabilities that are more time consuming to find (but just as serious) remain. This doesn’t mimic real-life where attackers have as long as they like to discover and exploit vulnerabilities.
Skillset vs Technology Stack
Skillsets across the pentester community can vary. Some specialize in websites while others may be better at testing infrastructure. Due to high demand and a tight pentester labor market, the pentester testing a website this year may not specialize in the technology stack used to develop it. This leads to certain vulnerabilities going undiscovered, which is completely normal. Often, a different pentester audits the site the following year and will spot a vulnerability that has been present but undetected for years. This has led to the practice of ‘cycling’ pentesting companies every few years to have a ‘fresh’ set of eyes each time.
Pentester Syndrome – Making Things Appear Worse Than They Are
Finally, there’s ‘pentester syndrome.’ Pentester syndrome is the act of making things appear worse than they actually are. Pentesting companies are under pressure to distinguish themselves from each other, and this rivalry often manifests as ‘look how many vulnerabilities we found’. On engagements where the asset tested reveals no significant vulnerabilities, the pentester is left in the awkward spot of having to demonstrate value added in an environment where the only customer metric is the number of vulnerabilities found. This leads to informational issues, such as missing HTTP headers, finding their way onto the list of ‘vulnerabilities’ with low and medium severity rankings.
Enter Crowdsourced Security
So does crowdsourced security actually improve on the weaknesses in contemporary pentesting?
Speed of Development & Time Limited Tests
Many crowdsourced programs are effectively ‘open-ended’ with no time limit; effectively translating to a ‘constant pentest,’ but only if the incentives are right. ‘Researchers’ (the term often used for pentesters in a crowdsourced environment) are paid ‘per vulnerability’. If the rewards are high enough, this can ensure a constant watchful eye over all versions of the site and application, no matter how often it’s updated.
An Individual vs the Crowd
This is by far the greatest advantage. Crowdsourced engagements have been successful in identifying critical vulnerabilities in the most visited sites today that previously had relied only on pentesting. The reasons are clear – the more people you have looking at something, the more likely you are to find it. Because of the wide mix in technologies in use today, the crowd acts as a big equalizer in this field, ensuring that you will eventually get someone looking at your site that has experience in a specific flaw that would have been missed by someone else.
Pentester Syndrome
Crowdsourced security programs increase the burden of proof on researchers and those submitting vulnerabilities, which effectively eliminates ‘pentester syndrome’. The ‘pay per vulnerability’ model of most crowdsourced programs means only vulnerabilities with impact will be rewarded. This means all the previous ‘hardening’ information one typically finds on a pentesting report is now ignored, since this has little to no impact on a site’s overall security posture.
The Downsides of Crowdsourced Security
For Now, External Assets Only
Due to the nature of the crowd, it is understandably more difficult to have your internal network pentested. Whereas a pentester physically enters your premises, internal assets don’t lend themselves to being tested by people outside of the organization as this would involve granting secure access, such as through a virtual private network (VPN), authenticated proxies, or even a model test environment. These take time and resources that companies may not be willing to expend. While the author has personally participated in many successful crowdsourced ‘internal’ pentests, the majority still remain focused on externally facing assets.
Controlling the Crowd and the Performance Hit
On many crowdsourced engagements, as soon as a ‘target’ is made available, the website in question slows to a crawl and then crashes. This is an important downside to crowdsourced testing. While certain tests allow the invitation of a precise number of researchers to control who is testing, many are open to ‘all researchers’. Due to the nature of pentesting (many people used automatic scripts or vulnerability scanners) this can significantly impact the performance of the asset being tested.
If Incentives Are Poor, the Crowd Will Stay Away
As previously mentioned, crowdsourced security differs in the way it pays the tester. In pentesting, you pay a company a daily rate based on the number of days the pentester is active. If no vulnerabilities are discovered, they are still paid. Crowdsourced pentests pay out ‘per vulnerability’ and that rate is usually dependent on how ‘impactful’ it is. If a remote code execution is found, it warrants a higher payout than a cross-site scripting vulnerability. This makes budget estimation difficult since there’s no way to know what is going to be found ahead of time. If the site has never had any kind of pentest and is opened to crowdsourced testing, the payout will be high being that many vulnerabilities will be found. When incentives are minor or insignificant (e.g., ‘hall of fame’ mentions or t-shirts) it’s unreasonable to expect the best and brightest researchers to flock to the site; they’ll be focusing on companies that pay out monetary rewards.
Can I replace my pentesting program with a crowdsourced security program? Can I use both?
Organizations that have never undergone either a pentest or any kind of crowdsourced program should start with a pentest. This will identify the most common vulnerabilities present in the site and provide remediation experience. After a few pentest cycles, organizations can switch to a crowdsourced security model, as its budget impacts will have been evaluated and the researchers can start picking at the less obvious vulnerabilities. If the site or application changes often, the organization should switch to a crowdsourcing model, as this obviates the time-based weakness of pentesting programs. If the internal network must be tested and the organization lacks the time or resources to set up remote access, it should use a pentester. In the long term, with the shift to cloud infrastructure, this will become less of an issue. If the organization is confident that it controls all of the above, it can switch entirely to crowdsourced pentesting or run both in parallel.
Alex Haynes
Leave a Comment