Penetration testing and low-cost freelancing | Hacker News Reader

Penetration testing and low-cost freelancing

sophron.github.io

152 pointssophron71 comments

The problem that strikes me here (as a professional pentester) is that these vulnerabilities are so pathological. There is no realistic series of errors that would lead you to really design an app with hardcoded sqli looking credentials or somehow return a cookie with a malformed header, let alone one that somehow automatically authenticated the user. I am extremely onboard with the idea that pentesting as a whole has a lot of scanner jockeys (and a lot of my current work is finding people who aren't.) But you're not testing for vulnerabilities that would actually show up in the real world, and it is hurting your analysis.

I suspect these would be realistic vulnerabilities introduced by freelance or inexperienced devs hired by a startup, for example.

They might have hardcoded admin-level passwords for debugging, then forgotten about it.

They might have mis-typed HTTP headers like 'set-cookie', after having written manually a lot of auth & session management that should really be done using well-vetted libraries instead.

Respectfully, you are incorrect. Delegating session and cookie management to the framework or (in php's case) the language is so much simpler that I have seen manual implementations of this behavior only 2-3 times in my career. And the idea that logging in incorrectly would somehow return a malformed, but correct, session-setting header is again, pathological. Most of the time, these types of profound errors come from simply taking the path of least resistance provided in the framework. Sometimes they do come from a complex homebrewed solution to a simple problem.

Back at my startup days, I once worked with a poor guy who was dynamically generating individual IDs for each element on a page and their corresponding CSS for each page. He was unaware that CSS also had _classes_, which perfectly encapsulated the behavior he was trying to create. His work was complex, and full of errors, but it had a certain logic to it- it was the path of least resistance that he saw available to him.

These don't look like that. They just look like weird errors designed to avoid showing up on, or triggering false positives on, DAST tools. I thinking testing people's skills without those tools is a good goal, i just think the methodology here is wrong.

> once worked with a poor guy who was dynamically generating individual IDs for each element on a page and their corresponding CSS for each page

That looks like what modern bigcorp webapps do, with random looking css class names that change every release.

I can see both of your viewpoints, here. I wouldn't make the blanket statement that OP is incorrect. The fact that you have only seen this a handful of times in your career is not surprising. This sort of silly bullshit is far less common nowadays. However, as OP has stated, these things were somewhat more common a while back. Mind you, much of this very poorly written software is still being used in dusty corners by large companies. You should keep an open mind when testing and not dismiss these things are outright impossible or else you're going to miss a lot of bugs :P

eh, ive seen pros ("pros") do eerily similar thimgs in production code

Some of the most interesting issues seem pathological at first blush. Can you really think of no scenario where there would be an sql injection string as a password? Perhaps some try-hard came along before you and attempted sql injection on the user creation form that resulted not in sql injection, but in the password being set to the literal string that the person used as input.

I realize that in this scenario it is literally designed-in, but I understand the point the author is trying to prove. If a "scanner jokey" gets results that tells him or her there is SQL injection, a competent tester will try to verify what their tool is telling them. If the tester is doing that in this case, they'll find other injection strings not working and (hopefully) start looking under the hood to see what is going on and discover this pathological hard-coded pw and be able to tell the client that. Maybe it's the work of a malicious dev?

I agree that a pentester with a lot of experience will have skills that are honed to find common bug patterns, but it's nice to be able to find these seemingly bizarre issues and have an explanation for the client. It shows you really understood the app and what it's doing.

Given this is your area of expertise, I'm genuinely interested in how/why you believe that these vulnerabilities wouldn't show up in the real world. Generally speaking, isn't the software developer community littered with developers who aren't adequately skilled/copy-paste-from-Stack-Overflow, etc?

As a thought experiment, I think the best course of action for these freelance pen testers - assuming they incur no actual liabilities for being wrong - is to simply declare everything they test "secure" without putting any actual effort into it at all.

If they can manage to do this at any scale whatsoever, there is basically no downside, because there's a huge disconnect between the impact of a false negative (site breached, data stolen, etc.) and the impact to them (a 1-star review - even in your nightmare scenario as a customer, you still give them a score of 20%!)

1) A large number of the people who come to them will have actually produced secure code, so broken clock theory prevails, and there's no downside.

2) Another large subset will never generate enough interest among hackers to exploit their insecure code; a tree falls in the forest and no one hears it.

3) Another subset still will never know they've been breached and have cause to action.

4) If a company does learn they've been breached, they still must connect the dots to the failure of the pen tester.

5) And even then ... 1 star and some bad PR!

And of course if there were actual liabilities, they wouldn't be charging $35 on a freelance site.

Counterpoint: if they find _nothing_ wrong, then whoever hired them has potentially failed to justify their job. Better to find minor, easily fixed, bugs I'd think.

You can't expect a manual pentest for €35 or anything near that ballpark.

A proper pentest includes manual labour (such as the mentioned examples). It also includes a lot of documentation in the report. For example, recon data. They could also include we checked X for Y but did not found any issues. Nothing is 100% secure, and if you don't find anything after extensive manual testing it is, well, going to be a boring report, because you still have to document everything. It also feels like a failure (which is why the honeypot was a little bit mean, though clever as well given goal of post).

CEH is a joke btw. No serious pentester puts that on their resume. Its a waste of time to get thst certificate. OSCP is the gold standard.

In short, you pay peanuts you get monkeys and CEH is a red flag. OSCP is a green flag.

People are paying pentesters because their payer’s policy asks them to. Of course there will be a huge market for someone with an alleged certification to run automated tools, the value is in the box checking and report generation, not the bespoke broken website fixing.

Obviously this is a bit of a reaching statement, some organizations understand the legal, career, and regulatory risk and proactively do it.

In every org I've sat down with the head to explain those risks, I can see how it can be made priority and budget.

will it survive budget review though?

Usually does - if I can get in front of them.

Edit: Followup - here is a primer method. The exact method differs if you are talking with leaders, or middle management, or space occupiers. (Leaders as in intelligent, aggressive forward thinkers). You have to make it personal, and show them how the thing that they care most about (eg: career, personal security, promotion, or liability exposure) is at risk; which now-a-days is a true statement.

Can we collectively refrain from making comments like this on Hacker News—or in any venue, for that matter? Is it that hard to read an assumed “In my experience, it is often the case that…” in front of everyone’s comment?

I was stating that I have great success showing how said items should be prioritized and budgeted, across numerous organization types, sizes, and industries - and making it happen.

That is my experience, it may not be others.

Let me demoralise you. There are industries [cough] where (for a good number of jurisdictions) you MUST have the pentests and other audit-like engagements conducted by a specially approved operator. Think of a royal charter.

These operators have no incentive to invest in proper skills. So they don't. They can charge almost anything they like. And sure enough, they do.

In an industry with such massive vendor lock-ins and regulatory capture regimes, organisationally you get to shovel a lot of money for sub-par pentests. The appetite to have yet another one, even from a known-good and reasonably priced provider, is quite hard to come by. After all, these approved providers will REFUSE to even look at a third-party pentest report, let alone let one guide their own testing.

So in aggregate the existence of these approved providers and their level of competence degrades the security posture of entire business domains.

Macchiavelli would have been proud.

Absolutely agree.

This annoys the hell out of me. The whole security theater /checkbox checking stuff is depressing.

You should be satisfied with the results of a pen test if you can get a real hacker to do it for you. Finding real white hat hackers is hard, but they are out there.

Most 'security guys' are not hackers. They do not have CS degrees and are not curious at all. Rather, they are 8 to 5 office workers that are compliance and risk oriented. They run automated vulnerability scanners (that are largely useless and filled with false positives based on the version string of Apache). They submit reports to management where they discuss how to make the 'red' things 'green' to impress the Board.

Real hackers are very curious about technology and like to see how they can break things or make things do something that they were not intended to do. That's how they find bugs.

Two totally different worlds. One is focused on 'compliance and risk and loss prevention insurance policies' (basically CYA) while the other is focused on actually being technically secure by trying to break things before the bad hacker comes along and does so.

I agree with all of that, except for the part about a CS degree. When I was in the military, a General was talking to us about the officers that they had doing pentesting, and they said that the people with CS and other tech degrees were usually outperformed by the people with degrees in things like political science. From their experience, the people with technical degrees would often get hung up on a problem if it didn't have a provably right/wrong answer, while the people with degrees in "soft" subjects were able to think through a problem and provide a "best" answer.

CS/EE degrees are optional. Many hackers do not have them. It's just an indicator that they probably know the basics (bits, bytes, systems, etc.) but certainly not required. Some of the best hackers I have worked with had dropped out of (very difficult CS/EE programs) and never finished college.

The only thing this article proves yet again is: if you pay peanuts you get monkeys. I mean, 100 dollar is maybe enough for 2 hours of a freelancers time. But I do not think that is enough time to find both issues and scan the server.

I agree with you - yet there is also a philosophy that some organizations take which is to intentionally hire quite subpar 'certified' practitioners who themselves can be duped or walked past issues to provide 'clean reports' to upper management, even if false.

This is not always true. I’ve seen a ton of costly engagements produce super low quality deliverables that yielded no real value except for a checkbox being filled.

Right, but the cheap ones are almost guaranteed to be bad.

So how would someone know they have connected to a honeypot? Presumably detecting this sort of thing is deliberately difficult, otherwise there isn't much point of the honeypot?

Some SSH honeypots accept more than one password for root. So, if 'letmein' and 'changeme' get you a root shell, it's probably a honeypot... but it could be a very broken PAM config as well (I have seen this firsthand).

PAM is one utter freaking nightmare to set up if one wants to stray from the distro defaults. Actually I'm astonished no one has dared to try and develop an alternative... god knows it's about time.

There are alternatives. Last time I checked OpenBSD uses BSD_Auth; not PAM.

"While the cost of penetration testing can be pretty high (typically between $1,000 and $100,000+),"

I've always been curious about costs for real pentesting. 1-100k is a HUGE range, that can't mean the range in cost for the same project can be that big, can it? What's it cost to hire someone to run a decent set of tests on a Rails app, for instance? Could I really see that big of a range? Would I get way better results for for $100k?

You will find exponentially less vulnerabilities the more money you spend. How much you should spend depends on how big the damage could be in case of a breach.

"A rails app" could be a rails default build, or Github.

You should get significantly better results the more one pays - because of the human element.

For example, $1k gets you a basic Nessus/Nmap (basic vulnerability/mapping) scans, maybe SQL vuln, with automated reporting. More gets you architecture, infrastructure, penetration, white/black scenarios, risk assessment, external organization liability, code repo, data, compliance, and regulatory footing. Those can surpass $100k easily.

This isn't really comparing the results to other pentests, so I have a hard time seeing the use of this. I think almost any pentest will miss some/many security issues.

And these issues look more like CTF style issues than real world security issues. If you're not thinking about the challenge in the right way, it is probably harder to see the vulnerabilities.

> What is surprising is the number of people purchasing these security services as well as the number of positive reviews.

Because for most people its not about security, its about liability and having a proven attempt at trying to "security" implies in their minds that they are not liable.

Sadly security is a mindset and a process like healthy eating but we socially model it as a eating a single apple which translates to hiring one person for less than $100 and taking what they say as authority.

Now a real interesting article would be what appsec service would have the highest impact for $100? Obviously not pentest. Code review? Architecture analysis? At $100 none of it would be deep of course. I could find a lot more stuff of interest in 2 hours of code review compared to pentesting.

For those prices (except maybe the 400$) I don't think anyone is going to perform manual actions or invest more than a couple of hours.

Number 3 wil surprise you!

Thank you for this, needed a chuckle this morning.

"Penetration Testing and Low-Cost Freelancing" would be a much better title and less clickbait-y.

if anything, the number 5 should be the surprising one

I have VMs specifically for running garbage that I find/need to but I don't trust. These have 0 personal information and gets wiped to a clean state afterwards.

Is this pen testing or a puzzle?

I can kind of get behind the malformed header. But I have very hard time thinking up any suite of actions that would lead to a code resulting in leaked obfuscated query with a hard coded password. All while password somehow using a LIKE statement matching a sql injection.

Is there any precedent for this?

Edit: rewrote my sentence because it was barely readable.

Not-invented-here (NIH) coders end up worrying about this kind of situation. People who develop "creative" systems by working ground-up starting with foundational system logic. They create proprietary systems really fast, and those proprietary systems often do some wackadoodle things in the name of internal logic.

As a result, those systems are typically harder to maintain or secure by just anybody, even by some experts who are not from the NIH mindset (here's where the author's approach isn't really fair, IMO). Proprietary systems are are less likely to present the typical bell curve security issues that an automated scanner would pick up.

What they are more vulnerable to is puzzle logic--the "what's going on here, hmm let's have a tinker" approach. Which is exactly what the author presented and it also matches his broader approach of "let's have a tinker...with freelancers".

It's a subjective security vs. objective security approach. Both sides are important. But both sides should know about how the other side works.

> What is surprising is the number of people purchasing these security services as well as the number of positive reviews. An incomplete security testing methodology results in a false sense of security for the assessed systems.

The same is happening in the "smart contract audit" space. The results of the audit do not matter to the audience that wants to see the checkbox of "audit".

You can currently make a lot of money undercutting other auditors providing these audits.

The person requesting the audit is making multiple orders of magnitude more money by having the audit.

What they didn't test is whether higher cost services would find the vulnerabilities.

This sort of thing happens even for high-end pentesting. Here is the same assessment done by four decent consulting companies. They all found risks that the other companies missed.

https://ostif.org/four-audits-of-randomx-for-monero-and-arwe...

This is what I'm giving to clients who have unreasonable expectation that all vulns should be found during an assessment. The usually "time boxed" nature of this sort of work does allows for 1.5 sigma when many companies always expect 3 sigma coverage.

You get what you pay for. Open ended work is hard.

And I wonder if it’s good business to be kinda bad for companies that are required to get a pen test or want to put “passed security audit” on their website.

I'm trying to imagine being a customer for this kind of service. When would I be satisfied with a pen testing service?

If I'm not familiar with pentesting methodologies, my only metric of satisfaction would be the pen tester's reputation i.e., if a well-reviewed pentester says my site is OK, then maybe my site is safe.

I work as a professional pentester. This question is difficult for both sides.

The business often gets in my way with budgets. I can spend it all on some strange behavior that I detect in the application. I like to spend my time like that. The more experience I get the more time I could spend, even on simple applications. The rabbit hole goes ever deeper.

When am I satisfied that I delivered good work? When I find a number of high impact vulnerabilities. Doesn't mean there aren't still more to be found, like I said you could go on and on, but it does mean that I fixed some bad. Sadly I have a fixed budget, and sometimes I find nothing, those are bad days.

In my experience customers are also satisfied with a report that contains few or no impactful findings. But an empty report is no guarantee that there really isn't anything to be found.

It's difficult to balance the desire to dive deep and to provide broad coverage. I once got bit by that.

I found and clearly documented no less than 3 absolutely critical issues (price adjustment in an e-commerce website). Felt pretty good about that test. Until, in production, someone else found an additional authentication bypass. Oops. I wasn't focused on that due the aforementioned input validation issues and got blinded by trying to find more of those. It's not easy.

This comes down to the fact that penetration testing has a high false-negative rate. There's no way for a customer to really evaluate what a mostly empty report means, if there actually are no bugs, or the penetration tester missed them/did a poor job.

It's just like going to a doctor to find if you have a disease. If they make a mistake, we have no way to find out.

This is a common problem when hiring supposed experts. Unless the one doing the hiring is reasonably skilled in the area being hired for, it is very difficult to judge the skill and quality of the candidate.

This statement ignores the prices. Just look at the prices and you'll see there's no way anyone thinks they're paying for an expert.

There are unfortunately many small business people who greatly underestimate the skill that goes into anything they don't personally do. Those are the foolish people who shop by price and think they are getting a good deal.

These are also the people who keep Walmart alive. They have limited reasoning ability, and even if they might suspect it's too cheap to be any good, they want it to be good enough and are willing to pretend that it is. These happen to also be the people who easily accept absurd and obviously false statements made by certain political leaders.

I know quite a few of these people, and I have tried countless times to advise them on reasonable choices, but it is like talking to a wall (if a wall could nod and pretend to agree).

I am curious how common would it be to get actual SQL (obfuscated or not) in http response.

How in the world could that happen?

Edit: I guess it could happen if a developer decided to debug SQL via http response which seems pretty insane

> How in the world could that happen?

Broken exception handling, for one.

Interesting test, but too unrealistic bugs.

Undergrad pen-test homework assignment that someone paid actual money for. Most people got C- or worse, one guy got maybe a B-.

The "reports" are absolutely hilarious. Even better, the blog post "analysis" is worse than the pen-test reports.

I’m curious which freelancing sites you got these from

I may be wrong, but looks like fiverr

Which platforms have you used to hire them?

This poor research project represents the state of the art out there in the wild.

> Some of them claim to possess industry-accepted certificates

I'll trust them on that claim. But is my $100 getting me an industry-certified analysis?

I could hire a licensed MD Engineer Pharmacist Cosmetologist Veterinarian to walk my dog for me, but I don't expect them to perform heart surgery for $100

Clickbait title. I refuse to click.

The title reflects the content. He hired 7 pentesters and shows the results. Why comment if you don't even want to read the article?

The title of the HN submission was changed. It used to be the article's subtitle, "How I Hired 7 Freelancers to Exploit this Weird Vulnerability".

"This one weird X" is a common clickbait pattern, and indeed, in this case, it is misleading: the article's descriptions of the purposely-introduced "weird" vulnerabilities aren't useful, only the freelance test results are.

trolling comment, I refuse to respond.

The article headline is now "Penetration Testing and Low-Cost Freelancing"

Reply on news.ycombinator.com