Hacker News Reader

Comment by tptacek | Hacker News Reader

If this stuff fascinates you and you're a solid software developer and you'd be interested in having this be your full-time job for awhile and you're willing to sink a little bit of your own time into ramping up, give us a ping. We'll help you get there.

This page has a lot of info on how we recruit. We're getting pretty good at turning systems programmers into breakers, and we love hiring from HN:

http://www.matasano.com/careers/

The great thing about this field is that it's always changing. A long-term dev job gives you a chance to master two or three different technology stacks. Your next three projects at an appsec shop might each be just two weeks apart, and each will use radically different technologies. Even (maybe even especially) with web software.

You could join one startup... or spend a couple years beating up all of all the startups.

Also: I understand why Cody didn't write it this way, but the reality is, if you're going to test web apps, Burp is the standard tool. You can use things like mitmproxy or even WebScarab, but most people end up in Burp. Burp is also extremely valuable for testing even if you're not doing appsec full-time.

View on HN

Oh, hey! And if you'd like to learn to break crypto at the same time as you work through Cody's web recommendations --- even if you don't want to be an appsec person --- mail sean at matasano dot com. He's got a pretty kick-ass set of ~40 crypto-breaking exercises. Something like 200 people have started them over the past 6 months; only a few people have made it through the end.

(They aren't deliberately hard; they just cover a lot of ground --- you're starting with basic substitution ciphers and ending with RSA signature block forgeries).

I helped design them, and I'm really happy with how they came out. They're neat. You should see how many sets you can get through.

I hope it goes without saying that if you crush Sean's crypto challenges for fun and are interested in being a full-time appsec person, you will have our full and undivided attention. :)

As one of the 200 incompletes mentioned, I gotta say Sean Devlin at Matasano is top notch. The puzzles are fun, possibly too much fun. It's easy to wander off into the weeds and spend a lot of time thinking through and testing the non-puzzle implications. I hope to get through the puzzles eventually, but I'm terribly slow and it will take me a very long time. Even if you're a sec-idiot like me, the puzzles are a wonderful way to spend time learning.

\o/

Any idea why he's doing this stuff only primarily by e-mail? It'd be great if this stuff was online like in a blog or whatnot.

I'd like to take the time and look over the crypto challenges.

We want to actually teach people how to do stuff, instead of giving people something they can toss around in message board and twitter arguments; we also want to track (in a macro sense) how people do with them, and to be able to tell people when we add more challenges (I'm working on 42-48 next week).

This sounds like an amazing win-win scenario you've got here. People can have fun and learn more about security and cryptography, and you guys get a channel from which to hire the best and brightest.

I sent Sean an email. Even if I'm not in that latter category, it still sounds like a great chance to learn a little something about a field which intimidates but interests me.

I like that answer. \O/

Also sent an e-mail.

+1 I'm working through these now and they're the most fun thing I've done with a computer for a long time.

Another person chiming in to say that these exercises are a lot of fun and educational too. Feel like I learned more than in some online cryptography courses.

Sigh. And there goes my productivity. :)

DON'T WORK FOR MATASANO!!! Reasons below:

#1 Lack of honesty. Seriously, they promised releasing those crypto challenges publicly 2 years ago (Blackhat 11: Crypto for Pentesters) and never done so: https://twitter.com/matasano/status/101714851633700864. And now they're using them as a recruiting tool.

#2 Lack of humility: Matasano guys seem to disregard common tools like Burp scanner or Sqlmap. It's fine to cherry-pick tools to suite your needs; but if you choose to disregard them completely just because you feel they're associated with "Security Rookies" then you're more than likely to miss something, consequently disservice your client (they expect you, as a consultant, to find the most vulnerabilities regardless of tools used). Matasano may have better fuzzer/scanner, but since they don't publicly release them, I found that going around and bashing other security tools to position themselves higher than their competitors is a sign of arrogance.

Go work for other companies, I don't want you HN people turn out to be like them!

For those of you who emailed sean at matasano dot com but haven't received any response, go play with Trustwave crypto challeges: https://github.com/SpiderLabs/CryptOMG - And yes they don't have BS subscription model.

If you want to run scanners all day, we're definitely not a great place to work.

We changed the format for the crypto challenges because:

* the "vulnerable" web app got in the way of what we were trying to teach people (it's easy to work CTR mode into "decrypt this cookie" but not so easy to work Diffie Hellman into that)

* the web parts got repetitive (there's only so many times you can show people "decrypt this cookie" before the "cookie" part of that gets in the way).

* some of the challenges involve implementing crypto constructions (which we found to be the best way to learn how to break them). We had features in 36 Chambers that tried to capture "building" as well as "breaking", but they were extremely clumsy and contrived.

* But mostly, because we'd rather put effort into tech supporting people learning crypto, as opposed to Ruby code running on Heroku.

Sean's crypto stuff has about 2.5x as much material as the 36Chambers crypto-for-pentesters site had, and that's mostly because we stopped wasting our time making it look pretty. The site was a silly way to spend our time. We'll have RC4 keystream bias challenges by the end of next week. If we were going to work them into a shiny web app, we might not have them by the end of the year.

If the pricing model we use for the challenges ("mail Sean and ask for them and he'll give them to you for free") is too much for you, I don't know what to tell you. Yes, making it to the end does incur the penalty of us begging you to come work with us. But if you're unwilling to surrender to a life of indentured servitude in the pentest mines at Matasano, we will happily accept a warrant on the blood of your firstborn child. That is, I think you'll agree, a tiny price to pay.

I guess thanks for giving me a chance to clear up this issue, which, if you follow me on Twitter, has maybe had you confused for awhile (at least until last November or so when we started telling people every damn week to mail Sean for the challenges).

>"Sean's crypto stuff has about 2.5x as much material as the 36Chambers crypto-for-pentesters site had..."

This site never existed, at least according to Matasano's offical website or Twitter account.

>"If the pricing model we use for the challenges ("mail Sean and ask for them and he'll give them to you for free") is too much for you, I don't know what to tell you"

You are deviating from the fact that you promised to give sth away at a national conference, then completely ignored it until s.b obviously pointed it out. What happened between BH-11 to last November - when you started telling everyone to send email to Sean?

Maybe if you keep reframing your questions you'll manage to weave your way past my original evasive answer of "we decided not to do it that way" to the truth of "we grind up the bones of the victims of our free crypto challenges into a fine meal we use to fertilize the fungus patches we rely on for sustenance down in the pentest mines".

"Free" challenges, indeed. Ah hah hah hah hah haaaaaa! Mwrhwprh -gulp-. Mmmm. Fungus.

What a great response from Matasano founder! No wonder they have 2 Ex-es (daeken,nbpoole) in this thread.

Please don't presume to speak for me. I left Matasano because I was returning to school; I greatly enjoyed my time there and I keep in touch with everyone I worked with because they are a group of intelligent, kind, and all around awesome people.

Also, I wholeheartedly endorse Tom's responses to your insinuations.

Totally agreed on all counts.

Cody, you're dealing with a coward that created an anon throw-away account just to troll with offensive remarks. I'm actually surprised how restrained Thomas has been with this jerk.

It's not the anon that I care about, it's that someone is being an asshole in public to people I consider to be friends, and using my (and nbpoole's) name to do it. I'm not fond of that.

It bothers me too.

You guys are all great; honestly, though, I think we're the only ones reading this part of the thread, and I was owed some kind of karmic retribution for turning Cody's post into a Matasano hiring thread.

I'm surprised none of you think I'm devious enough to have planted that anonymous commenter, though. "What, you're saying you test ALL the form inputs? What are you, some kind of atomic superman?"

DON'T BELIEVE EITHER OF THEM! THEY ATE THE FUNGUS!

The Fungus is made from people!

So, I'm ex-Matasano and now work for Accuvant LABS (a competitor). While I'd love to snag awesome people to join up on my side, I completely disagree here. Matasano is a great company and they do do good work. #1 they should remedy for sure, but #2 I disagree with -- they hold the same opinion I've found in other high-end consulting shops, just more vocally.

"We avoid tools like sqlmap"

"I wish Burp didn't have a Scanner. I might pay $25 more for a branded version of Burp that specifically didn't have that feature, so I could reassure clients I wasn't ever using it."

Tell me, how do you expect to find MOST instances of SQL Injection or XSS without using tools? Do you manually tamper with every cookie parameters? Unless Matasano has better tools and release them publicly, then I am interested in hearing about them.

My gosh. Actually understand every cookie parameter, instead of running some tool to generate a list of obvious SQL injection vulnerabilities? Next they'll say they actually base64-decode every cookie too. How do they find time to sleep?

We automate lots of things. We just don't automate things that remove judgement from testers.

I've been a vuln researcher since 1995 and so have my partners. I was a lead dev on the industry's second commercial vulnerability scanner (Ballista), and Jeremy worked at ISS on the first. I think we know what we're talking about. Here is what we've learned: when you give a smart tester a tool that purports to find "low hanging fruit" vulnerability X, testers get worse at finding vulnerability X on their own. They subconsciously lean on the tool. They make assumptions about what kind of vulnerability the tool will find that they shouldn't waste time looking for. They gradually start getting worse at finding even the clever variations of X.

So the challenge is to find ways to eliminate drudgery (for instance, in comparing large numbers of responses from a web app to a run of different metacharacter input vectors across every parameter) without introducing things that degrade tester judgement.

Burp Intruder: Fine (though we do better internally for some things). Burp Scanner: Not Fine.

You can script the scanner to auto-decode B64 cookie once you found them: http://blog.portswigger.net/2012/12/sample-burp-suite-extens...

It all boils down to: how can you be so sure if your tool/process is finding most vulnerabilities than others, and can you prove it?

If I were your client, I would be very worried by now.

You've got me. I've only ever tested up to 49 dynamic forms, and only 42 insertion points.

I worked as an IT security consultant for a bit over a year and found it rather boring after a while. In theory, you do lots of interesting and different things, get to know many different technologies. Maybe it was just the job I was working at, but the actual penetration testing became pretty boring after a few months.

While it sounds compelling to beat up startups, startups are probably not the clients. Our consulting day rates were very expensive and usually only large corporations or the government could afford it.

You just go and check for the top OWASP vulnerabilities. Sometimes, it requires some creativity, but oftentimes, you get a "feeling" for a web app after a while. Many PHP projects, many open source software projects that got a few custom made plugins. And then it's a bit dull. Testing every single parameter of a web app with many attack vectors...

I have to admit that there was one guy at our company who did a lot of reverse engineering, iOS security, testing a DRM system for an ebook online library (key takeaway: you can't control the client. DRM makes it harder, but it is never impossible to crack the system as long as the hardware is not custom made or something), stuff like that. So this was quite challenging and changing, but this was rare.

One gem I want to add to the original post: If you're interested in SQL injection, check out sqlmap. That tool is a real breaker and worked wonders for us and we downloaded entire databases by having a tiny little sql injection vulnerability in the signup form of a newsletter or something like that.

We work with lots of startups. We work with big companies, too, but that work is disproportionately with big west coast tech companies. The "east coast" BigCo work we do touches on trading protocols and order routing systems, which is fun for a lot of other reasons (wider application domain for findings, extremely high impact, complicated systems with message-oriented middleware, non-web inputs).

We do zero government work.

I don't feel like we're along in appsec shops for having this mix. I think one possible difference is between pure appsec shops like us, iSec Partners, IOActive, and Stach & Liu, versus general security practices. The work at general security practices might be more of a drag.

It's also the case that network security, being a race to the bottom (with Nessus and Metasploit "scanner jockies" and the like) is actively trying to push up into appsec. Maybe the web appsec work at a place like that is boring? We take it pretty seriously.

We avoid tools like sqlmap.

I'm answering this on the off chance that the conversation is a good glimpse into the working life of an appsec pentester (since that's what Cody's writing about).

> Maybe the web appsec work at a place like that is boring? We take it pretty seriously.

Could be. Your work description sounds different though. We did use Nessus and Metasploit for some things, but not for web app security, since ALL these tools suck on a web app security level. They do stupid request-response analysis and they usually have no capability to hold some sort of state, which gets increasingly important in modern web apps.

> We avoid tools like sqlmap.

I think there's actually no other tool like sqlmap. Sqlmap is pure gold as a time saver and also capability-wise. Exploiting a blind time-based SQL vulnerability manually is a pain. Why not use a good tool for that?

If you want to be super careful, just hook up Burp between sqlmap and the target host and check every statement manually. Still better than typing it out.

Note for non-security guys: Blind means that you don't get an error message from the host, which should be the default. Time-based means that you craft some SQL statements that take longer than other statements to get an idea which statement is true. So you could ask something like "does user 1 in table 1 start with letter a-f? if so, return it, if not, wait 3 seconds". This way, you get true or false based on the time it takes the host to respond.

Still, if you're interested in web app security, go and try it out. But if you feel you're some sort of pentesting monkey that does the same stuff day-in day-out, better leave and chase something more interesting :)

I think there's actually no other tool like sqlmap.

But there is no tool that will come close to competing with a top-notch pentester on a SQLi hunt. Automated tools just don't cut it. There is nothing like watching First Blood in action.

Matasano's site mentions office culture. Are remote employees impossible to work out? I'd LOVE to get into this sort of thing, but I'm not a US citizen and I don't qualify for any visas.

If I get in touch, would you point me towards some resources even if it won't lead to employment?

Of course I will!

Fantastic! I dropped you (and Sean) a line.

Is the work at Matasano (and security consulting in general) mostly attacking web apps? How often do you get to use tools such as IDA Pro to reverse binaries?

Either way, it must be fun doing that full time. Nothing in the world comes quite close to the feeling of breaking someone's system. The building excitement and anticipation as you realise you might just have found a place where they don't properly encode one protocol into another. The intense satisfaction when you get to demo an exploit. Unlike the rest of app dev, you can prove your attack is right.

I've been a security consultant for ~13 years or so (predominantly application focused), and I'd say that all other things being equal, a good portion of it is web apps.

That's mostly just an artifact of the fact that so much software over the past ten years is web-based. I'd say maybe 80% of the client work I've done has been web-based (with maybe 10-15% non-web application, and the remainder network stuff).

But it's not the same everywhere. I would posit that one of the differentiators is the size of the company (i.e.: bigger security firms probably do more web-based stuff than more boutique places, mostly due to the clients that big firms service).

At the last place I worked, I ran a 10-person consulting division, and it was maybe 50/50 web app/non web-app testing. We were eventually acquired by a giant telco (two actually), and fast-forward a couple years, and the now 200-person consulting division is mostly doing PCI-related web-app testing (I have since left, although I think I stayed longer than I should have).

The larger the company, the larger your clients (generally), and the less agility of your sales process (ie: sales people tend to have a much easier time selling web application testing, as there is a huge number of clients who need it, and it's easy to put together statements of work around it).

So my advice, if you're interested in the more interesting types of security work, is to look for a small-to-medium-sized place. Actually, regardless of the type of security work you're interested in, I'd recommend a smaller firm. I've worked at enough of both to think that there's a certain size (either of head count or revenue) where you start to do less interesting work.

We definitely do both things. You wouldn't want to work here if you hated web apps. You wouldn't need to know your way around IDA Pro on day 1 (for reasons that will become clear in a few months, you'd be comfortable with the basics of assembly language within a month or so of starting).

> Also: I understand why Cody didn't write it this way, but the reality is, if you're going to test web apps, Burp is the standard tool. You can use things like mitmproxy or even WebScarab, but most people end up in Burp. Burp is also extremely valuable for testing even if you're not doing appsec full-time.

I actually forgot to update that -- it was on my list of edits. Done now, thanks!

I feel like I could justify the expense of Burp for a random freelance developer, even if they weren't billing out as a security tester. Like, I feel like we could convince Patrick McKenzie that it was worth his money. What do you think?

Ex-Matasano chiming in! ;-)

I've been demonstrating web app security topics for the Intro to Security course at Brown University this semester. I've used Burp almost exclusively. I've even had the students use the free version of Burp for labs. I don't know that the functionality / price tradeoff would make sense for them even if they were full time freelancers: they use Repeater and Proxy more than any other tools.

That being said, any time someone complains about it being too expensive I point out that a single vulnerability found as part of a security bug bounty program (ie: Google, Facebook, Mozilla, etc) nets you more than the cost of a license.

Justifying the expense wouldn't be difficult at all. However, I think the free version is Just Fine (TM) unless you need stuff like the scanner or intruder (intruder works in Burp Free, but is limited to something like 1 request/second).

Burp Intruder is the fuzzer inside of Burp. All the Burp-like tools let you capture requests your browser sends, edit them, and replay them. Burp Intruder lets you take a captured request and set up rules to send hundreds or thousands of variant requests.

I am weird among Matasanos (and ex-Matasanos :|) in that I live inside of Burp Intruder; I use it instead of Repeater. Why replay a request once when I can replay it 1000 times? So for me, non-crippled Intruder isn't optional.

I wish Burp didn't have a Scanner. I might pay $25 more for a branded version of Burp that specifically didn't have that feature, so I could reassure clients I wasn't ever using it.

> I wish Burp didn't have a Scanner. I might pay $25 more for a branded version of Burp that specifically didn't have that feature, so I could reassure clients I wasn't ever using it.

Huh? Do you like wasting clients time/money?

No,

* which is why we don't charge billable hours to run off-the-shelf tools that our clients could just run themselves,

* I addressed why we don't "augment" with scanners downthread (shortened answer: it's a slippery slope to testers just running scanners),

* Our scoping and rates are dead square in the middle of the market, so if scanners are helping other firms deliver projects more cheaply than us, I don't think the savings are being passed along. (We also don't double- or triple- book consultants on multiple projects, and we don't pay overtime.)

I upvoted you, because while I thought that was a pretty snide way to ask the question, I sure am happy to get to say over and over again how our projects aren't just Burp Scanner results. :)

Well I don't want to turn this into a company-specific debate, so I'm just addressing the position that running burp's scanner or sqlmap is "low quality". I have a few issues with that position and your justifications.

I wouldn't bill a client for running a scan on them. I would start a scan and do manual testing at the same time, focusing on more intelligent attacks and understanding the application. By the time I am done, the scan would typically kill off a significant number of buggy parameters that I now don't have to test because I already know it's as vulnerable. For some projects, this can be quite substantial. Beyond creating a POC and documenting the issue, I now don't have to spend billable hours on all of that.

The fact that scans consistently find a lot of bugs tell you that clients aren't running tools themselves. They don't know the tools, don't understand the results, don't know how to use them beyond point-and-click. They don't know how to set up macros that validate the session and re-log in, etc.

Although it sounds good to say that they aren't paying you to just run a scanner, the reality is no other reputable testers are doing that either.

Yeah, it was a bit snide, but you were scoffing at testers who do use scanners, and I genuinely think not using them (properly) is a colossal waste of time

I wouldn't want to restate a whole bunch of points I made downthread (we think scanners degrade manual testing, we're not opposed to automation but instead only to automation that actually flags findings, we grind up the bones of candidates to fertilize the fungus we use for our pentest "trips", &c).

It would be fun to have this debate somewhere that wasn't 10 comments deep into an old thread.

I don't actually know you, or who you work for, so please don't think I could be calling you out as a bad tester. We just don't test with automated scanners. We're not the only shop that doesn't use scanners. It's just the way we work.

Huh, yeah, I've never seen anyone use intruder like that. I might use it once every other engagement, whereas I use repeater 24/7.

Also, if I'm justifying Burp to a non-security person, part of the reason why is that Intruder would allow me to do all sorts of wacky integration and stress tests without having to write fiddley code. A rule-based request generator is a pretty useful tool for the box.

$299/year seems pretty affordable, I was expecting to see something that cost thousands from the way you were talking. I know zilch about Appsec, but this appeals to the part of me that's good at breaking things.

So kind of like a magic 8 ball variant of ab? You know I'm rather surprised there aren't more open source tools like Burp and that it is so expensive.

You could use it to benchmark (it might be useful for that in cases where what you were benchmarking wasn't raw request handling speed, or the performance of simple SQL queries, but rather some backend event that would only be tickled by a particular pattern of requests), but the real thing it does that I think ab doesn't do is collect all the responses and allow you to compare them.

(It's actually not great at doing those comparisons, but I don't have a better alternative).

Burp costs money, but it costs so little money relative to its value that if you think it's expensive, I'm going to suggest you're doing something wrong with your bill rate.

> Burp costs money, but it costs so little money relative to its value that if you think it's expensive, I'm going to suggest you're doing something wrong with your bill rate.

Couldn't agree enough. Even if this is something you do as a hobby, Burp will more than pay for itself in a single bug bounty payout.

I meant expensive for somebody who is new to the topic and just wants to play with it. Its an inertia thing.

Check out the free version. It still has significant capabilities, including intruder.

I'd be interested in what you security guys think of ZAP, the OWASP/Mozilla attack proxy. It's probably not up to the level of Burp yet, but I've found it works pretty well at least for basic stuff. https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Proje...

"...you're a solid software developer..."

I think that's the area it's hard to break if you're already engulfed into security. I've been in architecture, research and pentesting across a lot of shops but I've always felt like minimum viable exploitation was all most places were after.

Fast forward years and you end up in something you don't feel like you've transitioned into something deeper. Sure, you can go there on your own but it has limited viability unless you're path forward is Pwn2Own or bounties in general.

I'd love to work for the Matasanos of the world but feel like I may be locked out based on the initial hurdle of being more proficient in code as a first class skill vs having a more honed skilkset in finding flaws on the systems as a whole. Also tracking the relevant things within the security landscape is a skill in and of itself and since most orgs don't understand how to find and cultivate that talent it open the doors foe the Risk.IO of the world.

Sure, I've done some reverse engineering, lots of (easy) pentesting and am proficient in Python. I've designed many F100 systems as it pertains to the security construct, yet I can't see a path forward to digging in deeper with those like Matasano. So @tptacek, any advice? The money is excellent on this side of the fence, but the real challenges are, seemingly, few and far between.

Edit: I need to stop writing posts from phone/tablet (spelling).

I wish you guys had openings in the DC area. I've been interested in this stuff since college, when I would try reverse engineering using Fravia+'s tutorials. That's how I learned assembly programming.

Me too! There are a bunch of people I'd love to drag into the company from DC. But we're sticking with NYC, Chicago, and Mountain View. Always happy to relocate people! :)

I wouldn't mind going to the west coast, but my family hates moving. Do you know of any similar organizations in the DC/MD/VA area? I'm sure the government has a bunch, but I'd eventually like to get out of the gov side of things.

Reply on news.ycombinator.com