Stripe-CTF 3 Writeup | Hacker News Reader

Stripe-CTF 3 Writeup

muehe.org

74 pointshenrikm8572 comments

I got stuck on Level 3, partially because Scala is a little wacky (but I learned to appreciate it!) and partially because their specs were super terrible. They didn't mention it required substring matching or was case-sensitive until I'd been working on it for 2-3 days and then was out of time. Given the bad debug output (the head of a diff which often didn't show anything and didn't include what term the failure was for) it was really frustrating to make any progress.

And he's right about the servers and scoring; that, too, was frustrating. I'd pass a test case locally then push and have it fail because it was slower for no apparent reason. Overall, it was awesome up until Level 3. I wish I'd made it to 4 because that strikes me as an actual distributed systems problem rather than a minor tweak to an existing example like the others.

It wasn't a bad experience and the system they have to manage it all is cool. I just wish a little more care had gone into making that level a bit smoother.

I just rewrote it in Go, you can easily do the same with your preferred language. the readme is actually well written, and you can easily check the actual input you get by just sniffing.

give it a try

I'd pass a test case locally then push and have it fail because it was slower for no apparent reason.

My theory was that the remote instances were really CPU constrained, and so much much slower than your computer, but for whatever reason local test runs were very misleading. The spec for level 3 was annoyingly vague at first, and the test output was really unhelpful as the diff started with lots of original data and was truncated.

Level 4 was interesting, you should have a look - just the links alone are interesting ones, and I liked the raft paper, which was pretty easy to read even for neophytes. I might try to do better at it locally now when I have time free, but again, don't take local scores seriously, I scored in the thousands locally on that, but below 100 on remote. Must. do. better.

Yeah same here. I wish I had given up on Scala and used something more familiar to me. I kept having my index serialization randomly die and it wasn't very clear what was going on. It was great to get a taste of the language, though.

Yeah, I ran into the same issue on Level 3 (the comment about Github search in the first edition of the readme threw me off since Github search does not do substring search). The worst part was that I knew what needed to be done from the very beginning but the simplest things were kinda hard to do in Scala and playing around with the setup to test things out was pretty frustrating since you had to recompile every time and the recompilation took aeons.

(This is not meant to be a critique of Scala, while working on the level I kind of understood the appeal)

> the head of a diff which often didn't show anything

They really needed "here are very small test cases to just find out if you doing things correctly." I spent at least a day wondering why things remotely were working locally but not working remotely when it turned out I wasn't giving file names in the proper format, something their "head of diff" format provided no feedback on.

To be fair though, there was an update to the level 3 README published on 1/25 which made some of the requirements (substring matching) a bit clearer.

Yup, but that was kind of after I didn't have a lot of time to devote to it unfortunately. They did clarify but only after I whined. ;)

I got stuck on level3 over the weekend and then had to put it down due to life stuff. But it was still a ton of fun, and I learned a lot.

If anyone from Stripe is here, do you plan to post the original problem statements and repos for the problems anywhere public now that it's done? I'd love to get a look at level 4... :)

I have a post on the CTF infrastructure that I'm working on; should be out in the next few days hopefully.

Looks like the original level sources have been published, though perhaps we'll do something official as well.

https://github.com/ctfs/write-ups/tree/master/stripe-ctf3/le...

Ha, I missed that this existed. Thank you!

Well it was published about half an hour ago ;)

Same situation here. I'd love to work through the problems at my own pace over the next few weeks though - I'd also like a way of replicating the solution testing that Stripe performs when you do an actual submission.

Is that infrastructure going to be released as well? Or would I need to just rely on the test harnesses and comparing my solution against the published ones?

You'll be stuck with the test harness for the foreseeable future, unfortunately. Releasing the submission infrastructure in a usable form would unfortunately be a looot of work. We're working on open-sourcing a lot of the dependencies, though.

Ok, that's probably enough for my learning purposes. Thanks!

Not having looked through all the problems yet - do they all have a reasonable test harness that gives you some sort of score?

Great article, I am really eager to see the solutions in github. I guess there won't be many with c++ solutions.

I only found the challenge two days before the end. My goal with the challenge was not to get the best solution, but rather use it as an exercise to practice the different languages and finding intuitive solutions.

Level 0 as you wrote was really just a two line change.

Level 1 was fun, because I tried to work with the existing bash structure as far as possible and just replaced the hot loop with inline python (something new learned). I found it interesting that this way I could directly inject the variables into the python program without having to load them as env variables (see [1]).

In Level 2 I did too much work, I did not think simple enough. I assumed that the different test cases would differ a lot and that a static divider would not help. My assumption was that the histogram of requests per second across the different IPs would have a bimodal distribution[2], thus I used the excellent "fast-stats" library to get a histogram and ban clients based on that. The library even offers approximate histograms, so even if there would be thousands of requests it would still scale.

To solve Level 3 with the given code required to changes:

-(i) Partitioning the data across the three server. I only loaded a subset of the files based on the server id (which was conveniently already available in the search servers) and

-(ii) Loading the files in memory so that they don't have to be read from disk for every search term. This was enough to pass this level. The code was too long for the gist but I can put it on github if someone wants to take a look at it. No fancy index structure required. I just scanned The files linearly on each request.

Mastering Level 4 was not possible for me, I tried working with the go-raft library and got the integration with unix sockets working, but somehow the system got unstable after a while. Fixing this was really frustrating and in the end I gave up because I could not find the problem and time was running out.

I congratulate all ~200 persons which also passed this last level. This was much more difficult than the other ones!

[1] https://gist.github.com/mkaufmann/8716922 [2] http://en.wikipedia.org/wiki/Bimodal_distribution

well to level3 I can tell you, I just used grep embedded in a nodejs server application :D

Everything boiled down to this: "cd " + data_path + "; grep -rno " + key + " | cut -d: -f1,2 | sort | uniq"

I read the result from stdout and transformed the result into json that was enough to make it. I did not want at that time to spend more time on that level ;)

Damn ok, so I also missed that opportunity. Great find!

I forgot to add: I cached every result for every key which was searched for, because there was quite often that it asked 3-5 times for the same term in a row. So I had an instant result in a simple javascript object :D

Mind posting your code for level3? I got that partitioning was the main trick but I am too unfamiliar with scala to reassemble the responses properly.

FWIW, It was possible to pass Level 3 with ~5 lines of code changed - https://gist.github.com/ajtulloch/e05f75aaa6ba3b5b0241#file-... for my solution.

Here is how I merged the responses: https://gist.github.com/mattnworb/8717911

Or a bit cleaner using lift-json: https://gist.github.com/mwylde/8724700.

I did both all my solutions in C++. I am going to clean them up a little and post them in my github account

github.com/dkhenry

I really enjoyed the parts of CTF I was able to work on. I didn't have enough time to finish, but I got through the first two levels and the second flag ( gitcoin ) was actually really fun to implement. I got a gitcoin hasher up to 2M hashes/second on a single machine, but even with that I was no where near fast enough to get any coins when competing against the rest of the field.

I limited myself to doing everything using only "C++11". I had to pull in boost_regex since no one has the c++11 regex support, but I am really liking the new standard and looking forwards to what we can get with c++14

I think this particular bit says it all. To get blazingly fast results...

> I can say with great confidence that we all cheated.

But I disagree with the word used.

Pre-computing the hash-table and embedding that into the final binary to eliminate relatively expensive calculations when it matters? That's not cheating, that's effing brilliant.

Hard-core software engineering, that's what it is. There's also a saying that the only people who play fair are those who don't know how to cheat well enough.

Well done.

(edit: typos)

Op here, just wanted to say thanks. Of course it is not cheating in the sense that I tricked the system to give me a better score than warranted but my solution only works for this ONE dictionary so.. maybe it is not cheating but using tighter assumptions than those which were provided by the original task description :-)

The problem description did say:

    # Our test cases will always use the same dictionary file (with SHA1
    # 6b898d7c48630be05b72b3ae07c5be6617f90d8e).

The cheat is that he used a separate process to build hash table for real time look up, which is different from reading the data from the file at run time.

After many hours I failed to complete Level 4. How frustrating! But what a great experience. I think Stripe did a great job at building a programming challenge. By going through the challenge I learned something new in each of the following: Python, Ruby, Scala, Go, Node/Javascript, Git, Hashes, DDOS, Consensus Algorithms, raft, transactional logs, Inverted Indexes, BitCoin, IRC Chat, and more. Thanks Stripe you made me a better programmer.

You're very welcome. Glad you enjoyed :).

I gave up on the last level, as I didn't have the time to commit. I'd say this CTF was too difficult/too much work for me. Last time with smaller challenges was more fun. The challenges were interesting in their respective subjects, but maybe they should have been a little bit more condensed in scope, particularly the last 2 levels. Although learning Scala and Go beforehand would have helped a bit.

I went up to level 3 and then work/life took over. Here's my approach to the problems:

(1) Python. Used a set() for the dictionary. Got ~200 points. I was planning to go back and implement this in Go to see the difference in performance but never got around to do it.

(2) Python. I wrote a single-threaded miner which could do around 300K hashes/s. My miner calculated 1M hashes (~3s), then fetched origin, then continued until it got a hit. I stopped after the first Gitcoin.

(3) Node. My solution was very simple and pretty much like the OP's. I kept a map between IP and count and always let through IPs with count <= 4. If the count was > 4, then I let them through with 0.3 probability to keep the backends working.

(4) I spend most of my time on this problem, largely trying to keep the memory within the constraints. I started with Node and created a simple inverted index of all words to their positions. Since we needed substring search I had to loop through the keys of the index which was too slow. My next attempt was to index all substrings but that used too much memory. My third, and best, Node attempt was using a prefix tree to index all suffixes, effectively building a suffix tree. This was quite fast, got ~2200 on local tests but was still over the memory limits.

I then gave up on Node and moved to Python. I used Flask for the server and built two solutions. A simple inverted index and a prefix tree (using datrie) with all suffixes. To keep the memory footprint low, I stored file/line locations as bit-shifted longs. Amazingly, the simple index and the prefix tree performed the same (!), so I submitted the inverted index solution and got a passing score.

Thanks for a great week Stripe. I learned tons! Looking forward to the next one.

I bailed out on level 3 after about 10 total hours into the contest, as I see was pretty common.

I really enjoyed level1. I ended up doing an ugly little modification to git itself to mess around with the last few bits of the commit until I got the desired number of prefixed zeros on the sha1: http://pastebin.com/RwTjjtTU

Looks great! Nice scores.

I focused on getting the minimal passing solutions (e.g. Level 3, the distributed grep problem, only took ~5 LoC changes to pass, Level 4 was 162 LoC additions, 69 LoC deletions).

I wrote up my solutions at http://tullo.ch/articles/stripe-ctf-golfing/ if anyone was interested.

I really liked CTF-3 but I'm astonished at the amount of time people are able to find for it. Just reading this write-up I kept wondering, "does this guy have a day job!?"

I got to level 3 but then figured I had already spent my entire weekend plus a day and had responsibilities like grocery shopping, going to work, laundry, etc.

Scored in the top 10 on level0 using a Bloom filter in C. Once you had an optimal solution, getting on the leaderboard came down to luck. 90% of the runtime of my program was operating system overhead that also happened on `int main() {}`. Today I realized I could've compiled with -nostdlib or -nostartfiles and halved that overhead - lesson for the future. If your binary needs to run faster than 1.5ms, libc is the next thing to cut :)

I held the top leaderboard spot of level1 for a day or so (my hashrate varied between 1-5GH/s) but was pushed out in the last few hours by pushrax and the Stripe server load. Used an OpenCL + Go solution. Was a close race there by all.

http://bochs.info/~aegis/rounds.tar.gz - tarball of all the Gitcoin round repos if anyone wants to play with the data.

Very fair feedback on moving the deadline. We'd debated doing it or not, and ended up moving it in the spirit of distributed systems education. A lot of people in the chat appreciated it, but that's obviously a selected segment.

So we know for next time, how strong are other people's feelings on this?

Personally, I would have been happier if the deadline sat -- if only because I needed to get back to regular life, and it's hard to resist the temptation to squeeze out those last few points.

But I might have felt differently if I hadn't finished...

I'm glad you did (but I wouldn't have finished if you hadn't).

More people learning & having fun is better. IMO people only complain about the extended deadline because of its effect on the rankings.

The scoring system was an interesting addition over CTF2, but it also created these complaints and silliness like people submitting the same code repeatedly to get the best score possible.

It may have been better to only tell players what broad percentile group they're in. There would still be an incentive to write faster code, without the frustration of being edged out of position #20 because someone else's submission was (arbitrarily) scored a few points higher.

I actually think there is a middle ground; just bump the deadline, allow people to still solve it and get a T-shirt but freeze the scores and everybody's happy.

Btw, I agree that the scoring needs to be consistent or less coarse grained (percentiles) so that people don't submit thousands of attempts to get the max out of their solution...

I think the write up is pretty good at stating the problems (along with the deadline). If none of the other problems existed there wouldn't be a reason to extend the deadline. I agree with the choice to extend the deadline due to load/bugs but a fixed revised deadline would have been nice. Also, maybe limiting the number of successful trials a person could perform in total would have helped with scoring.

That being said, I think you guys picked a hard CTF concept this time but I LOVED PLAYING. Thank you Greg and everyone else at Stripe that contributed!

I'm not completely sure if extending the deadline was a good idea or not, but what was awfully bad was the way it was moved.

If you move, commit to another firm deadline, not to a fuzzy, never know when it'll be done one, that's awful for planning and only increases stress.

I completely agree with this statement.

Added a link to your write-up and solutions to the collection here: https://github.com/ctfs/write-ups/tree/master/stripe-ctf3

I love his hash table solution to level0!

I would never have spent much time optimizing such a trivial challenge as level0, but after seeing the leaderboards with such ridiculous scores, curiosity got the best of me, and I had to figure out how people were achieving such speedups.

I used a precomputed bloom filter to get 2641 points on level0, because I couldn't think of another way to avoid doing the expensive processing of the dictionary ahead of time. That solution simply read in the filter from a file at run-time. I thought of embedding the data in the source code to avoid the file I/O, but instead moved on to work on some of the later levels.

Op here. I have done a lot of work involving hash tables in main-memory database research and therefore really like applying those optimizations. You are right though that it might be overkill :-)

I read through this and was amazed by the OP's approach to the problems even though I could barely understand 10% of the reading. How do I learn to think and code like this? I honestly have no idea where to begin.

Thank you. What you have to keep in mind is that I do databases research for a living so many of the problems involved in each challenge are very familiar to me. My suggestion to you is to get a stronger systems programming background, I figure that's what helped me the most.

I'm glad I participated in this challenge. It was fun to micro-optimize stuff, and I'm really happy to have been introduced to Go. The last level was a challenge, mostly due to Unix sockets and Goraft bugs and lack of documentation, but I enjoyed the language enough that I went back and redid level3 in Go, taking a top-30 score right at the buzzer. (Or where the buzzer would have been had they ended on time, which they should have.)

My family doesn't appreciate my long hours at the computer, and I haven't caught up on sleep yet, but I'll definitely do this again.

One recommendation I'd make, is to reveal all problems at the very beginning.

I logged in, saw that the first problem was nothing to do with distributed computing. Also realised it would take me longer to submit the code than it had taken to work out the answer... then I quit and didn't look back.

It's a shame, because levels 2 & 4 possibly look quite interesting, I might have persevered if I'd known. Sadly, limited time on my hands means I dismissed it too quickly!

I've blogged my extremely lazy solutions for the first 3 rounds. TL;DR is level0 used a hash table, level1 found an existing utility, level2 used a hash table. Grand total of about 15 lines of new code.

http://www.codesuck.com/2014/01/stripe-ctf-minimalist-soluti...

Nice article. I got stuck on level3 and perhaps got close to finishing it, but there were a few unforeseen things that came up. I'm glad I did what I could for this CTF, though. This is the first CTF I've done, and pretty much all the languages were new to me. Definitely a great experience!

I've got mixed feelings this time.

Disclaimer out of the way: No sense of entitlement here, every time I see a challenge like this I expect to fail. I'm really, really happy that people (Stripe, here) offer their time and resources to entertain us.

I'm a proud owner of the CTF2 shirt.

This time I couldn't finish level4. My takeaway is: I invested too much time and shouldn't have done it. I guess I'll pass next time when a challenge w/ a deadline starts.

Level0 was a nice warm-up.

Level1 was immensely satisfying and interesting, but the pvp challenge was .. well.. dead. By the time real life let me join others were throwing GPU based miners at the problem and traffic was constantly heavy anyway. So amazing level, the arena type thing was imho not a good idea.

Level2: Here the problems began. Local tests and remote tests diverged, plus this level was pushing you to use node.js. Will come back to this later. The actual challenge was interesting, so my issues here were infrastructure and .. language

Level3: This was absurd. Again, the challenge sounded interesting. The sample solution you're supposed to fix came in Scala and you had some constraints (time/memory). I had a couple of (locally) working solutions that never made it remotely. After hours and hours I gave up and solved this challenge - by executing grep for each request. Yes, I called grep and parsed the output, didn't index a thing. That felt .. bad.

Level4: Didn't solve it. Utterly frustrating. The consensus (hah.. hah.. get it?) seemed to be that raft (goraft) would be the way to go, but I invested again hours (8? Probably a lot more) and didn't make it. Problems with the deployment were mentioned on the channel again and again, goraft has even open pull requests and recent tickets from the stripe guys, as far as I can tell. Dependencies were a mess. Someone built the reference solution to this level and found bugs in goraft (see Github, goraft issues)? I .. don't think this was a good idea. Plus, forcing unix-sockets into the game (where everything around it doesn't support those and hey - if you want to have a 'curl for unix sockets' you end up finding another go project that just didn't work for me) was really messy.

So, putting aside the fact that I wasn't smart enough to clear level4 (it was obviously possible!): The templates came in nodejs/Scala/go. I knew a little about nodejs, nothing about the other two. Ripping the provided framework out was kinda possible, but a pain in the ass to debug (you could run a script that installed stuff remotely - I tried solving level3 in Clojure and it was a miserable experience). So challenging tasks plus new languages, plus deadline, plus 'servers are overloaded and cannot answer': This was just way too much for me. I still tried to solve level4 until the server fell over again and the CTF was finally closed.

My take away: All the ideas, the problems, were cool and interesting. But I guess even the Stripe guys were surprised to see the load/spikes/outages - probably a consequence of the 'distributed'/'load handling' type of problems, versus the 'try to break in' challenges last time. It hooked me with the leaderboard, nice website, good problems. But it was an exercise in frustration for me (and Scala/Go are _really_ not for me..).

That aside: Thanks a lot, Stripe. I cannot even begin to imagine how much work you guys invested in the preparation and administration/maintenance. I do consider the extension fair - and didn't gain a thing from that (was already at l4 at that point).

This is good feedback. Honest feedback like this helps us really understand how successful we're being with CTF.

One of our design goals was to make it hard to have something that worked locally but not remotely. It seems like we didn't quite make it, and that caused a lot of frustration. Sorry for that — it gives us something to work towards next time. In any case, I hope you walked away knowing some things you didn't before :).

If anyone else has feedback, I'd love to hear — feel free to ping me at gdb@stripe.com.

> Ripping the provided framework out was kinda possible, but a pain in the ass to debug

This is very different from my experience. They give you a build.sh script that does anything you want and a container with open access to the internet. Anything you print to stderr is shown to you. I didn't use the provided code for any of the challenges, and thought it was awesome that they didn't force you into a particular implementation. It's just a reference to help you get started.

And hackers staying up all night to write openCL gitcoin miners? That's just awesome. I don't know how you concluded that pvp "wasn't a good idea". I thought that challenge was just fantastic.

To clarify:

- Ripping out the framework caused lots of issues for me. Maybe my choice was wrong? I actually (in spite of my complaints) used two 'interesting' languages for myself. I polished my ocaml for the miner and switched to Clojure for level3. The latter was hard, because the infrastructure was down a lot and you already had a hard time getting past 'Kicking of your trial' messages - only to fail because leiningen complains about the inability to verify ssl certs for example. Try to fix it (doesn't happen locally...), push, fail to kick of the execution of the build script.. Loop.

- The miner: As I said, I loved the idea of that challenge. The problem was highly interesting. My problem with pvp was mostly "There's no way to compete". When I joined it felt already like installing a bitcoin miner on my home machine. openCL based miners? That's changing the field of course and the only way to participate at that point is to follow - and I'm running a laptop here with an integrated Intel chipset for everything else. That totally ignores the fact that I don't know a thing about openCL or CUDA and would've started from scratch (as w/ Scala, as w/ go).

I certainly understand the fascination, but .. how many people _could_ compete against each other? The implementation (gpu based) and the network (git pull was sooo slow, same for push) made this inaccessible. At least for me.

PvP challange dead .. but heavy traffic? Sounds like a contradiction. I think it's a flawed deduction that the arena was a bad idea because it took more than a casual glance at the challange to be competetive. For me, level 1 was definitely the highlight of the CTF. I was one of the few who wrote a GPU miner, and the constant competition of improving my miner vs. the other active participators was extremely exciting.

Good job. Did you have previous experience with writing GPU code?

Had zero experience with programming for the GPU, or with OpenCL. Took this as an opportunity to learn both. I was surprised both with how amazingly fast the GPU is, and how amazingly buggy the AMD OpenCL compiler optimizer is. (Found multiple bugs where it generates memory corrupting code.)

I personally enjoyed learning more about the different languages that were used at each level. Granted I'd be meaning to try out node/scala/go in a more meaningful manner and the ctf problems were the perfect setting. The downside to this is that it can be difficult to learn a new language and solve the problem at hand.

I agree level4 was a bit undercooked and not as didactic as it could be. One could simply clear the level by using raft as a black box. Personally I enjoyed level 3 the most given the trade-offs that were present in solving it.

I definitely learned a lot. I was able to pass all levels but unfortunately didn't have enough time to optimize them, even though I would have loved to. Thank you Stripe for putting this on! Looking forward to the next one!

I really had a blast doing level 1. I wrote my first real C program and it was as blazingly fast as I hoped. After that level 2 was easy but like many others I got stuck in level 3.

Anyone else seeing links to empty repos?

"Note: I’ll add the source once submissions are off."

Didn't catch that, thank you.

Here are my thoughts: I managed to scrape through level 4 so saw them all - this CTF was fun and educational. Level 4 in particular felt like a real education (for me at least), and a great way to learn. Unlike many others I was fond of level 0 because it brought up questions that even beginner programmers should address - how to organise your data so that you can search through it easily, which was in some ways a preparation for the other levels. For the more expert than me it probably felt tedious or inconsequential, but it was quick to solve, and I thought inviting for beginners and open ended.

Lessons learned for me

Distributed consensus was really interesting; I'd never heard of it and enjoyed reading about Paxos and Raft, even if I don't fully understand them yet. Distributed searches can actually be faster than searches in one process - I didn't believe this but as resources were limited on the server it was true in some cases (depends how fast you can make your search, mine wasn't as fast as Henrik's :).

Big scala projects take an aaage to compile - is it normally this bad in real projects? Anyone care to comment? The remote was pretty slow, so that might be why.

Golang needs package versioning! On level 4 there was a versioning bug with differing versions of the go-raft library being fetched locally and on the server which led to problems setting the heartbeat timeout of raft. That was confusing and annoying.

A few notes for the CTF authors

The dramatic variation between resources on remote and locally was a problem in tests - I felt like that needed to be adjusted for as the local test results were meaningless compared to the server. This particularly made level 3 and 4 difficult.

Code examples are great to get you started, but it'd be nice if there was also a clearly specified API for each one which you were required to conform to, to make it easier to use another language, and if the build scripts simply launched one binary at all times - level 3 had two different styles of starting the servers for remote and local for example, that was frustrating.

The varying test cases were sometimes frustrating, and were too varied and randomly chosen. I completely agree with Henrik that you need small tests. Each run should have say 5 tests which can be run individually (for speed), or as a suite. The downloading of tests broke a few times for me and left me stranded and constantly re-downloading new test data/tests seemed a bit pointless. Proper logging output when tests fail is really important - level3 was particularly bad for this as the diffs were useless; I had to edit the test harness to get decent output.

The use of the JVM on level 3 was pretty painful for me - compile times on remote were in the minutes, and I couldn't even get it to compile locally even after installing another JVM, nor on remote without compiling all the sbt stuff - this one was probably my fault, and I just rewrote with golang in the end. Explicit language independence for all the problems would be great.

Unix sockets felt like an unnecessary distraction which caused pain on level 4, it would have been nice if it used IPs, but perhaps for reasons to do with Octopus that wasn't possible, and on a deeper level, being given broken code to fix just felt a bit wrong and distracting - having slow code you have to speed up is one thing, but having code which feels like it has been sabotaged is another.

Don't bother changing the deadline even if people whine - the contest isn't remotely fair anyway or an objective test, so it's better to have a deadline and stick to it.

Oh, and thanks again Stripe, it was a blast.

The original code and some more writups:

https://github.com/ctfs/write-ups/tree/master/stripe-ctf3

http://blog.joneisen.me/post/75008410654

The reason the build times on level 3 were so bad was due to time spent downloading libraries. Stripe seems to have set things up to use a different ivy cache for each user, so every time you deployed code it had to redownload all of the dependencies. For me, this tended to take a couple of minutes. Actually compiling the Scala was pretty fast (< 10s for me).

So yes, Scala is generally slow to compile, but don't blame scalac for this one :).

Personally, I didn't quite manage to finish 4. I had a working solution locally, but didn't have the hours to debug why their remote environment differed so much from my local environment. My submissions also failed randomly about 4/5 times as the deadline got closer, presumably due to overload on their servers. And the unix domain sockets indeed felt unnecessary; I probably spent more than an hour just dealing with issues related to that.

The format (particularly on 3 and 4) of being given pretty broken code also sort of messed with my normal approach to code reading, which assumes that the code is more or less ok. This was particularly problematic for me on 4, since I had never before used Go and had a harder time figuring out what was wrong and what was just Go weirdness.

All around, though, I had a pretty fun time. Even if it did suck up way more time this week than I intended.

Note to Stripe: I would gladly pay $10 next time around if it could buy more/better ec2 instances.

Stripe seems to have set things up to use a different ivy cache for each user, so every time you deployed code it had to redownload all of the dependencies.

Thank you. So normally you'd have the dependencies prebuilt, and it wouldn't take long at all to deploy? That sounds more sane. The build time for golang was < 1 second for comparison, and at first I couldn't get rid of the sbt build due to confusion over start_server/start_servers being used to launch, which was a bit annoying. I couldn't get their solution to build locally so I just gave up on Scala, which is a shame, perhaps I'll try again sometime.

l4 was pretty hard, even just integrating raft in the time available, I started adapting what they had, then switched to inserting raft - I was still left with some issues, and it was humbling and awesome to see people scoring so highly on it for me with things like in-memory dbs, but also great just to read about the problem space, which isn't one I'd normally touch at all. So an education even if you didn't pass, which was pretty chancy anyway in my experience.

Note to Stripe: I would gladly pay $10 next time around if it could buy more/better ec2 instances.

I wondered if that was part of the challenge, particularly on l3, it meant it was harder to solve with just one instance, but yes it did cause frustration, particularly because a working solution locally would then not work remotely and scores were all over the place, even for the same code, due to contention and the random tests loading remotely. It's probably really hard to set this stuff up, and I look forward to the infrastructure posts to see how they did it.

> Explicit language independence for all the problems would be great.

I disagree, because the real world isn't all specs and cleanly defined APIs everywhere in the stack, and it definitely isn't mono-lingual. Oh sure, Stripe's actually pretty good with their public API, but I'm sure their stack has a few warts in different languages they'd like to rewrite.

Not knowing Ruby, or Scala, or Go, I had great fun getting into them for this CTF.

Yep, there is an argument for learning the languages too as part of the challenge, and it was interesting to see those languages in practice. I don't think I'd have made it through level 4 though without being familiar with Go - depends on the time you have available I guess.

Reply on news.ycombinator.com