Hacker News Reader

> For those curious, #4, my presentation was about how I reverse engineered the Robinhood API, spoofed requests to get around their “security”

This was probably against their TOS and even if not, trying to bypass security measures, no matter how weak, is unethical behavior, and possibly illegal.

That would be a red flag in an hiring scenario since you were essentially bragging about breaking the law.

Unless Robinhood had a bug bounty/responsible disclosure kind of program and you were fully acting within its limit, but your comment did not sound like it.

It's not "security" in the hacking sense. Perhaps that was a poor word choice. So when you hit an API, the server can reject your request for any reason, typically a malformed request. Maybe a header it's expecting is missing, or the API key is invalid, something like that. With Robinhood's API, you can't simply provide your bearer token and hit the endpoint. There are MANY headers it is expecting, and if any are missing or incorrect, you'll get a 4xx response.

Otherwise, breaking ToS is akin to jaywalking and anybody who sees a side project like this, and their takeaway is "wow they broke ToS" doesn't deserve to be hiring software engineers, and certainly doesn't deserve to be putting them through the ringer such that the first bar to pass is literally Triplebyte (RIP Triplebyte).

Hopefully you explained the unique weirdness with Robinhood's API parser during the presentation, such that it was necessary?

Otherwise it sounds more like a case of "Here's how to get around the rate limiting of Company Foo's API and abuse those limits anyway".

Without the explanation, that'd get you an immediate No at lots of places just from the implied legal exposure/liability. Though probably not all. ;)

I’m not sure I agree with that. If you’re a technical person, like say a “Senior Software Engineer”, you should pretty much know that’s not what’s happening with something like this. Rate limiting is server-side. “Getting around it” means making fewer requests than what would trigger the rate limiting/temporary blacklisting.

You could distribute the requests amongst multiple IP addresses via say a proxy, but that isn’t necessarily going to work, like if the rate limiting is done by bearer token or API key rather than IP address (they should be doing it by the token rather than the IP address), but I made no mention of that so I don’t think anybody would think that’s what I was doing.

It depends on how it's pitched. :)

If you went with something like "using caching to avoid hitting API limits" it sounds legit.

Whereas if you went with something like "getting around Company Foo's API rate limits" it sounds like you're exploiting a bug in their system.

If you did present the system as spoofing requests to subvert a rate limit / blacklist, as per:

  #4, my presentation was about how I reverse engineered the Robinhood API, spoofed
  requests to get around their “security”, and used two levels of caching (Redis and a
  typical relational DB) to subvert rate limiting/IP blacklisting

... then that doesn't sound like the legit use case scenario. ;)

Your further follow up discussion makes it sound like you were just using a self-developed caching system though, and presented it terribly for the target audience.

I don't know that there's a way you can pitch your reasoning to me where I would agree. The two statements sound equivalent to me. API rate limits are just that- rate limits. This is not akin to hacking and a software engineer I doubt would ever confuse it with hacking of any kind. What would be an example of "exploiting a rate limiting bug"? Either the server limits/throttles requests, or it doesn't.

> What would be an example of "exploiting a rate limiting bug"?

All sorts of possibilities spring to mind for that. :)

Here's a hypothetical example. Lets say you discovered a logic bug in the API limit code along these lines: "So, it turns out if you have an `X-Forwarded-For` header in the request, then the request is processed properly except it isn't counted towards the limit. Effectively a bypass."

Exploiting this one would of course be trivial, as you'd just need to set that header for all your requests.

If an interviewer whined about Wall Street TOS, I'd pull my application on the spot.

I think some companies just want to look for a reason to disqualify a candidate to make their decision process easier. If you have two roughly equal candidates, pick the one whose side project did NOT violate some random company’s TOS. I kind of get it. It’s incredibly lazy and arguably just as unethical as violating some random company’s TOS, but I get it.

Yeah. This sounds like the presentation requirement (ridiculous as it is) worked to filter someone based on unethical behavior.

To quote another commenter, “If an interviewer whined about Wall Street TOS, I'd pull my application on the spot.” But yes, that might be why I didn’t make it past that step. My internet connection also dropped in the middle of it (AT&T Fiber, connected via Ethernet), and I had to give the presentation again, that might have been why- though they explicitly said that wasn’t why, but… you never know.

> used two levels of caching (Redis and a typical relational DB) to subvert rate limiting/IP blacklisting

How do those subvert rate limiting / blacklisting?

The Robinhood API is of course the API for stock broker Robinhood. Their API, last I used it, probably could have been optimized to provide data in fewer API requests, but aside from that, there are many pieces of data that change constantly, like the current price of a stock. There are pieces of data that change occasionally, like the next earnings report date for a company. Then there are pieces of data that should never change, like a company's CUSIP (essentially the primary key in a database of stocks). When you create an app that relies on data from another API, there is usually a "setup" where all the data your app requires is brought in from the API. I was getting rate-limited during this process (downloading all my watchlists and current holdings, as well as my orders/transactions, metadata about the companies such as the next earnings date, and so forth).

By using Redis and MySQL, I minimized the number of calls the app has to make directly to the Robinhood API. For things like the next earnings date, which changes typically once per 90 days or so, I have Redis store the result of that endpoint from Robinhood for some number of days, probably not 90, but not 1. For things like the current price of a stock, Redis might store that for a minute or two, so that my app can "spam" for the latest prices (maybe I have a refresh button in the app) without triggering a 401 or 403 from the Robinhood API.

TLDR: Since rate limiting prevents my app from pulling all the data it needs at one time - it has to throttle requests - I don't want to have to wait a couple of minutes each time the app is started because it needs to download data from Robinhood, much of which probably hasn't changed since the last time I started the app.

> Then there are pieces of data that should never change, like a company's CUSIP (essentially the primary key in a database of stocks).

I hate to break it to you, but tickers and CUSIPs do change.

The world of 'corporate actions' turns out to be a catchbasin requiring human intervention.

Yeah fair enough. I see a ticker change somewhat often (eg ETH to ETD, FB to META), CUSIPs change much less often, but unlike tickers CUSIPs aren’t reused. A company’s CUSIP will always be their CUSIP, but it might not be their current CUSIP.

  Jesus Christ himself could come back to Earth ... Ooo, sorry, that gap is a huge red flag.

lol