Hacker News Reader

Comment by ericb | Hacker News Reader

* Better googling. Time-restricted, url restricted, site restricted searches. Search with the variant parts of error messages removed.

* Read the source of upstream dependencies. Fix or fork them if needed.

* They're better at finding forks with solutions and gleaning hints from semi-related issues.

* Formulate more creative hypothesis when obvious lines of investigation run out. The best don't give up.

* Dig in to problems with more angles of investigation.

* Have more tools in their toolbelt for debugging like adding logging, monkey-patching, swapping parts out, crippling areas to rule things out, binary search of affected code areas.

* Consider the business.

* Consider user-behavior.

* Assume hostile users (security-wise).

* Understand that the UI is not a security layer. Anything you can do with PostMan your backend should handle.

* Whitelist style-security over blacklist style.

* See eventual problems implied by various solutions.

* "The Math."

View on HN

> Assume hostile users (security-wise).

Ah, a personal favorite of mine which often makes other people uncomfortable.

1. Users are like Marines, if it can be broken, it will be broken. If it's use is not clearly marked, it will be used incorrectly. If it can be fucked (literally or figuratively), it will be fucked.

2. Users are malicious and are actively trying to backdoor everything you do. Why? Because it's the first thing I do with a new product / what I do with the competitions products.

3. It is the developers responsibility to mitigate the above cases.

> If it's use is not clearly marked, it will be used incorrectly.

In my experience clear markings aren't enough to stop it being used incorrectly.

Ah yes. Should be revised to "unless it is clearly marked, it will rarely be used as intended. Even when clearly marked, users will misuse in ways you couldn't imagine"

True. Clear markings are like a deterrent control in security. They can mitigate damage but something that prevents wrongful use needs to be used in the end.

Why is that like marines?

There’s an old story about a Marine left alone in a room with two bowling balls for thirty minutes.

At the end of the thirty minutes one bowling ball was in a bunch of little pieces scattered all over the room and the other one was pregnant.

In the Marine's defense, there was no bowling alley.

Exactly, what was he supposed to do?

I seem to be fortunate to work with almost exclusively top engineers, by your list. Interesting perspective!

> Anything you can do with PostMan your backend should handle

Can someone expand on this and explain this? I'm just starting to build my first API, and it's not clear.

PostMan allows you to send any query to a server. When building an API, you expect some required parameters (ie. "login" being a string) but postman can send or not send the required login parameter and it might or might not be a non-empty string as expected. On the other hand your web form will consistently send the login parameter and check it is a non-empty string before sending to to the server. So it's straightforward to implement the API assuming your required parameter will always be sent but in practice it's not always the case.

The point here is great engineers make sure the API answers correctly even if the query is messed up.

To add context to the other reply, it’s not uncommon for penetration testers to have Postman in their toolbelt, in addition to traditional security tools like Burp, Metasploit, sqlmap, etc.

Could you elaborate on The Math?

Sure. You can draw conclusions about what will be a good design, or where a problem is occurring, or what you should optimize from doing "the math."

The simplest example is probably using a Big O approach to view how to walk a large data structure. The math works out that certain approaches to walking it are cheap enough in terms of total operations at small scales, and incredibly expensive at large scales. By doing the math, I mean that you can both know when you can get away with sloppy approach, and when you have to be careful to not touch each object more than once.

I didn't just mean big O, though. You might do calculations like "if every user creates these structures in a database, every 5 minutes, we can expect that this table will have n records by time y. If that happens, one, is the size tractable, does it need an index, and do we want to reconsider this approach in favor of a cleverer design that can avoid this query? Things like that.

A recent example for me was researching max zookeeper writes per second and deciding if it can be used for super high volume writes in a system. The conclusion, based on some napkin math of the system needs, and some benchmarking of ZooKeeper was that a different tool needs to fill that spot in the architecture because it was too write-heavy for zookeeper.

Interesting, somewhat similar to what I do, but at a deeper level.

> Dig in to problems with more angles of investigation

Can you elaborate?

Let's say there's "slowness" on a route in a Rails app. A really junior engineer might start blindly optimizing.

A mid-skill engineer might start by trying to diagnose the problem, and usually will get somewhere, but occasionally they'll run out of ideas. They might assume it has to be one of the things they've seen before, but their list is not exhaustive.

A top engineer will start by looking at the big picture and generating hypothesis. They will usually be able to generate more hypothesis and have more ways to test them than less seasoned folks. Often this is because they've seen more things.

For example, off the top of my head, the slowness could be:

- n+1 query problems

- slow remote service calls

- streaming to slow clients

- object allocations

- loops in loops--bad big O

- resource limitation (memory, CPU, IOPS)

- database connection limits

- worker process/thread limit

- frequent cache misses

- noisy neighbor problem

- lack of indexing and data size growth

A top engineer will run through their own list, and for each, they have some ideas on how to check on that item or to rule out what is unlikely. Some of them are trickier than others. The key is they have more angles that they'll know off-hand to follow up on.

Reply on news.ycombinator.com