Comment by marginalia_nu | Hacker News Reader

Comment by marginalia_nu | Hacker News Reader

I've been hugged twice on 100 mbit domestic broadband, and it's honestly not been all that bad.

In general I've seen somewhere between 1-5 visitors per second, a couple of million over a day. I think keeping the number of requests per visit down is a big key to surviving this traffic with low resources.

The first hug was search.marginalia.nu, which used to load fonts and had a load of about 50 Kb/visit; but still only generated in maybe 4-5 GETs per visit. I measured about 2 Mbps.

The second time was search.marginalia.nu/browse/random, which loads 75 images at a resolution of 512x348 (mostly WebP; so anywhere between 2 and 30 Kb). These were randomly picked from a very large pool, so they can't be meaningfully cached. I presciently set them to lazy loading and I think that made a big difference, especially from mobile visitors as they'd only load a few at a time. I didn't measure the bandwidth usage at this point, but it stayed up and stayed responsive throughout the day.

I think something that possibly also makes a difference is that I don't do any sort of session cookies or mutable non-ephemeral state in my backend. I can only imagine having to validate sessions and manage access/roles for every request would make for a lot more work.

I don't think it's necessarily the bandwidth that's going to be the bottleneck, rather than something like contested locks in the database, resulting in back pressure in turn causing traffic jam of lingering connections. Reducing mutable server state helps a lot with that.

Wait, what, one of the best text search engines for the whole internet runs on a residential broadband connection on a home server? My mind is blown.

Sometimes I wonder what kind of scaling startups are scared of. I keep reading about popular websites running on a single server, and how cheap bare metal is compared to an equivalent AWS setup.

Am I missing something?

Serving static HTTP GETs is easy. The moment you start trying to make money off the generated traffic its no longer a read only affair. Your system is not stateless anymore and tracking costs spiral up quickly.

I largely agree with the analysis, with the caveat that a search engine is a bit more involved than serving static GETs.

Sorry, I lost context and was not thinking of the search engine but of a statically generated blog with small pages and little media. Also, thanks for confirming my hunch about the bigger factor being statefulness rather than dynamicity.

CPU, RAM storage are incredibly cheap nowadays. But so much can be going on at a time that _certainty_ is now what's pricey.

I don't think most startups really understand how their systems scale, so assume they'll scale like a social media platform (which do not scale well at all).

I attribute a lot of my performance to initially targeting very low powered hardware. I ran it on a raspberry pi cluster until the index was around 1 million documents (I think). Having those limitations early on excluded a lot of design mistakes that would have been a lot harder to correct later.

Serving search requests is a embarrassingly parallel operation that could be very easily load balanced to different instances. If so Max Schrems hectored every other search engine out of Europe, and Ursula von der Leyen, taking pity on my living room search operation, sent me a bunch of money, I'd probably be able to deal with 1000x the traffic about as fast as I'd be able to lease space in a datacenter.

I am a bit choked in terms of how big I can grow my index, though. I could probably double, maybe even a couple of times, but not too many times if I can't find serious optimizations.

Hey, I just checked your search engine and it's really cool! I googled my own website and ended up finding a lot of non-samey content.

A lot of startups' products are write-intensive applications with the following requirements:

- strong read-after-write consistency (or their users could receive inaccurate information)

- as-high-as-possible availability (because they lose money whenever the site goes down)

- low-latency operations that are difficult to optimize--such as recommendations, complex full-text search, or large graph traversal

And engineering teams don't notice that their solutions are suboptimal until their product hits scale.

> And engineering teams don't notice that their solutions are suboptimal until their product hits scale.

A lot of engineering teams scale prematurely. Maybe their complicated engineering playground will indeed scale to 100M users, at the expense of being extremely complex and costly from an infrastructure point of view, where as a more "boring" stack may not scale to 100M but will handle up to 10M in a much simpler & cheaper way.

The more complex solution makes sense if you're sure you'll hit 100M, but megalomaniac founders' dreams aside, this rarely happens and they end up dealing with the unnecessary complexity & expense of a stack whose scalability isn't actually required.

Startups don't always want the most cost-effective solution.

A lot of startups' intentions is to look "serious". If everyone has an engineering playground full of microservices with 5 different programming languages all hosted on AWS, you have to be the same if you want to join the "big boys" club.

Sometimes startups don't even care about solving the business problem. The objective is instead to get the "startup CEO" lifestyle for the connections and reputation regardless of whether you end up solving your business problem in the end. Same for a lot of engineers - they'll happily join and build the engineering playground while enjoying their salary regardless of whether it's the right option from an engineering point of view. If anything, an engineering playground gives you the opportunity to manage lots of people and solve (self-inflicted) problems that will look good on your resume in a way that a boring solution with a handful of engineers won't.

On the other hand, if you live and die by solving the business problem the most cost-effective way, all the "scalability" goes out the window and a simple stack on top of bare-metal makes much more sense, because the only thing that pays you is solving the business problem (in the cheapest way possible) as opposed to bragging about your self-inflicted technical complexity at AWS conferences.

I wouldn't discount Hanlon's razor.

There seems to be many people who simply fall for the marketing and think that kubernetes and hundreds of microservices are somehow necessary to run a small to medium sized web service. They can't comprehend how a technology stack with a 20 year track record could be even nearly as stable and reliable as a grab bag of 2 weeks old javascript microframeworks. It's absolutely inconceivable how something that performed well a decade ago on hardware that was a hundred times slower could keep up on modern servers.

And unfortunately the effect flows both ways. I'm having a heck of a time finding a simple, modern log aggregator not designed for kubernetes and infinite scale.

I have 4 load balanced VMs and literally just need to pull the logs into a central location sorted by time. There's a shocking amount of complexity around all the popular tools in this space.

Seems like an opportunity to build one ;-)

The neat thing about simple tools is that they're often relatively simple to build.

Define modern?

RSYSLOG has existed for a long time and is probably the simplest solution for sending logs somewhere.

You can use logstash, without elasticsearch, without kibana. Example: use the logstash file output plugin and have it collect logs (from filebeat, syslog, or whatever you prefer) and dump them into a directory named /logs/<date>/service ... then just use grep or whatever.

I interviewed with a company that was like this. Their Engineering Manager took me through a systems design quiz. The problem was simple:

* 100k users updating their gas / electricity meter readings every second

My first question was what unit was the meter reading in, he said it was the current total (called a totalizer in the biz). I'd worked for a Flow Metering company before. So I said:

"There's no need to upload the total every second reduce the frequency and the problem becomes easier"

Didn't accept that and I said ok a few optimized servers behind a HAProxy load balancer would the trick as the processing is very simple here. The database is the harder part as you can't have a simple approach here is 100k requests / second is going to cause contention for most DB's. You'd need to partition the database in a clever way to reduce locking / contention.

This answer was unacceptable he ignored it and asked me to start building some insane processing system where you use Azure Functions behind an api gateway to process every request into a queue. Then you have a scheduled job in Azure of which the name has changed by now. This job looks at the queue in batches of say up to 1000 records at a time and writes these in bulk into the DB Once it's finished it then immediately reinvokes itself to create a "job loop".

You would create multiple of the "job loops" that run in parallel. Then you would need to scale up these jobs to drain the queue at a rate that meant we could process 100k requests per second.

You would also need something to handle errors, for example requests that are broken somehow. These would go into a broken request queue and could be retried by the system automatically a certain number of times if they went over the number of retries then they would go into a dead letter queue to be looked at manually.

I'm pretty sure this was the actual process he would take in designing this type of system. Use every option the cloud gives him to make some overly complex thing which is probably quite unreliable. I also suspect that's the reason I wasn't hired as I was just looking at the thing sensibly rather than in the "cloud native" way.

He doesn't work for that company anymore and is now back to a normal (non-senior) software developer.

In his hypothetical system with queues and tasks, I wonder how much resources are being wasted on the management overhead (receiving tasks from the network, scheduling functions, the cold start of the Azure function, etc) compared to a boring solution in a cron on a high-performance bare-metal machine costing a fraction of his solution?

Yip, exactly! If anything it's a waste of energy right. A "greener" solution is the simpler architecture :D

I love your initial reflex, which is to question the requirement itself.

In general, this comment is exactly the sort of discussion I'd be interested in hearing if I interviewed people.

He was very upset when I questioned the requirement.

To give him the benefit of the doubt maybe it was an artificial question and the real question is how do you make a large scale "cloud native" system say something like Netflix / Facebook.

Of course the problem with those systems is that it's very difficult to explain it in detail of every part in an interview so his example was just a small part of that system. Also most companies do not require that level of scalability but I concede they are fun to build just not very fun to maintain/operate.

I have no doubt that the architecture I described was something he had personally built as part of what he was doing at the company. Me questioning that was probably taken as an attack.

I never got the job or even a single word of feedback.

Yup. Not even server hardware. If it wasn't for the unusual number of hard drives, 128 Gb RAM, $20 graphics card, and UPS, you could almost confuse it for a gaming rig.

I made a very pretty retro control panel for Kerbal Space Program a few years back, and made a nice writeup on my personal website about it, hosted from a low power Intel baytrail based server I keep in my garage. It got picked up by hackaday, then popular mechanics and a few other similar sites, and hacker news. I vaguely recall my upload rate was about 5 Mbps at the time, and my connection was quite saturated for a few days. The page has half a dozen jpg photos, so I suspect that was the bulk of the data.

I think your math might be off, or I misunderstood: 2mil/day = ~23/s?

You're correct. I was pulling the numbers from memory.

I think it may have been 2 mil requests/day, which would mean 23 rps for 1-5 vps (roughly lining up with the math of 5 requests/visit).

Reply on news.ycombinator.com