Hacker News Reader

Comment by pradn | Hacker News Reader

So we can use garbage-collected languages in trading systems without turning on the garbage collector - just collect it all at the end of the trading day. I'm joking but this is a real technique.

View on HN

Yeah this is dumb but it’s actually par for the course for firms I’ve worked at. Build up all the garbage during trading hours (and on days like this hope you don’t hit resource limits) then start collecting it and rebooting at the end of the day.

Most services are shut down after trading/brought up again before trading.

After trading you would have accounting and other processes that would take hours. These are not done in real time and rely on daily downtime.

I honestly can’t imagine these firms figuring out 24/7 trading hours.

If it's dumb but it works, it isn't dumb.

It is, there was this HFT firm I interned at once that rebooted all their servers at the end of the trading day to make sure they would start clean and fast the next day.

That makes a lot of logical sense. Every single ms counts in those cases.

It reeks of code base problems or general misunderstanding of systems administration. Easier to just make sure you're not leaking memory (profilers exist) and actually monitoring the servers than having to reboot your servers just in case.

Things like heap fragmentation still happen.

It also ensures you can come back up after an unexpected shutdown.

This is the best reason. Knowing your business systems will simply come back with the power, and having that tested on a daily basis will quickly put everyone at ease.

Is this what is called chaos engineering?

Did they also warm up caches before open?

Now I understand why Jane Street uses OCaml. It has less of the GC problems.

I've never used this before, but I suspect this is exactly what you would want to use if you were on .NET and building latency-sensitive systems that see daily reboots:

http://tooslowexception.com/zero-garbage-collector-for-net-c...

If I was building something that was on a short fuse like this, I'd also be using structs and stack allocation as much as humanly possible before leaning onto the "having tons of physical memory" crutch. I feel like virtual memory could cover your ass for a small period of time before the whole thing started to grind to a halt.

Reply on news.ycombinator.com