Hacker News Reader

Comment by hliyan | Hacker News Reader

For ten years, I worked on codebases written in C++ without using any third party libraries (no, not even STL -- we needed faster, thread safe versions of strings and maps which were not available at the time; Boost? What's that?), its own custom messaging middleware built directly on sockets, its on distributed process management system built with unix syscalls (obviously). There was not a single line of HTML, Java etc. anywhere. It was a comfortable, high-paid life and I solved cutting edge problems (HFT, distributed, concurrent computing). In 2013, I forced myself to quit, start with a small company that paid me half the salary so that I could learn the web tech stack from the ground up. It was all to avoid this fate. I'd like to think I've succeeded.

View on HN

I don’t think anyone working with low-level C++ for HFT can face similar fate. Most of skills gained at such job seem very transferable. The article seems to be more about jobs where 95% of work consists in implementing CRUD in accordance with very specific business logic of the company.

The only reason to change such job I can imagine is to sit out your non-compete package. But I am not in this industry so maybe I’m missing something.

Yeah, I find it amazing someone would transfer to standard web CRUD development from that. Sure you can have enough of all the custom stuff, but there are plenty of companies that will hire you for your current salary for non-web work. (you can probably guess how little I like web work)

Same! The poster's pivot reads like horror to me. Like they lost their mind and joined a cult where they worship Memnon with sermons of JavaScript.

> It never sat well with me. Colocated systems saw latencies no mom-and-pop investor could ever dream to achieve. That is one of the reasons why the second company I joined was a non-profit that built systems to help track human rights violations, enforce labor rights, monitor democratic elections etc. Third job was in the food industry and the last one, in housing / property management. So I'd like to think I've made sufficient amends.

Ah, nevermind. Turns out they were worshipping Memnon with C++ and renounced that cult with web tech! :p

It's not that you won't have the skills, it's somehow demonstrating them.

I also worked at an HFT C++ shop that used no STL. Every data structure has a weird name and API that you get used to. The whole architecture has its own idioms, and hasn't needed much upgrading in terms of language features.

Imagine you learn the grammar of English. You learn how to pluralise nouns, when to put an s at the end of a verb, how to to use commas, and so on. You then go to work at a place where they have a different word for coffee, drink, and order. Along with every other word. You end up learning where everything is and how to talk about it, and you're productive.

How do you apply for another job? Chances are you will be asked how to order a coffee and drink it.

Very real issue btw, I still have friends there.

This kind of issue cuts both ways - how do you hire new people and get them productive quickly on a code base where they will almost certainly understand nothing?

Getting rid of weird data structures that now have STL analogs then is a productive exercise - existing devs get to learn more STL and new devs have more familiar ground. You get to delete code too.

I work at a HFT shop and put C++ code is modern (C++23, GCC 13) and uses stacks of third party libraries

Yeah I'm not entirely sold on the NIH story. Plenty of things written by other people will work just fine for HFT.

Unrelated question: do hfts make you move out to east coast, or are there some in California?

HFT firms are primarily in Chicago / NYC.

Firms are mostly in the financial centers but maybe there's a remote role here and there.

There's at least one in the bay area though.

> so that I could learn the web tech stack from the ground up. It was all to avoid this fate. I'd like to think I've succeeded.

I'm not sure if that's intentional humour or not.

Out of the frying pan, into the cow manure.

There is a massive advantage in being able to see the web tech stack for what it is and wielding it without over-engineering or fanaticism, while your competitors are spending 2-3x on monstrosities that essentially does the same thing, but slower.

The day a professor made me implement a simple http server was the day I achieved code enlightenment.

At that level of abstraction it’s all just strings

That was the web, yeah. Absurdly simple. Now all new protocols are binary and require reading several academic papers instead of a single RFC.

> At that level of abstraction it’s all just strings

Exactly... I worry about the masses learning of this and gaining literacy over the landed gentry.

PHP/CGI basically had it solved. I emulate PHP in .NET6 today using verbatim and interpolated string operations.

Relevant: https://twitter.com/serbantanasa/status/1582019078533046272

I was originally concerned about your parent comment regarding your new foray into web dev (that C++ path sounds amazing), but this comment has put my soul at ease.

If you walk into web dev with aggressive principles and a willingness to be controversial, you can get a lot of shit done and it will feel great. Watching others struggle under the various JS framework circuses has almost an opulent/aristocratic vibe in 2023.

Fair point. The web stack is the one thing I actively avoid doing anything significant with for exactly the reasons you've just described.

I have a pretty similar story. I spent 7 years working as a subcontractor for a big telco with CORBA, pthreads and the ACE reactor library. First week, I halted a whole contact center service by adding a sleep call to the codebase (it was on UAT environment). Fast forward several years and I could effortlessly debug cores with thousands of threads, reason about semaphores, reentrancy and even tell the difference between the POA and the BOA Orbix adaptors. I began to become the mythical guy who knew what everything did not because I was technically good, it was because I understood the business and how it mapped to the different architecture elements.

Then I felt like I could spend my whole career there and decided there was a whole world out there in the web, began to learn PHP in 2004 until the mythical "blog in 5 minutes" Rails demo appeared. I switched jobs in 2007 and I have never looked back.

Interesting. I've used semaphores with pthreads (only once -- after that I realised plain old mutexes will do). Like your sleep call, I once brought an entire service to a grinding halt by adding a simple `fsync` call (to fix a recovery failure because once a sequenced message we thought we had written to disk before a crash had actually got lost in the buffer). Understanding things at that level makes any new tech fairly easy to grok.

> I understood the business and how it mapped to the different architecture elements

If you have access to key systems, you can definitely reverse-Conway to understand how an organization really works.

https://en.wikipedia.org/wiki/Conway's_law

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

— Melvin E. Conway

That's so correct, this was Conway's Law in action!

C++ is such a damn chore. This is partly why, everyone always assumes they can do it better, they don't trust any other code, so you end up with monoliths of NIMBY code and esoteric build setups. Code reviews are incredibly painful for the same sort of C++ reasons, everything can effect everything, the whole idea of encapsulation tends to fall apart when memory, time, ordering behavioral effects get relied on after some amount of time usage. Every change is a footgun to be found in production under just the right circumstances.

Hard to be productive when you can't trust the language, the tools, or other developers to be correct.

I did the same, I thought C++ was heading for obsolescence. Now a master of Java, Javascript and can write great full stack apps. Meanwhile HFT C++ jobs are paying double, I'm thinking of going back, even if the systems are boring.

been working on java jobs and easy code bases since I graduated 10 years ago, nothing had been really challenging and I'm not complaining. Would be interested in a job that pays more even do you think working on C++ jobs is more challenging or just boring?

Depends on the job of course. Probably more technical problems and tuning. Often code is low level which is interesting if its your thing but not if you like business problems.

Sounds like your old place had a good idea what's good about c++. It's not the STL and certainly not boost.

Legit, is that place hiring? Sounds like an ideal stack free from the tragedy of the commons.

Those stacks can be full of 'not invented here'. They are unique and have their own quirks. Quirks that you more than likely can not google for any sort of guidance as to what is going on. I wrote one of those stacks a few years ago. No std libs at all. Why? must fit in 32k and most of that space is needed to hold data. That means throwing a lot of those helper bits out or spend weeks figuring out how to bend the compiler to your will and not include everything. When it is easier just to roll your on CS version of whatever. But it sucks because you probably have odd edge cases.

Yeh it's a rough but rewarding life; I spend most of my hours on problems that aren't on Google, or any LLM, so I'm kinda used to it now. (Stockholm)

Would love to know of more spaces which require this level of programming - usually I feel kinda underworked at most shops.

Though, true enlightenment is combining the urge to NIH with some, light, sensible borrowing from those smarter than us :) very much a key skill unto itself.

Game engines have a lot of low-level C++... Often not the best working conditions though from what I hear.

The space I was in was IoT type items. Usually hub controllers. The thing is that is getting less and less as you can get a fairly robust thing now with a couple hundred meg of flash and ram for a very reasonable price. So those 'must fit in 32k' items are few and far between these days. With that sort of space you may as well use the proper libs that are usually nicely documented and all the quirks have been seen by hundreds of other poor souls such as yourself.

Embedded programming is this 99% of the time. Abstracting hardware away is a silly notion. Docker? Cloud? Hard to do when your code only runs at all on one specific custom board. The protocol you wrote to talk to that one chip isn't working? Time to break out the oscilloscope and look at the actual data going to the pins.

Join us, the pay is worse and the work is harder, but also the prestige is lower.

I sometimes really miss embedded programming; it felt like real engineering in a way that higher-level software development often doesn't. Alas, the shop I worked for could not keep me busy enough, the projects began to feel repetitive, and I got really bored; it'd be hard to go back.

> Join us, the pay is worse and the work is harder, but also the prestige is lower.

I tip my hat to you!

I know it's not all about the money, but over the course of your entire career, do you think that move was financially positive or negative?

I'm very roughly estimating that you could probably have worked in the HFT job for around two-thirds the time you're going to have spent working on generic web tech, and still been better off.

Absolutely 100% positive. I was largely an individual contributor all that time, and only did some unstructured, untrained, unguided people & project management. Perhaps because it was such a niche, engineer-dominated environment. Several years after leaving that company, I quickly learned to effectively manage people and projects, architect systems using more mainstream/open source tech and eventually reached exec level (but that's a different story).

That's really some fields I'd love to get into -- less need to talk to people and more need to talk to the machine. What a dream! Alas I only scratched the surface of C++.

Ah, that's the missing punchline: you had ambitions to be something else other than an engineer, and you (quite correctly) recognised that you were never going to grow your wider skills at such a place.

For someone who had wanted to stay an IC, I don't think the move would have been quite as wise. Being in a niche, engineer-dominated environment, and being paid extremely well for it, is nirvana for many lifelong ICs. It certainly feels very different from the experience of Arthur from the linked article.

I hope it worked for you, but I don't see any big difference. Web technology will become dated faster than the old C++ codebase. If you learn some web tech stack today I expect that it will be obsolete in 5 to 10 years. What are you gonna do next? Move to another small company for half the salary to learn everything again and compete with 20 year olds?

What about it would you like to avoid? I don't understand the fears that people have in this thread. To me this sounds like the Platonic ideal of a programming job.

EDIT: I'll reply to my replier here because /u/Dang rate limited me (again), though I must commend him for letting me have one more comment than usual before he pushes the button ;)

> I would say the ending -- the part where you are unemployed and unable to support yourself in your old age because you invested decades of your brainpower into a highly specialised, non-transferrable skill.

I would contend that such a person, unless very reckless with finances or extremely underpaid, should have enough money saved up to make what is mentioned there very unlikely. That said, I know some high-income earners struggle to not blow their entire bank account each month

I would say the ending -- the part where you are unemployed and unable to support yourself in your old age because you invested decades of your brainpower into a highly specialised, non-transferrable skill.

I should think that years of specialized C++ work in hft environments would have come with impressively high pay… in which case one could retire early and self study anything for fun, not hope to find a job for financial reasons.

Ideally, yes! Unfortunately, there is such a thing as being young and not knowing your own worth / not knowing how to negotiate / not having a manager who looks out for your career growth.

Understood.

In that case, if web development is your interest, then you have many possible paths.

The primary choices are full stack, backend, or frontend.

Obviously full stack is most flexible because you can theoretically do it all. But keeping up with so many technologies simultaneously is really tiring.

Take a look at Phoenix framework. It is a fantastic way to get started, and it will grow with you into just about any future project need. Plus Elixir is an excellent language built on top of the incredibly powerful Erlang ecosystem.

The web stack might give you more transferable skills but how did you calculate the trade off vs the 50% pay cut? Was the salary not actually very high and at a low ceiling compared to web work - i.e. you expect your salary to soon exceed what you made working in HFT? Or was there a serious risk of layoffs happening far in advance of your retirement? If the goal is to support yourself after retirement, are you saying in certain circumstances halving your salary 10 years into a tech career ultimately optimises your entire-career-earnings?

It wasn't such a purely material calculation. Less stress, more time to spend with family and friends, learning more things and doing something that you know actually adds value to the world -- those things matter.

That's mostly the guys fault and not the jobs fault.

Even in a bad code base you can start making things better, apply modern techniques and learn new things.

Also most of the skill you gain there is transferrable. E.g. just because you have to work with some legacy PHP project, it doesn't mean that you would suck at creating modern applications in JS. It would just take a couple of weeks to get use to it.

Making things better and applying modern techniques don’t map to a feature story. In the environments being discussed you’ll have to wait for a cadre of architects and program managers to convince the C levels to commit to a multi year, millions of dollar effort to have any hope of improving a code base.

You don't have to rewrite everything, to start fixing things.

Must be nice to live in your world of being treated as a professional.

'So your story ran over again help me understand where we can improve your work flow'.

> Even in a bad code base you can start making things better, apply modern techniques and learn new things

All th bad codebase I've seen were a result of fossils who are still with the company gatekeeping anyone from fixing their code.

A few years ago, I had one architect stubbornly refuse to use source control, rejected all PRs that didn't use his homegrown buggy C++ libs for strings, refused to let anyone do the http server bits unless they used his C++ server and so on.

If there isn't a gatekeeper, you wouldn't have the bad codebase in the first place.

I've got to disagree. not every contributor is making the code base better. I'm in a situation where we are such a small team that there really is no gatekeeper, and we only require one approve per pr...and I've come back from vacation to some monstrosities that I have to look at with no time to fix. After a pairing session with a a fellow developer, where I took the extra effort to refactor something along the way, probably an addition 15 minutes of effort...she quipped- no wonder it takes you so long to finish your stories sometimes. I mean, that's not why- I'm ADHD, so my lack of productivity sometimes is invariably due to a day where I forgot my meds or didn't get a full nights rest.

> "just because you have to work with some legacy PHP project, it doesn't mean that you would suck at creating modern applications in JS"

I disagree; working on a legacy PHP project isn't just opportunity cost (failing to sharpen your knife), it's actively harmful (leaving the knife in salt water, accelerating the rust).

Shouldn't you have ample savings, a private pension and social security?

? In the US? Maybe not. I'd bet a lot of hot shot young programmers think it will be gravy for ever.

Start saving now.

Being good at C++ is very much a transferable skill.

I'm sure he could learn STL (standard C++ library) in a fraction of time it would take someone else to become proficient in C++ in general.

It is a transferable skill. But not one many people want to pay for except in very niche areas these days. So if you are out of a job expect to grind for awhile trying to find someone to hire for C++. Unless you have network connections then you might be ok. Being good at C++ only goes so far, you have to have skills in techs people are willing to pay for. Not many companies are willing to wait while you skill up in their stack unless you are willing to work jr but they will not hire you anyway because 'you will jump ship'. Its not great.

that part is a complete fiction. People who have a lot of experience in the embedded world or performant but esoteric codebases are highly thought after. Even if the literal job isn't transferable, the skills for sure are.

My mother was a COBOL developer and didn't work for 20 years because she stayed home. She recently returned to her old bank she worked for because they are dragging people out of retirement as they can't find anyone else. She earns more than I do now.

People are crazy to give a job like that up for generic web dev work.

Most of my early programming work was with C and C++ (after I left academia, where it was mostly Fortran). I am, simply put, a better programmer for it, objectively so when compared to my peers who've never touched it, at basically every place I've worked at since (including a FAANG-level company, currently, where I'm mostly writing in go and python). At least, that's the feedback I get from them.

I think it's fair to say this skill difference is largely from that work in the C family, where I learned a number of different paradigms for design and development in that world, with all its footguns and low-level "gotchas". It has always been easier for me to parse and understand others' code, identify subtle bugs, to use debugging tools, and to identify some optimizations by examining the code that others might spend days using profilers to find.

Some of that would come with engineering experience in general, regardless of languages used, but not all of it, but I picked these skills up faster and earlier in my career, thanks to that early work. Skill with C++ is definitely transferrable, in at least some cases.

Are you aware that rate limiting is not something a moderator goes around doing to a single person, but instead a server managing its own workload?

Hacker News has per-user configurable posting rate limits. Moderators use them.

> learn the web tech stack from the ground up

Ouch. Should have stayed in legacy C++.

> in C++ without using any third party libraries (no, not even STL

As God intended it!

Was it something that ran in an embedded environment?

High frequency algorithmic trading. Often ran on colocated hardware within the exchange / trading system data centers. Started with stuff like Sun SPARC, Solaris and Oracle and we slowly made our way to Suse, Intel and... surprise, our own database optimised for fast writes.

Bloomberg? Comdb? If not, sounds very similar, anyway. Been there, done that!

Naive question: is this like creating fancy tech that is helping rich people to front-run the poor and middle-class?

Because that's what a lot of people think online. Maybe it's more nuanced. I'm curious how someone from that carreer sees it?

No, the poor and middle-class are not day trading stocks in any sort of meaningful volume.

It is to help the super-rich who own hedge funds to frontrun the rich upper-middle-class people with significant disposable income.

Unfortunately, yes. It never sat well with me. Colocated systems saw latencies no mom-and-pop investor could ever dream to achieve. That is one of the reasons why the second company I joined was a non-profit that built systems to help track human rights violations, enforce labor rights, monitor democratic elections etc. Third job was in the food industry and the last one, in housing / property management. So I'd like to think I've made sufficient amends.

I wonder why mom and pop are investing in individual stocks at all. Buying the index has been a passable "get rich slow" scheme for me.

The whole business model is misunderstood by your average internet commentator.

When you have a market, you need someone to be there to provide liquidity. Imagine if you're a farmer and you show up to the market with your wheat, but all the bakers have gone home that day. Or the baker shows up and there's no farmer. The market maker stands around all day offering to buy and sell so that you don't have to wait for the guy you're really trading with. Of course this middle man wants to get paid for it, but your cost as an average Joe is next to nothing. This is trading in time.

Now imagine you want to cook a meal and your ideal meat is beef, but actually you're ok with pork, so long as the pork is cheap enough to make it worth it. How much cheaper should it be? Well your fellow who knows all the pork and beef people will be able to gauge where the balancing spread is, given the amount of interest. In fact he will from time to time do the trade when the spread is out of line. This is trading in space.

So why all the fancy tech? After all market makers used to stand around in a pit in a colored jacket, and they didn't have degrees. My first boss in the market was one of these guys.

Well, things have gotten very tech heavy because as soon as prices are out of line, there is money to be made. Or rather, lost. As a market maker, you are constantly out there with your prices, offering to buy or sell at a very small spread. If some news happens that affects prices in a big way, you can be sure that you will buy when it's going down and sell when it's going up. In order to both have tight prices and avoid this "adverse selection", you really want to be able to react as quickly as you can when your system decides that something's up.

Ah, the ol' liquidity spiel. One wonders how markets were even possible before HFT...

I explained that. It doesn't matter whether it's a computer doing it or a guy in a jacket, but once one guy gets a computer all the market makers need it.

So take away the computers. Or more realistically, batch run all orders once an hour.

Retail consumers aren't exactly disadvantaged by the gradual narrowing of spreads, so what are you trying to fix? The latency arms-race is exclusively professionals fighting each other.

Getting back on topic, being a software developer for an HFT shop can be a lot of fun: serious technical challenges and a rapid feedback loop in a role where you may be the revenue stream rather than just another cost center. But posters highlighting NIH syndrome are spot on: you risk pigeonholing yourself.

Markets were possible. But what about the current offering of ETFs and similar products? Despite all the fancy talk, many HFT firms are just market makers for the products that enable average joes to invest well and cheaply.

You don't have to be that old to remember how terrible the experience used to be, and how easily even sophisticated individuals were ripped off at every point of the process.

Was that like Gomez Addams with the paper ticker coming out of that glass domed contraption, calling his guy to buy Consolidated Lint?

> You don't have to be that old to remember how terrible the experience used to be

I guess I'm not old enough. What was the system like? How did sophisticated traders get ripped off?

You could easily solve all of this without HFTs - you might introduce a few seconds of latency but virtually no real investors care about that level of latency and it would save them billions per year.

Would save money for the market makers too. Assuming someone couldn't get an advantage, and that's the rub.

Yes you could, but the point is that HFTs don't create any new problems that need solving. They don't make anything worse.

> and it would save them billions per year.

This is not true.

It absolutely would if you remove the cut HFTs take out of the market. I'm sure you'll make some argument about liquidity and spreads but, in the aggregate, the money from higher spreads goes to market participants so the market as a whole still saves by eliminating HFTs, even if the spreads go up a bit.

This is like saying if we eliminated millimeters from rulers, and only used centimeters, everything would get closer together.

HFTs do not take a cut out of the market, their cut comes 100% from other market makers. Those are the only folks who would benefit from eliminating HFTs, and unless you’re one of them, who cares?

I don’t care who buys when I am liquidating a position. That’s the whole point of a market. Therefore I don’t care about HFTs.

"Your average internet commentator" is ignorant by choice. No argument or description will convince them.

You're describing market makers, not all HFT participants.

True but trying to keep it simple.

Generally, it's not about front-running normal people. Instead its racing other HFTs. Either a race to capture arbitrage opportunities, or to update your orders on one market based on trades made in another market.

The aim of the game is to collect a small spread on a lot of transactions.

The scary thing for an HFT like that is being 'run over' by a big party selling or buying a lot over a day. If a pension fund is dumping 10٪ of their holdings of shell, it's hard to collect a spread because you need to find buyers for every stock you buy from them. Meanwhile the price is dropping as they sell off.

Hence you occasionally see 'front running' with price improvement. Instead of buying from the market at 1.02 a HFT gets to sell to you at 1.01, yielding you a better price. This is a good deal for the HFT, because a normal person isn't going to run the HFT over. And it's a good deal for the normal person because they get a better price.

No, it's a zero sum game. HFT firms fight to be the first to pick up a penny, but they can't force anyone to drop more pennies. They don't cause prices to go up more than they would have otherwise, or extract more value from the poor and middle class.

And now you are jumping from one "stack" to another, none the wiser, your knowledge hopelessly outdated just a year in. The principles you learned in HFT? Those apply all the same.

So you left a job that paid 2x and exposed you to hard problems that require reliable solutions for a job inside the horror clusterfuck that's the modern web where nothing really "works" and there's no real engineering to speak of besides monkeys throwing crap in a shared pile and shovelling it out the door ASAP.

Good for you I guess?

Most people I know, inc. myself, would call this a "bruh moment".

I can't fathom willingly entering the web mines if you've already sunk in the hours to master high perf native code, OSdev, dist systems etc. People usually enter web to avoid having to learn such things.

To discard that knowledge and instead compete with an endless stream of fresh grads, boot camp grinders and LLMs for less money? A madness if I've seen one.

Fun fact: I'm actually a bootcamp grad in web dev doing the opposite. I agree.

Speaking as someone who does both:

Web tech is more immediately rewarding. With C++ (and especially with modern web dev agile processes), you can spend a week building enough framework and infrastructure crap to stand up the barest inkling of a demo of the solution you plan to implement, only to have the whole thing pivot out from under you and become wasted work.

More complicated languages and frameworks that manage memory and do UI and everything for you let you get to a prototype much faster, so you can see whether it'll work or whether you want to toss out the idea much sooner.

Tell me you know nothing about web dev without telling me you know nothing about web dev

It's hyperbolic but it's a common sentiment that I tend to agree with. The web dev world changes too fast and too often, many stacks involve absurd complexity for dubious performance and maintainability and other metrics.

I've been doing web dev for 20+ years. Which part is wrong?

It's not hard to find feature support or implementation differences between browsers. But these days, that's not really the problem. I can go read the ES language specification, and generally find user agents/node/deno/bun have compliant implementations.

The problem is brittle "ecosystem" stuff. How are the create-react-apps going? How is module bundling going amidst the UMD/CJS/ESM war?

HFT is still CRUD, CRUD for the extremely wealthy.

If you want interesting, go to HPC large scale simulation.

The arrogance you are showing is disgusting.

Even if the decision he made was a bad one, you don't know anything about why it made sense for him at the time.

People aren't algorithms running in complete isolation and silence. They live in nuanced circumstances you can't possibly understand by his short history in a comment.

I hope you can find understanding and peace in the future.

Damn so it's not just me having that impression. Guess I better steer towards other things..

That first place sounds awesome

> so that I could learn the web tech stack from the ground up. It was all to avoid this fate

same fate

> I'd like to think I've succeeded.

possibly not

Reply on news.ycombinator.com