A lot of complex “scalable” systems can be done with a simple, single C++ server
twitter.com
twitter.com
This is way less in HW than most people in the trade (from web devs to devops) seem to think when asked about it.
SO ranks #36 in Alexa right now: https://www.alexa.com/siteinfo/stackoverflow.com
(SO and DailyWTF are very much into Microsofts ecosystem)
There's significant difference between ASP.NET and ASP.NET Core
https://meta.stackexchange.com/questions/316278/the-road-to-...
I guarantee you the bulk of traffic to SO is hitting a page and performing zero writes.
Yes it's read-heavy but there's still plenty of work done in assembling a page. It's definitely not as simple as caching at the CDN edge for every hit.
Ultimately, their service is (most likely) not compute-bound.
Looking at the specs of the machines, they are actually pretty basic as far as servers go. A server isn't barely worth the cost of it's chassis and motherboard if you put less than 64 G ram and 24 CPUs in it.
In other words, these are about the lowest specd proper servers you can get. So yeah, even their modest hardware is still overspecd for running their website.
Also, of course, the total cost per hour for 9 servers with 3 year depreciation is in cents.
this isn't a lot. Helps that site is mostly text data.
I know of a relatively small cloud security system that transfers petabytes/month to/from a handful of customers.
4 ingest pods, 8 pipeline pods, 7 time-series db servers, 2 sql servers
Compared to other similar sites like Reddit or Quora which are far slower yet running on more hardware, it shows what proper efficient architecture and code can do.
It would rank way higher if they would be more webscale. It needs more Kubernetes, GraphQL and Golang /s
For example, at work, our entire analytics ingest workload (HTTP) for a few hundred million users runs on 8 core VMs on GCP, written in Rust/Go, each node doing ~40k events/second.
Our RTC infra sustains >1m PPS per node on 4 core 3.9ghz 2014-2017 xeons.
Rust is slow to compile so it breaks my deep work when programming, also it costs me a lot in CI/CD. Also the Rust type system make implementing some things really hard.
For example I wanted to implement json requests logging. It took me more than 1 day in Rust, less than 2 hour in go.
To quote Rob Pike:
The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.
Go is aiming for the same niche that Python is. This has the accidental side effect of making it faster to write software in it than most of the ALGOL-derivatives. To quote Eric Raymond (sorry) on Python:
When you're writing working code nearly as fast as you can type and your misstep rate is near zero, it generally means you've achieved mastery of the language. But that didn't make sense, because it was still day one and I was regularly pausing to look up new language and library features!
Go and Python both allow for expression about as fast as you can type, by virtue of being designed for [children in Go's case, shell scripting in Python's case].
It's not a value judgement of Rust or anything, but Rust wasn't designed with the same goals in mind.
This may be a problem of what's idiomatic in each language, as opposed to a matter of language design per se. After all, Rust development can be made at least as easy as, e.g. Swift, simply by adding enough uses of .clone() and RefCell<>. Is this suboptimal? Of course, but it will still be plenty faster than Python, and perhaps even faster than Go.
Compilation time is a separate issue which apparently OP found problematic. It's being dealt with (for non-release optimized builds) via the cranelift project, which is a Rust-specific backend much like the Go compiler, with no reliance on LLVM.
It's absolutely a matter of language design. Again, I'm not criticizing Rust, but a language explicitly designed to be trivially written by anyone, fast, is going to be easier to write in quickly by anyone. Imagine making that comment about Logo instead of Go. Of course Logo is more painless to write than Rust! It's a child's language. So is Go!
Rust development can't be made as easy as Swift (and it is very unlikely that what you described would be faster than Go). Even Graydon Hoare agrees that Rust is inadequate in comparison to Swift in terms of development ease:
https://www.reddit.com/r/rust/comments/7qels2/i_wonder_why_g...
I'm no stranger to languages with vastly different idioms than normal (I write APL daily), but not all languages are as quick to develop in as every other, and pretending they are is silly. Rust has some innovations, and it's by no means a bad language in itself, but pretending it wins at everything under the sun (even things it's not trying to do) doesn't reflect reality or the perspective of the original author.
"Write as fast as you think" is a far better way to program than "Write much slower than you can think."
Eric Raymond has written some pretty substantial things, and he's not as clueless (on programming, at least, the rest of his views are...no) as you're implying.
The idea that intuitive languages are the only ones you should do development in is absurd. A single line of K can do what a hundred lines of C can, and you can write the line of K substantially faster than you could write the C to match. K only has something like 50 primitives. It's simple enough that you can keep it all in your head at once, and that allows you to develop much quicker than almost any ALGOL-derivative. Taking your comment at face value, everything written must be boilerplate. Looking at reality paints a different picture.
Good languages manage complexity in a way that allow you to express complex things in simple terms. That the languages you seem to be familiar with only allow you to describe simple things in simple terms isn't something that's inherent to every programming language. I'd recommend giving APL, J, or K a try.
To bring up k again, the language has already done about the maximum amount of abstraction possible. There's really no room for the programmer to make reusable abstractions; any useful ones have already been made. That allows you to do very useful things in very small amounts of code. Picking a random example, here's a complete Sudoku solver in 75 bytes:
We've become so accustomed to extremely inefficient software systems that we've lost all perspective on what is possible.
If your latency requirements are slack, then you can get away with one machine, because you can reboot or reprovision it and carry on processing without meet your requirements.
If your latency requirements are tight, you don't have time to fail over anyway, so you might as well run one machine and make sure you can deal with failing to meet your requirements.
Go back to two when maintenance is done.
Are you saying this can be optimised to fit inside a single 10 core server in terms of compute loads?
You use a cluster when your data and compute requirements are large and parallel enough that the tax paid on network latency trumps the 10-20X speedup you get on SSD and 1000X speedup you get from just keeping data in RAM.
250 Gigs is tiny enough that you could probably much get better performance running on high memory instance in AWS or GCP. You'll generally have to write your own multiprocessing code though which is fairly simple - your existing library may also be able to support it.
I once actually ran this kind of workload on just my laptop using a compiled language that performed better than pyspark on a cluster.
However, even if I could load it all into memory at once, and assuming it takes 200 gb, I'm still using a master's student access to a cluster. So I get preempted like it's nobody's business. Hence why I prefer a smaller memory footprint even if I take up cpus at variable rates through a single execution.
I did try to write my own multiprocessing code for this, but the operations are sometimes too complicated (like groupby) for me to rewrite everything from the ground up. If I'm not reliant on serial data communication between processes (like you'd need to sort a column), I can get it done pretty easily. In fact, I wrote my data cleaning code with this and cleaned up the entire file in half an hour because single chunks didn't rely on others.
However, if you have some idea of how to run these computational loads in parallel in python or any other language on single compute instances (like the size of a laptop's memory of 16 gb), I'd really love to see it. Thanks.
I do have fast SSD storage because it's on the scratch drive of a cluster and from what I've seen it can do ~300-400 MB/s easily. I haven't had a chance to test more than that since I'm mostly memory constrained in much of my testing.
My current attempt is to push this data into a pure database handling system like SQL so that I can query it. But like I said, I work with a less-than-stellar set of tools and I have to literally set up a postgres server from ground up to write to it. Which shouldn't be a big deal except when it's on a non-root user and I have to keep remapping dependencies (took 5-6 hours to set it up on the instance I have access to).
My other option was to write the entire 250 GB to a sqllite database using the sqlalchemy library in Python, but that seems to fail whether I do it with parallel writes, or serial writes. In both cases, it fails after I create ~64-70 tables.
> Generally speaking, Dask.dataframe groupby-aggregations are roughly same performance as Pandas groupby-aggregations, just more scalable.
The dask.distributed scheduler can also run on one high-RAM instance (with threads or processes) https://docs.dask.org/en/latest/setup.html
Pandas docs > Ecosystem > Out-of-core: https://pandas.pydata.org/pandas-docs/stable/ecosystem.html#...
Reading from Parquet into Apache Arrow is much faster than CSV because the data can just be directly loaded into RAM. https://ursalabs.org/blog/2019-10-columnar-perf/
If you have GPU instances, cuDF has a Pandas-like API on top of Apache Arrow. https://github.com/rapidsai/cudf
> Built based on the Apache Arrow columnar memory format, cuDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.
> cuDF provides a pandas-like API that will be familiar to data engineers & data scientists, so they can use it to easily accelerate their workflows without going into the details of CUDA programming.
Dask-ML makes scalable scikit-learn, XGBoost, TensorFlow really easy. https://dask-ml.readthedocs.io/en/latest/
... re: the OT: While it's possible to write C++ code that's really fast, it's generally inflexible, expensive to develop, and dangerous for devs with experience in their respective domains of experience to write. Much saner to put a Python API on top and optimize that during compilation.
There are a few C++ frameworks in the top quartile of the TechEmpower framework benchmarks. https://www.techempower.com/benchmarks/
Hardware/hosting is relatively cheap. Developers and memory vulnerabilities aren't.
"Dask on HPC, what works and what doesn't" https://github.com/dask/dask-blog/issues/5
Maybe you should spend some time developing a job visualization system for end users from scratch, for end users with lots of C, JS, and HTML experience https://jobqueue.dask.org/en/latest/interactive.html
You can create memory mapped ndarrays, these act like normal numpy arrays but don't need to fit into RAM. Numpy maps the array to a binary file on disk. The array otherwise acts like an ndarray so you can build a DataFrame with it. Whenever you access an array index Numpy in the background (essentially) seeks that many values into the file to grab the value of that index.
Since you're on a fast SSD and Numpy is fairly smart you'll be able to access your arrays close to your drive's speed. It's slower than if the whole database was in RAM but far faster than distributing the data over a network to a bunch of worker nodes. Memory mapped files let you have array-like access to data on disk as if it lived in RAM. When building a pandas DataFrame from a memmapped ndarray I believe you just need to set copy=False in the constructor for it to Just Work.
I don't know what your data looks like but I doubt loading it into SQLite is going to improve your performance.
Your second paragraph is essentially what I want. I'm willing to wait a day for code that may run in 1 hour from memory, so time isn't entirely an issue unless it's starting to bleed into weeks. The read_csv function in pandas has a parameter called memory_map, but when I tried using it on a smaller 7GB dataset, it read the whole thing into memory (32GB instance) even when I set it to True.
SQLite is definitely not my best option here. It was the only server-less implementation I could find, so I tried to use it and it didn't work. However, a database like implementation will be helpful because each operation I need to do will require data that satisfies certain timestamp and arithmetic conditions. I figured it'd be best to load the whole thing into a db and query it for every operation to train my model.
Nowadays there's another reason to use clusters: to autoscale your expenditure wrt workload. A little inefficiency might be acceptable if you don't have to pay for a huge beefed up server idling at, say, 30%.
Biased: you might want to look into DOE libraries. For IO I suggest ADIOS2 [0]. There's python bindings too.
One of the biggest things you can do is use a different storage type like BP (ADIOS) or hdf5. These are readable but binary. But to really determine how to speed up your problem you have to know where the bottleneck is. Is it IO or compute? With 100 workers (threads or nodes?) you aren't highly parallelized. I mean that could be a single node if it's threads.
I'm curious as to why this seems like a common mistake (I've seen it a few times already in this comment section).
If you have some good indexes and do some push-down work (give the database aggregation tasks to do instead of your python code), you should probably be more than fine.
For a 250Gb file.. should be ok.. maybe add some partitioning too.
https://pandas.pydata.org/pandas-docs/version/0.22/generated...
https://pandas.pydata.org/pandas-docs/version/0.22/io.html#w...
Obligatory read: https://aadrake.com/command-line-tools-can-be-235x-faster-th...
I'm currently employed to write software which does DNA analysis. DNA is known for being Big Data.
Some applications are very compute intensive and others are very data-relation intensive. The very compute intensive applications process about 1GB of data in about 30 minutes on a 32 core Xeon 6xxx with 32GB of RAM assigned to it. The data-relation intensive application processes about 350GB of data in 15 minutes on a single core of the same CPU but with 700GB of RAM assigned to it.
Both are heavily optimized but in different ways. So without knowing more about what you're doing with that data, it's hard to say.
The compute-heavy workload calculates edit distance [0] of short paired-end sequenced DNA [1] vs the human genome [2]. There is open source software to manipulate the FASTA/FASTQ [3] and SAM files [4] and run the calculations [5]. The aligned file is processed in a couple of minutes to genetic variation report [6] which is used for some of the analysis products that were purchased. One popular product will give you a haplogroup [7] which basically tells you where you are in a genetic tree.
The relationship estimator uses a different sequencing technology and basically consumes a CSV file from the sequencer's manufacturer. It uses a proprietary algorithm to calculate centimorgans [8]. That then gives relationship estimates between you and other people who've purchased the product.
[0]: https://en.wikipedia.org/wiki/Needleman%E2%80%93Wunsch_algor...
[1]: https://en.wikipedia.org/wiki/Next-generation_sequencing
[2]: https://en.wikipedia.org/wiki/Reference_genome
[3]: https://en.wikipedia.org/wiki/FASTQ_format
[4]: https://github.com/samtools/hts-specs
[5]: https://en.wikipedia.org/wiki/List_of_sequence_alignment_sof...
[6]: https://en.wikipedia.org/wiki/Variant_Call_Format
While I hope that each 250GB chunk has everything in order and I can separate it cleanly, I don't trust it. So I broke up all the files by stock in an embarrassingly parallelised code I wrote and got 500 separate files. However, the problem is, I'd want to process these 500 files parallely so each stock gets only so much memory (and hence, my original constraint remains). And for each stock, I want the data to be queried on timestamps, so I need a way to quickly say "I have data at 9am on Thursday, I want data from 4pm Wednesday to 8am Thursday to create a model".
I figured the best way to do this efficiently was to have the code create a massive database for all the stocks and query it efficiently in SQL. But I'm stuck there due to a lack of tools.
I have no doubt I could solve your problems. I honestly don't care to do so here though.
I imagine there's a software development/engineering team at your work or school you could ask for guidance though.
That's fair. I did not think it'd be such a difficult problem when I first set out to do it myself. But every single turn leading to a dead end kinda bummed me out. I'm going to finally resort to the database method of storing all the data in one query-able file and work off of that.
Segmenting your raw data and using memory mapped files will let you work with large data sets without needing huge amounts of RAM. From there it's a question of your single system's processing speed and IO capacity. This is only necessary if your processing needs random access to the entire dataset.
If your CSV data is more like a streaming data source, you're processing each record as it's read in, you can just stream it in through `stdin`. At 1GB/s you're looking at five minutes or so to process your 250GB of raw data. A SATA SSD might take twenty minutes to stream that raw data.
It's important to note that often your disks aren't directly attached to your compute. That's frequently the case in (particularly cheap) cloud instances.
It's also a domain where you can buy an off-the-shelf desktop for a few hundred dollars to do the work. That's the thrust of this whole thread, because scalable "cloud" systems exist and look cheap people obsess about throwing more instances at problems.
Modern commodity systems are ridiculously powerful and far more capable than people tend to assume. Even "the cloud" gets underestimated because people look at the low end cheap instances and assume they need to spin up hundreds of those when one beefy image for a short duration could do the same work.
Nit: 1GB/s is ok, not even solid let alone fast.
A fast SSD pretty much saturates 4x3.0 links (which explains why they universally tend to cap out at 3.5GB/s). In fact there are now a few PCIe4 SSDs (e.g. Corsair's MP600) which close in on 5GB/s.
https://www.usenix.org/system/files/conference/hotos15/hotos...
In my experience these problems are definitely not always toy problems while some of them can, easily, be ran on 1 server for the expected lifespan of the company because the company will never get that much users/clients/data even though immensely profitable.
Not everyone (almost no one) is a FAANG and I find it offensive to use the company money (which can be investor money) to realise the wet dreams of the cto while the case for that architecture and the cost of it, makes 0 sense to the business and bottomline.
Obviously in life, things are nuanced and differ case by case but nah, real world problems are, generally, not more complex. They are more complex in some (rare) cases, like the one you named. But most companies are not doing anything like that, but they do have their cto’s gearing up for that incredibly unlikely future.
1) We always wait for database, so performance of our code does not matter. This comes from places where ironically the whole database can fit into RAM. 10+ gbps connectivity - well it is almost commodity for business so the latency is not much of a bottleneck. Fast IO to store data - well imagine array of Optane drives. Not very cheap but really peanuts for normal business. All this means that purposely built data server which is a heart of many type of business residing on decent computer, can be blindingly fast and most of the businesses will never ever outgrow it. I've written quite a few of such servers serving loads of businesses in NA so hopefully my experience is not irrelevant.
2) Scripting languages are so convenient to use and save so much time and this is what matters as the salary is a main expense. - Personally I use those mostly for management/deployment/etc.etc type of scripts and maybe to quickly test some small ideas. I clearly see their benefits there. Anything that resembles a product that would actually run business. Sorry but experienced developer can implement those just as fast as in any scripting language and it will save a ton on maintenance.
3) And finally do not do premature optimization mantra . - In my opinion switching from scripting language to something like C++ has nothing to do with premature optimization. Developer can be very productive with compiled languages as they also have megatons of libraries for any imaginable task.
This is just a no true Scotsman argument. For 10 years I’ve watched python/ruby shops drastically outpace projects in C++/Java shops. What you’re failing to realize is how trivial most apps are and how fast fully functional back ends can be created with frameworks in those languages (django/rails).
Your bit about maintenance is also bullshit. I’ve spent entire days peeling apart complex C++ code bases to make a small change to some core abstraction. Ability to maintain code is entirely up to how well organized and documented it is. It has nothing to do with “scripting”.
A vast majority of web dev is incredibly simple. People tend to easily convince themselves that their app is very special and requires a lot of complex solutions to make it work. There’s a lot of incentives that lead people to that conclusion, but most of the time following the golden path laid out for you by an existing framework is going to produce a better product with much less effort.
Sorry but I have the exact opposite experience. First of all I would not call line of business applications incredibly simple. They're rather quite complex business rule wise. I saw floors filled with web developers constantly writing /rewriting endless stream of scripts often without any meaningful attempts to organize the code. In one example I was writing Python (yes guilty it is good for this type of tasks) scripts to process literally thousands of files of their source code to find and properly replace database access methods of which single app had not less then 5.
Ability to maintain code is entirely up to how well organized and documented it is. It has nothing to do with “scripting”
This I can 100% agree with
My guess, essentially holding their finger to the wind. Most orgs can't afford to do real engineering.
We were going to go all in on Snowflake with everyone on the team being for it. I sat down, read the original whitepaper, wrote a simulation of what the costs would look like with the current read/write statistics and tested a small batch of data on it to double check.
Turns out we would have paid between x500 to x10,000 what we were currently paying for using Postgres. I moved on three months ago, and last I heard they were trying to use Snowflake again.
Or perhaps not only there is market, but that's more or less what everyone is already doing.
I could manage my own server and fine tune everything, or I can throw it on snowflake. Snowflake means I've spent almost no time managing anything and it was costing less than an aws box running postgres but absolutely blew it out of the water performance wise. Depends on your workload but it's been perfect for one of my use cases. If they were just using postgres under the hood and I got the same en experience - fine.
Big things are time sharing resources and management/updates/etc. Can you charge me 10x the underlying cost but let me pay for two minutes on a wildly powerful machine? Great, that's a net win for me.
I’m a C++ veteran btw and understand the point but big data is about how to process petabytes of I/O not how to consume CPU.
> I’m a C++ veteran btw and understand the point but big data is about how to process petabytes of I/O not how to consume CPU.
I'm not sure this is cut-and-dry. Back when I was working on Spark workloads, there was some interesting research being done on where the bottlenecks were for jobs. I think it turned out for a lot of jobs, infinite disk / network io didn't give as much of an improvement as you'd expect.
https://databricks.com/session/making-sense-of-spark-perform...
These are not big data problems, however, by definition.
There's an incredible number of gigabyte and even terabyte scale problems that are consuming racks of blades when a little thinking and understanding of the problem being solved can be done pretty nicely on far fewer resources.
What's really happening is that to many people they think it's "easier" to simply rack more equipment into the cluster and end up shifting the complexity into cluster administration rather than programmer time.
Sometimes, that is the right thing to do. The problem is when the cluster solution also adds to the programmer-time complexity.
In our case latency and real time demands mattered a lot (NASDAQ feed parser), to the point of the (potential) slowdown of a garbage collector kicking in was enough to rule out Java and .NET. It runs entirely in memory and on 64+ cores.
We implemented our own reference counting system to keep some of our sanity, and to at least try to avoid the worst memory leaks.
This was an edge case, and for almost everything else you’re probably better off implementing it in something that handles memory for you. If performance is an issue, least try it in Go or Rust with a quick PoC before jumping the C++ wagon.
This is because the allocator itself can have an unbounded runtime that takes milliseconds, causing you to drop.
In the past I've replaced malloc with an implementation that asserts if called on certain threads after init time.
For constant time allocations, try TLSF: http://www.gii.upv.es/tlsf/
It's one benefit of a gced language with compaction, allocations are typically bounded (except when they trigger a gc).
Writing a trading system in Java is harder than C++ imo because where before you had an allocation problem, now you have a multithreaded randomly stalling allocation problem.
Virtu did it but everything I’ve heard about it nullifies the benefits of using java in the first place.
Still though, probably makes sense to do it in a lower level language. It's just far easier in C++ to decide that "Hey, you know what, I just want a big memory block that I control".
I've even heard of game devs doing things like having per frame allocators. They get super fast allocation because they just pointer bump and at the end they simply reset the pointer back to location 0. I'm sure trading systems could do something similar.
I don't consider it realistic to think that garbage collected languages might ever be usable in the context of game engines or HFT.
Game engines are already written in GCed languages. Java in particular.
You may be right that a 10ms pause is unacceptable for HFT. However, for a FPS, 10ms is more than acceptable. It translates to 1 or 2 missed frames in the worst case.
A bigger issue with using Java in particular for games is it's lack of value types. Writing high performance code for java is just that much harder because the language gets in the way.
Microsoft XNA and Unity's .net support were also pretty popular but the popular games written in those languages (like Bastion) didn't have many heavyweight assets or allocator pressure.
On the other hand imagine the security issues if shared_ptr was not thread-safe - you could just not reliably destroy a shared_ptr in any thread.
If you really know that you're going to get a graph of shared_ptr that do not move of a single thread, you can use boost::local_shared_ptr explicitely.
While it can’t (yet?) be generic over them, it lets you use the non-atomic Rc in thread-bound structures and know this is never going to be shared between threads, whereas the more expensive Arc can have handles be moved from one thread to an other.
[1] https://github.com/duneroadrunner/SaferCPlusPlus#reference-c...
[2] https://github.com/duneroadrunner/SaferCPlusPlus#tasyncshare...
This means people frequently need to reinvent their own reference counting mechanisms.
Rust does this right. Rc and Arc.
As with everything in the C++ STL, if you care about bleeding edge performance then you should be prepared to use more perform any (and less generic) components, whether it's data structure implementations or shared pointers.
I used sharedptr extensively in a game engine. Whoops: suddenly 20% of the frame time was gone, never to be recovered. Once that performance is gone it’s almost impossible to get it back, short of rewriting every system.
It was never actually an issue
You should not be doing any memory allocation in the critical path anyway, so garbage collection is hardly a relevant factor.
With modern tools and APIs it should even be possible to write alloc-free work loops in C# (though you're probably writing pretty alien C# at this point). AFAIK it'll remain impossible in Java until the "value types" effort bears fruits.
A few months back a study of sort made the rounds implementing a network driver in multiple languages (Ixy), C# did extremely well in it (better throughput than Go and near competitive with C and Rust at higher batch size though it was way behind on latency) while Java was pretty much in the dumps.
In one of the reaction threads (don't remember if it was on HN or Reddit) one of the people involved explained the discrepancy between Java and C# by not being able to go under ~20 bytes of allocation per packet forwarded in Java.
There were other odd / interesting results from the effort e.g. Rust was slightly slower than C, in investigating that they found out Rust executed way more instructions (especially significantly more stores) but had significantly higher IPC and much higher cache hit rates.
ixy also tried just converting the C code to Rust (using C2Rust) then compiling it and that turned out to be very slightly faster than the C code, which was funny: https://github.com/ixy-languages/ixy-languages/blob/master/R...
"Capable" is plausible.
Sorry, I doesn't make the rules.
But the fact of the matter is quite simply is that any single physical server solution will never be satisfactory for backups or HA purposes.
You can't store your backups in the same place as your data and call it good - what do you do when you have multiple disks fail and your RAID can't be rebuilt? This happens. What do you do when operator error accidentally destroys the array? This happens. What do you do when there's a datacenter fire and the server burns up? This happens. None of these things should be a business ending event, but if you only have a single server handling quintuple duty, that's what it has a real chance of being.
If you need HA, a single server isn't good enough even if you've got multiple VMs running the service. What do you do when the utility power is out and the generator fails to kick on properly or they run out of fuel? Both of those happen pretty frequently. What do you do when there's a network outage at the DC? This happens. When someone fucks up BGP somewhere and now the prefix you used is being routed to god knows where? This happens. When you have any sort of physical server failure that brings your single box down? This happens. Any of those situations will take you offline and render your HA meaningless.
I'm not saying that any of these things are impossible to do when colocating your hardware - but they're not free, and they're not mentioned even at a high level in this story. And since we're not talking about running benchmarks on price/performance and instead talking about a service that a business needs to keep available to their customers to make money, these are important aspects to talk about. Everywhere I've worked, keeping our services available and being able to recover from hardware failure are far more important priorities than being able to optimize performance. And doing those things properly takes more than one physical server.
Each and every one of your disaster scenarios is handled by the architecture. Each an every one of your disaster scenarios has happened and we've lived through them, as well as after the fact reviewed and optimized how we handled the events. As you are, we're professionals.
If you're backing things up to the same server, that's not enough. If the HA instances are running in the same server, that's not enough.
If there are other things besides that one physical server and it's power/network, you didn't include them in the cost, so the comparison is disingenuous there.
Awesome! Really glad to hear this is the case. But those are all added costs beyond the single physical server you gave the price of.
Server cost * number of physical servers you have deployed Cost of your off-server storage Cost of your network appliance doing load balancing
It's still probably way less than the AWS bill, but it's not really fair to compare the total price of infrastructure in one environment vs. just a portion of the other.
Back in 2010 I worked for a company whose system, in Java, ran on a single web server (with one identical machine for failover). We laughed at our nearest rivals, who were using Ruby, and apparently needed 60(!) machines to run their system, which had about 5x the average request latency of ours.
Then traffic doubled, and suddenly we were having to buy six-figure servers and carefully performance-optimize our code, and our rivals with the 120 Ruby servers didn't look so funny any more. And then traffic doubled again.
e.g. https://www.iea.org/reports/tracking-buildings/data-centres-...
"Global data centre electricity demand in 2018 was an estimated 198 TWh, or almost 1% of global final demand for electricity (Masanet et al., 2018)."
Isn't that the conceit underlying this whole argument, though? Many systems won't ever get to the point where you have to scale in that way, if you build them efficiently in the first place.
Perhaps more importantly, for many applications, you'll be able to see the limit coming some way ahead, and if you've reached a size where you do need a fundamental restructuring in order to start scaling horizontally, that's going to be a nice problem to have and you'll also have the resources to do it.
I know several online systems that are handling significant traffic volumes perfectly well on a simple, single-server basis. They don't get bogged down in infrastructure and tooling issues, ever. They don't get confused by complicated cloud hosting issues, ever. They are free to spend almost their entire development budget on actually developing useful functionality, which is like a breath of fresh air in today's dev culture.
Obviously in the more serious cases they probably also have some redundancy for backup/failover purposes, but even that is simple and if necessary can probably be handled manually when you only have a handful of servers to manage. Here I do slightly disagree with one of Carmack's later tweets, in that I would argue 100 servers instead of 1000 is just accounting, but 100 down to 10 or fewer is also more of a qualitative change (albeit not exactly the same qualitative change as 10 down to 1).
You must have had significant in-memory shared state to encounter that problem. Right?
Had you adopted a less stateful model, you'd have looked rather pretty with two Java servers.
"Significant" is in the eye of the beholder. The core of the system was easy to make shardable. But you'd be surprised how many implicit assumptions creep in, how easy it is for ancillary parts to end up sharing state when it's easy. Also note that just because your state's in a database doesn't mean having two instances of the thing that accesses it will work, you can easily end up with an access pattern that assumes only one reader (for example) even though on paper there's no in memory state.
> Had you adopted a less stateful model, you'd have looked rather pretty with two Java servers.
Maybe. At the point where we're running 4 or 8 servers we'd have been facing much the same ops problems that they were. Java bought us an extra year or two of not having to deal with that, but also a significant amount of migration work when it just became impossible to stick to the single process model. At the end of the day we still kept the 5x latency advantage, which is definitely not nothing. But there were also definitely features that they brought to market quicker, and I'm pretty sure Ruby played a part in that.
Tradeoffs, tradeoffs everywhere. I left before the final outcome of that fight (for all I know it's still ongoing), but I don't think either company was being dumb.
Yes and no.
You'd need some ops work, but you'd need to worry a lot less about managing your infrastructure provisioning to keep costs low, e.g. reserved instances, dynamic scaling, etc. and putting out fires when you inevitable exceed your tight perf margins.
You could overprovision 24/7 by 50% and write it off. Your competitors couldn't.
Maybe he really meant to say 'application server'. I.e that there was server side code with non trivial compute/memory requirements for running business logic.
Back when people were writing their backend business apps in C++, COBOL, Java, etc, if there was ever a performance problem, you could usually just get a slightly bigger machine and grow your thread pools a bit. But once the web took off and Ruby exploded onto it, you couldn't do that, because it's an order of magnitude slower, and doesn't really do multithreading. But, as long as you follow twelve-factor discipline, it scales horizontally like a champ. So, we took to horizontal scaling over multiple VMs (and caching things in Redis instead of local memory or Hazelcast or whatever), and that's been the unquestioned way to do scaling ever since.
I remember reading about their routing debacle and realizing just how much work went into trying to get ruby to scale.
Every new non-insignificant feature requires a new REST-API route, or database table or modification of the GraphQL schema. And depending on how you designed the backend, even small redesigns of the frontend might require changes on the backend. Consider a simple app showing car rentals, where you initially have something like /car/[id], then the frontend guys realise that we need to show cars rented by each customer and it's necessary to have /customer/[id]/rentals.
(Of course it's possible to design a system without any schemas or normalisations, which would make your statement truer, but that's rarely attractive for other reasons.)
I'll bite. I think it actually counts not only that, but also the probability that the product will actually get to be developed, and not need to significantly pivot, therefore throwing all developer time spent on perfomance to garbage.
http://web.archive.org/web/20111108170021/http://david.heine...
Modern C# in net-core 3 can even match C++ at manually vectorized numeric code: https://github.com/dotnet/coreclr/issues/27909#issuecomment-...
It's sometimes possible to design C# code in a way that doesn't stress GC too much, so the majority of bandwidth bypasses the GC.
For instance, here's my old .NET project which plays streaming media for hours without interrupts, with a hard limit of 15MB RAM for the whole process: https://github.com/Const-me/SkyFM
Doing so became easier in modern .NET, with these value tuples and spans.
> still hoping some day we'll be allowed both GC and Rust-like manual memory management in a single language
Microsoft tried, managed C++, then C++/CLI. The CLI is still supported if you want to try, but IMO they both were way too complex: 2 types of pointers, two runtimes with weird interaction between them, worst of both sides on safely and ease of use.
AFAIK most people only used these languages for a thin layer of glue to integrate native C++ with C#. And even for that limited use case, COM interop or C interop often worked better. Even MS switched to COM interop in the next iteration, C++/CX.
You can JIT Python also using https://www.pypy.org/ but it's not the default and it's not 100% official and compatible.
Back around 2001 or so, I was hired at an online travel agency to assist in porting their entire system from C++ to Java. Management at the time was frustrated at how long it took to add new features in the C++ codebase - it could take months to get something working in some cases, and they blamed the programming language. Once we got everything working in Java, we found that we could, in fact, turn around feature requests much faster than they could in C++. The trade-off was performance problems: they (we) found that in the old C++ codebase, if you fucked something up, the whole thing crashed, and you had to fix it before you could get it to run. In Java, programmers could paper over their fuck-ups pretty easily so that they wouldn’t be noticed until they had created a snowball effect that caused everything to slow down. Since management was pushing for more features faster, they were incentivizing developers to do as little testing as possible and kick the can down the road. For the most part, management was OK with this: they just bought more, and more, and more servers to make up for the performance problems we were having from the quick-turnaround features they wanted.
totally agree. if all your backend server is mostly complex serialization & de-serialization, and pushing bytes to other sub-systems, i think many other languages have advantages over python.
wondering why no mention of go or rust though...
https://blog.acolyer.org/2015/06/05/scalability-but-at-what-...
Type hints made large python codebase switch from "Any change will blow up to my face" to "I can touch it carefully with protection gloves".
Through, it is still pretty easy to fool mypy and very far of the compile time guarantees that provide most static typed AOT compiled language.
And unfortunately, it also does not bring any advantage in term of performance.
At that point it makes more sense to use a statically typed language.
Much of code's performance isn't really reflected on to the scalability since mostly a tiny part of code is really ran a lot of time, and the other parts are just glues or management stuff or rarely used(not used in scale) features.
Of course, it won't work out to replace 1000 Elasticsearch servers - that's where the advantage of a true "big data" tool will show - but none of the clients really have data "that big".
That's the reason why you need to go multi-process if you want to reach a similar level of concurrency in Python as multi-thread in C++. And that surely adds a lot of complexity.
As a very practical example of this, TensorFlow has a dedicated page with advice on how to make the Python part that reads the files from disk less slow. Think about that: The bottleneck for training a highly advanced AI with millions of parameters is in the 20 lines of Python code to read in a binary file...
There are a lot of ways to handle the IO intensive scenario in python:
* Threading - Works with python libraries written in C, but now you need to add locks to your code to prevent race conditions. Not good for CPU heavy work because it’s switching context every ~100 instructions.
* Gevent - Requires minimal code changes, but it usually blocks when running code from python libraries written in C. This uses event loop style concurrency and automatically patches internal python libraries to switch context when IO occurs. This means less time spent unnecessarily switching context, so it can scale better than python’s threading.
* multiprocessing - Better for CPU intensive work, but requires more memory than other solutions. You don’t need to worry about race conditions since the processes are separate.
* asyncio - Requires code changes and using compatible libraries. It does event loop style concurrency that allows you to specify specifically when to switch context. Like with gevent, this means less time spent unnecessarily switching context.
* ...And there are probably a bunch of other ways I’m missing.
I’ve heard that this sort of stuff is simpler in Go.
https://erikengbrecht.blogspot.com/2008/06/what-does-io-boun...
This person was actively arguing with me about what it was. It's one of many battle stories for why I pretty much just assume everyone on reddit is stupid unless shown otherwise.
I had a funny such experience lately in a job interview. I told the interviewer his misconception could be falsified with ~10 LOC summing a list with 2 threads.
If any operations were GIL-bound (which is extremely common, even if not intentional, by reliance on bytecode instructions dealing with CPython API under the hood, like attribute lookups or iteration special methods), then execution is probably interleaved serially and constrained by the interpreter’s GIL allocation, and slower than just summing serially.
I’ve seen a lot of people who are cocksure they know some contrarian “actually you can use threads” trivia about Python and just naively use the threading module or naively use the newish ThreadPoolExecutor stuff not realizing that no, in fact, it’s not somehow magically always GIL-avoiding to do so.
By having multiple concurrent IO requests from multiple threads? Can probably log detailed timestamps to see if IO requests were happening in parallel.
I imagine Python works roughly the same.
Parallelism is possible, but only for computations in native code.
FWIW Node.js is essentially the same situation.
Otherwise yes, Node or Ruby play by the same global lock rules.
Launching multiple processes without mutable shared memory is always an option.
Been a while since I've used Python, but as far as I remember the GIL only affects Python objects. So if you use Numpy for operations, you can avoid the GIL.
And, more importantly for us, while numpy is doing an array operation, python also releases the GIL. Thus if you tell one thread to do:
print "%s %s %s %s and %s" %( ("spam",) *3 + ("eggs",) + ("spam",) )
A = B + C
print A
Correct.
Threads cannot help Python-heavy code; they can help I/O-heavy or native-heavy code (assuming native code releases the GIL).
Even without multithreading, numpy programming can give amazing speedups. It's the reason why data science Python isn't dead in the water.
Spawn a thread and handle disk or socket IO requests there? You can get fancy and use a thread pool so maybe 15 lines of code.
Not sure why you used the word “supposedly”. Threads have been in the standard library for a very long time.
Yes and still run the risk of running into GIL problems. You claim to have shown 10 lines to an interviewer that proved he was wrong about the GIL still being an issue and I've yet to see an example of it not being one when using pure python. Yes there are certain cases where you don't hit the GIL, no that doesn't mean it isn't still an issue when dealing with threads.
Otherwise another trick is to identify a call which uses extensions or releases the GIL once it goes into C.
I had multiple of those experiences. Even then people wouldn’t believe me so we had to look at Python C code and read about threads and IO operations in C and such. The other alternative is to write a deliberately slow socket server that takes say 5 seconds to respond then then write a threaded client to issue requests and seeing how they complete in parallel.
# main.py
import random
from concurrent.futures import ThreadPoolExecutor as Pool
items = [random.random() for _ in range(10 ** 7)]
def run(items, n):
step = len(items) // n
with Pool(max_workers=n) as ex:
res = [ex.submit(sum, items[i*step : (i+1)*step]) for i in range(n)]
return sum(r.result() for r in res)
if __name__ == '__main__':
import timeit
import sys
n = sys.argv[1] if len(sys.argv) > 1 else 1
time = timeit.timeit('run(items, %s)' % n, 'from __main__ import run, items', number=10)
print("%s\t%.3f" % (n, time / 10))
$ for x in `seq 1 16` ; do python3 -m main $x ; done
1 0.172
2 0.170
3 0.166
4 0.155
5 0.149
6 0.142
7 0.144
8 0.140
9 0.136
10 0.135
11 0.135
12 0.137
13 0.135
14 0.136
15 0.136
16 0.136I think it does that pretty well but, well, that's just like my opinion.
Ideally, pure computation like the one in your example should get a speed up linear to number of threads. 2 threads -> 2x faster. 3 threads -> 3x faster, up to a number of cores.
If that's not happening then you have excessive locking.
In Python excessive locking is caused by GIL - Global Interpreter Lock. As the name implies, there's a single, global lock and execution in a single threads effectively blocks all other threads because it holds GIL captive.
What this particular benchmarks is showing is that GIL is as bad as everyone says: instead of getting 16x speedup, you get speedups that are almost within margin of error for such a coarse measure.
Your example I think is demonstrating the opposite of what you want to show. Those figures are atrocious.
I'm not even sure what the opposite of that is.
The way I read your post, it seemed like your interviewer told you that threads in python are not effective for parallel computation because of the GIL, and your example proves exactly that. The performance of your threads are absolutely horrible, if you were to do that in C++/Java/Go you would likely see a speedup in the order of min(16, cores). Your example proves that your threads are exactly serialized during the computation, which I assume was the point of your interviewer (but please clarify my assumption).
Perhaps your interviewer was instead proposing that threads in Python work like a charm?
1. There is a common misconception that any threaded python code is slower than sequential. This is trivially false for IO-bound, and possibly, potentially, in some cases false for CPU-bound. Said interviewer had that very misconception.
2. More importantly - you and another parent are right. The code does not demonstrate what I thought it does. This is proven by removing the ThreadPool and leaving the code intact:
import random
import math
from concurrent.futures import ThreadPoolExecutor as Pool
items = [random.random() for _ in range(math.factorial(11))]
def run(items, n):
step = len(items) // n
res = [sum(items[i*step : (i+1)*step]) for i in range(n)]
return sum(res)
if __name__ == '__main__':
import timeit
import sys
n = sys.argv[1] if len(sys.argv) > 1 else 1
time = timeit.timeit('run(items, %s)' % n, 'from __main__ import run, items', number=10)
print("%s\t%.3f" % (n, time / 10))
$ for x in `seq 1 11`; do python3 -m main $x ; done
1 0.799
2 0.749
3 0.671
4 0.715
5 0.730
6 0.704
7 0.649
8 0.631
9 0.689
10 0.613
11 0.616
(2) You're not measuring what you think you are measuring.
Take out the Pool altogether:
def run(items, n):
step = len(items) // n
res = [sum(items[i*step : (i+1)*step]) for i in range(n)]
return sum(r for r in res)
1 0.170
2 0.170
3 0.147
4 0.145
5 0.142
6 0.142
7 0.138
8 0.137
9 0.135
10 0.138
EDIT: As an aside, you can speed this up 20x by using Numpy:
items = numpy.array([random.random() for _ in range(10 ** 7)])
def run(items, n):
return numpy.sum(items)
1 0.007
I’m pretty sure the time “savings” you are seeing here come from somewhere else. At a first glance, your copying the huge list while submitting it to the thread pool, this has overhead. Maybe lots of smaller copies are faster on your machine.
The hard part is always that you have more data than can fit into memory, and need complex prefetch solutions to parallelize getting the next batch of data while one batch’s numerical linear algebra computation is in progress.
Many libraries handle this transparently for the user in an extension module, such as Keras & PyTorch.
I think it’s misleading to act incredulous that this is the bottleneck ... it doesn’t have anything to do with Python, GIL, etc. (and there are easy-to-maintain solutions for this in Python).
Neither of these are usually the case for web application servers.
If the choice is between hiring one good C++ programmer or 15 really dumb Python "backend engineers" (and a team of QA's and sysadmins to support them), what do you think your pointy-hared corporate boss would chose?
Last round, for the second best candidate who we hired at 130% the salary we initially offered, after the best candidate was snatched under our nose for what we were told was twice the salary we were offering:
>"Last question, I saw that you wrote a 100 line function here, is this because you ran out of time?"
>>"No. I don't like to break up my functions. Having many small functions is confusing and bad practice."
>What am I doing with my life?
Also ... Why'd you hire this person? Is there such a lack of "good" developers in your area?
Personal productivity is generally going to be slower in C++ than it is in Python. In most situations you would gain more productivity out of a similarly skilled Python dev as you are a C++ dev, so you'd probably need to hire more C++ developers.
Unless your argument is "C++ devs are smart and Python devs are dumb", in which case, let's not start calling people names over the language they use.
No, it entirely depends on the skill and experience level of the programmer.
> In most situations you would gain more productivity out of a similarly skilled Python dev as you are a C++ dev
No, a good programmer with equally good knowledge of both languages will code equally fast in both.
> This is a false dichotomy. You don't have to choose between 1 C++ developer and 15+ package of Python developers.
You failed to see the point. Large teams of clueless programmers doing things slowly and badly is a feature of the system, not a bug. KPI's for managers don't include lowering headcount and cost cutting as an incentive. (And trust me, you really wouldn't like it if they did.)
> Unless your argument is "C++ devs are smart and Python devs are dumb", in which case, let's not start calling people names over the language they use.
Yet that's effectively what you just did in your specious 'productivity' argument.
We're comparing programming languages. You control for the other variables - otherwise the comparison is meaningless. My argument is equally skilled programmers will generally be more productive in Python than C++.
>No, a good programmer with equally good knowledge of both languages will code equally fast in both.
Care to explain how? Huge amounts of the features in these higher level languages are explicitly to increase productivity, and it's generally pretty well accepted they're successful. In this very comment section there's multiple sets of people talking about having to implement their own reference counting systems, and all sorts of other things. Implementing those systems takes up productivity.
If any one language was the best at everything, we would only have one language. There's trade offs made, and that's why there frequently is a right language (or set of languages) for one job vs. another.
>You failed to see the point. Large teams of clueless programmers doing things slowly and badly is a feature of the system, not a bug. KPI's for managers don't include lowering headcount and cost cutting as an incentive. (And trust me, you really wouldn't like it if they did.)
This simply hasn't been the case anywhere I've worked at for a meaningful amount of time. Empire building was plenty discouraged at all of the places I have been employed long term and there were KPIs for reducing headcount, or maintaining it while taking on additional responsibility, and accounting was always happy to step in if costs were increasing without solid justification. Reducing headcount doesn't even have to mean firing people or managing them out - it can be not backfilling spots, helping people find teams with open headcount and transferring, etc. It's never bothered me any - if we have too many people for the amount of work, I'm more likely to get bored.
>Yet that's effectively what you just did in your specious 'productivity' argument.
You started this whole tangential discussion by taking shots at Python developers for no apparent reason. I'm not calling anyone names, or anything even remotely similar - different languages are frequently differently suited. There's obviously places where C++ makes a lot of sense.
I don't really have a horse in this race - right this moment, I'd do basically any serious project in rust, and cargo-script has me even doing small scripts in rust as well. Maybe Erlang or Elixir if OTP makes a lot of sense for the project.
128GB/sec? sure! and 256GB/sec is coming.
With 10Gb, 40Gb and 100Gb internet links becoming mainstream the roadblock to having cool stuff is developers to make it.
Opting for a performant but not scalable solution is basically an acknowledgement that:
- The project will only succeed up to a certain point.
- After the initial launch, no further changes will be made to the project since new additions are likely to cost performance and lower the upper bound on how many users the system can support.
Not many projects are willing to accept either of these premises. Nobody wants to set an upper bound on their success.
Also, it should be noted that no language is 'much faster' than any other language. Benchmarks which compare the basic operations across multiple languages rarely find more than 50% performance difference. The more significant performance differences are usually caused by implementation differences in more advanced operations and in any given language, different libraries can offer different performance characteristics so it's not fair to say that a language is slow because some of its default advanced operations are slow.
Usually, performance problems come down to people not choosing the best abstract data type for the problem. Some kinds of problems would perform better with linked list, others perform better with arrays or maps or binary trees.
Time complexity of any given algorithm is way more significant than the baseline performance of the underlying language.
For a guy like Carmack, it may be quite frustrating working with the constraints of pytorch etc. He ll probably end up making his own pytorh frontend in C, which as a bonus people will use to deploy models.
Said that, I believe Rust has a great future in data processing. Just take a look at Apache Arrow Rust bindings and Ballista (https://news.ycombinator.com/item?id=20456273)
That said, I think the benchmarks could use some tweaking. I'm not sure they are representative.
https://www.reddit.com/r/rust/comments/ap2047/john_carmack_w...
I think it’s a valid point that the reasons why C++ is good for perf - compiled, statically typed, low-level, manual memory management, high degree of control over memory layout - would also apply to Rust. If they don’t then that would really call into question positive claims being made about Rust.
IIRC, his conclusion was mixed: a lot of it was obviously beneficial and worth having turned on, but much of it wasn't, and going forward he intended to make it a limited but continual part of his toolchain.
So my interest in whether he'd tried Rust was whether he'd compared Rust's changes like the borrow checker against his earlier conclusions on writing good C/C++. Cute that he's tried and liked it, but I'd really like to see a more in-depth comparison from him.
2. Horizontal scalability is too much complexity/work. Just apply the correct amount of optimization when you initially write the code.
If you, like me, read this on mobile and did not click [more answers] link under that, then do it. That may save you a minute or two of derealization time.
It ends up boiling down to the fact that compute power is a lot cheaper than developer time and really good developers are more expensive and harder to find than inexperienced ones.
Yes, John Carmack is right that a bunch of scaleable systems that see reasonable load, with well-understood traffic requirements could be rewritten more efficient ways. But how long would it take? How much would the extra work cost? And what would happen if traffic goes up, to another 10x, 100x? Can you throw still throw money at that problem, with this new and efficient system already in-place?
One of my memorable stories comes from an engineer I worked with when we rewrote our payments system[1]. He told me how at a previous payments company, they had a system that was written in this kind of efficient way, and needed to run on one machine. As the scale went up, the company kept buying bigger hardware, at one point buying the largest commercially available mainframe (we’re talking the cost being in the multi-millions for the hardware). But the growth was faster and they couldn’t keep up. Downtime after downtime followed at peak times, making hundreds of thousands of losses per downtime.
They split into two teams. Team #1 kept making performance tweaks on the existing hardware to try to get performance wins, and increase reliability during peak load. Team #2 rewrote their system in a vertically scaleable way. Team #1 struggled and the outages kept getting worse. Team #2 delivered a new system quickly and the company transitioned their system over.
What they gained was not cost savings: it was the ability to (finally!) throw money at their growth problems. Now, when traffic went up, they could commission new machines and scale with traffic. And they could eventually throw away their mainframe.
Years later, they started to optimise the performance of the system, making millions of $ in savings. The new system - like the old one - cost millions to run. But that was besides the point. Finally, they stopped bleeding tens of millions in lost revenue per year due to their inability to handle sudden, high load.
It’s all about trade-offs. Is cost your #1 priority and are you okay spending a lot of development time on optimising your system? Go with C++, Erlang or some other, similarly efficient language. Is product-market fit more important, with the ability to have high uptime, that you can buy? Use the classic, horizontally scaleable distributed systems stack, and worry about optimising later, when you have stable traffic and optimisation is more profitable over product development.
[1] https://blog.pragmaticengineer.com/distributed-architecture-...
Most companies or startups or even bigtech are not big enough for these problems.
All these complexities are horrible, but good for the AMZN stock price, as devs keep spooning up a ton of VMs
Python programmers are much easier to hire, though - you already need a good C++ programmer on the team to hire another one, because HR and corporate management can't into proper hiring process.
This last factor is the overarching most important one for BigCorp Enterprise Inc., not development speed or cost.
This is obviously not strictly true, always. A skilled programmer will use the proper tools for the job.
If you for example is tasked with writing a backend service exposing a GraphQL API, I think it would be foolish to do this in C++, and would bet that the average Python programmer would do it quicker than even a top-tier C++ programmer (if the latter would be hellbent on doing it in C++).
Especially when working with MVP's (or new projects in general), the ability to leverage already existing tools and frameworks are key to rapid progress. This doesn't necessarily have to be scripting languages, but the Python/Node/Go/etc developer would have a working GraphQL server up and running connected to a database of choice within an afternoon while the skilled C++ developer would have to spend at least a few days implementing a GraphQL server mostly from scratch [0].
[0]: A quick Google show that schema parsers exists for C++, but nothing matching the frameworks/library available for more web-fashionable languages.
a) Writing a schema parser is not rocket science. In fact, for a good programmer implementing their own GraphQL library would be quicker than integrating some third-party library. So your first point ("average Python programmer would do it quicker than even a top-tier C++ programmer") is absolutely wrong.
b) There's no value in an MVP that does something generic that is already available in off-the-shelf libraries. Your GraphQL example is pretty pointless because it doesn't actually do anything.
> ...the Python/Node/Go/etc developer would have a working GraphQL server up and running connected to a database of choice within an afternoon
Well, no. By the end of the week they'll still be arguing about which package manager to use, whether TDD is a good idea, what makes a microservice 'micro' and how to configure Kubernetes.
I'm hiring 25 D programmers, so I suppose it very much depends on what you mean by practically!
At single-node scale, development costs dwarf hardware costs of scaling horizontally.
Modern SW trends are horrific
The second tweet of the discussion:
> JAVA or C# would also be close, and there are good reasons to prefer those over C++ for servers. Many other languages would also be up there, the contrast is with Really Slow (but often productive and fun) languages.
I'm afraid that Carmack has sided with his own anecdotal experiences of the 1990s to justify the use of C++ for server-side development for the 21st Century. This probably made sense at the time due to the availability of more C++ devs and less language choices, but today in the 2020s? I remain totally unconvinced by his argument.
He goes on to suggest Java or C# which still makes sense for many companies for generic server-side development if you are after a more secure backend. Kotlin is pretty much the most sensible for this. But for the sake of Carmack's engineering background however, it is unsurprising why Java/C#/Kotlin are technically unsuitable for high-performance gaming platforms if one was to create one. So what credible languages could be used to compete with C++? I hear Discord is having a great time with using Elixir (Erlang could also be used) and another gaming platform called 'Hadean' is using Rust for their platform.
This really is just not true.
Financial, real-time style High Frequency Trading apps are often written in Java - not C++.
Much of the JVM is not a VM, it compiles to machine code - in an optimised manner. For starters.
Given how difficult it is to develop safely in C++ I can hardly think of a reason to ever use it on the backend.
Generally, for a 3D game manual memory management and explicit data layout are critical. For example, it's common to use custom memory allocators with a region for each frame, a region for each loaded level, etc... This is then much cheaper to simply drop on the floor than any kind of object-by-object cleanup, whether that is reference counting, garbage collecting, a traditional heap, or whatever. Even Rust can't yet compete with this!
Similarly, many game engines use ECS systems or in-memory columnar data layout to (structure of arrays instead of arrays of structures) to enable SIMD instruction sets such as AVX.
Java can be coerced into doing much of the above, but it generally takes a ridiculous effort to approach what comes nearly effortlessly with a language like C++ or Rust.
Even C# is a better choice than Java, as it monomorphises more code and recently had a range of extensions[1] added to reduce GC pressure such as stackalloc, Span, Memory, MemoryPool, SequenceReader, ValueTask, etc...
I've bought and played several games written in C#, but other than Minecraft I'm not aware of any popular real-time games written in Java in the last 15 years or so. Meanwhile, Minecraft is not at all smooth on my very high end gaming PC in 2019 despite being 8 years old. (I'm sure this can be eliminating by tweaking some settings, but it's indicative of the problem.)
I say this as someone who used to professionally develop browser-based Java games back in the early 2000s and had to personally jump through hoops to reduce heap allocations to avoid GC pauses.
[1] https://docs.microsoft.com/en-us/archive/msdn-magazine/2017/...
Additionally, most multiplayer games have the same codebase for the server and the client for the obvious reasons. Single player is literally "online play" with an in-memory channel to a local server.
All Quake-based games work this way, including the derivative Source-based games and a bunch of other engine variants. Unreal works this way too if I remember correctly.
You can't realistically write a client in C++ and a server in Java. You'd be practically doubling your development time!
No. That is high level language propaganda and simply not true.
They're written in a very specific style of Java, where you take pains not to allocate, which looks a lot more like C++ than idiomatic Java. Operationally, strange rituals are followed to avoid recompilation or other otherwise normal runtime activity during trading hours. The app will be started on sunday night, and the team spends the week praying it doesn't need restarting during the week.
You can do HFT in Java, but it's a dancing bear.