Cogs bad

Maybe people succumb to the hype. Maybe they want the latest shiny running their own shiny new thing. I'm sure these things contribute greatly.

But I see another aspect to this whole "We Use Shiny Cogs" movement: high-level vs. low-level. As we make tools that abstract us away from the metal, we are able to spend less time thinking about the electrons flowing across silicon and more time thinking about building something John Q. Public will pay for.

We architect higher and higher abstractions for exactly this reason. And it comes with a price: at some point you stop running things as efficiently as possible and there's waste. If we were all studied CS students and could write kernels and compilers from scratch, we might spend five years building a very tight, efficient stack for Twitter that could run on a single box (maybe with a hot failover). But we're not. We're a collection of humans with differing levels of understanding and will power, many of whom just want to Get Shit Done and stop thinking about kernel task queues.

So lets turn his "rich man's problem" around a bit: you build your idea on top of a stack that you understand and keeps you happy, and when you bring in the capital (through revenue or investment - whatever) you put money into deeper, lower-level engineering. Until then, build your idea the way you know how.

This article, and the Hickey talk to a greater extent, present a coherent but one-sided argument about elevating simplicity and understanding above human concerns in software engineering.

It's true that, as a programmer, you should strive for simple, "correct by inspection" code when possible. And the better a programmer you are, the more you will see and take opportunities to write a bit of code instead of roping in a third-party library, to use a small library instead of a big library, or use a library instead of another process, thus avoiding large swaths of complexity, the bane of software development. On the flip side, poor engineers may make large errors of judgment in this area.

However, a bias against powerful, off-the-shelf tools or a disdain for the "familiar" over the "objectively simpler" is no better. The line between a one-man (or few-man) project and a bigger project is where this really starts to matter. News flash: You can't get that "my code feels correct" feeling (the one that's supposed to substitute for a formal proof your entire system works) when other people are writing it. When putting together a team, using technologies that are "familiar" doesn't seem so intellectually lazy -- and many popular technologies are actually very understandable and well-engineered. Finally, I'm taken by end-to-end testing and Eric Ries's "immune system" metaphor as a way to ensure correctness of a complicated system in practice.

If you're making something big, you might have to put down the microscope. If you're making a tapestry, you need to have multiple threads entwined and stay cool.

I have experienced this dilemma in a different domain: machine learning. You can either go the Hadoop way and commit to simpler algorithms that can be run in distributed manner or you can keep pushing the single machine more and more by use of ever more clever algorithms. Unfortunately, it is hardly ever possible to follow the route of push-single-machine-to-max -> thorw-more-machines-in. The distributed way and shared memory way of doing things often differ in fundamental ways.

Going with big data tool chains from the start is often a overkill for small experiments. But once you outgrow one machine, the pain of undoing all the nice (algorithmic) tricks is also quite severe.

Perhaps it is time to accept that we now produce data at a rate that distributed is going to be the way to process it. But this also means that some of the techniques available for scaling to larger data sets may need to be given up.

Developers (myself included) worry too much about future unlikely pain and suffering, especially the degree of said suffering. Maybe you spend a few weeks here and there rewriting things. Big Deal.

Scalability and performance is complicated. Unless you KNOW your product will have a big splash, premature optimization will kill your productivity. And you will get it wrong. You will get the implementation wrong. You will optimize the wrong things and not really understand your bottlenecks. Especially if you've never scaled anything before and haven't been bitten twice by all the compromises you have to make.

Distributed systems are hard. Multi-threading is hard. Sharding is hard. CAP is a bitch. If you can scale vertically, do it. Avoid the demons of distributed work until you require them.

Most of the services/apps we build today would do just fine with setups like Mailinator or gasp ACID-compliant data stores.

"The first enemy of performance is multiple machines. The fewer machines that need to talk to perform a transaction, the quicker it is."

That is not strictly accurate. He's taking one aspect of performance - communication latency - and expanding that to be a universal truth of performance.

Pixar's render farms are good. Google data centers are good.

When you're CPU bound, more CPUs can make you faster. Note the specific use of the word "can," as in "sometimes."

For most startups, embracing a cloud architecture just makes sense. You're building an MVP and want to get it out in front of users. Deploying on something like Heroku and using the add-ons is one of the best ways to focus on your product and not on your server. Then when you have a success, step back and evaluate your tech stack.

To me, the primary advantage of all this new-fangled web server technology is the improved developer experience, not the performance implications.

This article starts by saying that there is something badly wrong with modern programmers.

It then details and critiques the way a typical one-man, one-site 'startup' is using discrete 'cogs' to build his system, presumably whilst learning how to market, build customer relationships and develop a beautiful and compelling product that makes enough money to keep him afloat.

I think the author may be missing the point. An elegant and sustainable back-end does not directly correlate with an elegant and sustainable business.

I think this analysis is not quite as cut-and-dried as the author thinks it is.

1. Latency. Yes, going out-of-process, even to localhost, is very expensive, and the person the author was responding to should realize that. On the other hand, synchronization is also very expensive. How do they compare? I have no idea[0], and I'm not about to guess. The author shouldn't either.

2. Concurrency. Paul Tyma specifically talks about a "synchronized LinkedHashMap" as the implementation of a cache, so I'm going to take him at his word, understanding it might be a simplification. A synchronized Map is a poor implementation for a cache, because reads will block writes when they don't need to. A better implementation would be a ReentrantReadWriteLock protecting an unsynchronized LinkedHashMap. Redis gives you this behaviour for free (even if you don't know of the existence of ReadWriteLock).

3. Memory usage. Let's be honest--Java is a pig for memory[2] compared to C++, and this is nowhere more apparent than indexing and caching the guaranteed 8-bit strings you'd find in an email. If your whole purpose is to fit more lines into your cache it's genuinely worth considering breaking out of the JVM to exploit the smaller memory footprint of C++ strings (and this really only holds for caches).

Was using a LinkedHashMap a good idea for Mailinator? Probably, I definitely don't have any evidence or suspicion to the contrary. Is it sensible to say "COGS BAD! IN-PROCESS GOOD" for every use-case? Not really.

[0] If I had to guess I would imagine that going out-of-process is 1-2 orders of magnitude slower than contended synchronization. Anyone got any figures?

[1] http://docs.oracle.com/javase/6/docs/api/java/util/concurren...

[2] I would be very surprised if Redis was not more than twice as memory-efficient for this task as a LinkedHashMap. I'm pretty sure that one could implement a C++ solution that would be 3-4x as efficient. This is really only of paramount concern in a caching context, but in that context, it's paramount, because the cost of missing the cache is so phenomenal.

I think performance is far too often used as a reason to add cogs, others are far better. If you often replace parts, more cogs are great. We have a really low-yield setup at one of our clients that is nevertheless splitted: ane process imports media data from a huge number of content providers and is split into 3 parts - an importer that normalizes all data, a queue as a binding and a reencoding process. The reason why we did this is easy: the queue is running for two years straight now, the encoding process was deployed once last year (we changed our logging strategy) and the importer process is deployed around 4-5 times each week. Not having to bring the whole machine to a grinding halt on each of those occurences is a major benefit.

It's about daring to KISS. Only a select few dare to stand up against the hype.

> If you can use a local in-process data-store (sqllite, levelDB, BDB etc) you’ll be winning massively.

Hold on just a sec. SQLite? Isn't that essentially equivalent to saying "If you never have to concern yourself with database locking, you'll be winning massively?" How can we be talking about scalability and SQLite in the same article?

Maybe I'm being the kind of developer that the blogger was talking about, but would it be such a bad idea for a Mailinator like site to use Redis to store the email messages but hosted on the same box as the web server?

At least that way, if you started to hit memory limits it would be relatively simple to scale out to more machines by moving Redis to its own box. It would be a configuration change rather than having to re-architect your custom LRU cache.

Another benefit would be that you could get disk persistence for free while still staying fast. If Mailinator needs to reboot all the emails are lost. That wouldn't necessarily have to happen if he was using Redis.

As for the cogs argument, this is all I have to say: silicon is still cheap, and carbon is still more expensive.

I think it makes a lot more sense to focus on 1) what differentiates you, which may well involve lots of custom code, and 2) finding a market that works out for you.

For mailinator, they were already popular, so it makes sense to do some one-off coding to make things faster/more efficient. Perhaps, were mailinator starting up today though, using Redis as a good first step would have beaten whatever they had before, and would have been 'good enough' for longer.

Once you've got to the point where you're getting popular, then worry about making stuff scale up.

Scalablity is nice, but what about redundancy? Not every website can go down for 6 hours while a tech repairs a host in a data center.

There are also challenges with putting all components on a single host. It's simple, but the components are not all going to scale the same way. And depending on how the datastore is partitioned, you'll still make remote calls anyway.

By the same token, I've inherited projects where the developer did not take scalability into account and avoided "cogs" for quick development turn-around. Putting the "cogs" in afterwards was incredibly painful, and a lot of development effort could have been saved if some thought was put into the architecture.

Quips about premature optimization often make the assumption that any optimization is premature. If data or usage growth (or uptime guarantees, for that matter) is an inevitability, then its often worthwhile to have at least some plan to grow your system beyond a single machine.

This reminded me of http://teddziuba.com/2010/10/taco-bell-programming.html

the trouble is that the Cogs Model (as it is put here) allows you to get shit done with lower skill levels. sure, it won't be perfect. and yes, if you had better programmers you could do it better. but these are pragmatic timse for most of us.

so he's right (in the abstract).

but for the reality most of us operate in he's wrong because Done is Better than Perfect.

that said... the "New is Good" thing needs to die. We're not freaking magpies people.

Is mailinator a profitable business, or a volunteer project accountable to no bottom line? Both are fine, but the difference may inform design choices.

How expensively does the developer value his time? Has he already made in investment in learning certain technology?

I love posts like this because, love it or hate it, it gives me a checkmark of thoughts to compare my choices to.

I see three possible approaches, all with their advantages and disadvantages. (Of course people may fit between them with a mixture of attributes.)

1. Monolithic - Build it all yourself, purpose built and high performance. This is why mailinator and plenty-of-fish are able to produce high thruput on a surprisingly small number of machines.

2. Confederated - Completely distributed. Each machine is its own monolithic platform with everything from DNS to database, including web server on that node, but a cluster of nodes gives you scalability, and workload is distributed across the cluster. (I'm not aware of any examples of this, which is why I'm building Nirvana.)

3. COGS: You build your cluster of machines by architecting a system whereby you minimize (but not eliminate) single points of failure. You have N web server machines and X database machines and you seek out really high performance open source cogs to keep the number of machines low (e.g.: Redis, MongoDB, etc.)

The COGS approach is often taken with the idea that we need something really fast. MongoDB being fast (and "SQL") are the reasons its often chosen. Redis being fast is given as a key advantage (which is relevant for an in memory database, sure.) Node.js is often chosen for similar reasons.

But the ends result of the COGS approach is a brittle architecture. You may have multiple redundant web servers but the thing that distributes loads is a SPF. More specifically the architecture is complicated- each machine has a different configuration, etc.

With Monolithic, you get performance, and save hosting costs, and you can probably scale pretty well because you know your system really well, and you've squeezed out a lot of the inefficiencies that come from being generic (in the cogs approach) such that you can interoperate.

What I think we should see more of is confederated- no machine is a unique snowflake. Every machine is identical to every other machine. This way configuration becomes dead simple-- just replicate your model node, bring it up and data and load starts going to it.

This can be done with cogs- but they have to be fully distributed cogs. An example is Riak (Which hit 1.1 yesterday) which is open source and written in erlang and probably loses to nodeDB in every single single node benchmark you can come up with (not that the Basho people have designed it to be slow, quite the contrary.) But where's the fully distributed web platform for such an architecture? (If you have an answer, please make this question non-rhetorical. I'm putting a lot of time into building one because I couldn't find one.)

An interesting thing about the confederated approach is, because each machine is identical, it could be built in a monolithic fashion. Thus super optimized for its purpose. I'm using a sorta cogs approach because there are many good erlang cogs to use in my project.

But I think the big mistake is to focus on single node performance these days. Servers are relatively cheap, and you need more than one anyway for redundancy, so might as well have a cluster and no single points of failure.