The “Run Less Software” philosophy | Hacker News Reader

The “Run Less Software” philosophy

blog.intercom.com

210 pointsyrsamrith96 comments

"Here, read my ramblings about the "Run Less Software" philosophy on my blog that loads 10MB of data for what could be a plain html + basic css page, among which you'll find 800,000 characters of javascript code to execute. And that's not even counting 3rd parties, isn't that great?"

This is obviously marketing for the author's company. What do the four armies have to do with the point the autor's trying to make? "Hey, you should run as little software as possible. But first let's analyze the professional IT landscape with an analogy that does not make anything clearer but allows us to embed fancy pictures and videos."

Yes, the 4-armies analogy was a remarkable non-sequitur. I can't help but think he introduced it to talk about the hidden perils of "standard solutions" and "vendor lock-in", but then abandoned this idea when his conclusive paragraph brags about solving a problem with AWS Aurora (a cloud RDMS) and AWS DynamoDB (a cloud mongo-ish).

This trade-off between (apparent) simplicity and vendor lock-in is a fundamental force in the industry, and is worth talking more plainly about. (Another big area where this affects people is authentication - it's easier (for users) to use Google- and Facebook-backed OAuth, but what are the trade-offs involved? Does this increase my vulnerability in any way?)

I liked the idea behind this article, but it's definitely a missed opportunity. And yeah, as something of a web-minimalist myself the ridiculous page-weight was a rather ironic touch.

exactly what I felt, 'run less software' page seems is 'running a huge chunk of stuff' itself, how ironic, not to mention it's tediously lengthy, gosh.

1.6MB for one image of a map, that alone is a huge factor in the size. Unfortunately it's the state of many websites.

Bit of a shame as Intercom has been doing Content Marketing right for a long time. Older stuff by Des Traynor was pretty good though (and entertaining, his talks about Product Strategy are hilarious). Might have switched to a staff writer, hence the genericism (i hope they don't let engineering write articles on their own).

I guess that is the peril of content marketing in a narrow field - eventually you run out of fresh takes but still need to attract eyeballs. Hence the Buzzfeed-style headlines.

Initially Intercom was all about being great in a niche (on-site chat messaging), doing one thing well, focusing. Then growth came, demands for expansion, and so their mantra switched to all-in-one suites. So did their content marketing.

If all that bloat is generated because they are using third party frameworks or plugins and all they really are doing is writing the copy of the post itself then it still basically fits with the narrative.

I think it was one of the most nice-looking blog posts I've seen in a while.

Yes, I know - wrong medium for presentation over content.

Exactly

Hi! I’m Rich, the blog post author.

Thanks for the candid feedback :-)

This blog post is somewhat a written summarization of a 25 -> 30 minute conference tech talk that I gave at SRECON-EU last year. It was a real struggle to figure out what to leave in and what to leave out. I didn’t want to arrive at a blog post that takes 30 mins to read :-) So, I can see how the four armies and battlefield analogy might come across as disjointed or unnecessary. There is definitely some context missing from the blog post that was in the talk.

When I was putting together the original talk, the “four armies and battlefield” framing was something I felt was useful to help illustrate how competitive I feel the modern software business landscape is and thus how important it is that your company has the right engineering strategy to help them “win their own battle” / succeed in your chosen market.

Anyway, thanks again for the feedback, hopefully the above provides a little more insight on why I used the four armies framing.

Rich.

I don't see why AWS is alone in "Tier 1" on a list that claims to prioritize standard technology and outsourcing undifferentiated heavy lifting.

The post gives an example of switching from self-hosted MongoDB onto AWS DynamoDB. But AWS DynamoDB is proprietary technology, not standard. Amazon doesn't even claim that DynamoDB is standard; they argue that DynamoDB is the best in the world, worth the premium in price. It's not at all "undifferentiated heavy lifting."

It sounds like they made the right decision with DynamoDB, but they did it by violating the pillars of their own philosophy, adopting a differentiated proprietary solution for their scalability problems.

It's the "self-hosting" part that's the "heavy lifting" for MongoDB. They could've also chosen a hosted Mongo service, and that would've worked too. The idea is to spend time focusing on your own product and do as little as possible outside of that core competency. Setting up, optimizing, monitoring, upgrading and everything else that goes into running your own database, however standard, falls outside of that core. The more you can do to avoid doing work that isn't directly building features and experience for your customers, the more you'll be successful.

By undifferentiated, they mean outsource anything that doesn't differentiate their own product offering. Build things that give you a competitive advantage over other products in your space. Outsource everything else because wasting work on it won't help you get customers. Of course DynamoDB is differentiated for Amazon, but it's undifferentiated for Intercom...it's just a persistence strategy that customers should never know about.

DynamoDB is not even a good replacement for Mongo. DynamoDB only lets you index two (?) fields and everything else is a full table scan.

At least go with a hosted Mongo solution.

Actually out can have multiple indices, but they are eventually consistent:

https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

It’s certainly nowhere near as full featured. But in our case it is entirely appropriate. Over time we simplified our use of MongoDB to the stage where it’s effectively a k/v store. Swapping that simple pattern over to DynamoDB was easy, saves us a ton of money, and means we get to shed a ton of operational responsibility.

Also, you are incorrect about only being able to index two fields in DynamoDB. Though we luckily don’t need anything other than the primary key.

Thanks for taking the time to read our blog post and for sharing your thoughts!

For us, the concept of “standard” technology is not about whether it’s proprietary or open source, more that we have a “standard way of doing things. So in this case it was moving toward saying, “hey, if you’re an engineer at Intercom, and need a scalable key-value store for the thing that you’re building, we think you should lean toward using DynamoDB, rather than any of the other technology options, because that’s the one that we’re committed to making a first developer experience for internally for the long term”.

The reason for having AWS alone as a Tier 1 option is similar. We’re already very heavily invested in making developing on top of AWS a great developer experience at Intercom, and we’re committed to doing so for the long term. Therefore we think that it makes sense for us to advocate for always exploring relevant offerings that put out.

I recognize that I could have elaborated on the philosophy a lot more, there’s always a tension between writing too much and too little. And the philosophy is just that, not dogma and not right or wrong, just trying to give insight into how we think.

Thanks again for the feedback! Rich.

Somehow the recommendations feel like you are making them to the reader, and in that regards it doesn't feel right to just recommend AWS. Other public clouds provide equivalent or sometimes even better services.

Hi, Rich, blog author here again.

Your feedback is super fair. That is how it comes across in the blog.

In the talk that came before the blog post, I go into a lot more detail here. I have some slides where I explain that these are "our" standard technologies, and are not intended to be "the" standard technologies that everyone should use.

I gave some other examples of perfectly good sets of "standard" technologies which other companies could choose for lots of good reasons.

The main point that I/we were advocating for was that, in our opinion (just an opinion, not dogma, not definitely right or wrong), there are some strong benefits to making deliberate decisions about what technologies you consider standard, safe, easy and fast to use, versus those that are not.

Some of those benefits we see include: 1. speed of technical decision making 2. incentivizing engineers to develop deep technical expertise in specific technologies 3. incentivizing managers to fund training in specific technologies 4. creating fungibility within your engineering team to make it easier for engineers to transfer between teams and come up to speed quickly 5. minimizing ongoing operational overhead as there's fewer technologies that need to be battle-hardened with operational tooling

Hi Rich, thanks for the response. I guess I just found it a bit hard to see the lines between "here's a good strategy" and "here's our strategy". Where you are writing "we have a tiered set of recommendations for who to outsource to", you could add something like "for our developers, ..." or "at Intercom, ..." and it would be much better placed into context.

Otherwise I really enjoyed the article, it resonated well with my own thoughts and vision.

thanks again dajonker, that's more good feedback.

Keep in mind this is an article extolling the virtues of outsourcing critical components of your business by... a company that you outsource critical customer interaction components to.

that's right and instead you should outsource them to us

Oh my God, not just any company, the company responsible for those annoying chat popups on websites?

To be fair, if you want to annoy visitors to your site and make them hate you, you should probably choose to outsource that functionality to Intercom. Annoying your visitors may be dumb, but rolling your own chat widget would be both dumb and time consuming.

I am a customer of two SaaS companies that use Intercom, and with them it is nice to use and un-obtrusive. This is the first time I've noticed the pop-up, so it's really on this company for configuring it to pop-up.

I can honestly say I've never noticed an unobtrusive Intercom implementation.

"Annoying your visitors may be dumb"

By what metric is it dumb? We've found that our chat widget dramatically increased contacts, conversions, and is often used for support.

Personally I am not a fan of annoying chat widgets, but they do seem to work well.

Ah, I already wondered why my ad blocker has a rule for that very domain and tries to prevent loading the article. Now it makes sense.

Just to set things straight - you can use intercom without the popup. And on a more poetic level, a product can be annoying and useful simultaneously. Like a washing machine.

I have a washing machine that is not annoying, but, YMMV.

I love these things. SaaS companies who aren't explicitly clear about what their product does or it's limitations with pricing that is all contact us for a quote then BING "how can we help you today?". I have no idea because you don't provide a single point of reference for any of the things I need to evaluate before I care if you exist.

I opened the link and only skimmed it, but I left the tab open in the background since it seemed like an interesting read to return to later.

Then they started to blink the tab title.

It's one thing that you do irritating stuff to get my attention when I'm actively on the website, but when I'm doing other things? This is right up there with popunder ads, blocking right click, and autoplaying sounds.

Instant close.

My adblocker looked at me funny when i tried to click on a link.

The article is mostly about outsourcing "heavy lifting" like database hosting to cloud providers and non-critical software to SaaS providers. I agree with the article when it comes to database hosting/infrastructure but I would like to see more discussion about outsourcing to SaaS.

Outsourcing to SaaS means trusting your (and your customers) data to lots of different companies. And in the case of SaaS those companies could be small software companies or even one-man operations. This is not very compatible with upcoming data privacy regulations like GDPR. And I suspect that more and more companies are going to be looking at a model where they purchase/license/outsource non-core software but host it themselves in their own datacenter or cloud accounts.

The solution I am working towards in my current project is to build the software from day 1 with dual-mode operations in mind. Easy to host as SaaS but with the option of running Standalone on customer premises with the same codebase. Just different configuration/modules.

Well since this article is from Intercom, I can share that we outsourced our support / chat / things to Intercom, instead of (what was also proposed at the time) building our own.

A company for which I worked also leveraged intercom. I found it quite useful. Especially in the way it let non technical team members set up rules based customer interaction around onboarding and feature discovery.

See more of my thoughts here:

http://www.mooreds.com/wordpress/archives/2872

A great opportunity to move to open source. It will help you not only have more control on your assets in case of privacy and GDPR, but also avoid vendor lock-in.

I find it hysterically funny that companies think that operating their own mysql, postgresql, redis or mongodb is "a difficult task". It is like a home owner deciding that knowing how to operate their own toilet is a difficult task that must be outsourced.

Oh and before someone says "we outsource fixing our toilets to plumbers" - in this case we are not event talking about replacing a flap, we are talking about being able to use a plunger.

You're outsourcing sewage treatment when you run your own toilet. But, yah, I agree running a database on a computer is pretty easy.

Setting up a single postgresql is not that difficult indeed. But in a production scenario with decent uptime guarantees you are going to at least need master-slave replication, make sure they fail over when the master goes down, have backups so you can do point-in-time recovery, rotate and ship the logs somewhere, apply security patches, and more.

Sure, I can do all that by myself but that takes time, time that is better spent serving my clients instead of setting up yet another replicated db. It takes me a few minutes to setup an Amazon RDS instance with all that enabled out of the box.

Also, "just run your own X" starts to fall flat once you grow beyond having a single project in your life.

s/single project/single successful money making project/g

Oh to have those problems

Could you possibly provide the chapter and verse of this dogma? The harsh reality is that the vast majority of the companies *do not have traffic or need to have 99.9% available infrastructure because they:

a) do not have service that operates 24x7

b) do not have peopleware needed for 24x7 operations

c) service customers that do not have 24x7 operations

A metric boat load of now shuttered "startups" would have been going concerns providing very nice long term living (and retirement) to their owners had they been optimized for "not spending 109% of revenue for Yet Another Thing We Should Use As a Service".

Operating such services in a fairly static production environment is indeed simple, though in many cases I suspect it’s wasted effort that could be better spent elsewhere.

In our (Intercom) case these tasks are not simple.

We are lucky to have had exponential growth in most metrics (including database throughput) for the entire 7 years of our existence. This is a harsh environment for datastores! A 99.9% SLA makes it even harder.

We are also nowhere near finished building out our vision for the product. We prefer to focus on that instead of operating data stores, which we can pay someone else to do.

Mh, my current company does this, and it's a real pain and a massive innovation showstopper. It's nice as long as you have a single application and a single database. That's easy to do on premise.

Then you need search, so you either build the search yourself, or build on solr/elasticsearch or anything else building on lucene. Then you need a cache, and suddenly you need to worry about cluster setups, choose a cache implementation. Then you end up with a use case which would be easily solved with a persistent queue, or neo4j or whatever, but your on-premise solutions block you.

It sounds simple to have n different backend solutions, and to offer a binary choice - everything built-in and everything not-built in. But usually you end up with about n! * #on-premise different setups and they all end up with their own classes of bugs and difficulties to setup.

Everything about this post and the company behind it, is a contradiction of any proposed "minimalism in software" philosophy.

Personally I have found growth and simplification by replacing the complex with the simple, not necessarily less.

But certainly less software that does many things.

For example, I might replace one monolith app with many command line tools that each do a single job well.

Embracing markdown vs myriad of closed note taking tools has been a boon.

When I read "Outsource undifferentiated heavy lifting", I was skeptical, but outsourcing infrastructure and components to established companies makes sense (especially the list mentioned). Startups should not outsource to random body shops unless they either have experience doing such or have specific targeted/quantifiable/definable needs that a specialist can help with. Outsourcing, in general, sucks more time and money than the benefit it brings - specifically speaking to job/body shops.

Yes, agreed. For startups, writing software should rarely fall in to the "undifferentiated heavy lifting" category; it should mostly be in "enduring competitive advantage", either seeking it or creating it.

For very early stage startups, I think outsourcing software development can sometimes have merit. If the main point of the software is answering the question, "Do we have a real business here?" then if there isn't a technical cofounder I think outsourcing can be worthwhile as long as you assume that all the outsourced code will be thrown away.

But as soon as the founders are sure it's a real business, I think at that point you have to toss out the oursourced stuff and start building the real code base. Then you're reasonably sure that some of the code you're writing is going to be delivering customer value over the long haul, so it's worth investing in sustainability.

And you think thousands of companies didn't do their math before outsourcing.

Is that a joke?

I've seen management push for outsourcing simply on the basis of having worked with a particular company/group before when there was no need/sense. It's about knowing the trade offs and making the right decision for the company as a whole not due to personal preference.

And many companies used to make outsourcing decisions based off of simple head count cost and didn't factor in the extra time/hours/delays due to integration/poor work/other factors.

Not every company is like Intercom where they have enough funding to spend on 'outsourcing undifferentiated heavy lifting'. The only competitive advantage they have is that they are well funded and they have budget for marketing.

There are other software companies out there, smaller and leaner than Intercom that are eating their lunch by offering similar or a better product at a lower cost. Those companies aren't 'outsourcing undifferentiated heavy lifting' but rather have a lean and talented team that's able to do the heavy lifting themselves without sacrificing on product development.

Anybody know why the domain is on one of uBlock Origin's blocklists[1]? The styling is annoying as all hell but it doesn't seem to be an ad server.

[1]: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hos...

Ghostery has it marked as a "Customer Interaction" tracker, which apparently

> Includes chat, email messaging, customer support, and other interaction tools.

https://ghostery.zendesk.com/hc/en-us/articles/115000740394-...

The summary page has some privacy details:

> Data Collected:

> Anonymous (Browser Information, Cookie Data , Date/Time, Page Views )

> Pseudonymous (IP Address (EU PII))

https://apps.ghostery.com/en/apps/intercom

Yeah apparently they provide chat popups, didn't realize since reader view killed it. Thanks.

This company provides those contact chat popups (one of which you can see on that page too), so that's likely the reason.

Thanks. Reader view killed the chat popup, so I didn't notice anything untoward.

Because uBlock° is an annoying-crap blocker, not just an ad blocker.

And that is why I use rails and JQuery for my side projects. It is boring but I can pull it off on my own. If I run into some problems there are thousand already who have had same problem.

Yes! Boring stable tech is sometimes the right answer. I've heard you should either be pushing business boundaries or tech boundaries every time you solve a problem. But pushing both is too risky.

Great set of principles with some good examples. It's worth noting that leveraging third party providers is not fairy dust--choosing that path brings with it constraints and you may have to spend time working around those constraints.

I wish he'd talked more about the four armies and how they played into the choices Intercom has made.

"We quickly discounted this as it didn’t fit with our three pillars of Run Less Software and decided to consolidate on MySQL, specifically on AWS MySQL. As we were able to benefit from the pace of AWS innovation (and adopt AWS Aurora when it came along) we got 5x throughput at a 30% cost reduction."

It's worth noting that Aurora is compatible with PostgreSQL (as of Oct 2017) [1].

1. https://aws.amazon.com/blogs/aws/now-available-amazon-aurora...

The indented quote renders as a code block and isn’t readable on mobile. I like your reference, so it might be nice to change to formatting for easier reading on mobile.

Just updated it. I thought there might be a way to apply blockquote styling, but the traditional markdown syntax didn't work.

Thanks.

Dang: please consider removing the code-block functionality from HN.

Arguing simplification and then using DynamoDB, that is, opaque software handled by somebody else, does not make any sense or useful argument IMHO. It's like saying, we don't want to handle all this stuff, let somebody else handle it so that we can just be users. This is fine if is good for your company, but kinda negates the initial premise: you have a simplification problem if you own your infra, otherwise what's the point? Somebody else is handling all that for a premium price, I bet you have less software to handle.

I’m humbled that you read our post! We love your work, and heavily use Redis at Intercom. Thank you :)

The DynamoDB move is certainly a tradeoff, but for us it really makes sense. Moving to it will allow us to:

- scale out this storage layer trivially easily

- shed a ton of operational responsibility

- save lots of money

There are certainly downsides to moving to a proprietary database run by a single proivider, but we’re happy to make that tradeoff.

Also, we’re absolutely not recommending that people move from MongoDB to DynamoDB as some general rule. It just made sense for us at this time.

I feel a strange dissonance between the fantastic write-up, with which I 100% agree, and the ever so annoying intercom product... oh that horrible popup. What a terribly effective product though.

Nice piece with lots of takeaways but I think that they tend to lock themselves in AWS. This certainly adheres to the Run Less SW paradigm but they might suffer a vendor lock later on.

We've deliberately chosen to accept vendor lock-in with AWS. I probably should have made that explicit in the blog post.

We believe that the value we get from going all-in on AWS is greater that the potential negotiating power you may get by staying higher level and having the illusion of low cost switching to GCP or Azure.

This also accepts the risk that if AWS goes down, we go down.

I think both of those are acceptable risks/tradeoffs at this point.

In early stage you should outsource as much as possible but as you grow and get competition - in order to lower margins you have to incorporate full stack expertise and control the supply chains. However contraditional to the software solo-entrepreneur who's time is free, which allows almost zero operational cost vs crazy burn rates of some startups.

Related reading: "Ephemeralization", by Adam Wiggins (Heroku co-founder) https://adam.herokuapp.com/past/2011/4/7/ephemeralization/

Opening that link my ad blocker is going crazy. I guess the website surely runs less software, huh?

This philosophy fits well with a new open source framework I've been working on that is focused purely on simplicity.

https://github.com/jaequery/jasis

The goal is to have a framework that's so easy to use that even non-programmers can understand it. I'd really love to hear your thoughts, especially anyone that is new to programming. @jaequery

That article was a long winded way to say "Serve an underserved market efficiently, and execute sooner, better than competitors."

You don't say!!!!

That is a very concise summary! I still think the examples and detail are useful to explain how we do that at Intercom, rather than simply issuing a statement :)

This statement about software talent appears to be in direct contradiction with software development history:

"But as technology advances and provides us with more advanced software components it devalues and commoditizes what we’re working on and at worst makes us redundant."

Kinda wish my browser would follow your advice; the blog post is over 10 megs.

uBlock has blocked this page from loading due to this filter: ||intercom.com^ Peter Lowe’s Ad and tracking server list

You've missed out on a charming popup bubble offering you a... customer retention start kit, was it?

My readability bookmarklet doesn't remove the vertigo-inducing animated background. So I'm going to have to pass on reading this one.

Will someone please post a summary?

Annoying though it is, it's only on the title; the article text is readable.

And the <hr> elements between sections.

I also find that type of animation really annoying, even if it didn’t cause problems due to its high resource demand.

As a rule of thumb, I’d say animations should only be used for short-term things. Eternal animations will make a substantial fraction of people’s devices heat up, have fans speed up, &c. I think it is probably most.

And in the case of this article, animations don’t even fit with the art direction which is otherwise delightful.

If you're using Firefox, reader view works well (the note paper icon towards the right side of the url bar).

My adblocker blocks the whole domain, so I am also waiting for someone to post a summary.

It's because Intercom is sometimes classified as marketing software. Unfortunately their blog is hosted on their domain which is blacklisted by most adblockers.

The console browser w3m does fine. Others should work similarly well.

Good read and info

Does/Did Intercom use MySQL, PostreSQL and MongoDB at the same time?!

No :). Started with Postgres long time ago, switched to MySQL. Added MongoDB into the mix. Removing MongoDB from the mix.

Do you have a "Why MongoDB didn't work out for us." writeup?

No, not yet :)

What was the reason for switching from Postgres to MySQL?

I am interested as well — this is the first time I have ever heard anyone, without being under duress, switching to MySQL from PG. I would hope the reason is technical, but given it’s MySQL we’re talking about, it was likely a political decision.

This was a quite popular write-up of such a switch by a major company: https://eng.uber.com/mysql-migration/

With the HN comments being here: https://news.ycombinator.com/item?id=12166585

At the time we made the decision, we had a very small number of Postgres DB's in production and a large number of MySQL DB's in production.

We had built up a LOT of excellent operational tooling and expertise to enable us to operate MySQL to a really high standard and didn't have the same tooling/knowledge in place for Postgres.

Also, the Postgres instances were not chosen for any specific, differentiating Postgres features, more because the original developer preferred Postgres to MySQL.

We had recently encountered a few completely avoidable Postgres outages, mainly arising from the fact that they were not covered by the same level of supporting operational tooling that our MySQL DB's had.

We were faced with 2 choices: 1. spend a bunch of time leveling up our tooling and knowledge around Postgres 2. standardize on MySQL and migrate the small number of Postgres instances to MySQL

We chose option 2. And believe it provided the right level of improved operability at the lowest cost.

The icing on the cake was that soon after, AWS Aurora was launched with MySQL-only support (later they did also launch Postgres support for Aurora) which allowed us to get a 5X throughput improvement at a 30% cost reduction. These numbers were not just AWS marketing numbers, we benchmarked them with our workloads and found them to be true.

Reply on news.ycombinator.com