Choose Boring Technology
mcfunley.com
mcfunley.com
"Am I using this project as an excuse to learn
some new technology, or am I trying to solve a problem?"
Trying to solve a problem? I'm going to use what I already know. For web stuff, this'll be a super-boring, totally standard Rails web app that generates HTML (ugh, right? How last century), or maybe a JSON API if I'm trying to consume its output in a native app. For mobile stuff, this'll be an Objective-C iOS app.
Waffling about it and saying 'well, I am trying to solve a problem, and I think maybe a new whiz-bang technology is the best way to do it' is the simplest path to failing miserably. I've watched incredibly well-funded startups with smart people fail miserably at delivering a solution on-time because an engineer was able to convince the powers that be that a buzzword-laden architecture was the way to go.
You don't know what the 'right' solution is unless you understand the tools and technology you'll use to deliver that solution. Anything else is just cargo-culting.
My latest creation is a instagram-style feed of the beers a few friends of mine have been enjoying recently.
It works just fine and none of my non-technical friends have cared that they add beers in their mobile browser rather than via an app or something.
How about both?
If you're a PHP developer wanting to start some Python, your boss would love you to continue working with legacy codebase using PHP instead of migrating parts of it to Python. At the expense of your future career prospects, of course.
Sometimes I may say "can we do that on our 1K user side project instead?" but I think that if you are managing developmental resources and you are not allowing them learn new things, you are not following a strategy that is likely to work out long term.
How do you build unbounded learning time and an unguessable set of work items into a schedule?
Oh we need that in 2 weeks? And its about one weeks work the old fashioned way? Well try something else fun for a bit less than a week. Sometimes you win, sometimes you lose.
Sometimes you can do both in parallel. In the real world there's always wall clock time delays imposed by whatever. So when you're stuck waiting for whatever, learn as much as you can about XYZ till the main line is unblocked.
The truth is if you don't ensure your developers career development is baked into your schedules then you get a combination of high turnover, unmotivated engineers, and engineers who learn on the clock "in secret" by making decisions that are best for them rather than best for the company.
All of these cause far more problems and are harder to account for than being upfront in the first place.
Yes, and possibly not even consciously. Without that constant reminder of the pitfalls and learning curve of new technology, it's easy to convince yourself it's all upsides, or at the least undervalue the downsides.
I run an in house database with a Django front end.
Started off with the Django Admin application, which is great for getting things up and running quickly. Its definitely been worth learning more and more of Django, as forcing everything into the Admin gets pretty hacky, and basically generates a lot of technical debt.
Spending a bit of time and learning class based views for example (over the function based views) has paid off, as it leads to far more concise code, and basically less code to mange over the long term.
Today for example I know enough about Django and how the Admin application is built from it's components that I am adding features I wouldn't have assumed were possible a couple of years ago. (Stack Overflow hasn't got a decent solution to the question of a redirecting to a confirm page on save - I'll write one up assuming I get it working).
Then, when you make your decision about that new thing, you'll know why you like/dislike it. Your decision won't be clouded by arbitrary things like libraries inter-dependencies.
I'm the same way: I need a real problem to work whether I'm applying skills I know or skills I want to learn. I just don't ever promise anyone (except myself) anything when I'm applying technologies I don't yet know to the problem.
Its not really about a "last century" thing, "last century" most apps need not multiple front-ends like today, unless you are 100% you won't need multiple front-ends, doing a json api from the start simplifies things a lot.
unless you are 100% you won't need
multiple front-ends, doing a json
api from the start simplifies things
a lot.
Would you agree that it's easier[1] to write a web app that emits HTML than it is to write a web app that emits JSON which is consumed by a single page web app written in Javascript?
[1] Where I define "easier" in this context to mean requiring less wall clock time to get something showing up on-screen and doing something useful.
Relatively fewer apps were network-enabled, but even then, there were a bunch of technologies like CORBA, DCOM, and RMI to marshal communications. There were also a bunch of custom binary protocols; back then, it was common for software developers to work on all levels of the stack, and the culture of library re-use was not as entrenched (at least in the enterprise) as it is today.
To pick on one specific example in the post: Node.js is popular enough that there are lots of teams and engineers that are most comfortable and productive working with it. For these teams, choosing Node certainly wouldn't cost an innovation token, while deciding to build some service in Python, Ruby or PHP (if we take at face value that this is more "boring") may end up being more costly.
It absolutely does if it is only one team in the organization. If the entire organization is using PHP and, let's say, you acqui-hire a team based on NodeJS, unless they are doing something absolutely fundamentally different they should learn PHP and push code in your existing infrastructure. This way you have one way to deploy, one type of application server to support, one set of gotchas relevant to your domain, one set of QA tools etc. Building good products is about far more than just shipping the product, it's also about the cost of long term support. Because what you are doing is fundamentally automation, the less you have to manage the more benefit of the automation you are getting, the more you can forget about it and focus on shipping other things.
What you are describing is pretty much definitionally local optimization and is exactly what you shouldn't do in large engineering organizations.
No it still really does matter, because if your company needs to deploy those microservices in different ways then you need more people to support the deployment infrastructure. If you need to test those microservices in more ways you need more people to support the testing infrastructure. For engineers it's the amount of friction felt when trying to move around and work on new and different problems in your company because they're one of five people who knows how the hell the Java infrastructure works.
If you have an existing testing and deployment infrastructure etc. your team gets those for free and doesn't need to reinvent (and support) those wheels.
> As long as there are sufficient hire-able engineers who know that technology, it can be used.
Yikes. Hiring and firing engineers is pretty hugely expensive and something you want to help avoid having to do.
This is what PaaSes make a non-problem.
I should know, I worked on Cloud Foundry Buildpacks.
Here's how to deploy the PHP app:
cf push your-php-app
cf push your-nodejs-app
cf push your-ruby-app
cf push python-code-works-the-same-way
cf push your-go-code-with-a-godeps-file
cf push your-java-app-too
You know how making these surprisingly unalike systems deploy identically is really hard?
So does Heroku, from whom a large body of Cloud Foundry buildpacks code is derived. So did we, when we found issues specific to making code that assumes a connected environment work in a disconnected environment.
The point is, if you're doing this all by hand, you're doing it wrong. You should rent or install a PaaS and move along to the part where you create value instead of inventing a cool-sounding wheel.
That makes a lot of sense for small organizations, but I'm sorry PaaSs absolutely do not scale to the needs of many organizations.
> In general, these are all intended to Just Work™.
My emphasis added. When they don't Just Work it's really nice to own that infrastructure and be able to fix it yourself. It's also nice to be able to tailor things more specifically to your needs. Again, I agree that owning that infrastructure is not the right solution for organizations of all sizes, but neither are PaaSs.
Outside of the giants who rolled their own because there was nothing around in the early 2000s, who?
> When they don't Just Work it's really nice to own that infrastructure and be able to fix it yourself.
Cloud Foundry is specifically designed to run either in the public cloud, the private cloud, or both. You can get it hosted it from Pivotal or IBM, amongst others.
The work of my peers and I made that possible.
> It's also nice to be able to tailor things more specifically to your needs.
Cloud Foundry is opensource and the IP belongs to an independent foundation. I am personally aware of at least two companies who have private forks of buildpacks because that suited their extremely precise requirements. It took them about two developer days, tops.
And their modified buildpacks also Just Work™, because they're based on a robust design that Just Works™.
I still stick to my main point: your organization gets a massive benefit by all using the same toolset. If you are using Cloud Foundry, I'd still suggest the whole company stick with one language, one deployment infrastructure etc.
To be clear, if you're google I'm not suggesting the entire company all be forced to use one language or something. In that case your company is likely working on products that are different enough that it makes sense to do away with some global optimization. Some judgment is obviously required. But if you're in the sub-500 range (which the vast majority are) it makes a lot of sense to really optimize globally with your toolset, even if deployment infrastructure is relatively easy to setup.
PS I love that you are using the phrase Just Works - the company I work for is called Justworks :)
I think that the nice thing about something like CF is that a whole range of problems just goes away. On the other hand, as Weinberg observed, when you solve the worst problem, the second worst problem gets a promotion :)
Cloud Foundry doesn't get much buzz on HN. But I'm a one-eyed bigoted fan, so I mention it whenever I can. I'm actually a Pivotal Labs employee, my main work is agile consulting. But I've seen enough gigantoglobomegacorps who are choking on their own impossibly heavyweight deployment/ops mechanisms that I am a bit of a bore about talking up Cloud Foundry.
Sorry for being snooty up above. When you work hard on a product it's easy to become emotionally invested.
Know that feeling :)
E.g. we use erlang/cowboy for our web server and when there are bottlenecks changing that, they pretty much fall on two people who know erlang well enough to work on it.
It seems like it would have been better if the web server was written in something a larger chunk of the engineering org could modify so that when people needed changes to it for their project, they could make the changes, get them code reviewed by the maintainer, and get it shipped.
The other concern I have with microservices is that doing any wide-ranging changes is hard. Maybe it's always hard, but microservices seem like they would exacerbate the problem. I feel like I already run into this issue at my current job where we mostly have small (~10k) codebases, where people don't really want to make changes that will require making changes to more than two of them at once.
Substitute PHP with Java, and you've described the situation at my company exactly. The acquiring company had a legacy Java application and a lot of automation invested in making that platform work. The acquired company was a NodeJS shop that was using it long before this article or the comments in this thread would advise (this was pre-npm days). To give you an idea of the numbers, the acquired team was 4 engineers as compared to the 100 engineers of the acquiring company (50/50 split with an off-shore development team). I won't say which side of that divide I was on or go into the full year of culture shock that we went through, but fast forwarding these past 4+ years and now the bulk of the company's main product has been re-written in Node and developers are significantly more productive. Features that used to take months to push out in complex releases using a convoluted process of branching, meetings and tons of arguments are now delivered continually using the Github flow with little-to-no drama and far fewer production bugs/downtime. Our customers have never been happier with us and developers have never been happier to work here. All of this came from the fact that the CMO who advocated for the acquisition supported the small team of 4 in every effort to pervade the small team's technologies and practices across the larger organization. Having been in organizations that performed at a much higher level, he recognized just how much opportunity there was for improvement and recognized that the team of 4 had the vision to create the necessary blueprint for the rest of the organization to follow. It wasn't easy, and most of the developers who were here at the beginning of the shift are no longer part of the company. But it worked...and while a sample size of one is hardly conclusive, I have a hard time agreeing with your point having seen it play out so well in the real world.
I am not doubting that your scenario happens, but the possibility of changing a team of 100 (likely pretty jaded) engineers while certainly difficult is not necessarily what people think of when we're thinking acqui-hire. In fact, I'm barely entering my second decade as an engineer and I'm starting to think that surviving an acquisition intact and with career advancement somehow is probably far more lucky than hitting a start-up lottery jackpot in the first place.
There's gotta be a sort of trend of engineers that have gotten acquired so many times that their specialty now is to be able to scale / re-focus technology stacks and integrate and operationalize them better for other companies. Start-up companies typically want to see engineers that have a history of building stuff fast, growing rapidly, and the usual stuff that people get glory for as engineers. Established companies really aren't as picky. There's so many companies getting acquired you'd think that there's a niche for transitioning software over by now at least as contracting gigs.
I have a really hard time believing that Java was the culprit and Node the savior rather than the organizational stuff you mention...
Much like a change of location can help break someone's self-destructive habits, the change of platform helped break a lot of the toxic organizational habits that had built up over the years. The shift could have been to many other platforms. And if the platform had been something other than Java, a shift to Java could have improved the situation as well. The important part was that the new mindset and practices around more frequent/frictionless development and delivery.
I do think that it's easier to have that mindset when you use Node rather than Java, but Java has gotten better in this regard over the past few years.
In a project I worked on once, we went with "what we knew" (standard normalized SQL schema) to build an analytics engine. The problem with "going with what you know" is you are likely to badly reinvent well-established patterns. If we had stop for a minute and learnt about star schemas, the project could have ended in a much better shape than it did, and maybe the effective time to release would had been shortened.
BTW, learning "new things" is almost always useful but isn't always precisely exciting. Data warehousing is one example :-).
Becoming proficient with a selected set of technologies is still a good idea, but I'm willing to learn and try new things all the time. First thing I ask myself is if a problem was already solved by someone else, and how.
Put another way, the star schema was known to be a better approach only after the fact. Had your team researched “exotic” (unfamiliar) approaches early on, there is no guarantee you would have landed on a star schema.
Especially if you are working for other people, we are paid to innovate, and we are paid to learn stuff until it is boring. We've got to stick up for ourselves and learn on the job when we can. One new thing a project sounds great to me.
Using what you already know may not always be the best approach, because of "When your only tool is a hammer every problem looks like a nail" phenomenon. I would do my research and use what makes sense and what is best in the long run, despite of me being experienced or not in the technology. When you're a software engineer with years of experience under your belt, picking up the next one will not be a big challenge.
Both "innovative" and "boring" can shoot you in the foot. TFA focsues on "innovative" as a risk, but that's just daft. This industry is constantly rolling its lessons learned back into its shipped and shared technology. Ever gone back to a pre-Rails era web/backend codebase and screamed in horror? Ever gone to a "new" shop that never assimilated those lessons, used "boring" technology (thus dodging their shared/encapsulated forms), and recreated the old horror? (personally: check and check)
Trite policies are not a replacement for spending dedicated up-front (and occasional ongoing) time cycling between 1) evaluating/understanding your problem, 2) researching the current state of the art {processes, technology, etc.} related to your problem, and 3) using good engineering judgement to choose the best path then-and-there.
This is something worth fighting for, but by God, it's a hard fight, both in big corps and smaller "more agile" companies.
Great comment, anyway.
This is true, but it misses one of the points of TFA, which is that with boring tech you already know the ways it can shoot you in the foot, because lots of people have had their feet shot by it before you came along. You can learn what not to do just by looking around and seeing which sets of feet have the smoking holes in them. With exciting tech, you don't know; you get to be one of the people who discovers those things.
1. CGI
2. Python
3. Perl
As such I only use Python for scripting.
I'm not a huge fan of Ruby, but Ruby, its community, and rubinius are doing it much better.
PyPy is great, but until it gets to replace CPython it isn't something I would advise in production.
[0] https://www.tcl.tk/community/tcl2002/archive/Tcl2002papers/k...
What did you press it into service for in ye old .com boom?
Similar to AOL Server and already using quite a few patterns that people seemed to only have discovered years later with Ruby On Rails, but since Portugal isn't SV no one heard of them.
All critical path routines were actually done in C.
However, eventually the platform was migrated to .NET with focus on Windows.
The founders of this company went on to found OutSystems, with the lessons taken from this attempt.
No. Moving to Haskell did give me much nicer options for concurrency and performance. Was principally about productivity (area under a curve).
Java may be the worst example of a ‘bulb’ language I can think of. However, I recently spoke with a team which had an awesome response to all the things I hated about the language. The closest analogy I can think of is mechanics working on popular cars get to the point where they can diagnose problems in seconds because they know the kinds of things that break. Cars come with plenty of sensors to help diagnose problems, but in this case familiarity often beats better tools.
"You can’t pay people enough to carefully debug boring boilerplate code. I’ve tried." -Yaron Minsky
New hires don't need more than a week or two to get the gist of everything. The web frontend has been migrated from perl though. It got the job done, but it wasn't pretty and nobody dared touch it.
You shouldn't choose node.js or MongoDB because they are "innovative"-- but because they are poorly engineered. (Erlang did what node does but much better, and MongoDB is poorly engineered global write lock mess that is probably better now but whose hype way exceeded its quality for many years.)
The engineers are monkey's idea is that engineers can't tell the difference-- and it seems to be supported by the popularity of those two technologies.
But if you know what you're doing, you choose good technologies-- Elixir is less than a year old but its built on the boring 20 years of work that has been done in Erlang. Couchbase is very innovative but it's built on nearly a decade of couchdb and memcache work.
You choose the right technologies and they become silver bullets that really make your project much more productive.
Boring technologies often have a performance (in time to market terms) cost to them.
Really you can't apply rules of thumb like this and the "innovation tokens" idea is silly.
I say this having done a product in 6 months with 4 people that should have taken 12 people 12 months to do, using Elixir (not even close to 1.0 of elixir even) and couchbase and trying out some of my "wacky" ideas for how a web platform should be built-- yes, I was using cutting edge new ideas in this thing that we took to production very quickly.
The difference?
Those four engineers were all good. Not all experienced-- one had been programming less than a year-- but all good.
Seems everyone talks about finding good talent and how important that is but they don't seem to be able to do it. I don't know.
I do know is, don't use "engineers are monkies" rules of thumb-- just hire human engineers.
- The weaknesses of new tech may not be fully understood. A lot of new tech solves existing problems, while re-surfacing problems that the old tech solved. Everyone thinks it's great until they've used it for a bit longer, and run into those issues. - New tech runs a higher risk of disappearing/becoming unsupported. If you plan to support your product for a long time, that's a valid risk factor.
For myself, I'm wary of having very new tech as a fundamental underpinning of any piece of work I need to stick around. I'll likely adopt frameworks or database systems cautiously, unless their superiority is overwhelmingly obvious. On the other hand, I'd be a lot more willing to take risks on a simple library.
With a smaller, simpler piece of tech, it's easier to replace if something goes awry, and it's easier to evaluate in its totality prior to taking the risk.
The one that scrapes websites I did with Node since some sites are multi-steps and the latency of a single scrape, plus the database latency could've turned this into a multi-week run with PHP
Groups are a low-pass filter on the abilities of the individuals that compose them. To teach something to a group, you have to communicate it to every member; this communication is naturally bound by the ability to understand of the person who is least familiar (or least enthusiastic) about the particular tech.
You can often cut the time needed for a complex project in half simply by cutting the team in half and telling each group to work on it independently. The problem is that now you have two problems - or rather, two solutions. If you try to integrate them together, you end up reintroducing all the communication hassles and more. If you throw one out, you'll alienate and probably lose all the developers who worked on it. If you bring both to market, you confuse your customers and lose brand equity.
Actually, you have that exactly wrong[1].
"Behavioural economists and sociologists have gone beyond the anecdotic and systematically studied the issues, and have come up with surprising answers.
Capturing the ‘collective’ wisdom best solves cognitive problems. Four conditions apply. There must be: (a) true diversity of opinions; (b) independence of opinion (so there is no correlation between them); (c) decentralisation of experience; (d) suitable mechanisms of aggregation."
[1] http://www.diplomacy.edu/resources/books/reviews/wisdom-crow...
That difference is crucial. Markets function based on the wisdom of crowds; they work because if one person has the right information but everybody else is dumb, the one iconoclast stands to make a lot of money and force out all the dumb people. Statistics function according to the wisdom of crowds; it works because errors contribute little to the mean, while most people, arriving independently at their conclusion, tend to be closer.
Groups all have to agree on the same conclusion. When this condition occurs, the only conclusion that they can agree on is one that can be communicated to all members of the group, which is necessarily limited by the ability of the weakest group member to understand it.
This is why groups tend to be dumb. So much energy goes on hierarchical posturing and social signalling that there's relatively little left over for practical intelligence.
Orgs that can break through this can do astounding things. But the successes tend to be more rooted in the values of science and engineering as processes than in market processes.
Historically, every so often you get an org that works as intelligence amplifier and is more than the sum of individual talents.
But this configuration seems to be unstable, and so far as I know no org has ever made it stick as a permanent feature of a business culture.
If you want to be more precise, we often make assumptions of people belonging to a group that is not our own. The safest assumption to make is that all other groups are dumb. Ironically, this likely reinforces the problem: Why is this other group assuming our application doesn't have feature XYZ? Of course it does, because we're good at what we do. But obviously they must not be very bright to make such an assumption...
The problem isn't "we are starting a new codebase with 4 engineers, are we qualified to choose the right technology?" it's "we are solving a new problem, within a massive org/codebase, that could probably be solved more directly with a different set of technologies than the existing ones the rest of the company is using. Is that worth the overhead?" and the answer is almost always no. Ie: is local optimization worth the overhead?
Local optimization is extremely tempting no matter who you are, where you are. It's always easy to reach a point of frustration and come to the line of reasoning of "I don't get why we are wasting so much time to ship this product using the 'old' stuff when we could just use 'newstuff' and get it out the door in the next week." This happens to engineers of all levels, especially in a continuous deployment, "Just Ship" culture. The point of the article is that local optimization gives you this tiny boost in the beginning for a long term cost that eventually moves the organization is a direction of shipping less. It's not that innovative technologies are bad.
> But if you know what you're doing, you choose good technologies
No, if you know what you are doing you make good organizational decisions. It matters less what technology you use than that the entire organization uses the same technology. Etsy has a great engineering team and yet the entire site is written in PHP. I don't think there is a single engineer working at Etsy who thinks PHP is the best language out there, but the decision to be made at the time was "there is a site using PHP, some Python, some Ruby etc., how do we make this easier to work on?" Of those three python and ruby are almost universally thought of as better languages than PHP, but in this case the correct decision was picking a worse technology because more of the site was written in it, the existing infrastructure supported it more completely and so as an organization and a business we could get back to shipping products more quickly by all agreeing to use PHP. Etsy certainly does not think of its engineers as monkeys, quite the opposite.
But you're quite right. This is part of the reason why Java has been so successful - it's both boring and well supported.
In the startup world, we (rightly) focus on growth, but it's worth remembering that there are giant companies out there using really, really boring technology. In some segments IBM mainframes, DB2, and COBOL are still the technologies of choice.
These entities are now having huge problems trying to get off 1980s or 1990s non-boring non-standard technologies that are no longer supported.
There are places that have bought the company that was going insolvent that built their non-standard database or framework....
"Nobody ever got fired for buying IBM" had good reason behind it.
How many of those were really engineers, rather than managers, marketing people, consultants, support?
"9,199 employees as of December 31, 2014" -- that's probably close enough to his metric to call it a large-scale engineering organization.
Of course, there's the real question of why Facebook needs to be a 10k+ engineer organization. For a minute it looked like they'd grow past their MySpace 2.0 roots. That becomes less convincing every day.
Maybe 1/5th of that are software engineers of any sort. Sales, marketing, management, admin staff, QA, sysadmins, etc tend to fill up organizations.
[...] what it is about the current stack that makes
solving the problem prohibitively expensive and difficult
Plus, there are examples when the local optimization (i.e. staying with whatever legacy stack because it's proven) may lead to a global failure because of the unmaintainable "spaghetti blob" codebase with duck tape everywhere.
Analogously, if you promote by external hiring, you hemorrhage the kind of employee you'd want to promote internally. If you always stay in a particular sandbox, you lose the kind of employee that can work outside it.
Anecdotally, it usually turns out that good engineering practices can be brought into any medium. PHP comes with a higher than average number of foot guns, and there is a lot of terrible PHP code out there that is unfortunate to find when you Google something, but it's self evident that a good engineering organization can build solid systems in PHP. (See also: the diligent use of a specific set of C++ features in shops building cutting edge graphics/game technology.)
Of course, from a certain mindset I suppose anyone unwilling to sacrifice their happiness on the altar of your corporation's profit might be dubious... the question then becomes whether this affects recruitment and retention, and if so, whether you can still accomplish the things you want with mediocre talent and high turnover...
One way to keep that kind of work interesting and to provide growth and learning is to use new/different tools and technology to do it.
Nota bene, the creator of PHP works for Etsy
(Hey thanks for the comment, Will)
One thing I realized is these blogposts are consumerist. They talk about "Python" and "MongoDB". Very little about underlying ideas like "algorithms", "computational paradigms" or "expressive power".
And they have hypersimplified plans about "three innovation tokens". Instead of "risk analysis" or "evaluate tradeoffs".
One company shut me down with such blogposts... while it let devs run amok with an architecture which did n^2 (more?) network calls... where each call transfered one RDBMS row at a time. It dragged down the intelligence of everyone who really knew better; they spent "sprints" trying to find micro-optimizations, knowing exactly that the system was fundamentally ridiculous.
So I spent a weekend reimplementing it in the Scary Fun Language. Because it was my weekend dammit, and Embracing Boredom damaged my brain too much. Scary Fun was the only way to start mending it. And it succeeded.
So of course the first order of business was to rewrite it in the Embrace Boredom language.
Surely this is not the fault of boring technology, but rather poor design. You can easily make the same mistake using the latest and greatest.
It shouldn't be surprising that an organization that opted out of a discussion on technology stacks would also ultimately opt out of discussing algorithmic complexity as well, it speaks to a lack of sophistication and maturity at the institutional level.
They tried re-writing pieces in Java at a cost of tens of millions of dollars. Java was the new hotness. In addition, they built out a Java hosting environment using expensive, proprietary Unix hardware to reach the same production volume as the mainframe. However, it was grossly under-utilized because the Java code couldn't do much more than ask the COBOL code what the answer was to a question by using Message queues. More millions of dollars went to keep up licenses and support contracts on essentially idle hardware.
They tried moving it to Windows, using .NET and MicroFocus COBOL. But the problem was they would still be tied to COBOL, even though they (conceptually) had a path to introduce .NET components or to wrap the green-screen pieces in more updated UIs. But that in itself was a problem because all their people knew the greenscreen UI so well it was all muscle memory. Several workers complained because new GUI actually made them slower at their jobs.
They were stuck because they had no way to reverse engineer the requirements from the COBOL code, some of it going back 25+ years. Of course it wasn't documented, or if it was, the documentation was long gone. For the most part they were tied to that COBOL code because no one understood everything that it did and there were only a handful of COBOL programmers left in their shop (I think 6) and they were busy making emergency fixes on that + several other millions of lines of code in other systems.
They were, however, looking for an argument to retire COBOL and retire the mainframes. The cheapest solution would have been to stick with COBOL. Hire programmers. Teach them COBOL (because it was painfully difficult to find any new COBOL people and for various reasons they could not off-shore the project). Continue to develop and fix in COBOL (especially before the last remaining COBOL programmers died or retired). If you cleaned up or fixed a module, maybe move it to Java when possible.
The long story short is the decision to introduce a new technology, even in the face of an ancient, largely proprietary (since it's really about IBM COBOL on mainframes), and over-priced solution can actually lead to a worse outcome. Had they stayed with boring technology. Had they in-sourced more of their COBOL workforce. They might not have felt happy, but they would have been in a much strong, better position. Instead they were paying for a mainframe, and a proprietary Unix server farm, and software licenses on both Unix and z/OS.
When I last was there they were buying a new solution from Oracle which was supposed to arrive racked up and ready to go. Several weeks in they essentially said it would take months before the first of the new Oracle servers would be ready for an internal cloud deployment on which to try to re-host some software. I'm not even sure what they think they would be re-hosting but they talked about automatic translation of COBOL to Java.
Can you explain for people who never ever been close to such an environment how this can happen, and why do they still care about upholding the requirements they don't know about?
1982, Adds another port to the list of special port cities, but only if shipping goods of type JKL or MNO. That change was documented in an inter-office memo and filed away. Except the only time you have the type of goods information is in a different module - so even though it pertains to the original business process, it's in another module that prints the ship's manifest to (physically) mail it to the insurer.
1989, the original requirements binders are moved to a storage facility.
1992, The memo is also sent to an archive facility. Original manuals have been destroyed because the records retention policy is 10 years.
1994, There's a change in the law and an emergency fix was put in, and the comments were put into the source code.
1995, The source code with the comments is lost, so an older version of the source code is recovered with the just the code change.
And so on and so on
Until 2015. You have 5,000 to 10,000 lines of code that deal with the original requirement. They're split into multiple modules. They reside in a source code base of 5,000,000 lines of code. The people that use your software have a combination of the software + a whole bunch of unwritten rules like: "If it's this country, and this port, and this port of origin - PF10 to get to the override screen and approve the shipment. Add 'As per J. Randal' in the comments."
Just a small correction. Elixir was started in 2012. I assume you mean less than 1 year since version 1.0?
I recently went back to SQL from noSQL after I realized that a lot of noSQL was just reinventing wheels. I realize there might be cases where noSQL databases shine, but in most use cases SQL is better. It's slightly more work up front (only slightly) but it pays off later in keeping your data organized and making it easy to query. It's a great example of a very old technology with excellent longevity. That's in part because it's built on math and logic (set theory, etc.). There are universal mathematical/logical truths encoded elegantly into the structure of the SQL language, and they describe things you are going to need.
Your tools shouldn't be the exciting thing. The thing you are building with them should be the exciting thing.
"Pinboard is written in PHP and Perl. The site uses MySQL for data storage, Sphinx for search, Beanstalk as a message queue, and a combination of storage appliances and Amazon S3 to store backups. There is absolutely nothing interesting about the Pinboard architecture or implementation; I consider that a feature!"
> Can you explain why you think that's a feature?
> I believe that relying on very basic and well-understood technologies at the architectural level forces you to save all your cleverness and new ideas for the actual app, where it can make a difference to users.
> I think many developers (myself included) are easily seduced by new technology and are willing to burn a lot of time rigging it together just for the joy of tinkering. So nowadays we see a lot of fairly uninteresting web apps with very technically sweet implementations. In designing Pinboard, I tried to steer clear of this temptation by picking very familiar, vanilla tools wherever possible so I would have no excuse for architectural wank.
[1] http://webcache.googleusercontent.com/search?q=cache:98zuG6u...
Part of the problem is the jobs market.
I have been developing database applications for 12 years. Now many of teh jobs that I would seem suitable for want "experience in MongoDb" or "NoSQL skills" (whatever the hell that means - inability to design a schema?). I haven't used those because I have read up on them and decided they were not suitable for any use cases I have. I have used MySQL and Postgres successfully with billions of rows. I know when these new technologies might be useful.
If I need schema-less / JSON stores, I'll use Postgres. If I need things that won't fit onto one server easily then I'll look again at the NoSQL tech that is available and evaluate from there.
Another way to think about it is this: You get to change three axes in a product: new underlying technology, new product, or new process.
- Choosing one will allow you to progress with likely success.
- Choosing two opens yourself up to non-trivial risk.
- Choosing three means you will likely fail in this project.
There's a nifty talk by Steve McConnell about Software Engineering Judgement - https://www.youtube.com/watch?v=PFcHX0Menno - that goes into this kind of analysis.
You can debate which axes matter - you can debate the weighing and scaling of them - but you can't get away from the conclusion that "pushing all your risk boundaries at the same time equals failure". As a matter of fact, this is structurally identical to the famous "fast, good, cheap" triangle.
n.b., this analysis really starts hitting home in multi-team environments, say, over 50 engineers.
The author seems to be conflating the cost of innovation with the cost of doing something you're less familiar with, which are not necessarily the same things. The risk of chasing shiny new objects is real, but sometimes those shiny objects can actually reduce costs and time spent to accomplish a goal (like a MVP or new version).
Sometimes it's worth innovating if you already have experience in the area. Sometimes it's worth innovating even if you have to learn and try new things. Sometimes the time/monetary cost of innovation is 0, and sometimes it's so high that you shouldn't innovate even if it improves your product.
This idea of limited innovation resulting in cumulative costs is overly simplistic. The smart founder will recognize the difference between innovations that will yield net returns and those that won't.
Yes, you can look at this in terms of ROI. The author's point is that engineers - particularly ones who have never scaled & maintained a system over years and millions of users - consistently underweight the problems that they've never encountered before. With boring tech, other people have encountered them, and solved them, and you can Google for the answer or pull in a library. With bleeding-edge stuff, when you run into one of these, you have to drop everything you're doing and fix it, because nobody else will.
I agree with you completely. Furthermore, I'd add that the view on maintenance is too simplistic also. Effective maintenance requires more than just a tech stack where the limitations are known, you'll also want something testable and refactorable. Ballooning code bases are a real problem, sometimes the smart move is to clean up the cruft. If you're smart about integrating new tech into your stack there's no reason you can end up with a solution which is both more robust and efficient.
Furthermore, there's the whole scaling issue. Perhaps the mode du jour is just to assume increased server costs (regardless of where they're hosted) are just a necessary part of scaling a website to more users, but rethinking your tech stack can help keep these costs under control. Perhaps this is a decision that can wait until you have a decent userbase, but it's still a good reason to be open-minded about what benefits a new solution could bring.
Should be... "no reason you can't end up with a solution"
First, he is intertwining two seperate issues, limiting tech choices in an organisation, and incorporating new technologies. Keep in mind that you can have seperate strategies for both.
Secondly, there is no notion as to how big a change a token is worth. Obviously switching languages is a much bigger change than switching caching libraries.
Thirdly, there is no mention of project size. Should a 3 month project get the same tokens as a two year project? This year we have created ~300 microservices. If each was allowed 3 tokens we would have 900 new tech changes in this year alone. That's unmanageable.
Fourthly, what is your organisational strategy and culture? If an engineer prototypes in a new language is that a problem because it is seen as wasteful? Perhaps it is something that will make the other devs jealous? Or is it considered an investment in the company and a risk mitigation strategy? Do you have the kind of engineers and tech leads who will do a lot of this prototyping and experimentation on their own time?
For your third point specifically, I think taking a pragmatic view is the best. You mentioned you created ~300 new microservices this year. I imagine they're all based on the same pattern, so perhaps your tokens will apply to that pattern rather than each individual project (eg. you get to change the stack for future microservivces). On the other hand, at at least one new service per day, it's obviously pretty efficient for you, so consider why you'd change it unless necessary?
Want to innovate in the way your board is structured or remove standard protections from the term sheet? Or even set up your Twitter account in this never-before-seen way? Want to remove the idea of management, or rethink the way offices work? You lose an innovation token.
I'm fighting quota issues with my dumb host. I assure you migrating to something better is not my day job.
I just went with Wordpress. My personal blog is not my job, I just want to write down my thoughts. What specific technology would you recommend to generate a static blog with a WYSWYG editor and picture uploads, on my own server (not S3 or some proprietary paid hosting)?
More seriously, I work in vim+git all the time, so managing my blog with it feels natural to me. Editing in my favorite editor is more important than drag&drop image upload for me.
Seems like all the static site generators have lots of directory structure conventions and hoops to jump through for simple things like pagination and dates.
Well, git might on a time scale of decades. But I sincerely doubt we're going to see anything better in the text editor space than emacs and vi for the remainder of my professional career.
With smaller, more loosely-coupled modules, one can spend a fraction of a token here or there and still revert back to the boring way when necessary.
The whole article builds up on a point that people tend to fail more when they are using new tools. That point is false. In reality, when you use a wrong but ’accustomed’ tooling in inappropriate situation, you end up writing code that you would never write if you had chosen right tools. You are effectively reinventing the wheel.
You also have an idea about ’innovation tokens’ that builds up on a static representation of weight of a new technology in a project. That is ridiculous.
There is no definition of ’boring’ in this article. I don't understand why you call PHP, Postgres and Cron ’boring’. What is ’interesting’ then?
It seems like you have made a wrong choice while thinking about the problem. The problem is clear: people fuck up projects by using modern, hyped technologies that are inappropriate for project's domain. They are just as wrong as you are.
My hats of to its author for having the courage to write it.
It's plain wrong. Innovation is good for any kind of organization, if done properly. What the author should focus on is the lack of agility that prevents companies from experimenting and failing quickly. It's not the innovative technology that gets you in the end, it's your inability to evaluate/adopt/discard fast. Granted, that ability is hard to find in large-ish organizations, but to willingly limit your innovation sounds like a recipe for a slow death. It's like a gentleman boxer from the 19th century limiting himself to just jab, cross, hook entering a modern MMA fight.
In short, yes, I'm buying this. I think it's a perfectly sensible analogy; a somewhat leaky abstraction, if you will, since none of us actually have any "tokens" that we are trading in for a new database. But, the meaning is clear to me, and I can't find fault in it.
That's exactly what the article disputes - "It's not worth the trade if you had 3 of those already".
Sure, you could drown in meaningless (for your project) new technologies, and this is a risk you should be aware of. If that's the meaning of the article that you are referring to, I agree it is sane (and also wildly accepted). But that's not what the article actually says. What it says is that you get a superficial number of shots at new technologies, and that number is limited by time/growth.
Wow weird someone should totally think about that, maybe it doesn't even have much to do with the programming language
As you point out, the topic is not about programming languages or technologies at all. It's about how you handle and benefit from innovation.
Puppet with puppetdb can sorta fill this gap, too, as long as you don't need fast convergence (or fast puppet runs, if your puppetdb is more than a few milliseconds away from any of your nodes).
Consul may be new, but it's built on really solid ideas and technologies. You can read papers[1][2] about the underlying technologies to get a sense for how Consul will fail. I'd like to think that counteracts some of the problems you get with newness.
[1] https://ramcloud.stanford.edu/raft.pdf [2] https://www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf
Now as to marketing....
I look at things like projected read/write throughput, latency characteristics, total data volume, concurrency, and whether or not the problem domain actually requires highly relational queries.
I think that a lot of shops don't put enough thinking into figuring out what kind of data access patterns they'll need to support throughout the life-cycle of the business. This is no big deal if the product doesn't experience growth. But in terms of rich web applications which begin to experience growth the team inevitably ends up with a massive scaling problem unless their system architecture was designed to support these access patterns from the ground up.
It seems that this "growing pains" scaling nightmare has become almost a right of passage for successful tech startups. Founders are generally led to believe that it's a good thing for them to need to sell equity to outside investors in order to "scale out" a much larger team to build out the infrastructure required to perform in-flight rocket surgery on the application before it either explodes or becomes increasing cost inefficient.
While this whole process greatly benefits VCs, the high end tech engineering job market, and recruiters, it's absolutely terrible the founding team because it means they inevitably get massively diluted as a consequence of experiencing success. I'm not saying it's a conspiracy, but I am saying there is massive financial incentive to keep this kind of technical knowledge about best practices an open secret within the highly paid IT consultancy world.
TLDR: It's my supposition that small teams can build scalable, composable, systems by thinking about web scale data access patterns from the beginning.
I've been using it for maybe six years non-stop. I've thrown large data sets at it and it always runs fast; it's trivial to set up and always has more than enough options for my search purposes. It has also become a much better product over the time I've used it, with an active development group behind it. Sphinx works so well as is, I've never had a reason to look elsewhere at the latest hot search tech, it would be a waste of my time to do so.
Old boring tech (c. 2007): Rails or Django (the two are largely interchangeable, it's largely personal preference), JQuery, PostGres.
Old tech (c. 2012): Node.js, Angular, Express.js, MongoDB. Not boring, because you will still face lots of problems deploying this stack at scale.
Boring tech (c. 2012): Native iPhone/Android apps, JSON-RPC, often Java on the server. Usually Guice or Dagger is used for dependency injection with Java. Not really old (except for the Java part), there's still a lot of innovation going on in this space.
Bleeding edge stuff (today): React, Polymer, Go, Rust, Erlang/Elixir (Erlang is interesting in that the runtime and standard libraries are rock solid, but because it's so different from most mainstream languages, you can face a lot of integration pain when looking for third-party libraries), Haskell (old, but very different from anything mainstream). Basically everything you read about on Hacker News.
Erlang strings, for example, are lists of bytes, which will blow up your memory requirements and algorithmic complexity if you do any serious string parsing. You're shut out of common libraries like protobufs. There are libraries available for things like HTML parsing, MySQL, Postgres, and even Apple Push Notifications/Google Cloud Messaging, but many of them are some guy's personal project on GitHub rather than something that's gotten widespread use & testing and has plenty of StackOverflow posts for help.
Erlang does have a good Protocol Buffers library, by the way: https://github.com/basho/erlang_protobuffs. Even if it didn't, you'd use a more native serialization format like BERT.
As for abandoned projects and library sprawl, that is true. However, I'd say that this is far more bearable in Erlang than in other languages. For one, the module system makes deducing how to use a program's API from source code much easier even if there is no explicit documentation - every Erlang program basically gets a user interface for free just by virtue of being a module. In addition, if the library in question is a properly structured OTP application or if it uses vanilla process primitives efficiently, I can have relative confidence that it is less likely to blow up in my face than, e.g. a random Java library.
Even still, there's all sorts of libraries despite the small community. If you're doing web development and there's some RESTful API that has no Erlang bindings, those are relatively trivial to roll yourself.
Every stack will have some javascript in it.
I recommend Heroku/Postgres/Python/Django/jQuery as a good starting stack.
Sub Python/Django for Ruby/Rails if you want, keep the rest. Both of these stacks also make you very employable, if that's your goal.
If your goal is to seek employment 6 months from now, I'd say Django / Angular is a pretty safe bet. I guess if you want to focus on one language, you could use Node instead of Django and target frontend development.
I mainly recommend Django because of its incredibly friendly and well written documentation. It should teach the basics in an approachable way.
Also you might want to watch some lectures on how the web works at a high level as well, so you have a good idea of how all the pieces fit together.
All of the above, and then some.
If you're just starting out with web development, it's much more useful that you learn MVC and REST; those are architectural skills that will result in you being able to adapt to most frameworks rather quickly. I'd personally vouch for Rails (or perhaps Padrino) and Ruby as an introduction to those concepts, seeing as that's where my own personal experience is, and seeing as Ruby is generally regarded to be an easy and programmer-friendly language, but feel free to make a choice based on your own personal language preference (Django if you like Python, Catalyst or Mojolicious if you like Perl, Chicago Boss or Phoenix or Sugar if you like Erlang or Elixir (respectively), etc.). You can't go wrong, so long as you learn those underlying concepts.
Lots of folks around here express distaste for "polyglot" programmers like (probably, at this rate) myself, but I personally appreciate that focusing on thorough understanding of core concepts rather than the ins and outs of specific implementations thereof is really helpful in this context.
For example if my task is to create a typical website, that is relatively boring by default. Speaking about databases I choose postgresql not because it is boring but because that is the most convenient, elegant, and general-purpose solution I know. I dont't get excited about NodeJS or MongoDB by default. What I get excited about if I encounter a hard problem: maybe a very hard scalability issue that is impossible to solve with PostgreSQL. Searching for the solution to that problem is interesting and then I might get exited about some NoSQL solution. Also I don't really get excited about new programming languages as almost all programming languages try to solve the same problem. On the other hand I got excited about recent advances in Deep Machine Learning (and frameworks like Caffe, cxxnet) because with these new advances and tools it is possible to solve problems which we had absolutely no chance previously.
Also it is pretty standard for me that the stuff I do at the workplace is relatively boring, and the stuff I do at night by myself is exciting.
" Any server should have only one thing that is not installed via the default OS packages "
So you could have one weird bleeding edge version of your language or some unusual daemon that nobody else used but that was it. The idea was the rest of the server was stock and it had only one weird thing.
I think the issue I have with rules like yours, and that proposed in the article, is that they're fine when you're working with no information, but when an engineer lays out a need, shows how the "boring" package available does not fit that need, and then proceeds to choose an "interesting" package that meets the requirements of the problem, the last thing he wants is nebulous objections over how the choice is an "interesting" tech. For example, the article calls out consul (we also considered etcd and Zookeeper…) as an "interesting" choice, but need a multi-node distributed database with a good consensus algorithm for things such as service discovery and locking; what other techs fit the bill that aren't "interesting"?. Consul fits the bill. It's HTTP- and DNS-interfaces interest me because they play well with our existing boring tools, like http (or curl) and dig…
IIRC, [1]: default package lacks features [2]: unacceptable issues [3]: default package too far out of date [4]: default package is non-existant
(I'm also curious exactly how old your version of Ubuntu is, as testing the language on older versions of Linux is currently a bit of an annoyance and it's glad to know that someone out there is benefiting from that effort.)
While it isn't presently in a production setting, I would be completely willing to put it into production. There are some spots where we need more performance than Python can offer, and the memory safety and good static typing are very appealing (I'm vary wary of memory-unsafe languages anywhere near the path of user input…). (I personally use C++ presently here, but the included gcc in Precise/Trusty IIRC lacks some of the more modern C++11/14 stuff, so heavy use of Boost is needed. Also, some third party libs — mongo, to name one — have less than elegant C++ interfaces…) The Rust standard library has made great strides in the few months since I started using it (I started picking it up in December?), and since my normal work is in Python, the static typing is a very welcome change. I'm still wrapping my head around lifetimes[2][3], regex! tripped me up a bit[4]. I found some iterator based functions — such as zip — odd: it's a member function on the Iterator trait, and only allows two args. I find Python's free-standing zip function which takes any number of iterables much more natural. Compare Python:
zip(a, b, c, d)
a.zip(b).zip(c).zip(d)
Overall, it's shaping up to be quite a language. I sincerely hope it uproots C. (I'm a big proponent of modern C++, and so I'm still attached there. :D)
[1] I really wish upgrading happened faster; I tried to cull items that weren't relevant to Trusty because I do consider Precise out of date. We're limited to LTS simply because people aren't comfortable with non-LTS (and I don't know that we could upgrade quickly enough to stay on a supported OS…)
[2]: http://stackoverflow.com/questions/28956195/how-do-i-create-...
[3]: http://stackoverflow.com/questions/29001667/how-do-i-store-a...
[4]: https://github.com/rust-lang/rust/issues/23326 — (I had to follow up in the IRC room here, it's not quite as simple as the answer in the issue. Not only do you need to depend on the crate, you need the "#![feature(plugin)]" "#![plugin(regex_macros)]" as _crate_ attributes, and this can be confusing if your use of regex is in a module-in-a-modue-in-a-module; the use of regex in that module (foo/bar/baz.rs) causes you to need to edit a different file (lib.rs).
[5]: one of the things I miss about Gentoo's ebuilds in Ubuntu is that it's so darn easy to throw additional packages into the purview of the package manager. Building .deb files is vastly more complex than an ebuild.
For a 'wannabe entrepreneur learning WordPress', anything with more than 50 lines of code is insane. They may be able to build an ecommerce, forum,...., but usually no real value is seen.
Still, even if one takes an exception to this rule(example Groupon) it can't be compared to a real technical innovation.
"Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones."
Of course knowing the answer but not the question is best left to Douglas Adams.
The reality is, you need senior engineers, people who have built systems before, and are over the stage of their career where they just want to build stuff for fun, and are actually focused on building systems that provide value for the companies they work for.
It's not about new tech, or old tech, or boring tech, or exciting tech. It's about looking at the specific problem at hand and making an assessment about which technologies make sense to use.
As much as our industry doesn't want to admit, there are advantages to having real work experience.
"I dislike the use of the word 'sexy' to describe things which are considered attractive for non-sexual reasons, partly because it dilutes the original word and partly because it doesn't convey any reason for the attraction to the actual thing."
It was a world of pain.
You need to find a balance, but if you're doing something new in business you should use reliable tech, if you're doing new tech you need to isolate that.
Corollary: you can hire for standard tech, but have to ramp up in-house tech.
These days I generally optimize for fewer lines of code (boilerplate or not), as few dependencies as possible, and a general respect for the CPU cycles needed to run it.
We at GeoG are building a safe, easy and sustainable platform for IoT. We can totally do everything if we put time and effort into it - which we are ready for! What we want you to do, is start hacking and building things with us. No strings attached!
We recently set up a Community page http://community.geog.co where we want to start listening to you, your needs and suggestions.
Did we mention we have an API? We'd like you to start beaming data and create cool things with us. http://api.geog.co
Also, if you have reasons to hate us, bring it out! We're listening.
https://www.dropbox.com/s/1k7ngf52o822ccy/choose-boring-tech...
A lazy writing blemish on an otherwise good article.
"To be clear, fuck this guy."
Deleted comment
Fuck it. I'll do what's fun.
I get the point, if the end goal is the reason, don't get caught up in hype. But I won't lie, my job is not a end goal, as a employer never believe this.
Plus there's complicated issues like a bored worker is a bad worker.
Um. ;)