Ask HN: Has anyone implemented a zero-maintenance system?
I am more and more interested in designing and developing a zero-maintenance system - i.e. a system that once built requires no maintenance whatsoever.
Is this possible to achieve at all?
Is this possible to achieve at all?
A sun-dial, requires little maintenance after it has been set up. Cleaning off the bird feces and the leaves is all. It is not "zero" maintenance though. A trail duck is just a pile of rocks, and in the absence of getting knocked over requires no maintenance, but may require repair from time to time. A spacecraft on a mission to Pluto might require a software change or a course correction, but its hardware is not changed once it launches. A humming bird feeder sits on my porch and provides sugar water to birds in the neighborhood, when it runs out I pour in more.
So there are three things people often lump under the general heading of maintenance:
Repair - making something work after breaking.
Resupply - replacing consumable supplies, cleaning, and adjusting
Renew - giving something new capabilities or changing its operation by replacing parts or software
If you're being pedantic about the term, there is no such thing as 'zero maintenance' if you want a system to remain stable. Entropy is a thing.
So perhaps the more crisp question is, are people working on systems with really low maintenance requirements? Where "low" can be defined as hours of maintenance vs hours of operation or skill level of maintainers vs skill level of the builders, etc. Sure there are some really famous ones like the clock of the Long Now. Every museum in the world with interactive exhibits tries to design systems that will run with very little maintenance for a long time. Pretty much every operations lead spends time working on ways to keep large numbers of computer systems running with little human intervention.
Before internet use was common, there was just no really convenient way to deploy bug fixes, at least to consumers. Enterprise software updates were probably handled by a visiting rep who applied the updates, or a set of tapes with instructions to the local administrators.
But for consumer/office stuff, when you shipped something, that is what it was. You worked pretty hard to be sure did what it was supposed to do.
If you mean that software didn't change unless you manually altered it, well there's also OK maintenance. Unix systems would fill the disk with logs unless you cleared them out manually or wrote custom scripts to do so. Windows would need to be at least rebooted and more likely reinstalled periodically.
One way to set this time frame could be "life of the system" -- which might be say 2 years, or 5 years, or 50 years.
The more constrained your requirements and design are in terms of inputs and outputs and error states, the more you can validate and get closer to maintenance free (finite state machines are a nice pattern for this).
Many embedded systems (appliances, space program gear) achieve this, but I've noticed that as those are built more and more with off the shelf general purpose hardware and software they become more error prone and require maintenance (due to competition and time to market tradeoffs?)
edit: one final thing re error states : watchdog timers. These can be tremendously useful in reducing maintenance, and keeping functionality alive. When certain error states are entered, the watchdog timer triggers and the system can restart to a known good state and, hopefully, restore its core functionality until that particular error state is entered again.
Also I like what you said about timeframe. Why software doesn't have a warranty period after which the software is absolute?
Insistence on warranties would up the cost of software exponentially. Did you mean "obsolete" for absolute, or absolute meaning "fixed and immutable"?
Your question was a great one though -- and worthy of pursuit as a sw engineering goal in general!
Set a lifespan the product should be able to operate without maintenance, based on your goals and engineering constraints. Design with the lifespan in mind, perform lifecycle testing. Then release your product, stop working on it and don't offer support.
The feasibility of such depends on what you're trying to build. Customers have different expectations of lifespan, maintenance needs, and manufacture support depending on the product. A garden hoe is not an alarm clock radio, which is not a car, which is not a general purpose operating system.
The status quo of maintenance for any product domain is an inherent limitation to putting out something better. You can't just jump ahead of the competition because you will it so. You need some combination of: building with higher quality components, insight into the problem domain that none of the current commercial solutions have, or simply more productive (ie. more expensive) R&D.
If you're just looking at products that currently require maintenance and are licking your lips at the idea of selling a turn-key solution, join the party. Otherwise roll up your sleeves and make low maintenance a key design goal of your product.
At the time there was little computer hacking activity so you did not have to update the software either. Backups was done to floppy disks and there was a reserve hotswap backup computer. For remote control you would dial into the computer via modem, first via 300/1200 baud that is 1200 bits per second. Then later at lightning 9600 baud.
I'm also sure you find very good old engineering in the computers of Voyager.
https://books.google.com/books?id=le-M5s191moC&pg=PA126&lpg=...
So if security not an issue or removed from the project via some wrapper be that system locked in a room with control and other layers. Then you still have to plan for how long it is meant to last and being realistic.
WIth most business assets your talking 5 year write off at best tech wise with many 3 or maybe less, depending upon use.
SO possible - in the right situation - yes. But do not forget security.
THough remember time always a factor and also wear and tear and with that a pen eventually runs out of ink and required refiling. So dependant upon the timeframe doable but for many at what cost.
Another aspect would be look at what you have got that has fulfilled this criteria and not required maintenance and then ask, were you lucky or did you plan it that way as for many it usually ends up as some legacy system, hardly anybody uses or with low usage that has no internet connectivity at all. That is when you see your Amiga or C64 or early PC working away without issue.
But things fail and however well you plan things, things happen and to not at least check everything working and monitoring because it was designed to just work is something you should also plan. Expect the unexpected.
I had first thought that one of my projects was zero-maintenance as I truly gave it zero-maintenance in the three years between when I built it, and when I sold it. It was a website which indexed content from others. I had coded the "spider" in such a way that it was quite flexible if the content on the page changed a bit and it stood me well for those three years. I had a database back up system in place that backed up to another server.
But then I began to realise that although I had gave the site zero-maintenance, it wasn't truly zero maintenance. If I'd had a hardware failure, I'd have a backup of the database but no way to automatically spin up a new server and put it in place. But even if I had some monitoring in place for that, what if the monitoring service went down? I guess what I'm getting it as that there can never be a zero-maintenance self-sufficient service - just one that is incredibly well automated by great engineers.
I very much doubt that anything I buy from now on will last so well, mostly because of quite unnecessary use of electronic control systems that add unnecessary features and extra failure modes. Of course my new fridge uses only half as much power as the old one which means that the new one saves me about 125kWhr per year, which in turn means that it will take about twenty years to pay off the investment, good job that wasn't the reason I bought it.
As someone already pointed out you have to specify the expected life time to be able to evaluate 'zero-maintenance-ness' of a system.
The teams the develop these platforms (Java/Node.js/Ruby) have to balance support for old code branches with the development of new ones. Once the platform team leaves a code branch behind, the problems start to multiply. Zero day exploits are developed for zero day vulnerabilities. The underlying operating systems also evolve and leave the old versions of the platform behind as well.
At some point you have to open your applications code base back up and update it for the updated platforms. If you don’t the application dies with its platform. Sort of like the ship that sinks of rust or building that collapses due to the accumulation of deferred maintenance.
If your system is networked, there is always the risk of security updates being required. This can be minimised by extreme effort, but if you have to support SSL/TLS? At least you need a way of deprecating the vulnerable ciphersuites.
Would you want a web browser with zero updates? It would gradually become a handicap.
For consumer equipment, "zero maintenance" means "disposable": in the event of trouble, throw it away and buy a new one. Good for manufacturer turnover, not so good for the environment.
The other option is simply 100% outsourced maintenance, which has long been an option. Some mainframes would even do their own fault reporting, all you'd have to do is say hi to the technician and let him into the building.
I believe that software engineers failed in some ways because we always factor the maintenance as part of the development cost. In other words, we assume that we have to maintain a system once it is built. It is even more relevant with web applications which give the impression that they are in a constant flux. I don't buy the idea that web apps needs to adopt because standards don't move that fast and almost all browser are backwards compatible with sites that were developed in the 80s. I don't know what is behind HN but its simplistic, non-fluid interface hasn't been changed (I don't know actually) since it was made which for me shows that it is possible to make something useful without the need to constantly change. There are many example like that.
I also like the analogy that some of you wrote about physical systems. Indeed, there are physical systems that hasn't been changed for years and they still work. There is also a lot of software systems, mainly SCADA stuff, that are also designed never to be changed. Not long ago I heard a story about a pentest on a SCADA system that was controlling a dam. Despite that the pentest found a bunch of vulnerabilities, it was not possible to change the system because its author was dead long ago. The dam operated just fine even with the vulnerabilities in its software.
Another point I wanted to bing about is about defining time and version constrains. Back in the days software was not continues. In other words you get Doom 1. Doom 2 is a different story. Therefore, Doom 1 source code can go free. I would say that Doom was very much low to zero maintenance system, albeit a software one. Bugs were part of the character of the software. Hacking the software through its bugs was as close as it get to magic in the digital world. As we get more connected we require to maintain the software.
These are just a few things that came to my mind. I will add more thoughts as I go over your comments.
It is true that the visual UI of HN looks stable but behind the scenes they are always making changes to improve the community. Part of it is dealing with attacks & spam, part of it is trying to shape the behavior of users who are in the grey area between positive and negative contribution.
So far as security, that is another force for perpetual change.
IMO the key thing to keep in mind is that twenty years ago, it was very common to have systems that were truly "air-gapped" from everything else. E.g. in the 1980s or 1990s, someone probably had to operate that dam's SCADA system from a particular console in a particular room.
"As we get more connected we require to maintain the software."
Yes. As soon as everyone started networking everything together - and especially after they started putting everything on the internet - those old systems become a dangerous liability. IMO it is effectively impossible to have a truly fixed-design system continue to operate in a reliable way when it exists in a changing environment. Computing systems are too complex for any designer to understand all of the possible conditions in advance.
While it's true that if nothing goes wrong, the dam will probably continue to operate correctly, if it's now attached to a network, then there are all sorts of things that can make those vulnerabilities an issue later:
- Obviously there's the possibility of someone tampering with it for malicious purposes. Maybe they're a foreign government who wants to destabilize the US. Maybe they're a malicious stock trader who wants to short the stock of the dam owner.
- If it's a really poorly-written system from the 1980s, it probably crashes if it receives input beyond a certain length, because of e.g. a buffer overflow. Maybe someone mistypes an IP address in a configuration field and suddenly a database client tries to connect to it and triggers that bug.
- Does it represent time in some way? If so, is it going to enter unexpected territory when whatever counter it uses flips over? If it's old enough that the author is dead, it's probably using a 32-bit integer at most, and so is susceptible to the same kind of problem as legacy Unix systems.
Etc.
IMO, as soon as computing systems start to be networked together, they have to be treated more like plants in a garden than machines in a factory - if there isn't a "gardener" keeping things under control, it will turn into a disaster. It's still a deterministic system, but it's one in which the number of variables is far too high (IMO) to effectively plan out every possible contingency and response in advance. Maintenance can be minimized, but not eliminated.
The trick isn't eliminating maintenance, but in making maintenance as hassle-free as possible. The best way I've found to do this, the way I do it personally, is to build enough slack into the human system surrounding the automated system (in other words, the business you're working for) to adequately design, implement and iterate maintenance procedures. "Ship" the procedures by handing them off to an unskilled person so you can get feedback for iteration.
Best case scenario is a lot of automation and monitoring.
Where I work our dev / engineering team is investigating machine learning to automate repair a bit better. For example, over a 7 day period we record data for load on our backend and that ML model predicts and generates autoscaling rules according to historical data. Some of our services are about as low maintenance as I would think they can get (autoscaling, autoremoving poorly performing instances, etc) but I'd like to improve and only make an engineer get involved when the system can't be trusted to make the right call.
My wife is a financial person. Her billing system (it's a public utility) until 2009 ran on an AS/400 with 33mhz processor. It was probably rolled out in 1991 or 1992 and upgraded to support IP connectivity in the late 90s.
It's only connectivity was a modem that called IBM when something broke.
I made fun of her about it, but It actually worked great. The only reason it was replaced was that IBM ran out of spare parts for the printer.
Such a design makes sense only if the work the system has to do is predictably static though. Even in the case of Google Search Appliance, the moment your company adds a document in a new format the device will require some kind of external intervention to keep working.
It's just a matter of relation of cost versus maintenance work.
If you want to build a zero-maintenance system by yourself, just try to imagine your service and servers will be launched over to Pluto or a comet, and will never come back.
But remember that you have zero-maintenance if you pay someone to do the maintenance in your place. That's what people generally do when paying for cloud services.
Except that I suppose I'll still have to blow the dust out of it every so often, and make sure it's in an environment where it won't rust.
(To be less glib: what do you mean by "system"? What do you mean by "maintenance"?)
Also, zero is a very small number.
I don't think this is a good idea for a web server.. browsers change, you have to do some things to keep current..
So in a way, every time I realize I don't need to build a "system" for a given task, I am building "a zero-maintenance system" of sorts.
"No maintenance whatsoever?" I doubt it. Things break, get old, go obsolete, are insecure, and wear out.
Perhaps, you should worry about "nines of uptime" or "fixed requirements" or something along those lines.
Simplicity is first. The more complex, the more maintenance. "Everything Should Be Made as Simple as Possible, But Not Simpler" - http://quoteinvestigator.com/2011/05/13/einstein-simple/
"No moving parts."
Make it highly available and redundant. Power, cooling, networking, hardware, and software redundancies are needed.
Make it immutable. Change and mutable state will create maintenance. Implement functional programming, if you write software.
Monitor it and make it self-restart. Somebody already mentioned watchdogs, for hardware.
Make it ultra secure. No outside networking?
Program finite state machines....
If you imply a "hardware and software" solution, these points sound like you need redundant hardware and Erlang/OTP. Take a peek at OTP and the Erlang-based languages (Erlang, Elixir, Joxa, and LFE).
At least with redundant power, cooling, hardware, and Erlang/OTP (Elixir/OTP, etc.,) you gain the ability to do all of these things.
With Erlang/OTP, you can achieve very high uptimes, and if you design it correctly, you do have the ability to hot-patch running code, if you do have to (rarely) perform maintenance.
While you're at it, you also get distributed programming, concurrency, and parallelism, for free, with Erlang/OTP. This, in and of itself, can "reduce maintenance."
See https://pragprog.com/articles/erlang and http://stackoverflow.com/questions/8426897/erlangs-99-999999...
* do you need to keep supplying electricity?
* do you need to keep paying the bills so that someone else will supply electricity?
* will it continue working if there is a hardware failure? What kind of hardware failures can be tolerated? Do you need to intervene if the data gets migrated to a new system?
* will it continue working if there is a flaw (not just a bug) found in something? e.g. it uses TLS version X which turns out to have some flaw in Y years time
* what about it society changes in some way so that the load on it is quite different in the future?
* how long are you expecting to do zero maintenance? At some point the operating system you run on will be obsolete, and there will be no upstream fixes to critical security bugs - what will happen then?
Are you talking about software?
One of them is a piece of automation that looks for Active Directory accounts that are "inactive" based on a variety of criteria (including correlation with data not stored in AD). If they're considered "inactive", then they're disabled and their description is updated to indicate when they were disabled and why.
I originally wrote it as a 6-12 month stopgap until it was replaced by a fancy commercial account lifecycle product. I believe it's now 7 years later and it's still in use by the team I used to be on when I wrote it.
This may sound like a relatively simple task, but it actually wasn't, which is why (IMO) almost no one ever succeeds in building low/zero-maintenance systems. Some of the challenges I ran into:
- Some users only use their accounts in ways that don't update the last logon timestamp in Active Directory. I can't remember all of the specifics offhand, but one was that at the time, there were still BlackBerry users, and if they only ever used their AD account for email, and only read email on their BlackBerry, the timestamp wouldn't be updated, so the automation had to query the BES database to look at their last usage their too and use the most recent of that vs. AD. I think there were 3-4 things like this, and the automation used the most recent of them all.
- Employees go on maternity and military leave, and disabling/deleting their account would make for them being really unhappy when they got back. So the automation also has to check the HR system to see if their record is flagged as being on some sort of extended leave.
- The government requirement that spurred the development of the automation only applies to accounts for people. There are plenty of service accounts that log on infrequently enough that they would be disabled, so the automation also has to differentiate between those accounts and accounts that represent people.
In addition to the basic functionality, I also felt that it had to have significant safety features in place, because if something goes wrong and all accounts get disabled, then no one can log on to fix the problem. Among some of the other safety features:
- With each iteration, the automation calculates how many accounts will be disabled during the next iteration as well. If that number exceeds one threshold, warning emails are sent to the account administration team. If it exceeds a larger threshold, it will refuse to operate altogether.
- If no information was obtained from one of the data sources that it uses (e.g. a database was moved to a different server and the connection string wasn't updated in the account automation config), it will refuse to operate and generate warning emails.
- I don't remember the details, but there are special conditions for things like "too much time elapsed between iterations" and "the last iteration took place after the current iteration" to catch edge cases where there are problems with time synchronization.
I was able to build the automation in a way that made maintenance as close to zero as possible. In the ~7 years it's been running, AFAIK the only thing that's needed to be changed were the thresholds for number of accounts disabled in a single iteration, because the company expanded its use of AD and suddenly it became normal to operate against a thousand or more accounts at once instead of e.g. 200. A database connection string might have been updated when a DB got moved to another server as well.
Anyway, trying to predict all of the things that can go wrong even for such a simple system turned out to be a lot of work. My experience is that it gets dramatically worse as the system itself becomes more complicated (e.g. complicated enough to be a commercial product as opposed to an engineering maintenance task). I don't think it's really practical to do it with significantly more complex systems - modern computing involves too many changing variables.
For an analogy, consider trying to build a zero-maintenance system that waters and fertilizes a garden in a way that keeps the plants healthy. It wouldn't be that hard to build one that would handle the current garden. Imagine trying to build one that would handle literally any plant that someone could put in the garden. It's not impossible, but you'd need all kinds of wacky sensors and logic to figure out what each plant was and how frequently to water/fertilize it. It's much easier and less error-prone to just require that the gardener update a table that lists what types of plant are in which part of the grid, and maybe generate an alert if something changes that indicates that the table may no longer be up to date.
Not sure what is meant by "system" here.
What type of "system" is it? A database.
I got fed up having to do maintenance on my database, specifically worrying about having to expand its storage capacity. I'd constantly be woken up in the middle of the night because my entire web application had crashed because it ran out of disk space, because the database greedily pre-allocated excess space to improve performance. I'm not a DevOps guy, so figuring out LVM and MDADM on the fly was a nightmare. And as my app got more popular, things got worse and I just could not keep up.
But then I sat back and thought about it for a minute. I was deploying my app in the cloud, not on my own physical machines. Why was I worrying about expanding storage capacity when the cloud can sell me more of it than I could possibly keep up with? Why was I maintaining finite space when there was an infinite amount available? That doesn't make sense.
The other problem was that despite the fact everything else was working (the web server, the frontend javascript, etc.), if the database server was down then nothing would function properly because data is the life blood of my app. This sucks, if one piece breaks the whole thing is defunct.
So after a while, I decided to change all this. I decided to build a database that would be zero-maintenance, it hit the top of hackernews several times, some big name investors (Tim Draper) got behind it. Check out this demo of it automatically recovering from the complete loss of primaries - https://medium.com/@marknadal/gun-0-2-0-pre-release-auto-rec... .
So how is it zero maintenance? Well some other people in this thread mentioned some important points:
- There are fewer moving parts - there is no database server. Instead it gets embedded into your app server and your frontend (see next point), so the only thing you have to maintain is your app. If your app is flawless or (more likely) can auto-restart just fine, then you'll won't need to do any maintenance.
- It is highly available and redundant because it uses Peer-to-Peer architecture, like BitTorrent. So even if your server crashes, your app can continue to work in offline mode or with WebRTC (not implemented yet) continue to interact with other users. When the server auto-restarts at some point, the offline data will sync back up and resolve any conflicts. If you have more than one server running, then things will continue to work even if one or many of them crash.
- Immutable data allows the system to recover and resolve conflicts without your manual intervention. I'll let others talk about how awesome immutable data is, I'm sure you've heard enough about how many problems it helps with. Especially when it comes to maintaining systems, stuff can't get corrupted, and even when it does, the immutable data allows it to reconstruct itself.
- State machines allow the system in advance to know what is valid and invalid behavior, so it for the most part can avoid going down paths that are "incorrect" which lead to having to do manual maintenance, because it is already instructed in the first place what states to avoid or how to exit those states if it gets into them.
I'm super glad you asked your question, because I feel like a lot of software developers out there are super negative and bitter about it because most systems they have worked with have basically ruined their lives (like what happened to me, having to fix stuff in the middle of the night). But just because a lot of things have been like this, doesn't mean we can't borrow from engineers or mathematicians ways to make zero-maintenance systems. So I really hope you find what you are looking for or build one yourself, if you do please let me know mark@gunDB.io because I'm interested in that sort of stuff. Maybe we could start some group/forum around zero-maintenance systems!
I assume that that's not what you're looking for, though.
There's a fine line between "maintenance" and "improvement", and without the latter, you have stagnation. There certainly are systems that require very low levels of maintenance. I have a friend who built a program in Erlang that is still running, 10 years later. (I don't mean that the code is still in production. I mean that the program itself is still running.) Of course, Erlang allows the definition of "a program" to span multiple machines, and we're debating terminology here...
Pay-as-you-go maintenance is best. Don't allow technical debt if you can help it, push back against The Business on deadlines, certainly don't allow that micromanagement under the name of "Scrum" to get in or else you're just fucked when it comes to quality because you'll get a fuck-quality-I-need-to-complete-story-points culture, and create a culture of doing things right the first time.
Not that you'll necessarily use them, but learn a few things about strong statically typed languages like Haskell or Ocaml (Java doesn't count; that's shitty static typing). One of the great things about Haskell is that it allows safe refactoring. You're not holding your breath every time you change the code, because the compiler will usually tell you where your change broke things, and you can just go in a fix them. It is possible to write highly reliable software in dynamically typed languages (such as Erlang, mentioned above) and I don't mean to denigrate those tools at all, but it's a bit harder, especially when you're fairly new to programming, to do so.
Finally, once your system reaches a certain size, you will need tests no matter how good your type system is. They start to become an obvious win around a thousand lines of code. Consider generative testing (e.g. QuickCheck) rather than hand-written tests if you can.