A smarter SmartOS? (2016)
hypoalex.github.io
hypoalex.github.io
EDIT: Also, I rather like IPS packaging. Please explain how it sucks.
https://blogs.oracle.com/author/alibahrami
It’s true that the linker in SmartOS is pretty far behind, but that’s due to some choices by other parties. The linker in Solaris 11.4 is quite comparable (and in some ways superior) to GNU ld, at least on Solaris.
It's written in Python (because they wanted to attract developers) and it doesn't allow for pre- and postinstall scripts, nor does it allow pre- and postremove scripts. If you're doing very large scale configuration management with OS packaging, you're screwed really deep and really hard, because with IPS, all of that pre- and postinstall as well as pre- and postremove scripting has to be completely redesigned into self-assembly SMF methods, which is a gargantuan, extremely financially expensive effort.
It's slow, so the core parts had to be rewritten in C. Might as well have started with C to begin with.
Since they didn't want to allow scripting, they had to put in special hacks like "actuators" and driver hooks. At which point they might as well have put in scripting, but no, they remained stubborn. Another big sin. It's because both Hahn and Smaalders were/are for some bizarre reason absolutely convinced that scripting in packaging is bad and fragile, which is utter nonsense. So wrong assumption begat wrong, busted architectural decisions.
The design was actually an experiment by one Dr. Stephen Hahn. It had some good ideas like delta in-place upgrades (like SGI IRIX's inst(1M)), and incorporations (like IRIX inst(1M)'s depots) but the implementation sucks so horribly, with such horrible technical choices as Python that it should have never made it past the prototype stage. Apart from OmniOS and OpenIndiana, it is shunned by other mainstream illumos-based operating systems.
The automated installer which leverages it and was meant to replace JumpStart(TM) is an overcomplicated disaster.
The biggest sin of IPS though, is collateral damage: because IPS is so architecturally busted, sparse zones, a truly revolutionary and phenomenally useful capability had to be abandoned (to be brought back in SmartOS, thankfully).
I hate IPS with the burning passion of 30 trillion suns. Thanks, Dr. Stephen Hahn & Co.!
Post scriptum: I kept trying to tell them to license IRIX's inst(1M). They kept ignoring me. Now inst(1M) is bit rotting away in some vault and IPS is an obscure packaging mechanism preventing OmniOS from reaching its adoption potential. All around win-win.
Back in the days of SVR4 packaging, the scripts had to be extremely limited in what user-land features they could use because they had to be able to run in many different environments, including a) two minor releases back (so, S8 for S10 pkgs), b) netboot, c) boot media. This was insanely hard to support -- I know, because I was there at Sun.
By deferring execution of pkg scripts, IPS avoids the problem altogether. Instead, the scripts can only run in the running OS instance on which a pkg was installed (or from which it was removed).
This is much, much cleaner than traditional pkg scripting. So much so that I'm shocked this technique hasn't been more widely adopted.
This is patently wrong: if I'm running self assembly from SMF, that has nothing to do with IPS. If I invested the tremendous amount of engineering and funding, I could do the exact same thing from SVR4 packaging, which I have done in some packages. What do I need IPS for then? I don't! But wait, there is more!:
Back in the days of SVR4 packaging, the scripts had to be extremely limited in what user-land features they could use because they had to be able to run in many different environments, including a) two minor releases back (so, S8 for S10 pkgs), b) netboot, c) boot media. This was insanely hard to support
Auto- and thin client capability was ditched, so the very argument used to fuel this nonsensical course of action was made moot. And even with Auto- and thin client argument, that was not enough to justify the atrocities committed by IPS.
No thanks. If I can't have IRIX inst(1M) or HP-UX swinstall(1M), I'll stick with good old SVR4 packaging. Since Snoracle killed AutoClient and thin client support anyway, configuration management with SVR4 works wonderfully. I have my scripting in the immediate installation context for configuration management in tandem with SMF and it's great. IPS can remain an obscure packaging technology nobody's using anyway for all I care. And that is good so: it's a totally botched implementation of inst(1M) and swinstall(1M).
By deferring execution of pkg scripts, IPS avoids the problem altogether. Instead, the scripts can only run in the running OS instance on which a pkg was installed (or from which it was removed). This is much, much cleaner than traditional pkg scripting. So much so that I'm shocked this technique hasn't been more widely adopted.
That is exactly what killed the adoption of IPS and severely handicapped the adoption of OpenSolaris and all operating systems which were IPS based. I cannot believe you (plural) are still refusing to see it! As if the entire Solaris ecosystem didn't have enough adoption problems, this just piled on and made it far worse. To someone like me who grew up on and deeply, deeply cares about Solaris and illumos and SmartOS, it's absolutely infuriating. You made your own lives easier but it cost you the company (Sun Microsystems), and it cost me having to primarly work on GNU/Linux based operating systems these days. We all lost.
Empathy is still a core engineering value. With IPS, there was no empathy. All your customers wanted was more freeware bundled with the OS and automatic dependency resolution for easier patching, things which could have been easily added to SVR4 packaging. They didn't need or want someone's doctoral disertation implemented as an experiment.
I don't even remember how many userlands had to be supported by SVR4 packaging at its peak, but it was a lot, and it was difficult to test. That some of those were removed eventually hardly matters: the problem remains, only slightly reduced in magnitude.
Moreover, I want less package scripting. So many Linux packages want to do stupid things like... interact with me when Ubuntu/whatever is just updating the system. It's never appropriate for, e.g., a PostgreSQL package to create a postgres account and configure a postgres service running as that account -- this is not how I want to run postgres, and if you stop this madness then suddenly a postgres pkg is just code and no pkg scripts. If doing stupid things in pkg install scripts is what you call empathy, then I want none of it.
And by the way, IPS had nothing to do with Sun's failure. That can be chalked to a long line of terrible executive business decisions, such as:
- suspending/killing x86 Solaris support in 2002
- killing SunPS (especially considering how much money IBM made with theirs)
- acquiring Cobalt and then promptly killing it
- buying MySQL
- not seeing the writing on the wall for Java ME when iOS came out
- and much more
That is exactly how I want and do configuration management!
The main PostgreSQL package brings on the software and the SMF method. It depends on the postgres-user package which creates the technical user account and group. That's exactly what I want, because I plug that into JumpStart and I can churn out database servers all day long!
The postgres-application-db configuration overlay package depends on the postgres-user and postgres packages, starts the database via SMF, and then runs CREATE DATABASE in the postinstall. That's exactly what I want! It works a treat for Oracle databases, it works for anything. I can churn ready to run database servers all day long. I love it. That's exactly how it should work, infrastructure with no human interaction whatsoever: plug in the cable, scan the chassis serial barcode and call it a day.
Sun had many problems, but IPS was the final nail in the coffin, as a customer, and as a professional whose bread depended on that ecosystem, and who is now forced to work with substandard, shoddy Linux, I feel I have a much better insight and far more right to decide what has been done wrong at Sun Microsystems. That's for us as customers to say, not for anyone else. And ultimately customers voted with their wallets. I know I sure did, I ran Solaris 10 on generic intel 19" rack servers (and I still do) because it is a far better value for the money.
Overpriced hardware and unwillingness to listen to people like me, who were your core customers was like turbo acceleration of the company's demise. And if you lose your core customers, you lost your business. You lost the purpose of your business' existence. Proof is in the pudding, Sun is no more, and as usual, I'm right about the wrong thing.
If I'm paying you, you don't argue, you do as I say, or you have no business. Now there is no business and competition rules supreme. It's a terrible business decision to argue with one's customers, because they are the ones who decide what they will give you money for. IPS is not a thing I would ever give Sun or any other company which uses it money for. IRIX's inst(1M), I'd give tons of money in a heartbeat if I knew how; but IPS - never!
IPS is the single reason why I don't run operating systems which sport it, like OmniOS.
It is beyond awful that what should be an unattended operation (install, upgrade an OS) isn't. It's completely unacceptable, and utterly intolerable.
Yes, I'm aware of "answer" files. That was SVR4's compromise: you'd put all interaction into one kind of script that can be run separately from the actual pkg installation and which produces "answers", and then you can furnish those answers at actual pkg installation time. But in practice most developers were not really aware of all of this. The cognitive load of SVR4 packaging was too high. Interaction and package scripting really complicates packaging. Please stop.
The database engineering department decides database names and creates the appropriate packages; that is no longer the decision of the system or database administrators to make. It is no longer their job and hasn’t been for at least a decade and a half.
Because implementing and debugging the asynchronous execution of postinstall scripts is extremely difficult, and preinstall / preremove / postremove nigh impossible. IPS was an extremely bad idea by people who have never done large scale configuration management with OS packaging.
I don't mind it creating a postgres account, but what I do mind is the package automatically enabling the service, and starting the service.
Even worse, on Linux, most packages will be helpful, for example upgrading the MongoDB package on CentOS? It'll helpfully restart the service for you.
Nevermind that you've just scripted this and it's being run on all of the servers all at once so instead of doing a nice easy rolling upgrade WHEN you are finally ready, it's done at the same time.
I hate this behaviour. Let me worry about the lifetime of processes, don't auto enable them, don't auto configure them, let me worry about making sure systemd knows it is time to start/stop them.
FreeBSD let's me upgrade packages all day long, and until I restart the service, nothing happens. That way I can upgrade the packages, make sure everything looks good, then restart the service.
That’s exactly how it must work for total automation where humans are out of the picture. What you lack is a push, rather than a pull methodology, with a software deployment server and a capability maturity model level 3 (or higher) change management process around it. Then it works flawlessly, and I’m writing from experience of several decades of modelling and implementing such things on a very large scale (tens or even hundreds of thousands of servers).
That’s my specialty as a technical architect.
Why would I expect a package manager to also restart my services?
I need to make sure that other parts of the stack can either handle that service going away gracefully, or have automation around it, my package manager can't know the dependencies in which certain services should be stopped/started for example to allow for graceful failover. It's illogical to try and place all of that logic in the package manager, that is not its job. Especially if there are other config changes that need to happen too that are not part of that package management system.
Humans can still be out of the picture without a package manager starting/stopping/restarting services. Your suggestion of it being part of a maturity model don't make sense. I asked for the package to be upgraded, not for the service to be restarted.
For example: a package delivers /opt/abcd/sbin/pgadm, which would be your engineered command line interface to CREATE DATABASE, database backups, and so on; then the application-db package calls that in its postinstall to create a database specific to that application. Best of all, once one has such packages, they can be plugged into Kickstart, JumpStart, AutoYaST, or whatever the OS provides for automated provisioning and hardware whether physical or virtual, turned into fully configured systems from literally nothing with no special or custom provisioning required. You could lose entire networks of servers and prop them up in seconds again this way.
> I don't mind it creating a postgres account, but what I do mind is the package automatically enabling the service, and starting the service.
IPS is one part of a fairly well-integrated system. IPS has several different actions to deliver various things to the system. These include file, directory, user, group, etc. actions. Delivery of a postgres account in IPS is done via a user action, not via a post-install script or a service that runs after installation.
If you run a shop where you don't want the postgres account to be created, you can use pkgmogrify to transform packages. pkgmogrify makes it easy to remove actions (remove the postgres user) and transform others (set ownership of files to someone other than postgres).
IPS allows signed packages and allows you to specify which signature(s) are acceptable. A shop that wants to only install packages that have passed their QA process could require that their own signature is on every package.
Taken together, those features make it quite possible to ensure that your systems never get software (via IPS) that you've not blessed. Presumably you use automation to detect things (user actions) that you feel may be objectionable and only the new objectionable content requires manual intervention.
When you really do want postgres to start, that's pretty easy to do too. The odds are that you don't want it to run as postgres, but as some other user. Maybe you want many instances as different users. This becomes fairly easy to handle by delivering another package that depends on the postgres package. That package can deliver user action(s) and an SMF profile that creates and enables instances of the postgres service.
I did a bunch of work for Solaris 12^^11.4 to have profiles that may be applied at various layers. The enterprise-profile layer applies to things that are common across all your machines. The site-profile is for a particular site and takes precedence over the enterprise-profile layer. Then there's the node-profile layer that can be specific to a given machine. All of these layers are intended to accept packaged content (or unpackaged, perhaps delivered by puppet or similar). If you want to control the behavior via install-time profiles, you can do that with sysconfig profiles which are fully integrated with the automated installer, 'zoneadm install', etc. Details at https://docs.oracle.com/cd/E37838_01/html/E60998/dzhid.html .
IPS is part of an integrated system that is designed to solve the problems that many people assume should only be part of the packaging system. While some criticisms of it are valid, I can say that I became much more fond of IPS as I started using Linux again after a long hiatus.
Automatic startup on install is supremely annoying. There's a way to disable it on Debian at least, but it's terrible as default behaviour
Do you package all software by yourself? I mean, if the vendor-provided httpd package for example started and enabled itself upon installation, how would you structure your configuration packages so that httpd does not start until it's actually configured? Or (theoretically) if you needed to deploy a configuration change and an upgrade simultaneously and can't restart the software before the new configuration is in place.
I see this being rather easy if the vendor package simply puts the binaries in place so you can have your own packages run whatever scripts it is you need and depend on the vendor package, but I don't see how it wouldn't be a headache if the vendor package contained scripts to manage the service as well, which is what primarily annoys me on Debian-likes.
It is the *-conf package which depends on the application package, brings on the configuration and automatically (re)starts the application.
As a quick test, I installed httpd on my workstation, and it did not get automatically started or even enabled at boot.
I don't think there's anything wrong with using packages to control which software is running if you do it right, but I do think it shouldn't be default behaviour.