Hacker News Reader

Comment by capableweb | Hacker News Reader

Funny, I run Ubuntu on my laptop and tried just today to upgrade it (since I'm on 18.04 and have been for a while). But no matter how I try, `do-release-upgrade` won't do the upgrade, complaining about third party repositories (removed all of them as a result), mismatched python versions, and a load of other issues. When I fix one, another one appears. Tried for 5 hours today and eventually gave up.

On the other hand, my desktop machines run Arch and NixOS and I have never had any issues upgrading. Well, actually a few issues with NixOS upgrades, but it's so easy to rollback that I barely spend any time on it, and when I later try to upgrade again, everything just works.

I wonder why Ubuntu/Canonical can never get their upgrade game to work properly. I've had similar issues with Ubuntu upgrades since I started using Ubuntu (8.X) and seems it'll never be fixed. Maybe I'll just start using either Arch or NixOS on my laptop too.

View on HN

> I wonder why Ubuntu/Canonical can never get their upgrade game to work properly.

Ubuntu developer here. Sounds like you have a really hacked up system there, and that it's your existing system that has issues. Under these circumstances, I don't think it's reasonable to expect an upgrade to be able to work smoothly. Scripts that handle upgrade paths necessarily need to make assumptions that what they are upgrading from is what the distribution put there. If you hack it up, then those scripts aren't going to work.

Your Arch system, being rolling release based, I assume isn't as hacked up with newer versions of things because you get them straight from your distribution? The equivalent on Ubuntu would be to run the six-monthly releases, which I assume you choose not to do. If you do that and then you hack up your system, then I don't think it's fair to compare to an Arch system where you (presumably) don't do that.

If you want to run an Ubuntu LTS system then I suggest that you do your hacking inside containers (whether system containers like lxd, or app containers like Docker). Then you won't need to hack up your system itself, and then upgrades will generally work fine.

To be fair, some moving parts outside of canonical's work are necessary yet break often.

Prominent example: Nvidia drivers. Oh how often have I wiped a system because the precise combination of driver/CUDA/nvcuvid/nvenc and nvidia-docker broke through some freak update.

These are the things that often prevent smooth upgrades.

> Sounds like you have a really hacked up system there

Some people would call this 'a system that's actually being used to do something'.

Don't blame users unless it's clear they are doing something egregious with the system. Installing a different Python version shouldn't break installers, but here we are.

Ubuntu should have taken a page from CoreOS, NixOS and even Arch. CoreOS with their immutable system partitions (and two of them so it can revert to a good one if the current can't boot). NixOS with the immutable packages (and the split between user and system packages). And even Arch with its release model.

Sometimes you have to 'hack your system' to add a new apt source to try to get your GPU working with OpenCL, or you have to modify some config file on /etc to get something else working. Or whatever the use-case might be. Should that break upgrades then? Ubuntu really doesn't provide any guidance or utilities to help with this.

At least figure out a way to install a known sane system configuration, even if it breaks the user's customization. Having a separate system partition by default would actually help a little.

Even Windows allow you to snapshot and rollback changes. Something that Ubuntu users have to – guess what – "hack the system" to get.

> Ubuntu should have taken a page from CoreOS, NixOS and even Arch. CoreOS with their immutable system partitions (and two of them so it can revert to a good one if the current can't boot). NixOS with the immutable packages (and the split between user and system packages). And even Arch with its release model.

Actually Ubuntu Core is an edition of Ubuntu that has exactly these properties. It runs read-only, with packages that can be added and removed cleanly.

This does involve moving away from apt/deb, which is fundamentally incapable of this. I don't know of any plans to do this for the main Ubuntu distro itself, but consider the resistance if this were on the roadmap.

Obviously you put the effort into the Arch one to set up everything with Nix, and didn't do that in Ubuntu. A fairer comparison would be "a system that's actually being used to do something" in just Arch to Ubuntu or Nix on both.

I'm not a heavy linux user but I know 3 guys who use Arch at work and one of them had an upgrade issue that caused them half a day to fix, this was circa 2016. I know plenty of people who use Ubuntu no issue. Not that such anecdote really means anything, just to demonstrate that they can be used to back whatever bias.

> Having a separate system partition by default

This is just one simple extra step when you install. And I don't know if arch has this by default?

> Even Windows allow you to snapshot and rollback changes

just use brfs? This is more of a linux thing has nothing to do with ubuntu

> And I don't know if arch has this by default?

What do you mean by default? :) Arch doesn't have an automated installer that does partitioning for you. You extract some files to whatever partitioning setup you've dreamed up for yourself, setup a bootloader and reboot.

You are the one who asked for default.

Ubuntu has that but I don't think most people use it either. I don't see any harm as an option for beginners. Not that it actually happened but my parents should be able to set up ubuntu because they don't need to partitition.

    Installing a different Python version shouldn't break installers, but here we are.

laughs in https://xkcd.com/1987/ .

Hah I always imagine this is what we will finally determine DNA to mostly be.

I wouldn't expect nothing less than being blamed by Ubuntu developers for a hacked up system, even though I barely described anything.

Thanks for giving me another reason to leave the Ubuntu ecosystem behind me. The condescending tone that your message has seems to be relatively popular in the Ubuntu/Canonical community as well, at least compared to NixOS.

You described third party repositories and mismatching Python versions. That is a hacked up system, by definition.

Seems uncommon that those things would prevent upgrades. It sounds more like Ubuntu is beginning to suffer from an excess of complexity.

I think part of the problem is that third party apt repositories have become normalised as if they're a reasonable way to "add on" to your existing system.

In reality they're a hack on a packaging system that was never designed for third party plugability in this way. They very often break future upgrades. Third party apt repositories fundamentally cannot express all the necessary metadata to allow for upgrades to work in the general case. They don't even namespace properly; apt cannot even tell the difference between a third party installed package and one that came from the distribution. And third party packagers typically don't even consider future upgrades.

This is why Ubuntu is working on snaps: they are a mechanism that allows for third party plugability in a way that does not break the system.

If we want upgrades not to break, the system dpkg/apt installation needs to be left alone and not be interfered with by third parties.

This is at the core of why I've found Ubuntu (More broadly Linux?) to be frustrating to use. Installing programs or functionality beyond what it ships with leads to the system ending in a "hacked up", as you put it, state. Then reliability tanks.

So, what should I do if you don't decide to include critical release of a package that was out there for 6 months? https://github.com/swaywm/sway/releases

Sorry that wasn't included. Please understand that it's not about "you" and "me" though. Anyone can contribute to Ubuntu, and "any responsibility can be carried by any contributor who demonstrates the required capacity and competence" (https://ubuntu.com/community/code-of-conduct). So please join us and help to make this happen next time.

Of course there are processes and policies to help us ensure quality and that other user expectations (eg. stability, integration) are met as best as we can. These take time and effort to negotiate. But we'd be happy to help to explain what's needed to be contributed to make progress in Ubuntu itself for any specific circumstance. Find us in #ubuntu-devel on Libera Chat, or https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discu...

Here's some further information on sway specifically. Looks like it couldn't be updated because depending packages weren't ready for the switch: https://bugs.launchpad.net/ubuntu/+source/phoc/+bug/1938404

> Third party apt repositories [...] don't even namespace properly; apt cannot even tell the difference between a third party installed package and one that came from the distribution.

This is true. dnf and zypper handle this just fine in the RPM world, and they operate under basically the same paradigm. Are there no plans for apt to catch up?

I don’t know much about python, but for nodejs dev work I have about half a dozen versions of nodejs installed. It’s super handy having all those versions installed so I can test that a certain native nodejs package I develop builds and works properly. And to rule out nodejs itself when reproducing bugs. I use nvm (or fvm) or something to manage all that because the system package managers updated too slowly.

What are you saying here? That having multiple versions of python installed can put your whole OS into some sort of unrecoverable corrupted state, and that’s expected and ok? That’s a non starter for me.

If you replace the version of python that the upgrade process expects, then yeah, you'll probably run into problems. The upgrade process can't possibly support every single version of any given package, that would defeat the entire purpose of having a stable release cycle. If it's a core package like python which sits really close to the top of the dependency tree then the effects of swapping that out will cascade downward and that's why everything breaks so badly.

In my experience, this is the only safe way to have third party apt repositories:

- Don't install any third party packages which have reverse dependencies. This means no libraries, only applications. If you need newer libraries as a dependency then you have to statically compile the application.

- Don't install any third party packages which have the same name as an upstream package. If you want to install a newer third party version of an upstream package then you need to change the name of it and have it install all of its files in different locations. A prefix change is enough here, /opt is a popular choice.

- Delete all third party packages before doing an upgrade, in case a new upstream package happens to have the same name, or needs to put files in the same locations.

Do anything outside of that and you will risk getting your install into a broken state. Because even though you might think you are just installing some applications, what you are really doing is bolting things on to the upgrade process for the entire core system. Using nvm is perfectly safe though because that just installs the different versions to your home directory, it doesn't touch apt.

That makes sense; though to be clear, you're essentially saying that apt isn't appropriate for 3rd party packages. And that:

- The question of which files go in which directories in the system is a question for canonical (or whoever) to decide. Not end users or 3rd party developers

- The namespace of apt package names is special, and 3rd party / external dpkgs shouldn't add anything into it

I think this stance is entirely reasonable. Its essentially what windows and macos both do (and to increasing degrees with each version). But its a pity its taken the current apt packaging disaster to realise it. Its really convenient for users to have a package manager, and for 3rd party developers be able to write software for it. And these problems seem solvable:

- Namespace all 3rd party packages. Eg, @apple/foundationdb-server (as distinct from @canonical/git)

- Use overlayfs or something to essentially make /usr, /usr/bin, etc read only by normal processes. If these paths were only writable by system packages, the whole system would be much more predictable (and verifyable) - since that filesystem would only - could only contain a subset of the files in apt.

I get that flatpak and snap are both attempts to replace apt. But its a pity they're needed. I'm enjoying mint, but if I ever rebuild this machine I'm going to give nix a try.

Yes to all of the above. Although namespacing is not enough, the packages need to be built in a specific way so that they don't put files in any system locations. This isn't anything specific to Ubuntu, this is just how Debian works. If you want to do things "the right way" you're supposed to submit your packages upstream and be patient.

This is not exactly a new problem, it's why Docker is popular, it's why Fedora Silverblue and NixOS exist. It's just not done on the OS level in Debian because in some ways it doesn't really need to be.

Yes, because Ubuntu doesn't ship everything you need, third-party repositories are needed for my day-to-day job. None of the packages I get from that are overwriting existing software from official repositories, but still Ubuntu recommends turning those off, not sure how that should break an upgrade. On the other hand, my Arch installation runs with tons of custom software, with repositories and more added to it, but still somehow can handle upgrades 100 times better than Ubuntu.

Regarding the python, the error message was something like "python is gonna be python 3 after the upgrade, and we can't handle that so you should uninstall python before upgrading", or something similar.

I'm just using what Canonical/Ubuntu are providing me when it comes to Python, but still somehow it's up to me to remove/install stuff manually when it comes to upgrading.

I use Manjaro and different Python versions alongside each other regularly break my updates because all of them want to be the system python3 binary at the same time. I've had to nuke and reinstall tons of software because of version clashes, sometimes needing me to cast arcane package manager spells which every manual tells you to never ever do. Hell, a lot of really common AUR packages or dependencies sometimes just plain fail to compile. That's the danger of the AUR, I guess, but there's no real difference between the AUR and PPAs/Ubuntu repositories. At least Manjaro pins packages for a short while before releasing them onto my laptop, I can't imagine the stress an average Arch update would cause me.

Ubuntu is not a system designed for running multiple versions of packages like you want it to. You can make it do it if you really want to, but you can't expect the Ubuntu developers to provide support for your unique use case. They give you one version of Python, the version that the _entire system_ expects, and if you're not happy with that, you should probably move away from Ubuntu. It can't and probably won't satisfy your day-to-day job's requirements. Try a rolling distro instead, or run the Nix package manager next to the system package manager so you can work around the lack of support Ubuntu provides you.

In my opinion, it's a lot better to abort the upgrade than to knowingly break your system and make you suffer through the recovery process. Sure, the installer could ignore the problem, or remove your carefully crafted Python setup, but you'd still be complaining if it did that and you had to reconfigure your entire machine again.

Isn't the freedom of "hacking" your system one of the main reasons for using Linux to begin with?

I'm not sure it's a main reason, but it is indeed a key property of having full control over your own system that you are welcome to do this. But this level of control also comes with a curse: if you break your system, then you get to keep the pieces. I find it quite upsetting to be blamed instead.

You are misassessing how people use Desktop/Laptop computers. Ie you're assuming people will install Ubuntu, and leave it in its default state, or only use package managers.

I understand how people use their computers. I merely object to those people then blaming the distro for not writing software that can magically work around how they hacked it up.

Yeah, it's not the Distro's fault that they didn't design it for how people actually use their computers. How could they have guessed that would be an issue?

I don't understand how you can expect a distro developer to account for every single possible third party repository and package that is out there. That's multiple orders of magnitude more packages that they would have to support and test for, it's simply not feasible. It's reaching into combinatorial explosion. I would actually have serious concerns if a Linux vendor was somehow promising that they could support such an unrealistic thing.

The real solution was mentioned up thread though: they are designing it to be used that way, but you have to use snaps or flatpaks. Not third party apt repositories which can overwrite system components in an infinite number of ways.

Do programs installed via installer executables/scripts also contribute to this problem?

If you install them to your home directory, no. If they install to system directories (/usr, /lib, /etc, /var) then they could cause issues. I personally would not run any installer script that asks to use sudo or otherwise give it root permissions, that's a recipe for disaster. If you have to do that, run it in a container.

Regrettably, that's most software! When I start a new OS (Windows or Linux), I have a set of software I want to install. Examples: Firefox, Chrome, VsCode, PyCharm, SOlidWorks, KiCad, Em-Client, CubeIDE, Rust, Python (NOT the old system version), Steam etc. Almost all of them operate outside the package manager with their own installers!

I'm not sure what you mean, that isn't even most software out of the examples you gave. I think snap and flatpak have almost all of those. For Rust and Python I think you can use rustup and pyenv. All of that would be preferable to using a root installer.

If the only option they have is an installer that runs as root, then the safest way to handle that would be to run it inside a docker container. Those companies should probably not be shipping installers on Linux unless they target very specific distributions and are working closely with them to ensure they don't break the upgrade process, otherwise it's going to cause issues.

Edit: here are the snaps and flatpaks and docker images I could find.

https://snapcraft.io/firefox

https://snapcraft.io/chromium

https://snapcraft.io/code

https://snapcraft.io/pycharm-professional

https://flathub.org/apps/details/org.mozilla.firefox

https://flathub.org/apps/details/org.chromium.Chromium

https://flathub.org/apps/details/com.visualstudio.code

https://flathub.org/apps/details/com.jetbrains.PyCharm-Profe...

https://flathub.org/apps/details/org.kicad.KiCad

https://flathub.org/apps/details/com.valvesoftware.Steam

https://github.com/xanderhendriks/docker-stm32cubeide

AFAIK, Solidworks and eM Client are not available on Linux, but if they were you would probably see them available in one of those forms.

None of those you linked are recommended on the official websites:

https://www.rust-lang.org/tools/install: Curl an .sh

Steam: https://store.steampowered.com/about/: Download and install a .deb

Firefox: https://www.mozilla.org/en-US/firefox/all/#product-desktop-r... - Download a .tar.bz2

VsCode: https://code.visualstudio.com/download - Download a DEB

Pycharm https://www.jetbrains.com/pycharm/download/download-thanks.h... - download a .tar.gz2

Python - Install from source or use a 3rd party dep.

https://www.kicad.org/download/ubuntu/ - KiCad is the only exception that wants you to use an official package.\

Docker is a pain to use; it's a crutch used by web devs and Dev-ops for complex business stuff.

Right, and the ones that require root to install are steam, vscode and kicad. Kicad is probably in the worst situation because it uses a PPA. Please double check your links before posting, it will save us both time. It's also not really relevant what the official website says, as it's traditional in open source for the packaging to be handled by the distribution, not by the upstream.

You're free to "hack" it as much as you wish, but when you break it(if you don't know what you're doing) you get to keep all the pieces. And the blame.

I guess it depends what you consider as "hacking", exactly. To me "hacking" implies "customizing, but not as the makers had intended".

On my my recent Ubuntu attempt about 2 years ago, a major usability problem was diagnosed as "Not using an LTS release".

See, this is why Linux distributions piss me off. You have an upgrade path that breaks, and when it does the response is "oh, you used your Linux distribution in a way we said you could in our docs? why would you think the upgrade would work after that?!"

Distros just don't care about the user experience. They care about making it look fancy, or shifting around designs, or introducing brand new components that break old functionality. But they don't care if people actually have problems using it.

Every bug is the user's fault, and every reinvention-of-the-wheel (that actually works worse than the previous wheel) is a praiseworthy accomplishment. Regressions are nonexistent, hardware compatibility/stability is a guessing game, solving common problems is left to the user to Google, all the interfaces and apps are inconsistent, there are no interactive guides to design changes, and just getting some basic hardware like a Bluetooth adapter to work is a multi-day adventure. (And I'm a 20+ year Linux veteran.)

I still use Linux because I know that my hardware will mostly work, and that once I get past a month of fixing bugs on a new install, I can basically make it do what I want. But it's so frustrating to use compared to Windows. The only reason I don't use Windows instead is performance and troubleshooting is worse there (for the most part) and security is scarier. But I want to throw both Linux and Windows in the trash. And I won't buy a Mac because the price and incompatibility is absurd.

> "oh, you used your Linux distribution in a way we said you could in our docs?"

Huh? Adding third party repositories has never been a supported path. You're welcome to use them since you own your system, but the future stability of your system is then down to the ability of the third party to not break it. Usually third party apt packagers don't consider future release upgrades.

What do you expect users to do in this case, when apt has old or missing software? Are users expected to limit themselves to the software in the main repositories? Windows and macos don’t have this constraint. Installing applications from the internet never breaks windows update or macos. (Sometimes macos updates break homebrew or kernel extensions, but that’s kind of easier to deal with.)

Users can only realistically do what their third party software providers tell them to do.

The third party software providers, however, owe it to their users to provide reliable and responsible mechanisms for users to consume their stuff. Providing a third party apt repository that breaks the user doing a future release upgrade is not it.

Now we have snaps, AppImage, Flatpak. These mechanisms can be used by third party software providers to provide safe and reliable (for the system) software. Any of them are better than providing third party apt repositories.

> Windows and macos don’t have this constraint. Installing applications from the internet never breaks windows update or macos.

That's because it isn't the norm for third party software to hack up the OS in order to install themselves, rather than use the mechanisms provided. It used to be - remember DLL hell? They got away from that, and our ecosystem needs to do the same.

Just as another data point, to be honest, I've had nothing but similar issues with Ubuntu. It always found new, weird, sometimes downright bizarre ways to break, often in ways I never could figure out how to fix. Mind you, that was back when I didn't even do anything unusual and didn't know how to really dig into the nuts and bolts of linux... I just installed fairly normal apps from the repository with no custom configuration. ...That was all pre-snap.

Ironically, I tried again post-snap thinking that would make things harder to break. The first app I tried to install failed in some bizarre way. I fixed it eventually, but it left a pretty sour taste in my mouth. I've had upgrades pre 20.0x that basically nuked my whole server. I just can't in good faith recommend ubuntu to beginners anymore. Mind you, that's for desktop... The 20.04 server is actually pretty nice for a headless linux box, though usually I just run it in a container from a debian system via lxc.

Conversely, I've never broken my Arch or debian systems beyond repair despite really hacking the ever-loving shit out of them. I think I had one package I had to fix the version on and no other significant issues.

Agreed. Ubuntu aways broke more than my Arch system.

Funny how experiences change between systems. I stopped using Arch because it kept breaking every other day for months. On the other hand I’ve updated Ubuntu every 6 months for the last 8 years and it never broke.

I'm sure you don't mean it, but this reads very much like victim blaming and responsibility reduction. Most likely the assumption of what the distribution should do is different for the user and provider, which is going to lead to a lot of talking past each other and not many issues being fixed.

I've had dist-upgrade work 1 out of ~7 times I tried it. My count may be off, but that one time I had to help it along.

Yes, I was installing stuff into ~ the entire time.

> If you want to run an Ubuntu LTS system then I suggest that you do your hacking inside containers (whether system containers like lxd, or app containers like Docker). Then you won't need to hack up your system itself, and then upgrades will generally work fine.

Configuring additional apt sources isn't hacking anything up. It's utilizing the normal functionality of your operating system's package manager. It's part of the point of using an operating system with uniform and sane tools for managing software.

I've upgraded from 18.04 in the past, but via intermediary LTS releases (i.e. 20.04). Going all in one step is not possible IMO.

Also note that Ubuntu LTS releases only do upgrades after the first point release of a new version. You won't be upgraded to 21.10 until it hits 21.10.1.

[edit] Apparently the holdout until point release .1 only applies to LTS versions:

> Upgrades from one LTS to the next LTS release are only available after the first point release. For example, Ubuntu 18.04 LTS will only upgrade to Ubuntu 20.04 LTS after the 20.04.1 point release. [https://ubuntu.com/server/docs/upgrade-introduction]

I’ve learned that ubuntu upgrades only if you update to the next version as soon as it is released. I have a 14.04 server that I cannot upgrade anymore and it sucks.

Also, NVidia and Cuda drivers mess the upgrade path as well. And ubuntu is the most supported distro.

Yes, Nvidia is special (in a bad way).

They seem especially hell-bent on pushing new versions of CUDA on everyone way before any usable software supports it (i.e. try getting a usable tensorflow or mxnet when your machine auto-upgraded to 11.4.x one day after its release).

You can have multiple CUDA versions installed at once, it’s not exclusive.

> I’ve learned that ubuntu upgrades only if you update to the next version as soon as it is released. I have a 14.04 server that I cannot upgrade anymore and it sucks.

Actually you can upgrade from old releases. You do need to move from LTS to LTS, rather than jumping. So you can upgrade 14.04 to 16.04, then to 18.04, and then to 20.04.

See https://askubuntu.com/questions/91815/how-to-install-softwar... for details.

Even when a "version jumping" upgrade is allowed, I usually do them in order, unless I'm going from LTS to LTS.

Bold is the man that jumps across multiple major versions of any software.

I'll echo the NixOS experience. Upgrades are not even stressful since you can 'rollback' nearly everything.

For my part, I prefer NixOS, too, and it works just fine on a laptop.

However: do-release-upgrade is really conservative about shutting everything down and reversing as much as possible if anything goes wrong. But if you know your way around apt and dpkg and you know how to mount a borked system and chroot it, you don't really need do-release-upgrade. Just manually hack up your apt sources to the next LTS release and use your favorite apt frontend (aptitude is probably best for this) to upgrade and full-upgrade your way through it. Use dpkg with some --force flags if the state of things gets really messed up.

In my experience it's best to simply not attempt in-place major distro version upgrades and approach them instead as a backup/reimage/restore (or using an alternate/new machine)

This has been true for me across all distros, and honestly it's a main reason why I only run linux on servers anymore (easy to create a new VM for upgrades, harder to create a new laptop)

I have been using linux and Ubuntu for years, and not once has a full upgrade gone off without a hitch. I did upgrade to 21.10 , and of course it blew away my display drivers.

If you use btrfs, which Ubuntu defaults to, you can simply take a snapshot of your system before upgrading and roll it back if there's an issue.

That said, I use Ubuntu on many machines and VMs, and I haven't had issues with upgrading them at all.

I don't think Ubuntu defaults to btrfs.

I just looked and you're right.

In my patchy and incomplete experience this is true for Ubuntu but untrue for Debian. As long as I've followed the upgrade instructions I don't think I've ever had a Debian stable version bump fail. Whereas Ubuntu... yeah, backup and restore, because you're going to do it anyway.

Hmm, I have a couple of laptops with Ubuntu on them, they aren't my primary personal (or work) machines, but I generally haven't had trouble upgrading them - I did one today, via do-release-upgrade -d and it worked fine (though the FF snap, while maybe ok in theory, meant that it had lost all my tabs / bookmarks / etc., uninstalling and installing via apt got me the deb version which did pull those all in from my profile).

Interesting. I ran the upgrade over lunch on my laptop and it finished pretty quickly (and I say this as I've been considering moving to Arch anyway).

The only complaint I ran into is that I didn't catch the "new release available" message on the upgrade dialog the first time and had to go check again.

And this is why the year of the Linux desktop will forever be in the future. From the cryptic (to a normal user) error messages, the broken repositories, the (often) primitive UI/UX, I really don't see Linux becoming more mainstream popular anytime soon.

have a backup and skip the do-release-upgrade script, edit /etc/apt/sources.list directly for newer repos, outcomment anything 3rdparty (in sources.list.d/ too) and update+upgrade+dist-upgrade your system into the future, did it this way for years.. it's almost a rolling rhino

Reply on news.ycombinator.com