Hacker News Reader

Comment by geocrasher | Hacker News Reader

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong.

1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue.

2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning.

3) Because "FIX IT NOW OR ELSE" is a thing. Okay, sure. Null the file and then fill it with 7GB. Problem solved, for now. Everybody is happy and now I can work on the Real Problem: Bob won't stop hoarding spam.

That is all.

View on HN

This reminds me of the reserve tank toggle on some motorcycles. When you run out of gas, you switch the toggle and drive directly to a gas station.

always fun when you're barreling down the highway and the engine starts to lean out, prompting you to hurriedly locate and switch the petcock over before the engine stalls completely.

suppose then that you go fill up and forget to set the petcock back to normal. 8ball says: "I see a long walk in your future."

I once put a new fuel pump in a Chevy pickup with two tanks on the side of the road because I was switched to the empty tank. Good times.

IME it doesn't take too many hikes to learn that part of the procedure for turning off the engine is "turn the fuel switch off reserve".

out of years of riding it's only happened to me a couple times.

one time i was eastbound on the bay bridge when my bike started to sputter. i'd just reassembled the tank and had left the screw-style reserve fuel valve open, so there was no reserve fuel to be had. a very kind lady put her blinkers on behind me and followed as i coasted the last few hundred yards toward yerba buena island.

i pushed my bike up the ramp and looked in the tank to assess. it's a dirtbike, so the tank has two distinct "lobes" to accomodate the top tube of the frame. I had a few ounces in the tank but they were not in the lobe with the fuel pickup, so i dumped the bike on its side to get the fuel to slosh over to where i wanted it.

i got back on the highway and, going quite slowly and gently, managed to get to the gas station at west oakland bart, the engine leaning out and sputtering right as i rolled into their lot.

I think that driving on those last few ounces of fuel is a completely different feeling.

Normally you take for granted that the engine works for hours at at time.

When you've come to a stop and found those last few ounces of fuel, it's such a relief that the engine can run again, and you know it won't run for very long, but every minute that it continues running saves you many minutes of walking or pushing. You appreciate every minute that the engine produces that amazing amount of power (compared to your own power when you're pushing a 300+ pound bike)

It's a crazy amount of energy. The 2.5 gallons of gas that that tank holds has more energy than all the food I eat in a month.

Surprised there isn't a mechanism that mechanically switches the petcock over when you put a fuel nozzle up to the port

Most motorcycles are surprisingly manual. This was originally a necessity (like in cars), but remains aesthetically preferable for many riders.

OTOH, Honda Goldwings have stereo systems. They might grow an automatic fuel reserve switcher-backer someday too. :)

Fuel injected motorcycles don’t have reserve (at least, none that I’ve seen.) instead they have low fuel lights or full fuel gauges. I’m guessing it’s because the fuel pumps are in the tank and the fuel injection system needs high pressure.

Fuel injectors require filtered gas because even small particles can clog them, and said filter is more likely to be clogged or even compromised by sucking up the last drops of fuel (and scale and debris) in the tank, so the low-fuel warning is required.

Carb jets can get clogged, too, but are wider since they're not under as much pressure. Also, since they're a wear item they're a lot easier to clean and/or replace.

I think grandparent commenter had it right: it's because the pump is in the tank. There's just no good way to have an external petcock determine where a tank-internal pump gets its fuel from.

Many new bikes come with a lot of rider aids for safety (ABS, TCS) as well as all kinds of electronics (fuel maps), so this is changing. But of course manual transmission won't go away until bike are electric.

I am one of those who likes things old school. My bike still has a carburetor, has no fuel light or tachometer, and I have certainly had some practice reaching down to turn the fuel petcock to reserve while sputtering on the highway. If they didn't intend for me to do that, why did they put it on the left side? :)

> But of course manual transmission won't go away until bike are electric.

See multiple Honda bikes with DCT (dual clutch transmission). This is what I'm planning to get as my first bike.

Goldwings also have a reverse gear. Even more remarkable: I used to have an Aprilia scooter that had a remote release button (on the key fob) for the under-seat storage area. I think I used it once just to see if it works.

Typically there aren't two separate tanks - In one tank there are two tubes at different heights. As the fuel level falls below the height of the "main" tube the engine sputters, then turning the petcock engages the lower down "reserve" tube which is still below the fuel level. It's more of a warning than a true reserve, and most bikes with an actual fuel gauge don't have a reserve.

On bikes like that, there's a reserve-reserve trick sometimes. Sometimes, the tank is an inverted U shape so when the pickup runs dry there's still a little more fuel on the other side of the U. If the bike is light enough, you can stop the bike and lean it way over to pour that last bit over to the pickup side. Might get you another couple miles.

Some newer bikes, like mine, don't have a reserve petcock. They have a low fuel light. No forgetting about the petcock and an obvious warning light instead of sputtering.

Some older bikes, like my '99 Ducati Monster, don't have a petcock. It has a low fuel light that first failed in around 2002, and for which that part that fails (the in-tank float switch) stopped being available in about 2015 or so. No petcock _or_ warning light. (And that trip where the speedo cable fails so I couldn't even use thew trip meter to estimate fuel requirements was a fun one...)

Can you find someone who can adapt a float switch from a different bike? It seems like a very useful thing to have, even if it's not the original factory part.

I've just gotten used to it. I'm fairly reliable about always resetting the trip meter when I fill it up (and always fill it to full). I know it'll get 200km easy, maybe only 180 if I'm having _way_ too much fun. That's always about time I want to stop and stretch my legs anyway. It doesn't bother me enough to "solve the problem".

Most motorcycles with a manual petcock are very manual in nature. Often this is to minimize the number of moving parts that could die on you if you take it into rural areas. An automatic petcock adds more complexity that could cause a malfunction.

The bikes I've had that have had reserve tanks have also been old enough to raise the disconcerting follow-on question, which is: "is the reserve gas also full of sludgey crap that's settled in the tank and hasn't been disturbed really in a year, and am i about to run that through my poor carbs?"

My friend had a truck with a reserve tank, but it was the same size as the main tank, so he would just flip the switch at every fill up to make sure they both got used.

Had this in a 70s F150. A "Main - Aux" switch on the dash, right above the 8-track player. I used to let the main tank sputter out on fumes and then triumphantly shout "Rerouting auxiliary power to engine!" while sliding the switch. Letting them empty out alternately would have been a lot smarter.

My father drove a '95 F-150 for years that had the dual tanks. Shortly after highschool I got in accident that ended up totaling my vehicle and got a couple months I was using his truck (he runs an Auto repair shop from a garage behind the house so he almost always had something available to drive) and I ended up using it to go out on a date with someone I had met at work.

I noticed on the way to pick them up that the truck was running on empty in the main tank but I checked and the aux tank was full. Then I remembered the first time my dad let the tank run down and start sputtering down the road and decided to keep going on the empty tank.

Make it to pick them up and start heading down the highway(where we were it was a good 3-4 miles to the nearest gas station) and then the truck finally started to sputter. I proceed to play along with it pretending to panic for a good 20 seconds and then I turned and saw the look on their face and couldn't help but start laughing. Switched to the aux tank and when the truck started running again I turned and and the look I was getting indicated I was being mentally murdered. Then they punched the crap outta my arm and started laughing and calling me not so nice things.

Ended up being an awesome night out with someone I'd end up being friends with for a long time. It's weird how this kind of random conversation in an unrelated internet post can drag you way back down memory lane.

thank you for the journey! (I must be getting old, loving these)

This is typically used for agricultural/off-road fuel which is not priced with road taxes and as a result much cheaper. Off road fuel is dyed red in the US. If you get caught running dyed diesel on road you will be fined. Thus the switch on the dash, when you leave the highway to drive on your farm you flip over to dyed fuel to save $$.

You don't have farmers filtering the red out and selling it - its also whey a lot of UK farmers love Diesel Landy's

Oh, fascinating! My first vehicle was the family's 3/4-ton Diesel '84 Chevy Pickup from the farm, and I'd forgotten it had an Aux fuel tank! This makes a lot of sense.

It’s not a separate tank (in any of my bikes at least) so it gets disturbed every time you refill the tank?

the two-tube design of the tank on my 1975 honda CB meant that there was about an inch and a half of tank that sat below the primary fuel port. Tank crud (steel tank, theoretically passivated, 40 years old) settles faster than I ran through a tank of gas, so the bottom layer had sediment in it fairly regularly.

I kept spare inline fuel filters in a tool roll just in case after a while.

Motorboat fuel tanks have a reserve as well. It's just a raised area that splits the bottom of the tank into 2 separate concave areas. One of the concave areas contains the end of the fuel line, and the other doesn't. When you run out of gas, you tip the tank up to dump the remaining gas from the other basin into the main one, and then you restart the engine (or keep it from stopping at all if you're quick enough on the draw) and head for the docks.

Old SCUBA tanks didn't have gauges, they had a reserve tank with enough air to get you to the surface. You'd realize you were running low (which I'm sure was terrifying) then hit the switch and slowly surface (you don't want to surface quickly when diving).

Yeah, my dad had a tank like that. I dove with it exactly once - never again, yikes. It was coated inside and out so, despite being a steel tank, it was in excellent shape.

It is a shame that motorcycles have moved away from this model. My last bike had a manual petcock with a reserve setting. It was problematic because I’d forget to turn it from off to on, take off on what’s left in the carburetor bowl, and the engine would start sputtering just down the road. But I also never got stranded.

New bike has a vacuum-actuated fuel valve, no reserve. It does have a fuel gauge but since the tank is not a nice simple rectangle and the angle makes a difference the gauge is basically untrustworthy. So I go by the mileage and hope I don’t get it wrong. How hard would it be for them to add a reserve setting so it could just be between On and Reserve so I could just flip between them as needed?

buying a translucent gas tank has been one of the best investments I've made into my bike.

I bet that gets fun in a hurry in the event of a collision.

It is extremely thick plastic. I wouldn't be surprised if it dislodged from the frame before it burst. In any event, in any collision violent enough to rupture the tank, the rider will have already been thrown a hundred feet away (and be dead...)

In the Honda CBF125 group on Facebook, a fellow Indian shared a photo of his bike. A British guy asked what's the switch, he's never seen one before. Same bike, same country of origin, but only certain markets get the switch and the recessed panel.

The UK version of Indian bikes have deluxe items added like disk breaks.

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue.

If you don't have monitoring, will you even be aware that your disk is filling up?

If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?

An extra failsafe? You can do both. What if your cron/netdata are not forwarding emails for some reason (eg nullmailer gets errors from Mailgun)?

Right, but again, what good does the spacer file do if you're not aware that you're running low on disk space? That is: if your monitoring isn't working, how do you know that you need to quickly make room?

And if your monitoring is working correctly, the spacer file really serves no purpose other than lowering the available disk space.

1. When your DBMS is no longer responding to queries, your boss and your customers replace your monitoring system (unlimited free phone calls 24/7 included ;). Case in point: HN is often a better place to check than Google Cloud status page, for example.

2. Maybe you didn't get it, but "nullmailer not forwarding cron email due to mailgun problems" was a bit too specific to be an example I just made up, wasn't it? Again, the premise "if your monitoring is working correctly" is not a good one to base your reasoning upon. Especially if you have 1 VM (VPS) and not a whole k8s cluster with a devops team with rotational on-call assignments.

Somebody will notify you. If the service is just for yourself, you don't need monitoring at all.

Yes, yes, but they will notify you after your service is down (because that's when they notice), in part thanks to a spacer file that eats up available disk space without being of any use. A monitoring service would notify you before your service is down, users grab pitchforks and start looking for torches.

I understand the benefit to be able to quickly delete some file to be able to run some command that would need space, though I find that highly theoretical. If it's your shell that requires space to start, you won't be able to run the command to remove the spacer, and once you're in the shell, I've never found it hard to clean up space; path autocompletion is the only noticeable victim usually. And at this point, the services are down anyhow, and you likely don't want to restart them before figuring out what the problem was, so I don't see the point of quickly being able to make some room.

It feels like "having two flat tires at the same it is highly unlikely, so I always drive with a flat tire just to make sure I don't get an unforeseen flat tire". It's cute, but I'd look for a new job if anyone in the company suggested that unironically.

This is an additional safety net. It's like doing backups. Of course you should replace your hard drive before the other drive breaks down, but you want to have a backup in case your server burns down.

The reason was, I thought, discussed in the article.

When you actually fill up your disc, many linux commands will simply fail to run, meaning getting out of that state is extremely difficult. Deleting the file means you have room to move files around / run emacs / whatever, to fix the problem.

because sometimes you run things that don't really need monitoring.

I run bunch of websites for pet projects and for friends clubs etc. They don't need monitoring, and even if they go down for couple of hours (or days) doesn't really matter.

I do monitor them, but mostly as an excuse to test various software, that I don't get you use during my day job (pretty sure that bunch of static sites and low use forums don't need elstic cluster, for log storage :) )

And sometims you simply don't have the time to deal with this right now. So you do a quick hack, and do it later.

Because even if you have monitoring, some unforseen issue rapidly eating disk space at 3:00 am may not give you the time to solve it without downtime or degraded performance unless you can immediately remove the bottleneck while you troubleshoot.

Then why not automate the removal of the 8 GB spacer file when the disk gets full? Or in other words, just sound your alarms when there is 8 GB of free disk space.

I actually suggested exactly that in another comment, thoigtnto do it in stages: 4gb with an alarm, the more alarms and the other 4gb if not resolved.

Because if it is a broken process then it will fill up the disk again before you wake up and look at it.

I think the idea is that once you are at the system you can try to find out the cause without removing the file, or worse case remove the file and act fast (you may be on a short timer at this point). So for example if you find out that process X broke and is writing a ton of logs you can disable that process, remove the file, then most of your system is operational while you can properly fix the root cause or at the very least decide how to handle the data that filled up the disk in the first place. (You can't always just delete it without thought)

I think a more refined approach would be disk quotas that ensured that root (or a debugging user) always had a buffer to do the repairs. This file just serves as a system wide disk quota (but you need to remove it to take advantage of that reserved space).

Monitors can fail, you can miss an email, etc etc etc

There's always a big gap between what should never happen because you planned well and what does happen

One argument in favor of it is the 8GB file may cause a runaway process to crash, leaving you without it continuing to chew up space and able to recover.

A second argument is it's not opened by any process. One problem I've had fixing disk full errors was figuring out which process still had a file open.

(For any POSIX noobs: the space occupied by a file is controlled by its inode. Deleting a file "unlinks" the inode from the directory, but an open filehandle counts as a link to that inode. Until all links to the inode are deleted, the OS won't release the space occupied by the file. Particularly with log files, you need to kill any processes that have it open to actually reclaim the disk space.)

lsof

Except that you probably don't realize that a process had it open until after you deleted it.

Besides runaway log files that aren't being properly rotated, human error can cause it too. I managed to completely eat up the disk space of one of our staging servers a few weeks ago trying to tar up a directory so I could work on it locally. Didn't realize the directory was 31GB and we only had 25GB of space. By the time the notification for 80% usage was triggered (no more than 2 minutes after we hit 80%), the entire disk was full. Luckily it was just a staging server and no real harm was done, but such a mistake could have just as easily been made on a production server. In this case, the obvious solution is to just delete the file you were creating but if you're running a more complicated process that is generating logs and many files, it may not be so easy and this 8GB empty file might be useful after you cancel the process.

This is one of those great solutions where they got 90% of the value of the Real Solution(tm) with 5 minutes of work.

I agree with this assessment. Of course its not a solution. Its delaying the inevitable. But depending on the rate of "filling up the disk for unknown reason" it will buy you time.

So when you're running out of space, you immediately delete the junk file. Suddenly there's "No Problem" and you've reset the symptom back to hopefully well before it was an issue. Now you can run whatever you need to, do reports, do traces etc. Even add more storage if necessary.

More importantly, as soon as you delete that junk file now you have space for logs. You have space and time for investigation.

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue.

It doesn't do that though. If you don't have monitoring/alerting that can either a) give you sufficient notice that you're trending out of disk space, b) take action on its own (e.g. defensively shutting down the machine), or c) both of the above, then having your server disks fill up is bad whether you have a ballast file or not.

If your database server goes to 100%, you can't trust your database anymore whether you could ssh in and delete an 8GB file or not.

That's it.

Real Solutions (tm) are indeed nice, but hackers get shit done - this is an utterly shameless hack, and I do it myself.

It's a tool, and should be celebrated as such. It gives you breathing rom to actually solve the problem. It's an early warning system. :)

I find that either a server needs more space, or has files that can be deleted. For the former you just increase the disk space, since most things are VMs these days and increasing space is easy. For the latter you can usually delete enough files to get the service back up before you start the proper cleanup.

If you really need some reserve space (physical server), I'd much rather store it in a vg (or zfs/btrfs subvolume). Will you remember the file exists at 2am? What about the other admins on your team?

> Will you remember the file exists at 2am? What about the other admins on your team?

Hopefully if you were doing something like this it would be part of your standard incident response runsheet/checklist.

> Will you remember the file exists at 2am?

As someone who has been woken up at 2am for this exact issue, emphatically yes. I would much rather be back in bed than trying to Google the command to find large files on disk.

Setup proper monitoring and never get to the Real Issue to begin with. These sysadmin hacks are not helpful

In a perfect world, this is true. But we don't have one of those.

That’s not perfect world this is standard industry practice to have host level monitoring

THANK YOU. How are so many people in this thread content with saying “monitoring isn’t perfect, this solution is ingenious”. Ofc nothing is perfect and even when you do everything right things can still go wrong, but if you don’t have a ROBUST monitoring/alert system in place then you’re not even doing the bare minimum. They’re acting like it’s rocket science to set thresholds, and have meaningful alerts and checks in place. Not to mention if you wait until disk full you risk issues like block corruption among others and your 8GB of space doesn’t do anything. It’s why people in this industry are on call, it’s why they have monitoring on their monitoring systems. The bare minimum

Yeah it’s crazy. If someone does this on their homelab server it’s probably fine but if they run it in production I really want to know because Im not buying jack from them.

Of course! But do you put all your trust in your monitoring, 100%? You've never had monitoring fail for any reason at all? You've never had a server fill up before you can respond to the alert?

This 8gb file idea isn't to replace monitoring. It's to offer a quick stopgap solution so you can do things in a hurry and give yourself a little extra "out" when things go awry. Because believe me, they WILL go awry. And if you're not prepared for that eventuality, then I don't know what else to say.

> But do you put all your trust in your monitoring, 100%?

Yes. If I didn't feel that I can trust it, I would get another solution.

> You've never had a server fill up before you can respond to the alert?

I have. With the proposed hack in this article: it would fill up even faster: by that amount of time it would take the problem to write 8gb of data.

> Because believe me, they WILL go awry.

In my experience: not in any way that this would help. If your disk fills up, it's either slow (and your monitoring alerts you days or at least hours before it's a problem) or it's really, really fast. In the latter case, it's much faster than you can jump on your computer, ssh into the machine and delete your spacer file.

Invest in better monitoring, that's much, much, much, much better than adding spacer files to fill up your disk or changing the wall clock to give you more time.

Ah I see where you are coming from. You see the spacer as a way to prevent a problem that should be prevented by better monitoring. But that's not what it is for. It's for quickly providing a stopgap so that you have time to solve the root cause without enduring more downtime.

If you've had a disk go full on you, what's the first thing you do? For me, I log in and start looking for a log file to truncate to buy me a few megs of space, at least. This spacer file is just a guaranteed way to find the space you need without having to hunt for it.

Also it doesn't HAVE to be 8GB. On most systems I think a 500mb file would be every bit as effective.

"proper monitoring" is extremely broad. And, I would say, almost unreachable goal.

You have it mail you when it goes over 80% disk usage (and what if you are on holiday)? Does it mail all colleagues? Who picks it up (I thought Bob picked it up, but Bob thought Anne picked it up. So no one did)? Does it come and wake you in person when it reaches 92%?

Will this catch this async job that fails (but should never) in an endless loop but keeps creating 20MB json files as fast as the disk allows it to?

Is it an alerting that finds anomalies in trends? Will it be fast enough for you to come online before that job has filled the disk?

I've been doing a lot of hosting management and such. And there is one constant: all unforeseen issues are unforeseen.

> I've been doing a lot of hosting management and such. And there is one constant: all unforeseen issues are unforeseen.

I work in hosting too, and have been for a long time. I feel ya.

Slack warning/ticket at 75%, page at 85% (to oncall obviously). Don’t let user workload crap into your root partition. I’ve been doing this for over 10 years and managed many thousands of nodes and literally don’t recall full disk problem unless it was in staging somewhere where monitoring was deliberately disabled.

Your requirements for "proper monitoring" are not everyones requirements.

On a current gig, we host at heroku. Our monitoring is all about 95th percentile response-times, secondary services, backlogs, slow-queries and whatnots. For another job, "disk space filling up" is important. Again another job will need to monitor email-delivery-rates and so on and so forth.

Keep in mind that sysadmins are essentially babysitting software that they do not develop. The hacks that we come up with are to work around responsible party and help us get a good night's sleep instead of a 2am wakeup call. I try to cut you guys some slack, usually this proliferates when management decides they are willing to accept some inefficiency in favor of getting new features out the door. I get it, really.

My org is in the middle of a SRE introduction and for some reason I'm getting a lot of pushback on the topic of 'error budgets' and what to do with alerts when they are exceeded. Can't imagine why.

How does using proposed solution prevents a 2am wake up call? Your monitoring/alerting does, this just makes it easier to recover already broken software. And btw I’ve been carrying pagers for more than a decade so well aware of all the organizational dynamics here. Best way to prevent this is have devs carrying pager too (amazons “you built it you run it”) - and magically your nighttime oncall is much more pleasant ;-)

Reply on news.ycombinator.com