Hacker News Reader

Comment by Animats | Hacker News Reader

This is why the UNIX approach of using flat text files for dynamic information is obsolete. It seems so simple to have a text file. But then you need an editor, a locking system, access control, and a checker. And probably something to remove orphaned lock files and detect corrupted data.

Those are all database functions. In UNIX/Linux, each configuration file has its own mechanism for all that. They're all inferior to SQLite.

View on HN

Crontab is an excellent example of situations where text files are superior. You are welcome to try to design a database schema that is safer or easier to use.

Most of the things living under /etc are not databases. crontab certainly isn't:

- Only a single "table" (no table-relations)

- No standard database type exists for columns (how would you model the scheduling as database types, how would you model the shell command as database types, if not simply as sanitized strings)

So you can't get integrity for free. Need a custom parser and some integrity checks. Not so difficult (yes, cron will give instant feedback for the unlikely case that you did something wrong)

Re: editor, locking, access control:

I have an editor and prefer it to writing SQL DML any day.

File updates (rename(2)) are atomic, so no need for additional locking. And even if there is some broken tool writing to the old file instead of rename, the chances of that being simultaneous with a parse are microscopic. Don't overdesign. (there are many more things that can go wrong in a system that you can't possibly design out. Live with it).

Access control: Standard file permission modes are just fine. More complex permission model would be a serious overdesign. You can get granularity by using separate crontabs.

Checker: Never needed an additional tool to format the meaning as human-readable text. The syntax is a bit ugly but quite easy to remember and use.

A cron job kicks off that makes the whole system unresponsive, undergoing heavy swap (updatedb used to do this and ran every night on many default systems).

Someone edits a cron file in the most common editor, vim. On many systems it doesn't do an atomic swap on save without extra configuration: http://stackoverflow.com/questions/13563627/how-does-vim-sav...

Now maybe your unresponsive, undergoing swap, system misses a cron job that was present in both edits. Opps.

I'm not sure this could really play out, but the approach definitely sounds firmly on the "worse is better" side of the scale.

crontab -e is the way to edit your cron. A similar command exists for editing the sudoers file which is visudo. This will ensure your configuration is a valid syntax as well as do an atomic swap of the file.

Otherwise adding a file or deleting a file in /etc/cron.d/ is the way to go for automatic(ansbile, puppet etc) and package installations.

I'm surprised that people would edit cron in any other way, crontab -e is one of the first commands most intro to Linux manuals teaches.

However crontab -r has to be one of the most poorly thought out commands in UNIX. It removes the cron file and since r is only one character over from e on qwerty keyboards it's a painful and surprising typo.

first thing when logging into new server: place /etc under git. then loss of cron files (or any config file) is easily recovered from.

Except that crontabs are usually somewhere in /var. :)

Which is why I only use system-wide crontabs, in /etc/cron.d

Instead of per-user crontabs. Much easier to manage.

With cron.d you can specify what user each cron entry should run as, so it is equivalent.

This is a brilliant idea - thank you!

I cannot count the number of times this has helped or saved me. You can keep the /etc git repo local. Or if its a critical server, push to github private repo. Seeing a history of your /etc edits is a very reassuring feeling.

what if you are managing the crons externally, eg puppet, crontab -e is not always option

1. If you don't have a test process for changes made with puppet, this is hardly a problem with cron.

2. Puppet handled that parsing for you and will throw errors if your syntax is wrong

I can't tell you the amount of times I've hit -r by mistake and regretted it..

I can. 0. There, more anecdata.

Honestly though, I never use `-e` either. I keep a ~/.crontab and edit that, and then `crontab ~/.crontab`.

Voila, backup/time machine/git friendly crontab.

I got bit by this once, and had to rescue the crontab out off the disk with GNU strings :(

Have a cronjob that backs up your crontab?

Use /bin/cat as editor. That should work nicely from Puppet.

And there's "vipw" for the password file. Each of them with their own separate updating bugs.

"Worse is better" is an oxymoron. You can also read it as "better is worse". In "worse is better", the "worse" isn't really worse in practical settings (considering all tradeoffs, giving appropriate weights to design goals, etc.).

Nobody argues that there can't be problems. What all critics miss is to show how to avoid these problems.

What can you do about jobs that make the system unresponsive? Not a problem for cron to solve.

If someone writes a crontab (instead of rename()ing another version over it), that's not cron's fault. Sqlite databases can be just as easily corrupted or lose integrity. Now which format would you rather repair?

Missing a cron job can not be totally avoided, and it isn't the end of the world. What if your machine goes down or cron crashes? The answer is that the jobs themselves have to be provided with some resilience against that situation.

This isn't a case of "worse is better" since there weren't proposed solutions to the "problems" at all, however unrealistic.

This is a perfect example for why you ought to monitor all your cronjobs with something like https://wdt.io.

Not all databases follow a relational model with tables or schemas. I guess what he meant is that a key-value store would have some merits over files. Kinda like Window's registry.

He explicitly mentioned sqlite. Key-value stores could not improve on that since RDBMS generalize key-value.

Bad example. Windows registry is well know as a source of problems.

This comment doesn't make sense to me at all.

Editor: of course you need some way to input data, and an editor is the simplest interface. You need to generate SQL to store data in SQLite and that's certainly not simpler.

Locking system: the OS/filesystem can provide this just as well as a database can.

Access control: same answer

Checker: input needs to be validated for its intended task and SQLite won't magically make this requirement disappear.

Who edits crontab by opening in the editor directly? Using `crontab -e` is what the manual says to do (cron, fcron, dcron)

Or did we not read the manual, then complain about broken stuff and follow that with an over-enginered "solution" to a problem that didn't exist.

You probably wouldn't want to use text files for actual database information, but they're perfect for code and configuration.

The syntax of cron isn't the most intuitive but it's powerful and would make even less sense in a database.

That's the point though, if it's in a database you can have any number of awesome tools, text or GUI, to manipulate the state. And you don't need to reinvent the wheel to deal with atomic updates to the state, etc.

There is no need for a database. File updates are atomical. Editors rewrite the whole file.

Also the tools argument just isn't true. A well designed plain text format is easier to parse with a one-off parser than with ugly one-off SQL queries. The reason that not many GUIs exist for plain text formats is that they are already designed to be easy to use. There are few cases where GUIs make sense.

From the top of my head, the only configuration GUI I regularly use is firefox's. Not that it wouldn't be possible to come up with a clean plain text design that would be easier to use.

Also most of the stuff is not represantable as relations. I'm looking forward to see how you encode a shell command in relations.

There's more to atomic updates than "rewrite the whole file", note. The way that text editors usually rewrite whole files in place, preserving hard links, is most definitely not an atomic update.

The typical way to do it is to write a new file. Not that it would matter in practice for configuration files.

So what happens when your 'atomical' text update tool reads in the text file to memory, makes some changes, then the file changes on disk from another unrelated process, and your tool writes the file back out losing the previous changes?

Write a bash script to manage your cron and you can do the same thing in a terminal. Use http://freecode.com/projects/zenity or something similar for your new awesome GUI tool. I think for most people cron is simple enough to just google when they need the syntax. Although this crontab.guru site is 1000 time nicer than most google searches for help.

I don't think a database would offer anything unless it was managing 1000's of cron servers. Flat text is fine and easier to parse than querying a database and adding a dependency. A database is a huge requirement to ask for to manage a few simple files.

Can someone explain to me how crontabs, and cron syntax, are still so popular and one of very few options available?

It's not like crontabs are written all the time. They don't need to be so terse. They shouldn't require learning a language... they shouldn't require a cron.guru website. Systemd timer syntax (https://wiki.archlinux.org/index.php/Systemd/Timers#Example) is far more readable, and there's still a lot of room for improvement.

Sure. Spend a focused 10 minutes to really try and understand them and you'll realize that a cron line is a perfectly concise and flexible syntax for creating scheduled time events. It even handles that crazy case where you want to run on a specific day of the week. Cron lines may seem arcane at first, but then again have you ever tried to write code that was timezone aware? Welcome to a special kind of hell. What I'm trying to say is there's complexity there and you'll be a better programmer if you embrace it.

Unix programmers aren't in the habit of throwing out a working system because it's hard to grok, nor building something more complex then it needs to be. I took a look at your systemd timers link and it is a more complex syntax to create the same effect. From their own caveats section:

> Complexity: to set up a timed job with systemd you create two files and run a couple systemctl commands. Compare that to adding a single line to a crontab.

> Spend a focused 10 minutes to really try and understand them

I've spent the past years doing that. I do understand cron. I also strongly dislike it. You could build a thriving city within the bounds of the room there is for improvement.

> It even handles that crazy case where you want to run on a specific day of the week.

So does systemd's cron. Supporting "crazy cases" doesn't mean the syntax has to suck.

> Cron lines may seem arcane at first, but then again have you ever tried to write code that was timezone aware?

Yes...

Re systemd: It's one example; it's not perfect, I wouldn't even call it good, but it's leagues ahead of cron in terms of UX already. Bonus: usage of the ini format means it's programmatically much easier to interact with the cronjob itself.

It's ugly, simple and flexible. I can live with its ugliness (I don't change my crontab every day, after all) and simplicity and flexibility beat ini format/UX/etc. any time AFAIC.

My main problem with the cron syntax is that is error prone. I'd like it to be a bit harder to get correct syntax when you write something wrong.

But a simple GUI could take care of that.

Something curses based would be nice - no need for a GUI ;). Cron's syntax can be hellish. For example, % is treated as a special newline character (ugh) so that commands like this don't work - and it's very difficult to troubleshoot:

  appstartup.sh | tee -a app.`date +%Y%m%d`.log

Agreed. I've been a sysadmin for about ten years now and absolutely loathe them. At the surface, they're simple to use, but it very, very quickly becomes a headache as the server count increases. That, and just about everyone who uses cron forgets to clean up their old messes or adds random scripts here and there, which makes a machine's crontab very, very messy. It makes finding a rogue cron job very, very difficult to find sometimes.

> concise

Why is having the format be so concise and terse a virtue?

They could keep the flat file approach and everything about the system identical but just make it more explicit. Cron usage would become much simpler.

> It even handles that crazy case where you want to run on a specific day of the week.

Does it handle the, in my opinion, not crazy case of running something every, say, 100 seconds?

Sadly, no. The minimum granularity is in minutes.

What about every 100 minutes?

If you're really stuck with this requirement, and vixie cron is your only tool:

    */1 * * * * /usr/bin/bash -c '[[ $(( $(/usr/bin/date +%s) % 6000)) -eq 0 ]] && mycommand'

It's not designed for frequencies that don't align well with the partition of time in months, weeks, days, hours, minutes.

But you probably can tweak your requirement slightly.

With fcron the line looks like

    @100 /do/my/thing.sh

Is fcron stateful so that it remembers if the schedule started today at midnight, the first time it should run tomorrow is at 1am?

As far as I can tell and if IIRC its across reboots too. I can't remember all the reasons I moved to fcron (10+ years ago) but the @# syntax was one factor

> the @# syntax

Watch your language!

  */100 * * * * echo "every 100 mins"

Have you tried this? Since minute only counts from 0-59, it's never divisible by 100.

I have a similar problem with "every 2 days". */2 looks simple enough, but it fires on the 29th, miss 30th, fires on 31st, then fires the 1st. 3 times in 4 days isn't "every 2 days".

Cron only works for periods which are evenly divisible into the calendar. "every 7 minutes" doesn't work because 60 isn't evenly divisible by 7. "every 5 hours" doesn't work (in that it'll match 8pm and midnight, which are only 4 hours apart), etc.

You're right; thank you for pointing out my mistake. Every day's a school day.

Well in cron "every two days" is more like "every second day of a month", and "ever 5 hours" like "every fifth hour of a day".

I guess that's what I was getting at. "every 100 minutes" doesn't work, because cron reads it as "every 100th minute of an hour".

0 is divisible is by 100.

Because it's not nearly as bad as people describe, it's extremely powerful, it's quite clever, and there's really not even a language to learn -- it's just regex/grep for timestamps.

Cron ticks every minute and checks to see if any crontab entries match the current time, if so it runs the job. But standard regular expressions aren't well suited to the types of patterns people normally try to express which is why they use their own syntax.

Alternatively, there is the iCalendar recurrence rule syntax [0]. It shows how complex recurrency rules can get.

Cron could be considered as a good mix between complexity and brevity (readability).

Shameless plug: I tried to create a format in-between both in a JS project [1]. It is based on the ISO 8601 duration format [2], but it's better to stick to iCalendar.

[0] https://tools.ietf.org/html/rfc2445#section-4.3.10

[1] https://github.com/smhg/date-frequency-js

[2] https://en.wikipedia.org/wiki/ISO_8601#Durations

The crontab syntax is popular because it has been standardized by POSIX:

http://pubs.opengroup.org/onlinepubs/7908799/xcu/crontab.htm...

In a sense, crontabs are the lowest common denominator for scheduling tasks on POSIX-compliant systems.

Very few? There's a whole range of options available, from Paul Jarc's runwhen to Bruce Guenter's bcron.

* http://code.dogmap.org/runwhen/

* http://untroubled.org/bcron/

That's funny, because IMHO they both suck and require a certain time to master them. Systemd timers are just a bit different syntax, but as I first opened the linked cron.guru site, I thought about how cool it would be if it displayed the same for systemd timers as well.

Introducing a new, simpler design only simplifies things if you kill the old design. Otherwise everyone needs to know both and their lives aren't any simpler.

Killing off old designs is exceptionally difficult for Unix. Hell, Microsoft can barely force an upgrade on everyone and they don't have a hundred forked distros.

No, it is NOT a problem of using flat files. It is a problem of using files. The files are the serialized initial values for the run-time configuration, which may or may not be changed. SQLite is no superior to flat files in that, and in fact, can be damaging if the user managed to break the invariant and the invariant is not checked.

The configuration system absolutely demands the locking system (to alter it) and the checker (to correctly check if it is correct before the changes are ever committed). SQLite can provide some of them, but not all---most importantly, SQLite's locking system is independent of the runtime configuration which requires separate locking system (however rudimentary).

> It is a problem of using files.

In UNIX, everything is a file

Oh, if only. Network interface is not a file. Process is not a file (though some of its metadata is accessible through different files under different unices). IPv4 routing entry is not a file. User account is not a file.

Do you see a pattern?

Network interface: /sys/class/net/<name> Process: /proc/<id>/ IPv4 Route: /proc/net/route User account: /etc/passwd

OK, configure the network interface this way. Oh, you can't. So it is not a file, it just has some metadata exposed with a file interface.

Create a process in the same manner. Oh, again you can't do that. What a shame.

And with route and user account examples you confuse being a file and being a record in a file (which, couriously, is not a file itself).

You mean /etc/network/interfaces? I can set tx queue length of eth0 to 300 packets: echo 300 > /sys/class/net/eth0/tx_queue_len Or change frame forwarding behavior from switch to hub: cat 0x65535 > /sys/class/net/vnet0_11/bridge/group_fwd_mask

Granted, it's not possible to change IPv6 address through /proc/net/if_inet6, but just because I don't know about it, doesn't mean that it isn't possible.

> You mean /etc/network/interfaces?

No. It's a configuration file for a separate set of tools that actually configure the interface for you. You can't set its current IP address by writing to any file.

> I can set tx queue length of eth0 to 300 packets: echo 300 > /sys/class/net/eth0/tx_queue_len Or change frame forwarding behavior from switch to hub: cat 0x65535 > /sys/class/net/vnet0_11/bridge/group_fwd_mask

Sorry, no banana; you only can control two or three aspects this way, and not the most fundamental ones at that.

Network interface is simply not a file, it's a separate OS-level construct, it's not exposed to userland as a file. Not everything in Linux or unix is a file, even though many things (the ones more remote from kernel) are.

If I were not clear enough, it is a problem of treating configuration files as like the runtime configuration. Not a problem of file formats (though some formats ease the job)---does this make sense?

Let's have a "registry".

Yes, KDE and Gnome both have registries.

Just because Windows' registry is hard to manually edit doesn't mean they are a bad idea. Look at how much better Windows' GUI configuration tools are than anything on Linux. They all use the registry.

Registry has some serious practical problems. How do I easily list, or copy to another machine, the settings for application X? For technical users these are the most important operations. I know how to view or copy a text file...

An associated problem is that the registry accretes trash that is never weeded out (and it's hard to weed it out).

Also at least in former times it was slow (partly because of the trash thing, probably partly because of the implementation, partly because of the centralised design).

Right-click that program's config hive, export. Move file to new system, merge. I did this very frequently before PowerShell got amazing support for the registry (seriously, it's the best. Tab completion of registry values in get-childitem if you supply a registry path).

> ...accretes trash

If your app doesn't remove it's old settings when uninstalled how is that a problem for the registry to solve? It happens with /etc also.

> hard to weed it out

Search for the hives if you don't know where they are. Right-click the hives in question, delete.

Registry usability got leaps and bounds better in Windows 7, and again in Windows 10. Most registry problems you hear about are holdovers from the bad old days (XP and older).

>How do I easily list, or copy to another machine, the settings for application X?

with reg query, reg export and reg import commands, respectively. Or with a few clicks in the GUI, if you prefer.

> Just because Windows' registry is hard to manually edit doesn't mean they are a bad idea.

That's not the problem with Windows registry.

>> Just because Windows' registry is hard to manually edit doesn't mean they are a bad idea.

> That's not the problem with Windows registry.

What is?

https://blog.codinghorror.com/was-the-windows-registry-a-goo...

Your link explicitly says you're wrong, though. Read it. It says:

> My life would be a heck of a lot easier if per-application settings were stored in a place I could easily see them, manipulate them, and back them up. Like, say... in INI files.

> The registry is opaque and binary. As much as I dislike the angle bracket tax, at least XML config files are reasonably human-readable, and they allow as many comments as you see fit.

Now look at the comment you replied to:

> Just because Windows' registry is hard to manually edit doesn't mean they are a bad idea.

You said that's not the problem with the registry, but your own link says that's exactly the problem with the registry...

The filesystem is a type of database. It isn't automatically better or worse than sqlite.

> They're all inferior to SQLite

Which itself has all sorts of complications around locking. Can't write when another process has read-only lock, for example:

https://www.sqlite.org/lockingv3.html

Yes, text files are useless. This "programming" thing will never take off. It would require an editor and code checking. Totally inferior to SQLite.

Makes sense.

Hourly.d

Reply on news.ycombinator.com