NTP's Fate Hinges on 'Father Time'
informationweek.com
informationweek.com
"Apple Macintosh computers and servers running OSX use NTP, and Stenn said Apple developers have called him for help on several NTP issues. In the last such incident, he said he delayed a patch to give Apple more time to prepare OS X for it. When they were ready, he applied the patch and asked "whether Apple could send a donation to the Network Time Foundation," Stenn recalled. "They said they would do their best to see that Apple throws some money our way." But it hasn't happened yet."
Surely, he needs to say upfront that there is a consultancy fee. I'm sure most big companies can't make donations easily, what they can do is pay for services and products which they do all day, everyday.
Customer: I'd like new MacBook, I really need to get some work done.
Apple Employee: Here you go, this should solve your problem. (gives him a new MacBook)
Customer: Great
Apple Employee: Could donate to the Apple foundation?
Customer: Sure, I'll ask my partner what they think. (walks out the shop with the MacBook, without paying for anything).
I am, broadly, of the opinion that a non-profit either needs to take up (or form), an "infrastructure consultancy" firm with financial structure and incentives to ensure that projects like LibreSSL, GPG, NTP, etc are funded and maintained; some of that will involve consulting work for large firms for large piles of money.
Anyway, I don't have a lot of swing in that field, but... it's my conclusion. :-)
"With the Linux Foundation's $7,000 in monthly cash flow, Stenn finances his movement between his home lab, in Talent, Ore., and the NTP servers located in San Jose, Calif. In Oregon, Stenn lives with his wife and does most of his patch inspection, code writing, and release building three weeks a month. The fourth week, he stays in San Jose, close to two co-location data center providers that host NTP computers. He rents a room there to work on server and network administration, maintain the email list, and check on server backups.
Much of the travel, room, replacement hardware such as disk drives, or needed commercial software such as the Coverity system for security checking on code, must come out of the $7,000 monthly stipend or be charged to his consulting business.
Most of his 17-20 servers came out of a one-time, $10,000 grant in 2010 from the Internet Society, a policy and technology infrastructure advisory body for the Internet founded in 1991. Those servers are running at ISC.org in Redwood City, Calif., which hosts BIND and several other open source pieces of Internet infrastructure. For 15 years, it has provided space, electricity and some management "smart hands" to host NTP operations, without charging, said Stenn. "They would love for us to pay them," he said, and he once totaled the monthly bill at $1,400. But ISC.org also knows the NTP project can't pay and continues to host it, Stenn added."
And on the first page:
"Its ongoing development and maintenance now rest mostly on the shoulders of Stenn, and that's why NTP faces a turning point. Stenn, who also works sporadically on his own consulting business, has given himself a deadline: Garner more financial support by April, "or look for regular work.”"
Now although he has a donate link on http://www.ntp.org to:
http://networktimefoundation.org/donate/
he doesn't have a clear goal set like GnuPG had. GnuPG doesn't state the goal because it reached it. But until then you were able to actually see how your support moves the bar. Maybe setting the goal would help here?
The overall architectural goals are the same as every other FOSS project claims to follow: Simplicity, Quality, Security etc. etc. but I tend to think that we stick a little bit more closely to them.
This work is sponsored by Linux Foundation, partly in response to the HeartBleed fiasco, and after studying the 300,000+ lines of source-code in NTPD. I concluded that while it could be salvaged, it would be more economical, much faster and far more efficient to start from scratch.
Ntimed is the result.
http://phk.freebsd.dk/time/20140926.html
The problem with all such initiatives ("we start clean") is that they typically never even reach the feature set of the previous software (implementing features takes orders of magnitude more time than some simple "proof of concept") and that they often make the same errors the older software already solved. JWZ wrote nicely somewhere about that effect (something like "it's not 'hard to read code,' those are the actual features implemented") but I can't find the link.
"Back to that two page function. Yes, I know, it's just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I'll tell you why: those are bug fixes. ...
Each of these bugs took weeks of real-world usage before they were found. The programmer might have spent a couple of days reproducing the bug in the lab and fixing it. If it's like a lot of bugs, the fix might be one line of code, or it might even be a couple of characters, but a lot of work and time went into those two characters.
When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work."
Sometimes, had it been written well from the beginning, there wouldn't have been as many bugs in the first place, though.
I've seen codebases that really shouldn't be redone, just maintained and slightly patched over time - the risks are too high, the maintenance is low, patches are at least understood (mostly). I've also seen other codebases that should be scrapped and restarted. It really depends on the skills, commitment and expectations of the parties involved, and there's no one answer that fits all situations.
But if you don't work with talented/skilled/experienced people and there's "weird stuff" in the codebase it might well be for no good reason at all.
You can only invoke the Spolsky "it's bug-fixes!" argument if you're not cleaning up someone's horrific mess.
Is it the boy scouts that promote "leave the campground cleaner than you found it"? That's the attitude I try to bring to projects, but there are limits, both time and effort. I can leave you (client, employer) a much better system (by whatever metrics you want to establish) by rebuilding from scratch when what I'm starting with is a broken, unstable mess. not always, but the idea shouldn't be dismissed out of hand because of something Joel Spolsky wrote about Netscape in 2000.
But every once in a while we'd end up paying double for a shipment because the customer was way out in the sticks or whatever. Had we done more dynamic stuff like taking the whole address into account when quoting a price to a customer we'd never get bit by this kind of problem. But it was only a couple of times a month so I didn't worry too much about it.
My replacement found that this bothered him a lot and he figured he'd score points by fixing this problem. So he did exactly that, transitioned the entire quote system from local database lookups to remote UPS/FedEx/USPS/etc calls. 2-4 rates per shipper (Ground, Air, etc) for a total of about 10-15 every time a customer wanted a quote. And because we would repackage stuff (it was a logistics company) we often never knew the exact weight so we'd quote 3-4 prices so people could get a feel for which choice was their best bet for the best rate without delaying everything by an extra day or two in order to get a hard quote.
We cached these rates by country and weight (up to 1000lbs) so between all the service offerings and whatnot it was about 100,000 pieces of information in our actual, but occasionally incorrect database. So there were two choices:
1. Don't do any caching and just look them all up in realtime for customers. They're web APIs so there's latency associated.
2. Cache, but on a per-address basis. We had an address book for our customers so we knew the couple of addresses they would want to ship to and we could aggressively warm the cache so that all the rates would already be there. But there were about 10k unique addresses in the database * 100k total rates = 1 billion rates that needed to be cached.
When I presented this back-of-the-envelope calculation in a meeting do you know how he blew all of it off?
"Premature optimization is the root of all evil" -- Donald Knuth
I was so flabbergasted that someone could be aggressively ignorant and yet somehow twist Knuth's words to support their own position that I simply gave up. I was dealing with a powerful stupidity and it was stronger than me.
I later heard that during the transition it was touch-and-go for about a week and they had to issue a lot of credit to pissed off customers. The rate quoting page went from about 20ms to render (and maybe 300ms to load) now to about 4 seconds.
If my system requirements are that something run in 100ms, dismissing options that have a min 2s latency (however useful they are) is not "premature optimization". It's a system requirement, and options B, C and D don't make the cut. Simple as that.
http://www.jwz.org/doc/cadt.html
"that's what happens when there is no incentive for people to do the parts of programming that aren't fun. Fixing bugs isn't fun; going through the bug list isn't fun; but rewriting everything from scratch is fun (because "this time it will be done right", ha ha) and so that's what happens, over and over again."
But the longer one, containing more or less the quote I first approximated, I just can't find. If I remeber he wrote about Netscape, the code for FTP and how long it took to get it right in all edge cases, and then it was thrown away.
TDD can go horribly wrong, too -- but if your tests are on high-enough-level functionality, and you maintain them, then they can encompass the lessons you learn from fucking up every time you do it. Writing & Maintaining tests isn't fun, just as fixing bugs isn't fun -- so in spirit these kinds of arguments hold just as much sway: there are fun parts of programming, and there are parts of programming that are significantly less fun, but even if you fork and start from scratch, it is feasible to have a checklist of bugs/features developed in the order that they are developed initially.
Do most open source projects do this, even the more well-maintained? Of course not. However, if people are seriously worried about this phenomenon, that's probably one of the ways developed in the decade since this essay was published to approach it.
"This is, I think, the most common way for my bug reports to open source software projects to ever become closed. I report bugs; they go unread for a year, sometimes two; and then (surprise!) that module is rewritten from scratch -- and the new maintainer can't be bothered to check whether his new version has actually solved any of the known problems that existed in the previous version."
http://www.jwz.org/doc/cadt.html
He got the money to look for the bugs and to fix them (hard). He instead goes to make the fully new undiscoverd bugs in fully new (his own) code (easy).
E.g. perhaps it makes sense to have an entirely separately client-only package with a separate codebase.
NTP needs time sources too, not just clients.
Coverity is free for Open Source projects, why is he paying for it at all?
It's hard to sympathize when there are such frivolous expenses. There's what, a thousand bucks at least on travel and accommodation per month.
Going on-site is understandable when updates need to be performed that are too risky to do remotely. On-site on a fixed schedule, however, would quickly be over-ruled in any commercial operation.
There does not exactly appear to be a queue of people wanting to take over the reigns.
See the other comments here about that.
There are at least 4 simple fixes I can think of:
1) Move him and his family closer to the data center (Perhaps difficult)
2) Move the servers to a data center that's closer to his home (Not as difficult)
3) He could start using VMs like the rest of the world... physical access really isn't needed except for physical security (making sure the rack is locked and access is controlled), which data center employees can manage (Fairly easy)
4) Fire him and find someone else who would like a $7k/month stipend to do bugfixes (Fairly easy)
If you go with a superhero, then you have to deal with him. You can't just say, "Sorry bud, we need to uproot you and move you in-town." Even if he doesn't say no, which he very well could, leaving you and your plans fucked and the world unprotected, driving all the enjoyment out of the job will make him eventually say "fuck it, I'm retiring."
Okay, let's go with an organization. Who is going to build it? You still need the superhero. The critical nature of the infrastructure demands you use the guy whose been covering your ass all this time, because most anybody else you're going to find is going to fuck it up, and there can be no possibility of a fuck-up.
And you can't buy a superhero for any amount of money. There just aren't that many of them.
If there's a vulnerability it affects the entire world
Who is going to build it? You still need the superhero.
Hyperbole much?
If it really was so dire then that would be all the more reason to not rely on a single "superhero" who may get run over by a bus tomorrow.
Gladly NTP is not the mythical voodoo rocket science that you make it out to be. Most large corps run their own NTP servers, some of them public, e.g.:
time1.google.com
time2.google.com
time3.google.com
time4.google.com
The official NTPd impl doesn't even have a very good reputation (cf. the recent debate about that security vulnerability).
As I see it someone like Google should indeed just hire the guy and take pool.ntp.org under their wing. Throwing even more money at a single guy doesn't convince me as a good way to improve the situation here.
PS: And I don't mind him being paid well at all. I'm very much in favor of important OSS projects getting sponsored and rewarded. But if $7k/mo are not enough to maintain a package that others have re-implemented for free (openntpd etc.) then something seems seriously wrong.
Which is why he for a couple of years have tried to get corporate sponsors for a foundation to hire more people to work on it, but even that has been met largely with apathy.
> But if $7k/mo are not enough to maintain a package that others have re-implemented for free (openntpd etc.) then something seems seriously wrong.
If you think that's all he does, then you don't understand what he's doing.
You also seem to have missed that $7k also covers all his costs, including hardware replacements and hosting for parts of the servers.
Doesn't fix the problem. They have to sync to something, someone's gotta maintain the connection to actual atomic clocks. Something has to secure that connection. Someone has to maintain that something. You're severely underestimating the scope of the problem.
> As I see it someone like Google should indeed just hire the guy and take pool.ntp.org under their wing.
They should, but they probably won't, though they might as a result of this article. That would make protecting the world Google's responsibility and it doesn't make good corporate sense to do that. Google would have to find a real reason to throw $X0 million a year at this. The estimate given by Father Time was $4 million a year, that will only make sense if he builds it himself. If Google does it it will cost a lot more.
This is a resource allocation problem, one traditionally solved by governments. Nobody wants the government to do this, so we have to find some innovative way to fund critical technology projects.
As I understand it, they usually sync to GPS?
You can buy those boxes on Amazon[1], starting at around $299.
Google would have to find a real reason to throw $X0 million a year at this
$X0 million?!
To construct and run their own atomic clock, or what would they spend so much money on?
I may indeed be completely missing what this guy does. He runs an actual atomic clock in that rack, like the one from NIST[2]?
[1] http://www.amazon.com/TM1000A-GPS-Network-Time-Server/dp/B00...
I everyone had boxes like this, then yes, great, we'd not need reliable publicly accessible stratum 1 servers. But most people don't.
Um. We have "reliable publicly accessible stratum 1 servers".
Regardless of what happens with the NTPd software, Google will probably continue to provide time1-4.google.com.
NIST will not shut down their timeservers.
And the 3654 servers in pool.ntp.org[1] (which seems to be maintained by a different guy) also won't just disappear overnight, though I'm not sure if these are Stratum 1 (probably not).
I still don't understand what exactly this guy is doing that should cost more than the $7k/mo that he's getting, much less the "$X0 million a year" that vinceguidry wants to allocate to the task.
Neither is DNS, but running the root DNS infrastructure (which serve hundreds of millions of users) is a wee bit more complicated than hosting forward lookup zones or even a public caching resolver on a handful of corporate nameservers. So too with the stratum-1 and stratum-2 NTP servers that serve as national or regional standards. They are not simple bits of kit that one throws together and shoves onto the public Internet, especially when they are ultimately used by just about every modern Internet-connected computer out there. That this guy's doing not only all of the software/release engineering but also operating some critical time servers for USD 84000 per year is too good of a deal to be true---and that he's running through his personal savings to finance the NTP project is just shameful on the IT community's (our) part.
3) He could start using VMs like the rest of the world... physical access really isn't needed except for physical security (making sure the rack is locked and access is controlled), which data center employees can manage (Fairly easy)
For some stuff, you just need your own rack.
This goes to show that the person making the comment is thoroughly unqualified to make any kind of substantive comment on anything that this guy does.
Well, the article does a terrible job at explaining why that rack is needed and why it can't be in a datacenter in the same city as the NTPd maintainer.
Honestly, I have a hard time believing there is new NTP hardware coming out every month that the maintainer then personally has to write drivers for.
And even if that was the case, what prevents that hardware from working when it's located, say, on a desk in the house of the maintainer (I think most timeservers are fanless and don't consume very much energy)?
Perhaps there's valid reasons and explanations for his monthly trips to that datacenter, but the article only left me with questions.
> Well, the article does a terrible job at explaining why that rack is needed and why it can't be in a datacenter in the same city as the NTPd maintainer.
For starters, the majority of the servers are currently hosted for free at isc.org. You'd save a tiny amount on his monthly travel (I don't know what his costs are, but I used to stay in Palo Alto for weeks at a time in my last job, and it should be doable for him to travel to, and stay in, San Jose for a week for <$1k/month), in return for a as much or more in hosting costs unless you can find someone willing to give him free space.
That's assuming you can find a nearby suitable data centre.
> I wonder what prevents that hardware from working from, say, a desk in the house of the maintainer
Nothing. If you're willing to fund sufficient low latency, high capacity, redundant internet connections to his house. The price would almost certainly be far higher than flying him to San Jose once a month to maintain the servers there.
What for?
Does he run public timeservers himself?
High-accuracy/high-precision NTP servers are extraordinarily timing-sensitive, never mind the fact that they must be connected to some kind of external time standard. None of the currently available hypervisors support the kind of precision required by a stratum-1 NTP server. Even low-precision applications---such as Active Directory, which can handle clock skew of up to 5 minutes by default---recommend against running time servers in virtual machines.
To get an idea of the engineering tolerances involved, please refer to "Clock Quality" in the NTP FAQ:
3) We are talking about accurate time here. Using VMs is not so conducive to that.
4) You would need someone as knowledgeable as him, and preferably as passionate about accurate time. And he's also doing a bunch of other things.
You might not call it that, but for someone in devops with that length of experience, it's <50% of market value.
> Especially considering that the project isn't under heavy development.
According to the article, he spends ca. 100 hours a week on the project. It may not be under heavy development, but there's a constant stream of both development work and ops work.
> Also, I have to echo what other comments have said... he seems to be spending it oddly.
That is only relevant if you want to account for part of the $7000 as something other than salary, in which case he's even more woefully underpaid. Which he is anyway, given that he's also spending on some of the hosting and replacement parts out of his own pocket. Even so, his arrangement is not at all unusual at his level - I used to work for someone whose previous job involved flying from California to D.C. once a week for a couple of days. The cost of a travel arrangement like that is peanuts compared to his value. Especially given there's no established office or other admin costs.
> 1) Move him and his family closer to the data center (Perhaps difficult)
Why do you think he'd want to move for a job that's so underpaid, to one of the highest living cost regions in the world? Chances are good this would increase his monthly outgoings by more than he'd save on travel and board.
> 2) Move the servers to a data center that's closer to his home (Not as difficult)
Did you miss the part where isc.org that hosts the bulk of the server farm does not charge for it? If you can find someone with sufficiently low latency connection to major interchange points near him that are willing to host the bulk of the servers for free, I'm sure he'd be ecstatic to hear about it. If not, it'd almost certainly cost more than it'd save. It'd also represent a major risk to key internet infrastructure.
> 3) He could start using VMs like the rest of the world...
For stratum 1 servers? You clearly don't understand what it is he's doing. Stratum 1 servers needs a reliable physical connection to a reliable time source (such as a GPS receiver). He needs dedicated hardware. Try to get data center employees to properly handle non-standard hardware like that. I dare you.
> 4) Fire him and find someone else who would like a $7k/month stipend to do bugfixes (Fairly easy)
Good luck with that. Pretty much anyone prepared to take on that job for $7k/month including covering server costs and hardware repairs, is going to be woefully underqualified. In any case, if the situation doesn't improve, we'll find out soon given that he set an April deadline before he'll look for other things to do.
EDIT: Also note the part where it is unclear whether the $7k grant will even continue after April.
So he is apparently a family man, but he also spends 14 hours a day, 7 days a week on a project that hasn't largely changed in ages? Something doesn't add up.
> Did you miss the part where isc.org that hosts the bulk of the server farm does not charge for it?
It happened once. It could happen again. I refuse to believe there is no other organization willing to offer the same deal.
> For stratum 1 servers? You clearly don't understand what it is he's doing.
Yep, I realized my mistake after writing that, and the replies have made it clear that I was wrong. I still think there are options (remote access cards, for one) but I agree that VMs aren't the way to do it.
> it is unclear whether the $7k grant will even continue after April.
If that's the case, we'll see what happens in a month. If everything comes crashing down (it won't) I'll change my mind, but for now I believe one person being touted as the ONLY man who can do a particular job is incredibly wrong. Even if he were literally the only person on earth with the knowledge, it wouldn't be acceptable for everyone to rely on him... instead we should produce something as a community to replace this system. It's just not an acceptable option to have something this important in the sole hands of someone who could be in a plane crash tomorrow.
I agree.
If the article is a misrepresentation then they should correct that asap, it leaves a rather bad taste in the mouth.
It feels like this guy saw how well the sob-story fundraiser worked for GnuPG[1] and figured he'd give it a shot, too.
That's all good and well, but not while you already have a sponsor as generous as the Linux Foundation seems to be here.
Wouldn't it be considerably cheaper to hire someone at the data-centre to do the reboots or whatever it is he's in situ to do? [What do they call that "remote hands" or something.]
EDIT: The apple watch uses GPS.
I've also read a discussion with PHK somewhere that he didn't like OpenNTPD because it had no auth (sorry, can't find the source again). I guess this complaint has been mitigated recently now that TLS auth is supported in OpenNTPD [3].
/edit: not sure if it's clear but I was comparing OpenNTPD to Ntimed. But the named differences still exist when compared to ntp4.
[1] http://phk.freebsd.dk/_downloads/FOSDEM_2015.pdf page 13 and 14
[2] http://www.bsdnow.tv/episodes/2015_02_11-time_for_a_change at 30:30 and 34:00
[3] http://undeadly.org/cgi?action=article&sid=20150210103656
> I've also read a discussion with PHK somewhere that he didn't like OpenNTPD
> because it had no auth (sorry, can't find the source again). I guess this
> complaint has been mitigated recently now that TLS auth is supported in
> OpenNTPD [3].
Instead, this is just pointing your NTP server at an HTTPS server you trust so it can use the timestamps returned in the TLS packets as another highly trusted time source to make sure spoofed NTP packets can't skew your time. It's clever, that's for sure.
If ntimed was written in Rust though... THAT would've been excellent.
listen on *
Is it that he doesn't want to work for a company?