Excessive load on NTP servers
news.ntppool.org
news.ntppool.org
> Confirmed - starting up the iOS Snapchat app does a lookup to the domains you listed, and then sends NTP to every unique IP. Around 35-60 different IPs.
Hmm. Is that a fraud prevention thing or something? No way on earth a user app should be getting its own time
See my other post here, and the problems in the (third party) iOS NTP library "features" and its use.
There "createAssociations" in that library without any parameters contacts all the IPs behind the big domain list of the NTP servers!
And according to the forum every IP is contacted -- behind one server name there are 3-4 servers in the DNS in this case, I get 31 server! Real "distributed denial of service" attack.
Ironically, part of an HTTPS handshake involves sharing the server time in a cryptographically-verifiable manner. I am not sure why they don't use that! https://github.com/ioerror/tlsdate
This practice is becoming increasingly common for "time-sensitive applications":
https://github.com/lyft/Kronos/blob/master/Sources/NTPClient...
The third-party library used by Snapchat didn't have any "maximum of servers" (using 30 at once(!)) and defaulted to many in the ntp.org pool, across all the continents!
http://mailman.nanog.org/pipermail/nanog/2016-December/08962...
If you want to prevent users from altering their time use your server and do a time compare with your server.
NTP can be easily intercepted and altered so it would make a lot more sense to do this via a encrypted certificate pinned communication path increasing my work load drastically to alter the time.
I snapchat going to pay for the DDOS they created?
It doesn't say anything about synchronisation between phone and server.
EDIT: In fact it is easier to implement it this way than using NTP. I've implemented something similar and I found it easier to add an API endpoint that returns time() than to ship an NTP client...
And you don't care whether the time on the server is running accurately if you're just using it to generate tokens which are checked against the time on the server. It just needs to be consistent, it could be an arbitrary counter and would still work fine as long as it counted up reliably at consistent intervals.
The library was only doing a couple request per NTP server, so rate limiting really wouldn't have helped.
Also, NTP runs on UDP rather than TCP, and it only requires one packet from each party to exchange time, so it's harder to rate-limit it without making your server unreliable.
NTP has KOD (Kiss of Death) which was meant for that situation, but it is often ignored/not handled properly by applications :/
And of course, you don't get the page that states why when the website is served via https. Not that I need to see the page to know it was either blocked for "hacking" or "entertainment", and I'm guessing it's not entertainment.
Edit: This probably explains why our clocks have been off by 45 minutes since Monday. I guess it will be entertaining to see how long it takes for IT to figure this one out.
Why not just tell them. What have you got to lose? Hell, blame your charitable spirit on the holiday season if you must.
Edit: I realize "got told off" didn't really capture what happened. I came in early one day and noticed we were having a dns issue. I manually refreshed my DNS cache and it started to work. I sent him an email to let him know that the DNS cache was expired. He told me I was out of line and complained to HR.
I had to go meet with HR, which was pointless since they think he is on a power trip as well. Anyways he added a line to the IT policy that specifically prohibits "performing a diagnosis on the network or any of IT managed systems."
He told me I was wrong. So, I wrote a .bat with a netsend command and emailed it to all staff. Multiple staff clicked on the attachment.
Once they figured out it was me, they made me start a computer club with the IT manager as the supervisor of the club. First order of business was locking down .bat execution.
That seems like a great way to handle this situation. Some ignorant other schools might have kicked you to the curb because you were spreading "malware".
They also were way cooler than they had to be about the several times we took down the network or broke the porn filters, or the time we port-scanned a district tech's machine, or had a whole collection of malware on the network drive, or....
I just worry about how students like me would fare in schools these days.
Two hours later, I faced an angry teacher (who was also a math teacher and the lead teacher for our class). She said something about the next person after me freaking out that "the computer has viruses". Got a bad note for behaviour, spot-check of math homework leading to three F grades, and she also tried to take away my notebook with notes about Windows Registry -.-
Sincerity helps, both as a shield and as a way to diffuse the inevitable frustration that comes from troubleshooting tech.
Of course, I've got a strong force of will, too, so I tend to mix poorly with charlatains like the grandfather's post. I really feel for the souls who have to work under that jerk.
I'm definitely a friendly PITA. Most of my coworkers get it, though some definitely think that I'm just creating more work for myself. It's really only the IT director who is so defensive, and unfortunately he has the power to back it up. HR goes out of their way to find answers to questions, and my immediate coworkers are really driven.
A coworker sent me an email because some data I was in charge of adding to the system was missing. I looked, and somehow data in our database had gone missing. I use the data to add information to another database, and that database had the information in it. So somehow he managed to lose information in SQL Server.
A few weeks later, my boss brought me in to his office because about 25k rows had incorrect information. I went to check my notes, and all of my notes from a period of 10 days were missing. I had been creating a changed file log (because files have been lost in the past), and I could see I had created notes during those dates, but they didn't exist anywhere.
The only evidence I had done my job at all during that time were the emails I sent to other people. (Thankfully Microsoft hosts these, so the IT director can't mess it up.) I had sent somebody an email that basically said "I found error [x], but I fixed it."
For a little while I was questioning whether I was insane. I mean, I would never have believed it was possible for data that you've added to an ACID database to just disappear. If you can't trust ACID principles, what can you trust?
I got into trouble at university when I was running a CAD session on X (Cadence VLSI design FTW, not). Some asshat had telnetted in and was trying to brute force root because it was a faster machine than the crappy sun4's dotted around. It was spewing all over the framebuffer. So I logged into another box and sent him an email saying pack it in and that I was trying to work and that I'd report him for AUP violation to the Sysops.
He complained (?!!!) and the next thing I was in front of a tutor getting a bollocking for it. No explanation was allowed to be returned or appealed, permanent black mark on my record.
And that's when I learned about university politics, gave them the fat middle finger and got a job and left a massive 11 page long diatribe about the charlatans at the place.
I basically went through all my tickets and emails, took IT directors claims, read the MSDN articles on the topic, and pointed out all of the places where what he said was not only wrong, but grossly wrong. Things like "sometimes databases lose data." That's funny, cause I'm sure the team that built SQL Server 2012 would have something to say about that. Why don't we look at the documentation on ACID principles.
I've come to expect the IT director to be a moron at this point, and I had been trying to roll with the punches. However, my work has been going missing, and I got in trouble for it. He denied losing it, then he blamed the database for his incompetence. I was so angry I was awake for 3 days straight cross referencing everything.
The lost data wouldn't have even been a problem if he had just told me within a day or two. I only find out it's missing when we try finding it weeks or months later. Then I have to waste my time doing a forensic investigation in to the scope of the problem.
The most annoying this is that the last person in this role never had any of these issues because nobody ever audited our data. I have managed to instill the idea that data can be 100% correct, and we should always be checking our data to make sure it's right. So now I get blamed anytime information is missing or incongruous.
I know that otherwise well-reasoned emails seem like a rant once they reach a certain length. I kept the body of the email to 500 words, and included a pdf of supporting documentation. I let the email sit for a week before I sent it, and had another manager read it as well to make sure the tone was alright.
My entire argument was contained in the body of the email. I knew that was all it would take to get my boss on my side.
The PDF was aimed at HR. The IT director has a lot of power under the IT policy, and he has used it in a retaliatory manner in the past. My goal was to stop the IT director from retaliating so I could do my job while I look for a new one.
I write my emails as if they could be leaked to the general public at any time, and I'm certain that the email wouldn't reflect poorly on me, even out of context. Maybe there was a better course of action, but I don't think I did much to hurt myself.
The important people often see those things, assume it's an incoherent screed from a disgruntled piece of crap, and delete it without reading (I know from personal experience; such completely accurate and valid diatribes have gotten me fired on the 3rd day of my 2-week notice and, on a separate occasion, a running joke among the big shots for months after my departure, where one would specifically talk about how the email came in while he was on the toilet, at which point he cackled and promptly deleted it without reading).
To anyone non-technical, that letter is all mumbo jumbo. It seriously might as well be in a foreign language. They are not going to check a few of the cross-references and see that you're obviously right. Even though you might hope they'll do this just one or two times, they won't.
They are not going to ask the people called out in your letter to account for your accusations.
They are going to write you off as an unhinged, angry, and worthless nothing-tier employee/student/whatever, make fun of you for a long time, and then forget all about it.
Humans base their decisions on personal trust/credibility. The way to win against an evil IT director is to obtain far more trust/credibility in the eyes of his bosses than he has, and then to use that credibility to your advantage.
That's a lengthy and difficult process, especially when you start out as a subordinate and the boss has a lot more access and ability to frame your efforts to his advantage.
I personally have never had the patience to undertake such political subterfuge seriously and I find such undertakings both incredibly frustrating and soul-crushingly phony and hypocritical, but I am now convinced it is the only reliable way to get real career success and mobility. Thus I accept that any career success I enjoy will be lucky/accidental.
Employment and career IS a popularity contest. Good software is somewhere between the 10th and 20th most important career concern for a developer. The number one concern for anyone trying to make it in white-collar America is to be as well-liked and popular as possible. Most of the time, love of colleagues and love of bosses are symbiotic and they feed off of each other (as long as you're sycophantic efforts aren't TOO obvious), but to the extent that a situation arises wherein someone's love has to be preferred, prefer to get the love of the bosses.
This is the sure path to career success. Disregard truth, objectivity, and practicality. No one cares about you or what you think, they don't care about what you judge to be practical or wise. They didn't really hire you for your experience or insight even though they want to pretend they did. They hired you because they thought you would make them feel and look good.
Not only bosses, but people in general, care only that your presence and actions are generating pleasant feelings for them. Do this reliably, disregard everything that is not this, put only as much energy into the tasks required by your actual job description as you must to be passable in the unlikely event of a performance audit, and pour the rest of your energy into social development. If you're going to make it as a company man, that's the only reliable path.
I think the thing is that there are degrees here. A lot of the things that we are socialized to consider "bad taste" are not actually immoral and unfair (for example, aggressive SEO), and you have to meet on that playing field if you ever intend to be competitive. But you have to identify the limits of what's just getting into the nitty gritty of business and what's crossing the line into being a cheater/liar/phony.
There also seems to be some people for whom social camouflage comes naturally. These people don't feel like frauds or phonies when they cater their preferences and likes to match those of the group around them. It's hard to compete with these people because they have no compunction about being yes-men and they have no malfeasance behind it, since they don't really even realize they're doing -- they only realize that they're making the people around them happy. They truly just have nothing original or important to contribute and don't realize that they're mimicking everyone around them, which, as we've discussed above, is a great skill to have on the path to career success.
I think that developing an active, interesting conversational style and taking care to frame arguments in the most emotionally influential way possible without altering, distorting, or seriously misrepresenting them is perfectly in bounds. Most people do this intuitively to some extent or another, and I believe that intuition can be improved with practice and training, and that that's a great thing for anyone to develop.
The problem is that that's about where I stop. Even if there is nothing immoral about dragging yourself to a baseball game with the bosses, it still feels painful and phony to do that kind of thing. Your unscrupulous and/or unaware competitors, however, will waste no time immediately making themselves as likable as possible by adopting all of the boss's favorite things and habits, refusing to criticize him or anyone whom he holds in high esteem no matter how grossly dangerous their actions are, etc.
It's a tough game. Sometimes I try to believe that there's some cove of people that aren't this way out there, but it always gets shot down as soon as I start believing it again.
I'm starting to think that the only bonds that allow real honest participation are permanent and non-revocable bonds like parent-child and sibling relationships. People still get offended in these and rarely they may even fully withdraw, but most of the time everyone accepts each other and has to get over the perceived slights. I'm not sure there are many voluntary relationships (certainly not relationships where the continued relation is predicated upon a regular payment) where people do that.
I have the advantage to be able to leave a job and find another relatively easily, and I would definitely agree that I'd rather leave then suck up, but for those who don't have that kind of mobility, sucking up and playing politics can literally make the difference in making car / house payments, sending your kids to college, etc.
We're really spoiled in tech because the field is so abundant. We offend someone or get offended, and we're off to greener pastures within weeks if not sooner. I've come to believe that hopping around like this, which I've done for most of my career up to this point, is not healthy, but the availability means that technical people don't have to learn to conform as well as everyone else to survive. And while that means we may be able to keep a job, it's hard to move up.
These things that we struggle with are just normal life to a lot of people. They had to swallow these compromises early in life when it was apparent that good feelings were all they would be able to offer.
This contributes to the cycle because those people pay their dues to the establishment, go through the process, and get used to the circlejerk. They then expect everyone else to do the same.
When someone wants to come in and challenge some of their thoughts, opinions, or practices, even minor things, in what the challenger feels is a sporting way or a way to drive an interesting and inoffensive discussion, the "good feelings violation" siren fires off in the non-technical person's head.
This brings in a large flood of negative feelings and resentment, including but not limited to jealousy that you can express your thoughts openly while they've always had to kowtow, a sense that you're entitled for thinking you should be able to do this and "dictate from your expertise" (as was expressed about me once) instead of "climbing the ranks" the hard way and then quietly and subtly implementing your opinion after you've won the social clout like everyone else has to do, and a sense that you may represent a threat to the perceived competency of the challenged person (and those least competent will be the most aggressive protectors of this perception) in the minds of the people whose trust they live off, which is really everyone -- colleagues, subordinates, and superiors -- which means it's very difficult to overtly question or discuss anything done by anyone, even in what you believe is a polite or considerate way.
Exceptions are basically not made to this. The potential of substantial data loss, massive security holes, etc., are irrelevant. If someone grossly incompetent like this is on your team, the smart move is NOT to challenge or disprove, because again, no one evaluates proof on any basis except "which proponent do I trust more?". The smart move is to frame the situation such that his failures are opportunities for you to deduct from his social clout and add into your own, without ever firing an alarm in anyone's mind that you're trying to do this.
It's all an image game. As an individual contributor, you can avoid a lot of this game as long as you're non-threatening, churn out semi-reliable work, and are at least not annoying if not socially pleasant. Once you try to move up the ladder, even just one rung, image and likability goes from 65% of the equation to 99%.
I'm currently looking for a role, but I've worked here for 3 years because I have been able to basically do whatever I want. I'm a combination of analyst, data scientist, and marketer. I'm in charge of our appeals, from strategy all the way to the money coming back in house.
We have over half a million constituents. It's really great if you are interested in testing. I've sent out mailings where I'm testing 10 or 11 different factors. I've tested basically everything: the size of the font, the weight of the letterhead, the format of the coupon, how the letter is mailed, the dimensions of the envelope, the structure of the ask, etc.
Even with the added expense of testing, I've decreased our cost to raise a dollar by over 20 cents.
On top of that, I've automated all sorts of processes. At this point, I think my processes save about a man-month per month.
So, I have a lot to feel good about, but I'm really being limited on the technology side of things at this point. Biggest barrier is that I haven't quite decided what I want my next step to be at this point. I'm afraid I'll end up with a job with a much narrower focus and get bored.
The IT may be terrible, but I'm very thankful for the opportunity I've had to develop these skills. It's really unfortunate how one or two people can ruin an organization. The rest of the people here are incredibly kind and incredibly driven people.
I knew what the problem was because I had run in to that issue a few years back with my own computers. After I updated my DNS, all the Windows computers were having issues, but none of the Linux ones were. That's when I learned that Linux doesn't typically cache DNS records on local machines.
I know Chrome in Linux definitely doesn't cache DNS requests. I believe Linux only keeps the DNS info around until the socket closes, but I don't know the actual implementation. I looked in to the ncsd man page, and it looks like DNS info isn't cached, only open sockets.
I know Chrome uses the system DNS cache in Windows, and my understanding was that all browsers in Windows used the system DNS cache.
I don't know a lot about how sockets are handled. I thought they were discarded as soon as they were closed, but they could function similarly to a DNS cache. I though a DNS cache stuck around for a lot longer though.
They are very important people.
You're best off running your own NTP infrastructure that's isolated from the public Internet NTP servers if you want stable time sources. Put a CDMA NTP server in two locations, let them sync, and go. You can get cheap EndRun servers on eBay for a few hundred bucks.
I'm willing to bet they don't realize that they've blocked their server. What I want to know is how the drift got to be this bad in a matter of days. It's like whatever computer they are using as the NTP server doesn't have a real-time clock.
It sounds like their domain controller holding the PDCE FSMO role is a virtual machine. This advice might be outdated but last time I checked it was better to host it on a physical machine. If they're restoring DBs from backups without notifications and managing their Active Directory environment like this I can only imagine how fragile your infrastructure really is. Good luck, you're going to need it.
Called it. I didn't even think about the fact that it was a virtual machine until you mentioned it. Everything makes so much sense now. The randomly disappearing files, random issues with file locking. I just assumed the problem was they had no idea what rules to use for file replication.
> I can only imagine how fragile your infrastructure really is.
Yeah, it's a nightmare. Ignoring the "files disappear and are never found" issue, files get overwritten with earlier versions all the time. I'm not entirely sure what their hosting situation is, but I suspect they are running virtual machines in virtual machines.
On top of that, most applications are only available on VMs through a Citrix connection. This isn't inherently a problem, but they have it set up so everyone is logged on to the same machine. This causes issues with most MS Office applications, because they assume one user per computer is accessing the files. So file locking is completely broken, because Windows looks at the file and says "Hey, that's me, I have that open!"
It's the biggest nightmare with Access applications, (which we have a ton of), because Access assumes that everyone has their own copy of the front-end (client) database, and each independently accesses the back-end (server) Access database. Optimistic locking doesn't work at all with the "everyone uses the same file from the same computer" approach, and pessimistic locking barely works. I've set up the client databases to automatically create a copy for each user, but as you can imagine it's a bit of a nightmare to keep everything updated.
AD and a few other systems get very snippy if the time various servers have isn't in sync, although they don't have much of a problem if every server has the wrong time (within reason).
https://github.com/jbenet/ios-ntp
Specifically, all the servers(!) from here are contacted: https://github.com/jbenet/ios-ntp/blob/master/ios-ntp-lib/Ne...
Note that the library author wrote:
"ios-ntp is often (mostly?) used to make sure someone hasn't fiddled with the system clock. The complications involved in using multiple servers and averaging time offsets is overkill for this purpose. The following skeleton code is all that is needed to check the time."
And that "skeleton" contacts just "time.apple.com"
But the library really has the default possibility of contacting a lot of the ntp.org servers from a big list ("createAssociations" with no parameters!) and it's bad.
As we know, the developers like to just "copy-paste" whatever is where. Or use any defaults. "Hey it works."
It seems that the iOS library author "helpfully" provided the default of contacting 30 servers from the ntp.org pool.
You're wrong, see my upper comment in this very thread, I've wrote already: they used the third-party library which has as default in the call without any arguments the huge number of the ntp.org servers. The servers hit and the servers in the library sources match exactly. They used that default, resulting in 30 servers being hit at once from a single app (and the app is probably used by tens of millions of people).
https://github.com/jbenet/ios-ntp/issues/47
The original author writes:
" I wrote this library a long time ago for the iPhone 3GS and earlier. At that time, as best I could tell, the iPhone obtained its time from the phone company and, for me in South-East Michigan, it was not unusual for the iPhone time to be at up to two minutes variance from true time. Since my app was predicting the location of the International Space Station and a two minute error represented about 600 miles, I wrote this library so people wouldn't be staring at the wrong side of the sky!"
"Since then, the world has changed. First, iOS devices use the NTP protocol as a time source -- I stopped using my own library years ago because iPhones were delivering sub-second accurate times natively."
"I've thought for a while that this library had passed its "sell-by" date and was of minimal value; this Snapchat incident (about which I've heard only a little, and that obliquely), makes me want to remove it. I've been involved in network software, on and off, for about forty years and regard myself as a good net citizen."
The defaults up to now were really, really bad. His correction now is to remove the list. But if then the users just use any list, they are doing it wrong again. For effectively all the scenarios in practice, just time.apple.com instead of any other list and the limited number of IP addresses used from there, by default 1, should be enough for the iOS apps, or even better that they use their own servers. The swift library with the similar goals, mentioned elsewhere, has that IP-limiting loop, this library probably not.
/*┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Prepare a sort-descriptor to sort associations based on their dispersion, and then create an │
│ empty array for associations to fill .. │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘*/https://github.com/jbenet/ios-ntp/blob/master/ios-ntp-lib/Ne...
for use of two different styles of boxes, + some additional typographic experiments.
I actually kind of like it. Not enough to start using it yet, but then again, in Lisp code I make a judicious use[0] of ^L characters and form-feed-mode. Form feed character seems to be a forgotten but pretty neat invention.
I don't own an iPhone.
I think on the Nexus 4, Snapchat still ships with a warning that it doesn't work properly.
Mine would reboot about every second time I took a picture.
Still, I've got to wonder, what are they doing that's so different than other camera apps that seem to work fine?
To the unobservant user it looks like the OS is rebooting, although it's much quicker than a real reboot.
Two of them have been in the last month or so. Unfortunately the phone doesn't just reboot. It shuts down completely and refuses to boot back up with a low battery warning. I believe this is actually hardware related but I have had it crash when trying to call an Uber. I think Facebook may have caused a crash too.
Uber was especially painful because hailing taxis in my city (Seattle) is essentially impossible and pay phones are not a thing that exist in 2016. Luckily I was near a bus stop and know how the lines work without my phone.
It was a shocking illustration of how dependent I have become on my smartphone.
It took 4 days, to zero on the root cause. As is usual in a complex scenario like this there are a few false positives, some suspects abusing the protocol and alas final redemption. Amazing work by a dedicated group of technical folks in coordinating (just via emails, I suppose) and tracing the root cause.
> You must absolutely not use the default pool.ntp.org zone names as the default configuration in your application or appliance.
- http://www.pool.ntp.org/en/vendors.html#vendor-zone
Hopefully they were just unaware of the vendor zone policy.
It seems they didn't know, or didn't care, how both the third party iOS library they used and the NTP worked, see my other posts here.
They surely didn't need ntp.org pool at all.
The first one I had heard of was Netgear vs. UW-Madison.
It'd be harder to pull off on Google Play Store but I think Apple could make this happen if they wanted to.
Ethically, and morally, requiring the source to be available might be good.
to be clear, this is the reaction I anticipated when I wrote the gp. I don't think this is a cut and dried topic. I was merely expressing one side of the argument.
I don't think Apple compiling and signing binaries changes much when it comes to the average user and their security and trust model. It is my impression that Apple could easily intercept any keyboard input if they wanted to and frankly I wouldn't put it beyond the people at some other companies like Facebook or LinkedIn but I don't think we have found any good reason to lose faith in Apple's (or Google's) ability and will to do the right thing. Of course, ideally the idea of Trust No One is great but at some point you have to trust someone because who can read all the lines of all the code in the world?
I don't think Apple needs access to the source code to do things it needs to do. I was just saying that they have enough clout that the platform iOS has a solid foundation and can weather the storm that would inevitably rise from such a divisive decision.
I wouldn’t ever stake my reputation on signed code which hasn’t been signed by myself!
The only thing it can do do is show Apple you compiled the code.
There's no way you as the dev or the end user can verify the installed software really originated from you.
edit: snapchat apparantly isn't a facebook property, but it has very deep pockets and as such my comment still stands.
Excessive NTP queries aren't going to cause noticeable issues on the device or on Apple's test network, especially with only one or two people testing simultaneously.
Guess I know why now..