That UDID script I wrote - 350K hits... (w/source) | Hacker News Reader

That UDID script I wrote - 350K hits... (w/source)

blog.maclawran.ca

53 pointsbbunix32 comments

Security-wise, the traffic to his server is http, meaning any man-in-the-middle can collect tuples of UDIDs, IP and browser fingerprints, even if he doesn't collect anything.

If you worry about UDIDs being tracked I guess you should worry about the http to his page too.

However, according to somebody who apparently tested it a year ago (and posted it on HN), half of the iPhone apps pass UDID over the same http, allowing collection of the very same or similar data.

Which makes the whole subject of a government agency having UDID database without other convenient data, ummm, quite less probable. IMHO most of the "business entities" are able to collect much more than presented here.

The page size is just shy of 3kb. At 350k hits, that's around 1gb of data transfer.

If he had 100 requests in one second (average is 8/ps over 12 hours), that's 300kb per second transfer rate.

1gb of data transfer for a single day is relatively small. Server upload rates of <1mbps is also small.

If we make the page 100kb in size now, for example with a few nice images 350k hits turns to 34gb of data transfer. 100 requests in one second would require a 10mbps transfer rate. A $15p/m host likely wouldn't be able to cope with this.

> I think the combination of Lighttpd+php5-fpm is underappreciated...

So for me the real reason the site stayed up was not because he picked PHP or any other technology, but because the page was barebones and didn't put any strain on the server at all. Page file size is an underappreciated advantage for websites!

It's cool that he got so much press in such a short time, but applauding technology choices that allowed him to sustain 8 requests per second (assuming this was over a 12 hour period) is ridiculous.

I really doubt that these 350k hits were spread neatly over 12 hours.They most likely came as one giant rush. The author would have to confirm.

Considering that lots of peoples blogs die because they get 10k visitors from HN. His small $15 vps did fine.

The PHP script is super simple, so lighttpd (which is a great web server, in the same performance realm of nginx) and php-fpm on a shared VPS, is plenty of power and resources.

Never seen $found = `grep $udid FILE` using back-ticks in PHP land. Is that the same as the exec() function?

effectively the same, more like a system() call... impressively ugly I must say.

I've seen this used in a PHP git class to call git and capture the results. (Obviously) Just make sure you're not including any user input data in any variables you put in there.

It's also grossly insecure unless you sanitize $udid, which thankfully the author of this script does.

It's still relatively unsafe, as it invokes external programs through a shell, so you're dependent on the shell's environment, which can diverge on different systems. Last I checked, PHP did not have an easy, obvious way to safely invoke another program. You can do it manually with a mix of pcntl_fork and pcntl_exec, but capturing the output using that is more difficult.

PHP provides escapeshellcmd / escapeshellarg functions for escaping when building commands.

Is there any shell on which those functions are unsafe ? I think they assume cmd.exe on windows and sh on others.

Escaping is difficult to get right, especially in PHP... witness things like magic quotes that existed for so long.

The right thing to do is provide an output capturing API that does not require escaping because it takes a list of strings passed to the exec(2) system call.

All those blogs die because people don't bother to use WP Super Cache or the equivalent.

Just for comparison, in 1998/99 I ran an mp3 search engine that handled 200k daily queries at the peak. It was a Pentium with 128MB of memory. The search server was written in C, and it used an inverted index. The system load rarely went over 1.

Doesn't IndexTank still run on that Pentium with 128MB of memory? I kid. :)

You missed the real catch that he mentioned towards the end - Quoting from the article - "The lesson is really simple. Make it work, put it out. You can make it pretty later, maybe."

I had basically the same experience a few years ago when the Wikileaks stuff was first released and I hacked together a basic search engine for it. Interestingly, I got most traffic through Twitter which I hardly ever use personally.

Anyway, that was an EC2 Micro running PHP-FPM on Nginx, and it stayed up without problems through millions of hits over a few days as well. It's easy to write a well-scaling single-function site like that.

Yet a lot of blogs and more complex sites die instantly when they cross a couple of dozen hits per second. I suspect there are multiple reasons for it, but the most egregious one coming to mind is the typical PHP setup within shared hosting environments where every single request means about 30 file compilations and a lot of DB requests. Caching is probably not very popular either.

My http://pastehtml.com/udid got 130k hits yesterday. PasteHTML.com itself and this UDID tool are running on a moderate dedicated server, nginx/rails/postgresql.

What has impressed me most, is that a pretty crap script has received so much praise.

EDIT: Re-reading, that sounds a little bitchy.. I didn't mean it that way. Sorry

Please describe how this script is crap. It's short and to the point, it does its job, handles the load, and it doesn't pass bogus characters to the command line.

Perhaps it needs commenting? Could be formatted more nicely? I'm truly interested in your characterization because I'm just not seeing it.

I don't know if the 350k hits run that script, let's say they did.

The OP is praising the performance of lighttpd + php5-fpm, but the script actually spawns an external process to search for the UUID. That's like running an old GGI that needs to be executed with every request.

I may be wrong but using lighttpd and php5-fpm (with fastcgi) may not be as relevant as it seems. I'd say that the operating system is caching everything.

EDIT: yes, php-fom is "a simple and robust FastCGI Process Manager for PHP". I'm no doing much PHP lately!

Premature optimization. Obviously, this thing works well enough and handles the load. Why worry about whether it's launching a process?

When the system begins to show signs of stress, you look at where it's spending its time, then alleviate the bottleneck.

Finally ran the tool and the only issue I had was that the 'not found' message was at the bottom of the page, now below the fold.

When I first visited the site, before he started putting more text on the page, I don't think I would have run into the issue.

Browser window isn't maximized; if I scroll all the way up it cuts off the last two paragraphs/sentences (link to his wife's blog and the results).

Otherwise it's simple; it doesn't need to be flashy when it's a one-stop site (unless they release the remaining ones).

I agree. I had never heard of the site this post is referring to. I used a different site to check UDID's that let you only enter in part of the UDID. This site looks like it REQUIRED you to send the ENTIRE UDID. Not very good.

I also had a blog post hit the front page of Reddit as well as Gizmodo. ~400K hits over 24 hours. Page size was roughly 700KB in size. All hosted on a shared hosting plan at HostGator.

How? A free CloudFlare account to cache every image/css/js resource on the page, plus a custom page rule to force cashing of the actual html response from the server.

Did I think my situation was impressive? Not really. Just an average number of pageviews for a front page article.

What someone needs to do - ASAP - is do another one where people whose UDID's were on the list can list all the apps they've bought (it should be possible to get these out of itunes somehow) and we can use that to narrow down which ones most people have in common, and track down the organisation responsible for collecting this data in the first place. I would do it myself but I really don't have the free time at the moment.

Isn't that what http://fredericjacobs.com/identifying-the-traitor is?

Exactly what I was thinking of.

That demonstrates the value of launching early and "advertising" in the right spots ..: Not so much the capabilities of php or cheap VPS.

If your UDID script returns a positve, could you display a link to http://fredericjacobs.com/identifying-the-traitor and his "identifying the traitor" survey?

Done.

Pretty incredible that you were able to withstand that on a $15 VPS. Nice job, man.

The article mentions that it's difficult to oversell Xen boxes. Anyone care to comment on why this is?

Take a look here: http://www.freewebspace.net/forums/showthread.php?2246409-Xe...

+1 for lighttpd. We use it extensively here at Nextpeer and it's been super so far.

I wonder... Could he have done it on a free heroku instance?

Reply on news.ycombinator.com