I'd also like to say that I highly recommend everyone does projects like this from time to time. I don't know any way to gain programming skills faster than making small throwaway projects using new tools and techniques.
Although now that I think about it, I could probably get close by using an 8bit greyscale image and then apply the palette in the client. That would probably halve the image size.
Take the bare bones of this project:
- Canvas.
- Websockets.
That's literally it. You'll need to know how to draw on a canvas, and how to send and receive WebSocket information. You can quite happily keep the current state of the canvas in an in-memory array, perhaps saving it to a file every few minutes in case the server crashes. Then, perhaps, when that's done you can swap our your in-memory array for a REDIS bitfield, improve the web sockets to no longer use JSON, but instead binary? Both of which should be only a few tens of lines of changes, but after that you'll be able to support tens of thousands of simultaneous users with hundreds, if not thousands of changes per second.
The thing with this project that's complex is the number of users required to use this at once, lessen the requirements a little and you'll come up with a simple project.
Looks like you genuinely felt it is out of the scope of a lot of novices (which it well might be at least for a weekend project), and you genuinely were looking for tips.
Just look for projects that would make yourself or others happy. Doesn't matter how small or large, whether it would take an hour or a decade, the hardest part is getting past line zero. Once you have that the next hardest part is getting to line one; wash, rinse repeat.
The good news is everything is relative and everyone has different skill sets - what's hard for everyone else might be a cinch for you due to your childhood fascination blending fishing line with champagne corks - who knows? ;)
Try not to be discouraged, just stick with it and be infinitely inquisitive, throw caution to the wind and dive right in.
Good luck and happy sailing!
I think all my technology choices worked out fine. I dumped server-sent events halfway through in favour of websockets because WS support binary packets. But that was a pretty easy change affecting at most 50 lines of code.
I still wish we had an efficient (native) solution for broadcasting an event sourcing log out to 100k+ browser clients, with catchup ('subscribe from kafka offset X'). Nodejs is handling the load better than I expected it to, but a native code solution would be screaming fast. It should be relatively simple to implement, too. Just, unless my google-fu is failing me I don't think anyone has done it yet.
Seems to me like you just described long-polling, which you dismissed in the article as "so 2005".
Yes, I did dismiss it out of hand in the article. The longer response is this:
In this instance long polling would require every request to be terminated at the origin server. I need to terminate at the origin server because every connection will start at a different version. The origin server in this case is running JS, and I don't want to send 100k messages from javascript every second. Performance is good enough, but barely. And with that many objects floating around the garbage collector starts causing mischief.
The logic for that endpoint is really simple - it just subscribes to a kafka topic from a client-requested offset and sends all messages to the client. It would be easy to write in native code, and it would perform great. After the per-client prefix, each message is just broadcast to all clients, so you could probably implement it using some really efficient zero-copy broadcast code.
The other approach is to bunch edits behind constantly-created URLs, and use long-hanging GETs to fetch them. I mentioned that in the blog post, but its not long-polling - there's no poll. Its just old-school long-hanging GETs. I think that would work, but it requires an HTTP request/response for each client, for each update. A native solution using websockets would be better. (From memory WS have a per-frame overhead of only about 4 bytes)
Nice work on all this!
The server that both the app and my blog are on only has 1 core and 1.5gb of ram. I didn't pay close enough attention to kafka, and it turns out kafka was using half of the available ram starving the rest of the processes. And it didn't help people's bots were thrashing POST requests at nginx to edit the page.
I've just done some high fructose server maintenance, spinning up a new machine 4 cores and 8 gigs of ram. The old site is proxying all traffic across, and once DNS propagates it'll stop being hit at all. Hopefully that'll ease the congestion.
Edit: at the time of writing everything seems back up and happy. Nginx was running out of open file descriptors, kafka was eating all the ram and ghost (my blogging platform) wasn't sending the right cache-control headers.
A few tweaks and a bit more CPU to play with and everything seems happier now.
How did you go about preventing spam? If you feel it might give the spammers the information to circumvent your measures, consider writing a blog post later.
How many users did the site receive over the course of the past 2 days?
Really cool weekend project btw, this is literally the first time I have seen someone follow through on the "I can build X in a weekend" claims.
I can't really hide anything - all the code is on github, though its ... um, ... not pretty. It might be better if nobody looks. http://github.com/josephg/sephsplace
But the system at the moment is really simple: The site only accepts 10 edits per 10 seconds from any IP address. After that your edits get rejected until the next 10 second window begins. You can write bots to draw things for you (and lots of the images you see are drawn this way). But drawing big objects is slow. So thats ok. I think thats a reasonable compromise between bots being powerful and humans being powerful.
The giant Angela Merkel image (and some other smut that I deleted) was drawn by someone proxying edits through about 200 IP addresses. I don't know if they're using TOR, or have access to a botnet or are using an anonymizing proxy or something. I could tell they were all the same botnet because all the requests had the useragent of 'python-requests/2.10.0'. (I have an IP address list if anyone wants to take a look.)
Anyway, I figured those addresses are probably hard to replace - so I let them do it, harvested the addresses and banned them all. Worse, I made it impossible to tell which servers are banned - the server replies to banned servers like normal - their edits just never appear.
I caught about 2/3rds of their addresses before they started making their headers match real browser traffic, but I think I ruined their fun and they stopped.
I have a few more tricks up my sleeve I could pull out if that little war escalated further. For example, I could always add a captcha you have to fill out when you open the page. The captcha would generate a token that you would have to provide with each edit. Rate limiting would then be by-token. Bots would still work, but you would have to give the token to your script. But getting around the rate limiting would be harder.
> How many users did the site receive over the course of the past 2 days?
Um I'm embarrassed to say I don't know! I don't have a log of which IP addresses generated each edit. Monitoring and logging seemed much less important at the time. Its obvious in retrospect, but I wish I'd sent each edit along with a timestamp and some metadata into a separate kafka queue when I received them. That way I would have a complete audit log to play with now. I have all the edits - but I have no way of knowing who did what, or how many unique visitors I've had.
The site was called 'blograffiti.com' - Remnants of it still exist on the wayback machine.
Fantastic work by the way. And thanks for the post about it all. I love weekend projects like this. That's exactly what kicked blograffiti off. (And most of my most valuable projects, come to think of i!.)
I'm thinking of a grid of Litecoin addresses would work well to limit abuse of infrastructure to gain advantage, while still allowing bot activity. A payment to a given address would last the amount paid divided by cost of ownership amount per time period.