How We Built r/Place
redditblog.com
redditblog.com
I'm pretty active on both. What threads/subreddits are you reading?
- /r/Libertarian
Pretty funny considering the content of the post he wrote that on.
I do and don't agree with you. Whats really going on here is that development time scales linearly with the number of decisions you need to make. Decisions can take the form of a product questions - "what are we making?" and development questions - "how should we implement that?".
There are three reasons why people feel confident saying "I could copy that in a weekend":
- When looking at an existing thing (like r/place), most of the product decisions have already been made. You don't need to re-make them.
- If you have a lot of development experience in a given domain, you don't need to make as many decisions to come up with a good structure - your intuition (tuned by experience) will do the hard work for you.
- For most products, most of the actual work is in little features and polish thats 'below the water', and not at all obvious to an outsider. Check out this 'should we send a notification in slack' flowchart: https://twitter.com/mathowie/status/837807513982525440?lang=... . I'm sure Uber and Facebook have hundreds of little features like that. When people say "I could copy that in a weekend", they're usually only thinking about the core functionality. Not all the polish and testing you'd actually need to launch a real product.
With all that said, I bet I could implement r/place in less than a weekend with the requirements stated at the top of that blog post, so long as I don't need to think about mobile support and notifications. Thats possible not because I'm special, and not because I'm full of it but because most of the hard decisions have already been made. The product decisions are specified in the post (image size, performance requirements) and for the technical decisions I can rely on experience. (I'd do it on top of kafka. Use CQRS, then have caching nodes I could scale out and strong versions using the event stream numbers. Tie the versions into a rendered image URL and use nginx to cache ... Etc.)
But I couldn't implement r/place that quickly if reddit didn't already do all the work deciding on the scope of the problem, and what the end result should look like.
In a sense, I think programming is slow and unpredictable because of the learning we need to constantly do. Programming without needing to learn is just typing; and typing is fast. I rule out doing the mobile notifications not because I think it would be hard, but because I haven't done it before. So I'd need to learn the right way to implement it. - And hence, thats where the scope might blowout for me. Thats where I would have to make decisions, and deciding correctly takes time.
But in all seriousness, I agree with most of what you said - I think I'm just more bearish on people's ability to infer those many decision points without being given the blueprint like we were in this article.
If you are a senior engineer at FB and you decide to make Twitter in your spare time, I buy that lots of experience and knowledge gained at your job can probably get you going fairly quickly. But I have never seen an example of engineers discussing sophisticated systems like these where crucial aspects of its success in terms of implementation didn't rely on some very specific knowledge/study of the particular problem being solved that could only be gleaned after trying things out or studying it very carefully. The representation of the pixels is a great example in this case -- they go into wonderful detail about why they decided to represent it the way they did, which in turn informs and impacts how the rest of the stack looks like.
I think at one point Firebase had it as one of their example apps something which very closely mirrored what they did with r/Place, so I agree that one could probably build some "roughly" like it somewhat quickly. I agree that in general knowledgeable individuals could probably grok things which would in some form resemble popular services we know today. The devil is in the "roughly," though. I think that often what makes them be THE giant services we know are things that let them have the scale which very few of us have ever needed or know how to deal with, or because they have combined tons of "below the water" features and polish like you mentioned. When really basically most web apps that we use are CRUD apps which we all know how to build, I think maybe we need to give more weight to these "below the water" features in terms of how much they actually contribute to the success of the applications we use.
That's funny because I thought (assuming you're referring to the bit-packing of the pixels) that seemed to me one of the more obvious choices (to do something else would have been more remarkable to me).
Beyond that though, I have zero experience with real-time networked systems. Especially with a gazillion users and that everybody gets to see the same image, that seems hard.
The cleverest solution that I read in the article, that I really liked but probably would never have thought of myself (kinda wish I would, though) was the part where the app starts receiving and buffering tile-changes right away, and only then requests the full image state, which will always be somewhat stale because of its size, but it has a timestamp so they can use the buffered tile-changes to quickly bring its state up to time=now. Maybe this is a common technique in real-time networked systems, I don't know, but it was cool to learn about it.
It's getting the nitty-gritty details hammered out and then implemented that takes time and people. I'm currently trying to do something similar but directed in a very different way. Facebook feels like a CRUD app. But there's a lot of different moving parts that you have to deal with on top of the CRUD pieces.
And every single one of those things--while not necessarily difficult to implement individually--add up to tons of decision time.
When the expected outcome is already decided, the problem becomes much much easier to solve for a developer.
Assuming that you're dealing with some kind of a reasonable web stack, implementing any individual feature is often not that big a deal. Deciding to do it, and also perhaps making the choice to have a reasonable web stack to begin with, are potentially problematic.
And of course, building something that can massively scale is hard as well.
But from my experience with side projects and also my day job, the hard part is making decisions.
The challenge is: Meet the main requirements of r/place: 1000x1000 image, a web based editor, 333+ edits / second, with an architecture that can scale to 100k simultaneous users (although that part will be hard to actually test). I won't implement mobile support or notifications, and I won't implement any sort of user access control (thats out of scope). The challenge is to do it & get it hosted online before I go to bed on Saturday (so, I'm allowing myself some slop there). But its already 1:30pm on Friday, so I think that easily qualifies as "less than a weekend".
As a stretch I'm going to write all the actual code in nodejs while aiming for well into the thousands / tens of thousands of writes per second territory.
I'm willing to accept some fun stakes if anyone wants to propose them (ice bucket challenge level stuff). I'll live-tweet progress here - https://twitter.com/josephgentle
(Edit: fixed days)
Polish and perf tuning tomorrow.
Code: https://github.com/josephg/sephsplace (warning: contains evil)
I'll post a writeup about it all once I've eaten some real food and had a rest. But for now, enjoy!
It would be kind of neat if the place experiment could become something of a micro-benchmark for online customer facing distributed platforms.
Keep up the good work man, it's looking good.
Don't forget to hydrate and consume pizza.
Mobile support was a huge factor for /r/place. To reimplement the project while completely ignoring mobile is basically taking out maybe 30% of the effort required. It's a bit of a copout to claim "can implement in a weekend", while discounting one of the main reasons why it is nearly impossible to hammer out solo in a single weekend.
The other "big ticket item" is integrating such a system into an existing live stack. It's very convenient that you get to pick the tools you think are most efficient for the job from scratch; it's quite another to restrict yourself to an existing set of tech.
>> an architecture that can scale to 100k simultaneous users (although that part will be hard to actually test)
Also this. Reddit had a single opportunity to deploy live. You can't throw out many days of required dev/staging benchmarking for a system that must work on first deploy. You're not "done" after a weekend unless your first deploy both works 100% and scales as required.
tldr; Were I to do what you are doing, it would be out of interest and to put it live for others to use. What seems disingenuous is to do so with the primary goal of proving that "see, child's play - it's not difficult at all!". The goal should be to build something that works, not to rush the job as some way to show off and take something away from Reddit's efforts.
From my post above:
> But I couldn't implement r/place that quickly if reddit didn't already do all the work deciding on the scope of the problem, and what the end result should look like.
If anything, this only increases my motivation to replicate the project myself, whether it's during a weekend or two full weeks. It's interesting enough and at the right level of complexity - kind of simple, but not too simple - to make it a fun side project.
I thought I knew timezones... But that's 40+ hours ahead of me.
But there's no reason I couldn't use multiple partitions if I had to. To make it fully consistent I'd just need a consistent way to resolve conflicting concurrent writes on different partitions. Either adding a server timestamp to the messages, or using the partition number to pick a winner would work fine for this. It would just make the server logic a little more complicated.
Yes - today's Facebook could not be built by a dev in a weekend, but Facebook 1.0? Not so hard.
The secondary goal of pushing pixel updates to lots of users would probably want separate hardware. I would probably fake it so its not really realtime. If you send the web client batches of updates to replay on the board it looks realtime and doesn't require persistent connections.
I also like the way the article is broken down into the backend, API, frontend and mobile. This isolated approach really highlights the different struggles each aspects of the product has, while dealing with what is essentially a shared concern: performance.
What I also found interesting is the fact that they were able to come up with a pretty accurate guess in terms of the expected traffic.
> "We experienced a maximum tile placement rate of almost 200/s. This was below our calculated maximum rate of 333/s (average of 100,000 users placing a tile every 5 minutes)."
Their guess ended up being a good amount above the actual maximum usage, but it was probably padded against the worst case scenario. The company that I work for consistently fails to come up with accurate guesses even with our very rigid user base, so it's pretty impressive that Reddit could accommodate the unpredictable user base that is the entire Reddit community.
http://highscalability.com/blog/2016/4/20/how-twitter-handle...
https://redditblog.com/2017/1/17/caching-at-reddit/
https://blog.twitter.com/2017/the-infrastructure-behind-twit...
https://www.netlify.com/blog/2017/03/16/smashing-magazine-ju...
https://nickcraver.com/blog/2016/02/03/stack-overflow-a-tech...
https://engineering.linkedin.com/blog/2016/10/instant-messag...
http://techblog.netflix.com/2016/08/building-fastcom.html
http://highscalability.com/blog/2016/9/28/how-uber-manages-a...
Or a recent writeup on a challenge building IMDB's forums in 2001(!) https://www.beatworm.co.uk/blog/internet/imdb-boards-no-more .. fun and scary in equal measures.
(I'm plucking these from things were popular recently in our http://webopsweekly.com/ :-))
Edit: why am I getting down voted for praising a write-up?
I've done a fair amount of tech writing.
Documentation always takes me as long or longer than designing, coding.
Most recently, when I released a novel layout manager, the docs, examples, screenshots, etc took roughly twice as much effort as everything else combined.
I used /r/place from a few different browsers with a few different accounts, and they all seemed to have slightly different view of the same pixels. Was I the only one who experienced this problem?
When /r/place experiment was still going, I assumed that they grouped updates in some sort of batches, but now it seems like they intended all users to receive all updates more or less immediately.
Have you looked at NATS at all? We're using it as a message bus for one app and it's been fantastic. It is, however, an in-memory queue, and the current version cannot replace Rabbit for queues that require durability.
tbh, i never used clustering (because it's one of the shittiest clustering implementations i've ever seen) but we do use two servers (publishers connect to one randomly and consumers connect to both) and it seems to handle millions of messages without any issues.
of all servers i've ever used, rabbitmq is by far the most stable (together with ejabberd).
Right now, the main annoyance is that it's impossible, as far as I understand, to limit its memory usage. You can set a "VM high watermark" and some other things, but beyond that, it will — much like, say, Elasticsearch — use a large amount of mysterious memory that you have no control over. You can't just say "use 1GB and nothing more", which is problematic on Kubernetes where you want to pack things a bit tightly. This happens even if all the queues are marked as durable.
So it's awesome for realtime firehose-type use cases where a websocket client connects, receives messages (every client gets all the messages, although NATS also supports load-balanced fanout) for a while, then eventually disconnects.
NATS is ridiculously fast [1], too.
There's an add-on currently in beta, NATS Streaming [1], which [2] has durability, acking/redelivery and replay, so covers most of what you get from both RabbitMQ and Kafka. It looks very promising.
[1] http://bravenewgeek.com/tag/nats/
[2] https://nats.io/documentation/streaming/nats-streaming-intro...
If you already have a good Redis infrastructure then you can just use the pub/sub features built into it for your websockets communication.
BTW, thanks for the great post, it was a very interesting read.
You're already on AWS, why not use SQS instead?
And soon you get exactly what /r/place was.
As others have said subcommunities quickly formed or some subreddits themselves had a orga-thread to paint an iconic logo relevant to their niche.
Also, reminds be of the million dollar front page [1].
[1] https://en.wikipedia.org/wiki/The_Million_Dollar_Homepage
If it reached 5$, he'd have made 25,000$
>>> total = 0
>>> for i in range(5, 500, 5):
... total += i
>>> total
24750
I remember lueshi
[1] https://medium.com/@JohnWatson/real-time-collaboration-with-...
It was quite interesting for a little while.
> 500kB packed array, coupled with cooldown logic, should have a negligible performance cost and can easily be done on a single thread. That thread could interact via atomic channels with the CDN
That's not simpler.
We used tools that we're already using heavily in production and are comfortable with.
With respect to your experience in the matter, I strongly disagree. What I described is complicated to say, easy to implement. What the OP describes ("use redis") is easy to say, complicated to implement. Not just in terms of human work time (setting up the redis machine and instance, connecting everything together), but also in terms of number of moving parts (more machines, more programs, etc.).
> We used tools that we're already using heavily in production and are comfortable with.
That's entirely fair, and what I figured was the most likely explanation.
Because machines go down. If you don't expect your hardware to fail at the most inopportune time, you'll be screwed when (not if) it happens.
The crux of the problem is that they need to mutate a relatively tiny amount of memory and have a rolling log of events for which only the last 5 minutes needs fast access. Also, if you can put all your state on one machine its far less likely that the one machine will die, than it is that at least one will die in a cluster of machines. Given the nature of the problem keeping all state on one machine seems pretty rational to me, so long as you have the ability to switch to a hot spare within a few minutes or so.
If I were to architect this for speed I would have two tiers: a websocket tier, and secondly a 'database' tier. The database would be a custom program that would:
0. Provide a simple Websocket API that would receive a write request and return either success if the user's write timer allowed it to write or failure if it didn't. This would also broadcast the state + deltas. 1. Keep the image in memory as a bitmap 2. Use rocksdb for tracking last user writes to enforce the 5m constraint. You could use an in memory map, but the nice thing about rocksdb is that it shouldn't blow up your heap. 3. Periodically flush the bitmap out to disk to timestamped files for snapshots 4. Keep the hashmap size small by evicting any keys past their time limit 5. Write rotating log files rotated every 5m or so to record the history of events for DR and also later analysis
Backing this sort of thing up is very simple. You just replicate the files using rsync or something like it. You may have some corruption on files that are partially written, but since we're opening and closing new files often you can choose how much data-loss you want to tolerate.
Restoration is as simple as re-reading the bitmap and reading the log files in reverse up to 5 minutes ago to see who still isn't allowed to write yet (thus reconstituting the hashmap). Let's remember, redis replication is async, so this has the same tradeoffs.
Creating a "custom database program" is not a small task.
We like to use boring technologies that we know work well. We were already using Cassandra, had some experience with Redis, and had a lot of confidence in our CDN.
I'm not arguing that most problems need a custom database, only a minority do. I'd say that this problem is borderline on which direction to go.
Databases are very leaky abstractions as you all discovered. The nice thing about custom code is that you don't have leaky abstractions. The bad thing about custom code is that you have a large new untested surface area.
In the case of your application the requirements are so minimal, a bitfield plus a log, I'd say its a wash.
Programmers today forget that things like flat files exist and are useful. It's a shame, because you wind up with situations where people just assume they need a giant distributed datastore for everything.
What you're doing in that case is trading architectural complexity for code complexity. Now, if its the case that all data in your org goes in one data store to keep things consistent, great, that makes sense. But for a one off app I just don't buy it.
It only takes a couple of outliers to bring everything down. I'm not exactly well-versed in defining specs for large scale backend apps (not a back-end engineer) but it seems to me that preparing for the average would not be a wise decision?
For example, designing with an average of a million requests per day in mind would probably fail, since you get most of that traffic during daytime and far more less at the nightly hours.
Could anyone more experienced shed some light?
>We should support at least 100,000 simultaneous users.
This line makes me think that this is what they expected the peak (or near peak) to be.
>Users can place one tile every 5 minutes, so we must support an average update rate of 100,000 tiles per 5 minutes (333 updates/s).
So assuming that they mean that 100k is the peak and that clients are limited to 1 update per 5 minutes, they can expect 333 updates per second on average. The "average" is taken over this 5 minute period. This average represents the number of queries per second they will get if everyone's 5 minute cooldown is spread out evenly over each 5 minute period.
It is possible, for example, for half of the peak population's cooldown to expire at 1300 and the other half to expire at 1305. In this case the average updates/s over the 5 minute period from 1300-1305 would still be 333 updates/s even though there were really 2 bursts of 50k a second at 1300 and 1305. It's far more likely that cooldowns are not excessively stacked in this way, so you prepare for the average and hope for the best.
I didn't follow /r/place that much, but I haven't read any complaints about latency or failures so it looks like they did just fine.
Edit: you're an sre, you probably have more experience planning these things than I ever will. I just can't help but think about what would happen if some trolls realized they could synchronize their updates and bring down the service. (Although based on the infrastructure it doesn't seem possible to cause lasting damage.)
It's not an average: Reddit controlled the 5 minute user cool-down period, a.k.a. request throttling. 333 updates/s was the capacity. As alluded to in the article, the 5 minutes was dynamically configurable: if more than 100,000 had users showed up, they would have increased the cool-down period to a value that would yield 333 updates per second at most.
> For example, designing with an average of a million requests per day in mind would probably fail, since you get most of that traffic during daytime and far more less at the nightly hours.
That depends. If you're talking about a public website, then yes. But this is just a single API endpoint with a very fixed time-based rate limit.
I know they deal with insane scale yadayada but it's simply not ready.
That and the horrific dark pattern on the "We want you to have the best, massive red/orange button marked continue that takes you to the app store and the tiny weeny little "continue to mobile site" underneath".
I don't want your damn app, stop asking me.
If I was cynical I'd think they didn't care about the mobile site been awful as it drives people to the app.
I was trying to draw something, one pixel at a time, and all of a sudden, after a bunch of pixels, it stopped rate-limiting me! I could place as many as I wanted! So I just figured that they periodically gave people short bursts where they can do anything. This was backed up by my boss, who was also playing with /r/place, saying that the same thing happened to him not long before that (yes, my whole team at work was preoccupied with /r/place that Friday). So I quickly rushed to finish my drawing.
And then I reloaded my browser... and it wasn't there. Turns out that what I thought was a short burst of no rate limiting was just my client totally desyncing from Reddit's servers. Nothing was submitted at all.
Not too long after that, another guy on my team got hit by the same bug. But I told him what happened with me, so he didn't get his hopes up.
I really enjoyed the part about TypedArray and ArrayBuffer. And this might be a common thing to do, but I've never thought about using a CDN with an expiry time of 1 second, just to buffer lots of requests while still being close to real-time. That's brilliant.
This is the epitome of an anti-pattern .I sincerely hope that this approach was floated by somebody who had never used Cassandra before.
Even if individual requests were reasonably fast, you are sticking all of your data in a single partition, creating the hottest of hot spots and failing to leverage the scale out nature of your database.
It's literally like exploring the digital universe and reporting on some of your findings.
Great writeup!
[0] - https://github.com/reddit/reddit-plugin-place-opensource/com...
From a reddit "expert", so I guess that answers that ;)
Again, amazing write-up. Thank you!
How big was the team? How long did it take to complete this project? Is the code going to be open sourced?
> At the peak of r/place the websocket service was
> transmitting over 4 gbps (150 Mbps per instance
> and 24 instances).
Like
timestamp, x, y, color, username
https://www.reddit.com/r/place/comments/6396u5/rplace_archiv...
Complete?
I've been working with that.
Please explain.
I've exported it to imgur for you.
And for those interested, here's some additional stats:
- The original announcement about /r/place: https://www.reddit.com/r/announcements/comments/62mesr/place...
- Full timelapse of the canvas over the course all 72 hours: https://www.youtube.com/watch?v=XnRCZK3KjUY
- Heatmap of all activity on the canvas over the full 72 hours: https://i.redd.it/20mghgkfwppy.png by /u/mustafaihssan
- Timelapse heatmap of activity on the canvas: https://i.imgur.com/a95XXDz.gifv by /u/jampekka
- Entropy map of the canvas over the full 72 hours: https://i.imgur.com/NnjFoHt.jpg by /u/howaboot (explanation: https://www.reddit.com/r/dataisbeautiful/comments/63kuy6/oc_...)
- Map of all white pixels that were never touched throughout the event: https://i.imgur.com/SEHaUSJ.png by /u/alternateme
- Most common color of each pixel over the last...
- 72 hours**: https://i.imgur.com/C5jOtl1.png by /u/howaboot
- 24 Hours: http://aperiodic.net/phil/tmp/place-mode-24h.png by /u/phil_g
- 12 Hours: http://aperiodic.net/phil/tmp/place-mode-12h.png by /u/phil_g
- 6 Hours: http://aperiodic.net/phil/tmp/place-mode-6h.png by /u/phil_g
- 2 Hours: http://aperiodic.net/phil/tmp/place-mode-2h.png by /u/phil_g
- Atlas of the Final Image: https://draemm.li/various/place-atlas/ by /r/placeAtlas/ (source code: https://github.com/RolandR/place-atlas)
- Torrents of various canvas snapshots and image data: https://www.reddit.com/r/place/comments/6396u5/rplace_archiv...
- The post announcing the end of /r/place: https://www.reddit.com/r/place/comments/6382bb/place_has_end...
It took a while for members of the community to realize what was happening and start recording snapshots of the canvas, so there are a few time periods early on that got skipped
This is why you use a proper database.
I'd probably add a Postgres table to record all user activity, and use that to lock out users for 5 minutes as an initial filter. Have triggers on updates to then feed the rest of the application.
I lean towards just using the ratelimiting stuff we already have in place (via memcached, which we talked about in a previous post). We just overlooked it.
The simple fact of introducing a one-million-row read for the latest data of each "pixel cell" is fairly insane. You must have a cache for such data. "I'd still have have Redis cache, though" is not even debatable. It doesn't have to be Redis, but is definitely has to be a cache of one kind or another.
place=> explain analyze select * from board_bitmap ;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Seq Scan on board_bitmap (cost=0.00..14425.00 rows=1000000 width=6) (actual time=0.009..57.295 rows=1000000 loops=1)
Planning time: 0.160 ms
Execution time: 90.510 ms
(3 rows)
I don't think you understand how fast Postgres is on modern hardware. What took a large cluster 5 years ago can be done on a single system with a fast NVMe drive today. We really might not even need Redis in this situation.
And, yes, I have to deal with viral content, so this is right up my alley.
No, just using it to store the update log.
But I don't know if there's any obvious problem with querying a handful of megabytes once per second either.
All we needed to do here was add some simple locking and we would have been fine.
Ideally, you would have also used redis to limit the per-user activity without having to hit Cassandra. Also not sure why you hit Cassandra instead of redis for the single-pixel fetch endpoint (redis GETBIT operation rather than a database hit); if you already conceded to not-quite-atomic operations across the entire map, a GETBIT would have rarely returned a stale data point. But these are minor nice-to-have criticisms that would have pushed the scaling capabilities even further beyond your expected requirements. All in all, again I highly commend your results. You had one minor snafu, and managed to overcome it. Well done!
Aside: my brain is spinning as to how I would provide a 100% guaranteed atomic version of /r/place - without any point of failure such as a redis server not failing/restarting, or a single-server in-memory nodejs data structure. Really tough to do so without any point of failure or concession to atomicity. :)
Second aside: more than anything, I am surprised you have a CDN that allows 1-second expiries. While perfect for this kind of project, too many CDNs find a 1-second expiry as a risk to permit, as they tend to expect too much abuse/churn. ie: How is a CDN supposed to trust you enough to use a 1-second expiry for reasonably high traffic, rather than cycling so much caching effort for something that could have used a 5 minute expiration? I can't imagine being the developer of a CDN that trusts its users to use a 1-second expiry that wastes an insane number of CPU cycles for an origin that is not legitimately sustainable.
tldr (still long, but on point): You guys did an amazing job for something that lasted, what was it, 3 days? Great job! Many of your critical audience members would not have managed any better, let alone being viable and functional. I would submit my résumé to work for you, but I fear my personality is far too... um... abrasive... to get along with the organisation as a whole. In any case, your team as a cohesive unit - design, backend, and frontend (especially the mobile support) - did an incredible job. +1 to the Reddit team here, you should be immensely proud of yourselves for pulling this off.
We don't have a normal CDN, we have Fastly. They are really incredible at what they do, and this would not have been possible with our previous CDN partners.
I will say that it was an implementation bug which doesn't warrant the swapping out of entire data storage layers.
In 64ms.
Without caching.
Using PHP.
It's definitely doable, but you'll need to heavily fine-tune your queries. My first one was at over 2 hours for the same.
Misrepresentation; then it's not actually over 120 million rows. You're basically encoding which subset to actually search in the query, rather than building a proper overall schema that trivializes queries.
I mean stuff like using aggregation functions to let the database build a bitset out of them, and reading from every row a value into that bitset.
I mean stuff like CLUSTER table ON (pixel_y_index, pixel, x_index) to change the order in which they are stored.
Do those two optimizations alone and you improve speed massively.
Like people decide to use them from the get-go and then come up with a justification.
You're completely ignoring, or completely oblivious of the fact, that the entire 1000x1000 grid must be provided to every connected client. You're not going to read out one million aggregated rows by most recent timestamp per cell, from a billion rows of history, in a scalable amount of time.
Please post your GitHub link that proves your solution as superior, or even viable. Make sure it includes database triggers, for which you don't explain how they would help scale the app whatsoever. Are you going to have a denormalized table containing each of the one million cells' most recent rows? All you are doing is eliminating a GROUP BY on the indexed cell+timestamp columns. It's still a million rows returned per query. Please explain how that scales. Eagerly awaiting your proven solution that defies common sense scaling logic.
Myself, I just get so frustrated about the idiocy.
It's still there: https://www.reddit.com/r/place/
They have may have fixed one or two scrolling issues since, but the main issue is that if you
a) press LMB
b) move the mouse, and move outside of the pixelized area
c) release LMB
.. it does weird things.
The canvas trick with typed arrays is brilliant. Using requestAnimationFrame is what any front end dev who knows perf would do. Its like the front end version of cdn with 1 second time out trick.
Also using a layer of library whose code you don't understand in a perf critical application is quite risky. Its better to stick closer to the native browser APIs which you are familiar with. We once had to throw away a library and rewrite code from scratch because their assumptions failed when pushed to the limits. The rewritten code was 100x smaller in size and 10x more performant.
I would have loved to work on something like this but it sucks reddit doesn't have any presence in Seattle.
They used tools they knew and knew how to scale. Almost always the tool you know is better than the perfect tool. They know redis and websockets, and they made it work. Beats using some engine know one on staff has ever touched?