Blog post here: https://karmanivero.us/win-link-router/
41 karma · joined September 22, 2019
By "regular ole" I presume you mean some flavor of RDBMS. Those have significant issues at scale that the newfangled platforms don't... but as you see there's a price to be paid at design time.
If I do my job right, you get to have your cake & eat it too!
Recall that, in DDB, your index has two parts: hash & range key. If you want to have many entities in the same table, then you need a way of distinguishing between different entities, and a way of locating an individual record. In your primary index, those account for your hash and range keys, respectively: the hash key is your entity differentiator, and the range key is your entity id (which may come from a different record property from one entity to another). If you follow the development of the article, you’ll see how this plays out with variously constructed keys across different indexes.
Now, forget EM sharding for a minute and let DDB manage your sharding. Say you launch your application with little data and a single shard. Over time your data scales & spills over onto additional shards. When you perform a search, DDB has no way of knowing which shards are relevant so it has to search ALL of them.
But from the application side, your data scaled over TIME. Therefore, if you know which shards were created when, you could limit a time-based search only to the shards that are relevant to the search parameters. And a LOT of searches involve a time window.
Within the context of EM, when I say a “shard”, I am talking about a unique hash key value like `user!1F`, where `user` is the entity type and `1F` is the shard key. These may or may not map to physical DDB shards, and the good news is that you don’t NEED to care… DDB will flex if you don’t.
EM has a lot of features that greatly streamline the dev experience when operating against a DDB table with a multi-entity data model. You don’t HAVE to use the sharding feature… it’s literally just a config item, everything else happens behind the scenes. But when you DO use it, EM splits a search across sharded data into MANY parallel searches, one per shard, then assembles the returns into a coherent result with a “page key” that is actually a compressed map of ALL the underlying page keys. You don’t have to care about THAT, either… just pass the compressed string back to EM and it will rehydrate the page keys & perform the next set of searches.
So you get to choose your own adventure… you can run every entity on a single “shard” or run in parallel. I’d just keep an eye out for any drop in performance at scale and add a shard bump when I see it.
Also worth noting: EM is actually platform-agnostic. There is a companion repo that contains the DDB-specific client. This is still a bit in flux btw so be kind lol. Anyway the point is that other platforms that don’t have AWS’ resource footprint may not handle sharding as well, and EM will be able to render effectively the same result.
Hope that answers your question!
P.S. Worth noting: in addition to searching across multiple SHARDS, an EM query can also search across multiple INDEXES. Say you want to query on “name” and you want to query both your firstName and lastName indexes with the same “name” value. With EM, this is a SINGLE query that returns a combined, paged, deduped, sorted result set. Handy.
I've actually been using the JS version of EM in production for over a year. It's been working flawlessly.
The TS version is a complete rewrite that factors in a BUNCH of lessons learned and is completely--maybe obsessively lol--type-safe. The query builder got a LOT of attention, and the fluent API reduces even complex queries to a super-compact, declarative coding experience.
I'm pushing a big update tonight and will then resume my focus on the demo & docs, basically the companion stuff to this one. Should be ready for use in a couple of weeks.
Thanks for the interest, it really means a lot to me!
Entity Manager is a framework for defining, managing, and most importantly QUERYING an entity model with DynamoDB. It's actually platform-generic, so the DynamoDB-specific machinery is implemented at https://github.com/karmaniverous/entity-client-dynamodb
Entity Manager's most important feature is that it permits a simple, scheduled partition sharding configuration and then transparent, multi-index querying of data across shards with a very compact, fluent query API.
This resolves the biggest challenge of using DynamoDB at scale, which is that very large data sets MUST be sharded, and a given query can ONLY operate against a single shard. If you're querying on the basis of a related record, you won't know which shard your results will be on so you must query ALL shards.
Entity Manager reduces this to an effortless operation: once you've defined your sharding strategy for a given entity, you can forget sharding is even a thing.
For some more color on Entity Manager within the context of SQL vs NoSQL databases, please review this (much shorter!) article: https://karmanivero.us/projects/entity-manager/sql-vs-nosql
ELECTION WINNER Trump: 64% Harris: 36%
POPULAR VOTE WINNER Harris: 64% Trump: 37%
Looks like it's gonna be a bumpy ride lol.
I have come to the conclusion that shared decks don't work, at least not for me. The reason is that actually assembling the deck has turned out to be almost as valuable as practising with it.
I've got well over 1,000 cards in my Indo deck right now. I had to go find every translation and decide whether synonyms should be on the same card. That activity produces its own kind of learning.
There’s one aspect of all this I didn’t mention in my post but it’s germane here: this is a group of men and women whom I LOVE in a way you’d have to be another veteran or maybe a cop or a fireman to appreciate.
We’ve bled together. We’ve buried the same friends. So I don’t have to work very hard to muster up the emo stuff at the very top of Maslow’s pyramid. I FEEL it. And so the audience does too.
That’s probably pretty hard to fabricate if it isn’t already there. But if it IS there, it’s a pretty powerful foundation for a host relationship.
FWIW, I wanted to do that BECAUSE I had the sense that the format we were already using would work well for a combined online/IRL audience. Basically treat the whole IRL room as one Zoom caller, with the camera focused on whom ever had the mic.
If I were to try this again with an IRL component, I’d definitely use this framework as my starting point.
We solved that problem by giving every intro a VERY tight time constraint, 2 min I think. I’d make ‘em stay inside those lines, but then I would also find an interesting point in their speil & ask a leading question to draw them out for another minute.
So nobody drones on and you maintain control of the format, but everybody still feels like they got to say their piece in front of an interested audience.
Oh also: there was often a previous attendee on the call with a similar or complementary story. So I would often make that intro right there and use the opportunity to get a quick update from that person.
It made for a really organic flow and the sound bites were short enough that nobody got to be boring for long. Not even me. :)
I’m the author. This is the BEST compliment I have received on any platform. Thank you! :)
In that SPECIFIC use case remember: it was a Zoom call. So with a good headset and a quiet room I was already top 20%, audio-wise.
Once you get that far, what you have to say and how you say it are WAY more important than whether you can rock a little extra baritone.