745 karma · joined June 12, 2019
This is the perfect exit for precisely the problem you're pointing at: if the data is open it doesn't need to be massively harvested to make this work, and you can run the text-to-speech engine entirely offline on the device.
If what you want is just to add some uniform, seemingly random behavior to something (e.g. a game, generate some noise, whatever), I've used the Wolfenstein3D's FizzleFade effect algorithm [0] in the past.
It's boringly easy and works like a charm.
I'd suggest that OP use something like https://aws.amazon.com/fsx/lustre/ if they really need to mount S3 as a filesystem (do read the documentation regarding distributed consistency though), but other than that using AWS solutions in general (Athena, Presto on EMR, Spark on EMR) will tend to use specialized S3 committers such as https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spar... and are quite good at minimizing this problem.
Other than that, read the S3 docs for this behavior.
Huh, this is a weird one for me.
I believe there is a Terminal command to shorten the animation time which I've used, but other than that it would take more time for me to cycle through so many windows to find the one I want, than to just lay out task specific desktops and switch between them with Ctrl+Left/Right.
Not only that but I already am at a limit where I have so many windows in one of those (several Firefox windows, each with many tabs of stuff to read) where all of the windows become sluggish unless I move the one I'm trying to use to a less crowded desktop.
Did you do something to avoid this slowdown? How do you efficiently move between all those applications in a single desktop?
Sure there are good things about it, and many great enhancements, but the larger mechanic of the protocol (the most important part really) has been significantly worsened. I'd rate them as follows:
HTTP3 > HTTP1.1 > HTTP2
QUIC is an amazing protocol, I have no complaints about it, and I'm very happy they decided to go with it for HTTP3. However the decision to make HTTP2 traffic go all through a single TCP socket is horrible, and makes the protocol very brittle under even the slightest network decay or packet loss. Any level of head-of-line blocking severely degrades the entire stream, even for unrelated requests.
Sure it CAN work better than HTTP1.1 under ideal network conditions, but any network degradation is severely amplified, to a point where even for traffic within a datacenter can amplify network distruption and cause an outage.
HTTP3 however is a refinement on those ideas, and gets pretty much everything right afaik.
iOS13 had for the past two weeks fixed one of the biggest annoyances in Apple's horrible keyboard UX: For people like me who type in multiple languages, the keyboard switching button cycles between "English", "Portuguese", and "Emojis". iOS13 FINALLY put a dedicated emoji button in there and after 2 weeks living with it on the iPhone, iPadOS apparently completely forgot to port this over and the feature has been taken away from me.
This is just one example out of probably a crap-ton of things that they forgot to bring along from iOS. This crap happens all the time with Database vendors too, when they try to keep a separate closed-source Enterprise version. I get that feature flags are annoying, but thinking you can manually keep two different OS codebases synced and consistent is delusional.
_On a side note, the rest of the keyboard is still a mess._
No number row (not even as an option in Settings), no hold for punctuation, but worst of all (especially if you're coming out of Gboard to try their keyboard again), Slide-to-Type is just horrible. The accuracy is SO bad, I tried typing "hello world", and not only it just couldn't understand that I was trying to type "world", it wouldn't even suggest it as an alternative, the best I got was it suggested "world's" once, and if you tap it and try to delete the "'s" it just deletes the entire word.
Not to mention that since I use two languages, it even tried portuguese words before it tried "world", even though I had the english keyboard selected (and there is no way to disable multilingual typing like there is on Gboard, even IF it worked decently). Also not being able to turn off Memojis on the Emoji keyboard... why?
Intel is infamous for severely downclocking the processor for these and other AVX/SSE family instructions, to a point where sometimes using them makes the program slower than it would be otherwise, especially if you're constantly provoking frequency switches between them and regular instructions.
AMD might not have implemented AVX512 specifically yet (there's nothing legally keeping them from doing so however, they have patent sharing agreements with Intel regarding the entire x86/x64 ISA and extensions), but what they currently DO have is all common SIMD extensions implemented (up to SSE4 and AVX2 if I'm not mistaken) without incurring any frequency penalties on clock speeds for using them.
I can live without AVX512 for now, even though I'd be happier to have it. But I would really rather not have it if it came out in the same crap implementation that Intel has.
For a long time, the only reason ElasticSearch was not the ugly duckling of distributed systems was because MongoDB existed. It does not have a good history at all. There was a time that Elastic even claimed that it beat the CAP theorem, which always makes me chuckle when I'm reminded of it, and historically they also have been quite vague about the distributed properties (and shortcomings) of ES.
It works fine as a secondary datastore when you would benefit from all of the implemented search functions, but I wouldn't trust it for consistency (so I guess in a way it is a replacement for mongo \s)
Not only that but having operated ES clusters before, it's a tricky thing to manage, and depending on your workload they can be very resource hungry.
PostgreSQL is very solid though, and will scale well for the majority of people. I highly recommend this over ES, not to mention it does have some search features too, you should check them out.
You'll need to manage some stuff yourself, and assemble your own dashboards and stuff, so there will be some labor involved. That being said, I doubt it will be more painful than managing an ELK stack: there are just too many ways you can destabilize a cluster with it.
ClickHouse clusters from my experience are ridiculously scalable, fast, and stable. There are several other accounts to back that up, and a good case study is Cloudflare, which uses it to store and query all of their logs and metrics from all data centers (that's quite a few PB of data).
There are some projects on GitHub you can use to get inspired, but what you need is pretty much a ClickHouse cluster, Grafana, and a Log Shipper.
Some things I'd love to read about on your blog
1 - I'd love to read more on the local information retrieval strategies you use for Badger, especially if you have use cases that are comparable to sequential reads (something that Kafka does, but with a structure optimized for magnetic drives) or ISAM style indexation (traditional SQL databases), and how you leverage the metadata byte to speed things up (this has been touched on only superficially on the blog afaik) along with other specialized features/operations Badger supports;
2 - Some insights on how Dgraph's data model influenced the design of Badger, and what parts you decided to explicitly generalize for other use cases, and what use case constraints have you purposefully decided on (aside from the obvious ones like SSD optimized, etc);
3 - Any progress you might have made regarding why some queries in Badger take longer than in RocksDB. That so far has been the biggest cliffhanger in the tech blog, I've been waiting for the sequel to that one for some time now :)
It's difficult to find good resources where people go in depth on these database engineering topics, but your blog is a very good one and a joy to read. Thanks for not simplifying things and keeping it very technical!
Some questions relevant to the post and Dgraph itself
4 - When I (and I assume many others) see "Enterprise Features" on databases that is kind of a turn off. That usually correlates to keeping separate codebases that are being manually kept in sync (which has problems on it's own), and it sometimes feels like it's gatekeeping some pretty crucial features (e.g. better backups) which make people either accept a more vulnerable architectural situation, or makes it unsuitable for a few use cases for lack of such features. I do understand and respect the need to make money to keep Dgraph being awesome, but have you considered something like the CockroachDB approach (selling support + mandating a license to sell the DB as a managed service)? If you did, why did you turn it down and what would make you reconsider it? On a second note from a maintenance perspective, how are you planning to handle the "feature flagging" and "repo syncing" of community and enterprise editions?
5 - As someone who's considered using Dgraph in the past, something that I found interesting is that graph databases in some use cases alleviate some of the same data modelling woes that Datomic does[2][3]. How would you diff them apart and what would you say they differently excel at?
6 - Graph databases sit in a spectrum of use cases, where some people want a system to work similar to a traditional RDBMS but with more flexible/powerful data modelling capabilities, and some people might want to load up big graph-heavy datasets for analytics purposes (i.e. loading a giant product catalog to map product relationships and either do a bunch of batch queries to feed a secondary dataset or keep it live for realtime queries on a recommender system). How would you place Dgraph and it's strengths/weaknesses in this spectrum? What alternative solutions would you recommend for the places where Dgraph would be unsuited for?
Sorry for writing too many questions (which will probably take up a lot of time to answer) but many of these things (especially on the second half) are not explicitly said anywhere, and are valuable knowledge for making an informed decision on a database product.
[1] https://github.com/dgraph-io/badger
[2] https://augustl.com/blog/2018/datomic_look_at_all_the_things...
[3] https://augustl.com/blog/2018/datomic_look_at_all_the_things...
One question though, doesn't CockroachDB also have a holdout period of 250ms (default) when reading across shards, kind of similar to Spanner's 7ms holdout on writes? I remember reading something vaguely similar to that somewhere.
I think CockroachDB will be amazing when cloud providers start offering APIs for High-Resolution Clocks as a service, of even better, if I got to opt-in to the system clock being synced to highres, much like in GCP you can upgrade your instance's network to premium tier on the fly.
Everytime I look at git's documentation GPG seems very entrenched in there, to a point that for things that matter I'd use signify on the side.
Is there a better way?
Currently Git seems to be very much integrated with GnuPG and the same goes for GitHub's UX sprinkles over the signing feature. That is what I'd like a decent alternative to.
I considered using OpenBSD's signify but it does not integrate as nicely as GnuPG signing so I'd basically be rolling my own mechanism (which is fine I guess, but feels subpar)
Does anyone know good alternatives in this space?
I don't think we should use boring tech for sake of it. I think we should use boring ideas, and sometimes boring ideas come in new packaging that brings better tooling and community support to the table.
Single server architecture, relational databases, are an idea. Postgres is an implementation of such idea. What we understand about the failure modes of something like Postgres, are not actually specific to Postgres, but to the ideas behind it dating back to Codd himself, and most other databases that came along the way.
Same thing can be said from DBs that derived from the BigTable paper (Cassandra, Scylla, etc). Of course they have specific operational differences, but they have at least twice as much common ground. The way we understand the failure modes of these projects, is by looking at the ideas behind it (Gossip, Paxos, AP, SSTables, LSM-Trees). We understand things by stripping away their specifics, and seeing how their core ideas interact with each other. The specifics of each solution are merely tie breakers.
This is why I would never call something like Clojure or Elixir "shiny new hype technology".
Clojure is nothing but a modern implementation, backed by an excellent community, of mature ideas that have stood the test of time. Lisp itself is as old as the dinosaurs, and functional programming is not as new as people think. It has just been "rediscovered" recently, once the hype/aversion passed and we were able to look at it with sober eyes.
Elixir is also a modern implementation, with better tooling/community, of things that have been in BEAM for 20+ years. Apache's Samza/Storm for stateful stream processing are reimplementations of things the Erlang community has been able to do with Mnesia since at least 1999. It's also something we sometimes circumvent by using things like Redis to deal with stateful computations, and Google recently seems to have realized this and published a recent paper, trying to go back to that old idea[1].
In short, I think it's a really good thing that we have new tech that tries to make the old ideas better, and I generally feel pretty safe experimenting with those. You just have to learn to recognize these values, and adequately investigate solution candidates before jumping into what could very well be just hyped up crap, because sometimes, it isn't.
[1] https://ai.google/research/pubs/pub48030 discussion https://news.ycombinator.com/item?id=19823022