200 karma · joined June 29, 2020
i think this is where spark shuffling comes in? but how does it work here.
https://duckdb.org/docs/stable/guides/performance/how_to_tun...
"he was an early designer and engineering manager at Palantir (NYSE:PLTR), where he designed the company logo"
I just checked on their GitHub and it says "Additionally, it provides optional and fully encrypted synchronisation of your history between machines, via an Atuin server."
So you trust all of your shell commands to be stored on a server that you don't control?
Maybe I'm missing something here.
Who’s the “I”?
“Designed by Cozy Ventures” … “We're a company that creates advanced digital solutions for early-stage startups.”
provision spark on emr or duckdb on beefy ec2 -> run sqlmesh -> wipe resources.
i'm still in MVP phase of revamping my company's current data platform, so maybe there are better alternatives -- which i'd love to hear about.