crorella

Missed some basic info about distributed systems: Partitioning, bucketing, sorting, bloom filters, the fact that you normally create specific logic to backfill rather than run the same 'daily' process N times.

If you come from Dataswarm/Airflow then you can use JINJA for the templating, UPSERTS are better when you want to save space by not keeping the same data for each snapshot you take day after day (on multi terabyte tables the space savings are considerable).

More topics that are usually prevalent in distributed systems: - Join order - Type optimization for smaller tables (ones you can fit in memory to do broadcast joins) - Field codification for extra compression/smaller intermediate shuffles when processing data. - Field sorting for extra compression - Caching tables that participate in several parts of the process. - Rolling accumulating pre-aggregates for multi-day processing - T-Digest for performant aggs / cubes - HLL for performant aggs - etc

why the DB? Why not the client itself? How much data was wasted back and forth between the client and the backend? how many CPU cicles were spent in that poorly optimized SQL statement?

#wut MLM

Telemetry per se is not a bad thing when you make it with privacy at the center of it. If you offer users the chance to opt-in/opt-out of logging and then REALLY respect that decision by first) not generating data when the user does not want to and second) not using the data for purposes the user does not sign up for (in downstream, analytics mostly)

then telemetry can actually be beneficial to keep improving your product and features.

> "Weighing just a teaspoon of salt more" why not just say the glasses are 5 grams more?

I like that there is a light signaling the glasses are capturing photo/video, although it might not be noticeable in some cases, just like other devices

Cool to see the pictures/videos don't go directly to FB.

Hello Peter, thank you for offering this space. Is it possible to get a work permit in the US as a self employed? What are the steps? Are there limitations?

Thank you

“What if we could give customers a button. They’d press it at the end of the year and it would automagically file their taxes for them.”

This is exactly how it works in Chile, you pretty much check the numbers and make sure they match with your figures and done. For free.

Because they wanted the price to go down

This is a major issue that causes major frustrations to the people working on such companies.

Not only that,

- How do you make sure the users are not writing badly optimized tables/pipelines that end up consuming too many resources.

- How do you facilitate data discoverability so they don't end up creating a new table where 90%+ of the data is already present in another already existing one?

- How do you make sure they are mindful with the way the model the tables such there are not too many files due to bad partition/bucketing, compression is leveraged and good datatypes are picked?

This is a great idea! I love it.

Better to call them "non spy TV" sounds more accurate.

11 mbps in the middle of a rural location in the heart of Chile ? Count me in!

I thought it was about improving education so professionals formed in the US could be more competitive.

I am in a similar situation. I ended up investing overseas on a good real state deal with high yields. The tax side of things was simplified by the type of investment (debt investment). I think the pandemic created or accelerated a bunch of business opportunities, some of them out of the US.

missing ludens.cl

Really nice design, in the future I see a market for "privacy tools that also happens to ..." kind of software, for example a Privacy tool that also happens to browse websites, or a Privacy tool that also happens to help you to send messages (like Signal); in summary, privacy aware services.

What bout the tracking of phones and laptops? You'd have to have your laptop with a remote desktop software or something like that.. or to spoof the GPS and make sure you connect to an anon wifi that is not sharing its position.

All of them seem valid errors. I’m don’t get the idea of the post lol.

Auto tuning of table’s structure (partition, bucketing) based on access patterns

I can do both interchangeably and always assumed everyone could do the same!

Why not intercept the different system calls and randomize their outputs?

I was thinking the same!

I see, thanks. I was hoping for this to be more automatic.

You mean DataClasses?

Airflow just cover authoring, monitoring and scheduling. I don't recall it covered caching, lineage nor resource allocation/reuse.

"use cached versions of pre-computed artifacts" This is really nice. I wonder if this also covers partial pre-computations, for example when the same subquery is reused across several pipelines.

Is airbnb still a thing ?

Qualify ?

Buildings mostly, I guess they will move harmonically with the ripples as wood is pretty flexible but I still freak out thinking they could fall over my head lol