4,101 karma · joined August 20, 2012
https://aidnn.ai
My family lives outside of a tier 2 city border, in what used to be farmland in the 90s.
They have Asianet FTTH at 1Gbps, but most of the video/streaming traffic ends at the CDN hosts in the same city.
That CDN push to the edge is why Hotstar is faster to load there - the latency on seeks isn't going around the planet.
Similar to how git can serve a repo from a simple http server with no git installed on that (git update-server-info).
The frozen part is what iceberg promised in the beginning, away from Hive's mutable metastore.
Point to a manifest file + parquet/orc & all you need to query it is S3 API calls (there is no metadata/table server, the server is the client).
> Creating and publishing a Frozen DuckLake with about 11 billion rows, stored in 4,030 S3-based Parquet files took about 22 minutes on my MacBook
Hard to pin down how much of it is CPU and how much is IO from s3, but doing something like HLL over all the columns + rows is pretty heavy on the CPU.
There's a whole skit in the vein of "What have the Romans ever done for us?" about ZeroMQ[1] which has probably lost to the search index now.
As someone who has held a socket wrench before, fought tcp_cork and dsack, Websockets isn't a bad abstraction to be on top of, especially if you are intending to throw TLS in there anyway.
Low level sockets is like assembly, you can use it but it is a whole box of complexity (you might use it completely raw sometimes like a tickle ack in the ctdb[2] implementation).
That honestly might be an absorption issue, not an intake issue - you can hit aerobic limits enough for your body to skip digesting stuff & just shove protein directly out of the stomach instead of bothering to break it down.
My experience with this was a brief high altitude climb above 5km in the sky, where eating eggs & ramen stopped working and only glucon-d kept me out of it.
The way I like to think of it is that the fat in your body can be eaten or drank, but needs to be breathed out as CO2 to leave it.
The rate at which you can put it in and the rate of letting it go are completely different.
The "naturally sortable" is a good thing for postgres and for most people who want to use UUID, because there is no sorted distribution buckets where the last bucket always grows when inserting.
I want to see something like HBase or S3 paths when UUIDv7 gets used.
The REPL is what matters - also while being performant.
Someone asks you a question, you write something, you run it and say an answer, the next question is asked etc.
I've seen these tools be invaluable in that model, over "write software, compile and run a thousand times" problems which C/Fortran lives in.
This is, but only for someone who wants to do JIT work without writing assembly code, but can read assembly code back into C (or can automate that part).
Instead of doing all manual register allocations in the JIT, you get to fill in the blanks with the actual inputs after a more (maybe) diligent compiler has allocated the registers, pushed them and all that.
There's a similar set of implementation techniques in Apache Impala, where the JIT only invokes the library functions when generating JIT code, instead of writing inline JIT operations, so that they can rely on shorter compile times for the JIT and deeper optimization passes for the called functions.
This error can happen if there's an AAAA record, but it contains the ipv4 address packed inside a ipv6 mask.
If the AAAA record says ::ffff:10.0.0.105, then you can either fix DNS or do what's in the blog, which should stop checking for quad A records.
Technically competent doesn't always mean empathetic.
The decisions can sometime look like the xkcd cartoon about scientists[1].
[1] - https://xkcd.com/242/
> Utterly useless to people who have no clue what they're doing
> the same way that a fighter jet is not useless
AI is currently like a bicycle, while we were all running hills before.
There's a skill barrier and getting less complicated each week.
The marketing goal is to say "Push the pedal and it goes!" like it was a car on a highway, but it is a bicycle, you have to keep pedaling.
The effect on the skilled-in-something-else folks is where this is making a difference.
If you were thinking of running, the goal was to strengthen your tendons to handle the pavement. And a 2hr marathon pace is almost impossible to do.
Like a bicycle makes a <2hr marathon distance "easy" for someone who does competitive rowing, while remaining impossible for those who have been training to do foot races forever.
Because the bicycle moves the problem from unsprung weights and energy recovery into a VO2 max problem, also into a novel aerodynamics problem.
And if you need to walk a rock garden, now you need to lug the bike too with you. It is not without its costs.
This AI thing is a bicycle for the mind, but a lot of people go only downhill and with no brakes.
You can do that better if you don't use a random number for the hash, instead flip a coin (well, check a bit of the hash of a hash), to make sure hash expansion works well.
This trick means that when you go from N -> N+1, all the keys move to the N+1 bucket instead of being rearranged across all of them.
I've seen this two decades ago and after seeing your comment, felt like getting Claude to recreate what I remembered from back then & write a fake paper [1] out of it.
See the MSB bit in the implementation.
That said, consistent hashes can split ranges by traffic not popularity, so back when I worked in this, the Membase protocol used capacity & traffic load to split the virtual buckets across real machines.
Hot partition rebalancing is hard with a fixed algorithm.
[1] - https://github.com/t3rmin4t0r/magic-partitioning/blob/main/M...
No, this is a compatibility layer for future encoding changes.
For example, ORCv2 has never shipped because we tried to bundle all the new features into a new format version, ship all the writers with the features disabled, then ship all the readers with support and then finally flip the writers to write the new format.
Specifically, there was a new flipped bit version of float encoding which sent the exponent, mantissa and sign as integers for maximum compression - this would've been so much easier to ship if I could ship a wasm shim with the new file and skip the year+ wait for all readers to support it.
We'd have made progress with the format, but we'd also be able to deprecate a reader impl in code without losing compatibility if the older files carried their own information.
Today, something like Spark's variant type would benefit from this - the sub-columnarization that does would be so much easier to ship as bytecode instead of as an interpreter that contains support for all possible recombinations from split up columns.
PS: having spent a lot of nights tweaking tpc-h with ORC and fixing OOMs in the writer, it warms my heart to see it sort of hold up those bits in the benchmark
I've been writing python from the last century and this year is the first time I'm writing production quality python code, everything up to this point has been first cut prototypes or utility scripts.
The real reason why it has stuck to me while others came and went is because of the REPL-first attitude.
A question like
>>> 0.2 + 0.1 > 0.3
True
The REPL isn't just for the code you typed out, it does allow you to import and run your lib functions locally to verify a question you have.
It is not without its craziness with decorators, fancy inheritance[1] or operator precedence[2], but you don't have to use it if you don't want to.
[1] - __subclasshook__ is crazy, right?
[2] - you can abuse __ror__ like this https://notmysock.org/blog/hacks/pypes
The lack of executive function is overlapping, but this particular post might be more of an ADHD simulator.
The very first "Follow the morning routine" or not is where this veers off my experience of the spectrum.
"Changing plans because of situations internal or external" is hard.
The option should've been "Spend 20 minutes making eggs again, because the yolks weren't the right kind of runny", miss the train, take a cab to work, but tell the driver that you've now got a system for eggs which you didn't have today (yeah, fun fact, the recipe was off because they don't refrigerate them over in France).
I feel like what I felt with adaptive cruise control.
Instead of watching my speed, I was watching traffic flow, watching cars way up ahead instead.
The syntax part of my brain is turned off, but the "data flow" part is 100% on when reading the code instead.
Didn't OP end by picking Postgres anyway?
It's the right answer even for a naive developer, perhaps even more so for a naive one.
At the end of the post it even says
>> Having an interface for your cache so you can easily switch out the underlying store is definitely something I’ll keep doing
The curiosity hasn't disappeared from the culture, but it might not be brought in to a workplace anymore.
I think a lot of us have stopped bringing the tinkerer itch to work.
Outside of the workplace, there's an entire parade of tinkering by folks who at best post it on Youtube, not here (I watch "Stuff Made Here" for the code).
Of all the events of the past decade, the worst hit to the tinkering visibility has been Github making personal repos private by default.
Mostly the folks who were like me still have pet projects, most of them will share their code but only if you ask because it is "Well, not as nice as it should be".
I've got hundreds of repos in my github, but there's a sharp fall-off in what's public (there's ~113 public and 180 private) right when that happened and I'm sure I'm not the only one.
The tinkering is more active than ever now with vibe coding tools, where I can draw an svg out and then ask it to "I want to use manim.py to animate this" to get something which is a neat presentation tool to describe how data moves between systems.
But is it worth showing you when all the fun there was in making?
What if all I am likely to get "So what?" as the only response. Wouldn't that it make it less fun?
Yes, it picked a valid result and gave up.
I got a similar nonsensical result in my run-ins[1] in with SAT solvers too, until I added a lowerbound=0.
The "real life" follow-up question in interviews was how to minimize the total number of intermediate rows for a series of joins within in a cost-based optimizer.
[1] - https://gist.github.com/t3rmin4t0r/44d8e09e17495d1c24908fc0f...
Not how I read when I read on paper & this bothers me on a phone a lot.
I'm not functionally dyslexic, but all the words pop out at the same time, you could say I read verbs, then tense, nouns after and then I read adjectives in a sentence.
But English is also my second language, so my first language's grammar probably fits closer.
Oddly enough when I first encountered lisp, the nested pile of map, fold and filter was easier because of the well trained habit of restructuring after reading verbs.
The negative label is the old world pulling the new one back, it rarely sticks.
I'm old enough to remember the folks saying "We used to have the paint the background blue" and "All music composers need to play an instrument" (or turn into a symbol).
Even if you have experienced the joy, it might not always be rewarded & the disillusionment burnout is always a risk.
The most bitter folks I've worked with are the ones who started with a lot of passion, but got turned around.
Not to mention that this Passion comes in many flavours.
I would not put a label on my experience, but "autistic joy" is a good comparison to what drives passion in my work - for my partner it comes from the final unveil and other eyes landing on their work.
I only realized this when reading Andre Agassi's book and being stuck in close proximity during the pandemic (and to "bear witness" to provide joy).
From recent experience, 95% of changes are good and are done in 15 minutes.
5% of changes are made, but break things because the API might have documentation, but your code probably doesn't document "Why I use this here" and instead has "What I do here" in bits.
In hindsight it was an overall positive experience, but if you'd asked me at the end of the first day, I'd have been very annoyed.
I thought this would take me from Mon-Fri if I was asked to estimate, but it took me till Wed afternoon.
But half a day in I thought I was 95% done, but then it took me 2+ more days to close that 5% of hidden issues.
And that's because the test-suite was catching enough class of issues to go find them everywhere.
I was told that this was the polite honk triplet - the two honk call and one honk response.
"honk honk" / "honk"
"horn ok" / "please"
"Everything worth doing is worth doing badly"
And as a corollary, every complex system that works came from a simple system that works.
I learned this in programming, but now I apply it on everything from motorcycle maintenance, home appliance repair to parenting.
--
Often the easier way to fix a complex system is to pretend that it could be simpler and then reintroduce the complexity-inducing requirements.
I had a professor who taught debugging as a whole another skill from programming and used to say "Most of programming is starting from an empty editor and debugging until your code works".
The debugging "lab" in Java course (in the year 2000) was one of my transformational after-school classes - where I got a java program which fits within 2-3 pages of print code with a bug and was told to go find it in print for ~20 minutes, then given 40 minutes with a debugger instead.
No question, the I can't count the number of times I've sent the "Baby Race" to other parents who ask me "When did your kid start to talk?"
We watch the Bluey bike whenever the kids say "I can't do it!" or the Dad can't draw episode.
Or the curry quest, when I had to travel for work (after all the pandemic "no work travel").
Bluey's dad is an inspiration, like the puppets episode is so funny & I keep saying "Bauxite" in random times without context.
Most of my middle management experiences has been between Office Space and Better off Ted.
One with "Don't care" as an answer and the other says "Care more" as its.
Those are the two extremes of the genre.
Maybe it was not when printed in the first edition, but at least it was the same content shown to hundreds of people rather than something uniquely crafted for you.
The many eyes looking at it will catch it and course correct, while the LLM output does not get the benefit of the error correction algorithm because someone who knows the answer probably won't ask and check it.
I feel this way about reading maps vs following GPS navigation, the fact that Google asked me to take an exit here as a short-cut feels like it might trying to solve the Braess' paradox in real time.
I wonder if this route was made for me to avoid my car adding to some congestion somewhere and whether if that actually benefits me or just the people already stuck in that road.
They are tiny and often hit with a huge number of IOPS.
Ext4 supported external journals and moving it to a single SSD for a large number of otherwise slow SMR disks has worked great in the past.
However, when you hit a failure that SSD becomes a single root cause of data loss from several disks when losing that SSD (unlike a read cache).
Where I was working that didn't matter as I was mostly working with HDFS which both likes a JBOD layout of several disks instead of RAID (no battery backed write caches), tolerant to a single node failing completely and having a ton more metadata operations thanks to writing a single large file as many fixed-size files named blk_<something> with a lot of directories containing thousands of files.
SSDs were expensive then, but it's been a decade of getting cheaper from that.
I have a bunch of scripts in my git-hooks which have dependencies which I don't want in my main venv.
#!/usr/bin/env -S uv run --script --python 3.13
This single feature meant that I could use the dependencies without making its own venv, but just include "brew install uv" as instructions to the devs.
This is the core part of what's changing - the most important people around me used to be "People who know how".
We're slowly shifting to "Knowing what you want" is beating the Know-how.
People without any know-how are able to experiment because they know what they want and can keep saying "No, that's not what I want" to a system which will listen to them for without complaining supplying the know-how.
From my perspective, my decades of accumulating know-how is entirely pointless and wiped away in the last 2 years.
Adapt or fall behind, there's no way to ignore AI and hope it passes by without a ripple.