Wow, that's amazing and a bit funny - so heartwarming to me that it's a success!
9,188 karma · joined July 27, 2009
mali at akmanalp dot com
Still, the rabbithole goes much deeper. I highly recommend The Turkish Cookbook by Musa Dağdeviren, a food anthropologist of sorts who spent his formative years traveling the country town by town and collecting recipes that were often condemned to be forgotten due to the modernization and globalization of food and food culture: often they'd be passed down generationally and not written down, rely on manual and work intensive or just plain unusual methods, have ingredients that are easy to acquire in a rural household but hard to acquire in a grocery store, have flavors that would be considered unusual by the global standards of the 70s-90s, etc. It's a truly massive and amazing collection.
It isn’t that I’m perfect, or infallible, or even immune to frustration and the occasional bout of snark. It doesn’t mean that I’m a “better” person. It also doesn’t mean that I have to accept every viewpoint as true or equally likely. Nor does it mean that I can’t criticize. Nor that I can’t take a side on a considered opinion. It doesn’t mean that this always leads to perfect results or that I’m always successful. It doesn’t mean that it’s a perfect heuristic. It’s only one more tool in a toolbox.
I think it is undervalued - and sometimes angrily pushed back against - in this community, maybe because it gets dismissed as “feelings” (and not “reason”). The reason I think this is that I've encountered here the “experienced expert with healthy self doubt versus passionate amateur with little self awareness” dynamic in action over and over again.
There’s an ongoing, half century current in the field trying to examine whether reasoning may often involve a heuristic-based, adaptive process based on incomplete and imperfect information, and less so something resembling a pure application of formal logic.
In this universe, cognitive biases extend more clearly to things like which facts you consider as possible at all in the first place and by which methods you weight them as more or less likely or influential, as opposed to merely fallacies of formal logic that we’re most often exposed to. So, the thinking goes, it can help to try to fill some gaps in this area by trying to increase the breadth of what you’re exposing yourself to and allowing yourself to take into account.
In my experience this can be a frustrating and exhausting concept to contend with as a person. You certainly can tu quoque yourself - or me, like you and others in this thread have - into oblivion. After all, what business do I have talking about cognitive biases if I've been shown to be subject the very same, and am not an expert in it? I don't know where the line is but I'm not certain it makes the concept less helpful outright. At the very least I don't think I'm claiming something that'd be considered very controversial as far as I'm aware.
I'll admit I also had a chuckle at the expense of the parent post here (I'm sorry - the rest of this post isn't meant to pick on you or describe you specifically). I think this there is a lesson in here somewhere on the intellectual confidence that comes with being good at one thing, and the silliness that can ensue when it might not translate elsewhere. Not in even necessarily in a malicious or foolish way, but in a "it can catch you even when you think you're accounting for it" kind of way.
If not empathy for the sake of moral reasons or even good vibes, I wish that we could at least convince the "I read the abstract of 3 papers / I read a gwern post about this / I accomplished something hard in my life so I know everything" crowd of the utility of empathy as a tool for exploring your "unknown unknowns". You'd think that more people would be interested in a dialectic (in the classical philosophy sense) discussion as a cooperative if conflicting search for truth. This is harder to do if you systematically overestimate yourself and underestimate others. Of course as I write this I'm thinking of all the times I didn't practice what I'm preaching now.
Also parquet has lots of features that'll get you to the general vicinity of a single record tolerably fast without sacrificing much in terms of storage or computational complexity. It's a small price for a big win.
What this gets right: Part of the reason you want to store columns together is that similar values compress well, so you could reduce your IO: smaller files are faster to load into memory. However in many cases (e.g. Arrow, Parquet) lightweight compression formats are preferred here, e.g. run length encoding (1,1,1,1,1,1,5,5,5,5,3,3,3 -> 6x1,5x4,3x3) or dictionary encoding (if your column is enum-like, you can store each enum value as a byte flag) because they can be scanned without decoding, amplifying your savings.
What it misses on (IMHO): - There's a metadata field but it doesn't contain any offsets to access a specific column quickly. So if you have 8 columns of 2GB each, to just get to the 7th column you have to read 12GB first which is quite wasteful. If you store just an offset, you could be reading a handful of bytes. Massive savings. - Within each column, how do you get to the range of values you want? Most columnar formats have stripes (i.e. stored in chunks of X rows each) which contain statistics (this stripe or range of values contains min value A, max value B) that allow you to skip chunks really fast. So again within that 2GB you have to read not much more than you strictly have to.
If this reminds you of an on-disk tree where you first hop to a column and then hop to some specific stripes, yeah, that's pretty much the idea.
-----
Sidenote: I've generally concluded that "human readable" is only a virtue for encoding formats that aren't doing heavy lifting, like the API call your web app is sending to the backend. Even in that case, your HTTP request is wrapped in gzip, wrapped in TLS, wrapped in TCP and chunked to all hell. No one complains about the burden caused by those. So what's one more layer of decoding? We can just demand to have tools that are not terrible, and the result is pretty transparent to us. The format is mostly for the computer, not you.
When I hear about stuff like terabytes of JSON just being dumped into s3 buckets and then consumed again by some other worker I have a fit because it's so easy and cheap these days not to be that wasteful.
I've taken some liberties and likely made some mistakes, but it's aping the kind of "history of technology in its social context" that I love to read about.
I am actively taking suggestions for similar reads - one I love off the top of my head is "The Victorian Internet" by Tom Standage: https://www.smithsonianmag.com/arts-culture/review-of-the-vi...
I like this comment because it gets to the heart of a misunderstanding. I'd further correct it to say "for something it wasn't trying to be good at". DeWitt and Stonebraker just didn't understand why anyone would want this, and I can see why: change was coming faster than it ever did, from many angles. Let's travel back in time to see why:
The decade after mapreduce appeared - when I came of age as a programmer - was a fascinating time of change:
The backdrop is the post-dotcom bubble when the hype cycle came to a close, and the web market somewhat consolidated in a smaller set of winners who now were more proven and ready to go all in on a new business model that elevates doing business on the web above all else, in a way that would truly threaten brick and mortar.
Alongside that we have CPU manufacturers struggling with escalating clock speeds and jamming more transistors into a single die to keep up with Moore's law and consumer demand, which leads to the first commodity dual and multi core CPUs.
But I remember that most non-scientific software just couldn't make use of multiple CPUs or cores effectively yet. So we were ripe for a programming model that engineers who've never heard of lamport before can actually understand and work with: threads and locks and socket programming in C and C++ were a rough proposition, and MPI was certainly a thing but the scientific computing people who were working on supercomputers, grids, and Beowulf clusters were not the same people as the dotcom engineers using commodity hardware.
Companies pushing these boundaries were wanting to do things that traditional DBMSes could not offer at a certain scale, at least for cheap enough. The RDBMS vendors and priesthood were defending that it's hard to offer that while also offering ACID and everything else a database offers, which was certainly not wrong: it's hard to support an OLAP use case with the OLTP-style System-R-ish design that dominated the market in those days. This was some of the most complicated and sophisticated software ever made, imbued with magiclike qualities from decades of academic research hardened by years of industrial use.
Then there was data warehouse style solutions that were "appliances" that were locked into a specific and expensive combination of hardware and software optimized to work well together and also to extract millions and billions of dollars from the fortune 500s that could afford them.
So the ethos at the booming post-dotcoms definitely was "do we really need all this crap that's getting in our way?", and we would soon find out. Couching it in formalism and calling it "mapreduce" made it sound fancier than what it really was: some glue that made it easy for engineers to declaratively define how to split work into chunks, shuffle them around and assemble them again across many computers, without having to worry about the pedestrian details of the glue in between. A corporate drone didn't have to understand /how/ it worked, just how to fill in the blanks for each step properly: a much more viable proposition than thousands of engineers writing software together that involves finnicky locks and semaphores.
The DBMS crowd thumbed their noses at this because it was truly SO primitive and wasteful compared to the sophisticated mechanisms built to preserve efficiency that dated back to the 70s: indexes, access patterns, query optimizers, optimized storage layouts. What they didn't get was that every million dollar you didn't waste on what was essentially the space shuttle of computer software - fabulously expensive and complicated - could now buy a /lot/ more cheapo computing power duct taped together. The question was how to leverage that. Plus, with things changing at the pace that they did back then, last year's CPU could be obsolete by next year, so how well could the vendors building custom hardware even keep up with that, after you paid them their hefty fees? The value proposition was "it's so basic that it will run on anything, and it's future proof" - the democratization aspect could be hard to understand for an observer at that point, because the tidal wave hadn't hit yet.
What came was the start a transition from datacenters to rack mounts in colos and dedicated hosts to virtualization and very soon after the first programmable commodity clouds: why settle for an administered unixlike timesharing environment when you can manage everything yourself and don't have to ask for permission? Why deal with buying and maintaining hardware? This lowered the barrier for smaller companies and startups who previously couldn't afford access to such things nor markets that required them, which unleashed what can only be described as a hunger for anything that could leverage that model.
So it's not so much that worse was better, but that worse was briefly more appropriate for the times. "Do we really need all this crap that's getting in our way?" really took hold for a moment, and programmers were willing to dump anything and everything that was previously sacred if they thought it'd buy them scalability, schemas and complex queries to start.
Soon after, people started figuring out how to maintain all the benefits they'd gained (democratized massively parallel commodity computing) while bringing back some of the good stuff from the past. Only 2 years later, Google itself published the BigTable paper where it described a more sophisticated storage mechanism which optimized accesses better, and admittedly was tailored for a different use case, but could work in conjunction with mapreduce. Academia and the VLDB / CIDR crowd was more interested now.
Some years after that came out the papers for F1 and Spanner, which added back in a SQL-like query engine, transactions, secondary indexes etc on top of a similar distributed model in the context of WAN-distributed datacenters. Everyone preached the end of nosql and document databases, whitepapers were written about "newsql", frustrated veterans complained about yet another fad cycle where what was old was new again.
Of course that's not what happened: the story here was how a software paradigm failed to adapt to the changing hardware climate and business needs, so capitalism had its guts ripped apart and slowly reassembled in a more context-applicable way. Instead of storage engines we got so many things it's hard to keep up with, but leveldb comes to mind as an ancestor. With locks we got was chubby and zookeeper. With log structures we got kafka and its ilk. With query optimizer engines we got presto. With in-memory storage we got arrow. We got a cambrian explosion of all kinds of variations and combinations of these, but eventually the market started to settle again and now we're in a new generation of "no, really, our product can do it all". It's the lifecycle of unbundling and rebundling. It will happen again. Very curious what will come next.
My first Google Summer of Code project (aka my first internship ever during my freshman year of college) was implementing some of the animations (like Grow) in KDE Plasma. I'm not sure I did great work, but it felt awesome to be doing it - KDE 4 had just come out, it was extremely shiny and this was the era of the desktop cube and wobbly windows as desktop compositing had just become a thing in linux!
We're in the process of switching to this at work for some very high volume logs and I'm quite hopeful - other teams saw pretty decent improvements.
I guess I'm a bit confused about your position here - you note that it's predatory, and then you continue on as if it's an easy choice: maybe we disagree on what predatory means?
I can't recall a single class in school or anywhere else that taught me about the dangers of credit cards and debt that grows cumulatively. Yet credit cards are aggressively marketed like candy through the mail and every time I shop at a big box store or even online. Not to mention other kinds of debt like student loans, car loans, mortgages, etc.
A lot of people don't even begin to understand the things they need to be aware of to make a truly informed decision to take on debt: compounding interest, amortization schedules, inflation, their budget and runway in case their income stops, etc. They do not do some quick back of the napkin math on how much extra they'll be paying during the lifetime of the loan, and whether that extra cost is worth it. And credit card companies are fully aware of this, and leverage it for maximum benefit of course. They jump at you at the situation where you're least likely to stop and think: you're spending a surprise 1000 bucks to replace your busted heater which is really gonna eat into your food and gas money - wouldn't it be great if we gave you 100 dollars off of that right now? Just sign up for a store card, it takes 2 minutes! Also, the minimum payment will be so low! That'd be the "predatory" part.
Just "choose to live without them" is not effective if you're both underinformed and poor, AND companies are incentivized to push debt on you. Heck - when I was a (supposedly educated) college student, I was hit by an overdraft fee: I bought some lunch, and later a snack from the vending machine. And I paid 20 + 20 dollars in fees for the privilege of 2 transactions totaling about 7 dollars. It made no sense to me: how could you spend money that you didn't have in your account? It just wasn't a thing in my home country. Should I have known better and read 15 pages of small print? Yes. Did I? No. When I called, the bank said I should be happy they've opted me in to this "service". If I had known, I would have just not made those purchases. I also ran into trouble trying to opt out. Predatory.
How do you propose we bridge a massive gap in financial literacy when we as a society are struggling with basic literacy and financial stability? I don't mean like 20 years in the future ("better financial education" which is the same as "hopes and prayers"), I mean right now. So that leaves us with not much other than regulation.
What this rule cuts down on surprise fees, which isn't nearly a complete fix. But it's an okay incremental improvement: it's one less thing to watch out for. Either offer a simpler, easier to understand financial product, or don't. There's no god given right to a profitable business, especially when it's not based on a mutually beneficial transaction.
For some reason all this makes me reminisce about Fravia's Searchlores [1] which always felt a bit like if Umberto Eco was interested in computers. And the site felt a bit like the library labyrinth from The Name of The Rose where you'd turn some random corner and find something incredible, only to lose it forever later on :D
[0] http://ts.sesse.net/ [1] https://www.biostatisticien.eu/www.searchlores.org/indexo.ht...
If you get stuck and / or end up doing something cool, shoot me an email at hn at my website on my profile!
Here's the report with pics and videos if that sounds interesting!
This website seems like generated spam - why is it even being shown here?
I remember not really understanding at the time and then years later coming across Visual Basic and the name ringing a bell, and then I was off to the races! So much fun.
I certainly haven't read the entire thing but gosh there's some good bits. E.g. "B-trees Versus Hash Indexes" - tons of insights jampacked together into a couple pages for anyone who hasn't already been exposed to B-trees in detail (e.g. in a research context vs the usual data structures class)
Some things that help / helped me:
1. Some serious self-reflection about what the problem is: What are the things you do vs don't enjoy about your job? Do you really like what you're doing or are you doing it because of money, goals you set for yourself / you think this is the most prestigious type of company / problem / etc? Is the problem with yourself or the people you work with? Have you ever been happy at work before (at any job you had) and what was different about that? Or perhaps are you not sure about what you actually enjoy? Or are you just tired?
2. More self-reflection about what work means to you as a person and why: Is your job also your passion? Are you proud of your technical achievements? Do you have daily work hours or do the boundaries blur? Do you put in more work / care more about things than others (e.g. code quality, arguing for the right technical decision)? Do you feel especially good when people tell you you did a great job or somehow acknowledge that you're smart? Do you feel disappointed, unappreciated or like there's something wrong with you if you fail to get this kind of feedback? Not saying this is always the case, but for me they're warning signs for a type of relationship with work that's prone to burnout sooner or later. Maybe talk to a therapist about this kind of thing to help you sort out your stance on this, or just generally about a downward spiral in mood.
3. A big break - unfortunately not everyone can afford this but I'm assuming an engineer with a decade of work experience like you probably can. Go do something else for a while. Definitely a month. Maybe a year: e.g. take a job that's not the most technically or personally challenging but will give you other things: more work-life balance, meet interesting people you wouldn't, will expose you to different subjects you like, give you more freedom with your approach, let you play more. For example, I worked at an academic research center for a few years and it was not without its own challenges but overall broadened my horizons a lot. This isn't going to fix your problems in the long term but maybe you'll realize truly how things can be different - the benefit of perspective.
4. Some small "daily life" changes (obviously huge caveats here about lots of people not being able to afford / allowed to do this stuff but again software engineers usually have this kind of nice privilege, so why not take advantage): Some example ideas: Try to do less things at once, maybe work on one thing at a time even. Work on something with someone you like. Take 5-6 Fridays or Wednesday afternoons off in a row: just see how lessening the load a bit or breaking up the work helps. Enforce a hard-cutoff deadline in the evening. etc.
Now, to actually follow my own advice ...