9,188 karma · joined July 27, 2009
mali at akmanalp dot com
So far I'm within spitting distance of the winning entries without using any unsafe code or bit twiddling tricks or custom JVMs or anything like that, and having all the concerns nicely separated and modularized.
Excited to share soon!
For example, managing the context window has become less of a problem with increased context windows in newer models and tools like the auto-resummarization / context window refresh in claude code make it so that you might be just fine without doing anything yourself.
All this to say that the idea that you're left significantly behind if you aren't training yourself on this feels bogus (I say this as a person who /does/ use these tools daily). It should take any programmer not more than a few hours to learn these skills from scratch, with the help of a doc, meaning any employee you hire should be able to pick these up no problem. I'm not sure it makes sense as a hiring filter. Perhaps in the future this will change. But right now these tools are built more like user friendly appliances - more like a cellphone or a toaster than a technology to wrap your head around, like a compiler or a database.
There are other similar lightweight encoding schemes like RLE and delta and frame of reference encoding which all are good for different data distributions.
Now, product groups for which data is most frequently and easily available is the 4-digit level, which is quite broad. If you look at the code 3002 in the HS classification system (of which there are many versions but we'll ignore that for now), you'll find a category, succinctly named:
> "Human blood; animal blood prepared for therapeutic, prophylactic or diagnostic uses; antisera, other blood fractions and immunological products, whether or not modified or obtained by means of biotechnological processes; vaccines, toxins, cultures of micro-organisms (excluding yeasts) and similar products; cell cultures, whether or not modified:"
https://hts.usitc.gov/search?query=3002
People new to trade data, especially programmers, with some hubris, tend to think this is way too long a category name to fit in a title or dropbox, so they chop it at the semicolon and call it good, resulting in "Human Blood" or similar. Better data sources tend to shorten these based on the real world percentage of the subcategories, e.g. see here "Serums and vaccines":
https://atlas.hks.harvard.edu/explore/treemap?exporter=count...
If you search for 3002 (Serums and Vaccines) in the US's exports in 2023 you'll see the figure 1.58%:
https://atlas.hks.harvard.edu/explore/treemap?exporter=count...
Which seems to me to be how they arrived at that incorrect number - some other website showing comtrade / us trade data with bad category names.
Lesson here: classification systems are hard.
Had to reread that a few times to make sure
It costs even more to be reckless today.
Re: "whitey on the moon" - I'm not sure the space program would be my first target there but I think it makes a more poetic contrast and forces people to pay attention by targeting a beloved cultural narrative. Cyberpunk - by my reckoning a bit later - has been preaching a very similar message of massive inequality in the presence of incredible technology and wealth disparity and power concentration. And yet that doesn't draw the same ire. I guess in that case it's easier to dismiss the core message because robot limbs and cool neon lights are too much of a distraction.
https://99percentinvisible.org/episode/suits-articles-of-int...
Especially for a first time in all of humanity type of mission, half a century ago, which yielded brand new data on faraway objects we'd never had, and considering it's still going and reporting data, it's arguably a bargain basement price for such a thing.
I think this is the way.
Also quite similarly soon after VB, learned C# so I could make mods for RunUO, which was a reverse engineered server implementation for Ultima Online that people would run free game servers with. At that point I was pretty hooked, and tried making things like dragon eggs that would hatch over time and evolve and such. There's something about other people being able to experience your code in a virtual world and also the creative aspect that makes it somehow addicting.
I think there's probably some lesson in there about microsoft misunderstanding the strength of VB as a RAD tool for mom and pop shops and non-software firms who have a single tinkerer, rather than an Enterprise Language. It died a slow death in favor of C# at that point. Embrace, extend, extinguish, perhaps.
I'm the guy with the enthusiastic thread earlier on in this post. I'd love to sit down and chat with you for an hour on zoom and hear all about those times, which we could then we could post the video on here - I think people would appreciate.
I have absolutely zero experience in interviewing people, nor do I have a media channel of any kind, but I promise I'd do my best to ask interesting questions. If that sounds interesting, shoot me an email (you can find it in my profile).
But I'm not sure it was anything particular about the environment per se. Like, visual basic was written to do business software, truly the most boring thing imaginable. I think it's more about being left to your own devices with something that intrigues you for hours on end without an adult trying to control or direct what you're doing. Maybe. I'm not sure :-)
I found a copy of "Write Your Own Adventure Programs" (1983 - Usborne: https://colorcomputerarchive.com/repo/Documents/Books/Write%...) as a kid in my primary school's bookshelf. I remember the code was written in BASIC and my family didn't really own a computer back then.
Fast forward a few years later I saw this "Visual Basic" thing and thought it would be similar ... it was, but only sort of. I had no book to learn from at first so I remember clicking through every single menu and button available to see what it did. Then I remember using our dialup to download every possible 3rd party VB form control and throwing them in a Form to see what they did. I don't know why I found this entertaining enough to keep doing it.
Eventually by copy pasting and changing stuff I was able to write some basic "homework helper" programs: calculate the area of a circle and stuff like that. Soon after I tried to look up tutorials which taught me basic win32 programming to do things like have an icon in the status area next to the clock, and then hiding my window to run in the background and make annoying sounds so I could build a silly little prank program to install on my friend's computers which was fun but often would fail because they were missing some .dll file which wouldn't fit on the same floppy.
It could be frustrating at times but also I feel so blessed to have lucked myself into learning programming this way and my parents pretty much just letting me do whatever I wanted to this expensive device that probably was not a small thing for us to afford at the time.
Even tutorials felt more fun at the time, it'd be "hypnoMan37's windows registry tutorial!!! HEyyeyeyy Guuyzs :-)))) gzgzgz to my irc channel #blabla on EFNet! so first you call RegistryCreateNewKey32(...." because god knows I did not have an MSDN CD either.
Learning via a code camp feels way more efficient but also so much more dry in comparison. I wonder if there isn't a substantial cost to boring the newbies to death.
Brilliant observation that never occurred to me. Around that age being allowed to sit with and listen in on adults having normal adult conversations - being one of them - felt special.
Also this is not too dissimilar to how adult life that surrounds every child is to a child. You're sort of used to living in a world that has workings beyond your comprehension and just going along with it. I didn't get what exactly was going on but I did understand /something/ was.
I'm listening to a review of The Great Mouse Detective (1986) which has a similar ethos, as did other content targeting young people from that era. Also I recall picking up books as a kid that were certainly not meant for children and adults back then didn't even blink, and I think it stoked my curiosity and interests and pushed the boundaries of my understanding, as well as prepared me for growing up. I don't think I ended up being a worse person or being traumatized in any way. Part of me wonders if kids' content being much more sanitized these days is a mistake.
> The elevator pitch behind this performance difference is > straightforward: column-stores are more I/O efficient for read-only > queries since they only have to read from disk (or from memory) > those attributes accessed by a query.
> This simplistic view leads to the assumption that one can ob- > tain the performance benefits of a column-store using a row-store: > either by vertically partitioning the schema, or by indexing every column so that columns can be accessed independently. In this pa- > per, we demonstrate that this assumption is false.
https://faculty.cc.gatech.edu/~jarulraj/courses/4420-s19/pap...
:-)
It's really the perfect language to pick up on a visit even ... except the vocabulary doesn't resemble anything that most of the rest of the world speaks. There's lots of loanwords from farsi, arabic, french and english of course but beyond that and speakers of other Turkic languages, it's struggle for most people.
But yes, it's true that we're often over the moon that someone put in the effort to speak it :-)
As a long time sufferer, I want to support the general sentiment of this post, and I hope it helps someone who finds this that is currently suffering. My core message is that managing your psychological response to your condition matters immensely. I've noticed over the years that my perception of my tinnitus gets worse when my stress levels are high, even if for unrelated reasons. For example, reading about this post might remind me of it when I wasn't thinking about it for a long while. And suddenly this might bring back unpleasant feelings I have about it and it might suddenly feel very loud and overpowering when half an hour ago I wasn't even aware of it. But then an hour or two later I'll catch myself having a normal, quiet, unimpeded conversation with someone, which can feel like a contradiction.
Unmanaged, this kind of thing can turn into an unchecked, self reinforcing doom loop that's not always necessary or helpful. As with anxiety (another long term problem I manage), one unhelpful thing doctors might say when you first get tinnitus is that it doesn't go away - this is one of those technically true statements that patients can misinterpret. The reality is that just like anxiety, in many cases there's plenty reason for hope: it's often quite possible to improve quality of life that's mostly the same as it was before. It's still there if you check, and time to time you need to manage it, but also you might go days and weeks without even noticing, living a perfectly normal and happy life rather than a miserable existence.
On the other hand, leaving your primary care doctor with "yup, it's tinnitus" (my interpretation at the time: you're fucked for life, good luck) can lead to deep feelings of despair, panic, lack of sleep, over-sensitivity to noise of any kind even within normal thresholds, and many more terrible things. And "just ignore it, you'll be fine" is trite advice that can be difficult to achieve without help, like telling someone to ignore a siren in the next room. And it might be difficult for others to understand how debilitating it can be. But slowly and step by step it's possible to get there.
Sadly compassionate and competent care for tinnitus, even among audiology specialists, is few and far between and patients are left to suffer and / or try to figure stuff out on their own, which like a hypochondriac on webmd, can leave you feeling even worse. Luckily it is more possible to find tinnitus specific care these days and I encourage you to find a clinic that does this if you can - it's worth traveling for IMHO. There's a cocktail of potential causes that indicate different treaments so take this with a grain of salt but what worked for me was a combination of various things first to manage my current state and give me some relief during my initial crisis state (psychological help with how to deal with the reality of your condition, various things to help mask it and allow me to get normal sleep), clearing out wax (it's hilariously gross and also wild how it can affect your hearing perception and tinnitus in the most unexpected ways), common sense non-doomer information on the nature of the disease, what to do next and how to slowly transition to focusing on normal life ("playing playstation"): doing things with people, being outside, working out, focusing on your interests, as opposed to catastrophizing at home in bed. This feels like a non-cure cure but the results for me are night and day - if it's a placebo, I'll take it. I still have trouble with it time to time, but lead a normal happy life most days.
That aside, while we don't have the technology to make it fully go away yet, there have been a lot of recent changes and advances in our understanding of how it works which might lead to new therapies. Disclaimer: I'm not a doctor and might be butchering this so take this with a grain of salt: There seem to now be competing theories on the exact causes e.g. arguments about whether it's the the little hairs in the ear are sending no signal or incorrect signals. Also a recent understanding that you can have a seemingly unaffected hearing AND a normal audiogram but still have tinnitus is leading people to question if it's a different type of damage (different, larger hairs) that cause it which might mean different therapies. There's promising new treatments like Lenire which seem to be low risk and whose early study results seem to be much more promising than earlier versions of similar ideas (haven't tried it). I wouldn't be shocked if we eventually got much much more in our lifetimes.
TLDR: of course you'd rather not have it - please do be careful in protecting your hearing as much as possible. But if you're at the point where you do have it, just take a bit of solace maybe that it's not always a terrible prognosis and more improvement might be possible than you think. And the psychology and managing my emotional perception of it was very helpful for me. I hope this helps you, dear reader.
The thing is - and I imagine you know this - that when you run tens of thousands of database instances it's not a matter of if, it's a matter of when. I can't chase after thousands of engineers to make sure they're not hitting some obscure pitfall nor can I demand that they know the ins and outs of the 8 different types of locks in FooDB. And "read the docs lol" definitely doesn't scale. So we build automated mass-guardrails and linters as best we can and use various shenanigans to deal with the real world.
In the context of schema migrations, there's tools like pt-osc, gh-ost and pgroll that are excellent for what they are and I'm grateful to those who created them but these are still in effect huge hacks that are bolted on top of the database. They're teeming with footguns. I can tell you about a time where badly conceived timeout / error handling code in one of these tools in their golang database driver resulted in most of a database dropped instantly with no fast rollback, for example. That was fun.
If a third party bolt on tool can implement zero(ish) downtime migrations and cutovers, at least much much much better than the regular DDL, why the heck is the database not doing whatever that thing is doing? It's tech debt, history, crud, lack of caring, etc. Schema migrations is just the tip of the iceberg too, don't get me started on replication, durability settings, resource usage vs capacities ...
My point is that academics and vendors should be taking operational problems seriously and owning them, and solving them properly at the database layer.
And just like you I also make living off of the fact that the real world operational characteristics of truly busy databases are very different than what gets benchmarked and tested. I'm just saying that some of this stuff even I don't want to have to deal with. There's better problems to be working on.
It's a world of pain, and there are so much scar tissue of third party tooling doing crazy stuff dealing with this problem among large companies that really should be the DB vendor's problem.