The cloudy layers of modern-day programming
vickiboykis.com
vickiboykis.com
Many commenters are attributing this problem to the modern high-level tools we now have access to. But I don't think this is the crux of the issue. You can face the same issue (you're plumbing things, not a designing a system) whether you are working with low level or high level components.
Heck, you could be working on a hardware circuit, but if the only thing you had to do was make sure the right wires, resistors, capacitors, etc. were in place between the chips you're still just doing plumbing work.
To me, one of the most satisfying things about programming is when you can build something great by starting with a concept for your lower-level primitives, your tools, and then work up through the higher levels of design, ultimately having the pieces you designed fit together to form something useful to the world.
This building-things-to-build-things idea is even satisfying in other areas. Just gluing a bunch of wood together to make a piece of furniture is fine, but building your own jigs and tools to be able to do the kind of cuts that enable the end design you envision is way more satisfying, and opens up the design space considerably.
If I had to lament anything (and perhaps this is what's most in alignment with the post) it's that most of the high-level primitives you touch these days tend to be sprawling, buggy, not focused, and just generally not of high quality or performance. It's possible for high-level primitives to avoid these pitfalls (e.g. Sqlite, the canonical example) but it does tend to be the exception.
I think there is still plenty of interesting and satisfying software engineering work to be done when starting with high-level libraries and tools. You just need to think about how to use their properties and guarantees (along with maybe some stuff you build yourself!) to enable the design of something more than just the (naively-plumbed) sum of the parts.
It sounds unrealistic and I'm going to get flamed, but hear me out. It works. Most of my development these days is in reasonably complete languages like go, rust, zig, and various scripting languages, so your mileage may vary if you're writing in something like rune, hare or carbon that is still taking shape.
If you think I'm crazy but have a lingering skepticism, challenge yourself to spend one day, one week, or one month using the stdlib only. If that is unrealistic in your setting a compromise could be to only use libraries that were created by yourself. You wont come out of it as an enlightened samurai monk with a celebrity HN presence, but you will gain an immense sense of scrutiny that didn't exist before as you take a look at the libraries you want to use after that.
For those who think this is just 100% bull, consider how we survived before google, youtube, and stackexchange existed.
You can find cool technical problems anywhere as long as you are willing to take the path less traveled.
After doing that a few times, I'm no longer sure if the reward of tackling cool problems to create more robust, better, faster components is worth the stress of missing deadlines.
Looking back on what value of better work materializes and what is, per YAGNI, usually wasted, I just had a thought: perhaps the right way is to take the easy/dumb way and focus all available time/effort to optimize it for performance - instead of abstraction and extensibility. Because in my experience, nobody ever extends the code the way you envisioned - if they do it at all, they do it by first refactoring it to suit their own idea. And, nobody ever goes back to fix performance. Therefore, making things abstract and extensible is mostly a wasted work - but making things fast pays back for as long as the code is in use.
Instead, I want to do credit-based billing: you buy credits, and when you get to 0 credits, you are cut off (with an auto-refill option for the "metered billing experience," but with strict spending limits). This is, in my opinion, a much better UX than metered billing. From a distributed systems perspective, it's isometric to "metered billing with a hard spending cap" which may be the ultimate version.
The problem with credit-based billing (and why nobody does it) is that if you have a service in multiple datacenters, you have to consistently update a database to make sure that you don't drop below 0 credits, and that is very slow. However, by fiddling with the CAP theorem the way CockroachDB/Spanner do, I think we can do credit-based billing and make it feel like the AP system that metered billing is, and behave like a CP system only when we absolutely need to.
Also, this is basically all enabled by the fact that AWS has a precise time service.
In theory, version 1 of the API will be only in one region, version 2 will have active/passive redundancy, and around version 3 or 4 I want to switch to active/active in several DCs to give low latency and high reliability.
> Instead, I want to do credit-based billing: you buy credits, and when you get to 0 credits, you are cut off (with an auto-refill option for the "metered billing experience," but with strict spending limits). This is, in my opinion, a much better UX than metered billing.
How do your customers feel about that? Have you researched if your customers are comfortable spending their money upfront to buy credits they might only use much later (if ever)? For small amounts of money that might not be a big deal, but if we're talking about thousands of dollars that might look different.
The way many of these projects go is someone very smart works with subject matter experts to map out the problem space. The smart person (or people) then go and begin building this set of primitives and integrating the product, adjusting as they go along accruing some warts.
After 6-12 months we have this beautiful tool that improves developer velocity, new features are easy to code; then disaster strikes. It turns out the map of the problem space was wrong! A bunch of things the team believed to be invariants aren't! Suddenly business needs are forcing developers to tear down walls in their beautiful abstraction castle until all that's left is a tangled maze no other developer has a hope of understanding.
Now the project is a millstone around the dev team's neck rather than velocity boost they'd hoped for.
The way these projects more often succeed is some senior engineer pastes together an abstraction layer sends it out into the world. It gets heavily abused for years until finally the team says "We know this sucks, and there's a lot of business value if we make it nice, let's invest a bunch of time in retrofitting this" and fight like hell to make the business case. IMO this tends to lead to better projects and value (though unfortunately many companies make the fight harder than it should be)
My general advice is that software engineering has a lot of well worn patterns for problems, stick to those as much as possible. Their great advantage is that any experienced software engineer will recognize them and onboard quickly, allowing you to focus on those parts of the problem specific to your project/company.
In most cases, whatever common pattern you shoehorn your problem into will suffice for the purposes of the business. It will be ugly, and have warts, but will be generally maintainable and not often touched. Again, if this turns out to be wrong after it's been battle tested and shown value, you can begin to migrate away to something new.
There are exceptions to the above, and many companies don't actually go through the motions of following well-worn patterns even when they think they do e.g. many companies with public APIs make a common set of mistakes we've known how to solve for over a decade and have known they're easy to solve if you deal with them up front.
I didn't mean to imply a dichotomy. I don't think an intense planning phase up front solves the problem. But feature-sprints tend to crowd out refactor-sprints because of the demands from above for new features. Downtime working on backend stuff is not perceived by the customer as anything benefiting them. But here, the backend is a codebase no one enjoys working with and the customer is the C-suite expecting X features this month because X or X-1 features were pushed last month.
Need a hashtable? Write one.
You won't come out with a celebrity HN presence, but you may gain enlightened samurai monk status.
Need a conditional? Time to jump around.
I wrote a web server that scales to over 1 million requests per second, and I found out it's much more maintainable, scalable and environmentally friendly for our company.
Do you happen to know where (what source) you got that from? I'm genuinely curious, as to my knowledge, compilers are generally still easily fooled by things that causes them unable to vectorize code or factor out conditional jumps.
To give an example here, see Mike Actons' talk below (from about 43:10 onwards) in which he describes the compiler failures.
Nit: assembly is a language. The phrase you are looking for is probably "structured programming".
All before Google, Youtube and Stack Exchange
In the last 3 years I think I never saw even once a job description on any popular job board where they advertise that you will do some actually interesting programming. The only ones I've seen have been on Twitter but from companies doing things in areas I have no experience in (e.g. game engine programming).
It is definitely true for VFX Artists. And you will definitely get a pay cut compared to tech (50%-ish TC)
They basically test for distance from school, and not much else, as the algorithms aren’t really reflective of real-world work, which, as the article states, is really fairly simple “glue,” binding together prefab sections.
If someone is good at, and energized by, writing “from scratch,” and "learning the whole system," then they are actually not what you want. You want people that are good at rote, can learn fairly shallow APIs quickly, and are incurious as to why things work.
I have a friend who is writing code to run on a sort of exoskeleton meant to benefit disabled people and help them walk. He has never in his life "deployed to the cloud" and wouldn't have the foggiest idea of how to do it.
Now you have a cloud component.
A lot of modern hardware design feels like that, take a microcontroller, some peripheral chips, connect them together, and copy the datasheets for whatever support passives they need.
Lisp can enable fantastic development speed allowing you to build your own primitives. Racket's ecosystem's all high quality too.
I believe Julia has a less buggy ecosystem allowing you to pipe together natively written ML things, as opposed to Python's dumpster fire ecosystem
Love the Julia language. But its ecosystem is definitely less complete, less documented and arguably more buggy than the Python equivalences.
This is not a failing on Julias part, the Python ecosystem is 50x bigger and backed by multiple of the largest IT/ML firms on the planet.
People who are really serious about software should make their own hardware. - Alan Kay
Nitpick: ! for shell commands is an IPython feature, not Jupyter. It doesn't work with other kernels.
Nowadays when it's just gluing frameworks together and configuring AWS services... it doesn't really feel any different intellectually than cleaning toilets. Sure the pay is better but as a creative challenge they're pretty much on par.
Comparing your six figure white collar job to basic janitorial work is pretty damn cringe and pretty objectively untrue.
I've cleaned toilets professionally, and I'll say once and for all: writing software, no matter how monotonous or boring, is nothing like cleaning toilets. And I'd be willing to bet that someone trying to make such comparisons has ever had to do that kind of work.
Your standard CRUD applications and web services are largely just a rigamarole of reciting the right incantation and duct taping bits together. It's immensely non-stimulating work when done properly.
This isn't an insult by any means. It's a testament to the triumphs of decades of engineering efforts to turn the process of orchestrating extremely complex electronic systems spanning continents into a largely trivial task for most projects.
Not only this, but there'll always be an a**ole to say that we're doing that wrong, and add a few more steps in between just to make the process "better".
I grew up doing hard farm/ranch labor outside in 95-105 degree TX heat and humidity. I agree with the ancestor comment that manual labor is a hell of a lot more satisfying and stimulating than gluing together AWS services with IAM/RAM snippets from stack overflow and updating some design doc about it. If it payed adequately I might do manual labor in the day and solve actually challenging and fulfilling technical problems at night. Programmers don't get to program much anymore :(
There is a big difference. one is cleaning shit other more shit creating more shit :)
Let's imagine building a video chat:
Are you going to build an OS from scratch? Network protocols? Encryption? The rendering engine for it? Compression algos?
Just how much of this do you consider "not configuration"? I think this all sounds very much like "old man yells at cloud" pining for a past that never actually existed.
I already started using ChatGPT to solve problems because I can’t be arsed to read through ten vendors worth of documentation. It wrote me a fairly complete and accurate chunk of code the other day to solve a problem.
- someone about django custom inlines, the answer was mild but it integrated various aspects of the framework in a short answer which helped a lot (django is particularly horrendous, i cant suffer its strange style, so that played too)
No one is stopping you from writing 6502 assembly code in your spare time. That's actually still a somewhat popular hobby.
using a software framework like django for "rapid application development" gives me a feeling closer to writing configuration files rather than actually "programming" (where "programming" is writing hardcore algorithms).
but don't get me wrong, I liked doing that, I got paid to do it. But let's call it for what it is: that python code (django app) was really django framework config.
this is a simlar phenomenon but worse, at least I could look around all of the actual code of the program for which I was writing configuration as code.
then, the thing about hardcore algorithms is that they need to be written once, and then everybody can use them. this is a giant problem. as I think about this, such hardcore algorithms are digital artifacts, so they are subject to the same problems all other digital artifacts (media, videos) are. a problem also known as software piracy. but software has been about composing proven algorithms together since day 1. the problem by this point is socio-economic. not technical.
I see the whole debate around "who will pay for critical infrastructure software" as another instance of "how should artists make money in the digital era".
But this comment (which I'm editing) is already at 0 points. somebody doesn't like what I'm trying to say, but I knew this already.
Well, if you were writing abstract pseudo-code, then maybe. In practice, the same algorithms are often reimplemented multiple times.
> and then everybody can use them
ah, if only that were true... implementations are often not that flexible, nor are programmers that keen on utilizing existing implementations.
> this is a giant problem
Problem? To the extent it's true - it's a boon, not a problem. Image if whenever you made a chair - suddenly everyone could get a chair without taking your own chair or spending any time and resources on chair construction. It's a miracle!
> a problem also known as software piracy.
1. Piracy is when people on ships with guns rob other ships. Arrgh, matey!
2. Sharing and copying software or other media is a good thing, not a problem.
PS - I haven't downvoted you. Point taken about django "config-programming".
the problem I describe is not a technical one, but a social one. and it's only a problem due to the current way society works. It's only a problem for some people (e.g me); them who are well satisfied by this "status quo" don't see any issue beyond lacking enforcement of IP 'rights' and the need for better DRM, and copy protections, and other things like that.
you're focusing on the technical aspects (plus you seem to be deliberately using the wrong definition of piracy).
I'm talking about the social economic aspects: I'm saying that this boon brought about by digital technology is only benefiting a select few. I tend to think about this 'boon' as potential that we collectively seem to be choosing to forego; I think it's my life's mission to do everything I can to avoid this "foregoing" of the great potential unleashed by digital technology; I feel like I'm swimming against the current most of the time.
If you mean how most people in the world are under stricter social control with the advent of technology than they were in primitive tribal society (or maybe even middle-age agrarian society), then maybe.
But if you mean the benefit from being able to make these copies - then definitely disagree: People get to have tons of free (libre & gratis) software, and lots of free (gratis) cultural products like images, audio, video and text. Are you bemoaning what we do with this glut, and perhaps the problem of over-abundance, shallowness of cultural taste etc.? Otherwise I'm still missing your point.
About piracy: I'm using the right definition, it's the copyright crooks who are using the wrong one :-P
I suppose software is the least affected by this alleged "problem". after all, the 'free and open culture' movement spun out[1] of the free (and open) software movements.
[1] citation needed.
The problem is, that the tools and frameworks we use have such bad configuration.
That says something.
then ask yourself - what have you done in order to have interesting job, projects, challenges, etc?
I mean, if you decided at some point that $big_salary for throwing JSONs via REST from CRUD app
is what you want to do until you pay your loans (e.g decade), then it's fine, but don't be shocked that you aren't doing bleeding edge R&D at some fancy place.
Like what prevents you from putting effort for year? two? and switching from X to Y?
then ask yourself - what have you done in order to have interesting job, projects, challenges, etc?
These two things don't connect to each other at all
The connection is simple: If you don't like something, you're free to change it (in this case)
If you feel like your job is easy and you feel bored, then you're free to quit and apply to place which will challenge you.
If you expect that some magic will happen and one day you will come to your job and you'll be tasked with coding spaceship, then you're optimistic as hell.
So, the question stands - what did you do in order to improve your situation? if nothing, then do not expect magic to happen because that's unlikely.
I don't think it only takes 1 yr to switch to something interesting. Any R&D lab wants you have a phd with published papers.
Maybe over years you'll gain an expertise to work in research, idk.
Also there are lots of way for a company to say they are doing interesting/meaningful things or whatever and effectively not be. And no matter how great the work you may be doing, you don't want to be doing it if it doesn't pay enough.
That's just the world we live in.
Should we be mindlessly churning away taking no issue with the state we're in?
This thousand times for all the whiners. Learn an esoteric skill, get hired for esoteric job.
The reason society works is because you can reuse abstractions that other people have already invented. It allows you to scale. Without economies of scale, you end up with Baumol's cost disease, which is extremely obvious in the US in industries like child and elderly care.
We don't want most software devs to be doing anything because gluing stuff together. If they weren't, we really screwed up somewhere.
Are our abstractions I-beams or Pre-Fabs[0]?
We all know that the rise of pre-fabs is, at its heart, the story of cheap developments all lazily (and hastily) thrown together in arrangements that are of low quality, mid-to-low beauty and do not last for very long.
Skyscrapers of the early 20th century stand today (with I-Beams) and are considered by many to be beautiful, maintainable etc;
People want pre-fabs, since they're cheap, the economy will always be in the pre-fab.
But pre-fabs have a limited shelf-life, and reconstruction is more expensive than spending a little extra up front cost.
[0]: This is the sort of pre-fab I am talking about: https://en.wikipedia.org/wiki/Prefabricated_building#/media/...
Pre-fabs are massive frameworks where mixing them always looks hack-glued together. Same as if you glue two prefabs from different vendors.
That said, industrial plants aren't gothic cathedrals. They are a collection of buildings plopped together.
So I guess with that allegory you have to ask: Am I building a factory, an office, a home or a cathedral.
You can then plan your architecture accordingly.
Your catalog of beams, bolts, brackets, weld patterns, rebar, piles, and concrete pour standards are all abstractions over extremely difficult subfields of materials engineering and structural engineering.
They exist so that your engineer can focus on building the structure using parts and resources with known, standardised behavior under the conditions the building will be put in.
Of course you'll see engineers break away from these abstractions when they need to for a given structure but those abstractions do exist and are commonly used so that a given structural engineer doesn't also need a PhD in materials engineering and countless other specialized fields.
- the construction process.
- material composition.
- thicknesses of the flange.
- width of the flange.
- depth/height.
- max length.
- shear strength.
- tensile strength.
- elongation (stretch/sag when approaching the point of failure).
- corrosion resistance.
- tolerances.
- cost per foot.
And the list only goes on.
A structural engineer can say "I have a specific structure in mind and need a beam that can withstand XYZ conditions" and pick out the matching beam from a table without considering any of the details as to how one would actually achieve those properties and withstand those conditions.
Alternatively an engineer can know their budget, what grade of steel, and what size of beam they want to use ahead of time and simply design a structure using those abstracted pieces in coordination with the architect.
Beams and other construction elements are the structural engineer's equivalent of a Hardware Abstraction Layer (coming from the perspective of an engineer who works close to the metal when I can). It lets the engineer abstract most of the minutiae regarding working with the hardware itself by giving the engineer a largely uniform interface for choosing how their product interacts with the world. Generally the abstraction holds but sometimes it's leaky and details show through. Likewise sometimes the abstraction is insufficient and the engineer has to do some custom work that breaks away from the established path to meet the constraints.
But if we had a system that we owned, then it could be adjusted to fit the requirements, elegantly. But we don't, so we can't.
The other problem is the "Ops" part: vendor owned systems are opaque AF, and when something goes wrong, impossible to debug. Then you become a support ticket monkey, praying you can convince the powers that be on the other end that a. it is truly their stuff that's broken, not yours and b. we pay for it, so yes, you should support it.
When a. or b. fails, then you end up writing yet more glue code to try to work around the bugs and outages that your vendor just doesn't give a shit about.
I just wanted to shout at them to fix their damn system, but of course we ended up implementing retries instead…
Wheels. Absolutely wheels. Library gluing is what gets us garbage like Electron that needs to die in fire.
Imagine if a carpentry workshop gives a carpenter a bunch of IKEA kits and tries to conflate wanting to make furniture that isn't cost-cut prefabricated crap with insisting she can recreate all the other tools with just her whittling knife because it's in the 'true spirit of carpentry'. (Honestly, if anything, comparing Electron to IKEA is a insult to IKEA - there are cases where using IKEA is actually reasonable, they just aren't actually carpentry.)
(You use libraries when it makes sense to use libraries, just like you use 2x4s when it makes sense to use 2x4s. Sometimes you can make the whole thing out of 2x4s, just like you can make a whole program out of:
tr -cs A-Za-z '\n' | tr A-Z a-z |
sort | uniq -c | sort -rn | sed ${1}q
Perhaps we simply need to more clearly separate the goals of studying computer science from studying programming/development. For the time being, however, I'm left feeling like I may be doing students a disservice avoiding the reality of what "modern-day programming" has evolved into.
In my experience, though, the set of {software development jobs that can be performed exclusively with the notions acquired in CS} is pretty much the same as the set of {boring software development}.
CS graduates tend to work on information systems, and we all know information systems are the epitome of boring software: https://thedailywtf.com/articles/programming-sucks!-or-at-le...
That's why R&D departments are full of people who learned a trade and later taught programming and software development themselves, and the bits with no business value are outsourced to consulting companies.
Two ways to out from the existential dread: one startup on a problem statement that feels exciting to you, so that you get to pick your tools, processes and what not. But this is not a realistic/affordable option for most people. Also, if your startup achieves even a moderate level of success, you'll be back to solving boring problems.
The other is to find a pockets of software development outside work, where you can still solve interesting problems, like developing small games. Joseph White, who created the famed programmable fantasy game console Pico-8 alludes to this (https://youtu.be/87jfTIWosBw?t=1080) ! It was an explicit goal to build a tool to solve cute, interesting problems because routine software development feels like glueing things together. The whole youtube talk linked above is worth watching. But the trick here is to forgo commercial motivations. People make fun, cute little games on Pico 8, distribute them as cartridges. Occasionally, there's a hit like Celeste. But otherwise its a small community of people solving problems and building things just for fun.
Programming an information system is like flipping burgers; developing industrial software is not.
Now I realize there’s probably more than one flavor of engineer:
1. software eng: ships features, prefers to use ready-made abstractions
2. infrastructure-ish eng: ships things that enable shipping of features, prefers to invent abstractions
The things they value are very different! I suspect HN tends more toward SW engineers (because statistics) with some stuff for infrastructure engineers.
1. Is usually implementing business logic for the product 2. Is generally working on things #1 uses to work productively
I'm not sure about the composition on HN, but I see a lot of #2 in "DevOps"/SRE roles. Usually these roles are skipped for small startups but at some point they get big enough they need dedicated people to take them on
Still, they value the same things that the "pure infrastructure" people I were referring to do simply because they end up being on the hook for how it behaves.
Programming and software engineering and their subdomains and superdomains are no longer a priesthood or an academic exercise or playground for Levy-style oldskool hackers, or rather, no longer remotely those things exclusively.
Most of what most people who fit in the supercategory do is work. The specifics of the work vary. If they aren't the ones that give you joy, change categories is a good and readily available solution.
If you want to hack on beautiful code, there are 1000x more ways to do so today spanning the purely aesthetic defined as many ways as you like (live coding, avocational language design, exercises in new tricks for old tools like the demoscene and emulator worlds), to the aggressively useful (small tools made by small teams of solo devs upon which empires balance).
You don't need to be a mid-level IC at MassiveCorp unless you can't give up access to the mana-tap of massive scaling. In which case maybe that IS your thing and the complaint is flat.
The joy hasn't left. Maybe it's just harder for the author to find? Or the problem is simply that no matter what you do for work, it's work, "chase your bliss" being of course a lie, because work IS work, and dissatisfaction like water will always find its level.
There's also many other alternatives that could be used if memory efficiency is the goal, e.g. polars/data.table instead of pandas, an OLAP database instead of bigquery etc. While I agree that pasting together configs is tedious and annoying, a lot of this type of work is due to developers themselves trying to replicate new trends at $bigco$ instead of optimizing for their own needs.
I just don't see how you can arrive at today's productivity without running into this issue. Sure, it's not fun. Just like the author, it makes me miserable too. I have much more fun writing a parser or database from scratch as a side project. But it looks like you have to pick one: Do you want to be productive and get stuff done in time, or do you want to have fun and make art?
It's not a problem. It's an inevitable tradeoff.
No, but I understand why you’d think that. Both the author and commenters make strange conclusions about the situation.
The problem is much easier to phrase than that. We don’t have good building blocks and tools. What does good mean? It means reliable and performant implementations, predictable behavior, well designed API surfaces (does one thing well), ability to debug and inspect, among other things.
The basic tools are already like this. Standard libraries, compilers, unix tools, file systems, certain battle tested databases. They’re all open source & move very slowly.
However, we also have countless proprietary cloud services, dependency hell in languages like JS and Rust, services scattered over different networks, regions and vendors.
So.. how did we end up here? I think it’s a combination of different reasons:
- More data both in total and per time unit. Much more people online. The world also developed towards ad tech, ML and video which demands a lot more than text and images.
- Horizontal scaling is the most cost-effective (due to physics) which prompts orders of magnitude more complex distributed systems.
- Cloud is the only option for many and gets pioneered by massive rent-seeking cynical corporations, so we lost both FOSS and any serious opportunities for standardization and simplicity. The IBM years are back.
- Money is so heavily involved, so ad-tech giants slurp up the talent and use them for market dominance, tech contributions become a side effect.
- Package managers and GitHub makes distribution easy, but results in a ginormous amount of dependencies and vendors.
So how do we fix it? We need to change culture and attitudes. A couple of tricks: Choose your deps carefully. Prefer FOSS, check issues – are these good authors? Is the API surface good? Well documented? Only use what you need, don’t let trendy blog posts FOMO you into using anything. You probably don’t need click metrics, A/B testing frameworks, minification, uglification, code splitting. You probably don’t need microservices either, or 99.999% uptime. Chances are you don’t even need multiple machines, or more than basic monitoring.
I think the point is (at least for me it is), when you do that, have you done anything new that few or nobody else has done before?
No, it's just a well worn path which thousands of organizations have done already.
The world of software development is immense. You can treat it like a job, or like a creative path, but to synergize both requires a lot of effort. Complaining that there aren't interesting programming jobs comes across as uninspired. Go look for them!
I could sign up for that! But what interesting things? That cluster will in nearly all companies just be used for yet another CRUD API supporting yet another social or adware platform. Yawn, please no.
If you are aware of companies doing actual intellectually creative programming anymore, please share them. I have no doubt they exist, but they are becoming way too rare.
- a multiplayer web VR platform with hundreds of challenging problems across the entire stack
- a AI-powered information engine and ontological system with layered insights
- a multi-platform realtime communications research engine
- a web3 Patreon clone & a web3 universal messaging system
- a decentralized identity management application
- a decentralized, self-managed, open source YouTube/TikTok clone
- three game development competitions
- a novel computer vision research project for a sports company
- a graph-based end-user programmable productivity suite
- Multiple prototypes for new social media platforms
- a collection of web-based precision tools like an archiver, URL shortener, etc.
- an AI-generated blog
- an AI-generated wiki & knowledge base
- a few more things I'd have to think about
I love what I do, I love waking up to code, I love going to sleep thinking about architectural problems. I love spending all day in a deep state of problem solving.
If you have a specific kind of creative programming in mind, I can help you look around for something more fulfilling.
But a lot of times, I'm just connecting black boxes and slapping a coat of paint on top (aka our branding). And it leaves me with this sort of guilty feeling that I'm not actually doing anything valuable.
Ultimately I still do it because the money is great and overall it's a very privileged position to have in today's world.
But I think this is where having a hobby project comes into play. I can do things "my way" and not worry about needing to use the most efficient or industry standard components.
I am so, so grateful that someone else does all of that stuff, and I can pay cheap cheap rates for some space on that machine, and a few CPU cycles on another one, etc.
I hate it. But I also don't know if we can do anything about it. Like, for example the company I work at heavily uses AWS Lambda to build mobile app backends and, as a backend developer there, I feel very uneasy about how little I understand and control the system running our critical business logic. But it works well enough and the money savings are amazing when compared to using dedicated hardware, so I can't really make a good point why we shouldn't be doing it.
Sometimes I wonder if other professions get to have fun. Maybe I should explore design to scratch that artistic itch somehow.
Let capitalism be capitalism and give people the energy and time to do things without having to justify every little thing to some pencil pusher to try and make anything happen at all.
Back to the '70s with Serverless (2020) http://evrl.com/devops/cloud/2020/12/18/serverless.html (2020)
which I linked from
Summer Blog Backlog: Distributed Systems https://www.oilshell.org/blog/2021/07/blog-backlog-2.html (Kubernetes is our generation's Multics)
I'd say the key issue is that the cloud "abstractions" are leaky, and they don't compose. This leads to combinatorial explosions of code and frameworks.
Finally, I started testing it on Kubernetes because a lot of customers seem to use that in production. That forced me to work on the following things (after I had to learn details of how the cache works and how to configure it correctly, use its API, design a good integration for it into our product etc.):
- Kubernetes basics, like what a pod, service, deployment etc. is.
- Helm charts for configuring everything (lots of YAML).
- Terraform to kick off the infrastructure in the "cloud".
- Minikube to run things locally.
- Using Docker K8s as Minikube didn't work well on my Mac M1 for some "reason".
- Installing Lens, a UI to make sense of the hundreds of pieces I was now juggling.
- Configuring the cloud environment with the right permissions for devs to be able to use it.
- Network configuration so nodes can "talk" to each other, without exposing things on the Internet for no reason.
And probably a few dozen more little things that ate up all the time I had. Designing and writing the code to make things work locally and in a Docker container (running Docker compose) was pretty easy and took me a couple of weeks, including a lot of testing.... but once I started with the k8s, cloud stuff... I've been on it for a few months now :( and things are still not working reliably in "certain cases" (everything that changes seems to break something). It's a real nightmare and took all the fun out of the project for me. Really makes "programming" something else entirely that I would be very happy to leave to a good sysadmin to handle.
10 years? Try since the dawn of computing, this was always a problem. Wanna know the most famous one, older than 50 years? Here it is:
In 1969, when Apollo mission on moon was live broadcasted by all the TV networks around the world, the image is of a grainy white and black, despite the fact that both NASA and TV stations already had color transmissions. Why? Because of the incompatibility of NASA transmission (which was a format more closer to our current HD - btw, can you imagine having HD format being common in 1969?) with NTSC/PAL systems at the time.
And what was the "API" that bridged (or in terms of the article "VendorOps") married these 2 incompatible formats? Use a projector on a wall and all TV stations filmed that wall instead of getting the real transmission. What a shame. Also NASA handling of the role films is abysmal, to say the least (https://www.nasa.gov/feature/not-unsolved-mysteries-the-lost...)
The cloud takes inefficiency to a whole new level.
Cloud vendor tools are designed to sell cloud services, not solve developer problems.
Jupyter notebooks are buggy and unnecessary (You can execute Python without a browser) and won’t work with medium sized data sets.
Excel handles larger datasets than pandas and small datasets more efficiently.
You could parse the 2gb of data from goodreads easily on a 486 in a raw text file about as fast, but fetching the data over a 14400kbps modem would take all day.
Not that there isn’t a need for optimization and replacing glued pieces with bespoke pieces. But just don’t do it prematurely because it’s more exciting. It’s regularly not advantageous.
BigQuery and Colab are fantastic tools largely because they allow less technical folks to try things out with very little friction and BigQuery scales to ridiculous amounts of data making formerly impossible queries possible. The example fits neither of these as the author is quite apparently very technical and the data is minuscule.
The author could have applied some economics thinking to this and correctly concluded that they could have run this exploration on a laptop using a regular old file and Jupyter (or just normal Python if they want to avoid the IPython infrastructure). Post-exploration they can run their application on a properly specced VM. IDK if BERT requires a GPU these days or if it can run well enough on a CPU, but either way, these resources are easily available on GCP without all the vendor-ops the author talks about.
To cap things off, the author says at the end that "in no previous universe would I be able to try out Stable Diffusion". Stable Diffusion requires 6 GB of GPU RAM and ~10 GB of storage. That was modest 5 years ago for commodity hardware.
I am not a cloud-decrier by any means, I love the cloud, I use it every day, but most of what I use is basic storage (buckets) and VMs. With just a little bit of abstraction it's easy to make something that runs in the cloud run locally just as easily. That also makes it easier to switch clouds if you find a better deal somewhere else.
TL;DR: use the right tools for the job
I wonder actually, how far could you get today with ChatGPT for recommendations, or with character.ai to actually just generate the entire customized book chapters for you while you watch.
These days I am learning Arduino and thinking of learning AVR programming.
Is there a career path for this kind of intersection?
Some suggestions for your study;
1) Use C/C++ only and nothing else. See my past posts for books full of example code. This will allow you to carry your knowledge across MCU families.
2) Starting with Arduino and moving on to direct AVR programming is a great approach. Get Elliot Williams' book Make: AVR Programming for the latter.
3) Arduino programming is deceptively easy but extremely powerful if mastered properly. Do your sample programs first on Arduino and then redo it using straight AVR-C. This will teach you how to prototype something quickly using Arduino and then when you have strict cycle/memory/power/latency requirements how to drop down to the metal and program exactly what is needed.
Finally from the career pov, you will become one of the few who can do and understand everything from bare metal all the way to processing in the "Cloud". As an example; read sensor data using Arduino, send it over the Internet to the "Cloud", Apply ML algorithms and do "Predictive Analytics". This is Industry 4.0/IIOT etc.
I’d be inclined to process everything locally right up until the 1TB mark, and that’s only because my computer doesn’t have enough disk space.
That’s really the money quote, right there.
When we depend on the work of others, we also depend on the people and organizations behind that work. You really can’t get much bigger and more solid than Amazon (or Google, who are also infamous for drowning their babies), so “bus factor” is really a meaningless metric, when it comes to whether or not to depend on a service. I've been experiencing rug-pulls for most of my career (I'm an Apple developer, and Apple does this frequently).
I recently had to stop working on the app that I’m developing, and spend a month and a half, writing a new backend server and SDK, because the developer of a server I had depended on, has ghosted our project. I understand why they did it (life happens), but the reasons make no difference. They still ended up leaving us in the lurch. I actually have a huge amount of respect for the author, count them as a personal friend, and sincerely wish them nothing but the best.
That project ticked a lot of Buzzword Bingo boxes: Postgres, postgis, containers, Django, Python, etc. Also, the main author is fairly young.
But it was painfully slow. It was killing our app. It’s entirely possible that we could have sped it up, but it required working closely with the author for a while, and that relationship withered, as noted.
So I wrote my own server, using “boring” tech; PHP, MySQL, etc. It’s lickety-split fast, secure, simple, well-documented, maintainable, adaptable to multiple data inputs (the original server was for a single data source), and doesn’t require a container to deploy. It’s also fairly easy to switch the DB tech, and it can be deployed on a wide range of low-cost servers, which helps, as it’s a free app, for a nonprofit.
I’m not against dependencies, despite frequent sneers and insults, assuming that I’m some “out of touch boomer,” clutching the past. I think that modular software development (a very old technique, by the way) is how we can do high Quality, at scale, and rapidly. In fact, that’s exactly how I work.
It’s just that I tend to write my own dependencies. I have a fair bit of faith in myself.