jordanbeiber

What it used to be:

1. The rigidness of it all, which after a while is the main point ofc. Takes getting used to, but awesomeness for servers

2. The highly specialized filesystem layout requires some tinkering when trying to “just get things to work” that is not already packaged/wrapped for nixos

3. The language is… special. An acquired taste perhaps.

4. Flakes, home-manager, experimental flags galore.

LLM’s makes short work of it all, and having a devenv.sh setup of all your projects is :chefs-kiss:

My machines are essentially solid-state devices, and it all just becomes extremely predictable and stable.

I’ve been using NixOS exclusively the last few years. Its been a blessing and a curse!

My framework laptop, my NAS, my Home-assistant, etc

It’s how I’ve always envisioned “systems management”, but the struggle has been real.

Enter LLM’s - the barriers are kind of gone!

It’s actually making so much sense I don’t know where to start.

Just try it out, using a little bit of LLM support and you can reach a solid-state like setup for servers and workstations.

Well, if you work in an enterprise setting, and if this setting is just right, you can get paid well to build a lot of interesting things, and have all the choices in how to build.

Certainly not every enterprise setting is like this, but they do exist, or the setting can even be created if just enough people of the right stuff end up together.

I have kind of the same personal experience - a couple of days answering questions about the scalability and resilience of a fictional web app and enterprise integrations.

I’m very much a generalist and kept to standards and terms not specific to AWS infra. They ”passed” me but wanted a follow-up after going through some training packages to make sure I could translate everything to AWS.

At the time I was leaving a bigger software company , largely because I felt we completely locked down customers in solutions perhaps not in their best interest - so this put me off.

I realized AWS would be just the same, just on a different level.

And of course this makes sense - they want to sell their stuff!

I don’t want to sell stuff, I just want to build things.

Exactly this. Having spent almost three decades in enterprise context I see a lot of reinvention of something like a poor mans, unstructured, enterprise architecture - because AI agents.

I keep repeating ”what is good for humans in an organization is also good, or even required, for AI agents”.

Imagine every new instance of an AI agent as a new employee. With humans its ok to slowly accumulate knowledge through word of mouth, trail and error and the general inertia of larger orgs almost seem structured (or unstructured) knowledge-wise for this.

AI agents will never be useful in high value operations in a larger orgs without organizational knowledge available and reliable.

There’s a place for everything.

Most coding tasks take place outside of pure tech companies, if I’d venture a guess.

And let’s be honest, enterprises in general do not value that quality - and they face very little in terms of technical challenges that can’t be solved by code on stack-overflow or github.

What most enterprises lack is knowledge about themselves though - this is more a business problem than a technical one however.

This is most likely due to the fact that it is really bad at resetting blinker when the steering wheel is straight’ish again. Extremely annoying as any other car is much more sensitive (and sensible).

In a tesla an on-ramp to straight highway is rarely enough to stop the blinker, something I’ve never experienced in any other car.

Couple this with, IMO, the best baseline speaker system of any manufacturer… I’ve been driving with the blinker on for several kilometers at times!

Even if we still make a mess I think centralization of the mess is better than distributing it - what I mean is that polluting cities where millions sleep, eat, drink and breathe will probably be worse, net effect, than containing energy pollution to select places.

Running EVs in densely populated regions is probably a lot better for the population on the whole even if the net pollution would stay the same, IMO.

Still no EV is even better, but we’ve created a world where transport is often required so, one step at a time I guess.

Using AI doesn’t really change the fact that keeping ones and zeroes in check is like trying to keep quicksand in your hands and shape it.

Shaping of a codebase is the name of the game - this has always been, and still, is difficult. Build something, add to it, refactor, abstraction doesn’t sit right, refactor, semantics change, refactor, etc, etc.

I’m surprised at how so few seem to get this. Working enterprise code, many codebases 10-20 years old could just as well have been produced by LLMs.

We’ve never been good at paying debt and you kind of need a bit of OCD to keep a code base in check. LLM exacerbates a lack of continuous moulding as iterations can be massive and quick.

The problem is most likely not writing the actual code, but rather understanding an old, fairly large codebase and how it’s stitched together.

SO is (was?) great when you where thinking about how nice a recursive reduce function could replace the mess you’ve just cobbled together, but language x just didn’t yet flow naturally for you.

The argument is perhaps ”enshittification”, and that becoming reliant on a specific provider or even set of providers for ”important thing” will become problematic over time.

As go feels like a straight-jacket compared to many other popular languages, it’s probably very suitable for an LLM in general.

Thinking about it - was this not the idea of go from the start? Nothing fancy to keep non-rocket scientist away from foot-guns, and have everyone produce code that everyone else can understand.

Diving in to a go project you almost always know what to expect, which is a great thing for a business.

”Same here” meaning moving to Hetzner, but from Azure - could’ve made it less ambiguous!

Might throw together a post on it eventually:

https://news.ycombinator.com/context?id=43216847

Same here, but Azure. About 90% saved, with a very similar stack.

It is a great big cloud play to make enterprises reliant on the competency in their weird service abstractions, which is slowly draining the quite simple ops story an enterprise usually needs.

Cursor have a nice ”docs” feature for this, that have saved me from battles with constant version reversing actions from our dear LLM overlords.

We’ve gone full-on full-code.

Although we’re using temporal to schedule the workflows, we have a full-code typescript CI/CD setup.

We’ve been through them all starting with Jenkins ending with drone, until we realized that full-code makes it so much easier to maintain and share the work over the whole dev org.

No more yaml, code generating yaml, product quirk, groovy or DSLs!

Of all the paas providers Azure have the worst abstractions and services.

In general I think it’s sad that most buy in to consuming these ”weird” services and that there’s jobs to be had as cloud architects and specialists. It feeds bad design and loose threads as partners have to be kept relevant.

This is my take on the whole enterprise IT field though!

At my little shop of 30 so developers, we inherited an Azure mess, built abstractions for the services we need in a more ”industry standard” way in our dev tooling, and moved to Hetzner after a couple of years.

A developer here knows no different, basically - our tooling deals with our workflows and service abstractions, and these shouldn’t change just because new provider.

1/10-th of the monthly bill, and money partly spent on building the best DX one can imagine.

Great trade-off, IMO!

Only two cases come to mind for using big cloud:

- really small scale: mvp style

- massive global distribution with elasticity requirements.

Two outliers looking at the vast majority of companies out there.

Writing the code is NOT the problem with these enterprise project failures.

Usually decades of problem-solving have led to an absolute mess of blurry ownership and accountability.

This in turn leads to corner cutting and a road completely covered in Chesterton fences…

Tearing arbitrary fence down leads to consequences out of project scope, no one can answer questions, and no one can prioritize - this is a business problem, and no amount of fancy code (lo/hi/full/lo/left or right) will help.

If you run a bigger company and rely on IT and ERP flows, well, it’s a part of your core and you’d better treat it as such!

CRUD applications are not what most enterprises build though - they need interconnected flows of data & state-machines often supporting rather complex workflows and integration patterns.

Not realizing this is a mistake many enterprise IT shops make.

The boring crud-thingie that someone hacks together will sooner rather than later have to be integrated in a distributed system of state - this is where it gets hairy and most enterprises get stuck.

Just take the foot off the accelerator gradually.

The kia I drive still roll at a couple of km/h when accelerator is fully released and will not hit 0 unless you use a paddle by the steering wheel (or use the breaks).

But you rarely have to brake an EV though.

I’ve had two EV’s over the last 6 years, and I have to remember to occasionally hit the brakes to keep them from rusting.

One pedal driving is just money - saves power and is just much smoother over all than old school breaking. IMO.

In my experience people usually spend to much time trying to figure out what is most important instead of just doing an important thing.

But you wrote the docs and documented it’s place in the enterprise knowledge graph, because you’re a professional, right?

As an eng manager at an enterprise (not SV style tech) I differentiate between coders and developers.

Developers help our business by actively taking part in shaping software, automated workflows and integrations that makes us improve as a whole.

At lot of time is spent documenting and figuring out how that piece of software someone wrote 7 years ago, before us, works and what integrates.

This is why I firmly believe you need to tie documentation to running code in some way - at the very least a basic service graph that contains all services, dependencies and some metadata.

Tie this database to automatic firewall and proxy configuration and you’re on a good track.

We’ve on the other hand taken the strategic decision that tech and IT will help us win (is it the 90’s all over again?!) and we staff as such. A team is at least two, and no individual can own a service.

As an engineering manager I see problems with authority and process as something usually positive.

This usually leads to more things getting done “right” than “wrong”. IME.

Having the same issues/traits I’m not sure how that gets formed - my upbringing was limitless in many ways.

Question is - what are you using airflow for? My experience with airflow have been in data ETL, and if so you are not the target for something like windmill.

The target would most likely be automating HR, Finance and IT workflows and tearing down the shadow IT web of crazy integrations taking place at every larger organization I’ve ever experienced.

We’re talking “new hire” workflow for example, which at my current employer is about 25 activities in a workflow.

All assets have to be lifecycle managed in an enterprise and automated workflows will help you scale that. Far too many enterprises have a lot of people shuffle excel files and emails around to fulfill processes and workflows.

Airflow is a beast imho and usually not used in the same niche IME.

I’ve worked for years implementing tools like this in enterprises.

My most fond memory was introducing a workflow platform to an enterprise with a fully outsourced IT-ops department - it was ruining everyone else in terms of cost, speed and quality.

The security dept (this was a large bank) was gridlocked in this setup and wanted the ability to automate their way out of the sourcing mess.

I spent roughly three months building a few “hot path” workflows important to them which enabled them to take the ownership back of the processes and save an incredible amount of time and money.

Encapsulating these integrations as workflows makes them observable and measurable. The customer had in the first quarter after deployment 10’s of thousands runs and avg time to completion went from 2 weeks to 2 days. It also cut out an rather expensive middle man.

And this is not the worst enterprise customer I’ve worked with. One hade 4000 Windows servers manually provisioned and managed. There’s low hanging fruit out there!

You basically trade agility and quality for competence, unfortunately a lot of enterprise IT shops are not willing or capable to do so.

They have “not only one way or the other” - they have “all the ways” and this is exactly the problem you want a solution like this to fill.

We’re talking backoffice/cost-center workflows in IT, finance and HR - absolutely not business profit processes mind you.

A tool such as this can help an IT department take ownership of integrations and workflows, building them in a framework that can improve speed and quality. It will help with organizational scalability.

The alternative is in my experience a giant mess of integrations lacking ownership and observability.

Every large enterprise needs “something” like windmill, they might just not know it.

In a way it is (and also a humanity problem).

It would be great if a bunch of car parkings could be converted to secure bike storage spots.

> And I've had several bike stolen over the years (include being mugged with my bike)

So, let me get this straight - it’s quite likely a bike gets stolen and this experience, as shared with me, has led you to take precautions.

My precaution being not taking the bike some times and to some places.

How many times have you have a car stolen over that same time period?

I can emphasize “feels”, but if you’ve ever parked a moderately expensive bike in a dense city center for a few hours you know the feeling. Especially if you’ve had a few stolen. If not the bike, the saddle. If not the saddle a wheel. If not the wheel the battery. Etc etc. It’s all really annoying trying to get home not matter the insurance.

I had two strollers stolen from my backyard while at home last year… it’s kinda crazy over here!