Honeypot.
1,465 karma · joined August 25, 2017
Correct. "Shove it" is usually preceded by not liking something.
> They could negotiate with their manager to lessen the load.
Most of the time the manager will simply refuse. As a business owner it's my decision.
> They could upgrade the systems.
At big companies this is usually outside the scope of an on-call engineer. The on-call engineer often doesn't even have commit rights to that repository.
The specific example I gave was paying $10/month more. That can be a very hard sell at a large company because their service contracts are much more complicated/expensive.
> They could straight-up refuse on call.
A business owner has much more negotiating power than an employee does.
> They don't because they don't like the consequences of taking these options—and neither does the self-employed person!
In the vast majority of cases making changes to the on-call infrastructure has very little (if any) measurable impact on the business. Like spending a week making the systems better. Or changing deploy/release dates to be more convenient.
As a business owner I can take advantage of this and make my life easier.
As an employee I have layers of bureaucracy to wade through and will probably be refused. Not because it affects the business but for other reasons.
That's the difference.
Let's say I'm a business owner and I'm frustrated with the current state of the on-call system. I have options.
I can try negotiating with my clients to lessen the load in some way. Obviously this isn't always possible but it often is. I once had a freelance project that required 24 hours of on-call after a release. I negotiated release days that were convenient for me (never Fri/Sat/Sun). One time the client pushed back, I pushed back harder, and I won. In order for my push back to work I ensured that I had enough negotiating strength to do that which I planned for ahead of time.
I can upgrade my systems. For example if my current paging system is insufficient I can choose to pay $10/month more for another system that makes my life easier. I can set aside time to refactor my alerts code to make my life easier and I don't need to justify it to anyone but myself.
I can straight up refuse to do on-call and deal with the consequences to my business. Freelancer developers do this all of the time. We choose which client work to do and not to do. We can make these choices arbitrarily. Sometimes it's seasonal. Sometimes it's just based on vibes. Doesn't matter; it's our company.
Meanwhile the average on-call engineer at a large company has none of these freedoms. The underlying systems are chosen for them and they just have to deal with it.
It's quite easy to imagine a well factored codebase where all things are neatly separated. If you've written something a thousand times, like user authentication, then you can plan out exactly how you want to separate everything. But user authentication isn't where things get messy.
The messy stuff is where the real world concepts need to be transformed into code. Where just the concepts need to be whiteboarded and explained because they're unintuitive and confusing. Then these unintuitive and confusing concepts need to somehow described to the computer.
Oh, and it needs to be fast. So not only do you need to model an unintuitive and confusing concept - you also need to write it in a convoluted way because, for various annoying reasons, that's what performs best on the computer.
Oh, and in 6 months the unintuitive and confusing concept needs to be completely changed into - surprise, surprise - a completely different but equally unintuitive and confusing concept.
Oh, and you can't rewrite everything because there isn't enough time or budget to do that. You have to minimally change the current uintuitive and confusing thing so that it works like the new unintuitive and confusing thing is supposed to work.
Oh, and the original author doesn't work here anymore so no one's here to explain the original code's intent.
As soon as you split 1 repo into 2 repos you need to start building tooling to support your 2 repos. If your infrastructure is sufficiently robust with 2 repos then you might as well have 3 or 4 or 10. If it's built to _only_ support 2 repos (or 3 or 4) then it's brittle out of the gate.
The value of a monorepo is that you completely eliminate certain classes of problems and take on other classes of problems. Classic trade off. Folks that prefer monorepos take the position that multirepo problems are much harder than monorepo problems most of the time.
Hahaha
My major issue with your position is that, at least in my experience, good software is the sum of even the seemingly low risk parts. When I think of real world software that people rely on (the only type I care about in this context) then it's hard to point a finger at some part of it and go "eh, this part doesn't matter". It all matters.
The alternative, I fear, is 90% of the software we use exhibiting subtle goofy behavior and just being overall unpleasant to use.
I guess an analogy for my concern is what it would look like if 60% of every film was AI generated using the models we have today. Some might argue that 60% of all films are low stakes scenes with simple exposition or whatever. And then remaining 40% are the climax or other important moments. But many people believe that 100% of the film matters - even the opening credits.
And even if none of that were an issue: in my experience it's very difficult to assess what part of an application will/won't be low/high stakes. Imagine being a tech startup that needs to pivot your focus toward the low stakes part of the application that the LLM wrote.
The most financially rewarding project I worked on started out as an early stage startup with small ambitions. It ended up growing and succeeding far beyond expectations.
It was a small codebase but the stakes were still very high. We were all pretty experienced going into it so we each had preferences for which footguns to avoid. For example we shied away from ORMs because they're the kind of dependency that could get you stuck in mud. Pick a "bad" ORM, spend months piling code on top of it, and then find out that you're spending more time fighting it than being productive. But now you don't have the time to untangle yourself from that dependency. Worst of all, at least in our experience, it's impossible to really predict how likely you are to get "stuck" this way with a large dependency. So the judgement call was to avoid major dependencies like this unless we absolutely had to.
I attribute the success of our project to literally thousands of minor and major decisions like that one.
To me almost all software is high stakes. Unless it's so trivial that nothing about it matters at all; but that's not what these AI tools are marketing toward, are they?
Something might start out as a small useful library and grow into a dependency that hundreds of thousands of people use.
So that's why it terrifies me. I'm terrified of one day joining a team or wanting to contribute to an OSS project - only to be faced with thousands of lines of nonsensical autogenerated LLM code. If nothing else it takes all the joy out of programming computers (although I think there's a more existential risk here). If it was a team I'd probably just quit on the spot but I have that luxury and probably would have caught it during due diligence. If it's an OSS project I'd nope out and not contribute.
It reminds me of a time (long ago) when the trend/fad was building applications visually. You would drag and drop UI elements and define logic using GUIs. Behind the scenes the IDE would generate code that linked everything together. One of the selling points was that underneath the hood it's just code so if someone didn't have access to the IDE (or whatever) then they could just open the source and make edits themselves.
It obviously didn't work out. But not because of the scope/scale (something AI code generation solves) but because, it turns out, writing maintainable secure software takes a lot of careful thought.
I'm not talking about asking an AI to vomit out a CRUD UI. For that I'm sure it's well suited and the risk is pretty low. But as soon as you introduce domain specific logic or non-trivial things connected to the real world - it requires thought. Often times you need to spend more time thinking about the problem than writing the code.
I just don't see how "guidance" of an LLM gets anywhere near writing good software outside of trivial stuff.
I'm willing to accept that I just have "get off my lawn" syndrome or something.
But the idea of letting an LLM write/move large swaths of code seems so incredibly irresponsible. Whenever I sit down to write some code, be it a large implementation or a small function, I think about what other people (or future versions of myself) will struggle with when interacting with the code. Is it clear and concise? Is it too clever? Is it too easy to write a subtle bug when making changes? Have I made it totally clear that X is relying on Y dangerous behavior by adding a comment or intentionally making it visible in some other way?
It goes the other way too. If I know someone well (or their style) then it makes evaluating their code easier. The more time I spend in a codebase the better idea I have of what the writer was trying to do. I remember spending a lot of time reading the early Redis codebase and got a pretty good sense of how Salvatore thinks. Or altering my approaches to code reviews depending on which coworker was submitting it. These weren't things I were doing out of desire but because all non-trivial code has so much subtlety; it's just the nature of the beast.
So the thought of opening up a codebase that was cobbled together by an AI is just scary to me. Subtle bugs and errors would be equally distributed across the whole thing instead of where the writer was less competent (as is often the case). The whole thing just sounds like a gargantuan mess.
Change my mind.
My issue with most AI discussions is they seem to completely change the dimensions we use to evaluate basic things. I believe if we replaced "AI" with "new useful tool" then people would be much more eager to adopt it.
What clicked for me is when I started treating it more like a tool and less like some sort of nebulous pandora's box.
Now to me it's no different than auto completing code, fuzzy finding files, regular expressions, garbage collection, unit testing, UI frameworks, design patterns, etc. It's just a tool. It has weaknesses and it has strengths. Use it for the strengths and account for the weaknesses.
Like any tool it can be destructive in the hands of an inexperienced person or a person who's asking it to do too much. But in the hands of someone who knows what they're doing and knows what they want out of it - it's so freakin' awesome.
Sorry for the digression. All that to say that if someone believes it's a productivity boost for them then I don't think they're being misled.
Employees are not robots. They are human beings. Sometimes human beings have human problems that need the assistance of other humans. This makes humans happier and more productive.
It's depressing to think that there are people who actually believe that optimal use of work resources is even worth calling out as an issue. In 2024.
2) Rats nest of JavaScript callbacks.
3) Overlapping stylesheets with !important everywhere.
4) Elements used for style not their semantic purpose (<b>, <strong>)
5) Subtle and not-so-subtle browser compatibility issues.
The horror!
I love ambient tunes and listen to them for 1-10 hours a day while doing stuff. Any time I need to focus (even when writing an email) I'll throw on my favorite ambient music.
How is this any different? Why would I get a "premium" account for some random website when there's an entire catalog of ambient music on Spotify/Apple/etc that I could listen to?
The conception of an idea and the sharing of that idea could be completely different skills in some cases. I'm reminded of the classic Jobs/Wozniak duo that we often see repeated in the tech industry. There's definitely something there.
Claiming that a horse is majestic is not an "idea" in the "product idea" or "feature idea" sense.
But let's assume for a moment that someone imagines a majestic creature in their mind. The person sees this beautiful majestic creature in their head. They can't wait to tell everyone about this beautiful idea. And when they decide to share it... they share an objectively ugly ass stick figure drawing...
Do you see the problem?
How one presents an idea is critical in validating it for a broader organization or society. If the person with the idea can't tell the difference between an ugly ass stick figure drawing and the majesty in their mind then how are you supposed to trust their vision? Their understanding of majesty is obviously disconnected from reality.
Okay let's translate this to something more real world.
Suppose you're a principal engineer or architect or whatever those people are called these days. You come in to work on a Monday and a mid-level engineer asks to meet you because they have a great idea. They were so excited that they even created a little demo over the weekend to show it off.
The demo goes horribly. Every step of the way something breaks. The engineer can't clearly describe what the idea is as they stumble through layers of complex problems and solutions. The meeting ends and you walk away confused and mildly annoyed at having your time wasted.
Perhaps the engineer did discover something interesting. But if they're not able to distill a demo into what makes their idea awesome then it would be irresponsible for you to blindly trust that the idea would work in practice. Now obviously in this context "idea" doesn't mean "use this function instead of that function and you get a 10x perf boost". That's simple. I'm talking about something more complicated like a new paradigm or way to architect your systems.
My point is this. Presentation matters. It doesn't need to be pretty. You don't need to be eloquent. But when you present an idea it needs to do an exceptional job of demonstrating itself to peers. It's not the job of your peers to unpack your brilliance - that's on you.
Why would I do that?
There is absolutely no reason a single application should be using two versions of Typescript at the same time. What you're talking about is a combinatorial explosion of packages and versions - it's bad, bad, bad for software quality.
Upgrade paths can be done gradually. I've done and seen it done more times than I can count.
> For how long? Until it gets to what size?
That depends a great deal on the type of software and how it was engineered.
If you inherit a mess and your engineering team is incapable of not creating a mess then, sure, drawing heavy handed microservice boundaries might make sense. But in that case you're solving an organizational problem not a technical problem. All of the technical benefits you've been claiming are moot.
> So generating multiple executables from one repo? That is a monorepo, if you have a bunch of processes that communicate with each other through some channel, be it pipes or sockets or whatever, then I'm going to argue you have microservices. They may be running on one machine, but again, you have multiple processing talking to each other.
You could generate multiple executables. Or you can generate a single executable that's able to run in different "modes" (so to speak).
What you've shown throughout your writing is you don't quite understand what microservice architecture is. Multiple applications communicating over the network (or whatever) is NOT a "microservice". Why wouldn't you just call that a "service"?
A microservice is a very small application that encapsulates some "unit" of your larger application. Where that boundary is drawn is obviously up for debate.
I don't wanna digress but I can assure that two applications on the same server talking over a network is NOT a microservice. That's just... like... software, lol.
> Now I'll also argue that such code bases are more likely to have strong coupling between components. It doesn't have to be that way, but it becomes harder to resist (or to just stop junior devs from inadvertently doing).
Creating a network boundary (a HUGE performance cost) just to prevent people from doing stupid stuff is not good software architecture.
> So one executable with multiple network endpoints exposed? Sure. But you have lost some security boundaries in the process. If the only system that can ever have customer PII lives inside a firewall with no public IP address, you've gained security VS PII floating around in the same process memory address as your public endpoints.
Your PII lives in a database. Restrict read access to that database (or subset of that database) to applications running on the internal network. The most straightforward way to do that would be through configuration management.
Applications accessible to the outside world will never even have read access.
With microservices, there is nothing stopping some tiny internal service from exposing data to a public service. Maybe the internal service wrote their ACLs wrong and the data leaked out.
What you're describing, again, has nothing to do with microservices and is a problem you need to deal with either way.
PII is a data problem not a code problem. Microservice/monolith is a code problem not a data problem.
> Also, the Unix philosophy is small dedicated programs that do one thing really well. Kind of like... microservices.
So now all programs that communicate over the network are microservices. Do you not see how silly that sounds?
> Vs a monolith where you have everything running on one machine under one process.
That's not how anyone deploys monolithic applications. You're now trying to claim that monolithic applications only run one server. Confusing.
> Different problem domains are best solved with different programming paradigms.
That's why we create multiple applications that do different things. Microservices are something totally different.
> If you have well defined interfaces you don't need atomic builds. That is kind of the entire point! Unless there are some legal requirements which mandate "blessed builds".
The benefit of atomic builds is it's very easy to reproduce the behavior of your application at any given moment.
So, for example, a user reports a difficult to find bug and you're only investigating it 2 weeks later. You'll need to rewind your application's "code state" to whatever it was at the time that the user was using your application. For a monolithic application this is as easy as pointing the monolithic repo to some commit SHA.
With microservices this is much harder to do without tooling. This can be for many reasons. Sometimes every microservice is a separate repo. Sometimes your development environment isn't running every microservice.
> Pure functional peeps say no state at all, and arguably the functional paradigm often works, but the performance hit is insane at times.
I'm not even that into FP and this is painful to read. Such a gross misrepresentation of what FP is about.
> Microservices say you need to decouple components and that components can only talk through preestablished and agreed upon messages.
You mean like function calls?
> It enforces this decoupling by isolating different bits of code on the other sides of boundaries
You mean like... visibility? As in public/private interfaces?
> so that network connections of some type have to be used to communicate between modules
Enforcing correctness by placing a network boundary is too heavy handed. There are other ways to achieve the same thing that doesn't involve adding orders of magnitude latency for no reason.
> You get some benefits with this, I still maintain it is easier to update to new breaking changes of dependencies on smaller chunks of code rather than adopt a breaking change to a global dependency across possibly hundreds of thousands of lines of code.
You do realize that large software existed before microservices became a thing right? And it was maintainable, right? There are so many ways to solve this problem without affecting thousands of lines of code blindly.
There's also just as much risk in having 10 services that are running slightly different versions of the same dependency. In fact that's a freaking nightmare.
> Because you only need to maintain the API contract, microservices can be deployed in a true CI/CD fashion, with teams constantly releasing changes to their services throughout the day, every workday.
Sir, I'm afraid of you have a bad case of buzzworditis.
$ mkdir hello && cd hello
$ npm init -y
$ npm install react17@npm:react@17
$ npm install react18@npm:react@18
$ cat "var react17 = require('react17'); var react18 = require('react18'); console.log(react17.version, react18.version);" > index.js
$ node index.js
$ node index.js
17.0.2 18.2.0
> That website runs in its own microservice with a customized version of the org's build system, something that was possible because as an organization we have scripts that allow for the easy creation of new services in just a matter of minutes.
I don't see what this story has to do with microservices. That kind of velocity can easily be achieved with a single codebase too.
> Scaling, different parts of a system need to scale based on different criteria.
That's not at all unique to microservices.
A monolithic application can run in different modes. For example of if you run `./my-app -api` then it'll start an API server. If you run `./may-app -queue` then it'll run the message queue processor. And so on.
This way you can run 10 API servers and 50 queue processors and scale them independently. How an application is deployed isn't necessarily tied to how it's built.
> Another cool feature of microservices is you can choose what parts are exposed to the public internet, vs internal to your network. Holy cow, so nice! Could you do that with a monolith? Sure, I guess. Is it as simple as a command line option when creating a new service? If you have an absurdly well defined monolith, maybe.
I'm confused.
Is there some magical program called "microservice" that takes command line options? What does a systems architecture approach have anything to do ease of deployment?
The whole public/private networking thing is an infrastructure decision. Your monolithic application could easily have internal endpoints that are only exposed when certain flags are set.
> With microservices, impedance mismatches are hidden behind network call boundaries. Yes you can architect monoliths in vastly different fashions throughout (and I've done such), but there is a limit to that.
What are those limits praytell?
> E.g. with microservices you can have one running bare metal written in C++ on a hard real time OS, and other written in Python.
That has nothing to do with microservices. You can thank the UNIX architecture for that. Most computers run more than one program written in more than one programming language.
> Oh and well defined builds and deployments is another thing I like about microservices. I've encountered monoliths where literally no one knew how to completely rebuild the production environment (I overheard from another engineer that Xbox live services existed in that state for awhile...)
So it was architected poorly. Why is that a strike against monoliths? Are you saying that messy builds are impossible with a microservice architecture? One of the top arguments against microservices is how much of a rats nest they are in production - in practice.
Some companies are able to do it right with discipline and good engineering. The same can be said for monoliths.
By the way the big problems with microservice architectures is you don't get atomic builds. Very difficult problem to work around. Usually with lots of tooling.
It'd be nice to actually see billions of cells in action. Otherwise it's just marketing for their product disguised as a technical blog post. For all we know it doesn't actually work that well in practice.
C is a small language. There are benefits to that. But it also has a handful of historical oddities. Innovation here means to keep C small while also getting rid of those quirks.
C++ is enormous. Rust is headed in the direction of similar enormity.
> With a monolith, dependency updates, especially breaking ones, often mean either all development stops for a "code freeze" so the update can happen, or you have a team responsible for doing the update and they are trying to update code faster than other devs add new code.
In all my years I've never seen a code freeze due to a dependency update. Maybe the project you were working was poorly engineered?
> The result of this is that updates get pushed back to the last minute, or are never just done. I've seen old (ancient) versions of OpenSSL checked into codebases way too often.
There should be nothing stopping you from running multiple versions of a dependency within a single monolothic project.
> With microservices, you can have a team that isn't as busy take a sprint to update their codebase, carefully document best practices for fixing breaking changes, document best practices for testing the changes, and then spread the learning out to other teams, who can then update as they have time or based on importance / exposure of their maintained services.
Gradual adoption of new dependencies has nothing to do with microservices.
For the types of applications built with Django, async is actually more of a hinderance than a benefit. Async I/O can create massive back pressure in a distributed system.
A simple example would be a web service that gets a massive influx of traffic (organic or DDoS, doesn't matter). If the web service is using async I/O to manage database connections with an unlimited connection pool (often the default) then it will happily accept all incoming requests and push all of that pressure onto the database. The database gets overwhelmed because it's inherited all of that traffic. The database starts refusing connections not just to the web service but any service that connects to the database. And boom, you have system wide outage.
Obviously the database should guard against this kind of overwhelming traffic but that's often thought about last.
Diagnosing the problem becomes a lot harder since your bottleneck is further downstream. It gets messy.
Synchronous services on the other hand create a nice throttle. Your Django app is never going to achieve the kind of throughput that your database will. So, by design, your Django app will get overwhelmed first because it won't accept new connections (eg: maybe it runs out of memory).
That being said I still think it makes sense for Django to support it. Not for a chat app, but say... web sockets. You may want a websocket connection that pushes notifications back to a single-page-app. The notifications will contain a bunch of contextual data that the Django framework provides easy access to (think ORM, templates, etc). Obviously you could build the websocket thing as a separate service but then you lose all the goodies that Django provides to you.
There's a balance and it's more delicate than people give it credit these days. Don't get me wrong though... over-engineering and/or premature optimization is just as bad. But there is an opposite extreme as well.
When your business application reaches a point of "non-triviality" (for lack of a better term) you'd (hopefully) realize that you need to ditch Node ASAP or face growing pains.
To understand why requires some historical context.
During the time when Node was becoming popular most scripting languages had weird bottlenecks at the request level.
PHP: When a request was received, your whole application would load into memory. You know how an application might read a config file on startup? Yea, every single request would load that file. There was no concept of "global variables across all requests". That's one OS process per request. Sometimes process forking was used to make that faster but it didn't matter that much because your entire application needed to load for every request to be handled.
Python: You'd use a toolkit like uWSGI to launch N number of processes when your server started. The processes would stay up and continue to serve requests. You'd avoid the overhead of initializing your app on every request, but you were still locked to one process per request because of the GIL.
Ruby: Basically the same as Python.
Java, C++, Go, etc: Too slow to develop in, too complicated for rapid iteration. Type systems scary. Weak dynamic typing fun. Productivity was king and you needed some time in the hard languages to git gud.
Okay so the fundamental problem with PHP/Python/Ruby was that you needed to tie up a single OS process per request. Let's say you wrote an HTTP API that would fetch the current time from time.gov or something. While your HTTP handler was fetching data from time.gov that OS process would be STUCK. It's sitting around and doing nothing while holding your whole application state in memory just for that one request you're serving.
This wasn't actually a problem, not really. You could serve an ridiculous amount of web requests using this model. Computers are fast! Until... you needed concurrency... like a chat app. Because in a chat app you need to keep a TCP request open for every single user in the system. 500k active users? 500k OS processes just sitting there doing nothing with all of your application state in memory. Not a good fit for PHP/Python/Ruby where every open request was tying up all those resources.
So how would you solve it in C++/Java/etc? Easy, you'd use non-blocking IO. With some fancy connection pooling, threading, and use of the epoll_wait syscall (or whatever) you could handle all 500k users with a SINGLE OS process. That's because your chat app is really just routing bytes between client applications. Most of the time is spent in I/O, not in your application.
Except non-blocking IO is not easy. Threading is not easy. Connection pooling is not easy. Type systems, code compilation, etc etc are not easy.
Enter NodeJS.
The premise is simple. Node is a single threaded runtime environment that does all IO using non-blocking IO. All accessible via a well known language called JavaScript. No memory management, no types - oh and functions are first class primitives. Nice. Oh and don't try to calculate the Nth prime number because that'll block your whole single threaded application even if it's serving 500k requests.
All of the early demos of Node (that I saw) were basically chat apps. Applications with very little business logic that spend most of their time doing IO. It was more of a glorified router of bytes...
But then people realized that CPUs are actually really fast and you could start modeling basic web apps. All you're doing in most web apps is pulling some data out of a DB, decorating that data, and pushing it out of NodeJS. Single threading is no problem. So people starting building on it... and building on it... until they needed to scale for real.
Some unfortunate souls didn't realize any of this and built complex business applications in Node. They struggled a lot with the single-threaded nature of this environment. That's when you started seeing things like Node Cluster which basically used threading under the hood to distribute some of that CPU load. Over time better alternatives came along.
So the reason you don't see big complex frameworks in Node is because Node doesn't need them. Using a big complex framework means you're shoving way too much business logic into a technology that wasn't designed for it.
Express and Koa are basically peak Node "frameworks". They're effectively just syntax sugar for cleanly routing your requests somewhere else. Need something more than that? Look elsewhere or you'll regret it later!
EDIT: Fixed a bunch of typos. Did not expect to write so much.
Great point about package management. Go already had it figured out. Node was busy figuring it out (npm shrinkwrap; still sucked but less so).
Any time I talked to people about Python the conversation was about 2to3 and core team drama lol