Long Term Software Development

Is it? It seems like all the outsourced auth has lots and lots and lots of problems, issues, etc. along with being remarkably expensive.

I'm not saying I, personally, can do better, but it seems like auth is far from a solved problem. Especially since it seems like the difficult part of "auth" is actually "customer support" rather than any technical issue.

(I would argue that the lack of an open-source "Auth in a Box" seems to also argue that auth is far from "solved".)

It is solved until you have to integrate with third party home grown implementations and providers that implement the specs except for a little bit of behavior that is not in the spec.

And while I kind of agree with you on principle, I've also seen way too many shoddy OSS libraries from "experts" that I had to rewrite to really make it a no-brainer statement.

Just like OP, I'm leaning against shoveling in dependencies if I can help it as I age.

Imagine a doctor making such statement about their field of work. Do you have a degree?

I have been trying to get my head around OAuth, zero trust, Fido, SAML et al - and I am not sure I get it.

It’s not that say “JWT tokens might be dangerous” it’s how to hang it all together in a sane manner.

For example I was listening to Security Cryptography Whatever podcast and one of the guests (Chrome dev?) said obviously don’t trust everything that comes down an established TLS connection but verify / auth each request.

That’s a pretty tall order and seems to nullify quite a lot of established frameworks

But realistically I think a lot of frameworks and standards serve different needs than “smallish number high value clients”

This is a great resource to learn how to write it yourself, and things to take into consideration: https://lucia-auth.com/

IMO, this is one thing to implement carefully if need be (vs avoid altogether, as is the case of cryptography).

Probably because auth abstractions are so complicated is why developers started thinking auth is hard when all it was doing was putting a uuidv4 in a session_id cookie.

Like writing SQL without an ORM, developers have been psyop'ed into thinking things are so hard that they shouldn't even try.

All the projects you listed (Pandas, PyTorch, SKLearn) have a huge list of compatibility issues, on top of compatibility issues of Python itself. And I know this because I have to support various research projects, which typically use this stuff. My estimate is that the shelf-life of a project using just these three will not exceed 3 years in >90% of cases. My typical situation with these projects is that I'm called in because some former PhD student, who graduated a year or two ago left a project that now another student is picking up. And nothing works. Results cannot be reproduced. The dependencies cannot be installed. And so on.

And then, depending of the amount of effort I'm allowed to put into fixing the project, I end up either crafting some sort of a spec that tries to find a set of versions of used libraries that seem to make the project work again. Or, more commonly, have to vendor some code and add fixes on top to make the packages fit together at least somehow. Sometimes I even end up repackaging someone else's code (a dependency), where the version used before cannot be found anymore, or I cannot figure out the combination of version that could possibly make the program work.

I hear that it's worse in JavaScript world. And, my experience with LaTeX packages has also been marred by similar problems (but not anywhere near the extent of how bad it is in Python).

On the other hand, I had to build absolutely atrocious (math operations on a mix of signed and unsigned ints, modifying const char etc.) C code (pre-K&R, the kind where they write function argument types after parenthesis), and it was much easier to deal with than any "modern" Python (i.e. 3-4 years old).

If anything, Python is the poster child of the problem of unreliability of third-party dependencies.

I've lost count of the number of times runtime behavior and performance issues have been traced back to (for us) far-from-optimal design choices in third-party libraries that probably seemed like reasonable choices to the library authors. Particularly in open source, you see remarkably little investment in performance engineering or scalability, so it is always straightforward to write your own implementation that is many times faster on the same hardware. Even basic things like popular data format conversion libraries are surprisingly consistently suboptimal. In most cases, the kinds of changes we'd want to make cannot be upstreamed to that library; most open source libraries heavily optimize for simplicity and the widest set of users to the exclusion of maximum performance. In an age when hardware can deliver >100 GB/s of throughput, many popular software libraries deliver <1 GB/s per core even though far better can be achieved with a modicum of diligence.

Another issue is the impracticality of threading fine-grained metrics and observability through a hodgepodge of libraries that were never designed to be used that way or to work together. This is a big deal if you want to heavily automate operations.

There is also a real issue with supply chain risk in terms of both quality and security. The vast majority of open source libraries are implemented to a lower standard of quality and security than the rest of the code base. And those libraries are often not really designed to be testable to the extent that would give similar confidence in the implementation quality. Combine this with the reality that the software needs to be deployable in environments that require higher than average assurance makes this a business risk.

As for the manpower cost, it has been low in practice. An engineer-month here, a few engineer-months there, with the benefit of better performance, more optionality, and much easier maintainability. In most cases, we would only use a tiny subset of library functionality regardless, so it adds a large surface area for a narrow benefit. Much of this code is also reusable. The existing open source libraries are very useful here because they provide something to measure against.

I also like that this forces a discussion of actual requirements for that functionality. Far too much software makes their requirements whatever a library offers.

I arrived at this point gradually over many, many years. The practical maintainability and opportunity for optimization have been key. In far too many cases, it turns out that a library dependency could be replaced with a couple hundred lines of code that actually worked better and which matched the design and style of the rest of the code base. There is a learning curve but once you've written a common class of library once, it is straightforward to do it again.

For example, there are corners of the Perl ecosystem that have been really stable for a long time. Perl itself is very conservative with backwards-incompatible changes, and DBI, the module that abstracts away database interactions, has been around for a long time (the oldest changelog entry is from 1995, but that was for version 0.58, I'd be surprised if it weren't even older).

Of course, then you don't get the fancy, modern libraries, but going with DBI plus DBD::Pg is probably more efficient than coding your own, and stable enough for a ten-year horizon.

Another approach might be to move the really critical pieces of your software to a much simpler interface. For example, HTTP/2 looks much more complicated than HTTP/1.1, so your software might only implement (a subset of) a HTTP/1.1 server, and then you can use an off-the-shelf reverse proxy to expose it to the "modern" internet. This reverse proxy then is easier to replace, because it doesn't handle the core logic of your application.

Or you go really old-school and use CGI, which is an even simpler interface than writing a HTTP server, and any HTTP server that implements CGI would do as a frontend.

I ended up maintaining that thing and it was a great lesson in premature optimization. The bottom 3 or 4 layers remained a grey box that I did everything to avoid touching. Just changing the front-end query-builder flow from a series of forms to a flat panel that could do everything in one step involved ripping out half the back-end code. All those layers of abstraction and none of them considered a form-flow change, which is exactly why I never assume anything now.

It's so much easier to add a layer than remove an unneeded one, which can be practically impossible on a complex app.

Given the prevelance of mutating functions in Java this is welcome

This has never been common in any sensible environment. Only on projects with project leads who had zero clue about anything, and with silly catch-all rules such as "EVERY METHOD MUST BE DOCUMENTED!". This predictably leads to shitty documentation.

   /**
    * Sets foo to <code>foo</code>
    * @param foo new foo value
    */
   public void setFoo(int foo) {
     this.foo = foo;
   }

In practice, a lot of orgs just stopped including deprecated versions.

I keep running into this with the Rust 3D graphics stack. That's winit (windowing), wgpu (wrapper for Vulkan, Metal, WebGPU, etc.), egui (2D menu overlay), and support crates such as wgpu-egui, wgpu-profiling, tracy-profiler, and glam (vector and matrix math). Every time one of these has a breaking change, it takes 4 to 6 weeks before the whole stack works without patching. Then I have to fix three more levels of my own.

Base64 would require dynmaic memory, you would likely use a packed struct. How you would do it depends on your compiler, but we would use #pragma.

Most of my professional work over the last fifteen years has been on applications that were ten years or older when I was recruited. Currently I do work in public sector digital archiving, where it's common to retire systems that are still being updated and have been going since the nineties. This kind of long term storage aims for century or more.