Building data infrastructure that will last

A colleague of mine preaches this gospel. But we never actually delete anything. So now our life is lived on a shanty-town of under-engineered throwaway codebases.

I'm not saying that our life would be better if we gold-plated anything; it would probably just suck differently. But so far, IME, design for deletion hasn't really delivered.

> The question isn't whether You Ain't Gonna Need It, the question is whether when you do need it will so much have changed that other design-aspects won't be valid anymore.

This resonates with me. But IMO YAGNI isn't necessarily different from asking "whether when you do need it will so much have changed that other design-aspects won't be valid anymore." If "it" is reduced coupling (to make something easier to change or remove), or any other not-immediately-necessary abstraction, it's really the same question

Cannot agree more to this sentiment. I call it "throw away code" and it's always seemed like the easiest to change in the future, and we all know everything is gonna change in the future.

There is a Greg Young talk about this called 'The Art of Destroying Software'

You should be interested in building something that others don't want to replace as soon as you leave. That doesn't require predicting the future.

It should be obvious that people not caring what comes after them is not a good thing.

> If this is a core conceit of the thinking then my answer is who cares?

Yep. At the end of the day, it's very simple:

People working for a company are not ants or bees. A company is not a hive and people are not going to put down their own interests to serve the hive. We are a bunch of cooperating, but ultimately independent agents, who act in their own benefit.

It is up to the business owner to keep their employee activity in check. Does that mean giving them work to do? Checking on the progress of their tasks? Checking on their methodology and software stack sustainability? Making sure there are no single points of failure for the business? Making sure the "IT know-how" of the business is preserved when a person leaves? ALL OF THE ABOVE!

When a business owner can't do these periodic checks themselves, they're free to hire someone that will do this for them.

But the idea that individual developers should care about what happens to the business after they leave is just preposterous.

Also, the entire "resume driven development" thing is absurd. This has always happened in software development. People care a lot about what their resume will look like in 5 years. It's perfectly normal and the business benefits too ("we use modern tools, come work for us"). It doesn't mean the business should allow needless "shiny new thing" syndrome to thrive, but you should watch out to not stomp out innovation or you might find yourself unable to hire talented devs because no one wants to work on your shitty "php with jquery" web app.

Resume-building job-hoppers are annoying and self absorbed, yes. The idiots who enable them are even worse.

SaaS was born from comments like this. Paying to keep the lights on, effectively ensuring that an employee quitting won't undermine the entire operation.

A lot of the article relates to a key person dependency issue.

>Sure, they have built data infrastructure that works and solved the businesses current problems. They maintain it, and no one asks questions.

Probably all the "budget" has allowance for is current needs. Some of the key engineers may not even be paid very fairly considering the true magnitude of business problems being overcome currently. You can't really expect them to prepare for a longer future than they have already been fully staffed for, especially succession.

>Perhaps no one realizes that one of the team members has to wake up at 6 AM every morning to check and ensure all the reports and tables have been created.

>That all works until the day they leave.

If a talented engineer is regularly working overtime to get things going like infrastructure, or worse to keep things going, even worse to keep things from failing, then that engineer is definitely short two staff members. And has probably been short the entire time. Nothing less than an assistant engineer and a technical secretary if they want real documentation as they go along. Plus even more true investment if there's any need to make up for lost time.

If infrastructure is important, something like this is absolutely pure executive failure from someone who's just not in the proper league.

You can not paint a pretty picture, and it's reported to be very difficult to fix stupid.

Some people just should not be accepted as executives in technical endeavors.

It can be tough for lesser executives to accept a non-cutthroat non-business-ladder-climber as more of a "key person" than themselves, but it is far too often the case. Whether the bonehead executives realize it and shrewdly calculate how much more payroll would expand if there was to be better coverage, or are completely oblivious, as the article says about the overworked engineers:

>They maintain it, and no one asks questions.

Example article is from a more expert data "repairman" who knows better than to rely on a single company as an employer if it's got dingbat executives.

There's so many of the under-qualified executives to go around, he's got a lifetime of work ahead of him as a consultant fixing the lackadaisical way they let technical debt underlie a business to where it could topple unexpectedly.

You should care because, the vast majority of the time the person working with it after you, will be you.

But to your point, people think they want “flexibility” or some similar concept, and they end up adding immediate complexity that never pays off, or worse, they pick the wrong abstraction and have a mess that’s hard to undo later.

What they should be aiming for is simplicity. Instead of trying to predict future, keep it as simple as possible to give that future person a chance of tackling the future needs when they arise.

You can view the question as a proxy for "how do you provide value for money?".

If you build something that then gets replaced a few years later, maybe you did something wrong. Ideally you make something that evolves, or even better, that acts as a foundation others can build on. If you get a lot of assumptions right and the implementation doesn't get in the way of what people do - or better yet, meaningfully enables them to get work done, you've succeeded.

Here are some things I've observed in the wild.

Data infrastructure projects often fail, not because the technology doesn't work, but because the solution does not enable _organizations_ to work with them. I've seen many companies invest millions in solutions that eventually turned out to be useless because they failed to help make data and results accessible to complex organizations with lots of internal boundaries.

Too much too soon and too complex. You try to address every possible need from the start and in order to make the feature list as long and impressive as possible, you introduce lots and lots of systems that are expensive and complex. Then to use the system, you unload a huge burden onto the users. They have to learn all of these systems and spend lots of time and money training people and adapting their systems so they can interoperate with the rest.

I've helped a few companies design their data infrastructure. I usually follow an extremely minimalist approach. Here's how I start.

1) your long term data store is flat files, 2) you make real-time data available over streaming protocols, 3) by default everyone (inside the company) has access - access limitations have to be justified, 4) you document formats and share code that is used to interpret, transform and process data so the consumer can interpret the data. 5) you give people access to resources where they can spin up databases and run stuff. Data producers and consumers decide how they want to create and process data. You focus on the interface where they exchange data.

(I left security as an exercise to the reader because a) it depends and b) how to secure these kinds of systems is an even longer post)

Points 1 and 2 are sufficient to bootstrap databases and analytic systems at any time. Including systems that receive live data. It makes it possible to both support systems that are supposed to be up permanently and systems that perhaps only load the data, do some progressing and then get nuked. 5 provides the resources to do so.

3 usually meets with resistance in some types of organizations, but is critical. I've seen companies invest millions in "data lakes" and whatnot ... and then piss away the value because only 2-3 people have access to the data and they ain't sharing. You need executive management to empower someone to put their foot down. (One way to make people share data is to use budgets. If you don't share data, your department pays for its storage. If it is shared, it is paid for by a central budget.)

Point 4 requires you to also educate people a bit on data exchange. For instance in many areas there exists exchange standards, but these are not necessarily very good. If you find yourself in a situation where you spend a lot of effort expressing the data in format X and then spend a lot of effort interpreting the data at the other end, you are wasting your time. Come up with something simpler. Not all standards are worth using. And not everything is worth standardizing - don't lose sight of actual goals.

Point 5 is where you grow new core services. Producers and consumers get to pick their own technologies and do whatever they want. When they can show that they've built something that makes life easier for other parts of the organization, you can consider moving it to the "core" but this only happens when something has shown that it works and improves productivity across internal boundaries.

The "you need 5-7 different tools glued together to solve anything" is the CORE problem of the "modern data stack". It also ties very closely with Resume Driven Development.

It leads to a lot of anti-patterns.

For example, the 5-7 different tools are constantly changing, so after hiring some proclaimed expert.. they end up re-inventing the wheel by choosing a new combination of tools than they've used in the past, hitting various unexpected issues as they go.

VERY rarely in this space do you see someone come in and go "I used these 5 tools in previous roles, they work great, and I'm going to build the best solution because I have done it before."

These guys always think they need to reinvent the wheel, and then end up wrecking the car with some combination of v0.1 untested FOSS, up&coming SaaS, and their own in-house DSL.

In my opinion there are many tools and products in this space and they all seem somewhat confused in their target audience (is it marketed to management, analysts producing reports or developers supporting the analysts?) The boundaries between these projects is often fuzzy and they are often complicated (does it include a scripting language?) When you are starting from close to scratch with an application and it's backing database and being asked to produce timely reports, I think these tools aren't the best place to start.

My process has been to talk to stakeholders and sketch out reports they find useful, preferably getting a set of data together that many people find useful. Running these reports against a read-only replica of production data is typically not a big lift. If dashboards are required, write this data out to another database to back the dashboard, probably on some set schedule. It hasn't been long but now we have the bones of an ETL process that is already returning value.

At that point I think these tools start to look more compelling. Now we have a handle on the source data, what it looks like and any places where we need to do something tricky to connect the dots to get our data out. We know what the reports and dashboards look like.

In short, we know what we need these tools to do and where they can help us.

Having lots of options isn't a bad thing. The tools you happen to have chosen for your product are not objectively "the best". The entire business model of SaaS is "it's cheaper for us to manage this than you".

I am also in this space. Personally, I like piecing together my tools, rather than using some black box. YMMV.

Definite - seems interesting.

But what is the Definite warehouse? Is it built on open standards?

I'm not a founder in this space like yourself, but I do a solid mix of consulting and building on the modern data stack for AI/ML and always appreciate a well constructed data stack.

If your system is not getting replaced after your departure, it is likely your domain within the organization is just not attracting ambitious people.

>I tend to favor tools that have been around for a long time

If there are no early adopters, then no tools will ever make it to the "been around for a long time" stage. Someone has to use it.

This is a compelling testimonial. Someone recently posted a 'git ops' package here that was equally simple and bash based. Im interested to try, but im wondering what the rationale will be for avoiding github actions.

If you could share script or snippets I would be grateful.

One difficulty for me is my datastore and reporo g package is a python application using Prefect, and I manage dependency with Poetry.

The 'poetry install' phase isnt always ha ds off and my script fails.

For infrastructure, I try to keep the Lindy effect [1] in mind.

[1]: https://en.wikipedia.org/wiki/Lindy_effect

Everytime I look at S3/Glue/Athena I can’t help but feeling like the Glue layer shouldn’t be necessary and it’s instead just part of athena’s ddl

Exactly, and same as my comment below (parquet+iceberg+s3).

And yes Athena is a part of that. And we also use dbt but mostly for a place to commit and push queries. And I agree with the other question about glue, it is the ugliest part.

I guess a +1 is not per hacker news standard, but i still want to give it some strength, given that we came up with the same solution independently.

Is DBT really necessary? (serious question) If so, why? What would go wrong by skipping it?

I think that one of the biggest issues is that a lot of training is flat out horrible. The author preaches simplicity and working directly with the business, but mean while you have entire teams of developers being taught something like Clean Architecture, SOLID, DRY and all sorts of fancy things that when used poorly leads to extreme over-abstractions.

So even the most well meaning engineers with good training can go about building data structures which won’t last very long once the key people leave. Simply because what they were taught doesn’t work.

In many cases, that means gaining the freedom of creating a huge mess (speaking from experience)

> They quit and no one knows how it works

Agreed, this is a very low bar for "amazing"

The most praise I've ever received for my documentation was from the developers forced to read it after I moved on to a position with another company. It's a little unfortunate I didn't hear more about it as I was writing it, but then again it's the kind of stuff other developers find useful.

Still, in my opinion, spending the time writing documentation was a net positive: it didn't take me all that long to write up and it clearly made someone else's job easier, so much so that they mentioned it to me. And, of course, I've had to read my own docs more than once.

Should they care? I’ve come into organisations that had spent entire years worth of man hours on setting things up correctly so that they could potentially scale to millions of concurrent users. Organisations which would never reach more than 50.000 concurrent users in their wildest dreams.

On the flip side I’ve seen some extreme cowboy hacker man code run perfectly fine for its entire 10 year life cycles.

Now, I don’t think you should go completely cowboy, but I do think you should think about whether or not your “correctness” is getting in the way of your actual job. Which is typically as a service function where you’re supposed to deliver business value at a rapid pace. Obviously it depends on what you do. If you work in medical software you’re probably going to want to get things right, but just how much programming could be perfectly fine if it was just thrown together without any adherence to “correctness”? The theory tells you it’ll cost you down the line, and in some cases it will. In my anecdotal experience it’s not as often as we might like to think, and sometimes the cost is even worth it. In two decades I’ve only ever really seen two poorly build systems cost so much down the line that they would’ve been better off having been build better from the get go. In both cases they couldn’t have been build better upfront because the startups didn’t have the people to do so.

I think I write decent documentation, but my target audience is usually my future self. If I can rely on my own documentation to help quickly reacquaint myself with the project later on, I generally consider it sufficient.