Best policy I've figured out as a developer dealing with product teams is demo everything in a command line until the backend actually works.
No chance for suckass product guys to push if they can't understand what's going on.
2,569 karma · joined November 26, 2013
No chance for suckass product guys to push if they can't understand what's going on.
The correct response to "Yo, they fucked up." is not "Oh screw it. We'll just do it ourselves."
Only about 1% of tech workers are employed by companies headquartered in Silicon Valley, let alone San Francisco. The absolute vast majority of technology employment is at companies that don't have a direct technology product. Software engineers working at traditional businesses like banks and retailers far overwhelm the employees at FAANG and startup type companies.
You're buying into an echo chamber concept because you don't realize the extent of the universe you're characterizing. You're mistaking the echo chamber for the reality, kind of how people mistake what people say on twitter for what people actually think.
And if, in fact, you are storing a graph in a graph database, the QL makes a bit of sense.
But nothing in the post makes any sense out of any of that. It's just Python bad; REST bad; I read too much hacker news, and I feel like it's time for a change.
Like, when I complain about other people's REST APIs, that's out of my control. This guy is saying that his API is garbage, and instead of fixing it to make it better, he's just going to redo everything with a worse result. I don't get it.
pypi.org is another that's familiar. You know, every time you type `pip install x` yeah, that's pypi.
Although I think those are both powered mainly by Pyramid rather than flask. Still, same concept.
As others mention, large parts of google and youtube are still python. Dropbox was so invested in python that they employed Guido van Rossum for a while. Instagram, a lot of Yahoo! back when they were a thing, Spotify, Quora, Pinterest, Hipmunk, Disqus, and this really obscure satire site called The Onion that totally never gets any traffic at all.
All of them powered by python at their core, many of them Django, some Pyramid, and some Flask.
Yes, getting that big does require big teams. Becoming one of the top 100 or so sites on the internet always requires some special sauce as well as dedicated teams. But most of these companies started with Python and a framework and got to massive web scale along the way and never changed the core platform because there really wasn't a need. Handling scale isn't about your core language or framework. It's about dozens of other things that you can offload to other things if you're smart. But let's be real: sourcehut isn't close to any of that level of traffic.
My negativity on this isn't about stanning a particular language. I'm an agnostic in multiple ways. I'll use whatever tool seems like the best fit. I'm down on this because the explanation is tool-blaming, murky, unclear, and doesn't provide a lot of the detail I would want to have if I were depending on this service.
On the other hand, the guy has always said this is an alpha project and you should expect major changes. That's all fine. It's just weird to me to see a "why I'm changing from X to Y" post that doesn't really explain anything other than "I might be bad at this."
But claiming some nebulous backend that's more performant and robust than Postgres is like, WTF? Are you using an actual GraphDB like Neo4J? Are you putting a graph frontend on Postgres like PostGraphQL? None of the post really makes any sense because GraphQL is a Query Language, not a data store. What are the CAP theorem tradeoffs in the new backend? What does more robust mean? What does more performant mean? This is a source control app. Those tradeoffs are meaningful.
There seems to be a lot of conflation between API design and data store and core programming tools all mixed into a big post that mostly sounds to me like, "I don't get how to make this (extremely popular and well-known platform that drives many websites 10000x my size) work well, so I'm trying something different that sounds cool."
Which, again, the author has always said this is an experiment, and that's cool. But the conceptual confusion in the post makes me think that moving away from boring tools and trying new tools is not going to end up going well.
But this is a source control app, and it's hopefully backed up somewhere besides sourcehut so it should be fine if he needs to backtrack.
If you can't make that combination work well, there's another place to look for problems besides your tool kit. You might need to ask yourself if you really understand the tools you're trying to use.
But like I said, this has always been a very cool project. My "good luck" was meant more as actual good luck than a Morgan Freeman You're-trying-to-blackmail-batman kind of good luck.
So, I mean, good luck.
First thing I figure out is how I want to interact with a thing. Whether that's a program or a class or a function. How do I want to call it? What parameters do I want to pass? What do I want it to return? How do I want to use what it returns? So, basically, write the interface first.
If I'm building something with multiple parts rather than a single class or function, I'll map out how these things all work together. A loose graph of interactions. Invoke A and have it return X; invoke B and pass X to it and it returns Y; etc.
Then I'll consider failure modes and think about what should happen if something doesn't work out quite right. Is it possible to route failures to a centralized cluster of error handlers so I don't have to implement error handling at every level?
Finally, I'll think about whether I can map behaviors to defined data structures instead of controlling flow with if/else patterns.
Once I have all that written down or mapped out, then I'll start implementing from the outside in. Stub the object, methods and return dummy data structures that fit until I have a complete system that's interacting the way I want. Then I go in and implement the actual functionality I need.
The last part--implementing functionality--often implies modifications to my initial thought process. But it's easier to understand what those changes affect if you've already mapped your design. So you might think B can produce Y with I parameter. Maybe it turns out you can't. So now you need to add a new param. Where is that going to come from? Well, you don't have to invent that out of nowhere because you've already mapped out what is happening. You know that you need to either add another node to the call graph or change the return value somewhere else.
By the end of the process, you have a working program, and you've also done a lot of your documentation work as well.
Again, I have no idea if this is what anyone else is talking about here. But this is how I personally work. It annoys the hell out of some people. But it works for me and helps me create sane software with interfaces people can remember over time.
Does each day of 25 million records take 25GB of disk space? Or are you saying that it cumulatively takes 25 GB up until now?
1 million records per GB would indicate a really, really large record.
You have to put a semicolon in place of the comma or it is just plain wrong.
Developers don't need to care about threat models. Developers need to care about the attack surface their applications expose.
This article feels a lot more like it's directed at engineering managers than it does at individual developers.
It sort of pretends that there aren't best practices and that we all have to just invent shit from scratch every time we do anything. We don't have to do that.
If we all just considered OWASP top 10 and mitigated those, almost everything Fowler is suggesting here would be irrelevant.
Most people are going to get threat modeling wrong. Just follow the fucking rules. They aren't that hard. Just inconvenient if you are in a hurry.
I hope this is a joke.
It's a shame that Rethink did so many things right and failed as a company while Mongo continues to do almost everything wrong as a company and still gets business.
The fact that you put Kafka and RabbitMQ in the same category sort of makes me feel like you're out of your element, Donnie.
Really? You're going to go with "most languages implement static typing"?
That seems like a bold statement.
I was hired on to the team about 3 years after the nightmare tipping point. There were no tests. There were no real expected outcomes. Just angry customers sometimes. And angry product managers who were still pointing at the original workflow diagram telling us to just make it do that. And there were people who had been dealing with it who were all, "Yeah, that's not even close anymore."
So instead of trying to solve all that mess of code, I backed up the chain a little bit and looked at things from the point of view of what variables are getting passed into this shitshow and what are their possible valid values. I came up with about a dozen that matter. Really variable variables like the dollar amount of a transaction don't really matter. It's other things like has this transaction settled? Is this a chargeback? Etc. Lots of binary switches. Some of them could have 3-4 legit dispositions. But if you look at their sources and what they imply, the possible valid states isn't the combination of all possible positions of all 12 variables. Some of them are exclusive. So I figured out that there's about 100-ish valid ways for these 12 variables to be arranged. And that only those 12 variables were actually driving all the logical outcomes.
At that point I was able to go to the product managers with a spreadsheet and say, "Okay. Tell me what's supposed to happen when a=True, b=3, c=debit, d=whatever, etc." They would go huddle up for a day or two and then come back with an answer. X, Y, and Z should happen, but not A and B, and not-not-C. And over time we were able to get a mapping from all valid states to all valid outcomes.
We all know what a finite state machine is, and it would be trivial to implement in any language. But in this case, lots of different systems needed to be able to work with this, and the only thing they had in common was the ability to connect to the database. So that's where we did it.
I had two tables. One was the rule and one was the outcome. Each row of the rule table had vars/values of a valid state and a foreign key to the outcome table it mapped to. The outcome table columns where the names of transformation functions and bool flags on them. So the process was 1) query for the relevant variables, 2) bounce them off the rule table to find a match or error on no match (which was useful in finding bugs higher up the call chain), 3) query the outcome table from the foreign key and get the values of the transformations, 4) push all of that into a stored proc that had a totally flat if/else chain 5) exec state transformations where indicated. Very easy to grok, not hard to implement, and as I said, the rules and outcomes could be added to or changed very easily by the people who were expert in understanding the business rules.
I hope that's enough detail to be useful to the people who were asking for it. I don't really want to be more specific. I think it would be crude and maybe a tad unprofessional to go deeper than that.
If I may, however, I want to respond to the person that said logic wants to be code and not data. And that it's risky to do otherwise. Without getting on a LISPy high horse (I've ridden that horse only a few times), I disagree. Sometimes logic drives data and sometimes data drives logic. I don't think there's as clear a distinction as you are suggesting, and I don't understand how you draw the line.
E.g., if a ======== 1{ //No seriously, I mean really really equal return a }
What's happening here? Is your logic in the driver's seat? Or is the data in the driver's seat? Who is determining the behavior? Your holy code? Or the value that gets passed to it?
I feel like arguing about data and code is like arguing about the difference between putting a net in place on a river and letting the fish get caught and dragging a net behind a boat so fish get caught.
But I'm not going to dwell on it too much. I think it's kind of silly to get dogmatic about it.
The ideal use case for MongoDB is when you are a) collecting a lot of data, b) only need to retain a large enough sample for statistical purposes, and c) don't need historical archives.
In those cases, turn all the safeties off and go for it. It really does shine in that scenario. You want to get a heat map of clicks on a high-traffic website? Perfect. You'll get a good-enough picture of what's happening. And if you lose some or get some conflicting data, that's okay because we only care about a statistical approximation. Archive the results of the analysis instead of worrying about the underlying data.
People often get bogged down in the details of CAP theorem too early in their decision-making process. The reality is that database systems occupy a 2-d spectrum with raw speed on one end and safety on the other end. CAP theorem is a rigorous study of 3 dimensions of trade-offs you have to understand as you move that slider from one side to another. A lot of hand-wringing and debate could be shut down if people analyzed the lower decision first. Most use cases are binary and occupy one extreme or another: you either need them to be safe, or you don't. If you don't need them to be safe, then you want them to be as fast as possible. If you need them to be safe, your options are limited.
If you phrase the question only on the fundamental 2-d spectrum, your product and technology managers will almost always come up with a safe/speed answer. If you start at the CAP level analysis, you end up with endless discussions about what safe enough means--and it's usually a discussion among people who don't really understand the theorem at all.
There are tons of cases where any specific item of data doesn't matter. It's only the aggregate that matters. Great. Perfect. Use whatever it doesn't matter. And there are other times when you absolutely cannot afford to lose anything. These seem like obviously different things and I can't for the life of me imagine why anyone would try to use the same tool for both purposes.
Anyway, I'll shut this down now. To me, the bottom line is to identify what the business case is for data first. Fast or Safe. Pick one. Okay, now we basically know what our options are. Then we can get into the weeds among ourselves.
I think there are 4 buckets of data engineering problems, each with their own challenges and solutions.
Operational Data Engineering This is the detritus that grows like weeds as parts of other projects and often isn't recognized as a data engineering problem. We need to pull a file off an FTP server or hit an API and do something with it. Next thing you know, there are dozens of these little things that are not individually hard, but having visibility into dependency trees and failure cases becomes difficult because they are spread out everywhere and it's not obvious where to look when things go wrong. Tools like Apache Airflow are a good solution even if you don't use them in other ways because they can centralize monitoring, logging, and graphs. Scaling isn't resource intensive for these tasks because they are discrete. You can fan out. The scaling challenge for this type of data engineering is really about tending your garden and keeping things coherently organized.
Business Logic Data Engineering This is processing where the data is highly structured and sometimes even ordered or sequenced. It's hard to scale because you can't just throw things into a stream and apply multiple workers. You have to have a managed process and likely shared in-memory state that collects the worker results and applies strict rules to a process. This is the opposite problem from big data. It's small data, rigidly organized, and carefully managed.
Data Science Data Engineering This is sort of classic ETL with a twist. ETL systems are typically pretty static once the E, T, and L are known quantities. But working with Data Scientists requires that your pipelines have to be pretty flexible because scientists are doing experiments. But they also have to be repeatable and comparable, which means your pipeline has to maintain version. This is also the area where you are most likely to encounter Big Data, so you have to be prepared to change your mental model and be able to use tools like Hadoop and Spark to bring compute to where your data is.
Analytics Data Engineering This is classic ETL pipelines that move data from point A to data lakes or data warehouses. The key thing to understand here is what you are modeling at the endpoint. If it's a legit data warehouse, you are modeling business processes. If you aren't doing that, you are--by definition--pushing data to a lake. Understanding your endpoint is key to choosing your reporting and analytics tools to lay on top of your data source. Data lakes are a good use case for ad-hoc, SQL-driven reporting tools like MetaBase. But if you are sitting on top of a well-structured fact/dimension type of warehouse, you will want more formal tools like Tableau, Pentaho, or Cognos.