Hacker News Reader

> Because interfacing via API is expensive. Writing APIs for others to use productively isn't easy and change management also adds a lot of overhead.

I agree in principle, but there is a lot of “unseen coordination/communication” costs that it’s easy taken for granted.

When I was working on telecom doing interfacing with carriers (e.g T-Mobile, Verizon, etc.) on thing that I noticed was how simple was to work with those folks: Ok, this is the standard XML, those are the endpoints, that’s the list of error codes, the rate limit is X requests per second, a bunch of files will be on this FTP at 5AM daily basis, and if you face more than 100ms latency from our side just call this number.

Working with “product” companies without silos most of the time it’s design by committee, folks that won’t keep the service running wanting to have a say in our payload wanting us to change our overly reliable RabbitMQ to use their Kafka.

> When I was working on telecom doing interfacing with carriers (e.g T-Mobile, Verizon, etc.) on thing that I noticed was how simple was to work with those folks: Ok, this is the standard XML, those are the endpoints, that’s the list of error codes, the rate limit is X requests per second, a bunch of files will be on this FTP at 5AM daily basis, and if you face more than 100ms latency from our side just call this number.

To me, that probably reflects the maturity of the services the carriers provide. And presumably that there's an explicit customer-producer relationship? These things justify the complexity of maintaining a well curated and operated API.

> Working with “product” companies without silos most of the time it’s design by committee, folks that won’t keep the service running wanting to have a say in our payload wanting us to change our overly reliable RabbitMQ to use their Kafka.

If I understand what you're saying, you've experienced platform people telling you what tech to use, without having real skin in the game for operating your services? If so, that sounds very irritating. To me, a truly silo-less approach would not have that.

To the extent that there are platform teams with a say in architecture, I think they should develop requirements around the external characteristics of the deliverable (performance, cost, observability, contract with other teams, etc) and largely leave the implementation concerns to the people developing and running the service.

This is an interesting example, becase telco has an actual API standards committee (TM Forum). Telcos have decades of experience and extremely well defined (and to a large degree shared / interchangeable) domain model. It's an ideal scenario for APIs.

Meanwhile your product companies each develop different product, there's little standardization. API designers have only vague idea what the API will be used for and how. Fast evolution is important.

TM Forum provides a bloated format. It has everything and a kitchen sink. This means that something simple like customer name can be written in multiple places, and different architects will recommend different location. Two different silos within same company won't be able to talk to each other because although they are using same format, they interpret the usage different.

Additionally, this is XML/XSD, different teams will end up using different version of the standard. They won't be able to interface with each other without additional layers of translations. It is not uncommon for one team to need to load multiple versions of the XSDs because different end points use different versions.

> folks that won’t keep the service running wanting to have a say in our payload wanting us to change

This is my single biggest gripe with DevOps. If you’re not going to be fixing it, why on earth do you get a say as to how I build it? It’s nearly _always_ a one-way street, too – when’s the last time Ops successfully ordered Dev to change some specific part of their code (modulo things like, “hey, you really need to add a rate-limiter”)?

IMO, this phenomenon is not specific to DevOps. I've seen this happen when there are official architects or architecture committees.

One of the problems about debating DevOps is that there's no single agreed upon meaning. That probably also impedes its success.

To me, the essence is that there are engineers whose job it is to give product and service devs the ability to do their own operations through software that simplifies basic shared concerns. That does not mean mandating solutions.

“Working with “product” companies without silos most of the time it’s design by committee,”

With siloes you may end up with design by ego trip.

Egos only exists if the team let them exist. A strong leader will make sure this is not happening.

A strong leader at the same time can cause it to happen...

> Because interfacing via API is expensive. Writing APIs for others to use productively isn't easy and change management also adds a lot of overhead.

An under appreciated point. Something that is affordable for giant monopoly profit driven FAANGs, but harder in smaller orgs.

Even within my team, the "level of difficulty" of writing an API depends on its user base. If I am the primary user, it's easy. There's one really sharp kind on my team, if he is going to use it, then 2x the difficulty. If I need it to handle the median dev on my team, then 2x again. By the time you get to the below median dev, add another 2x. So even within my team, the intended user base can change the difficult from 1x->8x.

What are some of the pain points - input validation that drives useful informative errors, flexibility on inputs like having sensible defaults to reduce the number of inputs users must pass, performance/scaling, and edge cases.

>> In this sense, development teams weren’t so much empowered to deliver software faster; rather, they were over-encumbered with infrastructure tasks that were outside of their expertise.

And agreed on this point. Amusingly I have had HN thread arguments just this week with DevOps advocates telling me that akshully their job isn't to empower devs, but some sort of "tail that wags that dog" interpretation around devops organizational / standardization / cost / etc.

Yes APIs are expensive and require significant overhead.

But clean, maintained interfaces between separate domains are far LESS expensive than multi-domain/team mashups.

Maintaining separate teams AND well- maintained interfaces surfaces and makes explicit the myriad inter-silo interactions.

Instead of a quick backchannel conversation and a quick hack, it is an explicit conversation and an API adjustment.

Yes, it's more expensive up front, but far cheaper and more powerful in the long run.

So for high-speed prototyping, not so suitable- breaj the silos, do the backchannel hack & move on (this version will likely be binned or massively refactored anyway). But when the domain is more stable, modular with solid interfaces is the way to go.

> My overall reaction is that this is a great piece for products/teams that have reached significant scale, once the job to be done is too big and complex for one team to own, end-to-end, or there are truly reusable concerns that can be separated from the core product (e.g. auth, observability).

I'd say it's critical in the small scale too. When you get down to a single person size, this is just what well factored code is: you write down little reasonably independent pieces that can be learned about and thought about without having to think about all the other things. After all, we've all had that experience when you go back and revisit your code after vacation or something, and it might as well have been written by someone else.

I was on a tiny team not long ago where my teammates kept writing tightly coupled systems, then rewriting everything from scratch every time. It was hell. Our product moved slowly, broke constantly, and we couldn't build off of our prior work, and could barely even build off of each others' work, so velocity stayed constant (read: slow).

(as a tangent, the communication patterns of remote work seem to make this more important)

Siloing teams, siloing concerns, and writing modular code are all kind of the same thing, just at different scales.

As a coda to this, I think that we are entering a new world where services truly can have minimal operational overhead.

At my last company, we were all in on serverless, with AWS tools like SQS coupling things together. It worked extremely well for keeping the architecture simple to operate and approachable for new people.

But even better, I think we want logical services that interface with each other as though they were in a single process. We want the ability to write code as though it is monolithic (e.g. lexical scope and native-language APIs), while availing ourselves of the advantages of independently deployed services. I believe projects like Temporal point the way. I haven't had the opportunity to use it, but philosophically, I think it's the right direction.

I don't have experience with it myself, but Polylith architecture looks interesting. There you compose services based on shared components. You can start developing it as a monolith, and then extract components to separate services by just changing the interface between components.

https://polylith.gitbook.io/polylith/

If I'm understanding Polylith, it seems like a way to do monoglot monorepo in a way that avoids explicitly using libraries.

That looks pretty interesting and similar to what Artsy did with Ezel for front-end development.

https://github.com/artsy/ezel

I have been working mostly with Kafka for five years, and I found SQS to be a little weird to work with. How do you keep an overview of all services that consume a queue? What if you want multiple services to read the same data in the same order? Granted I am very new to AWS.

I think you're really looking to use kinesis rather than SQS.

SNS+SQS make a pub/sub setup, but every queue subscribed to the event is fully separate, and you wouldn't expect to try to couple the different listening queues together.

You could make all the queues into SQS fifo queues, and put them all on the same sorting group, but I think network time could still break the ordering on different subscribers?

The simple way to use SQS is to have a queue for each message type you plan to consume, and treat it as a bucket of work to do, where you can control how fast you pull the work out

>> as though they were in a single process

The problem is, you cannot abstract away a 3 order of magnitude difference in performance and latency. This problem increases with larger teams as people don't understand what is happening. Abstractions shorten the learning curve but also greatly slow the formation of an accurate mental model of how a system works.

The mental model of using SQS and serverless is extremely simple. You put a message in, get a message out, and have your code run.

This is something I’ve been able to teach teams how to do in a few hours. This is the a huge part of the value prop of these services.

Performance and latency have a trade off, sure. But you get teams who have complete ownership over their service. You get services that can scale to millions of requests per minute transparently. Some of the highest scale workloads I know use microservices behind SQS.

> You put a message in, get a message out, and have your code run.

In my experience it's more like put a message in, get an `AuthenticationException`, copy/paste some IAM stanzas into CloudFormation and redeploy, put a message in, get a configuration error, add dependency-injected config loaders to constructors, add env vars to CloudFormation and rededeploy, put a message in, get a deadlock from the threaded HTTP client due to class loading being single-threaded, refactor config loading to happen after construction but populate a cache, redeploy, put a message in, get a timeout due to a stampede on the cache, get a hefty bill from AWS for spinning up a load of long-lived serverless processes, ...

I'm not saying its a silver bullet, because sharing data via database makes change management way worse.