Hacker News Reader

Comment by CoolGuySteve | Hacker News Reader

When the packages and configuration files for the project are more complicated than the python code it would take to do the one simple thing that you need. A lot of frameworks tend to bloat to handle every use case and become overly complicated because of it.

Things like passing protobufs/gRPC when C structs, JSON, or CSV files will do, job scheduling systems that could be replaced by a postgres table, most Kafka applications.

I once worked with a guy who would pattern match whatever problem you had to the trendy Apache or Google library and then spend weeks/months setting it all up. He was easily the least productive person I've ever worked with.

View on HN

>Things like passing protobufs/gRPC when C structs, JSON, or CSV files will do

These will always get the job done, but at great cost to future maintainers and callers. Explicitly declared interfaces that give some thought to future evolution aren't that hard, and can save a lot of pain down the road.

I vividly remember the outage I caused by changing a struct field name, only to find out it was implicitly a critical part of the API "contract" when serialized to JSON. Another where I relied on a certain field from a dependency's response, only to find it out it was populated in some situations (the examples I used to learn the interface) and not others (revealed by production). Our Thrift services do not have these problems. Anyone can look at a diff and tell a) that the IDL is changing, b) whether the change is breaking, and c) whether a field is required or optional.

GRPC is great as an API generator, but there's a lot of machinery under the hood in its network layer. It's not small - there was an embedded project where we were considering using it to replace our roll-your-own, then we discovered that the compiled library for GRPC was larger than the entire application.

C structs are so brittle they should not be used for any kind of serialisation. You're just setting yourself up for upgrade nightmares. I wish BSON or the ASN.1 toolset were easier to use for this use case.

For C structs, start with a uint32_t 4 char code identifier and a size field. Then you can extend them as long as you only ever append, which is a constraint on protobufs anyways. This is how most media containers have worked since the Amiga days.

Meanwhile, their encoding/decoding is infinitely faster than most other serialization formats, allowing you to serialize data back and forth to disk or as IPC faster than anything else.

.. as long as you never change endianness, structure packing, word alignment requirements, and always use types with explicit sizes. And you have to keep the data in the structure itself, so as soon as you hit a pointer you need to write explicit serialisation.

> .. as long as you never change endianness, structure packing, word alignment requirements, and always use types with explicit sizes.

#pragma pack(1) and byteswapping solve all this and neither are more work than specifying some intermediate format and calling some external tool in your build system. In C++ you can make types that swap automatically if required.

> And you have to keep the data in the structure itself, so as soon as you hit a pointer you need to write explicit serialisation.

I have yet to see a serialization format that does this well and quickly. Just make a new message type for it that follows the first.

Or juste use riff which useful and provide just enough to implement wav, bmp and other file formats (so you'll find heaps of examples(.

Indeed. Using plain structs one can't just add a single field with a reasonable default and still be backward compatible - which is a very very common case in API design.

You could add completely new revision of the API. But having to maintain it - typically additionally to the old revision since both sides might not updateable in atomic fashion - is a lot of extra work.

Serialization which allows to add and remove fields in a backward compatible fashion is a mandatory requirement for any bigger sized project for me. If you need super high performance maybe use Flatbuffers or Cap'N'Proto instead of Protobuf/JSON/CBOR - but I really wouldn't recommend to go for plain structs.

>but there's a lot of machinery under the hood in its network layer

GRPC is designed to enable peer-to-peer communication in a microservices mesh, so it has hooks for a number of things like service discovery, authentication, health checking, retry policy, connection pooling, observability, etc. If you're just looking for a serialization format you might prefer Protobuf itself.

I hope the CBOR tooling is better https://cbor.io/tools.html

I vividly remember the outage I caused by changing a struct field name, only to find out it was implicitly a critical part of the API "contract" when serialized to JSON.

Lack of unit tests for serialization can be deadly for projects where the serialization format is defined implicitly via the application model. I've seen enough projects bitten by this to be convinced it's worth writing ser/des unit tests for every API object. For some reason it's a form of testing that even gung-ho unit testers tend to leave out. Arguably it should be covered by integration tests, but integration tests usually only test round-trip serialization, which rarely breaks when you're testing a codebase against itself.

But I think the issue you faced was a documentation issue as well. It should have been evident which objects defined the API. In some projects the objects have to be heavily annotated to make ser/des work at all, which makes it obvious, but when it's possible to separate concerns and define ser/des separately, there should be documentation or naming to make it just as easy to see.

I agree, but in this case I moved the interface in question to Thrift because it provides both documentation and compile-time safety for substantially less code than doing it manually with JSON. (There are also various ways to do some or all of it with JSON, such as JSON Schema and Swagger).

Oh yeah NIH is a problem, but "we can't invent it here" is also a problem

Person tries to find a library that does what one line of code would do (cough left pad cough and others) and then guess what, the library fails at that thing that maybe not 90% of people need but 50% need

And that even applies to some corners of famous apps or libraries big sigh

This is too common of a problem because many developers are functionally illiterate in their primary programming language.

Invented here syndrome: https://dev.to/mortoray/invented-here-syndrome-4mg8

>...pattern match whatever problem...

I agree with your comment. I would add that there are cases where your problem looks really easy right now, but will get increasingly complicated over time in ways you don't expect. In this case, "overkill" 3rd party solutions are actually better. For example, you could write an HTTP client using sockets pretty easily, and it will work in simple cases. But you will, over time, run into edge cases and problems that a mature library has already solved for you.

Also, this problem reminds me of having to decide whether to solve the integral, or look it up, back in the day. Solving from scratch was always preferable if you could do it, but sometimes it was just too hard.

> But you will, over time, run into edge cases and problems that a mature library has already solved for

It’s entirely possible that the new problems you encounter will be different than the ones the mature library was designed to address, and then you end up hacking around solutions to problems you don’t have in order to fix the ones you do.

I actually did have to write an HTTP library on sockets once. It wasn’t trivial (took a month to productionize), but it was core to our business. We were writing developer tools (Parse) and our file upload library had major problems on the latest iPhone due to an OS bug (if a kernel buffer filled before the modem could upload it, the file transfer would fail).

We became the third company I know about to ever solve this problem, but I had built what my colleague called “the hardest to use HTTP library ever”. I was happy to move back to AFNetworking once we could do so without a regression in performance or hitting that bug.

Can you shed a bit more light on this -

> job scheduling systems that could be replaced by a postgres table

I've never used Postgres in my life ever.

I think the point is to use a DB as a message broker. Schedule a job by adding a row to a table. Have services running that secure a lock on a row, run the task, and then update/remove the job from the DB.

Django-Q (the multiprocessing task queue framework, annoyingly named the same as Django Q objects), allows this as one of the configurable brokers. https://django-q.readthedocs.io/en/latest/brokers.html#djang...

+1. And I think specifically the SKIP LOCKED Postgres feature is relevant for this -https://tnishimura.github.io/articles/queues-in-postgresql/

Oh, I hadn't noticed that. I'm not sure how Django Q manages with a non-postgres DB, since it doesn't mention that as a requirement.

I have used nowait before to batch scheduled jobs together for a slow API call.

Why would you pick django-q over Celery, which also supports a database backend?

We've had a lot of mysterious "it just stops working" issues with Celery, and their # of open tickets is approaching scary levels.

Moving everything to rq, and so far so good.

Are you sure that changing your implementation didn't fix a bug?

I've noticed that a lot of "dependency" problems on "complex systems" are usually programmer implementation problems. Changing the implementation forces you to review the original implementation (or throw it out) and fix the bug that caused you to change the implementation in the first place.

We’ve had exactly the opposite experience. RQ was an absolute nightmare in every aspect across two companies I’ve worked at. And when you need something that Celery supports that RQ does not, you are in a pickle.

I wish you the best, but I cannot help but think you will regret that decision.

Also, Kiwi.com gave a talk[1] about their experiences with RQ and how/why they migrated to Celery at EuroPython last week. It's worth a watch.

1. https://youtu.be/SUQYuA9AJT0?t=7546

Have never experienced this and we sold a huge number of our app using celery. Based on other replies one should pause, take a step back and consider if their code or design is to blame..

Funnily enough, we also use this approach in Google. There's at least once chapter in the first (the unactionable) SRE book about this.

As long as I want to avoid very edgy race condition cases (low throughput), any DB will work right?

Either Mongo, MySQL or List in Redis.

Is there anything sepcific about Postgres when it comes to this? (except SKIP TABLES, as mentioned in other comments)

I said postgres because it's just all around good at a bunch of stuff. But I've used redis lists in the past to great success. Redis streams might be even better but they weren't ready the last time I needed a queue.

rq ("redis queue") for python does this with, obviously, redis.

API is pretty small and comprehensible.

It can't. Not easily.

Job scheduling is trivial. Doing it reliably, tracably and in a way that handled failure predictably is really a pain in the arse.

There are zillions of ETL frameworks out there, just use those. some are unspeakably complex (airflow I'm looking at you) Some are simple, but require glue (AWS batch) some are trendy (argo on kubernetes)

However, most CI tools are effectively job scheduling systems. Jenkins works surprisingly well as a cron replacement for >100 machines

What Jon Richards said. Also see https://github.com/collectiveidea/delayed_job which is a popular DB-based scheduling library for Ruby.

C structs are really, really brittle; CSV is alarmingly complicated (unicode); JSON can't pass "None". I like CBOR, myself.

> JSON can't pass "None"

What about null?

Ahh, then I might be talking crap. I'll check it out, thanks.

You can pass null, undefined, or remove the property.

JSON Schema – a whole topic unto itself – gives you further control over locking down how this can be represented (e.g. field MUST be present but can be string matching this regex or null).

CBOR has extremely slow encoding/decoding in C/C++ when compared to structs. Nothing is perfect.

For sure, compared to structs. Which you can almost certainly RDMA from one place to another. It's a little bit apples and oranges though since CBOR is self describing (and very lightly compressed).

Shameless plug: https://github.com/RantyDave/cppbor

Reply on news.ycombinator.com