Taming OpenAPI using Racket to create a DSL
developer.squareup.com
developer.squareup.com
> Scheme was originally called "Schemer", in the tradition of other Lisp-derived languages such as Planner or Conniver.
If there's any programming language family which condones trickery, I would think this would be it.
For context, the original title was:
> “This is “how I tricked my co-workers” into using Racket.”
But has now been changed to:
> Taming OpenAPI using Racket to create a DSL
Which is the original sub-title of the article. The real title is actually:
> Making OpenAPI / Swagger Bearable With Your Own DSL
I'm sure I'm going to get downvoted for this, but the use of Racket is a smell here. Why not use whatever language you were using before to turn the input DSL into the output OpenAPI spec? Sooner or later, another engineer is going to have to maintain that. You already have another language you're using for other things, and you're introducing an entirely new language and ecosystem to do one small thing. Is this something that couldn't be done in your existing stack?
Personally, my main problem with OpenAPI is that it is commonly used for documenting contracts, but very rarely for actually driving the implementation. In practice, I have seen a few implementations like OpenAPI Generator [0] but a) with dynamic languages like Python code generation is an anti-pattern, and b) even disregarding the former, the generated result is (in my experience) very incomplete and still needs a lot of manual coding.
I would like to see more frameworks that utilise OpenAPI for routing and request/response validation.
I built something like this at my last job by having an opinionated layer on top of Flask in Python. The idea was to have a small set of standard operation types (think:create, retrieve, update) and choices as to input/output types (via the marshmallow library ) and the function to process the route. This turns out to be enough to generate both the api contract and the boilerplate for routing, (un)marshalling, etc.
You can find the implementation in this repo: https://github.com/globality-corp/microcosm-flask
1. Just like any other documentation, it is difficult to verify that the documentation is actually correct and does not diverge from he underlying implementation. This is a lesser issue, as it can be mitigated with good integration testing practices.
2. The bigger issue, for me, is that any clients depends on the service implementation. In practice, it is very common to build clients -- especially complex ones like React or mobile UI -- alongside the service development, and if the spec is defined first both can be done in parallel. With the above approach you either need to a) wait for the service implementation to be complete before you can start implementing the client, or b) base the client on an OpenAPI spec which might potentially differ from the one your framework will generate based on the implementation.
I have recently worked on a project which had that same issue, and our initial solution was to build tests which would compare the generated OpenAPI with the design specification, but that turned to be extremely complex when we started running into all of the edge cases.
The alternative was to treat OpenAPI as the single-source-of-truth, using it to generate routes and execute validation over requests and responses. The first attempt used Connexion [1], which proved to be a bit too incomplete for our needs, so we implemented an alternative framework [2] (which includes a basic client-side support as well).
[0] https://fastapi.tiangolo.com/
[1] https://connexion.readthedocs.io
[2] https://github.com/berislavlopac/pyotr (still under heavy construction)
My approach is to assign projects to teams with members who can work on both the client and server layers so that the question is less of client blocking on server and more of client and server working together.
In addition, I like an "API first" approach, which in this case means building the signature of the API functions (e.g with mock data) before finishing the implementation.
That is: client is still blocked on server to define the API, but they are not blocked on implementing the API and client works closely enough with server (or is able to do both) such that they aren't blocked on the definition.
As always, many software problems devolve into people problems once you stare at them hard enough.
You could simply write the complete interface down in Python/FastAPI without actual implementation and generate the OpenAPI spec from that interface. That way both teams could start soon.
Sure the generated code is usually not readable which is against the zen of python. But writing code to generate code, especially if it's contract driven, is the correct approach no matter the language.
So really what I'm looking for is more substance of why generating code is an anti pattern in python but not in C.
With Python there's normally no build step – you just run the program. Adding code generation means adding another step, which you might forget to run, leading to confusion. If you make the build step mandatory you lose some of the upside of dynamic languages.
This is something that's baked into a gate check on merge to the source repo. Or rather it should be.
I've not seen any long lived python projects that don't end up implementing runtime checks to validate the shape of these generated objects.
Autogenerated code falls into two categories:
1. Code templates, which prepare some stubs but you need to complete it manually. This is what openapi-generator does, and this is anti-pattern because very rarely this approach can't be avoided in favour of simply using code constructs like inheritance or composition.
2. Intermediary code, which is not to be updated manually and is fed directly into interpreter or compiler. In compiled languages it is fine, as this intermediary code is not executed directly; what matters is whether it compiles. In interpreted languages, however, that code gets executed directly, and it can be really difficult to debug (as there is an additional level of abstraction, introduced by the macro). But ultimately there is no need for this type of code, as most interpreted languages (Python and JavaScript definitely) allow dynamic construction of various programming constructs (like classes and functions) on the fly.
Yes. If you look at the most popular code-gen for Python, which is protobuf - it does the generation very badly.
That generated code is not only unreadable by humans - it’s not readable by IDEs. Protobuf author later on made a point that was a mistake.
And that’s from one of the most popular projects.
I feel like the author is forgetting how software gets developed in the real world... or just human nature in general.
Documentation created with this approach will instantly go stale, and will likely get out of sync with the true API very quickly.
How is this a good solution at all? How is “tricking” people into this unsustainable situation a good thing?
Nothing in the article suggests that conversation didn't happen. The only mention of "tricked" is the headline the submitter gave it on HN.
This is self-contained enough that I could see the language not mattering too much. And while I don't use Racket at all it sounds like the ideal language for this task. I'm not saying it confidently, but I wouldn't be surprised if the Racket implementation is less buggy than any non-lisp. Generating YAML/JSON is a task which lisp might be uniquely suited for.
For the curious, we took and hacked https://github.com/metosin/spec-tools to output OpenAPI 3, and added a very thin shim to hook that up to our routes.
If Racket was added for this purpose then here are a couple boring[0] solutions that could have lower operational costs.
Generate OpenAPI specifications from your service code.
Use a YAML preprocessor/template engine such as [1]. I have used a similar technique to generate SQL queries.
It's language specific, but if a variation with minor attributing extensions existed that seamlessly round-tripped to yaml, the verbosity would just completely collapse.
Readability and creating from scratch could be an order of magnitude more efficient for anyone familiar with JS (reading a handful of interface definitions vs. human parsing back and forth between the docs and YAML).
- identify an isolated, low-risk need.
- develop a solution in your tech (usually on your own time).
- introduce to your boss/company as already developed.
The biggest risk for the business is needing to replace the component. In these cases usually an isolated, irregular solution is acceptable
Yes this is a very pedantic thing to complain about and I do understand what is meant. But it does bother me. Yes, I do realize it's using slanted quotes and not just " - still doesn't read right for me.
Never do this in prod without a heads up and approval from most of your team.
I worked with a guy who after completing his tickets instead of looking for more tickets or helping his team, spent his time rewriting the entire clientside application in typescript. He was let go the following month.