Smithy: A language for defining services and SDKs

Ended up building protoforce.io for the very purpose. It does support typescript + nodejs, which we used for the website itself as well.

Note: Principal at AWS.

Questions:

1. SCALA: is the generated code ONLY scala? are other languages supported?

2. CODE-GEN: is it designed only for code generation for a target http framework or does it actually provide an "API Server" itself ( e.g. like graphQL )

3. COMPARISON: the list of frustrations with other solutions, listed in your intro/technology goals are somewhat high-level. what are the "leaky abstractions" or "non-deterministic behaviour"?

4. SETUP: does someone need to know scala to use this ? ( depends on #1 above )

5. DSL : is the protoforce code implemented as a DSL in Scala or its a "language" in itself?

6. IDE : how do you check/compile the code? does it integrated with an IDE?

As a Scala Engineer myself ( though I mostly work in Kotlin now for Android/Server ), this looks great, but most Scala engineers i've met are focused on Spark, and using Play Framework/Http4S, etc. How big is the actual market for Scala API tools?

The services are free to serialize the data in any way be it json, XML or even protocol buffers.

So Smithy is more comparable to OpenApi than Protocal buffers.

We started working on Smithy around 2018 because we wanted to improve the scale of our API program and the AWS SDK team to deal with the growing number of services (over 250 now!) and languages we want to support in official AWS SDKs (like the newly released Rust SDK). We had a ton of existing services that we needed to be compatible with, but we also wanted to add new features to improve new services going forward too.

We needed a very flexible meta-model that allows us to continue to evolve the model to account for things like integrating with other systems and to model service-specific customizations that each AWS SDK team can implement independently. Smithy's meta-model is based on traits, a self-describing way to add more information to models. Lots of validation can be built in to custom traits, which helps to ensure that service teams are using traits properly and adhere to their specifications. Smithy's resource modeling helps us here too because it allows AWS service teams, as they adopt Smithy, to essentially automatically support CloudFormation resource schemas. Resources also help us to point service teams in the right direction to make their services work well over HTTP (which methods to use, URIs, safety, idempotency, etc).

We needed an integrated model validation, linting, and diff tool to keep services consistent and detect breaking changes, and it needed to support company-wide standards as well as service-specific standards. We use Smithy’s validation system to automatically enforce API standards, and service teams often create their own service-specific rules to keep their own internal consistency.

We needed built-in input validation constraints so that they're standard across services and clients (e.g., length, range, pattern, etc). We didn't want to rely on third-party extensions to provide this feature since validating inputs is important. AWS uses internal service frameworks that enforce these constraints and are compatible with Smithy models. We're working to create open source service frameworks for Smithy as well.

We also wanted to support various serialization formats so that clients work with all of our existing services spread across JSON, XML, query strings, RPC, and HTTP APIs, but we also wanted to be able to evolve our serialization formats in the future as new technology comes along. That's why Smithy is protocol agnostic (like gRPC actually). The serialization format is an implementation detail. Smithy has some support for MQTT as well.

And finally, we need our code generators to be really flexible to support service customizations. There's quite a few customizations across AWS services, and we needed a way to inject custom code generation logic in various parts of our generators.

Smithy is still in heavy development, and we're working on building out more of the tooling so it can be used easily outside of AWS SDKs too, including client and server code generation.

We're working to get projects like these to GA: https://github.com/awslabs/smithy-typescript, https://github.com/aws/smithy-go, and https://github.com/awslabs/smithy-rs. And we're also working on service code generation.

Because Taxi lets you describe how data from services relate, Vyne can work out how to connect services together automatically, and handle the integration for you. This is a realisation of the UDDI concept, where systems can autonomously work out how to operate with each other.

Java sdk is built on Futures, so somehow it's async as well.

Also we support server-to-client calls, which, effectively, are a better alternative to callbacks.

This is currently used in Amazon S3, Kinesis, Transcribe, and other services.

Smithy doesn’t have a service registry today. However, models can be vended and shared via Maven. Client codegen was designed explicitly to not require coordinated releases of clients and servers (that’s impossible for AWS SDKs).

> Waiters are a client-side abstraction used to poll a resource until a desired state is reached, or until it is determined that the resource will never enter into the desired state.

I havent seen them put behind an async interface but it seems like a good match for a lot of StartWork, DescribeWork(token) patterns.

That said, traits can be used in Smithy to enforce constraints on structures, so if you ever needed explicit indexing like that, it could be done via traits and protocols (Smithy's nomenclature for describing how clients and servers communicate). In fact, protocols are defined by traits, and traits can enforce requirements on the rest of the model using a DSL called selectors... Probably way too much info other than -- it's possible and easy to support this in Smithy if it's ever needed.

As for required vs optional -- today it's treated as server-side validation only and not used in client codegen. This allows service teams to remove the required trait from members if something ends up needing to be optional in future without breaking clients. We're working on some ideas too to see if we can generate even better code for SDKs in languages like Rust where optionality is very explicit, but without sacrificing the ability of being able to remove the required trait.

And, in general, backward compatibility issues are caught with Smithy diff, which also supports custom rules: https://github.com/awslabs/smithy/tree/main/smithy-diff

list Messages { member: Messages }

Then you can reference Messages from other places in the model, like from a structure:

structure Something { messages: Messages }

In contrast, the `[City]` syntax is used in other places in the IDL to define a relationship to a shape. This isn't something that gets sent over the wire, it's just used to form essentially a relationship in the service graph from a service to resources, a resource to operations, an operation to errors, etc. For example:

service Weather { resources: [City, Sensors] }