Deepkit – High-Performance TypeScript Framework
deepkit.io
deepkit.io
So as an experiment, I created a library that statically types raw SQL:
https://github.com/nikeee/sequelts
The idea is to parse the SQL queries using TS's type system. The parsed query is combined with the database schema and therefore, we know what type the query will return.
This is especially useful due to TS's structural type system. It's also zero-overhead due to the entire functionality being just TS types, which vanish as soon as we compile. It therefore also works JS-only.
However, it's just a proof of concept. I'm working on an ANTLR target for TS types, so that the SQL parser can be generated. A game changer will also be the integration with sql-template-tags [1] (which would make this actually usable).
This is just for selecting data. Time will tell if it's feasible to also type-check mutating queries.
The primary use-cases will target SQLite/D1.
It uses the actual PostgreSQL parser against your project's database schema to give 100% accurate results. Guarantees that all of your queries are valid, and that all input and output types are correct (and will also auto-generate/quickfix the output types for you in VSCode)
Similar idea - statically typing queries. Mine was mostly me playing around with recursive decent parsers seeing if I can actually parse SQL with it, and it seems to work OK, at least for Postgres.
It does require a "build" pass to generate the types, but I've added some additional bells and whistles like auto formatters and on hover info about columns / views / tables etc. Once you have the sql AST its pretty easy to build a lot of cool stuff around it.
It's all pure TS and doesn't use parser generators like ANTLR. We've been using it in prod for a while now and seem to be working alright - its mostly types anyway, but does have a runtime component, since sql parameters in node Postgres are too basic for our use case.
It all started from the amazing https://github.com/dmaevsky/rd-parse which showed me that you could build crazy complex parsers with like 100-200 lines of code.
> Requires knowledge about SQL & your database
For me that's a pros, because it is transferable knowledge.
And
> No type safety
Is a bummer, if you could check at compile time or some other way that your queries are valid it will be cool, in rust is a crate that does exactly that sqlx[0], and besides sql being verbose I found so easy and enjoyable to work with, it's so easy to know exactly what the query does, with ORM's is easy to have a query that's hard to know what does and the only way to be sure is running and printing the query.
My own library Zapatos[1] is less ambitious (it doesn't try to parse raw SQL) but has similar goals. It also has no truck with runtime types.
Thank you for your incredible work on this.
> Time will tell if it's feasible to also type-check mutating queries.
What do you mean by "mutating queries"? When the underlying schema changes? For instance when a new column is added to the `video` table, or if `video.id` changes to a varchar?
Haven't dived in deep enough yet, but is this using the DB engine's explain query, or I guess my question is how does it interrogate the underlying db/schema if the schema isn't supplied as code in the setup? Or at least, how would you envision all that happening?
It should be possible to parse those and check if they obey the schema, so raw SQL that updates/inserts/deletes data could also be checked.
Sometime when people use the term SQL, they mean DDL, DQL (data query lang) and DML (data manipulation lang), other times they mean just queries (DQL).
https://www.lloydatkinson.net/posts/2022/going-further-with-...
- D.I. https://deepkit.io/documentation/injector
- Topsort "A fast implementation of a topological sort/dependency resolver with type grouping algorithm" https://deepkit.io/documentation/topsort
- Templates https://deepkit.io/documentation/template
I'll stop there, I'm too lazy to do their work for them. Other red flags
- why is a very specific sorting algorithm bundled in?
- its claimed their RPC implementation is 8x faster than gRPC, specifically gRPC-js. Don't you think Google has the resources and the incentive to make gRPC faster? It's likely this is not an apples-to-apples benchmark.
- MongoDB ORM. It's a fine database, but it's also proxy indicator of web developers who only breath JavaScript. - Message broker. Very opaque, but it's likely this is coupled to a very specific pubsub provider.
- API console for testing HTTP calls. This is a solved problems with tools for testing APIs like Postman. The fact that the team is trying to solve all sorts of problems, each of which is standalone product, makes feel they are trigger happy at reinventing the wheel.
- For the view layer, it seems hard-coded to Angular. That's probably a no-go unless you don't care about what component library framework you have to use.
This project seems to be a single author project. Kudos for being ambitious, but I would heed caution at trying to solve every problem all at once. Maybe lean into the ORM aspect and make that best in class first.
Yet the author appears to be writing the parser in C++: https://mobile.twitter.com/MarcJSchmidt/status/1534323199147...
I don’t think this is a “Show HN” or “Launch HN” posting. It’s possible the author is still working on code and docs, and has no idea the site’s been posted here.
> Don't you think Google has the resources and the incentive to make gRPC faster?
You’d think that, but A) Gmail, and B) having worked for a Google competitor, I can assure you a passionate developer can write more performant code than what a massive corporation will generally produce.
> I would heed caution at trying to solve every problem all at once.
Agreed, but I hope the author throws caution to the wind. I’m tired of having either A) 1 million npm dependencies or B) boiling the ocean in my own projects, so it’d be nice to have a batteries included project you can rely on.
A lot of stuff is not yet documented, but they will. We are already writing new documentation/book which is roughly 40% done that covers already a lot more than what is currently seen on the website, but this HN post was posted at a somewhat bad timing.
> - its claimed their RPC implementation is 8x faster than gRPC, specifically gRPC-js. Don't you think Google has the resources and the incentive to make gRPC faster? It's likely this is not an apples-to-apples benchmark.
Yes, gRPC is fast. But not gRPC-js. I invite you to test it yourself, all our benchmarks are in the git repository. It is explained here on how to run them: https://deepkit.io/documentation/benchmark. Feel free to join our discord so I can guide you through the process. On the reasons why gRPC-js is slower: It uses Proto Buffers which is binary, and parsing binary is notoriously slow in JavaScript. By utilising the runtime type information of TypeScript we can generate JIT binary encoder/decoder for very specific types which are much faster.
> - why is a very specific sorting algorithm bundled in?
Why is providing something as a dedicated package a red flag? It's fundamental necessary for an UoW ORM for example and itself very useful. My other topsort package in PHP has 4 million installations, so it's only logical to bring the same functionality to TypeScript.
> Message broker. Very opaque, but it's likely this is coupled to a very specific pubsub provider.
It uses its own server, but will probably be replaced soon. Again, somewhat bad timing of that post.
> - API console for testing HTTP calls. This is a solved problems with tools for testing APIs like Postman
It doesn't solve what Postman solves. It's a way to automatically document your HTTP and RPC API plus allows to execute/try them right in the browser. There is Swagger UI which comes closer to what API Console is.
> - For the view layer, it seems hard-coded to Angular
There is actually no view layer. It has a server-side DI-enabled template engine based on JSX and a desktop UI kit (which is based on Angular). Both have their own use-case and are not required.
So performance would be one reason to chose Deepkit for me. The integrated framework looks interesting too. Something that make me hesitant is the relative small user base and size of the community (mainly one single developer?) considering this would basically be to commit to a new programming language. And if the Typescript team should decide to do anything similar then the project might be dead in the water?
(I don't see how you can cite Vue as an example; Evan is the main contributor to Vue, but he is certainly not the only one.)
On the other hand, I wonder what will be the level of support for each of these libraries - just the ORM or desktop UI seem like a lot of work.
I will never understand. Nor sympathize.
I suppose some developers enjoy doing this again and again. Many don't.
Let's say you build a little user management app and need a `User` type. So you define one:
interface User {
id: number;
username: string;
created: Date;
email: string;
}
If you want to build a REST API that returns that user, you can just do that: router.get('/user/:id', (id: number & Positive): User => {
//...
});
Deepkit automatically serialises the object into JSON using the type information in `User`.You want to accept a User object for creating a user in the database? No problem:
router.post('/user', (user: User): User => {
//...
});
You probably want to enrich the interface with additional constraints: interface User {
id: integer & Positive;
username: string & MinLength<3> & MaxLength<32>;
created: Date;
email: Email;
}
And with that the framework automatically validates all incoming requests against the User object and its containing validation constraints. It also deserialises it automatically from JSON for example. If the `User` is a class, it instantiates a class instance.But you don't want to require all fields, like id and created should be set by the server:
router.post('/user', (user: Omit<User, 'id' |'created'>): User => {
//...
});
This works equally well. And the best part: you get API documentation for free with the Deepkit API Console, just like you know from Swagger UI.Ok, but we need to save the User in our database, right? We do not need to duplicate the User in yet another language like with Prisma. Just re-use the User interface and annotate the fields.
interface User {
id: integer & PrimaryKey & AutoIncrement;
username: string & MinLength<3> & MaxLength<32> & Unique;
created: Date;
email: Email & Unique;
}
We still use the very same interface, but have added additional meta-data so you can use that exact same type now also for the ORM. router.get('/user/:id', async (id: number & Positive, db: Database): Promise<User> => {
return await db.query<User>().filter({id}).findOne();
});
Next, let's go to the frontend. We obviously have to request the data from our rest API. We can just do that and reuse the `User` interface once again. const userJson = await fetch('/user/2');
const user = cast<User>(userJson);
`cast` automatically validates and deserialises the received JSON. We import `User` wherever needed and this works because the meta-data is not tightly coupled to any library, so you won't pull in for example ORM code just because you use database meta-data. You can literally use TypeScript's powerful type-system to your full advantage in many small to super complex use-cases and let the framework handle all the validation, serialisation, and database stuff.And like that you are able to reuse types everywhere: From database to http router, configuration, RPC, API documentation, dependency injection, message broker, frontend, and more. That's not possible in this form in any other framework or when you combine lots of third-party libraries and glue them manually together.
Deepkit separated the functionality in libraries which would allow you to use their features in Fastify and Prisma, too, but that would mean you lose one of the biggest advantage of all: Reusing types.
When you start to hear people say of their PRs “it works but I just have to fix the types”, that usually means the codebase is trying to re-use too many types. In my experience.
The beauty of TypeScript is that you can have two different very narrowly specified types and the compiler will tell you if they’re compatible or not.
Reusing types is throwing away the most valuable feature of TypeScript.
For me, the type signature is part of the function signature and therefore should not be re-used.
Imagine what a disaster it would be if function signatures were reusable:
sig UserDatabase(user, db)
create: UserDatabase =>
db.write(“users”, user)
update: UserDatabase =>
db.get(“users”, user.id).update(user)
It’s too much. DRY can be taken too far.While it does sound quite great I have to try it in practise to understand would it work for me. To prevent framework fatigue I really try not to switch frameworks too often which is why I'd like to be able to reuse what I know already. And your scope sounds really big. Yet I always have liked how TS/JS community keep pushing the envelope constantly!
Wait, how can Deepkit's `cast` work on the frontend? Do you compile to WASM?
Quickly skimming the current Deepkit homepage and intro blog post didn't seem to hint at this.
My summary is that the DeepKit compiler is a JS transpiler (used as a ts-node or webpack plugin, or conveniently/invasively as a replacement of node_modules/typescript) that produces code like this:
//TypeScript
export type TypeA = string[];
//generated JavaScript
export const __ΩtypeA = ['&F'];
//TypeScript
function log(message: string): void {}
//generated JavaScript
function log(message) {}
log.__type = ['message', 'log', 'P&2!$/"'];
Where '&F' and 'P&2!$/"' are the compact bytecode translations of the types ("&" means string, "F" means array) and '__Ω' is a prefix used to avoid naming conflicts.Personally, I'd love to see a live example of code in & compiled code out on the homepage. I'm sure it's on your roadmap :)
> This does not work with Fastify and Prisma
The point of Prisma is that it generates types for you from the Prisma schema, right? Are the generated types not interchangeable throughout the application?
Fair point about expressiveness, but if your data must cross application boundaries then you are somewhat limited in expressiveness no matter what, because your types need to be understood by arbitrary clients.
Lack of generics is pretty annoying though. It's probably my biggest annoyance with GraphQL.
Prisma's DSL has no syntax for validation constraints, so even with openAPI generators it's incomplete.
[0] [url-redacted] [1] [url-redacted]
[0]. https://deepkit.io/assets/screenshots/debugger-http-light.pn...
It's still a project where the bus factor is extremely low: the author has 95% of all the commits. The project is basically reinventing every wheel in the typescript space, too, and it's going weird ways with a custom compiler and crazy reflection.
It's dogfooding hard, so it's all built on its own things, and all of those things are definitely going to have bugs because one guy wrote all of it, and it's just him to fix them.
I think it has a lot of neat stuff and cool things I'd otherwise like to have, but there is basically no way I can adopt this framework for anything serious because of the reasons above.
If I try the provided Dockerfile, Alpine can't install python for whatever reason.
It does not seem to have the code-generation (for the db table DTOs) though.
I also like typescript's "string & MinLength<3> & Email" type combining.
A quick edit-test-loop must be a great selling point of this kind of FW compared to FW in more strongly types languages (that have slower edit-test-loops due to compilation) that I usually prefer.
jooq is awesome. I was using that with DropWizard way back when. Now I'm just Go.
First, I really like that this leverages Typescript to its fullest. I'm a big fan of doing things like runtime reflection of compile time types.
However:
``` id: number & PrimaryKey & AutoIncrement = 0; ```
One thing that seems confusing is, (presumably) -- these compile time types are being translated into runtime type behavior. (I'm guessing by your Typescript transformer?). While this is very cool and concise, I suspect it could be super "magical".
In your case, has Deepkit felt too magical?
The places where I consider to use TypeScript are UI/CRUD/frontend centric, because I feel like the ecosystem around JS/TS is really rich and (finally) maturing when it comes to these things. So its really all about coordinating UI/user feedback, validation, serialization/parsing etc. And all that under regular change/evolution.
Now my conclusion so far has been to just try and keep it simple and separated and do the plumbing between all those parts explicitly. There is a benefit to that as well, but it does require more boilerplate and more bug surface. I will definitely tinker with some of the Deepkit libs to reassess this, because I think it might just do the right things in those areas.
How was it digging into the compiler api for that stuff? Every time I've poked around with that it's felt like a mess. Any resources you'd recommend?
I think you just did! (comment on something irrelevant to Deepkit, like the generic and uninformative landing page, that is)
I wish it had serverless support but I know that's a whole other beast, I have really enjoyed the majority of my typescript on serverless work and would love a formal framework instead of what I've cobbled together.
It does support serverless[1], as well as many other features you can expect from such a framework.
There's a general dislike towards Java/C# code nowadays, you can't argue that spring/springboot brought stability to the language and its mainstream usage.
Something like this in TS makes it feel familiar, and after trying the examples I am quite excited!
I like elixir/phoenix but I miss types and loving that this project is embracing types at its very core.
So... a hipster SpringBoot then? The history doesn't repeat but it rhymes indeed :)
why would I choose this over Deno?
`string & MinLength<3> & Unique;`
When one could use a Value Object Type?
Yes, I realize that the "loading" was really just Javascript effects to present each element individually. I'm sure that artists and designers love that stuff. But the people interested in a programming language framework are people who have things to do and value their time - otherwise they would not be looking for a framework.
Sorry to sound so negative.
Website is still not optimised though. It's a big SPA that loads most things in advance and followed a pragmatic mindset. Better done than perfect. It's low priority to make that faster, however shouldn't be a problem though as the target audience usually has big bandwidth.
There's also common sense. There are 33 words on that page, and it took 5 seconds for them to appear. No amount of design can justify that.
Additionally, the work will be served better if https://deepkit.io/framework is the home page, not the nothingness that is the home page
- Scripting language
Pick one.
https://www.techempower.com/benchmarks
Has entries for JS and PHP quick high in the ranks.
https://www.techempower.com/benchmarks/#section=data-r20&hw=...
I suppose you'd need to have Facebook-level scale for infrastructure to cost more than development in a "faster" language.
The whole sentence / tag line could probably do with improving, but maybe this is not a native speaker, making derision all the more unacceptable.
If you write junk like this to describe your software project I am immediately going to assume you are completely hopeless and not to be trusted at all.
Not one of these 6 words convey any meaning at all. It's like the bad old days of calling everything web scale.