Toasty, an async ORM for Rust
tokio.rs
tokio.rs
[1]: https://diesel.rs/
I'm mainly use sqlx, it's simple to use, there's query! and query_as! macro which is good enough for most of the case.
I remember fighting with handling enums in relations for a while, and now just default to manually mapping everything.
SQLx sucks at dynamic queries. Dynamic predicates, WHERE IN clauses, etc.
For SQLx to be much more useful, their static type checker needs to figure out how to work against these. And it needs a better query builder DSL.
It doesn't end up being too bad though, except for the loss of compile time syntax checking. Manually handling joins can be kind of nice, it's easier to see optimizations when everything is explicit.
The codegen part makes all columns and tables and stuff checked at compile-time (name and type) like Diesel, with a query builder that's more natural like SeaORM. I hope the query builder does not end up too magical like SQLAlchemy with its load of footguns, and stay close in spirit to Diesel that's "write sql in rust syntax".
I think time will tell, and for now I'm keeping my Diesel in production :D
Sooner or later we always hit the n+1 query problem which could only be resolved by a query builder or just plain old sql.
It always was a mess and these days I can't be bothered to try it even anymore because it has cost me a lot of hours and money.
Congrats, you now have your own little ORM.
Op is never implying they intend to maintain one to one correspondence between the DB and objects and do that through manipulating objects only. Mapping hand written queries results to structs and updating the DB yourself on the basis of what is in structs is not at all an ORM.
You just described a bad, home-grown Object Relational Mapper.
Through that lens, the parts where you load and save object state are redundant. You're going to throw those objects away after the request anyway. Just take your request and build an UPDATE, etc. Use record types merely as a way to define your schema.
When I was in RoR world, pretty much every N+1 query I saw was due to lack of RTFM.
[1]: I made this up
Because it's half a dozen joins and hence no N+1 query but actually N*6+1 queries...
And yes, RTFM is nice, problem is: it's my fucking partners that should've done this before we shipped it to the customer which they abandoned and I did not.
And not even that, really. I’m sure you can understand that it is possible to call functions that execute SQL statements inside a potentially nested loop? What’s that got to do with an ORM.
They all ended the same way and whenever I have a say, I'll oppose ORM, because of my subjective experience.
You wont change my mind, especially not with any snarky comment that starts with "I am sure you can understand"
Maybe you're just a way better engineer than me, maybe you could work on other projects where ORMs were a lovely fit and you didn't end up having to join half a dozen to a dozen tables because of ever changing requirements and a very convoluted domain. In either case: I am super happy for you. For me ORMs have never delivered on the promise. The easy things are easy, but the complicated things are severely underperforming and depending on the tech stack one cannot even fall back to raw queries (yes, you heard that right) within the framework. To me the easy mappings are nice, but the downside is that my colleagues are sometimes lazy (depending on the project) and don't even see an issue with "Well, then the client has to wait 2 minutes for this simple Webapp to render" (again, this EXACT thing happened twice to me now. Once I was finished with the refactoring the client had to wait 8 seconds with 10 times the data - really)
So while it may not be the fault of the ORM, it certainly didn't help the project as a whole, with the team members as they were.
With that being said: have a wonderful day/night and a lovely start of the week!
never saw that in any of the stacks I had to use... care to share what stack didn't let you fall back to raw queries ? So we can avoid those.
On the other hand an async orm sounds like (n+1)(n+2)+...+(n+m) Problem
> The common wisdom is to maximize productivity when performance is less critical. I agree with this position. When building a web application, performance is a secondary concern to productivity. So why are teams adopting Rust more often where performance is less critical? It is because once you learn Rust, you can be very productive.
> Productivity is complex and multifaceted. We can all agree that Rust's edit-compile-test cycle could be quicker. This friction is countered by fewer bugs, production issues, and a robust long-term maintenance story (Rust's borrow checker tends to incentivize more maintainable code). Additionally, because Rust can work well for many use cases, whether infrastructure-level server cases, higher-level web applications, or even in the client (browser via WASM and iOS, MacOS, Windows, etc. natively), Rust has an excellent code-reuse story. Internal libraries can be written once and reused in all of these contexts.
> So, while Rust might not be the most productive programming language for prototyping, it is very competitive for projects that will be around for years.
It is this culture thing makes adopting Rust for web apps worthwhile - it counters the drawback of manual memory management.
If you hire an engineer already familiar with Rust you are sure you get someone who is sane. If you onboard someone with no Rust background you can be pretty sure that they are going to learn the right way (tm) to do everything, or fail to make any meaningful contribution, instead of becoming a -10x engineer.
If you work in a place with a healthy engineering culture, trains people well, with good infra, it doesn't really matter, you may as well use C++. But for us not so lucky, Rust helps a lot, and it is not about memory safety, at all.
As time passes, the more I feel a minority in adoring rust, while detesting Async. I have attempted it a number of times, but it seems incompatible with my brain's idea of structure. Not asynchronous or concurrent programming, but Async/Await in rust. It appears that most of the networking libraries have committed to this path, and embedded it moving in its direction.
I bring this up because a main reason for my distaste is Async's incompatibility with non-Async. I also bring this up because lack of a Django or SQLAlchemy-style ORM is one reason I continue to write web applications in Python.
And in the opposite situation, if you call an async function then you are doing IO so your function must be either async or blocking, there's no third way in this direction, so when you're doing IO you have to make a choice: you either make it explicit (and thus declare the function async) or you hide it (by making a blocking call).
A blocking function is just a function doing IO that hides it from the type system and pretend to be a regular function.
There's a fundamental difference between CPU heavy workload that keep a thread busy and a blocking syscall: if you have as many CPU heavy tasks as CPU cores then there's fundamentally not much to do about it and it means your server is under-dimensioned for your workload, whereas a blocking syscall is purely virtual blocking that can be side-stepped.
Also, the IO and the execution being completely tied (the executor provides the IO) is a wrong choice in my opinion. Hopefully in the future there is a way to implement async IO via Futures without relying on the executor, maybe by std providing more than just a waker in the passed-in context.
It's more a consequence of having let tokio becoming the default runtime instead of having the foundational building blocks in the standard library than a language issue. But yes, the end result is unfortunate.
It’s really not that bad, you might just need a better mental model of what’s actually happening.
So you use gevent/greenlet?
Creating your own file format is always difficult. Now, you have to come up with syntax highlighting, refactoring support, go to definition, etc. When I prototype, I tend to rename a lot of my columns and move them around. That is when robust refactoring support, which the language's own LSP already provides, is beneficial, and this approach throws them all away.
In this case, for example, it looks like the generated code needs global knowledge of related ORM types in the data model, and that just isn't supported by proc-macros. You could push some of that into the trait system, but it would be complex to the point where a custom DSL starts to look appealing.
Proc-macros also cannot be run "offline", i.e. you can't commit their output to version control. They run every time the compiler runs, slowing down `cargo check` and rust-analyzer.
Prisma is popular enough it also has LSP and syntax highlighting widely available. For simple DSL this is actually very easy build. Excited to have something similar in Rust ecosystem.
I have the afternoons of my past week trialling to see if you could achieve something similar to Toasty with just structs and proc macros.
https://github.com/jayy-lmao/sql-db-set-macros
Still WIP but made it past the hurdle of inserts, which I decided to generate a type-state builder pattern to enforce non-nullable fields and skip auto-fields. This is more intended as a proof of concept but I’ll see how much I can grow it and whether I can dogfood at my job
Ideally I would use something akin to Go Jet.
Great to see some development in this for Rust, perhaps after it becomes stable I may even switch my SaaS to it.
I also have a relatively successful saas that uses Prisma and it’s been phenomenal. Queries are more than fast enough for my use case and it allows me to just focus on writing more difficult business logic than dealing with complex joins
Your code ends up using the driver raw in these cases, so why not just use the driver for everything? Your codebase would be consistent at that point
Making that 99% smaller, simpler and automatically mapping to common types makes development a lot easier/faster. This applies to pretty much any higher level language. It's why you can write in C, but embed an ASM fragment for that one very specific thing instead of going 100% with either one.
You can extend diesel (and probably many other orms, Diesel is just particularly easy here) to support any db feature you want.
> It is highly unlikely that an ORM provides support, much less a good abstraction, over features that only 1/N supported DBMS have.
That depends on orm flexibility and popularity. It may not provide support OOTB, but can make it easy to add it.
> Your code ends up using the driver raw in these cases, so why not just use the driver for everything? Your codebase would be consistent at that point
Main point of using orm for me is that I have type verification, raw (as in text) breaks too easily.
Might have improved since last I checked, but I was pretty confused.
Case in point Django is really good about DB-specific functionality and letting you easily add in extension-specific stuff. They treat “you can only do this with raw” more or less as an ORM design API issue.
My biggest critique of Django’s ORM is its grouping and select clause behavior can be pretty magical, but I’ve never been able to find a good API improvement to tackle that.
The simplest example is you can't build a Django object with a collection on it. Take the simplest toy example: a todo list. The natural model is simple: a todo list has a name and a list of items. You can't do that in Django. Instead you have to do exactly what you would do in SQL: two tables with item having a foreign key. There's no way to just construct a list with items in it. You can't test any business rules on the list without creating persistent objects in a db. It's crazy.
So yeah, Django lets you do loads with the relational side, but that's because it's doing a half-arsed job of mapping these to objects.
But then you have actual properties on your todo list. So even in your object model you already have two classes, and your todo list has a name and a list of items.
So there's not one class, there's two classes already.
As to "having a list", Django gives you reverse relations so you can do `my_list.items.all()`. Beyond the fact that your persistence layer being a database meaning that you need to do _something_, you're really not far off.
One could complain that `my_list.save()` doesn't magically know to save all of your items in your one-to-many. But I think your complaint is less about the relational model and much more about the "data persistence" question. And Django gives you plenty of tools to choose how to resolve the data persistence question very easily (including overriding `save` to save some list of objects you have on your main object! It's just a for loop!)
You can only do `my_list.items.all()` if you've already saved the related records in the db. And if you do something like `my_list.items.filter(...)` well that's another db query. A proper ORM should be able to map relationships to objects, not these thinly veiled db records. See how SQLAlchemy does it to see what I mean. In SQLAlchemy you can fully construct objects with multiple layers of composition and it will only map this to the db when you need it to. That means you can test your models without any kind of db interaction. It's the whole point of using an ORM really.
In my experience, Dynamo and other NoSQL systems are really expressive and powerful when you take the plunge and make your own ORM. That’s because the model of nosql can often play much nicer with somewhat unique structures like
- single table patterns - fully denormalized or graph style structures - compound sort keys (e.g. category prefixed)
Because of that, I would personally recommend developing your own ORM layer, despite the initial cost
Developing your own ORM is almost always a waste of time and a bad idea.
I'm not a rust person so I might just be exposing my ignorance here, just wanted to provide feedback since it's on early development.
It reminds me of Prisma and yet, it's all Rust. Also good to see that async is the focus point of the API so the usage feels ergonomic.
https://docs.sqlalchemy.org/en/20/orm/queryguide/relationshi...
the older I get the more I'm convinced this should be the default behavior.
Diesel hasn't been providing an async API for reason told in this thread: https://github.com/diesel-rs/diesel/issues/399
The situation might change some day though, once async support in the core language and surrounding ecosystem gets stronger.
Further, the project is focused on implementing the active record pattern, so it would be more appropriately called an async active record than an "async ORM".
Still to me they all suck and nothing beats SQLx
AI writes amazing SQL, modern SQL databases are incredible and the best way to get the most out of your DB is write SQL.
Invest your learning budget in SQL, not in some random developers abstraction.
There's plenty of value in knowing both.
"AI writes amazing SQL" and "AI writes amazing DB to Application Translation Layer Code" just means "AI can write your company's bespoke ORM".
ORM has nothing to do with queries. It is about the data. You will still want to map your database structures to your application structures (and vice-versa), else you'll end up with a weird and ugly chain of dependencies that will forever ruin your codebase.
2. I may be misinterpreting you, but you seem to be talking about composition of data, while the rest of us are talking about the composition of the language.
But, hopefully I've misinterpreted you. Perhaps you could demonstrate how query builder composition is best replaced in pure SQL?
I'm also not sure how to answer your question. Obviously a query builder is just spitting out queries that you can just write yourself. In the best case, they're a thin wrapper to give different names to SQL (e.g. where vs filter, select vs map, join vs flatMap). Perhaps an example would be how frequently, ORMs encourage you to do a load-modify-save pattern that turns into multiple statements to execute. This is usually more code, more error-prone, and worse performing than just doing an UPDATE WHERE. If you need complex reporting with a dozen joins and several window functions, you'll also see how difficult it is to write similar code in an application.
I'm not sure what you mean with composition of the language. The language consists of operators which you can chain together, and you can substitute expressions into each other to form larger expressions. E.g. you can join to a (select from where group by) expression in place of a table, and you can factor such things out into CTEs. What's not composable?
Duplicate rows, NULLs, ordering, etc. But there is no distinction to be made here, just calling attention to your grievous error so you don't make it again. We don't want to look like a fool again next time. Viva la education!
> I'm also not sure how to answer your question.
You put a lot of effort into a rather detailed response not once, but twice. You obviously know how to answer the question at a technical level, but perhaps you don't understand the nuance of my question? What I mean is: Show us some example code that demonstrates query builder composition and the pure SQL that you would use to replace it.
I'm not sure what you have in mind either for query builders or their composition. Like I said, some builders are really just wrappers to rename SQL operations and have a method chaining syntax. Those are always going to compile to obvious, analogous sql (e.g. `Users.filter(_.id==id).map(_.name)` will compile to `select name from users where id=?`. For the most part I think these are fine but maybe redundant. Then there are ORMs that do a bunch of state tracking and might produce multiple statements from one expression. These are usually what people get opinionated about. What's an example of query builder composition that you think can't be written in SQL?
The T in CTE literally stands for table. Even if you're right about the others, how could someone not think of that one as being a table? Regardless, now they can! Isn't education a wonderful thing?
> postgresql documentation uses
In fairness, Postgres originally implemented QUEL, which is relational. It didn't move to SQL until quite late in its life. It often takes longer to update documentation than to write code, especially in open source projects where the contributors tend to much prefer writing code over updating documentation.
> and have a method chaining syntax.
And this is often how composition is implemented. Not strictly so, but a common way to do it. Consider:
users = select("*").from("users")
admin_users = users.where("role = 'admin'")
recent_admin_users = admin_users.where("created > LAST_WEEK()")
Of course, you could do the obvious and write out 100 completely separate almost identical SQL queries, but that's not exactly maintainable and it's not the 1940s anymore. You are going to quickly hate everything about your existence as soon as those queries need to change. This is the reason people turn to query builders. If you only had to ever write one or two queries there'd be no point, but that never happens within the domain where these tools are used.
But perhaps there is a better way. This is where you would give us example code to show how you would replace that query builder code with a pure SQL solution.
create view admin_users as select * from users where role='admin';
create view recent_admin_users as select * from admin_users where created > LAST_WEEK();
I wouldn't use views for something so small, but I probably wouldn't use a query builder either. If you want a stable API, make a view to indirect access to the table(s). Don't break your view API. If you change the underlying table, update the view to keep it as a stable interface.
Query builders can be nice for generic code. E.g. you have a variable length list of predicates (e.g. from http query strings) and want to do `predicates.fold(_ => true)(_ and _)`. In that case you're basically using it as a macro system, which works because sql fragments compose. In fact IMO the most pleasant way to use a query builder is usually exactly as string interpolation macros.
ORMs, the original topic at hand, are an entirely different beast. I think generally people who bash ORMs don't have much issue with query builders.
That gets you halfway there, perhaps, but I'm not sure it explains how you would use it in an actual application. Are you back to writing hundreds of "SELECT * FROM recent_admin_users"-type queries in the application, once again exploding the development effort and maintenance surface – exactly what you're trying to avoid with these types of tools?
> ORMs, the original topic at hand, are an entirely different beast.
The original topic at hand is active record, not ORM. ORM is, as the name literally tells, about mapping between relations (or, in practice, tables) and objects. No sane person is going to bash ORM. They may dislike a particular toolkit that tries to help with ORM, but at very least they are going to do ORM by hand.
But that's not what we're talking about. The original comment that set the context for this particular discussion is about query building. It literally proposed using an LLM to generate queries instead. You can query build in the active record style: e.g. `User::all().admins().latest()`, but that's still query building, exactly like the earlier example except with a different API.
Every language has it's own (multiple) language specific ORMs.
There's something fundamentally broken with SQL syntax, and yelling at people to "Just Use SQL" doesn't really help.
There's a pattern of "We can't change X, so we'll write Y that transpiles down to X". It happens often with closed source tools and others that can't or won't implement new languages. Verilog, SQL, Javascript, all fit that bill.
E.g. Why is Javascript the only first class language for the browser? For the longest time JS was the only game in town.
? Are you referring to something specific?
Also, instead of a reactionary "all ORMs are trash," where ORM probably means different things to different people, maybe you could provide some value to the conversation by providing specific points and/or arguments supporting your feelings about ORMs. At the very least, you could provide some citation to an article that does the summarization.
For example I'm familiar with https://gorm.io and it does save me a lot of time and useless boilerplate.
And guess what, if I ever need to make a complex query, I also happen to know SQL, and I'm just going to make a "raw" query https://gorm.io/docs/sql_builder.html#Raw-SQL and be done with it.
It's not all that hard.
edit:
the other common complaint usually is: "but I don't know what query the orm is going to make..."
Use "Debug" https://gorm.io/docs/session.html#Debug and it will print out exactly what query it's making. Not happy with it? Make a raw query.
Objects are just bad abstractions for representing database interactions. They never map cleanly and any attempt to cover up the incoherence introduces further problems.
Avoiding boilerplate is understandable, but typed schemas and queries exist without hauling in a full ORM.
Of course you can pump out a lot of SQL very quickly with ORMs. There's a lot positive to say about this approach! But you don't tend to end up with code where you can easily tell what's going on.
But they're excellent abstractions for business entities.
> Of course you can pump out a lot of SQL very quickly with ORMs. There's a lot positive to say about this approach! But you don't tend to end up with code where you can easily tell what's going on.
15 years building up massive apps with Django and SQLAlchemy and this has never been a problem.
I guarantee you it is if the reader isn't already intimately familiar with SQLAlchemy.
You definitely make good points, and all my issues are over-comable with time and effort and knowing which tricks to use, so this truly is a matter of opinion and taste. I'm just pointing out that the idea that this produces more readable code seems far from obvious.
I'm back to using bespoke query builders and raw SQL since I don't use Python much anymore, but sometimes miss the syntax and migration system.
A query/DML that is badly optimized is usually very complex, which means that the ORM's syntax isn't a good fit. No problem, since they typically support raw SQL calls, so nothing is really lost.
You just have to know when to use the right tool for the job, as always.
I think the exact opposite is true, actually—because of the introduction of stuff like lifecycle hooks it becomes very difficult to figure out how the domain semantics translate to query/execution semantics. Of course some are lighter than that and just e.g. map a table to a domain type (which is much more readable), but that's not typically what catches criticism.
When was the last time your RDBMS was handling business logic and wasn't just a persistence layer for your application? SQL (the language) isn't composable so you often have a choice of building strings or having many slightly different queries.
Anyway, pros of using an ORM outweight it cons in my opinion.
Some ORMs are better than others but if if you have ever looked at a database created by a ORM it always has weird lookup tables, funny names and even with simple objects completely unusable without the ORM.
We live in a multi language world. There is a high chance you are going to want to access this data with a different language. This langue is not going to have the same ORM as such you will have a horrid time access the data.
ORMs often lay their data out in a way that is highly language dependent.
The fact of the matter is SQL is not hard and bypassing the main interface to the database and hand wave it away will always bring you regret when writing any sooty of software other than a toy project.
Your best case is you become an expert in the ORM. Resulting in a skill set that does not transfer easy, language locked and the worst of all bottle necks any changes to the data layer to your ORM expert who at first will be proud and happy and end smug and bitchy as all the data layers request changes will simply be redirected to them.
When people like me say ORMs are trash it’s much more than any of the surface level rebuttals listed here. It’s about the whole life cycle is your project. Adding up all the places ORMs can fuck you just makes it a bad proposition.
God forbid you need to upgrade your database or the ORM version.
You do realize that you can make your own migrations using raw SQL, and still use the ORM with your tables?
Which ORMs did you use? This doesn't sound normal at all. Never saw this with Rails, Ecto, EF, Sybase, or even the legacy project I once worked on that had 4 different ORMs for different parts of the same backend process, using the same database (some were very old and behind slowly phased out over time). Maybe you have ORM confused with CMS (content management system). A CMS can do those things, but that is not an ORM.
> There is a high chance you are going to want to access this data with a different language.
There are tools for that, such as ETL, read replicas, data warehouses, code generators, raw sql, stored procedures, views, just off the top of my head.