Validating Data in Elixir: Using Ecto and NimbleOptions | Hacker News Reader

Validating Data in Elixir: Using Ecto and NimbleOptions

blog.appsignal.com

114 pointsamalinovic59 comments

A lot of people don't think of Ecto when doing data validation outside of the database, but that's a mistake! It's a really powerful design pattern and implementation, and I've found it to be immenseley useful. It's clear from inception that they recognized it's general-purpose usefullness for data validation, so you don't have to hack it to make it work. Very useful tool in the toolbox

For sure, Ecto.Changeset "client-side" validations are a powerful tool even without any Ecto.Repo to insert the validated changeset into. You can just take the validated Ecto.Changeset, and "bake it down" to either the new/modified data, or the delta, and use that. And you get proper per-DDD-use-case type-casting "for free", through Ecto.Changeset.cast invoking Ecto.Type.cast on all your schema fields.

Though! I would say that you might just as well be making a mistake by not at least considering the question, when you reach for "raw" Ecto.Changeset validation, of whether whatever subsystem this is that has all these database-like needs that you're plumbing your way into, is just a Greenspunning of a database — something you could potentially rip out and replace with a database, and in so doing, end up with an easier-to-understand, easier-to-maintain system. (It's "easier-to-understand" not in the sense that your system is now less complex; but rather in that the complexity is now constrained by a formal abstraction — one that allows reasoning about the behavior of that component on a higher level. It's the same reason you'd turn a generic server "with state", into a finite-state machine; or turn a generic recursive-descent parser into a PEG grammar.)

The Erlang runtime is nice, in that it has in-memory database-like features (e.g. ETS), that let you get pretty far in writing internal state-management logic, without using any kind of database. But when you find yourself writing code that relies on 1. taking complex user-submitted data, 2. validating it and casting it into domain types, 3. inserting it record-wise into an ETS table, 4. and then manually indexing that table across multiple fields... maybe it's time for a database!

(It doesn't have to be a big database. Or even one that involves any external dependencies. There's an Mnesia adapter for Ecto, which is a combination I've always found fascinating — especially in the context of a single-node, memory-only Mnesia setup. It's an ORM over transactionally-joined ETS tables! But of course, if you have control over your deploy-time environment and so can suffer a few native deps, then SQLite works great as your "internal database" too.)

same here, so much so that i usually encourage teams to change any defstructs to embedded schemas.

Definitely. I’ve used it on general input validation for many endpoints over the years.

I tried out Elixir a couple months back, but had no end of problems getting the tooling set up in VSCode.

I have to say that my experience could not have been more different.

So far, everything I’ve done within the Elixir ecosystem has “just worked” (with the exception of some stuff with supervisors, the config and intuition for which are currently eluding me).

For example, I managed to code a full image processing pipeline (classification, labelling, and description) using Elixir, Nx, Bumbkebee, etc on the train on the way into work the other day. Which is about a 35 min trip.

If we’re talking about tooling, the last time I tried to just print “Hello, world!” in either Python or JavaScript, it took me down a hell dungeon of configuration and dependency nightmares. To be fair to both, tho, that was some time ago so I can only hope that situation has cleared up more recently.

I was mainly interested in working with Nx and friends, which look pretty nice. Good to hear about your experience with that.

The ElixirLS server always crashed for me on init. The problem is assuredly on my end, but I was never able to fix it.

https://elixirforum.com/t/elixirls-not-working-runtime-termi...

I just discovered elixir-tools which has an LSP and it works instantly, and seems to have a more robust architecture than ElixirLS. I used ElixirLS for a couple years now but it was the best option for a while, but elixir-tools (which I literally installed 30 mins ago) has been a delight to work with.

The instant feedback actually feels like how it always should've been.

i tried it out but wasn’t really happy with it in emacs. ended up going back to elixir-ls until elixir-tools is a little further along.

Next LS author here! I would love to hear about how your emacs experience went and what your setup/config is. My emacs skills are not the best (zero), so I've not been able to really work through any of the kinks.

I have recently gotten it working tho with vanilla emacs with eglot, as well as doom emacs (again, with eglot).

No pressure!

happy to provide feedback when i get a minute! i’m also utterly useless in elisp so not sure how useful it’ll be but ill try and make it helpful :)

I am a long time Elixir user and I also find the VSCode experience a bit frustrating ; sometimes the completion did not work, for hard-to-tell reasons (I'm switching computers often).

I have seen a couple of issues that have been worked-out.

In particular, it's nice to have completion in ".exs" files!

One thing that's been bugging me recently is completions in eex/heex template files. It seems like a tough problem to solve, since you need reasonable completion for html, heex syntax, and arguably also completion on things like schema fields for forms and then components that the project defines.

But man, if I could have a smart editor for those templates I would be like... three times as productive on my little side projects.

Getting set up a dev environment for me was painless, with ElixirLS being a big exception. For me the wrong version of erlang/OTP was causing crashes that weren't getting surfaced well. Fortunately I have enough Elixir experience that once I got the error message surfaced I was able to figure it out pretty readily, but yeah that experience can definitely be improved.

I had this problem too when I started, for me it was an incorrect Erlang version. Using asdf to install both of them fixed it for me completely, and was pretty much a good enough experience all around that I use it to manage Nodejs installs too now.

Yes, protip: Use asdf to manage erlang and elixir versions. The only exception might be if you're on Fedora the packages tend to stay pretty new, but anywhere else (and even there if you need a specific version) asdf is a godsend. I use it to manage go, node, python, and ruby as well.

> the last time I tried to just print “Hello, world!” in either Python or JavaScript, it took me down a hell dungeon of configuration and dependency nightmares.

On Mac:

    brew install node
    node -e "console.log('hello, world');"

Seems pretty easy?

You can also do "brew install elixir".

But for any serious work I will more likely use asdf or nvm/rbenv/rvm.

i basically never recommend the brew version. on the occasion it gets installed i’ve had no end of problems from mismatched erlang version requirements and path management. asdf is just easier.

You left out what I said next in the comment you pull-quoted. :)

The in-editor tooling definitely has rough edges. I've done a lot of Elixir and the ElixirLS stuff falls over a lot for me. There is a new effort with NextLS that seems very promising.

Thanks, I'll check that out.

Next LS author here! Feel free to reach out with any questions!

My contact links should be discoverable on the website: https://www.elixir-tools.dev

Elixirls is finnicky and works 50% of the time in my experience. Don't know why.

Try using Zed it comes with Elixir support out of the box, and uses nextls, which works always for me. Snappy as hell! Like Sublime Text 4.0 https://github.com/elixir-tools/next-ls

NextLS also has an extension for VSCode![1] I've found the ElixirLS works best with the projects I use at work, but there's a lot of exciting language server development in the space.

[1] https://marketplace.visualstudio.com/items?itemName=elixir-t...

This did not work for me. I installed the extension on Linux Mint, VS Code Insiders, and intellisense/autocomplete never worked no matter how many times I restarted the editor.

I never got it fully working either - it kept getting stuck on one of my dependencies, which happens to ElixirLS sometimes too.

Next LS author here, thanks for the kind words!

My experience, admittedly is not one where I am a full time developer just a hobbyist and sort of dev adjacent for work, is that getting _any_ tooling setup in VSCode is a path with no end of problems. So, at least when someone says that type of thing here I go "ah, so the norm." I think saying things like that is more an indictment against VSCode.

Don't get me wrong, I think getting vim/neovim setup to do the same types of things is possibly worse, but I haven't really had much of a better experience in VSCode either.

Had no problem with neovim but I think it uses the same LSP under the hood

It does. That's what I use. If you have trouble, check that the erlang version you're using is compatible.

That's a good point which also applies to tooling such as observer, etc.

That is why I don't use VSCode. While it's is a nice text editor, VSCode is lacking features that were standard in IDEs over twenty years ago.

What kind of features? With Elixir, I use the ‘click in stack trace to go to error’ feature, do some debugging, use code completion, test running - and not much else.

I’d be interested to know what other kinds of features there could or should be, as I’d like to be as productive as possible and I don’t feel like I really am.

I'm just starting to learn Elixir, but for other languages as you type in the code editor in an IDE it will auto-suggest method names and method arguments based on the libraries/modules you have included in your project.

Eclipse got this right, but the JetBrains IDEs (such as IntelliJ and CLion) are a bit heavy-handed.

The Elixir add-ons for VS Code provide this too.

That's an unrelated tangent better served by asking on elixirforum.com instead of polluting a thread with a drive by complaint.

This seems interesting.

I've been trying Goal[0] to validate API data at the controller edge, which uses changesets itself.

[0] https://hexdocs.pm/goal/Goal.html

That looks nice, but not a lot better than just embedded schemas. If it had more to offer for validating dates, I'd be interested.

Maybe suggest it in the Elixir forum thread https://elixirforum.com/t/goal-a-parameter-validation-librar...

Yeah, I could do that but I know how much of a tar-pit it really is so I'd be hesitant to inflict that on anyone else :)

While I'm no strong typing zealot and find Elixir is decent enough on that aspect, I feel NimbleOptions is just runtime type checking of keyword params, and we wouldn't need much of it with a better type system.

This is a nice article. I've used Ecto embedded schemas and nimble options separately but never thought to use them together.

Faced with this kind of issue, I'd use Ash with the `Simple` data layer. You can get all the power of an embedded Ecto schema and nimble options together in a single dsl.

If you need to validate with arbitrary logic you can do that with a 'validation'

If you need to transform the input data before you return the struct you can do it in a change set.

It's all there!

The only place I hear about Elixir is here on HN.

It's talked about a bit on Twitter with the hashtag #MyElixirStatus. Discord makes use of it as well. Same with Supabase.

The problem is the languages one would hear about, fall into the OOP category, functional languages tend to not be as fashionable when you do get into one, you become almost a zealot.

It's hard to fight the imprinting a language has with its first fans, Elixir because of its Erlang origins tends to have the identity of telephonic-based use cases. Similar to how Rust gets its reputation as a specialist language (cli,kernal, etc) even though you could use it for web apps as well.

The last reason is, it's really hard to fight the gravity pool of NextJS and React.

Personally my circle and company and country are solidly on the side of C#/Java/PHP/Python and React/Angular, the average developer who fits into that grouping might not have even heard of Svelt or SolidJs when they do hear about it, they immediately make some snide comment about JS fatigue, even though that hasn't been a thing for a while.

> it's really hard to fight the gravity pool of NextJS and React.

Indeed. I love Elixir and I even struggle with choice when I don't need much or any server-side functionality.

There was a Jekyll-style project for Elixir some years ago but it fell into bitrot and deprecation. I'm still hoping somebody resurrects it or builds an equivalent, because using eex to build a static site is a really great experience.

Have you seen Griffing, your post prompted me to scan for an active project (many graveyards)

https://github.com/elixir-griffin/griffin

Back in 2018 I took a graduate level distributed systems course from a professor who spent most of his lecture time rambling on about how we’d be better off reading HN than our textbook. There were no exams, only four projects which involved implementing various classic distributed algorithms (gossip, chord, etc.) in Elixir.

For what it’s worth, Elixir is now my preferred language for personal projects, although I still haven’t had a chance to use it professionally.

Interesting, are the syllabus + assignments published online somewhere?

Wish it was different but realistically I feel like you are right (outside of Elixir specific communities).

I love working with Elixir and have done so successfully for the past 4 years at various companies, however I have been trying to find a new client for a while now and I'm getting close to the point of looking for other types of work because things are looking bleak. From what I can tell there are also way more people looking for Elixir work than available jobs, unfortunately.

In my 15+ years of software development in various languages Elixir has been the most enjoyable developer experience for me. So I guess I'll just have to think of a side project where I can apply it and do less fun stuff to earn bread.

Yeah it's tough right now. One of the saddest things for me is the vicious cycle of "we can't use elixir because nobody uses elixir, so we can't use elixir because nobody uses elixir" ad infinitum. It's such a wonderful and productive stack, but people are just insanely comfortable with their typescript (or laravel, or rails, etc).

For what it's worth there are at any one time 50-75 people writing Elixir at my company, and we have projects ranging in age from 6 years to some things started this year.

The main reason is probably because Elixir USP is the stability of the apps written in it and how easy it is to build distributable software with it imo. You have to be an experienced engineer in order to appreciate those two things and have specific problems which most don't have.

However I think with Phoenix Liveview Elixir gives regular developers a productivity boost that could be it's own USP.

I have only written a small elixir app but I was mind blown over Liveview and still is convinced that most people would be better of using it over a SPA.

Sounds like to work in a boring monoculture, then. Thank pg you have Hacker News to keep you up to date with the rest of the programming world.

I built approximated.app on elixir, proxying 300k+ domains. It's a great ecosystem, I've never seen anything else handle really hard problems so well in my 15 years as a developer.

Concurrency? Trivial.

Reliability? Unbelievable. (YouTube Sasa Juric's "Soul of Elixir" talk)

Clustering/horizontal scaling? Built in, even across networks.

Web framework? Competitive with the best (I'd argue better, with liveview but YMMV).

AI/ML? Amazing support and tooling.

Embedded/IoT? Incredible support with the nerves project.

There's only 2 places where I feel it falls down a bit:

1) brittle dev tooling in ElixirLS that trips up new and experienced devs alike. Soon to get better though as competing LSPs are in the works.

2) no official release-as-binary tooling yet. Things are way better in the last few years for releases, and you're probably containerizing anyways, but I am jealous of e.g. Go's single file binaries.

It doesn't have a major corporate sponsor, which is probably why. It's a very active community, though.

It's a very good point. I consider it is a language of "connoisseurs" at the moment. I've been exchanging with computer science schools here in France to see how to increase adoption.

Students, presented with the capabilities of the language (scripting / machine learning / scalable web apps) are quite interested actually!

there’s a pretty well established community on mastodon as well as xitter. not sure where you hang out online obvs but it’s not exactly niche anymore, maybe branch out a bit?

Hm, I hadn't heard of NimbleOptions before and always used Ecto's embedded_schema for these types of things (as also mentioned in the article).

Looking at NimbleOptions based on this example I think I would prefer to stick to Ecto because I don't like the idea of putting the schema in a giant keyword list - but maybe soon I'll have a better play around with it myself, read the docs and see if there are better ways to use it.

Reply on news.ycombinator.com