Fundamentals of Object Storage in Elixir | Hacker News Reader

Fundamentals of Object Storage in Elixir

underjord.io

122 pointslawik39 comments

Good write-up!

I will add:

- make sure to “stream” your transfers in general, instead of keeping whole file blobs in RAM. This to avoid refactoring at the least comfortable moment when a file gets too big in production

- it is quite easy to use Mox for isolated testing in general, and it works nicely with ExAws, so make sure to integrate those tests early on

Seconding the Mox recommendation, it's pretty good and enforces good separation of concerns in your application. It was created by José himself, so you know it's got a well-designed API. You just need to get used with using behaviours, and using the Application config to tell your code when to use the mock vs the real thing.

I use it to separate and test external API calls (i.e. to test the payment layer without making any request), to maintain separation between my code and an external library I might want to swap out at a later date, or even between various components of the application. Then writing unit tests becomes a breeze, and you don't need many integration tests.

https://hexdocs.pm/mox/Mox.html

I hate Mox with a passion. Because I hate separating out the interface and implementation of something that is only ever going to have one real implementation.

The extra layer of indirection just for testing bugs me so much. And I just can’t justify it with “one day I might want to swap out the implementation”. YAGNI and if you do, deal with it then.

So much legacy elixir code is an unreadable mess because of all the extra layers of interfaces that only get used for Mox. There’s nothing I dislike more than diving into an unfamiliar section of code and having to keep 15 files open to follow the logic for something that should have been in one module.

That just means you're putting a boundary when it's not necessary. Mox exists to make tests simpler to write and reason about, if you keep hitting extra layers of indirections then, "you're holding it wrong."

Most things are best used sparingly. That said, the vast majority of developers, even senior, have a very limited understanding of testing practices, and tend to create millions of separated, isolated modules, for no reason whatsoever. That's not Mox's fault.

A stupid example in practice: a naive blog app does not need Mox at all. If you add SSO sign up, you might want to put a boundary there to unit test without needing to set up a fake auth server, but 99% of your code won't ever touch it, just the authentication tests. Tomorrow you add payments, you might want to put a boundary just before your StripeRestAPI module to test the entire payment flow without contacting Stripe in your unit tests. Again, only the payment module tests ever need to touch Mox, the rest of your code doesn't. You place it before modules in the outer edge that have real-world side effects, not to isolate your functional and mostly pure core.

I recommend the DestroyAllSoftware talks, especially https://www.destroyallsoftware.com/talks/boundaries

> And I just can’t justify it with “one day I might want to swap out the implementation”. YAGNI and if you do, deal with it then.

I agree with that. Using Mox to swap out implementations is a possible use case, but in practice it's better to hardcode things rather than trying to make it generic and swappable the first time around. Junior devs have a tendency to over-engineer code "because we might need to change it later." My motto is "copy and paste it three times before adding a layer of abstraction"

I hope you see the irony in using “you’re holding it wrong” here?

If real world use of a library tends to push towards a certain outcome that I dislike, I’m going to tend to dislike that library.

You’re also never going to convince me that creating an interface for a single implementation is the best way to facilitate unit testing.

Mox is nothing more than a kludge to get around the limitations of the language and I think there are much less invasive kludges available. To be honest when I’m working with devs who are new to elixir I’m embarrassed when I have to explain how Mox works.

>but in practice it's better to hardcode things rather than trying to make it generic and swappable the first time around. Junior devs have a tendency to over-engineer code "because we might need to change it later." My motto is "copy and paste it three times before adding a layer of abstraction"

This I agree with 100%

> Mox is nothing more than a kludge to get around the limitations of the language and I think there are much less invasive kludges available. To be honest when I’m working with devs who are new to elixir I’m embarrassed when I have to explain how Mox works.

There are other ways, more similar to Ruby etc (e.g. https://github.com/edgurgel/mimic, or with_mock etc).

And also there is the risk to "over-mock", for sure.

But a good middle ground brings the right value here!

I’ve used mimic an I think it’s a much better option than Mox.

We spoke about this topic for far too long today, but looking at Mimic and its "nothing is mocked unless I tell you to" design, it works only iff you are using mocks the wrong way and at the wrong boundaries.

There’s no reason you can’t setup your tests so that mimic stubs out your module or a subset of functions in your module everywhere where you don’t explicitly override it.

The main difference is that the extra dispatch layer is only injected at test time, so there’s no extra indirection in the production code.

> I hope you see the irony in using “you’re holding it wrong” here?

Not really. If you use a hammer on a screw, is it the hammer's fault? Sounds like the code base you are working on is badly-organised, rather than Mox being the root of all evil. There are far too many stories of people picking up Elixir and coding in it as if it were Python or Ruby. You cannot base your experience with it on a codebase written by people new to functional languages.

And no one forces you to use Mox, anyway. It's far from being a core Elixir component. It is just one implementation of the mock object pattern that comes on a standalone, optional library.

> Mox is nothing more than a kludge to get around the limitations of the language

Mocks are not an Elixir invention, and exist in pretty much all languages.

https://en.wikipedia.org/wiki/Mock_object

“You’re holding it wrong” was a response from Steve Jobs blaming users for a design flaw.

>Mocks are not an Elixir invention

I have no problem with Mocks, I have a problem with Mox. Mox is the kludge, not Mocks in general.

Could you share the less invasive kludges you’ve found? I share your frustration with Mox, although maybe not so passionately.

I’ve had a good experience with Mimic for one.

> Mox is nothing more than a kludge to get around the limitations of the language and I think there are much less invasive kludges available.

You're definitely using or setting up mox wrong. Mox is incredibly good: it lets you do things in elixir that you definitely can't do in other languages: concurrently running tests with different mocks.

You can do that in every language I’ve worked with. You can even do it with Mimic in Elixir without the extra production indirection layer.

> You’re also never going to convince me

Such confidence, how do you approach testing your clients? Personally I have never used Mox, but I read the originating blog post when it came out and have followed it's principles ever since.

By either using Bypass, passing in fake adapters to Req, or injecting a dispatch layer at test time with Mimic.

I’ve been doing this for 20 years, you’re definitely never going to convince me that an interface for a single implementation is the best way to create the dynamic dispatch you need for testing.

Bypass is a nice alternative if the client is what's under test. Often it's not.

> passing in fake adapters to Req, or injecting a dispatch layer at test time with Mimic.

I see, so you buy into DI for the purposes of testing but you prefer a more terse approach than what behaviours offer. The extra module never really bothered me and the contents are mostly what would go into the implementation anyway. LSP provides documentation all the same.

> I’ve been doing this for 20 years, you’re definitely never going to convince me

I certainly appreciate your input, I like to learn new things, but have little interest in trying to convince anyone that makes such a dogmatic statement. An unfortunate thing to hear from a principle developer. I hope your coworkers agree with you.

I am Mox author and, honestly, I rarely use it. The most important part is to think strongly about what you are mocking and your interfaces (explicit or implicit). Once I do that, I do my best to use plain anonymous functions, modules, or protocols to test it. If you already have this discipline, then Mox only adds “paperwork”.

If you have layers of interfaces, it is worth asking why you need mocking across several layers.

YMMV, of course. As long as you are mocking with care, use whatever floats your boat. :)

You can write a using macro to automatically create a behaviour from specs in a module with https://hexdocs.pm/elixir/Module.html#spec_to_callback/2

It's been a minute but I've done it.

The behaviour/implementation setup reminds me of Java, especially when they're exactly the same.

I do recommend only using mox as far down as possible (only around libraries).

Hammox is Mox on steroids, check it out too.

I agree though I wouldn't say I hate Mox, rather I hate working with it (the things it does do it does well). As you say it forces you to do stupid stuff, like creating an interface for something with one implementation, but also the way you have to then add that to your configuration. I've also seen people only add "@callback" to modules, without any behaviour, because then it works with Mox.

> I hate Mox with a passion. Because I hate separating out the interface and implementation of something that is only ever going to have one real implementation

Are you injecting mox or substituting it as a dependency at compile time using an attribute?

I hear your pain: if boundaries are not correctly established, it can be problematic, as with any "boundary/interface" system.

Finding the right spot is indeed important here!

Kinda horribly half serious, but can you use Elixir's Metaprogramming features to demoxify mox?

https://www.youtube.com/watch?v=2Bjzml_Hpvk

Nice post. One part that raised a red flag for me was:

     defp bucket!, do: System.fetch_env!("BUCKET_NAME")

This will look at that environment variable at build time. For many workflows this doesn't work because you may build the app in CI and run it with different buckets in production and staging.

I always default to using Application.get_env/2 and setting the value in runtime.exs, now that that's a thing in recent Elixir.

For things where performance matters in a hot loop, it might be worth compiling in the value, but I wouldn't imagine that's the case here where it's a part of a network call to AWS.

As pointed out, it's not compile time but runtime here unless I'm missing anything. And it would be a good reason to build a Credo check exactly for this (I've done a bit of research about this).

Any calls to System in regular compile time config (e.g. config/prod/dev/test or included) should be forbidden, same for attribute-level / meta code.

I hope we will get this one day!

This will look at it at runtime not build time. `System.fetch_env!("BUCKET_NAME")` is still in the body of the function and won't get executed until it's called after application start.

It is still a good reason to use Application.get_env, and it should be preferred over peppering system environment variable fetches all over the application.

Oh, of course. Brain fart moment. I pattern matched (ha!) to code I've seen too much where that's at compile time.

Yeah, runtime here works but I agree is still not as good as Application.get_env. Aside from (IMO) code clarity reasons, last I checked it is (or can be) much quicker. Application is a quick ETS lookup, while System reaches out to the system and I forget the details but at one point that was much slower.

There are a bunch of things that are kind of quick and dirty about these snippets. I think I mentioned that in the post, that these are not best practices. But not quite so ill as compile-time in this case :)

what's up with the swedes and their insistence of using : in lieu of apostrophe?

It is often used in swedish technical writing to separate and foreign word or an acronym from the plural so "VPS:es" in english is VPS:erna in swedish. I do not know if this is a swedish tradition, it most probably exists in Germany or the US. I've seen this use in Sweden for several decades.

I've seen it used by Swedes and Swedish speaking Finns, at least.

Even in English, the use case you’re referring to would not have an apostrophe—apostrophes are used for contractions and possession, not plurality.

Meet our hero: https://www.bbc.co.uk/news/av/uk-39459831.

It's a fairly old tradition specific to Swedish and Finnish. Wikipedia has an explanation. [1]

[1] https://en.wikipedia.org/wiki/Colon_(punctuation)

If I knew what was up with us swedes I'd be consulting on that instead.

All Elixir projects I've seen, when you run "mix xref" you will see spaghetti.

You are not wrong but you may be hinting at the wrong cause. It is hard to generate a useful graph or summary that gives actionable feedback once you reach a certain amount of nodes. Even mix app.tree, which has no cycles, is barely useful after 30-50 deps because it becomes dense.

Using mix xref to evaluate your project will, at best, give you a very blurry picture. Think of it more of a database: it will give you good information when you need to answer certain queries.

Oh boy another underjord post. It's always such a joy getting to read his writing.

Reply on news.ycombinator.com