How to implement a disk cache plugin for Elixir's Req HTTP client?
thibautbarrere.com
thibautbarrere.com
Req.get!("https://hex.pm/api/packages/req").body["meta"]["description"]
In Elixir, bang functions per convention will raise on error. `get/2` will return error tuples allowing you to handle errors. In fact, get!/2 just calls get/2 and raises for you[^1].
> no mention of JSON at all
Req is the most "batteries included" Elixir HTTP lib out there. I can't speak for Wojtek, but I believe the goal was to make Req extremely easy to use in scripting or things like LiveBook without having to do much work. That being said, the automatic decoding is mentioned in the readme[^2] and the docs[^3].
> if "body" is JSON, how do you even get the raw body, or can you?
Per the docs[^3], you can either skip with the `:decode_body` option, or just build your own request using only the steps you want.
> just seems over engineered/over fitted whatever you want to call it.
Fair, but again, this library is designed to be on that end of the spectrum. There are plenty of other libraries further down the stack that you can use. I am partial to Finch[^4], upon which Req is built.
To address the sibling comment about "Let it Crash", the language allows you to easily recover from crashes, but that is for resiliency, not error handling. In practice you would use the non-bang get/2, pattern match on the response, handle any errors, perhaps use Kernel.get_in/2 to safely traverse the map, etc. The example provided by the author is not "production ready".
[^1]: https://github.com/wojtekmach/req/blob/v0.3.11/lib/req.ex#L3...
[^2]: https://github.com/wojtekmach/req
Indeed, that + pattern matching does error checking just right.
I must admit that the automatic JSON decoding initially turned me off a bit!
case Req.get!("https://hex.pm/api/packages/req") do
%{body: %{"meta" => %{"description" => desc}}} ->
# use the value of desc here
%{body: body} ->
# report for unexpected body
end
On production code or scripts where I want to be sure (which is the case mostly), I would indeed use something like:
%{status: 200, body: json} = Req.get!(".....")
And then later restructure as needed.
Sometimes it would be:
meta = json |> Map.fetch!("meta") |> Map.fetch!("description")
Or pattern matching as you did!
E.g. Was it a network issue, content type issue, malformed JSON, unexpected JSON format, etc?
I mostly enjoy Go, Python, and Java (in this order), but I must say the self-contained, self-dependency-installing script thing in TFA seems pretty appealing.
The standard implementation is called supervisor, which you can read more about here, the Erlang documentation is very good: https://www.erlang.org/doc/design_principles/sup_princ
Fred Hebert's blog post called "it's about the guarantees" makes this more clear: https://ferd.ca/it-s-about-the-guarantees.html
No that's just wrong. If you want to let it crash, just pattern match on the :ok tuple.
You shouldn't have to write exception handler blocks to deal with functions that raise, instead of returning a tuple, that's bad Elixir style.
All functions that can fail should return a tuple. For convenience, one might also add a ! version that raises.
Pattern matching on ok tuple is awful because you will get a generic MatchError with an inscrutable message that requires reserialization of the exception struct and not a more useful Error where you can have an informative Exception.message protocol implementation.
> All functions that can fail should return a tuple. For convenience, one might also add a ! version that raises.
This is absolutely good advice for library writers. But if you're using a library, most of the time, use the raising version.
iex> {a, b, c} = {:hello, "world"}
* (MatchError) no match of right hand side value: {:hello, "world"}
I always pair my production Elixir apps with Sentry, which gives me a backtrace, so I have everything needed to track down an error.
> But if you're using a library, most of the time, use the raising version.
I use the raising version to explicitly "I want this to crash." So this would happen only at the edges of the system. All internal code should operate on tuples. You don't want a core, internal function to raise and have to catch it on higher-level modules, because that's terrible form. In other words, feel free to raise, as long as there is no one that has to catch the exception. If you raise, the goal is to crash, not to leverage exceptions as control flow. This isn't Java.
I've been using Elixir since 2016 and I don't think I have written more than a couple exception handlers since.
Anyways. The whole point of BEAM is that you code the happy path and if you can't actually do anything about the error, you're wasting effort and making your code harder to reason about.
Nobody is suggesting using errors like control flow. Writing a try/rescue is indeed an antipattern. It's like the Toyota factory line. If something irrecoverable happens, just pull the emergency stop and start over.
If you've been handling errors like that since 2016, well, I guess that's your choice, man. I wouldn't do it.
Once you run it in a task, then it makes less sense to do {:ok, something} on everything
What actually are you going to be doing with that error? Most of the time it gets raised at the end of the error chain resolution.
Yes, but do you want to kill your process and restart? I've been doing quite a bit of LiveView recently, and a thrown error will kill the process and reload the page. I... don't want to do that.
> What actually are you going to be doing with that error?
Depends on the error :) If it's "record not found", I redirect to a 404. If it's "can't resize image because file format is incorrect" I log it, and display an error message. And so on.
Ok. That's 100% real and I've run into it myself and been annoyed by it. Probably liveview should do something about that because it's breaking BEAM idioms.
> redirect to 404
Raise an error with plug_status set to 404. I believe there's a protocol way of doing it too if you dont own the exception and thus can't add plug_status as a field.
It really doesn't. It's the `one_to_one` restart strategy for supervision.
> Raise an error with plug_status set to 404.
Honestly, first time I've heard about it :) It's somewhat well hidden in the docs I think.
But 404 is just one of possible responses. I might want to redirect instead. Auth errors should be also handled by redirects. And so on.
Not everything is an exception for the framework supervisor to handle through custom fields.
https://hexdocs.pm/plug/Plug.Exception.html#actions/1
This is how phoenix does those fancy error reports in dev (but not prod)
> Probably liveview should do something about that because it's breaking BEAM idioms.
Actually now that I think about it, failable actions should probably be wrapped in a task. It's not hard for the liveview to subscribe to task failures and bubble it up to the user as a flash. Not doing so is breaking processes-as-failure-domains concept. The phoenix liveview documentation should probably hoist that sort of guidance to the top and put flashing lights around it.
But why would I do that for things that 100% predictable, can and will happen in specific flows?
> failable actions should probably be wrapped in a task. It's not hard for the liveview to subscribe to task failures and bubble it up to the user as a flash.
How does LiveView know an action is failable? What you propose is akin to trying to solving the halting problem :)
Also, not all failures need to be propagated to the user as a flash:
- an auth failure should result in a redirect, not a flash
- a record not found failure could result in an in-place 404, a redirect to a 404, a redirect to a search page, in a cache lookup etc.
- validation errors should be propagated to the form and displayed inline, not as a flash
and so on. There are many failure modes and there are just as many recovery modes, the vast majority of which entirely depend on the business logic of your app that LiveView has no idea about
Sorry I should have been more clear. You should wrap failable actions in a task if you're calling it from a liveview.
> an auth failure should result in a redirect, not a flash
That should happen before the socket plug and perform the redirect before even the websocket upgrade.
For complex, one-off and schduled tasks? Sure.
But when my stuff is literally "get one record from the db and display it" why should it be in the task? Just because you're a "let it crash" purist? I don't think even Erlang advocated this.
> That should happen before the socket plug and perform the redirect before even the websocket upgrade.
Again, that depends.
- An unauthed person accessing a link behind login? Yes. Redirect before anything happens
- An unauthed person accessing an expired link (e.g. invitatoon, or private preview)? Oh, I have to acces the DB to check that the link is valid/invalid, and show the page saying that the link has expired
- A person with some authorisation accesiing some resource that is only partially available to them? Oh, I still have to show some stuff, and hide some other stuff.
The world is significantly more complex than "just run it in a task, throw an error, and let Phoenix show a 404 page automatically"
The idea of hiding information from the user, just for me to get it from bug tracking and support tickets is not something I could jive with.
Most errors are not?
Suppose you get KeyError. Is that information useful to the user?
The specific error (going to say in response to API given the article) you gave would likely be returned through the system as API response error tuple.
The parts of call stack higher up could determine if they want to show the error or swallow it. If it's a generic KeyError or MatchError, there's a 0% chance anything outside of the immediate call site could handle the error.
That being said, you return errors in a similar way to other languages if they're expected during the system's operation rather than crashes.
Again it's been a while since I had to work with it.
First of all: this example is a one-off script, so error handling is not as much of a concern (you'll notice the first expression installs packages, this is not how production apps should handle dependencies).
Second, functions that raise always end in `!` in Elixir, or at least they are supposed to. Most have alternatives that return error tuples instead which you can pattern match on (this is what I recommend). You'll hear "let it crash" a lot as it applies to Erlang/OTP programming (and by extension Elixir), and this is a good high-level organizing principle. You shouldn't go out of your way to program against the unknown in Elixir. However, HTTP requests failing is common enough that in my opinion at least you really should just check the error. You can read the docs for `get/2` (as opposed to `get!/2` which raises) here: https://hexdocs.pm/req/Req.html#get/2.
A common pattern is for the `!` version to call the version that doesn't raise, check the result, and raise on error, which is the case here: https://github.com/wojtekmach/req/blob/9de30de0df481ee557ccc...
> and if "body" is JSON, how do you even get the raw body, or can you?
You would set `decode: false` when calling `get!/2: https://hexdocs.pm/req/Req.html#new/1. You can also set this as configuration with https://hexdocs.pm/req/Req.html#default_options/1.
As a closing note I'll mention that Req is intended to be a very high-level, scripting-friendly requests library, similar to Requests in Python (the author also mentions in the OP that his use case is scripting). If you don't want conveniences like Req provides, you can either turn them off or use something different, like Finch (which Req is based on, https://github.com/sneako/finch).
Other than Req and Finch I'm personally only familiar with HTTPoison, which is significantly older than all of the libraries derived from Mint (like Finch and Req, https://github.com/elixir-mint/mint) but is very popular and still works. There are many others though, like Gun and Tesla and such. I just think it's important for folks to know that this level of "magic" is far from the default in Elixir, but it's there if you want it.
Cool GitHub, btw, though I couldn't help but notice you have only Go repositories, which I found amusing (never seen a profile quite like yours before!). Cheers.