Easy Ergonomic Telemetry in Elixir with Sibyl
Feb 02, 2023
10 min. read
Elixir
Project
Refining Ecto Query Composition
Nov 08, 2022
13 min. read
Elixir
Absinthe
Ecto
Compositional Elixir—Ecto Query Patterns
Jun 10, 2021
18 min. read
Elixir
Ecto
Stuff I use
May 28, 2021
13 min. read
General
A minimal Nix development environment on WSL
Apr 04, 2021
13 min. read
WSL
Nix/NixOS
Elixir pet peeves—Ecto virtual fields
Dec 15, 2020
5 min. read
Elixir
Ecto
My usage of Nix, and Lorri + Direnv
Nov 26, 2020
5 min. read
Nix/NixOS
Build you a `:telemetry` for such learn!
Aug 26, 2020
19 min. read
Elixir
Erlang
Project
Drinking the NixOS kool aid
Jun 30, 2020
9 min. read
Nix/NixOS
Testing with tracing on the BEAM
May 20, 2020
8 min. read
Elixir
Compositional Elixir—Contexts, Error Handling, and, Middleware
Mar 06, 2020
13 min. read
Elixir
GraphQL
Phoenix
Absinthe
Taming my Vim configuration
Jan 14, 2020
7 min. read
(Neo)Vim
Integrating GitHub Issues as a Blogging Platform
Nov 06, 2019
6 min. read
Elixir
GraphQL
About Me
Nov 05, 2019
1 min. read
General

Elixir's `with` statement and Railway Oriented Programming

I discovered programming as an aspirational game developer at the age of nine, using Game Maker 5. Still, as I grew more experienced, I naturally tinkered around with the basics of Javascript, PHP, and more.

It really was during University where I had falling in love with Erlang. It was my first taste of functional programming, and despite some initial difficulties because of the syntax, I really came to appreciate the guarantees provided by the BEAM virtual machine.

My love for Elixir came about differently: I don't believe syntax (so long as its reasonable) matters when making decisions about whether or not to use a language—I actually personally dislike Elixir's Ruby-inspired syntax compared to the minimalism of Erlang's—but developer experience is everthing. For Elixir, syntactic sugar is a big piece of the pie.

Almost everything in Elixir can be said to be a macro, and in my mind, one of the most elegant macros in Elixir is the with statement.

The `with` statement

The with statement is a great example of how syntactic sugar can make code both more concise and maintainable; it was introduced Elixir v1.2 and was the last of the control flow statements (think case, if, cond) added.

One common problem in Erlang code (and thus Elixir also) is code which reads like this:

my_function(Email, Password) ->
  case get_user(Email) of
    {user, _UserFields} = User ->
      case validate_password(User, Password) of
        true ->
          case generate_access_token(User) of
            {token, _data} = Token ->
              {ok, Token};

            AccessTokenError ->
              AccessTokenError
          end;

        false ->
          {error, <<"Incorrect password">>};

        Error ->
          Error
      end;

    nil ->
      {error, <<"User does not exist">>}

    Error ->
      Error
  end.

It's very nested and very easy to accidentally miss a clause when reading this, and honestly, it's just a bit jarring to work with. Elixir's with statement cleans this up nicely, being a macro which is designed to compile down to nested case statements such as the above one, maximising readability and linearity:

You can see a simple example of the with statement below:

# The same control flow as the Erlang example, written with a `with` statement
def my_function(email, password) do
  with %User{} = user <- get_user(email),
       true <- validate_password(user, password),
       %Token{} = token <- generate_access_token(user_id) do
    {:ok, token}
  else
    nil ->
      {:error, :user_does_not_exist}

    false ->
      {:error, :invalid_password}

    error ->
      error
  end
end

If the |> operator is the honeymoon phase of becoming an Elixir developer, the with statement is surely twenty years happily married 😝

As you can see, at a super high level, you get two main benefits of using with over case:

  1. You primarily focus on your happy path. You describe a pipeline of steps that you want to happen, and in the case that they do occur, your pipeline succeeds, and you get put into the do ... end part of the statement.

  2. You flatten out your potentially deeply nested unhappy paths into one pattern matching block. We no longer have to add repetitive "bubbling up" matches to expose errors deeply nested in our control flow, and no longer do we have to spend much time parsing N-level deep nesting.

The caveats of the `with` clause

There are arguably two major (related) issues with the with clause presented above:

  1. Because pattern matching against errors is decoupled from the execution of any single given function, it might be confusing to know which error patterns in the else clause correspond to which function in the case of failures.

  2. Any unmatched pattern in the else clause will cause an exception. You must take care to match any patterns you expect will be thrown (or add a catch-all clause), but this leads to another issue: it can be easy to write overly permissive error patterns which shadow other, potentially important, error cases.

Basically, in short—with clauses have the potential for making your code terse but obscure, which is never a good thing. One should never optimise purely for length, but instead for related factors such as readability, maintainability, clarity, etc.

For something the size of the example above, perhaps the issues are small enough that we feel it's ok to ignore them, but this issue is only exacerbated by the growing size of any given with clause.

However, there are ways we can work around these issues and perhaps abuse them for the sake of simplicity itself! 🦉

An ok pattern

There is a simple idiom in Erlang/Elixir (though a pet peeve of mine: it's not enforced nearly enough), which we can use to start refactoring our with example above make it a lot cleaner.

Whenever I'm writing Elixir code that has a happy path and an unhappy path, I tend to implement the function with the following spec:

@spec my_function(some_argument :: any()) :: {:ok, success_state :: any()}
                                           | {:error, error_state :: any()}

What this means is, for any successful invocation of my_function/1, the result returned is expected to be wrapped inside a tuple: {:ok, _result}, otherwise any error will be wrapped inside a tuple: {:error, _result}. This is similar to the concept of an Option Type, though not one strictly enforced.

Using this pattern, our with clause example can be updated as follows:

def my_function(email, password) do
  with {:ok, user} <- get_user(email),
       {:ok, _response} <- validate_password(user, password),
       {:ok, token} <- generate_access_token(user_id) do
    {:ok, token}
  else
    {:error, nil} ->
      {:error, :user_does_not_exist}

    {:error, false} ->
      {:error, :invalid_password}

    error ->
      error
  end
end

This helps us remove a bit of the verbosity from our pattern matches: we don't bother pattern matching the response of validate_password/1 in our example because so long as a function returns {:ok, _response}, we know it succeeded and that we can proceed. Likewise, we don't bother pattern matching against any of the struct definitions.

An oft-missed quirk of the with statement is also the fact that error handling clauses are, in fact, optional!

Take a look at the else clause of our most recent example: we're mapping very concrete errors into domain-specific error atoms such that we can tell what happened. I would argue this is very wrong: the error {:error, :user_does_not_exist} is indeed a domain-specific error; just not of the function we're defining; instead, this error is specific to the domain of the get_user/1 function!

Taking that aboard, we simplify our example further:

def my_function(email, password) do
  with {:ok, user} <- get_user(email),
       {:ok, true} <- validate_password(user, password),
       {:ok, token} <- generate_access_token(user_id) do
    {:ok, token}
  end
end

This, in my mind, is pretty clean 💎

A Railway Oriented Architecture

An introduction to railway oriented programming is probably out of the scope of this particular blog post, but for lack of a better word, its a design pattern for describing your business logic as a series of composable steps—pipelines— focusing primarily on the happy path, coupled with the abstraction of error handling away from the core business logic.

It's a pretty common pattern in the functional programming world, and there are nice libraries in languages such as F#, but even C++, Java, etc., which facilitate this approach to programming.

The cool omission in the list above is that Elixir doesn't need a library to facilitate railway oriented programming, Elixir is batteries included. The following code is, in my mind, pretty idiomatic Elixir code, for example:

@spec login(String.t(), String.t()) :: {:ok, Token.t()} | {:error, term()}
def login(email, password) do
  email
  |> get_user()
  |> validate_password(password)
  |> generate_access_token()
end

@spec get_user(String.t()) :: {:ok, User.t()} | {:error, term()}
def get_user(email), do: ...

@spec validate_password(User.t(), String.t()) :: {:ok, User.t()} | {:error, term()}
def validate_password(%User{} = user, password), do: ...

def validate_password(_error, password), do: ...

@spec generate_access_token(User.t()) :: {:ok, Token.t()} | {:error, term()}
def generate_access_token(%User{} = user), do: ...

def generate_access_token(_), do: ...

This approach is pretty nice since your top-level API is literally a pipeline of simple functions. It becomes very easy to think about each step as a transformation being applied to a given input.

I've seen this pattern done again and again on several projects I've consulted on. However, I wasn't entirely speaking in jest when I described the |> operator as the honeymoon phase for someone learning Elixir...

The main issue with this way of doing things is that you have to pollute your function heads to pattern match errors which are returned by other, completely unrelated functions; in turn, breaking the semantics of your function as defined by its function name: yes, naming is hard, but why must validate_password/2 handle errors returned by get_user/1?

I'd make the very strong argument that the |> operator should only be used for genuine pipelines: data transformations that cannot go wrong. For anything else, we should be using with statements instead:

@spec login(String.t(), String.t()) :: {:ok, Token.t()} | {:error, term()}
def login(email, password) do
  with {:ok, user} <- get_user(email),
       {:ok, true} <- validate_password(password),
       {:ok, token} <- generate_access_token(user_id) do
    {:ok, token}
end

@spec get_user(String.t()) :: {:ok, User.t()} | {:error, term()}
def get_user(email), do: ...

@spec validate_password(User.t(), String.t()) :: {:ok, User.t()} | {:error, term()}
def validate_password(%User{} = user, password), do: ...

@spec generate_access_token(User.t()) :: {:ok, Token.t()} | {:error, term()}
def generate_access_token(%User{} = user), do: ...

Each function is entirely responsible for handling its own errors, and your code is shorter and clearer to boot. Nice.

Of course, this pattern is not dogma. I try to follow this approach whenever I can, but when writing Phoenix controllers or Absinthe resolvers, it's entirely understandable to want to handle errors coming from functions you call, transforming them, perhaps, into appropriate domain-specific GraphQL errors or JSON error objects.

The client I'm currently working consulting with, for example, has a project with several layers of nested with statements:

  • An outer with statement implemented as an Absinthe middleware to catch errors that bubble up from various places in our execution pipeline to transform them into error tuples that Absinthe can understand.

  • An inner with statement which implements authentication error handling on the level of specific GraphQL resolvers.

  • An internal with statement which encapsulates any non-trivial business logic in our actual contexts.

This is by no means a complete picture, but even so, it is by no means an extreme misuse of the with statement. We should isolate our functions into appropriate domains, and compose them. That is the zen of this pattern.

One minor shortcoming

There is precisely one case where this pattern makes life a little less convenient: string interpolation. This pattern forces you to choose one of the following approaches:

"Hello, #{get_name(user_id) |> elem(1)}"

"Hello, #{{:ok, name} = get_name(user_id); name || "default"}"

"Hello, #{case get_name(user_id), do: ({:ok, name} -> name; _ -> "default")}"

None of which are, in my opinion, particularly nice ways of solving the problem.

  1. The first approach is problematic if the result of get_name/1 is {:error, error} as we'd be displaying error to the viewer of the string.

  2. The second approach is fine but will crash if get_name/1 doesn't return {:ok,name}.

  3. The final approach is probably the best, but it is pretty messy to read.

Elixir, thankfully, has an idiom that can rescue us once again! In Elixir, typically, you can expect functions to be in one of two flavours (though again, this isn't always true and not as consistent as I'd wish for it to be):

  1. Functions that have the suffix !

  2. Functions that don't have the suffix !

Functions that belong to the latter class are expected to return tagged values as we described previously. Functions belonging to the former class are expected to return untagged values but may choose to throw an exception if they fail.

For any functions which I want to be able to interpolate cleanly, I can implement a !-variant of it as follows:

def login!(email, password) do
  case login(email, password) do
    {:ok, user} ->
      user

    {:error, error} ->
      raise error # This can always be made into a real exception and raised too
    end
end

def login(email, password) do
  with {:ok, user} <- get_user(email),
       {:ok, true} <- validate_password(user, password),
       {:ok, token} <- generate_access_token(user_id) do
    {:ok, token}
end

But honestly, this case doesn't happen very much. If it does, it means I modelled something that can never fail using this pattern, which is fine, but unnecessary, or I need to wrap my string interpolation inside a with clause and handle failures properly.

It might be nice to write a library that can automatically generate !-variants of functions for you; perhaps that'll be a project I start one day 🤔

I hope this helped outline a pattern I find myself reaching for literally whenever I have to write new code or refactor existing code. Feel free to let me know what you think!

Thanks for reading 💪