Chris Bailey

Elixir's `with` statement and Railway Oriented Programming

Syntactic sugar is awesome. Honestly, there isn't really much different between using Erlang versus
using Elixir. Elixir gives us macros, but other than that its pretty much just syntactic sugar upon syntactic sugar!

The with statement is a great example of how syntactic sugar can make code both more concise and maintainable. The with statement was introduced Elixir v1.2 and was the last of the control flow statements (think case, if, cond) introduced to the language.

If the |> operator is the the honeymoon phase of falling in love with Elixir, the with operator surely is the twenty years of fulfilment stage 🤣

You can see a simple example of the with statement below:

# Some control flow with nested `case` statements
case get_user(user_id) do
  %User{} = user ->
    case validate_password(password) do
      true ->
        case generate_access_token(user_id) do
          %Token{} = token ->
            {:ok, token}
          access_token_error ->
            access_token_error
        end
      false ->
        {:error, :invalid_password}
    end
  nil ->
    {:error, :user_does_not_exist}
end

# The same control flow with a `with` statement
with %User{} = user <- get_user(user_id),
     true <- validate_password(password),
     %Token{} = token <- generate_access_token(user_id) do
  {:ok, token}
else
  nil ->
    {:error, :user_does_not_exist}
  false ->
    {:error, :invalid_password}
  error ->
    error
end

As you can see, at a super high level, you get two main benefits of using with over case:

  1. You primarily focus on your happy path. You describe a series of steps that you assume will occur and will pass. This is pretty similar to a normal Elixir pipeline.

  2. You flatten out your potentially deeply nested sad paths into one pattern matching block which means removing duplicate 'bubbling up' logic you'd have to implement with case statements if you don't want to just crash.

You can probably tell that there are a few caveats with the snippet above though. It is important to note that with is not some perfect panacea to solve all the world's problems. It lets us simplify our code while being arguably more expressive with it but it requires a bit of planning before using. The simplification above is harder to reason about than needs be and we'll explore further refactoring work to make it a lot nicer.

Exploring some caveats

There are arguably two major issues with the with clase presented above:

  1. The person writing the with statement needs to know exactly what the function they're calling returns to pattern match against it if they don't want to match everything.

  2. When we look at the else clause, we can't tell at a glance where the patterns matching nil or false come from. Are they returned from generate_access_token/1? Or get_user/1? Maybe they're being returned in multiple places? How do I handle a nil being returned from two different functions in different ways?

Basically, in short—incorrect or hasty uses of with can reduce readability and maintainability quite a lot...

For something the size of the example above, we might be tempted to move on, but for particularly complex with statements the problem only gets more pronounced.

Thankfully, there exists a simple idiom in Erlang/Elixir (which whilst not dogmatically enforced and present everywhere), is common enough to be of use. With some reasoning we can use it to work around both issues described above.

Typically whenever I'm writing Elixir code which has a happy result or a bad result, I'll implement that function such that instead of the function signature looking like get_user(id :: String.t()) :: User.t() | nil, I try to make it look like get_user(id :: String.t()) :: {:ok, User.t()} | {:error, term()}.

This might seem like an arbitrary and simple change, but it allows us to refactor the with clause presented above to be as follows:

with {:ok, user} <- get_user(user_id),
     {:ok, _response} <- validate_password(password),
     {:ok, token} <- generate_access_token(user_id) do
  {:ok, token}
else
  {:error, nil} ->
    {:error, :user_does_not_exist}
  {:error, false} ->
    {:error, :invalid_password}
  error ->
    error
end

As we can see, on the second step of the with we don't even bother pattern matching the response of validate_password/1 because we make the assumption that if the function returns a tuple tagged with the atom :ok, its deemed a successful invocation of that function. We don't need to bother pattern matching against the struct types because again, they're tagged with :ok and thus the result is what we assume it to be.

This is a little better already, but theres still more we can do! A cool (and lesser known) quirk of the with statement is that the else clause is optional. If the else clause is ommited, then any non-matched responses to the pattern matches in the statement will simply be returned.

We can abuse this quirk (though this is really just utilising it the way it was supposed to be utilised) by rethinking why we're handling errors at all in our large, originally deeply nested conditional flow. Why should we have to handle errors ourselves? If get_user/1 fails to find a user, we shouldn't have to handle that to be able to return a suitable error to the caller of the function... get_user/1 should return a meaningful error which bubbles up to the caller of the function instead because given the fact an error is thrown by a step in pipeline of operations, the error arguably is the responsibility and within the domain of that function rather than the site of invokation.

Taking that onboard, we can further simplify our with to the following:

> user_id = "1234567890"
> password = "incorrect_password"
> result =
.   with {:ok, user} <- get_user(user_id),
.        {:ok, true} <- validate_password(password),
.        {:ok, token} <- generate_access_token(user_id) do
.     {:ok, token}
.   end
{:error, :invalid_password}

This simplification basically decouples error handling from the invoker's context to the failing functions context, which in my mind is how it should be. Nice!

A Railway Oriented Architecture

An introduction to railway oriented programming is probably out of the scope of this particular blog post, but for lack of a better word, its a design pattern for describing your business logic as a series of composable steps—pipelines—, focusing primarily on the happy path, alongside the abstraction of error handling to a lower level context than the railway's own context.

Its a pretty common pattern in the functional programming world, and there are nice libraries in languages such as F#, Erlang, Java (and more) to facilitate this approach.

When I described Elixir's pipeline operator (|>) as the honeymoon phase of an Elixir lover's adventure down this highly-concurrent scalable rabbit hole of a language, I didn't mean that entirely in jest. The following code could be described as a pretty idiomatic way of implementing railway oriented programming in Elixir:

@spec login(String.t(), String.t()) :: {:ok, Token.t()} | {:error, term()}
def login(email, password) do
  email
  |> get_user()
  |> validate_password(password)
  |> generate_access_token()
end

@spec get_user(String.t()) :: {:ok, User.t()} | {:error, term()}
def get_user(email), do: ...

@spec validate_password(User.t(), String.t()) :: {:ok, User.t()} | {:error, term()}
def validate_password(%User{} = user, password), do: ...
def validate_password(_error, password), do: ...

@spec generate_access_token(User.t()) :: {:ok, Token.t()} | {:error, term()}
def generate_access_token(%User{} = user), do: ...
def generate_access_token(_), do: ...

This approach is pretty nice, since your top level API is literally a pipeline of actions. It becomes very easy to think about each step as a transformation being applied to a given input.

The main issue with this is that you're having to add function heads to your functions, to handle errors which are thrown in other functions; arguably breaking the semantics defined by your function name: why should validate_password/2 need to handle the case where the first parameter is an error?

I'd argue that for any cases where we expect errors to be thrown, the with statement as described in the sections above is the perfect piece of syntactic sugar for implementing raleways:

@spec login(String.t(), String.t()) :: {:ok, Token.t()} | {:error, term()}
def login(email, password) do
  with {:ok, user} <- get_user(user_id),
       {:ok, true} <- validate_password(password), # returns {:error, :invalid_password}
       {:ok, token} <- generate_access_token(user_id) do
    {:ok, token}
end

@spec get_user(String.t()) :: {:ok, User.t()} | {:error, term()}
def get_user(email), do: ...

@spec validate_password(User.t(), String.t()) :: {:ok, User.t()} | {:error, term()}
def validate_password(%User{} = user, password), do: ...

@spec generate_access_token(User.t()) :: {:ok, Token.t()} | {:error, term()}
def generate_access_token(%User{} = user), do: ...

I try to follow this approach whenever I end up having two nexted conditional operators; its nice to be able to not worry about error handling outside of the site of the error. You might want to handle/translate internal errors into external errors at the API call site level (i.e. an Absinthe middleware as we're doing on a GraphQL heavy project at the moment) or something, but otherwise you don't really have to worry about it.

The project I'm currently working on now has several layers of nested with statements:

This isn't even that much of an extreme case, but the nice thing is we can selectively handle failures if we want/need to and otherwise we just let errors bubble up to our top level. It's an extremely composible pattern 😄

Drawbacks

There are times, however, where this kind of pattern makes life quite a bit more difficult. Using functions which are implemented this way means you lose the ability to cleanly interpolate them into binaries, for instance.

# You are now forced to do one of the following in your string interpolations:
> """
. Hello, #{get_name(user_id) |> elem(1)}
. """
"Hello, 1"

> """
. Hello, #{{:ok, name} = get_name(user_id); name || "default"}
. """
"Hello, 1"

> """
. Hello, #{case get_name(user_id), do: ({:ok, name} -> name; _ -> "default")}
. """
"Hello, 1"

None of these are particularly nice ways of solving the problem:

  1. The first approach is problematic if the result of get_name/1 is {:error, error} as we'd be displaying error to the viewer of the string.

  2. The second approach is fine but will crash if get_name/1 doesn't return {:ok,name}.

  3. The final approach is probably the best but it is pretty messy to read.

Thankfully Elixir also already has an idiom for this (or at least, with similar semantics to this). In Elixir, typically you can expect functions to be in one of two classes (very broadly speaking):

  1. Functions which have the suffix !

  2. Functions which don't have the suffix !

Functions which belong to the latter class are expected to usually return tagged values, like we discussed above: get_name() :: {:ok, name}

Functions which belong to the former then, as per Elixir convention, is expected to directly return the value. The actual meaning of the ! suffix is to denote that that particular variation of the function call uses exceptions as failure states as opposed to the form {:error, error}, which for our purposes is fine. You can pretty easily create an idiomatic ! variant of a function as follows:

def login!(email, password) do
  case login(email, password) do
    {:ok, user} ->
      user
    {:error, error} ->
      throw error
    end
end

def login(email, password) do
  with {:ok, user} <- get_user(user_id),
       {:ok, true} <- validate_password(password), # returns {:error, :invalid_password}
       {:ok, token} <- generate_access_token(user_id) do
    {:ok, token}
end

Which we can then use for interpolation in strings and other cases where the fact that return values are tagged prove problematic.

I'm not actually sure off the top of my head whether or not the idiomatic way of implement this convention is to wrap the non-! variant of the function with a case, or wrapping the ! variant of a function call with a try, but in practice the way demonstrated is how we implement this most often.

Something that might be nice to play with in the future as well is a macro which automatically creates ! variations of functions with typespecs which are known to contain tagged :ok tuples... I'll get to it one day 😉


Return to Posts →