Elixir pet peeves—Ecto virtual fields

Ecto is an amazing ORM that is super easy to use, and with good practices and conventions, it becomes extremely composable and flexible while at the same time being super light weight with minimal magic.

At times though, the lack of "magic" when working with Ecto can prove to be a little limiting: one such time this is an issue in my opinion is when using virtual fields.

What is a virtual field?

Virtual fields are just schema fields that aren't persisted/fetched from the database. For example, if we define the following Ecto.Schema:

def MyApp.User do
  use Ecto.Changeset

  schema "users" do
    field :first_name, :string
    field :last_name, :string
  end
end

We could define a virtual field full_name which is defined as: field :full_name, :string, virtual: true. This effectively tells Elixir that there can exist a field called full_name but won't do anything beyond that.

This is useful because even though we're forced to write a function for resolving this virtual field (with or without the Ecto schema field definition added), it makes working with the struct itself in Elixir a little more annoying without defining it:

def MyApp.User do
  use Ecto.Changeset

  schema "users" do
    field :first_name, :string
    field :last_name, :string
  end

  def resolve_full_name(%User{first_name: first_name, last_name: last_name} = user) do
    Map.put(user, :full_name, first_name <> " " <> last_name)
  end
end

iex(1)> MyApp.resolve_full_name(%MyApp.User{first_name: "Chris", last_name: "Bailey"})
%{__struct__: MyApp.User, first_name: "Chris", last_name: "Bailey", full_name: "Chris Bailey"}

Notice how that breaks the rendering of the MyApp.User struct? It also messes with autocomplete/dialyzer at times as well as locks you out from being able to reason about what fields a struct might have after one resolves all the custom fields one might want to attach to said struct.

If we define the schema with the virtual field, we can do the following instead:

def MyApp.User do
  use Ecto.Changeset

  schema "users" do
    field :first_name, :string
    field :last_name, :string

    field :full_name, :string, virtual: true
  end

  def resolve_full_name(%User{first_name: first_name, last_name: last_name} = user) do
    %User{full_name: first_name <> " " <> last_name}
  end
end

iex(1)> MyApp.resolve_full_name(%MyApp.User{first_name: "Chris", last_name: "Bailey"})
%MyApp.User{first_name: "Chris", last_name: "Bailey", full_name: "Chris Bailey"}

Note that the inspection of the struct appears unbroken. Doing things this way also allows us to have default field values set up as well as peace of mind when we consider that we essentially enumerate every field we will want to ever work with (plus type definitions) on the schema itself.

Using virtual fields

Certain virtual fields can be resolved during query time, but more often than not (at least on projects I've seen or worked on) virtual fields are resolved after query time (i.e. in your contexts).

Virtual fields that are resolved at query time can be done via using Ecto.Query.select_merge/4: I could probably resolve MyApp.User.full_name at query time via the following Ecto query:

# This is certainly ugly and unneccessary to do like this, but more realistically if your
# virtual fields are just an average over some aggregate or join, this makes more sense to do.
# This is just to follow through with the example we've used thus far
from u in MyApp.User,
  where: u.id = 1,
  select_merge: %{full_name: fragment("? || ' ' || ?", u.first_name, u.last_name)}
  # better example would be:
  preload: [:friends],
  select_merge: %{friend_count: count(u.friends)}

But for simplicity and readability sake having it in your contexts is much nicer for simple virtual fields:

defmodule MyApp.Users do
  def get_user_by_id(id) do
    User
    |> User.where_id(id)
    |> Repo.one()
    |> case do
      %User{} = user ->
        {:ok, User.resolve_full_name(user)}
      nil ->
        {:error, :not_found}
    end
  end
end

The only annoyance with the latter approach is composability and repetition: wherever we want to resolve this virtual field (likely everywhere), we need to make sure to call User.resolve_full_name/1, and if we have multiple virtual fields to resolve this can become unwieldy (though one can have a User.resolve_virtual_fields/1 function instead, but the former point still stands. This is a longstanding and common pain point to using virtual fields in my opinion.

The pre-query approach to resolving virtual fields thus has the advantage that you don't necessarily need to worry about it, since select_merge calls can be composed and you build your query multiple times for your context functions in any case. Whether or not all virtual fields can be resolved this way, or whether or not it's worth the reduction of readability should be done on a case by case basis however.

Introducing Ecto.Hooks

In the past, Ecto actually provided functionality to execute callbacks whenever Ecto.Models (the old way of defining schemas) were created/read/updated/deleted from the database. This is exactly the kind of functionality one would like when trying to reduce duplication and points of modification when it comes to resolving virtual fields.

I wrote a pretty small library called EctoHooks which re-implements this callback functionality ontop of modern Ecto.

You can simply add use EctoHooks.Repo instead of use Ecto.Repo and callbacks will be automatically executed in your schema modules if they're defined.

Going back to the example given above, we can centralise virtual field handling as follows, using EctoHooks:

def MyApp.User do
  use Ecto.Changeset

  schema "users" do
    field :first_name, :string
    field :last_name, :string

    field :full_name, :string, virtual: true
  end

  def after_get(%__MODULE__{first_name: first_name, last_name: last_name} = user) do
    %__MODULE__{user | full_name: first_name <> " " <> last_name}
  end
end

Simply add the following line to your application's corresponding MyApp.Repo
module:

use Ecto.Repo.Hooks

Any time an Ecto.Repo callback successfully returns a struct defined in a module that use-es Ecto.Model, any corresponding defined hooks are executed.

All hooks are of arity one, and take only the struct defined in the module as an argument. Hooks are expected to return an updated struct on success, any other value is treated as an error.

A list of valid hooks is listed below:

• after_get/1 which is executed following Ecto.Repo.all/2, Ecto.Repo.get/3, Ecto.Repo.get!/3, Ecto.Repo.get_by/3, Ecto.Repo.get_by!/3, Ecto.Repo.one/2, Ecto.Repo.one!/2.
• after_insert/1 which is executed following Ecto.Repo.insert/2, Ecto.Repo.insert!/2, Ecto.Repo.insert_or_update/2, Ecto.Repo.insert_or_update!/2
• after_update/1 which is executed following Ecto.Repo.update/2, Ecto.Repo.update!/2, Ecto.Repo.insert_or_update/2, Ecto.Repo.insert_or_update!/2
• after_delete/1 which is executed following Ecto.Repo.delete/2, Ecto.Repo.delete!/2

The EctoHooks repo repo has links to useful places like the Hex.pm entry for the library as well as documentation!

I hope this proves useful!