Configuration is one of the rougher edges of the Elixir ecosystem even after the many improvements over the years. Because it's always been rough, you may not expect, and therefore miss, the hidden improvements nested in recent Elixir releases. This post will shine a light on ways to leverage these improvements for better configuration.

Note on terminology:

• “application environment” refers to application variables that are accessible with Application.get_env and are generally set with Config.config by elixir scripts inside the config directory.
• “environment variables” are part of the environment in which an Operating System process runs (see environment variables on Wikipedia). In Elixir environment variables can be accessed via System.get_env and System.put_env (plus a couple other functions).

A brief history of configuration in Elixir

At one point it was a recommended practice to read application environment variables at compile-time (technically build-time) with module attributes. For example:

This was recommended for the following reasons:

• This is similar to a common pattern in Ruby/Rails with instance variables.
• There’s a (very small!) performance benefit in not reading configuration on every function call.
• Note: Application env is backed by :ets which is stored in-memory and is very fast, but if you have a very tight loop or performance critical code you may want to either cache the configuration value in process state or look into :persistent_term.

Despite this being recommended, I got the sense that relying on environment variables is generally frowned upon in the wider Elixir community. The community preference was to write application configuration scripts directly. This is why, at the time, Phoenix would generate a <p-inline>config/prod.secret.exs<p-inline> in fresh Phoenix projects. This file isn’t generated in newer versions of the Phoenix generators since the configuration style has been refined. In my experience, using configuration scripts to deploy secrets for Elixir/Phoenix applications was difficult because most deployment pipelines aren’t setup to deploy a special secrets file to each environment and this approach doesn’t work with shared secret stores like Hashicorp’s Consul.

Over time, some issues with this approach began to crop up. Configuration that is read at compile-time means that the module reading the configuration needs to be recompiled if the configuration changes, which is not automatic (and even if it was, it would incur a time cost). Eventually this leads to a general understanding that there are multiple downsides to reading configuration at compile-time. So, successive Elixir versions have been improving the ability to not rely on compile-time configuration.

Elixir 1.9:

• Adds <p-inline>config/releases.exs<p-inline>.
• Provides a place to put configuration that is read at runtime, but only for releases.
• mix new no longer generates <p-inline>config/config.exs<p-inline> by default.
• Release notes include “Relying on configuration is undesired for most libraries.”

Elixir 1.10:

• Adds <p-inline>Application.compile_env<p-inline>.
• This makes it explicit when the developer is expecting a to rely on a specific configuration at compile-time (and allows the Elixir compiler to warn when that configuration has changed).

Elixir 1.11:

• Adds <p-inline>config/runtime.exs<p-inline> which allows for use of runtime configuration in non-release environments (such as development) instead of only in release-based environment.
• <p-inline>config/releases.exs<p-inline> is deprecated.
• "Developers not using releases must use the <p-inline>config/config.exs<p-inline> file, which often loaded too early at compilation time. For any dynamic configuration, developers had to resort to third-party tools or workarounds to achieve the desired results."

Configuration and libraries

Elixir’s official Library Guidelines have a couple sections on avoiding application configuration in general, and also compile-time application configuration. Beginners should note that the configuration that is defined in a library is not used when you depend on that library. So if the library has a <p-inline>config/config.exs<p-inline> file, that config file has zero effect on an application that is using the library.

Other configuration approaches

Not everyone makes very heavy use of the Application Environment. There are a couple alternatives that are worth mentioning. One of the main alternatives I see is setting configuration at application startup within your supervision tree by passing the configuration directly to processes via their <p-inline>start_link/2<p-inline>.

In code, this might look like:

(In practice you’d probably generate the configuration in a small helper function rather than fetching all the configuration inline)

Then you’d handle the configuration options in your <p-inline>start_link/2<p-inline> and <p-inline>init/1<p-inline> functions (assuming MyApp.Transmogrifier follows the common GenServer patterns).

The downside to this approach is that you’ve now caused problems for the system operator. It becomes effectively impossible to determine what num_instances and api_key were passed to the process at runtime. You can use :<p-inline>sys.get_state/1<p-inline> but the process may have done additional calculations on the options or may not have needed to store the options in its state at all. Fundamentally, the issue with this approach is that it makes it difficult to introspect the state of the running system.

Another recommended approach is to localize the configuration in the places where that configuration is used.

Here’s one example from Saša Jurić’s Rethinking app env:

First of all, the configuration naturally belongs to the place which uses it. So if I’m interested in db connection parameters, I’d first look at the repo module. And if I want to know about the endpoint parameters, then I’d look at the endpoint module.

A developer won’t always know where to look to find things that have been configured, however. I was talking to Nerve's Co-Creator Frank Hunleth about configuration in Elixir and he drove this point home stating, "Passing config through supervision trees makes it less accessible."

The core problem is that in a stateful system there is no fool proof way to know the configuration unless the system adheres to specific practices–and Elixir doesn’t give us an easy way to force the code follow specific practices.

Elixir configuration pitfalls

Due to the global nature of Elixir configuration, implicitly relying on configuration means you can no longer use <p-inline>async: true<p-inline> in your tests because if you modify the application environment in one test, another test that is running at the same time could read that configuration value. This results in a nasty race condition in your tests. And, needing to use <p-inline>async: false<p-inline> means that your tests run slower because they need to run in series instead of in parallel, which will increase your cycle time and slow your momentum.

Another pitfall is reading configuration inside module attributes (e.g. <p-inline>@my_mod Application.fetch_env!(:my_app, :my_mod)<p-inline> which used to be very common). This pattern is bad because if you change your configuration (by editing your <p-inline>config/config.exs<p-inline> file), then the <p-inline>@my_mod<p-inline> in your module will be out of date because your file will not be recompiled. Luckily, Elixir 1.10 added <p-inline>Application.compile_env/3<p-inline> which helps with this issue by warning you if the configuration no longer matches. You still need to manually recompile when the configuration changes, however.

Another reason to avoid this approach is because a release shouldn’t be specific for a specific environment. Ideally, a release is environment agnostic (especially between staging and production).

Configuration recommendations

1. Don’t read environment variables in random places and times throughout the code base. This makes it difficult to know all the configuration that the application has. Also, you’re distributing the environment variable parsing logic (often environment variables need to be cast from a string to another data type like a boolean or a number.
2. Minimize references to code in your config scripts. As stated in Rethinking app env, "MFAs or atoms that are implicitly tied to modules in the compiled code are an accident waiting to happen. If you rename the module, but neglect to update the config, things will break. And sometimes this will only affect production, because the config for production is different than the config for development or staging."
3. Avoid compile-time configuration when possible–especially any compile-time configuration that depends on environment variables. This will cause issues, especially in development because you’ll need to recompile your project manually when you change any environment variables that are read at compile-time.
4. Avoid storing secrets in your git repository. This is a way that secrets can easily leak out (so, this obviously applies to more than Elixir). My general approach to handle this is to use environment variables in a .env file for development environments (the <p-inline>.env<p-inline> is in <p-inline>.gitignore<p-inline> so it doesn’t get accidentally committed). I’ll talk about how the .env file is used in a future section
5. Don’t use too much code to deal with configuration. Using too much code can make the flow less clear, and it should always be easy to ascertain what the code is doing.

Using runtime configuration

As we’ve covered, since Elixir 1.11 a <p-inline>config/runtime.exs<p-inline> file has been supported

Benefits of runtime configuration:

• We can reliably work with environment variables.
• In development they are loaded by <p-inline>config/runtime.exs<p-inline> by a library like dotenv_parser.
• Note: The environment variables are also accessible to any programs launched by watchers in Phoenix.
• Compile-time configuration is completely avoided, thus avoiding its resultant headaches of needing to force recompilation.
• Since the configuration is only available at runtime code cannot accidentally rely on the configuration at compile-time.

Setting up configuration for complex projects

Note: This section is applicable for large, complex projects that require thoughtful management. This approach isn't as necessary for simple projects where configuration is minimized, because with only a small amount of configuration, how you manage it is not very important.

1. Move as much configuration as possible into runtime configuration. In development, I like to use dotenv_parser to load a .env file from the root of the project. The .env file is where any environment variables for things like api keys should go.
2. Use configuration helpers. I like to use configuration helpers which are inspired by this file. Using configuration helpers like this eases the burden of converting string environment variables into their proper form which may be a boolean or an integer. Once I have the config helpers imported I can read an environment variable with <p-inline>get_env("RUN_SEEDS_ON_STARTUP"<p-inline>, false, :boolean) where false is the default if the environment variable <p-inline>RUN_SEEDS_ON_STARTUP<p-inline> is not provided and :boolean means to parse the environment variable into either false or true (any values that can’t be parsed to one of those is treated as an error).
3. Use a separate runtime configuration file for each environment. I’ve found that once I moved the majority of my configuration into runtime configuration, my runtime.exs, and I follow the Elixir core teams recommendation to use if <p-inline>config_env() == :prod<p-inline> do in your runtime.exs that my runtime.exs file grows very large. The length of the file makes it difficult to follow because you’ll have indented sections that are very long, sometimes as much as 2 or 3 screen lengths. Instead, I prefer to have a separate runtime configuration file for each environment so there’s runtime.dev.exs, runtime.test.exs, and runtime.prod.exs. However, this doesn’t work out of the box and if you try to set this up in the same way that the compile-time configuration is setup (by using <p-inline>import_config "#{config_env()}.exs")<p-inline> you’ll get an error:

** (RuntimeError) import_config/1 is not enabled for this configuration file. Some configuration files do not allow importing other files as they are often copied to external systems

So instead we will need to use <p-inline>Code.require_file/2<p-inline>:

Additionally we need to copy <p-inline>runtime.prod.exs<p-inline> as part of our release process. To do so open up your mix.exs and add:

Congratulations! You now have per-file runtime configuration setup! Enjoy your flexible configuration.

‍

Working at Felt

If you like Elixir and you like maps, join us!