The Missing Bit

Creating an elixir nif in zig
2022-11-26

zig erlang elixir

Zig is a language that has excellent interoperability with C, and for this reason, I use it a lot to interact with C libraries. I also use elixir as a daily driver for server applications.

In this post, I'll detail the step to be able to call zig code from elixir.

Zigler

First I'll point to zigler which is a project that allows for easy embedding of zig code in elixir. I am not using it for a few reasons:

But it is a very nice way to get some zig code into your elixir and I recommend it.

Doing a NIF from scratch

NIF is a way to write erlang code in C. The first reference you want to read is the tutorial.

The idea is to have C functions that receive elixir arguments and return data as elixir types.

For this, you need two things:

Let's start with the elixir module as it is the simplest.

As folder paths and such are a bit of a pain to figure out, I'll describe every single steps.

Elixir module

mkdir nif_test
cd nif_test

Now edit nif_test.ex and put this inside:


defmodule NifTest do
  @on_load :load_nifs

  def load_nifs do
    :erlang.load_nif('./build/libnif_test', 0)
  end

  def foo_test(_a) do
    raise "NIF foo_test/1 not implemented"
  end
end

If you fire iex and load the module you will get an error:


$ iex
iex(1)> c "nif_test.ex"

10:48:35.666 [warning] The on_load function for module Elixir.NifTest returned:
{:error,
 {:load_failed,
  'Failed to load NIF library: \'./build/libnif_test.so: cannot open shared object file: No such file or directory\''}}

[NifTest]
iex(2)>

This is normal of course, the dynamic library does not exist yet.

You will notice that the extension .so is added by the loader.

Zig module

Initialize the folder with a build.zig and a main.zig.

We are going to replace them fully, but you can do:

zig int-lib

It will create them in the current folder.

Now update build.zig with the following:


const std = @import("std");
const Pkg = std.build.Pkg;

pub fn build(b: *std.build.Builder) void {
    const mode = b.standardReleaseOptions();

    const nif_step = b.step("nif_lib", "Compiles erlang library");
    const nif_lib = b.addSharedLibrary("nif_test", "./src/main.zig", .unversioned);
    nif_lib.setBuildMode(mode);
    nif_lib.setOutputDir("build");
    nif_lib.addIncludePath("/usr/lib/erlang/usr/include/");

    nif_lib.install();
    nif_lib.linkLibC();
    nif_step.dependOn(&nif_lib.step);
}

This will add the erlang header path to the build environment and also link with the C library.

Finally, in main.zig:

const std = @import("std");
const erl = @cImport({
    @cInclude("erl_nif.h");
});

export fn foo_test(
    env: ?*erl.ErlNifEnv,
    argc: c_int,
    argv: [*c]const erl.ERL_NIF_TERM,
) erl.ERL_NIF_TERM {
    var test_binary: erl.ErlNifBinary = undefined;

    if ((argc != 1) or
        (erl.enif_inspect_binary(env, argv[0], &test_binary) != 1))
    {
        return erl.enif_make_badarg(env);
    }
    var test_slice: []const u8 = &.{};
    test_slice.len = test_binary.size;
    test_slice.ptr = test_binary.data;

    std.debug.print("Test slice: {s}\n\n", .{test_slice});

    return erl.enif_make_int(env, 2);
}

const func_count = 1;

var funcs = [func_count]erl.ErlNifFunc{
    erl.ErlNifFunc{
        .name = "foo_test",
        .arity = 1,
        .fptr = foo_test,
        .flags = 0,
    },
};

var entry = erl.ErlNifEntry{
    .major = erl.ERL_NIF_MAJOR_VERSION,
    .minor = erl.ERL_NIF_MINOR_VERSION,
    .name = "Elixir.NifTest",
    .num_of_funcs = func_count,
    .funcs = &funcs,
    .load = null,
    .reload = null,
    .upgrade = null,
    .unload = null,
    .vm_variant = "beam.vanilla",
    .options = 1,
    .sizeof_ErlNifResourceTypeInit = @sizeOf(erl.ErlNifResourceTypeInit),
    .min_erts = "erts-10.4",
};

export fn nif_init() *erl.ErlNifEntry {
    return &entry;
}

Finally, zig build nif_lib to build the module and run it with:

$ iex
iex(1)> c "nif_test.ex"
[NifTest]
iex(2)> NifTest.foo_test("1234")
Test slice: 1234

                2
iex(3)>

We see the output from zig. Of course, you should not output like that in real code, and instead always return data to erlang/elixir.

Detailed explaination

As we have a running minimal example, I will explain the code in details.

nif_test.ex


defmodule NifTest do
  # This is called when the module is loaded
  # It is important to call load_nifs from within the module, as
  # NIF can only be loaded within the module it will populate
  @on_load :load_nifs

  def load_nifs do
    # In a real nif, you would adjust this path
    # Second argument is what you C code will receive, we are ignoring it
    # so just pass 0 which is NULL on the C side
    :erlang.load_nif('./build/libnif_test', 0)
  end

  # Our first function, it needs to have the same name and arity as C version
  # Implementation is not important, but it is good manner to raise
  # if tthe NIF is not loaded.
  # The NIF version will come override this one.
  def foo_test(_a) do
    raise "NIF foo_test/1 not implemented"
  end
end

build.zig

const std = @import("std");
const Pkg = std.build.Pkg;

pub fn build(b: *std.build.Builder) void {

    // Add a build step, this will now be callable with `zig build nif_lib`
    // Steps are optional, you can ommit it, and `zig build` will invoque it
    const nif_step = b.step("nif_lib", "Compiles erlang library");
    // Create a shared library, the output name will have `lib` and `.so` added to it
    // It is not versioned as we will usually use elixir package versioning
    // instead, and we do not plan to distribute the library by itself
    const nif_lib = b.addSharedLibrary("nif_test", "./src/main.zig", .unversioned);

    const mode = b.standardReleaseOptions();
    // Use release mode
    nif_lib.setBuildMode(mode);

    // Override build dir, this is a convention of mine as it plays well with
    // IDE, but the default `zig-out` is fine
    nif_lib.setOutputDir("build");

    // We need to tell zig where to find the erlang headers
    nif_lib.addIncludePath("/usr/lib/erlang/usr/include/");

    // As erlang nif library is in C, we need this
    nif_lib.linkLibC();

    // Install the lib inside the
    nif_lib.install();
    nif_step.dependOn(&nif_lib.step);
}

main.zig


// Load zig standard library, here we need it for debug print to stderr
const std = @import("std");

// Load ERL library header
const erl = @cImport({
    @cInclude("erl_nif.h");
});

// Declare a NIF function
// Every function in your module will have the same zig signature
export fn foo_test(
    // This is the erlang environment, we need this for all NIF operations
    env: ?*erl.ErlNifEnv,
    // The number of argument your function was called with
    argc: c_int,
    // An array of arguments
    argv: [*c]const erl.ERL_NIF_TERM,
) erl.ERL_NIF_TERM { // Returned value is also a erlang term
    // Our test function is going to accept a single argument, a binary
    // We need to allocate the stack for it
    var test_binary: erl.ErlNifBinary = undefined;

    // We check that we have one argument and that we can transform this
    // argument into a binary, a binary is just a serie of bytes
    if ((argc != 1) or
        // we try to convert the argument at position 0 into a binary
        (erl.enif_inspect_binary(env, argv[0], &test_binary) != 1))
    {
        // If it fails, we return a bad argument error
        return erl.enif_make_badarg(env);
    }
    // If we are here, it's a success, we got our binary
    // Note that the binary data contained into `test_binary` will be
    // deallocated when this function returns
    // If you want to keep it longer, you need top copy it
    //
    // What we do here is that we build a slice and point it to the binary
    // this does NOT copy any data and this slice would be invalid after this
    // function returns
    //
    // Create an empty slice (pointer + len)
    var test_slice: []const u8 = &.{};
    // Set len
    test_slice.len = test_binary.size;
    // Set ptr
    test_slice.ptr = test_binary.data;

    // Now our slice is printable as a C string by zig formatter
    // This prints to stderr by default
    std.debug.print("Test slice: {s}\n\n", .{test_slice});

    // We return an int, 2, to demonstrate how to return data
    return erl.enif_make_int(env, 2);
}


// How many functions will our module have?
const func_count = 1;

// Create an array of function
var funcs = [func_count]erl.ErlNifFunc{
    // Our test function
    erl.ErlNifFunc{
        // The erlang/elixir name, this must match the name in the module
        .name = "foo_test",
        // The arity, note that you can declare multiple functions with same
        // name and different arity
        .arity = 1,
        // The function implementation, this must match the name in the zig code
        // above.
        // You can use the same zig function for multiple arity, thus the
        // argc argument to the zig function
        .fptr = foo_test,
        // Flags can be set to mark the function as dirty
        // Dirty functions can take more CPU time
        // Regular function must be fast, <1 ms
        // You can set this to erl.ERL_NIF_DIRTY_JOB_CPU_BOUND or
        // erl.ERL_NIF_DIRTY_JOB_IO_BOUND
        .flags = 0,
    },
};


// All the following code is required because it is implemented as macro in the
// C version With C, you just call the magic macro ERL_NIF_INIT
// but we cannot do that in zig, so here is a full manual implementation

// This is the NIF entry for out module
var entry = erl.ErlNifEntry{
    // We just the version of the header
    .major = erl.ERL_NIF_MAJOR_VERSION,
    .minor = erl.ERL_NIF_MINOR_VERSION,
    // If this is an elixir module that will be available as `NifTest`
    // on the elixir side, we need to add `Elixir` in front of it
    .name = "Elixir.NifTest",
    // Hor many functions
    .num_of_funcs = func_count,
    // Pointer to function definition array
    .funcs = &funcs,
    // Callbacks for the lifecycle of our NIF, they can be nil
    .load = null,
    .reload = null,
    .upgrade = null,
    .unload = null,
    // Which VM variant
    .vm_variant = "beam.vanilla",
    // This can be erl.ERL_NIF_DIRTY_NIF_OPTION
    .options = 0,
    // This must be defined as is
    .sizeof_ErlNifResourceTypeInit = @sizeOf(erl.ErlNifResourceTypeInit),
    // Minimum erts version
    .min_erts = "erts-10.4",
};

// This is the function that will be called by the erlang runtime when loading
// the module
export fn nif_init() *erl.ErlNifEntry {
    return &entry;
}

The repository with sample code is available on my github.

If you wish to comment or discuss this post, just mention me on Mastodon or Twitter.