Max Fierke

never met a *nix system I didn't want to be friends with

So You Want to Compile Crystal on ARM64 (macOS)

Max Fierke

April 13, 2021

Mid-February, I was able to get a PR merged upstream into Crystal for aarch64-apple-darwin support (i.e. Apple Silicon) based on some great work by Romain Franceschini to get the libc bindings and a lot of the foundational stuff ready. This PR also shipped with Crystal 1.0.0, and I was hoping that the cross-compilation story would get a lot easier, so I put off writing a post on targeting arm64. Unfortunately, there's been some speed-bumps I won't get into here, but suffice to say it's not quite possible to brew install crystal via Homebrew yet or run via an official M1 binary build either.

However, I wanted to revisit the process for cross-compiling your copy of the Crystal compiler on the Apple M1 for those that are daring and not willing to wait. My hope is to keep this somewhat updated, so that as things evolve and change, eventually this'll be a pretty simple set of steps. Already, writing this in early/mid April, things have gotten a little bit easier. bdw-gc in Homebrew now ships with the necessary multi-threading patch for Crystal. In addition, I landed another PR to rewrite the Crystal segfault handler in Crystal, instead of C, making it easier to cross-compile (this will ship in 1.1.0.)

Environment preparation

Installing Homebrew(s)

We'll need two Homebrews, one for each CPU architecture, in order to cross-compile Crystal on the same machine. This is because we need to bootstrap the compiler from somewhere, and what better way than to bootstrap it from the x86_64 compiler via Rosetta 2, which we can also run on the machine.

Install x86_64 Homebrew (aka. Intel Homebrew)

Run the following to install Intel Homebrew, which we will occasionally refer to as ibrew throughout this post:

$ arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Once this finishes, you should have a working Homebrew install in the default x86_64 prefix of /usr/local

ibrew calls specified later will be in reference to this Intel Homebrew install.

Add an alias for Intel Homebrew

In your shell or your shell's config (.bashrc, .zshrc, etc.), add an alias for ibrew:

alias ibrew="arch -x86_64 /usr/local/bin/brew"

Install arm64 Homebrew (aka. ARM Homebrew)

Run the same line again, this time removing the explicit arch call in front of it. It should run under arm64 by default, unless your Terminal is setup to run under Rosetta (Stop doing that now, if you are using your Terminal in Rosetta mode. It'll make the rest so much harder/impossible, and I cannot help you.)

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

This should prompt for installing into /opt/homebrew, the default prefix for Apple Silicon Homebrew.

Regular brew calls specified later will be in reference to ARM Homebrew.

Add ARM homebrew to your PATH

In your shell or your shell's config, add ARM brew to your path:

$ export PATH="/opt/homebrew/bin:/usr/local/bin:$PATH"

This is to ensure ARM software from Homebrew runs before Intel software.

Install Crystal dependencies

LLVM 11+ is required for targeting Apple Silicon, so we need to make sure we install LLVM on both installations of Homebrew, and all the other dependent libraries needed, including build-time dependencies.

$ ibrew install llvm bdw-gc gmp libevent libyaml [email protected] pcre pkg-config
$ brew install llvm bdw-gc gmp libevent libyaml [email protected] pcre pkg-config

Grab an official macOS binary build of Crystal for x86_64

We'll need this for later, when compiling the compiler the first time. You could also install with ibrew as well.

$ mkdir -p ~/crystal-prebuilt
$ curl -fsSL https://github.com/crystal-lang/crystal/releases/download/1.0.0/crystal-1.0.0-1-darwin-x86_64.tar.gz > ~/crystal-prebuilt/crystal-1.0.0-1-darwin-x86_64.tar.gz
$ cd ~/crystal-prebuilt
$ tar -xf crystal-1.0.0-1-darwin-x86_64.tar.gz

Compiling Crystal for ARM64

Setup compile environment variables

Since we'll be targeting two different CPU architectures, we need to be extra careful about what build environment we specify. Intel Homebrew lives in a higher priority directory for most build tools, so it is often checked first and will be used instead of ARM Homebrew's directories unless we specify ARM Homebrew's build tools instead. We'll need to do this a lot, so lets specify some environment variables up-front:

$ export INTEL_BREW_PREFIX="$(ibrew --prefix)"
$ export INTEL_LLVM_ROOT="$INTEL_BREW_PREFIX/opt/llvm"
$ export INTEL_LLVM_CONFIG="$INTEL_LLVM_ROOT/bin/llvm-config"

$ export ARM_BREW_PREFIX="$(brew --prefix)"
$ export ARM_LLVM_ROOT="$ARM_BREW_PREFIX/opt/llvm"
$ export ARM_LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config"

Compile for x86_64

The first step is to compile for x8664. Yep, x8664. We need to do this, because the official binary builds of the Crystal compiler ship compiled against [email protected] and without the ability to cross-compile for AArch64/ARM64 on Darwin, so we need to re-compile the compiler against LLVM 11 so that we can have the ability to cross-compile for arm64. In the future, this may not be necessary but for now it is.

$ LLVM_CONFIG="$INTEL_LLVM_CONFIG" \
   LDFLAGS="-L$INTEL_LLVM_ROOT/lib" \
   CPPFLAGS="-I$INTEL_LLVM_ROOT/include" \
   CC="$INTEL_LLVM_ROOT/bin/clang" \
   AR="$INTEL_LLVM_ROOT/bin/llvm-ar" \
   CRYSTAL="$HOME/crystal-prebuilt/crystal-1.0.0-1/bin/crystal" \
   arch -x86_64 make

Cross-compile to arm64

The next step is to take our x86_64 compiler and use it to cross-compile for the arm64 target triple (aarch64-apple-darwin).

$ LLVM_CONFIG="$INTEL_LLVM_CONFIG" \
   LDFLAGS="-L$INTEL_LLVM_ROOT/lib" \
   CRYSTAL_CONFIG_TARGET=aarch64-apple-darwin \
   CRYSTAL_CONFIG_LIBRARY_PATH="$ARM_BREW_PREFIX/lib" \
   arch -x86_64 ./bin/crystal build src/compiler/crystal.cr --cross-compile --target aarch64-apple-darwin -Dwithout_playground

Next, we need to compile the LLVM extension libraries for arm64:

$ LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
   LDFLAGS="-L$ARM_LLVM_ROOT/lib" \
   $ARM_LLVM_ROOT/bin/clang -I$ARM_LLVM_ROOT/include -c -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc

Finally, we need to link our cross-compiled compiler object (crystal.o) with our arm64 llvm_ext.o and the rest of our arm64 libraries, in order to get a final arm64 executable for the Crystal compiler.

$ LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
   LDFLAGS="-L$ARM_LLVM_ROOT/lib" \
   CPPFLAGS="-I$ARM_LLVM_ROOT/include" \
   $ARM_LLVM_ROOT/bin/clang crystal.o -o .build/crystal  -rdynamic -L$ARM_BREW_PREFIX/lib src/llvm/ext/llvm_ext.o `"$ARM_LLVM_ROOT/bin/llvm-config" --libs --system-libs --ldflags 2> /dev/null` -lstdc++ -lpcre -lgc -lpthread -L$(brew --prefix libevent)/lib -levent -liconv -ldl

In this final step, it's super easy for x8664 libraries to get picked up instead of or in addition to the arm64 libraries. If you get a working compiler at the end but have a bunch of warnings about the linker trying to link against an x8664 library, you can safely ignore those. If you don't get a working compiler at the end, it's possible you're missing an arm64 version of a required library, and you'll need to specify a more direct path via the linker arguments.

Verifying the compiled compiler

Now, at the end, we should be able to verify that we have a working arm64 crystal compiler:

$ file .build/crystal
.build/crystal: Mach-O 64-bit executable arm64

(Optional): Running compiler and stdlib test suite

This step is probably optional if you just want to compile some of your Crystal programs for arm64, but if you want to be extra sure the compiler works okay or are working on Crystal compiler or standard library development, you may want to run the test suite.

We're doing this directly to avoid triggering unnecessary rebuilds via the Makefile, but it's likely possible to just call make spec with the same environment variables specified and have the right thing happen.

Compiling and running the compiler specs
$ CRYSTAL_PATH=$(pwd)/src \
   CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \
   CC="$ARM_LLVM_ROOT/bin/clang" \
   LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
   LDFLAGS="-L$ARM_LLVM_ROOT/lib" \
   CPPFLAGS="-I$ARM_LLVM_ROOT/include" \
   PKG_CONFIG_PATH="$(brew --prefix [email protected])/lib/pkgconfig" \
   ./bin/crystal build -o spec/compiler_spec spec/compiler_spec.cr -Di_know_what_im_doing

$ CRYSTAL_PATH=$(pwd)/src \
   CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \
   CC="$ARM_LLVM_ROOT/bin/clang" \
   LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
   LDFLAGS="-L$ARM_LLVM_ROOT/lib" \
   CPPFLAGS="-I$ARM_LLVM_ROOT/include" \
   PKG_CONFIG_PATH="$(brew --prefix [email protected])/lib/pkgconfig" \
   spec/compiler_spec
Compiling and running the stdlib specs
$ CRYSTAL_PATH=$(pwd)/src \
  CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \
  CC="$ARM_LLVM_ROOT/bin/clang" \
  LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
  LDFLAGS="-L$ARM_LLVM_ROOT/lib" \
  CPPFLAGS="-I$ARM_LLVM_ROOT/include" \
  PKG_CONFIG_PATH="$(brew --prefix [email protected])/lib/pkgconfig" \
  ./bin/crystal build -o spec/std_spec spec/std_spec.cr --target aarch64-apple-darwin -Di_know_what_im_doing

$ CRYSTAL_PATH=$(pwd)/src \
   CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \
   CC="$ARM_LLVM_ROOT/bin/clang" \
   LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \
   LDFLAGS="-L$ARM_LLVM_ROOT/lib" \
   CFLAGS="--target aarch64-apple-darwin" \
   CPPFLAGS="-I$ARM_LLVM_ROOT/include" \
   PKG_CONFIG_PATH="$(brew --prefix [email protected])/lib/pkgconfig"
   spec/std_spec

Compiling Crystal programs on arm64

Now that we've got a working Crystal compiler for arm64, we can go about compiling our arm64 binaries.

Or we could. But we probably won't get very far yet. We still need to compile shards for arm64 so that you can install dependencies. However, compiling shards is a fairly good demo for how to compile any Crystal program for arm64, as it is basically just another Crystal program.

Compiling shards

$ git clone [email protected]:crystal-lang/shards.git
$ cd shards
# If $HOME/src/crystal isn't where Crystal is cloned for you, update the path in CRYSTAL_PATH and CRYSTAL
$ CRYSTAL_PATH=lib:$HOME/src/crystal/src \
   CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \
   CRYSTAL=$HOME/src/crystal/.build/crystal \
   make install

This should install it into /usr/local/bin, which we can verify by running the below command:

$ which shards
/usr/local/bin/shards

$ file $(which shards)
/usr/local/bin/shards: Mach-O 64-bit executable arm64

Compiling other programs

In general, you can probably get most of the way there with your programs by providing CRYSTAL_PATH, CRYSTAL_LIBRARY_PATH, and CRYSTAL as specified above if you have a Makefile-type setup where CRYSTAL can be provided. Otherwise, you can invoke the arm64-compiled crystal binary directly while providing CRYSTAL_PATH and CRYSTAL_LIBRARY_PATH. These two values are needed, because we're not using an omnibus build where these values would already be set by a wrapper script. Instead, we have to provide them and be explicit so that the compiler knows where to look up Crystal code and linkable libraries, respectively.

Depending on the libraries used in your application, you may need to supply other typical environment variables that you might be expected to supply anyone, the most common being for OpenSSL 1.1, e.g. PKG_CONFIG_PATH="$(brew --prefix [email protected])/lib/pkgconfig"

HACK: Mimicking the Omnibus Crystal wrapper

Instead of passing the CRYSTAL_PATH and CRYSTAL_LIBRARY_PATH all over, we can also add a thin shell wrapper to provide those as part of the crystal command. This is similar to what the official omnibus builds do, where there's a top-level crystal shell script which sets the environment properly and passes off to the actual crystal binary.

Add this script to your PATH somewhere (don't forget to make it executable):

#!/bin/sh

# Again, if you installed somewhere other than $HOME/src/crystal, change it:
export CRYSTAL_PATH=lib:$HOME/src/crystal/src
# This should be ARM homebrew's lib
export CRYSTAL_LIBRARY_PATH=$(brew --prefix)/lib
# Mad temporary. You might want to copy this build else where and point it to that less-temporary location
export CRYSTAL=$HOME/src/crystal/.build/crystal

exec "$CRYSTAL" "[email protected]"

This is a bit of a temporary improvement to ergonomics, but should reduce the amount of pain when working in Crystal regularly on an M1.

Future Work

In future, I will be looking at trying to get aarch64/arm64 macOS support into the omnibus setup, but long-term this will require having Apple Silicon in CI, which would require some involvement from the core team.

In the meantime, I'm may try and get an initial version of the omnibus package working, that maybe I can provide builds for in a lagging fashion until it's possible to either cross-compile for arm64 in Homebrew via bootstrapping from the official x86_64 build (will require someone figuring out the LLVM 11 issues in release mode) or the core team is able to provide official builds.