Sanitizer support

Sanitizers can be used in custom Julia builds to make it easier to detect certain kinds of errors in Julia’s internal C/C++ code.

Address Sanitizer: easy build

From a source-checkout of Julia, you should be able to build a version supporting address sanitization in Julia and LLVM as follows:

$ mkdir /tmp/julia
$ contrib/asan/build.sh /tmp/julia/

Here we’ve chosen /tmp/julia as a build directory, but you can choose whatever you wish. Once built, run the workload you wish to test with /tmp/julia/julia. Memory bugs will result in errors.

If you require customization or further detail, see the documentation below.

General considerations

Using Clang’s sanitizers obviously requires you to use Clang (USECLANG=1), but there’s another catch: most sanitizers require a run-time library, provided by the host compiler, while the instrumented code generated by Julia’s JIT relies on functionality from that library. This implies that the LLVM version of your host compiler must match that of the LLVM library used within Julia.

An easy solution is to have a dedicated build folder for providing a matching toolchain, by building with BUILD_LLVM_CLANG=1. You can then refer to this toolchain from another build folder by specifying USECLANG=1 while overriding the CC and CXX variables.

The sanitizers error out when they detect a shared library being opened using RTLD_DEEPBIND (ref: google/sanitizers#611). Since libblastrampoline by default uses RTLD_DEEPBIND, we need to set the environment variable LBT_USE_RTLD_DEEPBIND=0 when using a sanitizer.

To use one of of the sanitizers set SANITIZE=1 and then the appropriate flag for the sanitizer you want to use.

On macOS, this might need some extra flags also to work. Altogether, it might look like this, plus one or more of the SANITIZE_* flags listed below:

make -C deps USE_BINARYBUILDER_LLVM=0 LLVM_VER=svn stage-llvm

make -C src SANITIZE=1 USECLANG=1 \
    CC=~+/deps/scratch/llvm-svn/build_Release/bin/clang \
    CXX=~+/deps/scratch/llvm-svn/build_Release/bin/clang++ \
    CPPFLAGS="-isysroot $(xcode-select -p)/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" \
    CXXFLAGS="-isystem $(xcode-select -p)/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1"

(or put these into your Make.user, so you don’t need to remember them every time).

Address Sanitizer (ASAN)

For detecting or debugging memory bugs, you can use Clang’s address sanitizer (ASAN). By compiling with SANITIZE_ADDRESS=1 you enable ASAN for the Julia compiler and its generated code. In addition, you can specify LLVM_SANITIZE=1 to sanitize the LLVM library as well. Note that these options incur a high performance and memory cost. For example, using ASAN for Julia and LLVM makes testall1 take 8-10 times as long while using 20 times as much memory (this can be reduced to respectively a factor of 3 and 4 by using the options described below).

By default, Julia sets the allow_user_segv_handler=1 ASAN flag, which is required for signal delivery to work properly. You can define other options using the ASAN_OPTIONS environment flag, in which case you’ll need to repeat the default option mentioned before. For example, memory usage can be reduced by specifying fast_unwind_on_malloc=0 and malloc_context_size=2, at the cost of backtrace accuracy. For now, Julia also sets detect_leaks=0, but this should be removed in the future.

Example setup

Step 1: Install toolchain

Checkout a Git worktree (or create out-of-tree build directory) at $TOOLCHAIN_WORKTREE and create a config file $TOOLCHAIN_WORKTREE/Make.user with

USE_BINARYBUILDER_LLVM=1
BUILD_LLVM_CLANG=1

Run:

cd $TOOLCHAIN_WORKTREE
make -C deps install-llvm install-clang install-llvm-tools

to install toolchain binaries in $TOOLCHAIN_WORKTREE/usr/tools

Step 2: Build Julia with ASAN

Checkout a Git worktree (or create out-of-tree build directory) at $BUILD_WORKTREE and create a config file $BUILD_WORKTREE/Make.user with

TOOLCHAIN=$(TOOLCHAIN_WORKTREE)/usr/tools

# use our new toolchain
USECLANG=1
override CC=$(TOOLCHAIN)/clang
override CXX=$(TOOLCHAIN)/clang++
export ASAN_SYMBOLIZER_PATH=$(TOOLCHAIN)/llvm-symbolizer

USE_BINARYBUILDER_LLVM=1

override SANITIZE=1
override SANITIZE_ADDRESS=1

# make the GC use regular malloc/frees, which are hooked by ASAN
override WITH_GC_DEBUG_ENV=1

# default to a debug build for better line number reporting
override JULIA_BUILD_MODE=debug

# make ASAN consume less memory
export ASAN_OPTIONS=detect_leaks=0:fast_unwind_on_malloc=0:allow_user_segv_handler=1:malloc_context_size=2

JULIA_PRECOMPILE=1

# tell libblastrampoline to not use RTLD_DEEPBIND
export LBT_USE_RTLD_DEEPBIND=0

Run:

cd $BUILD_WORKTREE
make debug

to build julia-debug with ASAN.

Memory Sanitizer (MSAN)

For detecting use of uninitialized memory, you can use Clang’s memory sanitizer (MSAN) by compiling with SANITIZE_MEMORY=1.

Thread Sanitizer (TSAN)

For debugging data-races and other threading related issues you can use Clang’s thread sanitizer (TSAN) by compiling with SANITIZE_THREAD=1.