gdb debugging tips

Within gdb, any jl_value_t* object obj can be displayed using

The object will be displayed in the julia session, not in the gdb session. This is a useful way to discover the types and values of objects being manipulated by Julia’s C code.

Similarly, if you’re debugging some of Julia’s internals (e.g., compiler.jl), you can print obj using

This is a good way to circumvent problems that arise from the order in which julia’s output streams are initialized.

Julia’s flisp interpreter uses value_t objects; these can be displayed with call fl_print(fl_ctx, ios_stdout, obj).

While the addresses of many variables, like singletons, can be useful to print for many failures, there are a number of additional variables (see julia.h for a complete list) that are even more useful.

In your gdb session, set a breakpoint in jl_breakpoint like so:

Then within your Julia code, insert a call to jl_breakpoint by adding

where obj can be any variable or tuple you want to be accessible in the breakpoint.

It’s particularly helpful to back up to the jl_apply frame, from which you can display the arguments to a function using, e.g.,

Another useful frame is to_function(jl_method_instance_t li, bool cstyle). The jl_method_instance_t argument is a struct with a reference to the final AST sent into the compiler. However, the AST at this point will usually be compressed; to view the AST, call jl_uncompress_ast and then pass the result to jl_:

Julia requires a few signals to function properly. The profiler uses SIGUSR2 for sampling and the garbage collector uses SIGSEGV for threads synchronization. If you are debugging some code that uses the profiler or multiple threads, you may want to let the debugger ignore these signals since they can be triggered very often during normal operations. The command to do this in GDB is (replace SIGSEGV with SIGUSR2 or other signals you want to ignore):

The corresponding LLDB command is (after the process is started):

If you are debugging a segfault with threaded code, you can set a breakpoint on jl_critical_error (sigdie_handler should also work on Linux and BSD) in order to only catch the actual segfault rather than the GC synchronization points.

Errors that occur during make need special handling. Julia is built in two stages, constructing sys0 and sys.ji. To see what commands are running at the time of failure, use make VERBOSE=1.

At the time of this writing, you can debug build errors during the sys0 phase from the base directory using:

You might need to delete all the files in usr/lib/julia/ to get this to work.

You can debug the sys.ji phase using:

By default, any errors will cause Julia to exit, even under gdb. To catch an error "in the act", set a breakpoint in jl_error (there are several other useful spots, for specific kinds of failures, including: jl_too_few_args, jl_too_many_args, and jl_throw).

Once an error is caught, a useful technique is to walk up the stack and examine the function by inspecting the related call to jl_apply. To take a real-world example:

The most recent jl_apply is at frame #3, so we can go back there and look at the AST for the function julia_convert_16886. This is the uniqued name for some method of convert. f in this frame is a jl_function_t*, so we can look at the type signature, if any, from the specTypes field:

Then, we can look at the AST for this function:

Finally, and perhaps most usefully, we can force the function to be recompiled in order to step through the codegen process. To do this, clear the cached functionObject from the jl_lamdbda_info_t*:

Then, set a breakpoint somewhere useful (e.g. emit_function, emit_expr, emit_call, etc.), and run codegen:

Module precompilation spawns a separate Julia process to precompile each module. Setting a breakpoint or catching failures in a precompile worker requires attaching a debugger to the worker. The easiest approach is to set the debugger watch for new process launches matching a given name. For example:

or:

Then run a script/command to start precompilation. As described earlier, use conditional breakpoints in the parent process to catch specific file-loading events and narrow the debugging window. (some operating systems may require alternative approaches, such as following each fork from the parent process)

Julia now works out of the box with rr, the lightweight recording and deterministic debugging framework from Mozilla. This allows you to replay the trace of an execution deterministically. The replayed execution’s address spaces, register contents, syscall data etc are exactly the same in every run.

A recent version of rr (3.1.0 or higher) is required.