Metaprogramming

The : character has two syntactic purposes in Julia. The first form creates a Symbol, an interned string used as one building-block of expressions, from valid identifiers:

The Symbol constructor takes any number of arguments and creates a new symbol by concatenating their string representations together:

In the context of an expression, symbols are used to indicate access to variables; when an expression is evaluated, a symbol is replaced with the value bound to that symbol in the appropriate scope.

Sometimes extra parentheses around the argument to : are needed to avoid ambiguity in parsing:

The second syntactic purpose of the : character is to create expression objects without using the explicit Expr constructor. This is referred to as quoting. The : character, followed by paired parentheses around a single statement of Julia code, produces an Expr object based on the enclosed code. Here is an example of the short form used to quote an arithmetic expression:

(to view the structure of this expression, try ex.head and ex.args, or use dump as above or Meta.@dump)

Note that equivalent expressions may be constructed using Meta.parse or the direct Expr form:

Expressions provided by the parser generally only have symbols, other expressions, and literal values as their args, whereas expressions constructed by Julia code can have arbitrary run-time values without literal forms as args. In this specific example, + and a are symbols, *(b,c) is a subexpression, and 1 is a literal 64-bit signed integer.

There is a second syntactic form of quoting for multiple expressions: blocks of code enclosed in quote ... end.

Direct construction of Expr objects with value arguments is powerful, but Expr constructors can be tedious compared to "normal" Julia syntax. As an alternative, Julia allows interpolation of literals or expressions into quoted expressions. Interpolation is indicated by a prefix $.

In this example, the value of variable a is interpolated:

Interpolating into an unquoted expression is not supported and will cause a compile-time error:

In this example, the tuple (1,2,3) is interpolated as an expression into a conditional test:

The use of $ for expression interpolation is intentionally reminiscent of string interpolation and command interpolation. Expression interpolation allows convenient, readable programmatic construction of complex Julia expressions.

Notice that the $ interpolation syntax allows inserting only a single expression into an enclosing expression. Occasionally, you have an array of expressions and need them all to become arguments of the surrounding expression. This can be done with the syntax $(xs...). For example, the following code generates a function call where the number of arguments is determined programmatically:

Naturally, it is possible for quote expressions to contain other quote expressions. Understanding how interpolation works in these cases can be a bit tricky. Consider this example:

Notice that the result contains $x, which means that x has not been evaluated yet. In other words, the $ expression "belongs to" the inner quote expression, and so its argument is only evaluated when the inner quote expression is:

However, the outer quote expression is able to interpolate values inside the $ in the inner quote. This is done with multiple $s:

Notice that (1 + 2) now appears in the result instead of the symbol x. Evaluating this expression yields an interpolated 3:

The intuition behind this behavior is that x is evaluated once for each $: one $ works similarly to eval(:x), giving x's value, while two $s do the equivalent of eval(eval(:x)).

The usual representation of a quote form in an AST is an Expr with head :quote:

As we have seen, such expressions support interpolation with $. However, in some situations it is necessary to quote code without performing interpolation. This kind of quoting does not yet have syntax, but is represented internally as an object of type QuoteNode:

The parser yields QuoteNodes for simple quoted items like symbols:

QuoteNode can also be used for certain advanced metaprogramming tasks.

Given an expression object, one can cause Julia to evaluate (execute) it at global scope using eval:

Every module has its own eval function that evaluates expressions in its global scope. Expressions passed to eval are not limited to returning values — they can also have side-effects that alter the state of the enclosing module’s environment:

Here, the evaluation of an expression object causes a value to be assigned to the global variable x.

Since expressions are just Expr objects which can be constructed programmatically and then evaluated, it is possible to dynamically generate arbitrary code which can then be run using eval. Here is a simple example:

The value of a is used to construct the expression ex which applies the + function to the value 1 and the variable b. Note the important distinction between the way a and b are used:

As hinted above, one extremely useful feature of Julia is the capability to generate and manipulate Julia code within Julia itself. We have already seen one example of a function returning Expr objects: the parse function, which takes a string of Julia code and returns the corresponding Expr. A function can also take one or more Expr objects as arguments, and return another Expr. Here is a simple, motivating example:

As another example, here is a function that doubles any numeric argument, but leaves expressions alone:

Here is an extraordinarily simple macro:

Macros have a dedicated character in Julia’s syntax: the @ (at-sign), followed by the unique name declared in a macro NAME ... end block. In this example, the compiler will replace all instances of @sayhello with:

When @sayhello is entered in the REPL, the expression executes immediately, thus we only see the evaluation result:

Now, consider a slightly more complex macro:

This macro takes one argument: name. When @sayhello is encountered, the quoted expression is expanded to interpolate the value of the argument into the final expression:

We can view the quoted return expression using the function macroexpand (important note: this is an extremely useful tool for debugging macros):

We can see that the "human" literal has been interpolated into the expression.

There also exists a macro @macroexpand that is perhaps a bit more convenient than the macroexpand function:

We have already seen a function f(::Expr...) -> Expr in a previous section. In fact, macroexpand is also such a function. So, why do macros exist?

Macros are necessary because they execute when code is parsed, therefore, macros allow the programmer to generate and include fragments of customized code before the full program is run. To illustrate the difference, consider the following example:

The first call to println is executed when macroexpand is called. The resulting expression contains only the second println:

Macros are invoked with the following general syntax:

Note the distinguishing @ before the macro name and the lack of commas between the argument expressions in the first form, and the lack of whitespace after @name in the second form. The two styles should not be mixed. For example, the following syntax is different from the examples above; it passes the tuple (expr1, expr2, ...) as one argument to the macro:

An alternative way to invoke a macro over an array literal (or comprehension) is to juxtapose both without using parentheses. In this case, the array will be the only expression fed to the macro. The following syntax is equivalent (and different from @name [a b] * v):

It is important to emphasize that macros receive their arguments as expressions, literals, or symbols. One way to explore macro arguments is to call the show function within the macro body:

In addition to the given argument list, every macro is passed extra arguments named __source__ and __module__.

The argument __source__ provides information (in the form of a LineNumberNode object) about the parser location of the @ sign from the macro invocation. This allows macros to include better error diagnostic information, and is commonly used by logging, string-parser macros, and docs, for example, as well as to implement the @__LINE__, @__FILE__, and @__DIR__ macros.

The location information can be accessed by referencing __source__.line and __source__.file:

The argument __module__ provides information (in the form of a Module object) about the expansion context of the macro invocation. This allows macros to look up contextual information, such as existing bindings, or to insert the value as an extra argument to a runtime function call doing self-reflection in the current module.

Here is a simplified definition of Julia’s @assert macro:

This macro can be used like this:

In place of the written syntax, the macro call is expanded at parse time to its returned result. This is equivalent to writing:

That is, in the first call, the expression :(1 == 1.0) is spliced into the test condition slot, while the value of string(:(1 == 1.0)) is spliced into the assertion message slot. The entire expression, thus constructed, is placed into the syntax tree where the @assert macro call occurs. Then at execution time, if the test expression evaluates to true, then nothing is returned, whereas if the test is false, an error is raised indicating the asserted expression that was false. Notice that it would not be possible to write this as a function, since only the value of the condition is available and it would be impossible to display the expression that computed it in the error message.

The actual definition of @assert in Julia Base is more complicated. It allows the user to optionally specify their own error message, instead of just printing the failed expression. Just like in functions with a variable number of arguments (Varargs Functions), this is specified with an ellipses following the last argument:

Now @assert has two modes of operation, depending upon the number of arguments it receives! If there’s only one argument, the tuple of expressions captured by msgs will be empty and it will behave the same as the simpler definition above. But now if the user specifies a second argument, it is printed in the message body instead of the failing expression. You can inspect the result of a macro expansion with the aptly named @macroexpand macro:

There is yet another case that the actual @assert macro handles: what if, in addition to printing "a should equal b," we wanted to print their values? One might naively try to use string interpolation in the custom message, e.g., @assert a==b "a ( b)!", but this won’t work as expected with the above macro. Can you see why? Recall from string interpolation that an interpolated string is rewritten to a call to string. Compare:

So now instead of getting a plain string in msg_body, the macro is receiving a full expression that will need to be evaluated in order to display as expected. This can be spliced directly into the returned expression as an argument to the string call; see error.jl for the complete implementation.

The @assert macro makes great use of splicing into quoted expressions to simplify the manipulation of expressions inside the macro body.

An issue that arises in more complex macros is that of hygiene. In short, macros must ensure that the variables they introduce in their returned expressions do not accidentally clash with existing variables in the surrounding code they expand into. Conversely, the expressions that are passed into a macro as arguments are often expected to evaluate in the context of the surrounding code, interacting with and modifying the existing variables. Another concern arises from the fact that a macro may be called in a different module from where it was defined. In this case we need to ensure that all global variables are resolved to the correct module. Julia already has a major advantage over languages with textual macro expansion (like C) in that it only needs to consider the returned expression. All the other variables (such as msg in @assert above) follow the normal scoping block behavior.

To demonstrate these issues, let us consider writing a @time macro that takes an expression as its argument, records the time, evaluates the expression, records the time again, prints the difference between the before and after times, and then has the value of the expression as its final value. The macro might look like this:

Here, we want t0, t1, and val to be private temporary variables, and we want time_ns to refer to the time_ns function in Julia Base, not to any time_ns variable the user might have (the same applies to println). Imagine the problems that could occur if the user expression ex also contained assignments to a variable called t0, or defined its own time_ns variable. We might get errors, or mysteriously incorrect behavior.

Julia’s macro expander solves these problems in the following way. First, variables within a macro result are classified as either local or global. A variable is considered local if it is assigned to (and not declared global), declared local, or used as a function argument name. Otherwise, it is considered global. Local variables are then renamed to be unique (using the gensym function, which generates new symbols), and global variables are resolved within the macro definition environment. Therefore both of the above concerns are handled; the macro’s locals will not conflict with any user variables, and time_ns and println will refer to the Julia Base definitions.

One problem remains however. Consider the following use of this macro:

Here the user expression ex is a call to time_ns, but not the same time_ns function that the macro uses. It clearly refers to MyModule.time_ns. Therefore we must arrange for the code in ex to be resolved in the macro call environment. This is done by "escaping" the expression with esc:

An expression wrapped in this manner is left alone by the macro expander and simply pasted into the output verbatim. Therefore it will be resolved in the macro call environment.

This escaping mechanism can be used to "violate" hygiene when necessary, in order to introduce or manipulate user variables. For example, the following macro sets x to zero in the call environment:

This kind of manipulation of variables should be used judiciously, but is occasionally quite handy.

Getting the hygiene rules correct can be a formidable challenge. Before using a macro, you might want to consider whether a function closure would be sufficient. Another useful strategy is to defer as much work as possible to runtime. For example, many macros simply wrap their arguments in a QuoteNode or other similar Expr. Some examples of this include @task body which simply returns schedule(Task(() -> $body)), and @eval expr, which simply returns eval(QuoteNode(expr)).

To demonstrate, we might rewrite the @time example above as:

However, we don’t do this for a good reason: wrapping the expr in a new scope block (the anonymous function) also slightly changes the meaning of the expression (the scope of any variables in it), while we want @time to be usable with minimum impact on the wrapped code.

Macros, just like Julia functions, are generic. This means they can also have multiple method definitions, thanks to multiple dispatch:

However one should keep in mind, that macro dispatch is based on the types of AST that are handed to the macro, not the types that the AST evaluates to at runtime:

Julia’s base library has an internal sub2ind function to calculate a linear index into an n-dimensional array, based on a set of n multilinear indices - in other words, to calculate the index i that can be used to index into an array A using A[i], instead of A[x,y,z,...]. One possible implementation is the following:

The same thing can be done using recursion:

Both these implementations, although different, do essentially the same thing: a runtime loop over the dimensions of the array, collecting the offset in each dimension into the final index.

However, all the information we need for the loop is embedded in the type information of the arguments. This allows the compiler to move the iteration to compile time and eliminate the runtime loops altogether. We can utilize generated functions to achieve a similar effect; in compiler parlance, we use generated functions to manually unroll the loop. The body becomes almost identical, but instead of calculating the linear index, we build up an expression that calculates the index:

What code will this generate?

An easy way to find out is to extract the body into another (regular) function:

We can now execute sub2ind_gen_impl and examine the expression it returns:

So, the method body that will be used here doesn’t include a loop at all - just indexing into the two tuples, multiplication and addition/subtraction. All the looping is performed compile-time, and we avoid looping during execution entirely. Thus, we only loop once per type, in this case once per N (except in edge cases where the function is generated more than once - see disclaimer above).

Generated functions can achieve high efficiency at run time, but come with a compile time cost: a new function body must be generated for every combination of concrete argument types. Typically, Julia is able to compile "generic" versions of functions that will work for any arguments, but with generated functions this is impossible. This means that programs making heavy use of generated functions might be impossible to statically compile.

To solve this problem, the language provides syntax for writing normal, non-generated alternative implementations of generated functions. Applied to the sub2ind example above, it would look like this:

Internally, this code creates two implementations of the function: a generated one where the first block in if @generated is used, and a normal one where the else block is used. Inside the then part of the if @generated block, code has the same semantics as other generated functions: argument names refer to types, and the code should return an expression. Multiple if @generated blocks may occur, in which case the generated implementation uses all of the then blocks and the alternate implementation uses all of the else blocks.

Notice that we added an error check to the top of the function. This code will be common to both versions, and is run-time code in both versions (it will be quoted and returned as an expression from the generated version). That means that the values and types of local variables are not available at code generation time --- the code-generation code can only see the types of arguments.

In this style of definition, the code generation feature is essentially an optional optimization. The compiler will use it if convenient, but otherwise may choose to use the normal implementation instead. This style is preferred, since it allows the compiler to make more decisions and compile programs in more ways, and since normal code is more readable than code-generating code. However, which implementation is used depends on compiler implementation details, so it is essential for the two implementations to behave identically.