Metaprogramming

The : symbol has two syntactic functions in Julia. The first form creates Symbol, https://en.wikipedia.org/wiki/String_interning [saved string] used as one building block of expressions based on valid names:

Designer Symbol takes any number of arguments and creates a new symbol by concatenating the representation of their strings:

In the context of an expression, symbols are used to indicate access to variables; when an expression is evaluated, the symbol is replaced by the value associated with that symbol in the corresponding regions.

Sometimes additional parentheses are required around the : argument to avoid ambiguity during analysis.:

The second syntactic function of the symbol : is to create expression objects without explicitly using the constructor Expr. This is called quoting. The character : followed by a pair of parentheses around a single Julia code expression produces an 'Expr` object based on the included code. Here is an example of a short form used to quote an arithmetic expression:

(to view the structure of this expression, try ex.head and ex.args' or use `dump, as shown above, or Meta.@dump)

Note that equivalent expressions can be constructed using Meta.parse' or the direct form `Expr:

Expressions provided by the analyzer usually have only symbols, other expressions, and literal values as their arguments, while expressions constructed by Julia code can have arbitrary runtime variables without literal forms as arguments. In this particular example, + and a are characters, *(b,c) is a subexpression, and 1 is a signed literal 64-bit integer.

There is a second syntactic form of quoting for several expressions: code blocks enclosed in quote ... end.

Direct construction of objects Expr is effective with value arguments, but using the Expr constructors can be tedious compared to Julia’s "normal" syntax. Alternatively, Julia allows interpolation of literals or expressions into quoted expressions. The interpolation is indicated by the prefix $.

In this example, the value of the variable a is interpolated.:

Interpolation into a non-quoted expression is not supported and will cause a compile-time error.:

In this example, the tuple (1,2,3) is interpolated as an expression in the condition check.:

Using $ to interpolate expressions intentionally resembles string interpolation and command interpolation. Expression interpolation provides convenient and readable software construction of complex Julia expressions.

Note that the $ interpolation syntax allows only one expression to be inserted into the enclosing expression. In some cases, you have an array of expressions, and you need them all to become arguments to the surrounding expression. This can be done using the syntax $(xs...). For example, the following code creates a function call in which the number of arguments is programmatically determined:

Naturally, it is possible for citation expressions to contain other citation expressions. Understanding how interpolation works in these cases can be somewhat difficult. Consider this example:

Note that the result contains $x, which means that x has not been calculated yet. In other words, the expression $ "belongs" to the internal quoting expression, and therefore its argument is evaluated only when the internal quoting expression is the following:

However, the external expression quote can interpolate the values inside the $ in the internal quotation. This is done with a few '$`:

Note that the result now displays (1 + 2) rather than the x character. Calculating this expression yields an interpolated 3:

The intuition behind this behavior is that x is calculated once for each $: one $ works similarly to eval(:x), assigning a value to x, while two $ perform the equivalent of eval(eval(:x)).

The usual representation of a citation (quote) in the AST is Expr with the title :quote:

As we have seen, such expressions support interpolation with $. However, in some situations it is necessary to quote the _ code without performing the interpolation. This type of quoting does not yet have a syntax, but it has an internal representation as a QuoteNode object`:

The analyzer outputs a QuoteNode for simple quoted elements such as characters:

The 'QuoteNode` can also be used for certain difficult-to-implement metaprogramming tasks.

If there is an expression object, you can make the Julia environment calculate (execute) it in the global scope using eval:

At everyone module has its own function eval, evaluates expressions in its global scope. Expressions passed eval are not limited to return values — they can also have side effects that change the environment state of the enabling module.:

Here` the evaluation of the expression object results in the assignment of a value to the global variable `x'.

Since expressions are just Expr objects, they can be constructed programmatically, and then it can be calculated whether it is possible to dynamically create arbitrary code, which can then be executed using eval. Here is a simple example.

The value a is used to construct the expression ex, which applies the function + to the value 1 and the variable b. Note the important difference between how a and b are used.:

As noted above, one of the extremely useful features of Julia is the ability to generate and manipulate Julia code in the Julia language itself. We have already seen one example of a function that returns objects. Expr: function Meta.parse', which accepts Julia’s code and returns the corresponding `Expr'. The function can also take one or more `Expr objects as arguments and return another `Expr'. Here is a simple motivating example.

Let’s give another example. Here is a function that doubles any numeric argument, but leaves the expressions untouched.:

Here is an extremely simple macro:

Macros in Julia syntax have a dedicated character: @ (commercial at), followed by the declared in the block macro NAME ... end. In this example, the compiler will replace all instances of @sayhello with:

When @sayhello is entered into the REPL, the expression is executed immediately, so we only see the result of the calculation.:

Now let’s look at a slightly more complex macro.:

This macro takes one argument: name'. If `@sayhello occurs, the quoted expression is expanded to interpolate the value of the argument in the final expression.:

We can view the returned quoted expression using the function macroexpand (important note: it is an extremely useful tool for debugging macros):

You can see that the literal "human" is interpolated into the expression.

There is also a macro '@macroexpand`, which is perhaps a little more convenient than the macroexpand function:

We have already seen the function f(::Expr...) -> Expr in the previous section. Actually macroexpand is also such a function. So why do macros exist?

Macros are necessary because they are executed when the code is analyzed, therefore, macros allow the programmer to create and include fragments of user code before the program is executed as a whole. To illustrate the difference, consider the following example.

The first challenge println is executed when called macroexpand. The resulting expression contains only the second println:

Macros are called using the following general syntax:

Note the distinctive @ before the macro name, the absence of commas between the argument expressions in the first form, and the absence of a space after the @name in the second. These two styles should not be mixed. For example, the following syntax differs from the examples above; it passes the tuple (expr1, expr2, ...) as a single argument to the macro:

An alternative way to call a macro for an array literal (or inclusion) is to contrast one against the other without using parentheses. In this case, the array will be the only expression passed to the macro. The following syntax is equivalent (and different from @name [a b] * v):

It is important to emphasize that macros receive their arguments as expressions, literals, or symbols. One way to examine the arguments of a macro is to call a function show in the body of the macro:

In addition to the specified list of arguments, each macro is passed additional arguments named `__source__' and `__module__'.

The __source__ argument provides information (in the form of a LineNumberNode object) about the location of the @ sign analyzer from the macro call. This allows macros to include better error diagnosis information and is commonly used, for example, by logging, string analysis macros, and documents, as well as for implementing macros. @__LINE__, @__FILE__ and @__DIR__.

Location information can be accessed by referring to __source__.line and __source__.file:

The __module__ argument provides information (in the form of a Module object) about the context of the macro call extension. This allows the macro to search for context information, such as existing relationships, or insert a value as an additional argument into a run-time function call that performs self-checking in the current module.

Here is a simplified definition of a macro @assert Julia:

This macro can be used as follows:

Instead of the written syntax, the macro is expanded during analysis to its returned result. This is equivalent to the following entry:

That is, in the first call, the expression :(1 == 1.0) is woven into the slot of the test condition, while the value of string(:(1 == 1.0)) is woven into the slot of the approval message. The entire expression, constructed in this way, is placed in the syntax tree in which the macro @assert is called. Then, during execution, if the value of the test expression is true when it is evaluated, it is returned nothing, whereas if the value is false, an error is returned indicating that the expression being validated had the value false. Note that it is impossible to write this as a function, since only the value of condition is available, and it is impossible to display the expression that calculated it in the error message.

The actual definition of @assert in Julia Base is more complicated. It allows the user to specify their own error message options, rather than just displaying a failed expression on the screen. Just like in functions with a variable number of arguments (Functions with a variable number of arguments (Vararg)), this is indicated by an ellipsis after the last argument:

Now '@assert` has two modes of operation depending on the number of arguments received! If there is only one argument, the expression tuple captured by msgs will be empty and will behave the same as the simpler definition above. But now, if the user specifies the second argument, it is displayed in the body of the message, and not in the failed expression. You can examine the results of macro expansion with an appropriately named macro. @macroexpand:

There is another case that the @assert macro handles: what if, in addition to displaying "a should equal b", we want to display their values? You can naively try to use string interpolation in a user message, for example, @assert a==b "a ( b)!", but this will not work as expected in the above macro. Do you understand why? Remember that in the section String interpolation it was said that the interpolated string is overwritten in the call string. Compare:

So now, instead of getting a simple string msg_body, the macro gets the entire expression, which needs to be calculated in order to display the data as expected. This can be woven into the returned expression as a call argument. string; for the full implementation, see by the link https://github.com/JuliaLang/julia/blob/master/base/error.jl [error.jl].

The '@assert` macro makes extensive use of interweaving quoted expressions to simplify operations with expressions inside the macro body.

The problem that arises in more complex macros is https://en.wikipedia.org/wiki/Hygienic_macro [hygiene]. In short, macros must ensure that the variables they represent in returned expressions do not accidentally conflict with existing variables in the surrounding code they are deployed to. Conversely, expressions that are passed to macros as arguments are often expected to perform calculations in the context of the surrounding code, interacting with existing variables and changing them. Another problem arises from the fact that a macro can be called in a module other than the one in which it was defined. In this case, we need to ensure that all global variables are resolved into the correct module. Julia already has an important advantage over languages with text macro expansion (such as C), which is that it is necessary to consider only returned expressions. All other variables (such as msg in @assert above) follow normal behavior of the scoping block.

To demonstrate these problems, consider writing the macro @time, which takes an expression as its argument, records the time, evaluates the expression, records the time again, displays the difference between the time before and after, and then gets the value of the expression as the final value. The macro may look like this:

Here we want t0, t1 and val to be private temporary variables, and also for time_ns to refer to the function time_ns in Julia Base, and not to any variable time_ns' that the user may have (the same applies to `println'). Imagine the problems that could occur if the custom expression `ex also contained assignments to a variable named t0 or defined its own variable `time_ns'. We could get errors or mysteriously incorrect behavior.

The Julia macro expander solves these problems as follows. First, the variables in the result of the macro are classified as local or global. A variable is considered local if it has a value assigned to it (and it is not declared global), it is declared local, or it is used as the name of a function argument. Otherwise, it is considered global. The local variables are then renamed to be unique (using the function gensym, which generates new characters), and global variables are resolved in the macro definition environment. Therefore, both of the problems described above are solved; local macro variables will not conflict with any user variables, and time_ns and println will refer to Julia Base definitions.

However, one problem remains. Consider the following usage of this macro:

Here, the custom expression ex is a call to time_ns, but not to the same function time_ns that the macro uses. It explicitly refers to MyModule.time_ns. Therefore, we need to make sure that the code in ex resolves to the macro environment. This is done by adding escape sequences to the expression. esc:

An expression wrapped in this way remains untouched by the macro expander and is simply inserted verbatim into the output. Therefore, it will be resolved in the macro invocation environment.

This mechanism of using escape sequences, if necessary, can be used to "violate" hygiene rules in order to introduce or manage user variables. For example, the following macro sets x to 0 in the call environment:

This type of variable operation should be used with caution, but in some cases it is quite convenient.

Proper formulation of hygiene rules can be a daunting task. Before using a macro, you may want to consider whether closing the function will be sufficient. Another useful strategy is to postpone as much work as possible until completion. For example, many macros simply wrap their arguments in a QuoteNode or other similar ones. Expr. Examples of this are @task body, which simply returns schedule(Task(() -> $body)), and @eval expr, which simply returns eval(QuoteNode(expr)).

To demonstrate, we could rewrite the @time example above as follows:

However, we don’t do this, and it’s no coincidence: wrapping expr in a new scope block (anonymous function) also slightly changes the meaning of the expression (the scope of any variables in it), while we want @time to be used with minimal impact on the wrapped code.

Macros, as well as Julia functions, are universal. This means that they can also have multiple method definitions due to multiple dispatching.:

However, it should be remembered that multiple dispatch is based on the AST types that are passed to the macro, and not on the types that the AST computes at runtime.:

The Julia base library has an internal function sub2ind' to calculate a linear index in an n-dimensional array based on a set of n multilinear indexes — in other words, to calculate the index `i, which can be used for indexing in array A using A[i] rather than A[x,y,z,...]. One of the possible implementations is the following.

The same can be done using recursion.:

Both of these implementations, although different from each other, do pretty much the same thing: a run-time loop over the dimensions of the array, collecting the offset in each dimension into the final index.

However, all the information we need for the loop is embedded in the information type of the arguments. This allows the compiler to move the iteration to compile time and eliminate run-time cycles altogether. We can use generated functions to achieve the same effect; in compiler terms, we use generated functions to manually collapse the loop. The body becomes almost identical, but instead of calculating a linear index, we create an expression that calculates the index.:

What code will be generated?

An easy way to find out is to extract the body into another (regular) function.:

Now we can execute sub2ind_gen_impl and examine the expression it returns.:

So, the body of the method that will be used here does not include a loop at all - only indexing into two tuples, multiplication and addition or subtraction. All loop execution takes place during compilation, and we completely avoid loop operations during execution. Thus, we run the loop only _ once for each type_, in this case once for each N (except in borderline cases where the function is generated more than once — see disclaimer below).

Generated functions can achieve high efficiency at runtime, but their use is costly at compile time: for each combination of specific argument types, a new function body must be generated. Julia can usually compile "universal" versions of functions that will work with any arguments, but this is not possible with generated functions. This means that static compilation may not be possible for programs that use the generated functions intensively.

To solve this problem, the language provides syntax for writing common, non-genericizable alternative implementations of generated functions. If you apply it to the example with sub2ind above, it will look like this:

Internally, this code creates two implementations of the function: the generated one, which uses the first block in if @generated, and the regular one, which uses the else block. Inside the 'then` part of the 'if @generated` block, the code has the same semantics as other generated functions: argument names refer to types, and the code must return an expression. There may be several if @generated blocks, in which case the generated implementation uses all the then blocks, and the alternative implementation uses all the else blocks.

Note that we have added error checking on top of the function. This code will be common to both versions and is the runtime code in both versions (it will be quoted and returned as an expression from the generated version). This means that the values and types of local variables are not available during code generation — the code for code generation can only see the types of arguments.

In this style, defining a code generation function is mostly an optional optimization. The compiler will use it if it is convenient, but otherwise it can use a normal implementation instead. This style is preferred because it allows the compiler to make additional decisions and compile programs in more ways, and normal code is more readable than code generated using code. However, which implementation will be used depends on the details of the compiler implementation, so it is important that the two implementations behave identically.