Engee documentation

Julia Build (detailed description)

Downloading Julia source code

If you are behind a firewall, you may need to use the https protocol instead of the git protocol.

git config --global url."https://".insteadOf git://

Be sure to configure the appropriate proxy server settings in the system, for example by setting the variables https_proxy and `http_proxy'.

Julia Build

Upon first compilation, the build automatically downloads the pre-built external dependencies. If you prefer to build all dependencies yourself or build on a system without network access during the build process, add the following to the `Make.user' file.

USE_BINARYBUILDER=0

It takes 5 GB to build Julia, if all dependencies are assembled, and about 4 GB of virtual memory.

To perform a parallel build, use make -j N' and specify the maximum number of parallel processes. If the default settings in the build are not suitable for you and you need to set specific parameters (make), you can save them to the `Make.user file and place this file in the root of the Julia source code. The build will automatically verify the existence of the Make.user file and will use it if it exists.

You can create Julia builds outside the tree by specifying make O=<build-directory> configure on the command line. A directory mirror will be created with all the Makefiles needed to build Julia in the specified directory. These builds will share the source code files in Julia and deps/srccache'. Each build directory outside the tree can have its own `Make.user file, overriding the global Make.user file in the top-level folder.

If everything is working correctly, you will see the Julia banner and an interactive window where you can enter calculation expressions. (Library-related errors may be caused by the presence of old, incompatible libraries in the PATH variable. In this case, try moving the julia directory closer to the beginning of the PATH.) Note that most of the instructions above apply to Unix systems.

To launch julia from anywhere, you can:

  • add an alias (to bash: echo "alias julia='/path/to/install/folder/bin/julia'" >> ~/.bashrc && source ~/.bashrc), or

  • add a flexlink to the julia' executable in the `julia directory to /usr/local/bin (or any suitable directory that is already in the path), or

  • add the julia directory to the executable file path for this shell session (in bash': `export PATH="$(pwd):$PATH"; in csh or tcsh: set path= ( cwd )), or

  • permanently add the julia directory to the executable file path (for example, in `.bash_profile'), or

  • write prefix=/path/to/install/folder to the 'Make.user` file, and then run make install'. If the Julia version is already installed in this folder, it should be deleted before running `make install.

Some parameters that can be set to control the Julia build are listed and documented at the beginning of the 'Make.inc` file. But it should never be edited for this purpose. Use Make.user instead.

Makefiles in Julia define convenient automatic print-<VARNAME> rules for outputting variable values, replacing <VARNAME> with the name of the variable whose value needs to be outputted. For example:

$ make print-JULIA_PRECOMPILE
JULIA_PRECOMPILE=1

These rules are useful for debugging.

Now you should be able to run Julia as follows.

julia

If you are building the Julia package for distribution on Linux, macOS, or Windows, read the detailed notes in the file. distributing.md.

Updating an existing source code tree

If you previously downloaded julia using git clone, you can update the existing source tree using git pull instead of starting over.

cd julia
git pull && make

Assuming that you have not made any changes to the source code tree that could conflict with updates from the upstream repository, these commands will run the build update to the latest version.

Troubleshooting common problems

  1. Over time, there may be enough changes in the base library for the boot process to fail when building the system image. If this happens, the build may fail with an error similar to the following.

      *** This error is usually fixed by running 'make clean'. If the error persists, try 'make cleanall' ***

    As already mentioned, it is usually enough to perform `make clean&& make'. Sometimes a stronger cleaning is required, performed using `make cleanall'.

  2. New versions of external dependencies may be introduced, which from time to time may cause conflicts with existing builds of older versions.

    1. There are special make files to help clean the existing build of dependencies. For example, make -C deps clean-llvm will clean the existing build of llvm, so that llvm will be rebuilt from the downloaded source distribution the next time make is called. make -C deps distclean-llvm is a stronger cleanup that will also delete the downloaded source distribution, ensuring that a fresh copy of it will be downloaded and that any new fixes will be applied the next time `make' is called.

    2. To delete the existing julia' binaries and all its dependencies, delete the `./usr directory in the source code tree_.

  3. If you have recently updated macOS, be sure to run the command xcode-select --install to update the command line tools. Otherwise, errors may occur where headers and libraries are missing, such as `ld: library not found for -lcrt1.10.6.o'.

  4. If you have moved the source code directory, errors like CMake Error' may occur.: The current CMakeCache.txt directory ... is different than the directory ... where CMakeCache.txt was created.. In this case, you can delete the conflicting dependency in deps.

  5. In extreme cases, it may be necessary to restore the source code tree to its initial state. The following git commands may be useful.

     git reset –hard #Принудительно удаляет любые изменения в любых файлах в системе управления версиями
     git clean -x -f -d #Принудительно удаляет любые файлы или каталоги вне системы управления версиями

To avoid losing the results of your work, make sure you know what they are doing before executing these commands. git will not be able to undo these changes!_

Platform-specific notes

Notes for different operating systems:

Notes for different architectures:

Necessary build tools and external libraries

To build Julia, you need to install the following software:

In Debian-based distributions (for example, Ubuntu), they can be easily installed using apt-get:

sudo apt-get install build-essential libatomic1 python gfortran perl wget m4 cmake pkg-config curl

Julia uses the following external libraries, which are automatically downloaded (or in some cases included in the Julia source code repository), and then compiled from the source code when you first run make'. The specific version numbers of these libraries that Julia uses are listed in https://github.com/JuliaLang/julia/blob/master/deps /[`deps/$(libname).version]:

  • https://www.llvm.org [LLVM] (15.0 + https://github.com/JuliaLang/llvm-project/tree/julia-release/15.x [patches]) — compiler infrastructure (see note below).

  • https://github.com/JeffBezanson/femtolisp [FemtoLisp] — is bundled with the Julia source code and is used to implement the front-end of the compiler.

  • https://github.com/JuliaLang/libuv [libuv] (custom fork) is a portable, high-performance event-based I/O library.

  • https://github.com/JuliaLang/openlibm [OpenLibm] is a portable libm library containing elementary mathematical functions.

  • DSFMT  — fast Mersenne Twister pseudorandom number generator library.

  • OpenBLAS  — fast, open and supported [basic linear algebra subprograms (BLAS)]

  • LAPACK  — a library of linear algebra routines for solving systems of linear equations, solutions of linear systems of least squares equations, eigenvalue problems, and singular value problems.

  • MKL (optional)  — OpenBLAS and LAPACK can be replaced by Intel’s MKL library.

  • SuiteSparse  — a library of linear algebra routines for sparse matrices.

  • PCRE  — A Perl-compatible regular expression library.

  • GMP  — The GNU multiple precision arithmetic library required to support 'BigInt'.

  • MPFR  — GNU library for multiple precision floating point calculations, necessary to support arbitrary precision floating point ('BigFloat').

  • https://libgit2.org /[libgit2] — the Git library used by the Julia package manager.

  • https://curl.haxx.se [curl] — libcurl provides download and proxy support.

  • https://www.libssh2.org [libssh2] is an SSH transport library used by libgit2 for remote SSH packages.

  • https://tls.mbed.org /[mbedtls] is a library used for cryptography and transport layer security, used by libssh2.

  • utf8proc  — a library for processing Unicode strings encoded in UTF-8.

  • LLVM libunwind  — fork of LLVM from https://www.nongnu.org/libunwind [libunwind], a library that defines a chain of program calls.

  • ITTAPI  — Intel instrumentation and tracing technology and Just-In-Time API.

Build Dependencies

If you already have one or more of these packages installed on your system, you can prevent Julia from compiling duplicates of these libraries by passing USE_SYSTEM_...=1 in `make' or by adding this line to `Make.user'. A complete list of possible flags can be found in `Make.inc'.

Please note that this procedure is not officially supported, as it introduces additional variability in the installation and versioning of dependencies, and is recommended only for system administrators. Unexpected compilation errors may occur because the build system will not perform further checks to ensure that the correct packages are installed.

LLVM

The most difficult dependency is LLVM, for which we need additional patches from developers (LLVM is not backward compatible).

For Julia packaging with LLVM, we recommend either:

  • include the Julia-only LLVM library in the Julia package, or

  • add patches to the LLVM distribution package.

    • The full list of patches is available at https://github.com/JuliaLang/llvm-project [Github] in the julia-release/15.x branch.

    • The only patch specific to Julia is the renaming of the library (llvm7-symver-jlprefix.patch), which should not be applied to the system LLVM.

    • The remaining patches are bug fixes in the source code and have been transferred to the LLVM source code.

Using an uncorrected or different version of LLVM will lead to errors and/or reduced performance. You can compile another version of LLVM from a remote Git repository using the following options in the Make.user file:

# Force source build of LLVM
USE_BINARYBUILDER_LLVM = 0
# Use Git for fetching LLVM source code
# this is either `1` to get all of them
DEPS_GIT = 1
# or a space-separated list of specific dependencies to download with git
DEPS_GIT = llvm

# Other useful options:
#URL of the Git repository you want to obtain LLVM from:
#  LLVM_GIT_URL = ...
#Name of the alternate branch to clone from git
#  LLVM_BRANCH = julia-16.0.6-0
#SHA hash of the alterate commit to check out automatically
#  LLVM_SHA1 = $(LLVM_BRANCH)
#List of LLVM targets to build.  It is strongly recommended to keep at least all the
#default targets listed in `deps/llvm.mk`, even if you don't necessarily need all of them.
#  LLVM_TARGETS = ...
#Use ccache for faster recompilation in case you need to restart a build.
#  USECCACHE = 1
#  CMAKE_GENERATOR=Ninja
#  LLVM_ASSERTIONS=1
#  LLVM_DEBUG=Symbols

The various stages of the build are controlled by specific files.:

  • deps/llvm.version: change to check the new version, make get-llvm check-llvm

  • deps/srccache/llvm/source-extracted: result of executing make extract-llvm

  • deps/llvm/build_Release*/build-configured: result of executing make configure-llvm

  • deps/llvm/build_Release*/build-configured: result of executing make compile-llvm

  • usr-staging/llvm/build_Release*.tgz: result of executing make stage-llvm (re-creation using `make reinstall-llvm')

  • usr/manifest/llvm': result of executing `make install-llvm (re-creation using `make uninstall-llvm')

  • 'make version-check-llvm': runs every time to warn the user about the presence of local changes

Although Julia can be built with newer versions of LLVM, their support should be considered experimental and unsuitable for packaging.

link:@id libuv[libuv]

Julia uses a custom fork of libuv. This is a small dependency, and it can safely be included in the same package as Julia; it will not conflict with the system library. Julia _ builds should not use the libuv system fork.

link:@id BLAS-and-LAPACK[BLAS and LAPACK]

As a high-performance numerical computing language, Julia should be linked to multithreaded BLAS and LAPACK, such as OpenBLAS or ATLAS, which will provide much higher performance than reference implementations of libblas, which may be used by default on some systems.

link:@id Source-distributions-of-releases[Source code distributions for releases]

Each pre-release and Julia release has a complete source code distribution and a lightweight source code distribution.

The full distribution contains the Julia source code and all dependencies, so Julia can be built from the source code without an internet connection. The lightweight distribution does not contain the source code of dependencies.

For example, julia-1.0.0.tar.gz ` — this is a lightweight source code distribution for the release of `v1.0.0 Julia, and `julia-1.0.0-full.tar.gz ` — this is a complete distribution of the source code.

link:@id Building-Julia-from-source-with-a-Git-checkout-of-a-stdlib[Building Julia from the source code using the git checkout library stdlib]

If you need to build Julia from source using git checkout for Pkg, Tar, and Downloads, use make DEPS_GIT=NAME_OF_STDLIB when building.

For example, if you need to build Julia from source using git checkout for Pkg, use make DEPS_GIT=Pkg when building. The 'Pkg` repository is located in stdlib/Pkg and was created initially with HEAD disabled. If you are building from an existing Julia repository, you may need to perform a make clean first.

If you need to build Julia from source using git checkout of several stdlib libraries, DEPS_GIT should be a list of stdlib library names separated by spaces. For example, if you need to build Julia from source using git checkout for Pkg, Tar, and Downloads, use `make DEPS_GIT='Pkg Tar Downloads' when building.

link:@id Building-an-[Building an assembly of Julia statements]

The Julia statement assembly is an assembly that was created using FORCE_ASSERTIONS=1 and LLVM_ASSERTIONS=1'. To build an approval assembly, define both of the following variables in the 'Make.user file.

FORCE_ASSERTIONS=1
LLVM_ASSERTIONS=1

Please note that Julia approval builds will run slower than regular (non-approval) builds.

link:@id Building-32-bit-Julia-on-a-64-bit-machine[Building a 32-bit version of Julia on a 64-bit computer]

Sometimes there may be errors specific to 32-bit architectures. In this case, it will be useful to be able to debug the problem on a local computer. Most modern 64-bit systems support the execution of programs designed for 32-bit systems. Therefore, if you don’t need to recompile Julia from source (for example, you just need to test the 32-bit version of Julia without touching the C code), then most likely you can use the 32-bit Julia build for your system, which can be accessed at https://julialang.org/downloads /[official download files page]. However, if you need to recompile Julia from source, one option is to use a Docker container with a 32-bit system. At least for now, building the 32-bit version of Julia is relatively simple using https://hub.docker.com/r/i386/ubuntu [32-bit Docker images from Ubuntu]. In short, after configuring docker, you need to perform the following steps.

$ docker pull i386/ubuntu
$ docker run –platform i386 -i -t i386/ubuntu /bin/bash

At this stage, the console of a 32-bit computer should be open (note that the uname command tells the architecture of the host computer, so the 64-bit architecture will be output, but this does not affect the Julia build). You can add packages and compilation code; when executing the exit command, all changes will be lost, so you should complete the analysis in one session or configure the copied script to configure the environment.

Next, run the following command:

# apt update

(Please note that the sudo program is not installed, but it is not needed, since you are working with the root user rights and sudo can be omitted in all commands.)

Then add all the build dependencies, a console editor of your choice, the git program, and anything else you might need (for example, gdb, rr, etc.). Select the working directory, run the git clone command for the Julia repository, extract the branch you want to debug, and build Julia normally.

link:@id Update-the-version-number-of-a-dependency[Update of dependency version number]

There are two types of assemblies.

  1. Assembly of all components (deps/ and src/) from the source code. (Add the USE_BINARYBUILDER=0 parameter to the 'Make.user` file; see Julia Build.)

  2. An assembly from the source code ('src/`) with precompiled dependencies (by default).

If you want to update the dependency version number in deps/, use the following checklist.

### Контрольный список
Version numbers:
 - [ ] `deps/$(libname).version`: `LIBNAME_VER`, `LIBNAME_BRANCH`, `LIBNAME_SHA1` and `LIBNAME_JLL_VER`
 - [ ] `stdlib/$(LIBNAME_JLL_NAME)_jll/Project.toml`: `version`

Checksum:
 - [ ] `deps/checksums/$(libname)`
 - [ ] `deps/checksums/$(LIBNAME_JLL_NAME)-*/`: `md5` and `sha512`


Patches:
 - [ ] `deps/$(libname).mk`
 - [ ] `deps/patches/$(libname)-*.patch`

Note.

  • For some dependencies, certain elements of the checklist may be missing.

  • The checksum file can be a separate file without a suffix or a folder containing two files.

Example: OpenLibm

  1. Updating version numbers in deps/openlibm.version

    • OPENLIBM_VER := 0.X.Y

    • OPENLIBM_BRANCH = v0.X.Y

    • OPENLIBM_SHA1 = new-sha1-hash

  2. Updating the version number in stdlib/OpenLibm_jll/Project.toml

    • version = "0.X.Y+0"

  3. Updating checksums in deps/checksums/openlibm

    • make -f contrib/refresh_checksums.mk openlibm

  4. Checking the existence of patch files deps/patches/openlibm-*.patch

    • if there are no corrections, the action is skipped;

    • if there are fixes, check if they have been merged with the new version, which is why they need to be deleted. When deleting the fix, do not forget to change the corresponding Makefile (deps/openlibm.mk).