Uploading the code

This chapter discusses the technical features of downloading packages. To install the packages, use 'Pkg` — Julia’s built-in package manager, which allows you to add packages to the active environment. To use packages already available in the active environment, use the import X or using X operator; see module documentation.

Definitions

Julia has two code loading mechanisms.

Include code, for example include("source.jl")'. Enabling allows you to split the program into multiple source code files. When using the expression `include("source.jl"), the contents of the file source.jl are calculated in the global scope of the module in which the include call occurs. If include("source.jl") is called multiple times, the source.jl file is calculated the same number of times. The inclusion path source.jl is interpreted relative to the file in which the include call occurs. This makes it easy to move a subtree of source code files. In REPL, the inclusion paths are interpreted relative to the current working directory. pwd().
Package download, for example import X or using X'. The import mechanism allows you to download a package — an independent, reusable Julia code block enclosed in a module — and make this module available by the name `X inside the importing module. If the same package X is imported several times during the Julia session, it is loaded only the first time — later, the importing module receives a link to it. However, keep in mind that the import X operator can load different packages in different contexts: in the main project, X can refer to one package named X, and in dependencies, to other packages also named `X'. For more information, see below.

The inclusion of the code is very simple: the specified source code file is calculated in the context of the calling module. Downloading packages is based on the inclusion of the code, but it serves another goal. The rest of this chapter discusses how package downloads work.

The pack is a source code tree with a standard structure that provides functions that can be used in other Julia projects. The package is loaded using the import X or using X operator. They make the package named X available in the module where the import statement was called. The meaning of X in the expression import X depends on the context: the downloaded package X depends on the code in which the operator is used. Thus, the processing of the expression import X takes place in two stages: first, it is determined which package is the package X in this context, and then where this package X is located.

To answer these questions, a search is performed in the project environments listed in the constant LOAD_PATH' for project files (`Project.toml or JuliaProject.toml), manifest files (Manifest.toml or JuliaManifest.toml or the same names with the suffix -v{major}.{minor}.toml for certain versions) or folders with source code files.

Federation of Packages

As a rule, a package is uniquely identified by its name. However, sometimes a project may need to use two different packages with the same names. Although you can simply rename one of the packages to do this, such an action can cause serious problems in a large shared code base. Therefore, the Julia code loading mechanism allows you to use the same name to refer to different packages in different parts of the application.

Julia supports federated package management. This means that different independent parties can maintain their own registries of public and private packages, and projects can use a combination of public and private packages from different registries. A common set of tools and workflows are used to install and manage packages from different registries. The 'Pkg` package manager that comes with Julia allows you to install and manage project dependencies. It allows you to create project files (describing other projects on which your project depends) and manifest files (which contain versions of the complete dependency graph of your project), as well as work with them.

One of the consequences of federation is the lack of a centralized package naming system. Different objects can use the same name to refer to unrelated packages. This is unavoidable, as the objects do not coordinate their actions and may not even know about each other’s existence. Due to the lack of a centralized naming system, different packages with the same name may end up being used in the same project. The package loading mechanism in Julia does not require package names to be globally unique, even within the dependency graph of a single project. Instead, packages are defined by https://en.wikipedia.org/wiki/Universally_unique_identifier [universal unique identifiers] (UUIDs) that are assigned to each package when it is created. It is usually not necessary to work directly with these rather cumbersome 128-bit identifiers, since Pkg takes over the task of generating and tracking them. However, thanks to the UUID, it is possible to get an unambiguous answer to the question — which package is X referring to?

Since the problem of decentralized naming is quite abstract, it may be useful to analyze a specific situation to understand it. Suppose you are developing an application called App, which uses two packages: Pub and Priv'. `Priv is a private package that you created yourself, and Pub is a public package that you use but do not control. When you created the Priv package, there was no publicly available package named Priv. However, later on, someone created a completely different Priv package, which was published and became popular. Moreover, it has started to be used in the Pub package. So the next time you update Pub to get the latest features and fix bugs, App will end up using two different packages named Priv, and that’s just as a result of the update. The App' application has a direct dependency on your private `Priv package and an indirect dependency, via Pub, on the new public Priv package. Since these Priv packages are different, but both are required for the proper operation of the App, the expression import Priv should refer to different Priv packages depending on where it is applied: in the App code or in the Pub code. To do this, the package loading mechanism in Julia distinguishes between two Priv packages by their UUIDs and selects the appropriate one depending on the context (of the module that invoked the 'import` statement). How this happens depends on the environment and is described in the following sections.

Wednesday

The environment defines what the import X and using X operators mean in different code contexts and which files they load. There are two types of environments in Julia.

The project environment is a directory with a project file and possibly a manifest file. They form an implicit environment. The project file defines the names and identifiers of the direct dependencies of the project. The manifest file, if available, describes the complete dependency graph, including all direct and indirect dependencies, the exact versions of each dependency, and the information needed to find and download the correct version.
Package Directory contains nested directories with source code trees of a set of packages. It forms an implicit environment. If 'X` is a nested directory in the package directory and the file X/src/X.jl exists, then the package X is available in the package directory environment, and X/src/X.jl is the source code file through which it is downloaded.

These environments can be used in combination to create a multi-layered environment: an ordered set of project environments and package catalogs that overlap to form a single composite environment. In this case, the priority and visibility rules that determine which packages are available and from where they are downloaded are combined. For example, the Julia download path forms a multi-layered environment.

Each of these environments serves its own purpose.

Project environments provide reproducibility. By adding the project environment to a version control system, such as the Git repository, along with the rest of the project’s source code, you can reproduce the exact state of the project and all its dependencies. In particular, the manifest file contains the exact version of each dependency, which is identified by a cryptographic hash of the source code tree. This allows the Pkg manager to get the exact versions and download exactly the right code for each dependency.
Package catalogs provide convenience when a carefully monitored project environment is required. They are useful if you need to place a set of packages somewhere and use them directly without creating a project environment for them.
Multi-layered environments allow you to add tools to the main environment. You can put the development tools environment at the end of the stack so that they are accessible from REPL and scripts, but not from packages.

At the most general level, each environment defines three schemas: roots, graphs, and paths. When determining the meaning of the expression import X, the package X is identified using root diagrams and a graph, and the location of its source code is determined using the path diagram. The purpose of each of the three schemes is described in detail below.

roots (roots): name::Symbol ⟶ uuid::UUID

The environment root schema assigns package names to the UUIDs of all top-level dependencies that the environment makes available to the main project (that is, which can be loaded into the Main). When Julia sees the expression import X in the main project, she identifies X as roots[:X].
graph (graph): context::UUID ⟶ name::Symbol ⟶ uuid::UUID

The environment graph is a multi-level schema that assigns a name mapping to a UUID to each UUID in the context. It works the same way as the root schema, but within a given context (context'). When Julia encounters the expression `import X in the package code with the UUID context, she identifies X as graph[context][:X]. In particular, this means that the expression import X can refer to different packages depending on the context.
paths (paths): uuid::UUID × name::Symbol ⟶ path::String

The path scheme assigns to each pair of UUIDs and package names a location where the source code file with the entry point to this package is located. After the X element in the import X expression has been resolved into a UUID by means of a root schema or graph (depending on whether the download is from the main project or from a dependency), Julia determines which file to download in order to get X by searching for paths[uuid,:X] in the environment. When you include this file, a module named X is detected. After downloading the package, all subsequent import operations that resolve to the same uuid will create a binding to the already downloaded package module.

For each type of environment, these three schemes are defined differently, as described in the following sections.

For clarity, the examples in this chapter provide complete data structures for root, graph, and path diagrams. However, the Julia package download code does not explicitly create them. Instead, it "lazily" calculates only that part of each structure that is sufficient to load a given package.

Project Environments

The project environment consists of a directory containing a project file named Project.toml and an optional manifest file named Manifest.toml'. These files can also be named `JuliaProject.toml and JuliaManifest.toml; in this case, the files Project.toml and Manifest.toml are ignored. This allows you to simultaneously use tools that rely on files named Project.toml and Manifest.toml'. However, for projects exclusively on Julia, the names `Project.toml and Manifest.toml are preferable. However, starting with Julia version 1.11, (Julia)Manifest-v{major}.{minor}.toml is recognized as a format in which a specific manifest file should be used in a certain version of Julia, i.e. in the same folder Manifest-v1.11.toml will be used by version 1.11, and Manifest.toml - by any other version of Julia.

The schemes of the roots, graph, and paths of the project environment are defined as follows.

The root schema of the environment is determined by the contents of the project file, in particular its top-level name and uuid elements and the [deps] section (all of them are optional). Consider the following example of a project file for the fictional app described above.

name = "App"
uuid = "8f986787-14fe-4607-ba5d-fbff2944afa9"

[deps]
Priv = "ba13f791-ae1d-465a-978b-69c3ad90f72b"
Pub  = "c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1"

This project file assumes the following root scheme, presented as a Julia dictionary.

roots = Dict(
    :App  => UUID("8f986787-14fe-4607-ba5d-fbff2944afa9"),
    :Priv => UUID("ba13f791-ae1d-465a-978b-69c3ad90f72b"),
    :Pub  => UUID("c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1"),
)

With this root scheme, the import Priv operator in the App code instructs Julia to search for roots[:Priv], which will result in the value ba13f791-ae1d-465a-978b-69c3ad90f72b — the UUID of the Priv package to be loaded in this context. The UUID defines which package Priv should be downloaded and used when calculating the import Priv statement in the main application.

The dependency graph of the project environment is determined by the contents of the manifest file, if any. If there is no manifest file, the graph is empty. The manifest file has a separate section for each direct or indirect dependency of the project. For each dependency, the UUID of the package and the hash of the source code tree or the explicit path to the source code are specified in the file. Consider the following example of a manifest file for the App application:

[[Priv]] # частный
deps = ["Pub", "Zebra"]
uuid = "ba13f791-ae1d-465a-978b-69c3ad90f72b"
path = "deps/Priv"

[[Priv]] # общедоступный
uuid = "2d15fe94-a1f7-436c-a4d8-07a9a496e01c"
git-tree-sha1 = "1bf63d3be994fe83456a03b874b409cfd59a6373"
version = "0.1.5"

[[Pub]]
uuid = "c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1"
git-tree-sha1 = "9ebd50e2b0dd1e110e842df3b433cb5869b0dd38"
version = "2.1.4"

  [Pub.deps]
  Priv = "2d15fe94-a1f7-436c-a4d8-07a9a496e01c"
  Zebra = "f7a24cb4-21fc-4002-ac70-f0e3a0dd3f62"

[[Zebra]]
uuid = "f7a24cb4-21fc-4002-ac70-f0e3a0dd3f62"
git-tree-sha1 = "e808e36a5d7173974b90a15a353b564f3494092f"
version = "3.4.2"

This manifest file fully describes the possible dependency graph for the `App' project.

The application uses two different packages named Priv: a private one, which is the root dependency, and a public one, which is an indirect dependency via Pub. They differ in their unique UUIDs and have different dependencies.
- The private package Priv' depends on the packages `Pub and `Zebra'.
- The public package `Priv' has no dependencies.
The application also depends on the Pub package, which in turn depends on the public Priv package and on the same Zebra' package that the private `Priv package depends on.

The dependency graph, represented as a dictionary, looks like this:

graph = Dict(
    # Частный пакет Priv:
    UUID("ba13f791-ae1d-465a-978b-69c3ad90f72b") => Dict(
        :Pub   => UUID("c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1"),
        :Zebra => UUID("f7a24cb4-21fc-4002-ac70-f0e3a0dd3f62"),
    ),
    # Общедоступный пакет Priv:
    UUID("2d15fe94-a1f7-436c-a4d8-07a9a496e01c") => Dict(),
    # Pub:
    UUID("c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1") => Dict(
        :Priv  => UUID("2d15fe94-a1f7-436c-a4d8-07a9a496e01c"),
        :Zebra => UUID("f7a24cb4-21fc-4002-ac70-f0e3a0dd3f62"),
    ),
    # Zebra:
    UUID("f7a24cb4-21fc-4002-ac70-f0e3a0dd3f62") => Dict(),
)

With this dependency graph, when Julia encounters the expression import Priv in the Pub package with the UUID c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1, the following search is performed:

graph[UUID("c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1")][:Priv]

As a result, the value 2d15fe94-a1f7-436c-a4d8-07a9a496e01c is returned, which means that in the context of the Pub package, the expression import Priv refers to the public package Priv, and not to the private one, on which the application depends directly. This is how the name Priv in the main project can refer to packages other than those that this name indicates in the dependencies, which makes identical names possible in the package system.

What happens when calculating the expression import Zebra in the main code of the App'? Since the Zebra package is not listed in the project file, the import will fail despite the fact that Zebra is in the manifest file. Moreover, if the 'import Zebra statement had been used in the public package Priv with the UUID 2d15fe94-a1f7-436c-a4d8-07a9a496e01c, it would also have failed, since this package Priv has no declared dependencies in the manifest file and therefore cannot download other packages. The Zebra package can only be loaded by packages for which it is explicitly specified as a dependency in the manifest file, namely the Pub package and one of the Priv packages.

The Path diagram of the project environment is extracted from the manifest file. The path to the package with the identifier uuid and the name X is determined according to the following rules (in the specified order).

If the project file in the directory matches the identifier uuid and the name X, then the following happens.
- If it has a top-level path element, then the uuid is mapped to that path, which is interpreted relative to the directory containing the project file.
- Otherwise, the uuid is mapped to the path `src/X.jl' relative to the directory containing the project file.
If the above conditions are not met, but there is a manifest file for the project file that contains a section corresponding to the uuid identifier, the following happens.
- If it has a path element, this path is used (relative to the directory containing the manifest file).
- If it contains the git-tree-sha1 element, a deterministic hash function is calculated for the uuid and the git-tree-sha1 value, taking into account its result (let’s call this value slug) and the packages directory is searched in each directory in the global Julia DEPOT_PATH array./X/$slug`. The first such found directory will be used.

If any of these conditions are met, the path to the source code entry point will be either the result obtained, or the path relative to that result with the addition of src/X.jl'. Otherwise, there is no path corresponding to the `uuid identifier. If the path to the source code cannot be found when downloading package X, the search fails and the user may be prompted to install the appropriate version of the package or fix the situation in another way (for example, declare X as a dependency).

In the example of the manifest file above, the path to the first Priv package (with the UUID ba13f791-ae1d-465a-978b-69c3ad90f72b) is defined as follows: Julia searches for a section of this package in the manifest file, finds the path element, searches for the deps/Priv path relative to the App project directory (let’s say that the App code is located at the path /home/me/projects/App), finds the directory /home/me/projects/App/deps/Priv and therefore downloads the Priv package from it.

If it were necessary to download another package Priv' (with the UUID `2d15fe94-a1f7-436c-a4d8-07a9a496e01c'), the Julia environment would find its section in the manifest, which, however, does not contain the `path element, but contains the git-tree-sha1 element. Therefore, the slug value for this pair of UUIDs and the SHA-1 hash would be calculated and the HDkrT result would be obtained (it doesn’t matter how this calculation is performed, but the function is consistent and deterministic). This means that the path to the Priv package will be packages/Priv/HDkrT/src/Priv'.jl in one of the package repositories. Let’s say the 'DEPOT_PATH` variable contains the value ["/home/me/.julia", "/usr/local/julia"]. In this case, Julia checks if the following paths exist:

/home/me/.julia/packages/Priv/HDkrT
/usr/local/julia/packages/Priv/HDkrT

Julia then tries to download the public package Priv from the file packages/Priv/HDKrT/src/Priv'.jl in the repository where the first of these paths is found.

Below is a possible path diagram for an example of the App project environment as it is presented in the manifest for the dependency graph above, after searching the local file system.

paths = Dict(
    # Частный пакет Priv:
    (UUID("ba13f791-ae1d-465a-978b-69c3ad90f72b"), :Priv) =>
        # относительная точка входа внутри репозитория `App`:
        "/home/me/projects/App/deps/Priv/src/Priv.jl",
    # Общедоступный пакет Priv:
    (UUID("2d15fe94-a1f7-436c-a4d8-07a9a496e01c"), :Priv) =>
        # пакет, установленный в системном хранилище:
        "/usr/local/julia/packages/Priv/HDkr/src/Priv.jl",
    # Pub:
    (UUID("c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1"), :Pub) =>
        # пакет, установленный в пользовательском хранилище:
        "/home/me/.julia/packages/Pub/oKpw/src/Pub.jl",
    # Zebra:
    (UUID("f7a24cb4-21fc-4002-ac70-f0e3a0dd3f62"), :Zebra) =>
        # пакет, установленный в системном хранилище:
        "/usr/local/julia/packages/Zebra/me9k/src/Zebra.jl",
)

In the example diagram, there are three different types of package locations (the first and third locations are part of the default download path).

"https://stackoverflow.com/a/35109534 A [proprietary copy of] the "private package Priv is hosted in the `App' repository.
The public packages Priv and `Zebra' are located in the system storage, where the packages are installed and managed by the system administrator. They are available to all users of the system.
The Pub package is located in the user storage, where the packages are installed by the active user. They are available only to the user who installed them.

Package Catalogs

Package directories are a simpler type of environment that does not allow name conflicts to be resolved. In a package directory, a set of top-level packages is a set of nested directories that "look" like packages. Package X exists in the package directory if one of the following entry point files exists in that directory:

X.jl
X/src/X.jl
X.jl/src/X.jl

Which dependencies a package can import in the package directory is determined by the presence of the project file in the package.

If there is a project file, you can import only the packages specified in its section [deps].
If there is no project file, you can import any top-level packages, that is, the same packages that can be loaded in the Main module or in the REPL.

The root schema is determined by analyzing the contents of the project catalog and creating a list of all available packages. In addition, each package found in the X folder is assigned a UUID as follows.

If there is an X/Project.toml' file and it contains a `uuid element, then this uuid element will be the identifier value.
If the file X/Project.toml exists, but there is no top-level UUID element in it, the dummy identifier uuid is generated by hashing the canonical (real) path to `X/Project.toml'.
Otherwise (if there is no Project.toml file), the uuid identifier will consist of https://en.wikipedia.org/wiki/Universally_unique_identifier#Nil_UUID [just zeros].

The dependency graph of the package catalog is determined by the presence and contents of the project file in the subdirectory of each package. The following rules apply.

If there is no project file in the package’s subdirectory, the package is not included in the graph, and import operators in its code are treated as top-level operators, both in the main project and in the REPL.
If there is a project file in the package’s attached directory, then the schema is used as an element of the graph for its UUID [deps] from the project file. If this section is missing, the element is considered empty.

Suppose the package directory has the following structure and contents:

Aardvark/
    src/Aardvark.jl:
        import Bobcat
        import Cobra

Bobcat/
    Project.toml:
        [deps]
        Cobra = "4725e24d-f727-424b-bca0-c4307a3456fa"
        Dingo = "7a7925be-828c-4418-bbeb-bac8dfc843bc"

    src/Bobcat.jl:
        import Cobra
        import Dingo

Cobra/
    Project.toml:
        uuid = "4725e24d-f727-424b-bca0-c4307a3456fa"
        [deps]
        Dingo = "7a7925be-828c-4418-bbeb-bac8dfc843bc"

    src/Cobra.jl:
        import Dingo

Dingo/
    Project.toml:
        uuid = "7a7925be-828c-4418-bbeb-bac8dfc843bc"

    src/Dingo.jl:
# there are no imports

In the form of a dictionary, the corresponding root structure will be presented as follows:

roots = Dict(
    :Aardvark => UUID("00000000-0000-0000-0000-000000000000"), # файла проекта нет, нулевой UUID
    :Bobcat   => UUID("85ad11c7-31f6-5d08-84db-0a4914d4cadf"), # фиктивный UUID на основе пути
    :Cobra    => UUID("4725e24d-f727-424b-bca0-c4307a3456fa"), # UUID из файла проекта
    :Dingo    => UUID("7a7925be-828c-4418-bbeb-bac8dfc843bc"), # UUID из файла проекта
)

In the form of a dictionary, the corresponding graph structure will be represented as follows:

graph = Dict(
    # Bobcat:
    UUID("85ad11c7-31f6-5d08-84db-0a4914d4cadf") => Dict(
        :Cobra => UUID("4725e24d-f727-424b-bca0-c4307a3456fa"),
        :Dingo => UUID("7a7925be-828c-4418-bbeb-bac8dfc843bc"),
    ),
    # Cobra:
    UUID("4725e24d-f727-424b-bca0-c4307a3456fa") => Dict(
        :Dingo => UUID("7a7925be-828c-4418-bbeb-bac8dfc843bc"),
    ),
    # Dingo:
    UUID("7a7925be-828c-4418-bbeb-bac8dfc843bc") => Dict(),
)

Pay attention to a number of important rules.

A package without a project file can depend on any top-level package, and since all packages in the package directory are available at the top level, it can import all packages in the environment.
A package with a project file cannot depend on a package without such a file, since packages with project files can only load packages available in the graph schema, and there are no packages without project files in it.
Only packages without project files can depend on a package with a project file, but without an explicitly specified UUID, since the dummy UUIDs assigned to such packages are intended solely for internal use.

Let’s see how these rules are applied in practice in our example.

'Aardvark` can import any of the Bobcat, Cobra or 'Dingo` packages; in fact, Bobcat and Cobra are imported.
'Bobcat` can and does import packages Cobra and Dingo, which do not have project files with UUIDs and which are declared as dependencies in the [deps] of the `Bobcat' package.
'Bobcat` may depend on Aardvark, as the Aardvark package does not have a project file.
'Cobra` can and does import the Dingo package, which has a project file and a UUID and is declared as a dependency in the [deps] of the 'Cobra` package.
'Cobra` cannot depend on the Aardvark or `Bobcat' package, as they do not have real UUIDs.
The 'Dingo` package cannot import anything, as it has a project file, but without a section [deps].

The path scheme in the package directory is very simple: in it, the names of the nested directories are mapped to the paths of the entry points. In other words, if the path to the project directory in our example looked like /home/me/animals, then the paths schema could be represented by such a dictionary:

paths = Dict(
    (UUID("00000000-0000-0000-0000-000000000000"), :Aardvark) =>
        "/home/me/AnimalPackages/Aardvark/src/Aardvark.jl",
    (UUID("85ad11c7-31f6-5d08-84db-0a4914d4cadf"), :Bobcat) =>
        "/home/me/AnimalPackages/Bobcat/src/Bobcat.jl",
    (UUID("4725e24d-f727-424b-bca0-c4307a3456fa"), :Cobra) =>
        "/home/me/AnimalPackages/Cobra/src/Cobra.jl",
    (UUID("7a7925be-828c-4418-bbeb-bac8dfc843bc"), :Dingo) =>
        "/home/me/AnimalPackages/Dingo/src/Dingo.jl",
)

Since all packages in the package catalog environment are, by definition, nested directories with corresponding entry point files, the elements of their paths schema always have this form.

Media Stacks

The third and final type of environment is one in which other types of environments overlap each other, forming a single environment where the packages of each of the component environments are available. Such composite media are called media stacks. The Julia global variable LOAD_PATH defines the stack of environments in which the Julia process is running. In order for the Julia process to have access to packages from only one project or package directory, do not specify other environments in the LOAD_PATH. However, it is often useful to have access to some tools - standard libraries, profilers, debuggers, auxiliary programs, etc. - even if they are not dependencies of the current project. If you add an environment with these tools to the download path, they will immediately be available in the top-level code, so you do not need to add them to the project.

The data structures of the roots, graph, and paths belonging to the components of the media stack are combined very simply as dictionaries, and in the case of a key match, priority is given to earlier entries. In other words, if there is a stack stack = [env₁, env₂, …], then the following happens.

roots = reduce(merge, reverse([roots₁, roots₂, …]))
graph = reduce(merge, reverse([graph₁, graph₂, …]))
paths = reduce(merge, reverse([paths₁, paths₂, …]))

The variables rootsᵢ, graphᵢ and pathsᵢ with subscripts correspond to the environments envᵢ with subscripts included in the stack. The reverse function is used for the reason that when keys match in dictionaries passed as arguments to the merge function, priority is given to the last argument rather than the first. This approach has several noteworthy features.

The main environment, that is, the first environment in the stack, is fully integrated into the multilayer environment. The dependency graph of the first environment in the stack is guaranteed to be included in the layered environment completely and unchanged, including with the same versions of all dependencies.
Packages from non-core environments may end up using incompatible dependency versions, even if their own environments are fully compatible. This can happen if one of their dependencies is replaced by a version from an earlier environment in the stack (according to graph, path, or both).

Since the main environment is usually the environment of the current project, and later environments in the stack contain additional tools, this compromise is justified: it is better that the development tools stop working than the entire project. As a rule, if such incompatibilities occur, it is necessary to update the development tools to versions compatible with the main project.

Package Extensions

A package extension is a module that is automatically loaded when a certain set of other packages (its "triggers") are loaded in the current Julia session. Extensions are defined in the project file in the [extensions] section. Extension triggers are a subset of other packages listed in the [weakdeps] section (and possibly, but rarely, in the [deps]) of the project file. These packages may have compat entries, just like other packages.

name = "MyPackage"

[compat]
ExtDep = "1.0"
OtherExtDep = "1.0"

[weakdeps]
ExtDep = "c9a23..." # uuid
OtherExtDep = "862e..." # uuid

[extensions]
BarExt = ["ExtDep", "OtherExtDep"]
FooExt = "ExtDep"
...

The keys in the extensions' section are the names of extensions. They are loaded when all packages listed on the right side (triggers) are loaded. If the extension has only one trigger, the list of triggers can be written as a string for brevity. The entry point for the `FooExt extension is in ext/FooExt.jl or `ext/FooExt/FooExt.jl'. The contents of an extension often have this structure:

module FooExt

# Загрузка главного пакета и триггеров
using MyPackage, ExtDep

# Расширение функциональности главного пакета посредством типов из триггеров
MyPackage.func(x::ExtDep.SomeStruct) = ...

end

When an extension package is added to the environment, the weakdeps and extensions sections are saved in the manifest file in the section for that package. The dependency search rules for a package are the same as for its "parent", except that triggers in the list are also considered dependencies.

Package and environment preferences

Preferences are dictionaries of metadata that affect the behavior of packages in the environment. The preference system supports reading preferences at compile time. This means that before downloading the code, you need to make sure that the pre-compilation files selected by Julia were built with the preferences of the current environment. The public API for changing preferences is contained in the package https://github.com/JuliaPackaging/Preferences.jl [Preferences.jl]. Preferences are stored as TOML dictionaries in the (Julia)LocalPreferences.toml file next to the currently active project. When exporting, the preference is transferred to the file (Julia)Project.toml. The idea is that shared projects can have common preferences, but users can override them using their own settings in the LocalPreferences.toml file, which, as its name implies, should be added to the file.gitignore.

The preferences accessed during compilation are automatically marked as compile-time preferences. Any change to them will force the Julia compiler to recompile all cached pre-compilation files (.ji and corresponding files .so, .dll or .dylib) for this module. To do this, during compilation, the hash of all compile-time preferences is serialized, and then, when searching for the necessary files, it is checked for compliance with the current environment.

Default values can be set for the preferences at the storage level; if the Foo package is installed in the global environment and has the specified preferences, they are applied provided that the global environment is included in the LOAD_PATH. Preferences related to environments higher in the environment stack are overridden by lower-level entries in the download path up to the currently active project. This allows default preferences to exist at the project level, and when inheriting for active projects, they can be combined or even completely overwritten. For more information on how to allow or disable combining preferences, see the docstring of the Preferences.set_preferences!() function.

Conclusion

Federated package management and accurate application reproducibility are difficult to implement, but important mechanisms. Together, they complicate the package loading system compared to most dynamic languages, but at the same time provide the scalability and reproducibility typically associated with static languages. As a rule, when using the built-in Julia package manager for project management, you do not need to know exactly how these mechanisms work. Calling Pkg.add("X")`adds the package `X to the corresponding project and manifest files selected using Pkg.activate("Y"), so that on further calls to import X it loads without additional effort.