Uploading the code

Note This chapter discusses the technical features of downloading packages. To install the packages, use Pkg — Built-in Julia Package Manager, which allows you to add packages to the active environment. To use packages already available in the active environment, use the operator import X or using X; see module documentation.

Definitions

Julia has two code loading mechanisms.

Code inclusion, for example include("source.jl"). Enabling allows you to split the program into multiple source code files. When using the expression include("source.jl") file contents source.jl it is calculated in the global scope of the module in which the call occurs. include. If include("source.jl") it is called several times, and the file is calculated the same number of times. source.jl. The activation path source.jl interpreted relative to the file in which the call occurs include. This makes it easy to move a subtree of source code files. In REPL, the inclusion paths are interpreted relative to the current working directory. pwd().
Package download, for example import X or using X. The import mechanism allows you to download a package — an independent, reusable Julia code block enclosed in a module — and make that module available by name. X inside the importing module. If there is the same package during the Julia session X It is imported several times, and it is loaded only the first time. Later, the importing module receives a link to it. However, keep in mind that the operator import X It can load different packages in different contexts: in the main project X it can refer to a single package with the name X, and in dependencies - to other packages, also with the name X. For more information, see below.

The inclusion of the code is very simple: the specified source code file is calculated in the context of the calling module. Downloading packages is based on the inclusion of the code, but it serves another goal. The rest of this chapter discusses how package downloads work.

The pack is a source code tree with a standard structure that provides functions that can be used in other Julia projects. The package is loaded using the operator import X or using X. They make a package with the name X available in the module where the import statement was called. Meaning X in the expression import X depends on the context: the package being downloaded X depends on the code that uses the operator. Thus, the processing of the expression import X It occurs in two stages: first, it is determined which package is a package. X in this context, and then where is this package located X.

To answer these questions, a search is performed in the project environments listed in the constant LOAD_PATH for project files (Project.toml or JuliaProject.toml), manifest files (Manifest.toml or JuliaManifest.toml or the same names with a suffix -v{major}.{minor}.toml for certain versions) or folders with source code files.

Federation of Packages

As a rule, a package is uniquely identified by its name. However, sometimes a project may need to use two different packages with the same names. Although you can simply rename one of the packages to do this, such an action can cause serious problems in a large shared code base. Therefore, the Julia code loading mechanism allows you to use the same name to refer to different packages in different parts of the application.

Julia supports federated package management. This means that different independent parties can maintain their own registries of public and private packages, and projects can use a combination of public and private packages from different registries. A common set of tools and workflows is used to install and manage packages from different registries. Package Manager Pkg, which comes with Julia, allows you to install and manage project dependencies. It allows you to create project files (describing other projects on which your project depends) and manifest files (which contain versions of the full dependency graph of your project), as well as work with them.

One of the consequences of federation is the lack of a centralized package naming system. Different objects can use the same name to refer to unrelated packages. This is unavoidable, as the objects do not coordinate their actions and may not even know about each other’s existence. Due to the lack of a centralized naming system, different packages with the same name may end up being used in the same project. The package loading mechanism in Julia does not require package names to be globally unique, even within the dependency graph of a single project. Instead, packages are defined by https://en.wikipedia.org/wiki/Universally_unique_identifier [universal unique identifiers] (UUIDs) that are assigned to each package when it is created. It is usually not necessary to work directly with these rather cumbersome 128-bit identifiers, since it takes over the task of generating and tracking them. Pkg. However, thanks to the UUID, you can get an unambiguous answer to the question — which package is referring to X?

Since the problem of decentralized naming is quite abstract, it may be useful to analyze a specific situation to understand it. Let’s say you are developing an application called App, which uses two packages: Pub and Priv. Priv — this is a private package that you created yourself, and Pub — a public package that you use but do not control. When did you create the package Priv, a publicly available package named Priv there was no. However, later someone created a completely different package. Priv, which was published and became popular. Moreover, it has started to be used in the package Pub. So the next time you update Pub to get the latest features and fix bugs, in App as a result, two different packages with the name will be used. Priv, and this is only as a result of the update. Application App it is directly dependent on your private package Priv and indirect dependence, through Pub, from the new public package Priv. Since these packages Priv they are different, but both are required for proper operation. App, expression import Priv it should refer to different packages. Priv depending on where it is applied: in the code App or in the code Pub. To do this, the package loading mechanism in Julia distinguishes between two packages Priv by UUID IDs and selects the appropriate one depending on the context (of the module that invoked the statement `import`How this happens depends on the environment and is described in the following sections.

Wednesday

The environment defines what the operators mean import X and using X in different code contexts and which files they load. There are two types of environments in Julia.

The project environment is a directory with a project file and possibly a manifest file. They form an implicit environment. The project file defines the names and identifiers of the direct dependencies of the project. The manifest file, if available, describes the complete dependency graph, including all direct and indirect dependencies, the exact versions of each dependency, and the information needed to find and download the correct version.
Package Directory contains nested directories with source code trees of a set of packages. It forms an implicit environment. If X — this is a nested directory in the package directory and there is a file X/src/X.jl So, the package X available in the package catalog environment, and X/src/X.jl — this is the source code file through which it is downloaded.

These environments can be used in combination to create a multi-layered environment: an ordered set of project environments and package catalogs that overlap to form a single composite environment. In this case, the priority and visibility rules that determine which packages are available and from where they are downloaded are combined. For example, the Julia download path forms a multi-layered environment.

Each of these environments serves its own purpose.

Project environments provide reproducibility. By adding the project environment to a version control system, such as the Git repository, along with the rest of the project’s source code, you can reproduce the exact state of the project and all its dependencies. In particular, the manifest file contains the exact version of each dependency, which is identified by a cryptographic hash of the source code tree. This allows the dispatcher to Pkg get the exact versions and download exactly the right code for each dependency.
Package catalogs provide convenience when a carefully monitored project environment is required. They are useful if you need to place a set of packages somewhere and use them directly without creating a project environment for them.
Multi-layered environments allow you to add tools to the main environment. You can put the development tools environment at the end of the stack so that they are accessible from REPL and scripts, but not from packages.

Note When loading a package from another environment on the stack other than the active environment, the package is loaded in the context of the active environment. This means that the package will be loaded as if it were imported into the active environment, which may affect the resolution of versions of its dependencies. During pre-compilation, such a package will be marked as a pre-compilation job. (serial), which means that its dependencies will be precompiled sequentially within a single job, which is likely to be slower.

At the most general level, each environment defines three schemas: roots, graphs, and paths. When determining the value of an expression import X The package is identified using root schemes and a graph. X, and using the path diagram, the location of its source code is determined. The purpose of each of the three schemes is described in detail below.

roots (roots): name::Symbol ⟶ uuid::UUID The environment root schema assigns package names to the UUIDs of all top-level dependencies that the environment makes available to the main project (that is, which can be uploaded to Main). When Julia meets the expression import X in the main project, it identifies X how roots[:X].
graph (graph): context::UUID ⟶ name::Symbol ⟶ uuid::UUID The environment graph is a multi-level schema that assigns each UUID in a context (context) mapping names to UUIDs. It works the same way as the root scheme, but within the given context (context). When Julia meets the expression import X in the package code with the UUID context, it identifies X how graph[context][:X]. In particular, this means that the expression import X it can refer to different packages depending on context.
paths (paths): uuid::UUID × name::Symbol ⟶ path::String The path scheme assigns to each pair of UUIDs and package names a location where the source code file with the entry point to this package is located. After the element X in the expression import X It has been resolved into a UUID by means of a root scheme or graph (depending on whether the download is from the main project or from a dependency), Julia determines which file needs to be downloaded in order to receive X, by searching for paths[uuid,:X] in the environment. When you include this file, a module named X. After downloading the package, all subsequent import operations that resolve to the same identifier uuid, will lead to the creation of a binding to an already downloaded package module.

For each type of environment, these three schemes are defined differently, as described in the following sections.

Note For clarity, the examples in this chapter provide complete data structures for root, graph, and path schemas. However, the Julia package download code does not explicitly create them. Instead, it "lazily" calculates only that part of each structure that is sufficient to load a given package.

Project Environments

The project environment is formed from a directory containing a project file named Project.toml, and an optional manifest file named Manifest.toml. These files can also have names JuliaProject.toml and JuliaManifest.toml; in this case, the files Project.toml and Manifest.toml they are ignored. This allows you to use tools that rely on named files at the same time. Project.toml and Manifest.toml. However, for projects exclusively on Julia, the names are Project.toml and Manifest.toml preferably. However, starting with Julia version 1.10.8, (Julia)Manifest-v{major}.{minor}.toml It is recognized as a format in which a specific manifest file should be used in a certain version of Julia, i.e. in the same folder. Manifest-v1.11.toml it will be used by version 1.11, and Manifest.toml — any other version of Julia.

The schemes of the roots, graph, and paths of the project environment are defined as follows.

The Root schema of the environment is determined by the contents of the project file, in particular its elements name and uuid top-level and section [deps] (all of them are optional). Consider the following example of a project file for a fictional application App, which was described above.

name = "App"
uuid = "8f986787-14fe-4607-ba5d-fbff2944afa9"

[deps]
Priv = "ba13f791-ae1d-465a-978b-69c3ad90f72b"
Pub  = "c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1"

This project file assumes the following root scheme, presented as a Julia dictionary.

roots = Dict(
    :App  => UUID("8f986787-14fe-4607-ba5d-fbff2944afa9"),
    :Priv => UUID("ba13f791-ae1d-465a-978b-69c3ad90f72b"),
    :Pub  => UUID("c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1"),
)

With this root scheme, the operator import Priv in the code App instructs Julia to perform a search roots[:Priv], resulting in the value ba13f791-ae1d-465a-978b-69c3ad90f72b — The UUID of the package Priv which should be loaded in this context. The UUID identifier determines which package Priv must be loaded and used when calculating the operator import Priv in the main application.

The dependency graph of the project environment is determined by the contents of the manifest file, if any. If there is no manifest file, the graph is empty. The manifest file has a separate section for each direct or indirect dependency of the project. For each dependency, the UUID of the package and the hash of the source code tree or the explicit path to the source code are specified in the file. Consider the following example of a manifest file for an application App:

[[Priv]] # частный
deps = ["Pub", "Zebra"]
uuid = "ba13f791-ae1d-465a-978b-69c3ad90f72b"
path = "deps/Priv"

[[Priv]] # общедоступный
uuid = "2d15fe94-a1f7-436c-a4d8-07a9a496e01c"
git-tree-sha1 = "1bf63d3be994fe83456a03b874b409cfd59a6373"
version = "0.1.5"

[[Pub]]
uuid = "c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1"
git-tree-sha1 = "9ebd50e2b0dd1e110e842df3b433cb5869b0dd38"
version = "2.1.4"

  [Pub.deps]
  Priv = "2d15fe94-a1f7-436c-a4d8-07a9a496e01c"
  Zebra = "f7a24cb4-21fc-4002-ac70-f0e3a0dd3f62"

[[Zebra]]
uuid = "f7a24cb4-21fc-4002-ac70-f0e3a0dd3f62"
git-tree-sha1 = "e808e36a5d7173974b90a15a353b564f3494092f"
version = "3.4.2"

This manifest file fully describes the possible dependency graph for the project. App.

The application uses two different packages named Priv: private, which is the root dependency, and public, which is an indirect dependency via Pub. They differ in their unique UUIDs and have different dependencies.
- Private Package Priv depends on the packages Pub and Zebra.
- For a public package Priv there are no dependencies.
The application also depends on the package Pub which, in turn, depends on a publicly available package Priv and from the same package Zebra which the private package depends on Priv.

The dependency graph, represented as a dictionary, looks like this:

graph = Dict(
    # Частный пакет Priv:
    UUID("ba13f791-ae1d-465a-978b-69c3ad90f72b") => Dict(
        :Pub   => UUID("c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1"),
        :Zebra => UUID("f7a24cb4-21fc-4002-ac70-f0e3a0dd3f62"),
    ),
    # Общедоступный пакет Priv:
    UUID("2d15fe94-a1f7-436c-a4d8-07a9a496e01c") => Dict(),
    # Pub:
    UUID("c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1") => Dict(
        :Priv  => UUID("2d15fe94-a1f7-436c-a4d8-07a9a496e01c"),
        :Zebra => UUID("f7a24cb4-21fc-4002-ac70-f0e3a0dd3f62"),
    ),
    # Zebra:
    UUID("f7a24cb4-21fc-4002-ac70-f0e3a0dd3f62") => Dict(),
)

With this graph (graph) dependencies when Julia encounters the expression import Priv in the package Pub with a UUID c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1, the following search is performed:

graph[UUID("c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1")][:Priv]

As a result, the value is returned 2d15fe94-a1f7-436c-a4d8-07a9a496e01c, which means that in the context of the package Pub expression import Priv links to a public package Priv, rather than a private one, which the application depends on directly. Just like that by name Priv in the main project, you can refer to packages other than those that this name indicates in the dependencies, which makes identical names possible in the package system.

What happens when calculating the expression import Zebra in the main code App? Since the package Zebra not specified in the project file, the import will fail despite the fact that Zebra It is in the manifest file. Moreover, if the operator import Zebra it was used in a public package Priv with a UUID 2d15fe94-a1f7-436c-a4d8-07a9a496e01c it would also fail, since this package has Priv there are no declared dependencies in the manifest file and therefore it cannot download other packages. Package Zebra it can only be loaded by packages for which it is explicitly specified as a dependency in the manifest file, namely the package Pub and one of the packages Priv.

The Path diagram of the project environment is extracted from the manifest file. The path to the package with the identifier uuid and the name X It is determined according to the following rules (in the specified order).

If the project file in the directory matches the ID uuid and the name X, then the following happens.
- If it has an element entryfile at the top level, then uuid it is mapped to this path, which is interpreted relative to the directory containing the project file.
  - Otherwise uuid is mapped to the path src/X.jl regarding the directory containing the project file.
1. If the above conditions are not met, but there is a manifest file for the project file, which has a section corresponding to the identifier uuid, the following happens.
  - If it contains an element path, this path is used (relative to the directory containing the manifest file).
    
    If it contains an element git-tree-sha1, for uuid and the values git-tree-sha1 a deterministic hash function is calculated based on its result (let’s call this value slug) and in each directory in the global array DEPOT_PATH Julia is searching for a catalog packages/X/$slug. The first such found directory will be used.
If it is a directory, then uuid compared with src/X.jl, except in cases where there is an entry in the corresponding line of the manifest entryfile, which is used in this case. In both cases, these entries are relative to the directory in 2.1.

If any of these conditions are met, the path to the entry point to the source code will be either the result obtained, or the path relative to this result with the addition of src/X.jl. Otherwise, the path corresponding to the identifier uuid, no. If when downloading the package X the path to the source code cannot be found, the search fails, and the user may be prompted to install the appropriate version of the package or fix the situation in another way (for example, to announce X as an addiction).

In the example manifest file above, the path to the first package is Priv (with UUID ba13f791-ae1d-465a-978b-69c3ad90f72b) is defined as follows: Julia searches for a section of this package in the manifest file, finds the element path, looking for a way deps/Priv regarding the project catalog App (let’s say that the code App located on the way /home/me/projects/App), finds the folder /home/me/projects/App/deps/Priv and therefore downloads the package Priv out of it.

If it would be necessary to download another package Priv (with UUID 2d15fe94-a1f7-436c-a4d8-07a9a496e01c), the Julia environment would find its section in the manifest, in which, however, there is no element path But there is an element git-tree-sha1. Therefore, the value would be calculated slug for this pair of UUID and SHA-1 hash, the result is obtained HDkrT (it doesn’t matter exactly how this calculation is performed, but the function is consistent and deterministic). This means that the path to the package is Priv There will be a way packages/Priv/HDkrT/src/Priv.jl in one of the package repositories. Let’s say the variable DEPOT_PATH contains the value ["/home/me/.julia", "/usr/local/julia"]. In this case, Julia checks if the following paths exist:

/home/me/.julia/packages/Priv/HDkrT
/usr/local/julia/packages/Priv/HDkrT

Julia then tries to download the public package. Priv from the file packages/Priv/HDKrT/src/Priv.jl in the repository where the first of these paths is found.

A possible path diagram for an example project environment is shown below. App in the form in which it is presented in the manifest for the dependency graph above, after searching in the local file system.

paths = Dict(
    # Частный пакет Priv:
    (UUID("ba13f791-ae1d-465a-978b-69c3ad90f72b"), :Priv) =>
        # относительная точка входа внутри репозитория `App`:
        "/home/me/projects/App/deps/Priv/src/Priv.jl",
    # Общедоступный пакет Priv:
    (UUID("2d15fe94-a1f7-436c-a4d8-07a9a496e01c"), :Priv) =>
        # пакет, установленный в системном хранилище:
        "/usr/local/julia/packages/Priv/HDkr/src/Priv.jl",
    # Pub:
    (UUID("c07ecb7d-0dc9-4db7-8803-fadaaeaf08e1"), :Pub) =>
        # пакет, установленный в пользовательском хранилище:
        "/home/me/.julia/packages/Pub/oKpw/src/Pub.jl",
    # Zebra:
    (UUID("f7a24cb4-21fc-4002-ac70-f0e3a0dd3f62"), :Zebra) =>
        # пакет, установленный в системном хранилище:
        "/usr/local/julia/packages/Zebra/me9k/src/Zebra.jl",
)

In the example diagram, there are three different types of package locations (the first and third locations are part of the default download path).

"https://stackoverflow.com/a/35109534 [Own copy of the]"private package Priv hosted in the repository App.
Public packages Priv and Zebra They are located in the system storage, where the system administrator installs and manages packages. They are available to all users of the system.
Package Pub It is located in the user storage, where packages are installed by an active user. They are available only to the user who installed them.

Package Catalogs

Package directories are a simpler type of environment that does not allow name conflicts to be resolved. In a package directory, a set of top-level packages is a set of nested directories that "look" like packages. Package X exists in the package directory if this directory contains one of the following entry point files:

X.jl
X/src/X.jl
X.jl/src/X.jl

Which dependencies a package can import in the package directory is determined by the presence of the project file in the package.

If there is a project file, you can import only the packages specified in its section. [deps].
If there is no project file, you can import any top-level packages, that is, the same packages that can be loaded in the module. Main or in the REPL.

The root schema is determined by analyzing the contents of the project catalog and creating a list of all available packages. In addition, each package found in the folder X, a UUID is assigned as follows.

If there is a file X/Project.toml And it has an element uuid, then this element uuid and it will be the ID value.
If the file X/Project.toml there is, but there is no top-level UUID element in it, a dummy identifier uuid it is generated by hashing the canonical (real) path to X/Project.toml.
Otherwise (if the file Project.toml no) ID uuid It will consist of https://en.wikipedia.org/wiki/Universally_unique_identifier#Nil_UUID [just zeros].

The dependency graph of the package catalog is determined by the presence and contents of the project file in the subdirectory of each package. The following rules apply.

If there is no project file in the package’s subdirectory, the package is not included in the graph, and import operators in its code are treated as top-level operators, both in the main project and in the REPL.
If there is a project file in the package’s attached directory, then the scheme is used as an element of the graph for its UUID. [deps] from the project file. If this section is missing, the element is considered empty.

Suppose the package directory has the following structure and contents:

Aardvark/
    src/Aardvark.jl:
        import Bobcat
        import Cobra

Bobcat/
    Project.toml:
        [deps]
        Cobra = "4725e24d-f727-424b-bca0-c4307a3456fa"
        Dingo = "7a7925be-828c-4418-bbeb-bac8dfc843bc"

    src/Bobcat.jl:
        import Cobra
        import Dingo

Cobra/
    Project.toml:
        uuid = "4725e24d-f727-424b-bca0-c4307a3456fa"
        [deps]
        Dingo = "7a7925be-828c-4418-bbeb-bac8dfc843bc"

    src/Cobra.jl:
        import Dingo

Dingo/
    Project.toml:
        uuid = "7a7925be-828c-4418-bbeb-bac8dfc843bc"

    src/Dingo.jl:
# there are no imports

In the form of a dictionary, the corresponding root structure will be presented as follows:

roots = Dict(
    :Aardvark => UUID("00000000-0000-0000-0000-000000000000"), # файла проекта нет, нулевой UUID
    :Bobcat   => UUID("85ad11c7-31f6-5d08-84db-0a4914d4cadf"), # фиктивный UUID на основе пути
    :Cobra    => UUID("4725e24d-f727-424b-bca0-c4307a3456fa"), # UUID из файла проекта
    :Dingo    => UUID("7a7925be-828c-4418-bbeb-bac8dfc843bc"), # UUID из файла проекта
)

In the form of a dictionary, the corresponding graph structure will be represented as follows:

graph = Dict(
    # Bobcat:
    UUID("85ad11c7-31f6-5d08-84db-0a4914d4cadf") => Dict(
        :Cobra => UUID("4725e24d-f727-424b-bca0-c4307a3456fa"),
        :Dingo => UUID("7a7925be-828c-4418-bbeb-bac8dfc843bc"),
    ),
    # Cobra:
    UUID("4725e24d-f727-424b-bca0-c4307a3456fa") => Dict(
        :Dingo => UUID("7a7925be-828c-4418-bbeb-bac8dfc843bc"),
    ),
    # Dingo:
    UUID("7a7925be-828c-4418-bbeb-bac8dfc843bc") => Dict(),
)

Pay attention to a number of important rules.

A package without a project file can depend on any top-level package, and since all packages in the package directory are available at the top level, it can import all packages in the environment.
A package with a project file cannot depend on a package without such a file, since packages with project files can only load packages available in the schema. graph, and there are no packages without project files in it.
Only packages without project files can depend on a package with a project file, but without an explicitly specified UUID, since the dummy UUIDs assigned to such packages are intended solely for internal use.

Let’s see how these rules are applied in practice in our example.

Aardvark can import any of the packages Bobcat, Cobra or Dingo; actually imported Bobcat and Cobra.
Bobcat It can and does import packages Cobra and Dingo, which do not have project files with UUIDs and which are declared as dependencies in the section [deps] The package Bobcat.
Bobcat may depend on Aardvark, since the package has Aardvark there is no project file.
Cobra can and does import a package Dingo, which has a project file and a UUID and is declared as a dependency in the section [deps] The package Cobra.
Cobra it cannot depend on the package Aardvark or Bobcat since they don’t have any real UUIDs.
Package Dingo He can’t import anything because he has a project file, but without a partition. [deps].

The path scheme in the package directory is very simple: in it, the names of the nested directories are mapped to the paths of the entry points. In other words, if the path to the project directory in our example looked like /home/me/animals, then the scheme paths it could be represented by such a dictionary:

paths = Dict(
    (UUID("00000000-0000-0000-0000-000000000000"), :Aardvark) =>
        "/home/me/AnimalPackages/Aardvark/src/Aardvark.jl",
    (UUID("85ad11c7-31f6-5d08-84db-0a4914d4cadf"), :Bobcat) =>
        "/home/me/AnimalPackages/Bobcat/src/Bobcat.jl",
    (UUID("4725e24d-f727-424b-bca0-c4307a3456fa"), :Cobra) =>
        "/home/me/AnimalPackages/Cobra/src/Cobra.jl",
    (UUID("7a7925be-828c-4418-bbeb-bac8dfc843bc"), :Dingo) =>
        "/home/me/AnimalPackages/Dingo/src/Dingo.jl",
)

Since all packages in the package catalog environment are, by definition, nested directories with corresponding entry point files, their schema elements paths they always have this form.

Media Stacks

The third and final type of environment is one in which other types of environments overlap each other, forming a single environment where the packages of each of the component environments are available. Such composite media are called media stacks. In the Julia global variable LOAD_PATH The environment stack in which the Julia process is running has been defined. In order for the Julia process to have access to packages from only one project or package directory, do not specify in LOAD_PATH other environments. However, it is often useful to have access to some tools - standard libraries, profilers, debuggers, auxiliary programs, etc. - even if they are not dependencies of the current project. If you add an environment with these tools to the download path, they will immediately be available in the top-level code, so you do not need to add them to the project.

The data structures of the roots, graph, and paths belonging to the components of the media stack are combined very simply as dictionaries, and in the case of a key match, priority is given to earlier entries. In other words, if there is a stack stack = [env₁, env₂, …], then the following happens.

roots = reduce(merge, reverse([roots₁, roots₂, …]))
graph = reduce(merge, reverse([graph₁, graph₂, …]))
paths = reduce(merge, reverse([paths₁, paths₂, …]))

Variables rootsᵢ, graphᵢ and pathsᵢ The subscripts correspond to the environments envᵢ with lower indexes included in stack. Function reverse it is used for the reason that when the keys in the dictionaries passed as arguments to the function match merge, priority is given to the last argument, not the first. This approach has several noteworthy features.

The main environment, that is, the first environment in the stack, is fully integrated into the multilayer environment. The dependency graph of the first environment in the stack is guaranteed to be included in the layered environment completely and unchanged, including with the same versions of all dependencies.
Packages from non-core environments may end up using incompatible dependency versions, even if their own environments are fully compatible. This can happen if one of their dependencies is replaced by a version from an earlier environment in the stack (according to graph, path, or both).

Since the main environment is usually the environment of the current project, and later environments in the stack contain additional tools, this compromise is justified: it is better that the development tools stop working than the entire project. As a rule, if such incompatibilities occur, it is necessary to update the development tools to versions compatible with the main project.

Package Extensions

A package extension is a module that is automatically loaded when a certain set of other packages (its "triggers") are loaded in the current Julia session. Extensions are defined in the project file in the section [extensions]. Extension triggers are a subset of the other packages listed in the section [weakdeps] (and, perhaps, but rarely, in the section [deps]) of the project file. These packages may have compat entries, just like other packages.

name = "MyPackage"

[compat]
ExtDep = "1.0"
OtherExtDep = "1.0"

[weakdeps]
ExtDep = "c9a23..." # uuid
OtherExtDep = "862e..." # uuid

[extensions]
BarExt = ["ExtDep", "OtherExtDep"]
FooExt = "ExtDep"
...

Keys in the section extensions — these are the names of extensions. They are loaded when all packages listed on the right side (triggers) are loaded. If the extension has only one trigger, the list of triggers can be written as a string for brevity. Entry point for expansion FooExt located in ext/FooExt.jl or ext/FooExt/FooExt.jl. The contents of an extension often have this structure:

module FooExt

# Loading the main package and triggers
using MyPackage, ExtDep

# Extending the functionality of the main package through types from triggers
MyPackage.func(x::ExtDep.SomeStruct) = ...

end

When an extension package is added to the environment, the sections weakdeps and extensions they are saved in the manifest file in the section for this package. The dependency search rules for a package are the same as for its "parent", except that triggers in the list are also considered dependencies.

Workspaces

A project file can define a workspace by specifying a set of projects included in that workspace.:

[workspace]
projects = ["test", "benchmarks", "docs:", "SomePackage"]

Each project listed in the array projects, is indicated by the relative path from the root of the workspace. It can be a direct child directory (for example, "test") or a nested subdirectory (for example, "nested/subdir/MyPackage"). Each project contains its own file Project.toml, which may include additional dependencies and compatibility limitations. In such cases, the package manager collects all dependency information from all projects in the workspace, generating a single manifest file that combines the versions of all dependencies.

When Julia downloads a project, it searches up through the parent directories until it reaches the user’s home directory to find the workspace that includes that project. This allows workspace projects to be nested to any depth in the workspace directory tree.

Additionally, workspaces can be "nested," which means that the project defining the workspace can also be part of another workspace. This scenario still uses a single manifest file stored with the "root project" (a project that does not have another workspace that includes it). An example of a file structure might look like this:

Project.toml # projects = ["MyPackage"]
Manifest.toml
MyPackage/
    Project.toml # projects = ["test"]
    test/
        Project.toml

Package and environment preferences

Preferences are dictionaries of metadata that affect the behavior of packages in the environment. The preference system supports reading preferences at compile time. This means that before downloading the code, you need to make sure that the pre-compilation files selected by Julia were built with the preferences of the current environment. The public API for changing preferences is contained in the package https://github.com/JuliaPackaging/Preferences.jl [Preferences.jl]. Preferences are stored as TOML dictionaries in a file (Julia)LocalPreferences.toml next to the current active project. When exporting, the preference is transferred to a file. (Julia)Project.toml. The idea is that shared projects can have common preferences, but users can override them using their own settings in the LocalPreferences.toml file, which, as its name implies, should be added to the file.gitignore.

The preferences accessed during compilation are automatically marked as compile-time preferences. Any change to them will force the Julia compiler to recompile all cached pre-compilation files (.ji and the corresponding files .so, .dll or .dylib) for this module. To do this, during compilation, the hash of all compile-time preferences is serialized, and then, when searching for the necessary files, it is checked for compliance with the current environment.

Default values can be set for the preferences at the storage level; if the Foo package is installed in the global environment and has the specified preferences, they are applied provided that the global environment is enabled in LOAD_PATH. Preferences related to environments higher in the environment stack are overridden by lower-level entries in the download path up to the currently active project. This allows default preferences to exist at the project level, and when inheriting for active projects, they can be combined or even completely overwritten. For more information on how to allow or disable combining preferences, see the docstring function. Preferences.set_preferences!().

Conclusion

Federated package management and accurate application reproducibility are difficult to implement, but important mechanisms. Together, they complicate the package loading system compared to most dynamic languages, but at the same time provide the scalability and reproducibility typically associated with static languages. As a rule, when using the built-in Julia package manager for project management, you do not need to know exactly how these mechanisms work. Challenge Pkg.add("X") adds a package X to the corresponding project and manifest files selected using Pkg.activate("Y"), so on further calls import X it loads without additional effort.