Engee documentation

Eliminating pre-compilation freezes due to open tasks or I/O

The page is in the process of being translated.

In Julia version 1.10 and later, you can see the following message:

It can be displayed repeatedly. If this continues to happen without any hints of self-resolution, there may have been a "pre-compilation hang" that needs fixing. Even if this is a temporary phenomenon, it is better to eliminate it so as not to bother users with this warning. On this page, you will learn how to analyze and fix such problems.

If you follow the advice and press the keyboard shortcut Ctrl-C, you will see the following.

^C Interrupted: Exiting precompilation...

  1 dependency had warnings during precompilation:
┌ Test1 [ac89d554-e2ba-40bc-bc5c-de68b658c982]
│  [pid 2745] waiting for IO to finish:
│   Handle type        uv_handle_t->data
│   timer              0x55580decd1e0->0x7f94c3a4c340

This message highlights two key points.:

  • the hang occurs during the pre-compilation of Test1, the dependencies of Test2 (the package that we tried to download using `using Test2');

  • during the pre-compilation of Test1, Julia created a Timer object (use ?Timer, if you are not familiar with timers), which is still open. The process will hang until it closes.

If this is enough to understand how the timer = Timer(args...), it is recommended to add wait(timer)`if the `timer eventually completes on its own, or close(timer), if you need to force it to close before the final termination (end) of the module.

However, in some cases, it’s not that simple. It’s usually best to start by determining if the hang is related to the code in Test1 or to one of the Test1 dependencies.:

  • Option 1. Pkg.add("Aqua") and use https://juliatesting.github.io/Aqua .jl/dev/#Aqua.test_persistent_tasks-Tuple{Base.PkgId}[Aqua.test_persistent_tasks]. This will help you identify which package is causing the problem, and then run see below instructions. If necessary, you can create a PkgId as Base.PkgId(UUID("..."), "Test1"), where ... is taken from the 'uuid` entry in `Test1/Project.toml'.

  • Option 2. Diagnosing the cause of the hang manually.

To perform manual diagnostics:

  1. Pkg.develop("Test1")

  2. Comment out all the code included (include') or defined in `Test1, in the context of the using/import statements.

  3. Try using Test2' (or even `using Test1 if it freezes too) one more time.

Now we stand at a crossroads: either

Diagnosis and elimination of freezes due to package dependency

Use binary search to identify a problematic dependency: start by commenting out half of the dependencies, then, when you determine which one is responsible, comment out half of that half, etc. (you don’t have to remove them from the project, just comment out the using/import statements).

Having identified the suspected problem (here we will call it ThePackageYouThinkIsCausingTheProblem), first try to pre-compile this package. If it also freezes during pre-compilation, keep looking for the problem in the opposite direction.

However, most likely, the pre-compilation of ThePackageYouThinkIsCausingTheProblem will be performed normally. This suggests that the problem is with the function ThePackageYouThinkIsCausingTheProblem.__init__, which is not executed during the precompilation of ThePackageYouThinkIsCausingTheProblem, but is executed in any package that loads `ThePackageYouThinkIsCausingTheProblem'. To test this theory, create a minimal working example (MWE), something like the following:

(@v1.10) pkg> generate MWE
  Generating  project MWE:
    MWE\Project.toml
    MWE\src\MWE.jl

where the source code of MWE.jl looks like

module MWE
using ThePackageYouThinkIsCausingTheProblem
end

and you added ThePackageYouThinkIsCausingTheProblem to the MWE dependencies.

If this MWE reproduces the hang, you have found the culprit: ThePackageYouThinkIsCausingTheProblem.__init__ should create a Timer object. If the timer object can be safely closed (`close'), this is a good option. Otherwise, the most common solution is not to create a timer while any package is being pre-compiled: add

ccall(:jl_generating_output, Cint, ()) == 1 && return nothing

as the first line, ThePackageYouThinkIsCausingTheProblem.__init__, which will avoid initialization in any Julia process, the purpose of which is to pre-compile packages.

Fixing the package code to prevent freezes

Find the leading words in the package (for example, Timer) and try to determine where the problem occurs. Note that the definition of the method is of the form

maketimer() = Timer(timer -> println("hi"), 0; interval=1)

it is not problematic in itself: it can only cause this problem if `maketimer' is called during module definition. This may come from a top-level operator such as

const GLOBAL_TIMER = maketimer()

or it can happen in https://github.com/JuliaLang/PrecompileTools .jl[pre-compilation workload].

If it is difficult for you to identify the causal strings, use a binary search: comment out sections of your package (or include (include) strings to skip entire files) until you reduce the scale of the problem.