Engee documentation

Working with Python

General information

Python is a high-level, interpreted, general-purpose programming language. In this article we will look at the usage of Python within the Engee platform, specifically:

Calling Python in Engee

Python integration via the PyCall.jl package

PyCall.jl is a Julia package that allows you to call functions and objects from Python directly in Julia code. It provides an interface between Julia and Python and makes it possible to import Python modules, call their functions, create objects and write code with them in Julia. For more information about supported call commands, literals, and Python packages in Julia, see PyCall.jl.

*PyCall is built into Engee by default, you do not need to install or import it.

Engee uses the computational kernel Julia, but you can write Python code in Engee scripts (see [Julia]) using the following methods. Features when working with interactive Engee scripts) using the following methods:

  • Оборачивание кода на Python строковым литералом py"""…​""", основанном на пакете PyCall. This literal executes the string as Python program code, returning no value and executing the passed code.

  • Wrapping Python code with the string literal py"…​", based on PyCall. This literal takes a string as an argument and executes it as a Python expression.

  • Usage of the commands pyimport, pyeval, pycall, @pycall, pybuiltin, @pywith, @pyinclude, @pysym, @pydef.

Let’s look at these methods with examples.

You can write code in Julia using familiar Python syntax. Для этого оберните ваш код на языке Python литералами py"""…​""" и py"…​", например:

py""" # оборачивает в литерал
import numpy as np # импортирует пакет numpy
"""
a = py"np.zeros(2)" # оборачивает в литерал, в котором создает одномерный массив из двух элементов, заполненный нулями

For the pyimport command, the code would look like this:

np = pyimport("numpy")
a = np.zeros(2)

In both examples, the a variables will have the same value. The difference between the examples is that in the case of the pyimport command, Julia has an np object that can be accessed directly.

In Julia, Python commands like import numpy as np do not work natively, but with string literals you can insert Python code directly into Julia code, provided the PyCall package is installed.

You can also use other Python calling commands in Julia:

  • pyimport - imports a Python module into Julia, allowing you to use its functions and variables in Julia code.

  • pyeval - executes Python string code in the Julia context, returning the result of the execution.

  • pycall - calls a Python function with the specified arguments from Julia.

  • @pycall - A macro that provides a convenient syntax for calling Python functions from Julia. Typically used more often than pycall because of its simpler invocation.

  • pybuiltin - provides access to built-in Python functions such as print, len, range, etc.

  • @pywith - macro creates a Python execution context in which you can work with blocks of Python code inside Julia.

  • @pyinclude - a macro that includes a Python code file in Julia and executes it.

  • @pysym is a macro that converts a Julia symbol name to the corresponding Python object.

  • @pydef - creates a Python class whose methods are implemented in Julia.

For example, the pybuiltin command:

pybuiltin("print")("Hello, world!")

Code outputs the message Hello, world!.

Features when working with interactive Engee scripts

Read an example of Python usage in Engee scripts:

Open example in Engee
To see an example of Python usage in Engee scripts, see here.

Interactive Engee scripts (with the .ngscript extension) are a tool for working with code in the Engee environment (see Script editor). Let’s consider the peculiarities of working with Engee interactive scripts for Python:

  • Engee scripts can be created directly in Engee itself. This is the only supported format for creating interactive scripts.

  • Adding packages in one Engee script allows them to be used in other Engee scripts without re-installation.

  • Engee scripts support Python magic commands such as %run.

You may need many packages to work with Python in Engee scripts, such as the scientific computation support package SciPy. Use the package manager Pkg and Julia syntax to install it:

using Pkg
Pkg.add("SciPy") # will install the SciPy package in Julia

To remove packages, use the rm command. For example, Pkg.rm("Pluto"). Read more at Working with Julia packages.

To find out which packages are installed in Engee scripts, use:

import Pkg
Pkg.status()

Let’s look at an example. To do this, let’s install the SciPy package and use it to find the root of an equation via the string literal py"":

py"""
from scipy.optimize import root_scalar

def func(x):
    return x**2 - 4

result = root_scalar(func, method='brentq', bracket=[0, 2])

print("Результат (корень уравнения):", result.root)
"""

Let’s understand the code in more detail:

  • The from scipy.optimise import root_scalar line imports the root_scalar function from the scipy.optimize module. This function is designed to numerically find roots of equations with one variable.

  • The def func(x): line defines the func function for which the root is sought. Through return the value of the equation is returned to the func function, and x is the variable.

  • The result = root_scalar(func, method='brentq', bracket=[0, 2]) line uses the root_scalar function to find the root of the equation given by the func function. The 'brentq' method indicates that it is the Brant method that we want to use. Using the parameters bracket=[0, 2] we specify the interval for searching the root of the equation (in our case from 0 to 2).

  • The print("Result (root of the equation):", result.root) line prints the found root of the equation. result.root contains the root value found by the root_scalar function.

In summary, we find the root of the equation using Brent’s method from the SciPy package. The value of the root of the equation will be obtained:

Результат (корень уравнения): 2.0

Similarly, you can use other Python packages like NumPy, Matplotlib, Pandas, etc., but keep in mind that in Engee you cannot create different Python environments and switch between them.

The only limit on the amount of packages you can install is the RAM memory that is allocated to your current session in Engee. For more information about available memory, see Starting an Engee session.

Working with Python variables in the Engee environment

Variables created in Python notebooks (see "Python Variables in Engee") or in Engee scripts using a Python call. Running Python notebooks in Engee) or in Engee scripts using a Python call - will be of type PyObject. This typing of variables is due to the fact that data types in Python are not equal to Julia and Python variables are assigned their own unique type to save data.

You can import variables from a Python module into Julia variables by calling methods directly. For example:

math = pyimport("math")
math.sin(math.pi / 4)

The resulting value is: 0.7071067811865475.

In this code, the math = pyimport("math") command is used in Julia using the PyCall package. It allows you to import the math module from the standard Python package and store it in the math variable for further usage in Julia code:

python straight method 1

Variables of type PyObject cannot be exported to .mat format, but can to .jld2 (see Import and export of variables for details :

python variables 1

Python variables cannot be cleared using the built-in Engee tools, but can be overwritten. Deleting variables requires restarting the session.

PyObject variables can be used further in Engee scripts if needed, but be aware that Python data types are different from Julia. If type conversion is necessary, it is recommended that the types be given explicitly:

var_jl = py"int(var_py)" # var_py is a variable with the PyObject type.
println(var_jl)

where var_jl is a Julia variable. After conversion, the var_py variable remains with the PyObject type, so we created a new var_jl variable in Julia from the old var_py of the PyObject type.

You can work in Engee scripts with variables of type PyObject without setting them explicitly. For example, we have a variable deg of type PyObject. We can access it via the println(deg) command, and, for example, add 10:

println(deg + 10)

# we get PyObject 11.

The variable has retained its PyObject type and we have successfully performed the mathematical operation.

Running Python notebooks in Engee

You can load your Python notebook (Jupyter Notebook) into Engee, and run it in script editor. Let’s take a look at the specifics of working with Python notebooks:

  • Python notebooks can be created directly in Engee using script editor interactive script icon Engee. To do this, open the editor, click +, then Create Script and select the .ipynb format:

    img36c en -> img36c 1 en

  • Packages installed in Engee scripts cannot be imported into Python notebooks. The required packages need to be re-installed for each new Python notebook.

  • The Julia magic commands and the two user input functions, input() and getpass(), cannot be accessed from a Python notebook in Engee.

  • Python notebooks do not support delay in code (e.g. via the sleep function) and output the result of code execution instantly.

  • You cannot use Python scripts (extension .py).

  • You cannot use the conda package management tool, only pip.

Use the magic !pip command to install packages in Python notebooks:

!pip install scipy
*Magic commands are special commands in various software development tools. These commands are designed to perform additional actions that can be useful for interactive programming and data analysis.

If necessary, install packages from the WHL file:

!pip install path/to/file.whl

WHL is an archived format for packaging and distributing Python packages. To install a WHL file, first transfer it to File browser Engee. WHL files contain pre-compiled package binaries, making them easier to install and manage project dependencies. You can find WHL files on the official websites of the respective Python packages.

To find out which packages are installed in Python notebooks, use:

!pip list
Check out example usage of .ipynb in Engee:

Importing custom modules

In order to import custom modules into Python notebooks, you need to make sure that the path to them is added to the sys.path variable. By default, this variable contains paths to the standard libraries, installed packages and the current working directory of the Python interpreter, not the directory where the script resides. To find out the current directory, you can use the os.getcwd() function. If necessary, the current directory can be changed using os.chdir("path"). If the module is in a different directory, its path must be added to sys.path manually. This can be done with the command:

import sys
sys.path.append('/path/to/your/module')

After adding the path to sys.path, the module can be imported and its functions and data can be used. If you try to import a module from a directory that is not added to sys.path, a ModuleNotFoundError error will occur.

Python neural networks and their integration with Engee models

See an example for training a neural network using the scikit-learn library and generating training data from a model:

Open example in Engee

To work with neural networks using Python, Engee uses the PyCall package and Python calling commands in Julia. For example, for the library scikit-learn:

# installing the sklearn library in Julia
using Pkg
Pkg.add("ScikitLearn")

# importing necessary modules from Python to Julia
@pyimport sklearn.datasets as datasets
@pyimport sklearn.neural_network as nn
@pyimport numpy as np

To read another example of training a neural network in Engee using PyCall, please go to link.

  1. Calling a Python function via PyCall in Engee.

  2. How to use Python in Engee scripts

  3. How Julia differs from other languages

  4. Official Python documentation