Working with Python
General information
Python is a high-level, interpreted, general-purpose programming language. In this article we will look at the usage of Python within the Engee platform, specifically:
-
Python integration via the PyCall.jl package - write Python code in Engee scripts thanks to the PyCall package;
-
Features when working with interactive Engee scripts - learn the features of Engee scripts and install the necessary Python packages;
-
Working with Python variables in the Engee environment - work with Python variables in Engee scripts;
-
Running Python notebooks in Engee - run your own Python notebooks, bringing them into Engee directly from an external development environment;
-
Python neural networks and their integration with Engee models - build and train neural networks with Python libraries based on Engee models.
Calling Python in Engee
Python integration via the PyCall.jl package
PyCall.jl is a Julia package that allows you to call functions and objects from Python directly in Julia code. It provides an interface between Julia and Python and makes it possible to import Python modules, call their functions, create objects and write code with them in Julia. For more information about supported call commands, literals, and Python packages in Julia, see PyCall.jl. *PyCall is built into Engee by default, you do not need to install or import it. |
Engee uses the computational kernel Julia, but you can write Python code in Engee scripts (see [Julia]) using the following methods. Features when working with interactive Engee scripts) using the following methods:
-
Оборачивание кода на Python строковым литералом
py"""…"""
, основанном на пакете PyCall. This literal executes the string as Python program code, returning no value and executing the passed code. -
Wrapping Python code with the string literal
py"…"
, based on PyCall. This literal takes a string as an argument and executes it as a Python expression. -
Usage of the commands
pyimport
,pyeval
,pycall
,@pycall
,pybuiltin
,@pywith
,@pyinclude
,@pysym
,@pydef
.
Let’s look at these methods with examples.
You can write code in Julia using familiar Python syntax. Для этого оберните ваш код на языке Python литералами py"""…"""
и py"…"
, например:
py""" # оборачивает в литерал
import numpy as np # импортирует пакет numpy
"""
a = py"np.zeros(2)" # оборачивает в литерал, в котором создает одномерный массив из двух элементов, заполненный нулями
For the pyimport
command, the code would look like this:
np = pyimport("numpy")
a = np.zeros(2)
In both examples, the a
variables will have the same value. The difference between the examples is that in the case of the pyimport
command, Julia has an np object that can be accessed directly.
In Julia, Python commands like |
You can also use other Python calling commands in Julia:
-
pyimport
- imports a Python module into Julia, allowing you to use its functions and variables in Julia code. -
pyeval
- executes Python string code in the Julia context, returning the result of the execution. -
pycall
- calls a Python function with the specified arguments from Julia. -
@pycall
- A macro that provides a convenient syntax for calling Python functions from Julia. Typically used more often thanpycall
because of its simpler invocation. -
pybuiltin
- provides access to built-in Python functions such asprint
,len
,range
, etc. -
@pywith
- macro creates a Python execution context in which you can work with blocks of Python code inside Julia. -
@pyinclude
- a macro that includes a Python code file in Julia and executes it. -
@pysym
is a macro that converts a Julia symbol name to the corresponding Python object. -
@pydef
- creates a Python class whose methods are implemented in Julia.
For example, the pybuiltin
command:
pybuiltin("print")("Hello, world!")
Code outputs the message Hello, world!
.
Features when working with interactive Engee scripts
Read an example of Python usage in Engee scripts: |
To see an example usage of Python in Engee scripts, please see the following example here. |
Interactive Engee scripts (with the .ngscript extension) are a tool for working with code in the Engee environment (see Script editor). Let’s consider the peculiarities of working with Engee interactive scripts for Python:
-
Engee scripts can be created directly in Engee itself. This is the only supported format for creating interactive scripts.
-
Adding packages in one Engee script allows them to be used in other Engee scripts without re-installation.
-
Engee scripts support Python magic commands such as
%run
.
You may need many packages to work with Python in Engee scripts, such as the scientific computation support package SciPy. Use the package manager Pkg and Julia syntax to install it:
using Pkg
Pkg.add("SciPy") # установит пакет SciPy в Julia
To remove packages, use the rm
command. For example, Pkg.rm("Pluto")
. Read more at Working with Julia packages.
To find out which packages are installed in Engee scripts, use:
import Pkg
Pkg.status()
Let’s look at an example. To do this, let’s install the SciPy package and use it to find the root of an equation via the string literal py""
:
py"""
from scipy.optimize import root_scalar
def func(x):
return x**2 - 4
result = root_scalar(func, method='brentq', bracket=[0, 2])
print("Результат (корень уравнения):", result.root)
"""
Let’s understand the code in more detail:
-
The
from scipy.optimise import root_scalar
line imports theroot_scalar
function from thescipy.optimize
module. This function is designed to numerically find roots of equations with one variable. -
The
def func(x):
line defines the func function for which the root is sought. Throughreturn
the value of the equation is returned to the func function, andx
is the variable. -
The
result = root_scalar(func, method='brentq', bracket=[0, 2])
line uses theroot_scalar
function to find the root of the equation given by the func function. The'brentq'
method indicates that it is the Brant method that we want to use. Using the parametersbracket=[0, 2]
we specify the interval for searching the root of the equation (in our case from0
to2
). -
The
print("Result (root of the equation):", result.root)
line prints the found root of the equation.result.root
contains the root value found by theroot_scalar
function.
In summary, we find the root of the equation using Brent’s method from the SciPy package. The value of the root of the equation will be obtained:
Результат (корень уравнения): 2.0
Similarly, you can use other Python packages like NumPy, Matplotlib, Pandas, etc., but keep in mind that in Engee you cannot create different Python environments and switch between them.
The only limit on the amount of packages you can install is the RAM memory that is allocated to your current session in Engee. For more information about available memory, see Starting an Engee session. |
Working with Python variables in the Engee environment
Variables created in Python notebooks (see "Python Variables in Engee") or in Engee scripts using a Python call. Running Python notebooks in Engee) or in Engee scripts using a Python call - will be of type PyObject. This typing of variables is due to the fact that data types in Python are not equal to Julia and Python variables are assigned their own unique type to save data.
You can import variables from a Python module into Julia variables by calling methods directly. For example:
math = pyimport("math")
math.sin(math.pi / 4)
The resulting value is: 0.7071067811865475
.
In this code, the math = pyimport("math")
command is used in Julia using the PyCall package. It allows you to import the math
module from the standard Python package and store it in the math
variable for further usage in Julia code:
Variables of type PyObject cannot be exported to .mat format, but can to .jld2 (see Importing and exporting variables for details :
Python variables cannot be cleared using the built-in Engee tools, but can be overwritten. Deleting variables requires restarting the session. |
PyObject variables can be used further in Engee scripts if needed, but be aware that Python data types are different from Julia. If type conversion is necessary, it is recommended that the types be given explicitly:
var_jl = py"int(var_py)" #var_py — переменная с типом PyObject
println(var_jl)
where var_jl
is a Julia variable. After conversion, the var_py
variable remains with the PyObject type, so we created a new var_jl
variable in Julia from the old var_py
of the PyObject type.
You can work in Engee scripts with variables of type PyObject without setting them explicitly. For example, we have a variable deg
of type PyObject. We can access it via the println(deg)
command, and, for example, add 10
:
println(deg + 10)
# получим PyObject 11.
The variable has retained its PyObject type and we have successfully performed the mathematical operation.
Running Python notebooks in Engee
You can load your Python notebook (Jupyter Notebook) into Engee, and run it in script editor. Let’s take a look at the specifics of working with Python notebooks:
-
Python notebooks cannot be created in Engee, so export them from external software to File browser Engee.
-
Packages installed in Engee scripts cannot be imported into Python notebooks. The required packages need to be re-installed for each new Python notebook.
-
The Julia magic commands and the two user input functions,
input()
andgetpass()
, cannot be accessed from a Python notebook in Engee. -
Python notebooks do not support delay in code (e.g. via the
sleep
function) and output the result of code execution instantly. -
You cannot use Python scripts (extension .py).
-
You cannot use the conda package management tool, only pip.
Use the magic !pip
command to install packages in Python notebooks:
!pip install scipy
*Magic commands are special commands in various software development tools. These commands are designed to perform additional actions that can be useful for interactive programming and data analysis. |
If necessary, install packages from the WHL file:
!pip install path/to/file.whl
WHL is an archived format for packaging and distributing Python packages. To install a WHL file, first transfer it to File browser Engee. WHL files contain pre-compiled package binaries, making them easier to install and manage project dependencies. You can find WHL files on the official websites of the respective Python packages.
To find out which packages are installed in Python notebooks, use:
!pip list
Check out example usage of .ipynb in Engee: |
Importing custom modules
In order to import custom modules into Python notebooks, you need to make sure that the path to them is added to the sys.path
variable. By default, this variable contains paths to the standard libraries, installed packages and the current working directory of the Python interpreter, not the directory where the script resides. To find out the current directory, you can use the os.getcwd()
function. If necessary, the current directory can be changed using os.chdir("path")
. If the module is in a different directory, its path must be added to sys.path
manually. This can be done with the command:
import sys
sys.path.append('/path/to/your/module')
After adding the path to sys.path
, the module can be imported and its functions and data can be used. If you try to import a module from a directory that is not added to sys.path
, a ModuleNotFoundError
error will occur.
Python neural networks and their integration with Engee models
See an example for training a neural network using the scikit-learn library and generating training data from a model: |
To work with neural networks using Python, Engee uses the PyCall package and Python calling commands in Julia. For example, for the library scikit-learn:
# установка библиотеки sklearn в Julia
using Pkg
Pkg.add("ScikitLearn")
# импорт необходимых модулей из Python в Julia
@pyimport sklearn.datasets as datasets
@pyimport sklearn.neural_network as nn
@pyimport numpy as np
To read another example of training a neural network in Engee using PyCall, please go to link.