Working with Python
General information
Python is a high—level, interpreted, general-purpose programming language. In this article, we will look at the use of Python within the Engee platform, namely:
-
Python integration via the PyCall.jl package — write Python code in Engee scripts thanks to the PyCall package;
-
Features when working with interactive Engee scripts — learn the specifics of working with Engee scripts and install the necessary Python packages;
-
Working with Python variables in the Engee environment — work with Python variables in Engee scripts;
-
Launching Python notebooks in Engee — run your own Python laptops by transferring them to Engee directly from the external development environment;
-
Python neural networks and their integration with Engee models — create and train neural networks with Python libraries based on Engee models.
Calling Python in Engee
Python integration via the PyCall.jl package
|
PyCall.jl is a Julia package that allows you to call functions and objects from Python directly in Julia code. It provides an interface between Julia and Python and allows you to import Python modules, call their functions, create objects, and write code with them in Julia. For more information about supported Python invocation commands, literals, and packages in Julia, see the article PyCall.jl. PyCall is built into Engee by default, you do not need to install or import it. |
Engee uses a computing core Julia, but you can write Python code in Engee scripts (see Features when working with interactive Engee scripts) using the following methods:
-
Wrapping Python code with the string literal
py"""…"""based on the PyCall package. This literal executes a string as Python program code, without returning a value and executing the passed code. -
Wrapping Python code with the string literal
py"…"based on PyCall. This literal takes a string as an argument and executes it as a Python expression. -
Using the commands
pyimport,pyeval', `pycall,@pycall,pybuiltin,@pywith,@pyinclude,@pysym, `@pydef'.
Let’s look at these methods using examples.
You can write code in Julia using the familiar Python syntax. To do this, wrap your Python code with the literals py"""…""" and py"…", for example:
py""" # wraps in a literal
import numpy as np # imports the numpy package
"""
a = py"np.zeros(2)" # wraps in a literal, which creates a one-dimensional array of two elements filled with zerosFor the pyimport command, the code will look like this:
np = pyimport("numpy")
a = np.zeros(2)In both examples, the variables a will have the same value. The difference between the examples is that in the case of the pyimport command, an np object appears in Julia that can be accessed directly.
In Julia, Python commands like import numpy as np do not work natively, but using string literals you can insert Python code directly into Julia code, provided the PyCall package is installed.
|
You can also use other Python invocation commands in Julia.:
-
pyimport — imports the Python module into Julia, allowing you to use its functions and variables in Julia code.
-
'pyeval' — executes Python string code in the Julia context, returning the result of execution.
-
'pycall' — calls a Python function with the specified arguments from Julia.
-
'@pycall` is a macro that provides a convenient syntax for calling Python functions from Julia. It is usually used more often than
pycallbecause of the simpler call. -
pybuiltin— provides access to built-in Python functions such asprint,len,range, etc. -
'@pywith` — the macro creates a Python execution context in which you can work with Python code blocks inside Julia.
-
'@pyinclude' is a macro that includes a Python code file in Julia and executes it.
-
'@pysym` is a macro that converts the Julia symbol name to the corresponding Python object.
-
'@pydef' — creates a Python class whose methods are implemented in Julia.
For example, the pybuiltin command:
pybuiltin("print")("Hello, world!")The code outputs the message Hello, world!.
Features when working with interactive Engee scripts
|
Check out an example of using Python in Engee scripts*: |
| You can familiarize yourself with the example of using Python in Engee scripts. here. |
Interactive Engee scripts (with the .ngscript extension) are a tool for working with code in the Engee environment (for more information, see Script editor). Let’s look at the features of working with interactive Engee scripts for Python:
-
Engee scripts can be created directly in Engee itself. This is the only supported format for creating interactive scripts.
-
Adding packages in one Engee script allows you to use them in other Engee scripts without having to reinstall.
-
Engee scripts support magic Python commands, such as `%run'.
To work with Python in Engee scripts, you may need many packages, for example, a scientific calculation support package. SciPy. Use the package manager to install it. Pkg and Julia syntax:
using Pkg
Pkg.add("SciPy") # will install the SciPy package in JuliaTo remove packages, use the rm command. For example, `Pkg.rm("Pluto")'. Read more in Working with Julia packages.
To find out which packages are installed in the Engee scripts, use:
import Pkg
Pkg.status()Let’s take an example. To do this, install the SciPy package and use it to find the root of the equation using the string literal py""":
py"""
from scipy.optimize import root_scalar
def func(x):
return x**2 - 4
result = root_scalar(func, method='brentq', bracket=[0, 2])
print("The result (the root of the equation):", result.root)
"""Let’s analyze the code in more detail:
-
In the line
from scipy.optimize import root_scalar, the functionroot_scalaris imported from the modulescipy.optimize. This function is designed to numerically find the roots of equations for one variable. -
In the line
def func(x):the func function is defined, for which the root is being searched. Using thereturnvalue of the equation the function returns func, and 'x` is a variable. -
In the string
result = root_scalar(func, method='brentq', bracket=[0, 2]), the functionroot_scalaris used to find the root of the equation specified by the func function. Thebrentq" method indicates that we want to use the Brent method specifically. Using the parameter `bracket=[0, 2], we set the interval for searching the root of the equation (in our case, from0to2). -
The line
print("Result (root of equation):", result.root)displays the found root of the equation. 'result.root` contains the root value found by theroot_scalarfunction.
As a result, we find the root of the equation with the help Brent’s method from the SciPy package. The value of the root of the equation will be obtained:
Result (root of the equation): 2.0Similarly, you can use other Python packages like NumPy, Matplotlib, Pandas, etc., but keep in mind that you cannot create different Python environments in Engee and switch between them.
| The only limitation on the amount of packages to install is the RAM memory that is allocated for your current session in Engee. Read more about the available memory in Starting the Engee session. |
Working with Python variables in the Engee environment
Variables created in Python notebooks (see Launching Python notebooks in Engee) or in Engee scripts using a Python call will have the PyObject type. This typing of variables is caused by the fact that data types in Python are unequal to Julia, and Python variables are assigned their own unique type to save data.
You can import variables from the Python module into Julia variables by calling methods directly. For example:
math = pyimport("math")
math.sin(math.pi / 4)As a result, we get the value: `0.7071067811865475'.
In this code, the command math = pyimport("math") is used in Julia using the PyCall package. It allows you to import the math module from the standard Python package and save it in the math variable for future use in Julia code.:
Variables of the PyObject type cannot be exported to .mat format, but they can in .jld2 (for more information, see Importing and exporting variables:
| Python variables cannot be cleaned up using the built-in Engee tools, but they can be overwritten. Session restart is required to delete variables. |
If necessary, PyObject variables can be used further in Engee scripts, but keep in mind that Python data types differ from Julia. If type conversion is necessary, we recommend that you specify the types explicitly.:
var_jl = py"int(var_py)" # var_py is a variable with the PyObject type.
println(var_jl)where var_jl' is the Julia variable. After conversion, the variable `var_py remained with the PyObject type, so we created a new variable var_jl in Julia from the old var_py of the PyObject type.
You can work in Engee scripts with PyObject type variables without setting them explicitly. For example, we have a variable deg with the PyObject type. We can access it via the println(deg) command, and, for example, add 10:
println(deg + 10)
# we get PyObject 11.The variable has retained its PyObject type and we have successfully performed the mathematical operation.
Launching Python notebooks in Engee
You can download your Python notebook (Jupyter Notebook) to Engee and run it in script editor. Let’s look at the features of working with Python laptops:
-
Python notebooks can be created directly in Engee using script editor
Engee. To do this, open the editor, click +, then Create script and select the format .ipynb:
→ 
-
Packages installed in Engee scripts cannot be imported into Python notebooks. The necessary packages must be re-installed for each new Python laptop.
-
From a Python notebook in Engee, you cannot access Julia’s magic commands or the two user input functions 'input()` and `getpass()'.
-
Python laptops do not support code delay (for example, through the
sleepfunction) and output the result of code execution instantly. -
Python scripts (the .py extension) cannot be used.
-
You cannot use the conda package management tool, only pip.
To install packages in Python notebooks, use the magic command '!pip`:
!pip install scipy| Magic commands (magic commands) – special commands in various software development tools. These commands are designed to perform additional actions that can be useful in interactive programming and data analysis. |
If necessary, install packages from the WHL file.:
!pip install path/to/file.whlWHL is an archived format for packaging and distributing Python packages. To install the WHL file, first transfer it to File Browser Engee. WHL files contain precompiled binary package files, which makes it easier to install and manage project dependencies. You can find WHL format files on the official websites of the corresponding Python packages.
To find out which packages are installed in Python notebooks, use:
!pip list| Check out the example of using .ipynb in Engee: |
Importing custom modules
In order to import your modules into Python notebooks, you need to make sure that the path to them is added to the sys.path variable. By default, this variable contains the paths to the standard libraries, installed packages, and the current working directory of the Python interpreter, rather than the directory where the script is located. To find out the current directory, you can use the os.getcwd() function. If necessary, the current directory can be changed using `os.chdir("path")'. If the module is located in another directory, its path must be added to sys.path manually. This can be done using the command:
import sys
sys.path.append('/path/to/your/module')After adding the path to the sys.path, the module can be imported and its functions and data can be used. If you try to import a module from a directory that has not been added to sys.path, the error 'ModuleNotFoundError` will occur.
Python neural networks and their integration with Engee models
|
Check out the example for training a neural network using the scikit-learn library and generating training data from the model: |
To work with neural networks using Python, Engee uses the PyCall package and Python invocation commands in Julia. For example, for the library scikit-learn:
# installing the sklearn library in Julia
using Pkg
Pkg.add("ScikitLearn")
# importing necessary modules from Python to Julia
@pyimport sklearn.datasets as datasets
@pyimport sklearn.neural_network as nn
@pyimport numpy as npTo get acquainted with another example of neural network training in Engee using PyCall, go to link.