Custom Engee support packages.Integrations

Page in progress.

Subsystem Engee.Integrations allows you to organize the interaction between Engee and external equipment using a client program and hardware support packages.

You can create your own custom support packages by extending the functionality of the Engee subsystem.Integration. The support package is a Python module that runs on a client program and returns the result back to Engee.

So that the Engee subsystem.Integration accepted custom packages, Engee automatically generates code for them in Julia language. This allows you to call your functions directly from command line Engee img 41 1 2 or from blocks.

Technology is at the heart of this mechanism. RPC (Remote Procedure Call). You are calling a function in Julia that sends arguments over the network to the client computer. There, your support package executes the corresponding Python function with these parameters and returns the result back to Engee.

Execution Architecture

It is important to understand the distribution of code execution:

Python code — is executed directly on the user’s computer during the client program process;
Julia code — runs in the Engee subsystem and coordinates the interaction.

For example, when calling:

using Main.EngeeDeviceManager.Devices.EXTDEVICE
device = EXTDEVICE.Extdevice()
device.function()

Julia code is executed in the Engee subsystem, and Python is a function function() it is executed on the client computer, and its result is returned to Engee.

Available Python Packages

When developing custom support packages, all standard Python packages are available to you, as well as the following third-party packages:

list of available Python packages

Package	Description
aiohttp	Asynchronous HTTP client/server framework (asyncio)
autoflake	Deletes unused imports and variables
bandit	Static Python Code Security Analyzer
beartype	Fast hybrid type checking at runtime
black	An uncompromising code formatter
certifi	Mozilla’s CA Bundle Package
cffi	Interface of external functions for invoking C code from Python
docformatter	Formats documentation strings in accordance with PEP 257
flake8	Modular source Code Checker: pep8 pyflakes and others
gpib-ctypes	GPIB interface for Python, implemented using ctypes
hid	Ctypes bindings for hidapi
httpx	Next generation HTTP client
intelhex	Python library for manipulating Intel HEX files
isort	Utility/library for sorting Python imports
jinja2	A very fast and expressive template engine
jupyter-client	Jupyter protocol implementation and client libraries
jupyter-core	Jupyter Basic Package
jwcrypto	Implementation of JOSE Web standards
mdurl	Markdown URL Utilities
msgpack	The MessagePack Serializer
multidict	Implementation of multidict
numpy	A fundamental package for computing arrays in Python
ordered-set	An OrderedSet that remembers its order
patchelf	A utility for changing dynamic linker and RPATH ELF executable files
pathspec	A utility for matching file paths in the gitignore style
platformdirs	Identifying suitable platform-specific directories
pydantic	Data validation using Python type hints
pydantic-core	The main functionality for Pydantic validation and serialization
pydantic-settings	Managing settings using Pydantic
pyduinocli	The wrapper around the arduino-cli
pyflakes	Passive Python Program Checker
pymodbus	Full-featured Modbus protocol stack in python
pyserial	Python Serial Port Extension
pyusb	The USB access module from Python
pyvisa	Python VISA bindings for GPIB, RS232, TCPIP and USB tools
pyvisa-py	Pure Python implementation of the VISA library
pyyaml	YAML Parser and Emitter for Python
pyzmq	Python bindings for 0MQ
redis	Python client for Redis database and key-value storage
requests	HTTP for people in Python
setuptools	Easily download, build, install, update, and remove Python packages.
toml	Python Library for Tom’s Obvious, Minimal Language
untokenize	Converts tokens to source code (while preserving spaces)
urllib3	HTTP library with thread-safe connection pool, file sending, etc.

Package

Description

aiohttp

Asynchronous HTTP client/server framework (asyncio)

autoflake

Deletes unused imports and variables

bandit

Static Python Code Security Analyzer

beartype

Fast hybrid type checking at runtime

black

An uncompromising code formatter

certifi

Mozilla’s CA Bundle Package

cffi

Interface of external functions for invoking C code from Python

docformatter

Formats documentation strings in accordance with PEP 257

flake8

Modular source Code Checker: pep8 pyflakes and others

gpib-ctypes

GPIB interface for Python, implemented using ctypes

hid

Ctypes bindings for hidapi

httpx

Next generation HTTP client

intelhex

Python library for manipulating Intel HEX files

isort

Utility/library for sorting Python imports

jinja2

A very fast and expressive template engine

jupyter-client

Jupyter protocol implementation and client libraries

jupyter-core

Jupyter Basic Package

jwcrypto

Implementation of JOSE Web standards

mdurl

Markdown URL Utilities

msgpack

The MessagePack Serializer

multidict

Implementation of multidict

numpy

A fundamental package for computing arrays in Python

ordered-set

An OrderedSet that remembers its order

patchelf

A utility for changing dynamic linker and RPATH ELF executable files

pathspec

A utility for matching file paths in the gitignore style

platformdirs

Identifying suitable platform-specific directories

pydantic

Data validation using Python type hints

pydantic-core

The main functionality for Pydantic validation and serialization

pydantic-settings

Managing settings using Pydantic

pyduinocli

The wrapper around the arduino-cli

pyflakes

Passive Python Program Checker

pymodbus

Full-featured Modbus protocol stack in python

pyserial

Python Serial Port Extension

pyusb

The USB access module from Python

pyvisa

Python VISA bindings for GPIB, RS232, TCPIP and USB tools

pyvisa-py

Pure Python implementation of the VISA library

pyyaml

YAML Parser and Emitter for Python

pyzmq

Python bindings for 0MQ

redis

Python client for Redis database and key-value storage

requests

HTTP for people in Python

setuptools

Easily download, build, install, update, and remove Python packages.

toml

Python Library for Tom’s Obvious, Minimal Language

untokenize

Converts tokens to source code (while preserving spaces)

urllib3

HTTP library with thread-safe connection pool, file sending, etc.

An example of creating a multi-file support package

For your convenience, we have prepared archive with the finished project structure and code examples. You will find the complete folder structure in the archive. devices and targets with working examples.

Let’s create a support package with our own file hierarchy. To do this, we use a template folder. module, which has directories inside it devices and targets.

Device is an arbitrary user class that does not interact with Engee models and does not use their data.
Target is a user class that interacts with the Engee model, receives data from it for processing, and executes on another platform (microcontroller, separate computer, etc.).

In our example, we will create a device with a multi-file structure.

Create in the folder devices a new folder extdevice.
Inside extdevice create a file extdevice.py with the following code:

import time
from devices.base_device import BaseDevice
from .models import DeviceConfig, CalculationResult

class Extdevice(BaseDevice):
    def __init__(self, device_id: int, calibration_factor: float) -> None:
        self.device_id = device_id
        self.calibration_factor = calibration_factor

    def __del__(self) -> None:
        pass

    def complex_calculation(self, config: DeviceConfig) -> CalculationResult:
        # Complex calculations using configuration
        result_value = (config.parameter_a * config.parameter_b +
                       config.parameter_c) * self.calibration_factor

        return CalculationResult(
            success=True,
            value=result_value,
            timestamp=time.time()
        )

    def get_status(self) -> str:
        return f"Device {self.device_id} operational with factor {self.calibration_factor}"

Create a file models.py in the same folder extdevice:

from devices.base_models import BaseModel

class DeviceConfig(BaseModel):
    parameter_a: float
    parameter_b: int
    parameter_c: float

class CalculationResult(BaseModel):
    success: bool
    value: float
    timestamp: float

For custom RPC classes, it is mandatory to inherit from BaseDevice (if it’s a device) or from BaseTarget (if this is a target)

Data structures must be inherited from BaseModel!

All methods must have complete type annotations for parameters and return values. For example:

# WRONG - without annotations
def __init__(self, param1, param2):
    pass

# THAT'S RIGHT - with full annotations
def __init__(self, param1: int, param2: float) -> None:
    pass

def calculate(self, x: float, y: int) -> str:
    return "result"

Without annotations, the system will not be able to correctly generate Julia code and convert data types between Python and Julia.

You can clearly see how the mechanism described above for creating your own support package is applied in practice to work with real hardware using the example of the Community.: Development of a hardware support package for Engee.Integration.

Debugging support packages

You can use the built-in logging system to debug your extension. Add to your code:

from main_logger import MainLogger

class Extdevice(BaseDevice):
    def __init__(self, device_id: int, calibration_factor: float) -> None:
        self.logger = MainLogger()
        self.logger.info(f"Initializing device {device_id}")
        self.device_id = device_id
        self.calibration_factor = calibration_factor

    def complex_calculation(self, config: DeviceConfig) -> CalculationResult:
        self.logger.debug("Starting complex calculation")
        # ... your code ...
        self.logger.info("Calculation completed successfully")
        return result

Available logging levels:

logger.debug("message") — debugging information;
logger.info ("message") — information messages;
logger.warning("message") — Warnings;
logger.error("message") — mistakes.

The messages will be displayed in the graphical interface of the client program in the logs panel. You will also find examples of logging usage in the archive attached above.

Registration and use of the Engee support package

Before registering, make sure that your folder structure matches the example from the archive.

After creating the support package, you need to generate the appropriate code on Julia and register the package in Engee.

First, let’s look at the list of existing devices.:

engee> using Main.EngeeDeviceManager.Devices.

CAN                   COM                   ECHO                  HID
HTTP                  L502BOARD             LOGITECHG29WHEEL      MODBUSMASTER
SOCKET                TFLEXDOCS             THRUSTMASTERJOYSTICK  THRUSTMASTERTHROTTLE
UM                    UTILS                 VISA

Our support package EXTDEVICE Not here yet.

We will specify the path to our folder module and download the package (for more information about the methods used below, see the article Software management for working with the file system and support packages):

engee> module_path = "/path/to/module"
engee> using Main.EngeeDeviceManager.Devices.UTILS
engee> using Main.EngeeDeviceManager.UTILS_API
engee> utils = UTILS.Utils()
engee> UTILS_API.loadExtension(utils, module_path)

If everything was successful, the following message will appear in the logs of the client program:

INFO     | Extension with name: module was loaded successfully!

Support Package update procedure:

If you have only changed the implementation of existing functions (without changing signatures, return types, or function names):

It is enough to call UTILS_API.loadExtension(utils, module_path) again;
OR restart the client program, and the package will load automatically.

If you have added new functions, classes, or models (BaseModel inheritors):

Call UTILS_API.loadExtension(utils, module_path);
Then run engee.clear_all();
Restart the Engee subsystem.Integrations: engee.package.start("Engee-Device-Manager");

The support package is now registered. Reboot the Engee core and restart the Engee subsystem.Integrations:

engee> engee.clear_all()
# Waiting for the kernel reboot
engee> engee.package.start("Engee-Device-Manager")

Let’s look at the list of devices again:

engee> using Main.EngeeDeviceManager.Devices.

CAN                   COM                   ECHO                  EXTDEVICE
HID                   HTTP                  L502BOARD             LOGITECHG29WHEEL
MODBUSMASTER          SOCKET                TFLEXDOCS             THRUSTMASTERJOYSTICK
THRUSTMASTERTHROTTLE  UM                    UTILS                 VISA

Now the support package EXTDEVICE in the list and it can be used:

engee> using Main.EngeeDeviceManager.Devices.EXTDEVICE
engee> device = EXTDEVICE.Extdevice(123, 1.5)
engee> device.get_status()
"Device 123 operational with factor 1.5"

engee> config = EXTDEVICE.DeviceConfig(2.5, 10, 3.14)
engee> result = device.complex_calculation(config)

Your support package will be automatically downloaded every time you run the client program. If it is no longer needed, then you can delete it from auto-upload.:

engee> UTILS_API.deleteExtension(utils, module_path)

Function deleteExtension removes the support package from the startup list. For the changes to take effect, the Engee subsystem must be completely restarted.Integration.

Thus, we created a "bridge" between Engee and the custom Python code, performed calculations on the client program and got the result in Engee.