Creating tools with Python

Minimal example

The following code snippet provides a minimal implementation of a Python tool plugin. It implements a tool that passes an input image to its output and registers it to the VisionAppster runtime. If you save this code to $HOME/VisionAppster/plugins/tool/mytoolplugin.py, you will find the tool in the Builder’s tool box as “PassImage”.

import visionappster as va

class PassImage:
    def process(self,
                inputs: [('image', va.Image, va.Image())],
                outputs: [('image', va.Image)]):
        outputs.image = inputs.image

va.Tool.publish('com.acme.python/0', PassImage)

To be able to use the VisionAppster Python interface, the code imports the visionappster package. This package is built into the VisionAppster runtime.

Every tool is represented as a Python class. When a tool is added to a processing graph, an instance of it will be created. The simple example tool here has no constructor function (__init__) should it need to initialize instance variables. The name of the class will be shows as the name of the tool in the Builder.

The tool class must define an instance method called process. The first argument to the method is self as per Python’s conventions. The next two arguments declare the input and output parameters of the tool. Python’s type annotations are used to provide a rich description of the interface. The example tool has a single input parameter called “image” whose type is Image. The parameter has a default value: an empty image. The single output parameter is declared the same way except that it has no default value.

The body of the process function simply copies the value of the image input parameter to the output. In a more realistic situation, the tool would make calls to e.g. NumPy and OpenCV functions.

Finally, the VisionAppster runtime must be made aware of the existence of the tool. This happens by calling the Tool.publish function. The first argument is the full component ID of the tool plugin. The rest are the names of the Python classes to be registered in that plugin.

NOTE: If you package your Python code into a component, make sure that the full component ID you use in the Python code (here “com.acme.python/0”) matches that of the component you create. Python imports may fail to work otherwise.

Describing a tool’s interface

In essence, each tool in a VisionAppster processing graph is a function that will be called by the runtime when needed. At a minimum, the runtime only needs to know the names and types of input and output parameters as shown in the simple example above. A richer description of the interface however makes using the tool more convenient as it allows the Builder to automatically provide suitable UI components for editing the tool’s parameters, for example.

The example below shows how meta-data can be used to provide a the Builder hints on how to best handle each parameter.

class RichMetaData:
    """Tool with Rich Meta-Data

       The first line of the documentation comment is used as the
       default user-visible name of the tool. If there is no
       documentation comment, the class name will be used.

       The rest of the documentation comment will be shown
       as in-line help in the Builder.
    """

    # Tags can be used to categorize tools in the tool box.
    __va_tags__ = ['pass']

    def process(self,
                inputs: [('int', int, 1, {'min': 0, 'max': 100}),
                         ('float', float, 1.0,
                          {'choices': [{'name': 'Value 1', 'value': 1},
                                       {'name': 'Value 2', 'value': 2}]}),
                         ('bool', bool, True),
                         ('str', str, 'ABC'),
                         ('image', va.Image, va.Image(va.Image.GRAY8, 1, 2)),
                         ('tensor', va.Tensor, va.Tensor(va.Tensor.FLOAT, [1, 2])),
                         ('intMatrix', va.Matrix.Int32, va.Matrix.Int32(2, 2),
                          {'minRows': 1}),
                         ('floatMatrix', va.Matrix.Float, va.Matrix.Float(1, 1)),
                         ('doubleMatrix', va.Matrix.Double, va.Matrix.Double(4, 4),
                          {'typeName': 'Matrix<double>/frame'}),
                         ('boolMatrix', va.Matrix.Bool, va.Matrix.Bool(1, 1)),
                         ('floatComplexMatrix', va.Matrix.FloatComplex,
                          va.Matrix.FloatComplex(1, 1)),
                         ('doubleComplexMatrix', va.Matrix.DoubleComplex,
                          va.Matrix.DoubleComplex(1, 1)),
                         ('list', list, []),
                         ('dict', dict, {'key': 'value'})],
                outputs: [('int', int),
                          ('float', float),
                          ('bool', bool),
                          ('str', str),
                          ('image', va.Image),
                          ('tensor', va.Tensor),
                          ('intMatrix', va.Matrix.Int32),
                          ('floatMatrix', va.Matrix.Float),
                          ('doubleMatrix', va.Matrix.Double,
                           {'typeName': 'Matrix<double>/frame'}),
                          ('boolMatrix', va.Matrix.Bool),
                          ('floatComplexMatrix', va.Matrix.FloatComplex),
                          ('doubleComplexMatrix', va.Matrix.DoubleComplex),
                          ('list', list),
                          ('dict', dict)]):
        """
           Copies every attribute from the input object to the
           corresponding attribute in the output object.
        """
        for attr in dir(inputs):
            setattr(outputs, attr, getattr(inputs, attr))

The type annotations for inputs and outputs are provided as an array of tuples. Semi-formally, each input declaration is a tuple (name, type, default?, meta?), where

• name is the name of the input, which must be unique among inputs. The name must be a python str object.

• type is a Python type class that defines the type of the parameter.

• default is optional. If it is given and not None, the input is optional. The default value will be passed to the function if the input is not connected in the processing graph. The value must be an instance of the Python type class given as the type.

• meta is an optional dict instance defining meta-data for the input. If you want to define meta-data but no default value, give None as the default.

Output declarations are otherwise similar, but they cannot contain a default value.

When a tool is invoked, the runtime passes the process function inputs and outputs objects that have attributes matching the declared input and output names. The value of an input comes either from an earlier function in the processing graph or from the default value, if the input is unconnected. The function implementation must write a value to all outputs.

Data type conversions

Compared to C, Python provides fewer built-in data types. The following table lists the supported conversions between Python data types and VisionAppster C types.

Python type	C type
int	int64_t
float	double
bool	va_bool
str	va_string
list	va_array
dict	va_object
va.Array	va_array
va.Object	va_object
va.Image	va_image
va.Tensor	va_tensor
va.Matrix.Int32	va_imatrix
va.Matrix.Float	va_fmatrix
va.Matrix.Double	va_dmatrix
va.Matrix.Bool	va_bmatrix
va.Matrix.FloatComplex	va_fcmatrix
va.Matrix.DoubleComplex	va_dcmatrix

Python’s int is a variable-length, signed integer type. If you specify it as the type of a parameter, the C type used will be int64_t. Communication with JavaScript code and subsequently some user interfaces further limits the available range to 53 bits. Unless you really need larger numbers, please make sure your integers occupy no more than 53 binary digits.

When a va_array is passed to a Python tool, the VisionAppster runtime uses a special Array class that works like a Python list object. Similarly, an input parameter of type va_object will be represented by an instance of Object, which works like a Python dict. These classes are thin wrappers that hold a reference to the original data to avoid copying it. When writing data to outputs, it is OK to use the list and dict types. In parameter type definitions, dict and va.Object are interchangeable and work the same way. Similarly, you can use either list or va.Array for the same effect.

Matrices are special in that every matrix type is represented by a different Python type class. Matrices can be created by constructing a typed instance directly or by passing a type id to the generic factory function va.Matrix:

from visionappster import Matrix
# These are equivalent
mat1 = Matrix.Int32(3, 3)
mat2 = Matrix(Matrix.INT32, 3, 3)

Multi-dimensional array-like types (Image, Tensor and Matrix) implement the Python buffer protocol. This means that their data buffers can be directly used as NumPy arrays, for example. This makes passing data between the VisionAppster runtime and Python code efficient. The following code snippet illustrates how the Image type can be used as an input and output argument to NumPy calls.

import visionappster as va
import numpy as np

class ImageProcessing:
    def process(self,
                inputs: [('image', va.Image, va.Image()),
                         ('threshold', int, 127)],
                outputs: [('binarized', va.Image),
                          ('negated', va.Image)]):
        # Shallow reference to input image
        in_arr = np.array(inputs.image, copy=False)

        # Convert RGB to gray by taking average of three channels
        gray = in_arr if in_arr.ndim == 2 else np.mean(in_arr, axis=2)

        # Initialize an image with cloned header ...
        outputs.binarized = va.Image.uninitialized(va.Image.GRAY8, inputs.image)
        # ... and use it as an output array (shallow reference).
        # This is the most efficient and thus preferred way.
        np.greater(gray, inputs.threshold, np.array(outputs.binarized, copy=False))
        outputs.binarized.channel_max = 1

        # Construct a new Image out of an existing Numpy array. The ownership
        # of the array cannot be transferred. Thus, this requires copying every
        # pixel.
        outputs.negated = va.Image(inputs.image.type, np.subtract(255, in_arr))

Runtime errors

If the tool received inputs it cannot process, it needs to cause a runtime error. This happens by simply raising a Python Exception as shown by the example below:

class Error:
    """Error

       This tool does nothing by default but causes a runtime error
       if the input string is non-empty.
    """
    def process(self,
                inputs: [('message', str, 'This is an error')],
                outputs: []):
        if inputs.message:
            raise Exception(inputs.message)

Tool state

As described in Execution modes, the VisionAppster runtime automatically runs tools in parallel. This also applies to Python tools. The evaluation of Python bytecode is currently serialized, but native extension calls from Python code open up parallelization possibilities. This has the consequence that Python tools should be made reentrant just like C tools. That is, the outcome of the process function should depend on its input parameters only.

Sometimes, the tool may need to one-time initialization that would be too costly to do each time the process function is called. Such initialization made in the constructor (__init__ function) of the tool class do not necessarily render the tool instance non-reentrant. If the process function only reads the values of instance variables, you are on the safe side. In the same vein, referring to immutable global variables is OK.

It may however be necessary to maintain modifiable state in the tool. In such a case the order of process calls makes a difference and the calls cannot be parallelized. To make the runtime aware of this restriction, the tool class must list the allowed execution modes in a static class attribute called __va_execution_mode__. Its value may be one of the constants defined in va.Tool.ExecutionMode or a list containing one or more of them.

If the tool maintains internal state, it may need to know when the processing graph starts and stops. If the tool class defines a function called notify, it will be called by the runtime whenever the state of the tool changes.

Finally, the tool class can be derived from va.Tool. This will give the code access to self.inputs and self.outputs. These objects reflect the statuses of the tool’s inputs and outputs in the processing graph. For example, self.inputs.trigger.connected is a Boolean flag that indicates whether the input parameter called trigger is connected in the graph.

The tool below illustrates how to manage internal state in a tool implementation:

import visionappster as va

class StateChanger(va.Tool):
    """State Changer

       Counts the number of times the tool instance has been started,
       stopped and invoked. Also inspects connection statuses and
       default values.

       The StateChanger class derives from va.Tool to gain super
       powers, a.k.a access to self.inputs and self.outputs.
    """

    # This is not necessary as the existence of `notify` will make
    # the tool non-threaded by default. The default value is
    # [Tool.NON_THREADED, Tool.SINGLE_THREADED, Tool.MULTI_THREADED]
    __va_execution_mode__ = va.Tool.NON_THREADED

    def __init__(self):
        # NOTE: Superclass constructor must be called explicitly!
        va.Tool.__init__(self)
        # Initialize instance variables
        self.start_count = 0
        self.stop_count = 0
        self.connected_inputs = 0
        self.process_count = 0

    def notify(self, event):
        """This function is called by the runtime when the
           processing graph is started or stopped. `event` is
           either Tool.STOPPED or Tool.STARTED.
        """
        if event == va.Tool.STOPPED:
            self.stop_count += 1
        elif event == va.Tool.STARTED:
            self.start_count += 1
        self.connected_inputs = 0
        # Count the number of connected inputs by iterating
        # the `inputs` object.
        for name in dir(self.inputs):
            if getattr(self.inputs, name).connected:
                self.connected_inputs += 1

    def process(self,
                inputs: [('trigger', int),
                         ('another', int, 123)],
                outputs: [('starts', int),
                          ('stops', int),
                          ('calls', int),
                          ('connectedInputs', int),
                          ('defaultValue', int)]):
        outputs.starts = self.start_count
        outputs.stops = self.stop_count
        # This makes the tool stateful and botches parallelization.
        self.process_count += 1
        outputs.calls = self.process_count
        outputs.connectedInputs = self.connected_inputs
        # This reads the current default value of `another`,
        # which may be different from `inputs.another` if
        # the input is connected.
        outputs.defaultValue = self.inputs.another.default_value