Creating tools with Python🔗
Minimal example🔗
The following code snippet provides a minimal implementation of a Python
tool plugin. It implements a tool that passes an input image to its
output and registers it to the VisionAppster runtime. If you save this
code to $HOME/VisionAppster/plugins/tool/mytoolplugin.py
, you will
find the tool in the Builder’s tool box as “PassImage”.
import visionappster as va
class PassImage:
def process(self,
inputs: [('image', va.Image, va.Image())],
outputs: [('image', va.Image)]):
outputs.image = inputs.image
va.Tool.publish('com.acme.python/0', PassImage)
To be able to use the VisionAppster Python interface, the code imports
the visionappster
package. This package is built into the
VisionAppster runtime.
Every tool is represented as a Python class. When a tool is added to a
processing graph, an instance of it will be created. The simple example
tool here has no constructor function (__init__
) should it need to
initialize instance variables. The name of the class will be shows as
the name of the tool in the Builder.
The tool class must define an instance method called process
. The
first argument to the method is self
as per Python’s conventions.
The next two arguments declare the input and output parameters of the
tool. Python’s type annotations are used to provide a rich description
of the interface. The example tool has a single input parameter called
“image” whose type is Image
. The parameter has a default value: an
empty image. The single output parameter is declared the same way except
that it has no default value.
The body of the process
function simply copies the value of the
image
input parameter to the output. In a more realistic situation,
the tool would make calls to e.g. NumPy and OpenCV functions.
Finally, the VisionAppster runtime must be made aware of the existence
of the tool. This happens by calling the Tool.publish
function. The
first argument is the full component
ID of the tool plugin. The rest
are the names of the Python classes to be registered in that plugin.
NOTE: If you package your Python code into a component, make sure that the full component ID you use in the Python code (here “com.acme.python/0”) matches that of the component you create. Python imports may fail to work otherwise.
Describing a tool’s interface🔗
In essence, each tool in a VisionAppster processing graph is a function that will be called by the runtime when needed. At a minimum, the runtime only needs to know the names and types of input and output parameters as shown in the simple example above. A richer description of the interface however makes using the tool more convenient as it allows the Builder to automatically provide suitable UI components for editing the tool’s parameters, for example.
The example below shows how meta-data can be used to provide a the Builder hints on how to best handle each parameter.
class RichMetaData:
"""Tool with Rich Meta-Data
The first line of the documentation comment is used as the
default user-visible name of the tool. If there is no
documentation comment, the class name will be used.
The rest of the documentation comment will be shown
as in-line help in the Builder.
"""
# Tags can be used to categorize tools in the tool box.
__va_tags__ = ['pass']
def process(self,
inputs: [('int', int, 1, {'min': 0, 'max': 100}),
('float', float, 1.0,
{'choices': [{'name': 'Value 1', 'value': 1},
{'name': 'Value 2', 'value': 2}]}),
('bool', bool, True),
('str', str, 'ABC'),
('image', va.Image, va.Image(va.Image.GRAY8, 1, 2)),
('tensor', va.Tensor, va.Tensor(va.Tensor.FLOAT, [1, 2])),
('intMatrix', va.Matrix.Int32, va.Matrix.Int32(2, 2),
{'minRows': 1}),
('floatMatrix', va.Matrix.Float, va.Matrix.Float(1, 1)),
('doubleMatrix', va.Matrix.Double, va.Matrix.Double(4, 4),
{'typeName': 'Matrix<double>/frame'}),
('boolMatrix', va.Matrix.Bool, va.Matrix.Bool(1, 1)),
('floatComplexMatrix', va.Matrix.FloatComplex,
va.Matrix.FloatComplex(1, 1)),
('doubleComplexMatrix', va.Matrix.DoubleComplex,
va.Matrix.DoubleComplex(1, 1)),
('list', list, []),
('dict', dict, {'key': 'value'})],
outputs: [('int', int),
('float', float),
('bool', bool),
('str', str),
('image', va.Image),
('tensor', va.Tensor),
('intMatrix', va.Matrix.Int32),
('floatMatrix', va.Matrix.Float),
('doubleMatrix', va.Matrix.Double,
{'typeName': 'Matrix<double>/frame'}),
('boolMatrix', va.Matrix.Bool),
('floatComplexMatrix', va.Matrix.FloatComplex),
('doubleComplexMatrix', va.Matrix.DoubleComplex),
('list', list),
('dict', dict)]):
"""
Copies every attribute from the input object to the
corresponding attribute in the output object.
"""
for attr in dir(inputs):
setattr(outputs, attr, getattr(inputs, attr))
The type annotations for inputs
and outputs
are provided as an
array of tuples. Semi-formally, each input declaration is a tuple
(name, type, default?, meta?)
, where
name
is the name of the input, which must be unique among inputs. The name must be a pythonstr
object.type
is a Python type class that defines the type of the parameter.default
is optional. If it is given and notNone
, the input is optional. The default value will be passed to the function if the input is not connected in the processing graph. The value must be an instance of the Python type class given as thetype
.meta
is an optionaldict
instance defining meta-data for the input. If you want to define meta-data but no default value, giveNone
as thedefault
.
Output declarations are otherwise similar, but they cannot contain a default value.
When a tool is invoked, the runtime passes the process
function
inputs
and outputs
objects that have attributes matching the
declared input and output names. The value of an input comes either from
an earlier function in the processing graph or from the default value,
if the input is unconnected. The function implementation must write a
value to all outputs.
Data type conversions🔗
Compared to C, Python provides fewer built-in data types. The following table lists the supported conversions between Python data types and VisionAppster C types.
Python type |
C type |
---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Python’s int
is a variable-length, signed integer type. If you
specify it as the type of a parameter, the C type used will be
int64_t
. Communication with JavaScript code and subsequently some
user interfaces further limits the available range to 53 bits. Unless
you really need larger numbers, please make sure your integers occupy no
more than 53 binary digits.
When a va_array
is passed to a Python tool, the VisionAppster
runtime uses a special Array
class that works like a Python list
object. Similarly, an input parameter of type va_object
will be
represented by an instance of Object
, which works like a Python
dict
. These classes are thin wrappers that hold a reference to the
original data to avoid copying it. When writing data to outputs, it is
OK to use the list
and dict
types. In parameter type
definitions, dict
and va.Object
are interchangeable and work the
same way. Similarly, you can use either list
or va.Array
for the
same effect.
Matrices are special in that every matrix type is represented by a
different Python type class. Matrices can be created by constructing a
typed instance directly or by passing a type id to the generic factory
function va.Matrix
:
from visionappster import Matrix
# These are equivalent
mat1 = Matrix.Int32(3, 3)
mat2 = Matrix(Matrix.INT32, 3, 3)
Multi-dimensional array-like types (Image
, Tensor
and
Matrix
) implement the Python buffer protocol. This means that their
data buffers can be directly used as NumPy arrays, for example. This
makes passing data between the VisionAppster runtime and Python code
efficient. The following code snippet illustrates how the Image
type
can be used as an input and output argument to NumPy calls.
import visionappster as va
import numpy as np
class ImageProcessing:
def process(self,
inputs: [('image', va.Image, va.Image()),
('threshold', int, 127)],
outputs: [('binarized', va.Image),
('negated', va.Image)]):
# Shallow reference to input image
in_arr = np.array(inputs.image, copy=False)
# Convert RGB to gray by taking average of three channels
gray = in_arr if in_arr.ndim == 2 else np.mean(in_arr, axis=2)
# Initialize an image with cloned header ...
outputs.binarized = va.Image.uninitialized(va.Image.GRAY8, inputs.image)
# ... and use it as an output array (shallow reference).
# This is the most efficient and thus preferred way.
np.greater(gray, inputs.threshold, np.array(outputs.binarized, copy=False))
outputs.binarized.channel_max = 1
# Construct a new Image out of an existing Numpy array. The ownership
# of the array cannot be transferred. Thus, this requires copying every
# pixel.
outputs.negated = va.Image(inputs.image.type, np.subtract(255, in_arr))
Runtime errors🔗
If the tool received inputs it cannot process, it needs to cause a
runtime error. This happens by simply raising a Python Exception
as
shown by the example below:
class Error:
"""Error
This tool does nothing by default but causes a runtime error
if the input string is non-empty.
"""
def process(self,
inputs: [('message', str, 'This is an error')],
outputs: []):
if inputs.message:
raise Exception(inputs.message)
Tool state🔗
As described in Execution modes, the
VisionAppster runtime automatically runs tools in parallel. This also
applies to Python tools. The evaluation of Python bytecode is currently
serialized, but native extension calls from Python code open up
parallelization possibilities. This has the consequence that Python
tools should be made reentrant just like C tools. That is, the outcome
of the process
function should depend on its input parameters only.
Sometimes, the tool may need to one-time initialization that would be
too costly to do each time the process
function is called. Such
initialization made in the constructor (__init__
function) of the
tool class do not necessarily render the tool instance non-reentrant. If
the process
function only reads the values of instance variables,
you are on the safe side. In the same vein, referring to immutable
global variables is OK.
It may however be necessary to maintain modifiable state in the tool. In
such a case the order of process
calls makes a difference and the
calls cannot be parallelized. To make the runtime aware of this
restriction, the tool class must list the allowed execution modes in a
static class attribute called __va_execution_mode__
. Its value may
be one of the constants defined in va.Tool.ExecutionMode
or a
list
containing one or more of them.
If the tool maintains internal state, it may need to know when the
processing graph starts and stops. If the tool class defines a function
called notify
, it will be called by the runtime whenever the state
of the tool changes.
Finally, the tool class can be derived from va.Tool
. This will give
the code access to self.inputs
and self.outputs
. These objects
reflect the statuses of the tool’s inputs and outputs in the processing
graph. For example, self.inputs.trigger.connected
is a Boolean flag
that indicates whether the input parameter called trigger
is
connected in the graph.
The tool below illustrates how to manage internal state in a tool implementation:
import visionappster as va
class StateChanger(va.Tool):
"""State Changer
Counts the number of times the tool instance has been started,
stopped and invoked. Also inspects connection statuses and
default values.
The StateChanger class derives from va.Tool to gain super
powers, a.k.a access to self.inputs and self.outputs.
"""
# This is not necessary as the existence of `notify` will make
# the tool non-threaded by default. The default value is
# [Tool.NON_THREADED, Tool.SINGLE_THREADED, Tool.MULTI_THREADED]
__va_execution_mode__ = va.Tool.NON_THREADED
def __init__(self):
# NOTE: Superclass constructor must be called explicitly!
va.Tool.__init__(self)
# Initialize instance variables
self.start_count = 0
self.stop_count = 0
self.connected_inputs = 0
self.process_count = 0
def notify(self, event):
"""This function is called by the runtime when the
processing graph is started or stopped. `event` is
either Tool.STOPPED or Tool.STARTED.
"""
if event == va.Tool.STOPPED:
self.stop_count += 1
elif event == va.Tool.STARTED:
self.start_count += 1
self.connected_inputs = 0
# Count the number of connected inputs by iterating
# the `inputs` object.
for name in dir(self.inputs):
if getattr(self.inputs, name).connected:
self.connected_inputs += 1
def process(self,
inputs: [('trigger', int),
('another', int, 123)],
outputs: [('starts', int),
('stops', int),
('calls', int),
('connectedInputs', int),
('defaultValue', int)]):
outputs.starts = self.start_count
outputs.stops = self.stop_count
# This makes the tool stateful and botches parallelization.
self.process_count += 1
outputs.calls = self.process_count
outputs.connectedInputs = self.connected_inputs
# This reads the current default value of `another`,
# which may be different from `inputs.another` if
# the input is connected.
outputs.defaultValue = self.inputs.another.default_value