Tool APIs🔗

The VisionAppster platform comes with a large set of system tools for many common vision tasks. Due to the high variability of vision apps, no prebuilt tool set can however cover everything. The VisionAppster Store often helps, but sometimes you just need to resort to plain old coding. The existing solutions may not be fast or accurate enough, or you already have code that does the same thing. Or maybe you need to interact with some special hardware.

VisionAppster Tool APIs are used for implementing new vision algorithms or other functionality and integrating existing code to the VisionAppster platform. The APIs make it easy to build dynamically loadable libraries containing new tools that can then be used as building blocks in vision applications.

All native tool APIs are built on top of a C API. Another option is to write tools in Python using the Python API.

An important (perhaps obvious) thing to note is that unlike the system tools that come with the VisionAppster runtime, user-defined tools are not automatically available everywhere. It is the developer’s responsibility to build the tool plugin into a component package and install it where needed.

Terminology🔗

Tool: A wrapper for a low-level function. Defines an arbitrary number of input and output parameters that work as connection points in a processing graph. Unlike functions, tools provide modifiable default values and rich meta-data. They may also contain instance variables that keep the state of the tool. In the C API, the structure making up a tool is automatically generated, but it is possible to extend it. In the Python API, each tool is a Python class.
Processing graph: A directed acyclic graph that declares dependencies between tools.
Tool instance: A node in the processing graph. Different instances of the same tool may have different ideas of default values of arguments and the internal state of the tool, if the tool has state variables. In the C API, a pointer to the tool instance is passed to the underlying function. In the Python API, each tool is an instance of its defining class.
Input parameter: A parameter that is passed to the tool. May come from another tool in the graph, from a default value or from an external source through an API call.
Fixed input parameter: An input parameter that is programmer-adjustable but not connectable in the processing graph. The value of a fixed parameter cannot be changed at run time.
Optional input parameter: An input parameter that has a default value but is also connectable. The default value may be changeable through a public API.
Required input parameter: An input parameter that has no default value and must be connected in the graph either to another tool or to an API function. Required input parameters are needed only if there is no logical default value or if generating such value is not feasible.
Output parameter: A parameter that is produced by calling the underlying function with the current input parameters. Output parameters can work as inputs to subsequent tools and/or as signals in a public API.
Plugin: A dynamically loadable machine code library that implements a known entry point function (va_get_plugin_info).
Component: A redistributable unit that may contain an arbitrary number of plugins. The VisionAppster platform uses components for sharing all content, not just plugins. The ID and the major version of a component (e.g. com.visionappster.demo/1) defines a namespace for the tools inside it.

Execution modes🔗

The VisionAppster runtime uses a pool of threads to distribute work across processing cores. Most built-in tools in the platform are reentrant. In practical terms this means that the outcome of a tool invocation only depends on its input parameters and not on any static or global non-constant data.

Whenever all input parameters required by a tool are available, the runtime schedules the underlying function for execution. Since the outcome of a reentrant function depends only on the input parameters, it is always possible to execute any number of invocations to the function simultaneously. Therefore, one should always make tools reentrant if possible.

Sometimes, a tool must maintain mutable state variables. An example of such a tool is a counter that is incremented on each invocation. Such tools cannot be executed simultaneously unless the programmer manually ensures that concurrent access to the same state variables won’t break the program. There are also technical limitations that may make it impossible to implement a tool in a reentrant way. For example, it may be necessary to hold state in thread-local variables, which are only visible if the call is always made from the same thread.

It is important to understand the difference between a tool implementation and a tool instance in a processing graph. Multiple instances of a tool may appear in different locations in the graph. For reentrant tools this makes no difference. If the tool needs mutable state, each of the instances has a different idea of the state of the tool.

The underlying function may be called from different threads, but each thread will maintain different state data. This means that all wrapped functions must be thread-safe even though they may not be reentrant. In practical terms, the state of a thread-safe tool must not be kept in global or static variables, but only in a storage that is specific to a tool instance.

The VisionAppster runtime makes it possible to change the threading mode for each individual tool separately. If the default multi-threaded execution mode isn’t suitable, there are two alternatives:

Single-threaded: Each instance of the tool will be assigned a dedicated processing thread. No concurrent calls with the same instance will be made. Successive calls to the same tool instance always come from the same thread. In other words, if your graph contains multiple instances of the tool, they may still be called simultaneously, but with different instance pointers. This mode should be avoided because it may cause a large number of mostly idle threads to be created. Sometimes, there is no choice if you use a library that insists on storing its state in thread-local storage.
Non-threaded: No concurrent calls to the tool will be made with the same tool instance, but successive calls may come from arbitrary threads. The difference to multi-threaded execution mode is that simultaneous calls with the same instance will not be made. The difference to single-threaded mode is that the thread making the calls may be different from time to time. Note that it may make sense to use the non-threaded mode for a reentrant tool; the tool may be so simple that queuing the input data and performing a context switch would cost more than just calling it whenever data is available. If making your function re-entrant is difficult or impossible, prefer non-thread mode to single-threaded.

Meta-data🔗

To be able to call a function the VisionAppster runtime only needs to know the types of its parameters. Therefore, other meta-data is not strictly required. It does however make it easier for the application programmer to use the tool in the Builder.

Meta-data are either static or dynamic values that give the Builder and other GUI tools hints on how to handle the parameter. Each input and output parameter can have an arbitrary number of meta-data entries. Each meta-data entry is identified by a text string that is composed of the name of the parameter the entry refers to, a separator and a key that identifies the entry. In the C API, underscore (_) is used as the separator. For example, the min meta-data for a parameter called value would be identified by value_min.

Meta-data entries can be either static or dynamic. The need for dynamic meta-data usually arises when the meta-data depends on the current value of a fixed input parameter. For example, it may be useful to selectively hide certain parameters.

It is important to note that meta-data entries are only hints for the GUI, and it may choose not to respect them. At run time there is no protection whatsoever: your tool implementation may receive a value that is larger than the declared min, for example. If it is important that this does not happen, the tool implementation must check the validity of its input data and return an error code if needed.

Recognized meta-data keys🔗

The following list contains the meta-data keys most commonly used and supported by the default VisionAppster user interface components. It should be noted that no user interface is required to respect any of these and that custom components may provide additional values not listed here.

readOnly

A Boolean flag that indicates whether the value of the parameter can be written to or only read.

typeName

A type specifier for the parameter.

doc

A text string containing documentation for the parameter. May contain markdown formatting.

enabled

A Boolean flag that indicates whether the parameter should be available or not. The user interface may hide or gray out parameters that are not currently enabled.

stored

A Boolean flag that tells whether the value of an optional or fixed parameter should be stored permanently when the processing graph is saved. By default, the current values of all optional or fixed read-write input parameters are stored.

min

Minimum allowed value. The type must be the same as that of the corresponding parameter.

max

Maximum allowed value. The type must be the same as that of the corresponding parameter.

step

If the value can only take discrete values, this parameter specifies the step between successive values. Usually used with min and max. The type must be the same as that of the corresponding parameter.

suggestedMin, suggestedMax, suggestedStep

Sames as min, max and step, but impose no hard limits. This is useful for parameters that usually take a certain range but can also take different values in special situations.

choices

An array of valid choices for the parameter. Usually used for enumerated parameters. Each entry in the array is an object with name and value keys. The type of value must match the type of the parameter. See structured meta-data for details.

specialValues

An array of values that have a special meaning. The most common use for this meta-data is to provide a special meaning for Inf or NaN in a double type parameter. The format is the same as that of choices: each value of the array is an object with name and value keys.

columnNames

An array of text strings that are shown in the column header of a matrix or table, replacing the auto-generated header. If columnNames is given, it is assumed that the number of columns is fixed. For convenience, columnNames can also be given as a single, comma-separated text string. For example, a matrix representing a coordinate frame may define columnNames as “x,y,z,origin”.

firstColumnIndex

The index of the first column of a matrix or table, if auto-generated column headings are used. If firstColumnIndex is given, the default alphabetical column headings will be replaced by numbers. Must be an int32_t.

columns

A fixed number of columns as an int32_t. Unnecessary if columnNames or columnDefinitions is given.

columnsEditable

A Boolean flag that enables or disables UI controls for changing the number of columns in a matrix or table.

minColumns

Minimum number of columns if columnsEditable is true. Zero by default. An int32_t.

maxColumns

Maximum number of columns if columnsEditable is true. Unlimited by default. An int32_t.

columnDefinitions

An array of objects specifying meta-data for individual columns in a table. See structured meta-data below.

defaultValue

The value used for newly created table or matrix cells when new rows or columns are being added. The type of defaultValue can be anything, but it must match the typeName of the column definition object, if there is one.

firstRowIndex

The index of the first row of a matrix or table. Auto-numbering is started from one by default. Often, there is a need to change this to zero. Must be an int32_t.

rowNames, rows, rowsEditable, minRows, maxRows

Same as the corresponding meta keys for columns.

imageType

An array of text strings containing supported or produced image types for an Image type parameter. The types can also be specified as a single (comma-separated) string. This lets a tool to specify which kinds of images an input accepts or which kind of image an output produces. Valid values are raw, encoded and compressed. The default is raw, which only needs to be specified if an input parameter accepts e.g. both raw and compressed images. In this case the value for the meta-data entry would be “raw,compressed”. See image types for details.

linkedTo

Establishes a link between two associated input or output parameters. This makes it easier for the UI to create editor widgets such as a region or a line that can be draw on top of an image. Examples:

Input parameter called frame specifies typeName = frame. A related input parameter called size specifies typeName = size and linkedTo = frame/region. This makes the Builder aware that the size is related to the coordinate frame and that it can be edited using a “region” widget.
There are two output parameters, startPoint and endPoint, both of which specify typeName = point. In addition, endPoint specifies linkedTo = startPoint/line, which tells the Builder that the two points are the end points of a line and can be displayed using a line widget.

blockSize

If the output of a tool is a matrix, the existence of the blockSize meta-data specifies that the matrix contains multiple results stacked on top of each other. Each result consists of blockSize rows. For example, if an output specifies typeName = frame and blockSize = 4, the Builder knows to display each block of four rows of the matrix as an individual coordinate frame.

Types🔗

Built-in types🔗

To make all tools compatible with each other, the VisionAppster runtime defines a set of common C types every function must use when passing data. To ensure matching word lengths across different architectures, the stdint types of the C99 standard are used. Compatible definitions are provided for compilers that do not support the C99 standard. Tool API implementations for different programming languages use the core C types behind the scenes.

Type specifiers🔗

Often, semantically different types are represented by the same C data type by the VisionAppster runtime. For example, coordinate frames, points and one-dimensional time series are all represented as matrices. To make the Builder and possible other user interfaces aware of the semantics, the type of a parameter can be augmented with a typeName specifier.

The following types are recognized by standard user interface components:

enum: A list of mutually exclusive choices. Each choice is an object with the name and value keys. name is a String and value can be any data type, although “enums” are typically integers in programming languages. The existence of a choices meta-data entry implies enum.
enum/editable: A value that has a predefined list of choices but also allows editing the value manually.
multiline: A multi-line string.
escaped: A string that may contain escaped control characters such as \n or \t.
filename: A string that is used as a file name. This lets the Builder to provide a file selection GUI. The type may optionally be followed by a MIME type or a file name extension that limits the choices given to the user. For example, filename;image/* will let the user to pick any image file. filename;.txt will let the user to pick a file name with a .txt suffix.
directory: A string that is used as a directory name. This lets the Builder to provide a directory selection GUI.
frame: A 3D coordinate frame with translation (4-by-4 matrix).
size: Width and height relative to a coordinate frame (1-by-2 matrix).
point: A single point in two or three dimensions (1-by-2 or 1-by-3 matrix).
pointset: A set of related points in two or three dimensions (N-by-2 or N-by-3 matrix).

In the C API, a type specifier can be attached to a parameter using the STATIC_META macro. For example: STATIC_META(va_string, frame_typeName, "frame"). In the Python API, the same effect is achieved with function argument annotations. For example: (va.Matrix.Double, 'frame', va.Matrix.Double.identity(4), {'typeName': 'frame'})

Data type conversions🔗

When any 3rd party library is integrated to the VisionAppster platform, the need for data type conversions arises. To make this easy for the user, the platform comes with some built-in conversions to common types. Currently, OpenCV types and DLIB’s image type are supported in the C++ API. In the Python API, buffer-like data types implement the buffer protocol which allows them to be passed as shallow copies to functions expecting NumPy arrays, for example. Generally, one should avoid making deep copies of data whenever possible.

Debugging native tools🔗

In rare cases, a carefully designed and meticulously implemented algorithm doesn’t work exactly as intended once it compiles. If console output doesn’t help, a desperate developer may reach to a debugger as a last resort. But there is a problem: you cannot easily start the VisionAppster Builder or Engine in a debugger.

What you can do, however, is to compile your tool plugin with debug symbols, deploy it normally and attach your debugger to a running VisionAppster process.

Linux🔗

To debug a plugin in the Builder, launch the Builder normally and then attach the debugger to the va-builder process. To debug it in a locally running Engine, attach the debugger to the va-run process.

Windows🔗

To debug a plugin in the Builder, launch the Builder normally and then attach the debugger to the va-builder.exe process.

The VisionAppster Engine service runs in the LOCALSERVICE account and administrator rights are required to debug your tool in an Engine running in it. So, launch your IDE with “Run as administrator” command and then attach the debugger to the va-engine-service.exe process. You may need to select option “Show processes from all users” to see the service process.