Tool C API

Getting started

The VisionAppster SDK comes with example projects. To get started, you may want to have a look at a "hello world" equivalent of a tool plugin at sdk/examples/exampletoolplugin. On Linux, you'll find the example wherever you installed the SDK. On Windows, you should make a local copy of the sdk folder under your installation to %USERPROFILE%\VisionAppster\sdk.

The example comes with a Visual Studio project file and a Makefile for the GNU compiler. If you installed the platform to a non-default location on Windows, you need to change the BIN_PATH variable in the Makefile. In the VS project, there is a global property called VisionAppsterSDKRoot that needs to point to your local copy of the sdk folder.

This documentation describes how to build tool plugins in plain C and specifies the required functionality for a compatible tool plug-in.

Installing a C++ compiler

To create a new tool you need a C compiler. The tool API is in C because C has a standard binary interface that lets one to link together binaries generated by any standards-compliant compiler. C++ (or any other language that is compatible with C) can be used to implement the tools, but there is a caveat: linking to C++ libraries that come with the platform SDK is only possible if you use exactly the same compiler that was used to build the SDK.

Linking problems may arise for example if you use the built-in OpenCv libraries in the SDK. On Linux, this is less of a problem as you can usually install a compatible gcc compiler from your system's package manager. On Windows, things are a bit more complicated: the gcc compiler comes in many flavors, and not all of them are compatible.

If you are using MSYS2 the default gcc package won't work. Instead, go to the MSYS2 terminal and install the mingw-w64 version. You also need GNU make:

pacman -S mingw-w64-x86_64-gcc make

MSYS2 installs three different terminals into the Windows start menu. Make sure to use the "MSYS2 MinGW 64-bit" version when compiling.

MinGW-w64 can also be installed directly and used from the Windows command prompt. For this, you need gcc version 7.3.0 with Posix threads and structured exception handling. You may have luck with other versions, but this is currently the only officially supported one. The official download page is here, but it is suggested that you download the correct version directly. Unzip the file to the root of your C: drive and add C:\mingw64\bin to PATH.

Implementing new tools

Let's start with a simple tool that sums two integers. For this, one would need a single C (or C++) source file with the following contents:

// MyPlugin.cc
#include <va_tool.h>

VA_IMPLEMENT_PLUGIN("com.visionappster.demo", "1.0.0")

VA_REGISTER_TOOLS(
( IDENTIFIER     (MySum),
  NAME           ("Totally Cool Sum"),
  PROCESS        (MySum_process),
  OPTIONAL_INPUT (int32_t, a, 0),
  OPTIONAL_INPUT (int32_t, b, 0),
  OUTPUT         (int32_t, sum)
))

int MySum_process(void* instance, void* arguments)
{
  (void)instance; /* ignoring, tool has no state */
  struct MySum_args* args = (struct MySum_args*)arguments;
  args->out.sum = args->in.a + args->in.b;
  return VA_SUCCESS;
}

We'll ignore the VA_IMPLEMENT_PLUGIN macro for a while and concentrate on the rest. The VA_REGISTER_TOOLS macro takes a comma-separated list of tool declarations, each of which is enclosed in parentheses.

Tool declaration

Each tool declaration contains a variable number of entries that are created using special declaration macros (e.g. IDENTIFIER and NAME) as shown in the example. The first macro in all declarations must be IDENTIFIER. In addition, the PROCESS member function must be defined. Everything else is optional.

Supported declaration macros:

IDENTIFIER(ID)
Gives the tool a unique name in the context of the enclosing component. The tool identifier can be thought of as a class name and should follow the rules for naming structures in C.
NAME(NAME)
Gives the tool a user-visible name as a text string.
TAG(TAG)
Tags can be used to give the VisionAppster engine additional hints about the tool. Currently, tools marked with TAG("internal") will not be shown in the Builder's tool box. This is useful if you build apps or compound tools that require native functionality that is not intended to be used directly. The tool declaration can include any number of tags.
PROCESS(FUNCTION)
The name of the C function implementation. The signature of the function must be int32_t FUNCTION(void*, void*). To ensure correctness, this macro will declare the function with the correct signature.
EXEMODE(MODE)
The execution mode of the tool. One of the three predefined execution modes or a bitwise OR of all supported modes.
REQUIRED_INPUT(TYPE, NAME)
Declares a required input parameter with the given type and name. TYPE must be one of the supported types, and NAME a valid variable name according to the rules of the C programming language.
OPTIONAL_INPUT(TYPE, NAME, DEFAULT)
Declares an optional input parameter with the given type, name and default value. The default value can be any code that evaluates to the correct type. Usually, DEFAULT is a constant value (such as 1.23), but it can also be a function call that returns a value of TYPE (e.g. sin(M_PI)). Complex types can be allocated using functions such as va_image_alloc(), va_imatrix_alloc() or va_dmatrix_alloc(). The ownership of the returned pointer will be transferred to the platform.
FIXED_INPUT(TYPE, NAME, DEFAULT)
Declares a fixed input parameter with the given type, name and default value.
OUTPUT(TYPE, NAME)
Declares an output parameter with the given type and name.
STATIC_META(TYPE, NAME, VALUE)
Adds a static meta-data entry. See Meta-data.
DYNAMIC_META(TYPE, NAME, FUNCTION)
Adds a dynamic meta-data entry. See Meta-data.
CONSTRUCT(FUNCTION)
Provides a custom constructor for allocating new instances of the tool. See member functions.
DESTRUCT(FUNCTION)
Provides a custom destructor for freeing instances of the tool. See member functions.
NOTIFY(FUNCTION)
Provides a notifier function that will receive events of tool instance state changes. See member functions.

Tool implementation

The VisionAppster platform requires that the actual function to be called has a known signature: the function should take two void pointers as input parameters and return an int. The first input parameter is a pointer to a function instance. The second parameter points to function call arguments. The return value is an error code.

Both input arguments point to auto-generated structures. The name of the auto-generated function argument structure (second parameter) is formed by concatenating the tool's identifier and "_args". By default, the instance structure contains the default values of input arguments. Its name is formed by concatenating the tool's identifier and "_in".

Usually, the function implementation casts the argument pointer to the correct "_args" structure type, which gives it easy access to input and output parameters. The names of parameters in the structure match those provided by the declaration macros except for "in." and "out." prefixes that separate input and output parameters.

Useful tools are generally more complex than the simple sum shown in the example. They may also involve data types such as proprietary image structures that need to be converted from/to the built-in types. The purpose of the interface function is to parse the input parameters, make the necessary conversions and to allocate memory for the underlying operation. If everything goes fine, it writes the result to the output parameters in the argument structure and returns VA_SUCCESS. Other possible return values are:

  • VA_ERR_OUT_OF_MEMORY - As the name implies.
  • VA_ERR_INVALID_PARAMETER - At least one input parameter was invalid. This actually sets the most significant bit of the return value, and more specific information can be given by oring to this value.
  • VA_ERR_NOT_INVERTIBLE - A matrix given as an input parameter is not invertible. This is an example of a specifically invalid input parameter.

It is worth noting that changes made to the input arguments are not retained between function calls. Changes made to the instance structure are retained, but the function is allowed to modify only those values that it declares and initializes itself. An example is given below.

Building and deploying a plugin

The process of building a plugin will be different depending on the compiler/IDE being used. The goal is however to build a shared library (.dll/.so/.dylib) that contains some interface functions the VisionAppster platform will recognize.

In the simplest case, a compatible plugin consists of a single C source file as shown in the example above. The VA_IMPLEMENT_PLUGIN macro takes two parameters:

COMPONENT_ID
A globally unique identifier for the component the plugin will be placed into. If a component contains many plugins, the same component ID must be used in all of them. The component ID is used as a namespace for the tools that are defined within the component. The fully-qualified name for a tool will be composed of the component ID, its major version number and the name of the tool, for example com.visionappster.demo/1/MySum.
VERSION
Version number of the component. The version number has three parts: major.minor.patch. If a change in the plugin breaks existing functionality, the major version number must be incremented. Features can be added but not removed when changing the minor version number. If a release does not add new functionality but just improves or fixes the implementation, patch must be incremented. Version numbering can be started at any number, but usually 1.0.0 marks the first release.

Only the VA_IMPLEMENT_PLUGIN macro is strictly required for a plugin. Even the VA_REGISTER_TOOLS macro is optional, but without it the plugin would be quite useless.

Since the interface functions are written in C, it is possible to use any standards-compliant C compiler to build a tool plug-in. It doesn't need to be the same as that used for building the VisionAppster platform. However, you cannot mix and match binaries for different architectures. For example, a plugin compiled for a 64-bit architecture cannot be loaded by a 32-bit version of the platform.

If the tool implementations have no external dependencies, it is sufficient to add sdk/include to include the project's include path and compile the source file to a shared library. The vatypes library must be linked in if any of the built-in complex types are used. For this, sdk/lib must be in your linker's library search path.

The platform comes with example projects that support many different compilers. Please take a look at the sdk/examples directory. The following summarizes the required compilation and environment setup steps.

Linux

On Linux, the VisionAppster platform runs in a sandbox and one cannot directly access the SDK that is inside. If you have not installed the SDK, run the following command:

va-install --sdk

The script will ask you where to put the files. By default, the SDK will be installed to $HOME/VisionAppster/sdk. You also need a compiler. If you don't have one installed, use your system's package manager to install the build-essential package. On the command line:

# Ubuntu, Debian etc.
sudo apt install build-essential
# Red Hat, Fedora etc.
sudo yum install build-essential

Once the SDK and a compiler is installed, the compilation command will be something like this:

export VA="$HOME/VisionAppster"
g++ -shared -o libmyplugin.so MyPlugin.cc \
  "-I$VA/sdk/include" \
  "-L$VA/sdk/lib" -lvatypes

Replace g++ with clang++ if you wish. If your tool uses OpenCV functions, you need to include OpenCvMat.cc in sources:

g++ -shared -o libmyplugin.so MyPlugin.cc \
  "$VA/sdk/src/opencv/OpenCvMat.cc" \
  "-I$VA/sdk/include" \
  "-L$VA/sdk/lib" -lvatypes -lopencv_core

By default, your local VisionAppster Engine searches $HOME/VisionAppster/plugins/tool for tool plugins. Thus, copying your plugin there and restarting the service will make your tools available to the Engine:

mkdir -p $HOME/VisionAppster/plugins/tool
cp libmyplugin.so $HOME/VisionAppster/plugins/tool
systemctl restart --user va-engine

If your plugin depends on external shared libraries, you also need to make them visible to the sandbox. This is most easily done by copying dependent libraries to $HOME/VisionAppster/lib.

The corresponding directories for a system-wide installation are /opt/visionappster/plugins/tool and /opt/visionappster/lib.

The default search directory can be changed by setting the VA_EXT_PLUGINS_PATH environment variable. Tool plugins will then be searched also in $VA_EXT_PLUGINS_PATH/tool. System environment won't however be visible inside the sandbox, so just setting VA_EXT_PLUGINS_PATH doesn't help. One needs to pass the environment variable to the sandbox:

flatpak run --env=VA_EXT_PLUGINS_PATH=$HOME/plugins \
  com.visionappster.Builder

Environment variables inside the sandbox can also be changed in $HOME/.config/VisionAppster/engine.conf. Example contents:

[environment]
VA_EXT_PLUGINS_PATH=/home/l33tcoder/deploy/plugins

The file does not exist by default, so you need to create it.

Windows

On Windows, the SDK is installed to the platform's installation directory and should be copied to your home directory. To compile, you need to add sdk\include to your project's include paths and sdk\lib to linker search paths. We'll assume you have made a local copy of the SDK (typically at C:\Program Files\VisionAppster\sdk to your home directory (C:\Users\YourUserName\sdk). If you use MSYS2, compiling is more or less equivalent to Linux:

export VA="/c/Users/$USERNAME/VisionAppster"
g++ -shared -o libmyplugin.dll MyPlugin.cc \
  "-I$VA/sdk/include" \
  "-L$VA/sdk/lib" -lvatypes

A typical command line for the Visual Studio compiler would look like this:

set VA=%USERPROFILE%\VisionAppster
cl /I "%VA%\sdk\include" MyPlugin.cc vatypes.lib ^
  /LINK /LIBPATH "%VA%\sdk\lib" ^
  /LIBPATH "C:\Program Files\VisionAppster\bin" ^
  /DLL /OUT myplugin.dll

You also need vatypes.lib that isn't included in the SDK. To generate it, give the following commands in Visual Studio command prompt:

cd %VA%\sdk\lib
lib /def:vatypes.def

Once the plugin is compiled, you can make it available to the VisionAppster Engine service by copying it to the plugins\tool directory under the installation directory:

copy myplugin.dll "%VA%\plugins\tool"

Then, restart the VisionAppster Engine service using Windows service manager. To replace an existing plugin you need to stop the service first as Windows does not allow overwriting of libraries that are being used.

To deploy a plugin for use in the Builder only, copy it to \VisionAppster\plugins\tool under your home directory. Typically:

copy myplugin.dll %USERPROFILE%\VisionAppster\plugins\tool

The VA_EXT_PLUGINS_PATH can be used to change the default plugin search folder. If the variable is set, tool plugins will be searched in %VA_EXT_PLUGINS_PATH%\tool. To test your plugin in the Builder, you can set the environment variable in a configuration file that is stored at %APPDATA%\VisionAppster\engine.conf:

[environment]
VA_EXT_PLUGINS_PATH=C:\temp\plugins

The file does not exist by default, so you need to create it.

If your plugin depends on external dynamic libraries, make sure all dependencies are found in PATH. It is not a good idea to place dependent libraries in plugins\tool as the Engine will then try (and fail) to load them as tool plugins.

Meta-data

In the C interface, each meta-data entry is identified by a text string that is composed of the name of the parameter the entry refers to, an underscore (_) and a key that identifies the entry. The meta-data identifier may contain an "in_" or "out_" prefix. If none is provided, "in_" will be assumed.

Note that since underscore has a special meaning as a separator, neither variable names nor meta-data keys can contain underscores.

Let us assume the input parameters of the cool sum tool should only take positive values. We can achieve this either statically or dynamically:

VA_REGISTER_TOOLS(
( IDENTIFIER     (MySum),
  NAME           ("Totally Cool Sum"),
  PROCESS        (MySum_process),
  OPTIONAL_INPUT (int32_t, a, 0),
  OPTIONAL_INPUT (int32_t, b, 0),
  FIXED_INPUT    (int32_t, bMin, 0),
  OUTPUT         (int32_t, sum),
  STATIC_META    (int32_t, a_min, 1),          /* constant */
  DYNAMIC_META   (int32_t, b_min, MySum_b_min) /* varying */
))

void MySum_b_min(const void* instance, void* value)
{
  *(int32_t*)value = ((struct MySum_in*)instance)->bMin;
}

The need for dynamic meta-data usually arises when the meta-data depends on the current value of a fixed input parameter. For example, it may be useful to selectively hide certain parameters. In the example above, the minimum value for "b" is given by the current value of the fixed "bMin" parameter:

  1. The minimum value for "b" is declared to be dynamic and fetched using the MySum_b_min function.
  2. The implementation of the MySum_b_min function takes two parameters: a pointer to a function instance structure and a pointer where the meta-data value should be written. The implementation writes the current value of the "bMin" fixed input parameter there.

Structured meta-data

Not all meta-data can be represented using native data types. For example, choices or columnDefinitions are arrays that can't be easily constructed in C. To make things easier, arrays and objects can be created out of JSON strings. If you want to replace the default toggle button control used for Boolean parameters with a drop-down list, you can do something like this:

STATIC_META(va_array, input_choices, va_array_from_json(
            "["
            "{\"name\":\"NO!\",\"value\":false},"
            "{\"name\":\"YES!\",\"value\":true},"
            "]"))

The example may be a bit hard to read due to escaping. Since C++11, it has been possible to use raw string literals:

// "%" can be replaced with an arbitrary string.
static const char* choices = R"%(
[
  {"name": "NO!",  "value": false},
  {"name": "YES!", "value": true},
]
)%";

// ...
STATIC_META(va_array, input_choices,
            va_array_from_json(choices))

The columnDefinitions meta key can be constructed in a similar manner. The meta-data entry consists of an array of objects, each of which provides a number of meta-data entries for a single column in a table or a matrix. Either typeName or defaultValue is required for table columns. If both are provided, the type of defaultValue must match the given typeName. In a matrix, the type is the same for all cells and does not need to be specified.

The following example specifies a table that lets the user to pick the ingredients of her favorite juice. The table has two columns: "Concentration" and "Flavors". The first one is the fruit/water ratio that ranges from zero to 100. The second lets the user to pick any number of flavors to the mixture using bit flags.

static const char* columns = R"%(
[
  {
    "name": "Concentration %",
    "typeName": "double",
    "defaultValue": 50,
    "min": 0,
    "max": 100
  },
  {
    "name": "Flavors",
    "typeName": "enum/multiple",
    "choices":
    [
      {"name": "Apple",  "value": 1},
      {"name": "Banana", "value": 2},
      {"name": "Orange", "value": 4},
      {"name": "Potato", "value": 8}
    ]
  }
]
)%";

// ...
STATIC_META(va_array,
            inputTable_columnDefinitions,
            va_array_from_json(columns))

Member functions

The tool declaration may include member functions for different purposes. To actually perform some useful job, the processing function must always exist, and this is usually enough. Other functions are needed if internal state needs to be maintained.

The first argument to member function calls is a typeless pointer (void*) that points to instance data. It serves the same role as the this pointer in C++. By default, the instance pointer points to an argument structure that contains the default values of input arguments, but this can be overridden by providing custom constructor and destructor functions.

The VisionAppster platform assumes that the layout of the custom instance is the same as the auto-generated input argument structure. Therefore, the custom instance structure should contain the auto-generated input argument structure as its first member (or derive from it, if the code is written in C++).

In addition, the tool may need to receive notifications about state changes. For this purpose, it is possible to add a notification function. In addition to the instance pointer, the notification function takes two parameters: the type of the event (one of the predefined events) and a void pointer to event data. The contents of the data pointer varies depending on the type of the event.

Let's add another tool that calculates a cumulative sum of past values, optionally negating the input if negate is set to true:

#include <va_tool.h>
#include <stdio.h>
#include <stdlib.h>

VA_IMPLEMENT_PLUGIN("com.visionappster.demo", "1.0.0")

VA_REGISTER_TOOLS(/*
( IDENTIFIER     (MySum),
  ...
  The rest of MySum declaration would go here.
  Omitting for brevity.
),*/
( IDENTIFIER     (CmlSum),
  NAME           ("Cumulative Sum"),
  PROCESS        (CmlSum_process),
  CONSTRUCT      (CmlSum_construct),
  DESTRUCT       (CmlSum_destruct),
  NOTIFY         (CmlSum_notify),
  REQUIRED_INPUT (int32_t, a),
  FIXED_INPUT    (va_bool, negate, va_true),
  OUTPUT         (int32_t, sum)
))

/* Custom structure for holding instance data. */
struct CmlSum
{
  /* This must be first to ensure correct memory layout. */
  struct CmlSum_in in;
  int i_sum;
};

/* Constructor function.
   If "other" is non-zero, it must be copied.
*/
void* CmlSum_construct(void* other)
{
  struct CmlSum* self = 0;
  /* Allocate an all-zero block from the heap. */
  self = (struct CmlSum*)calloc(1, sizeof(struct CmlSum));
  /* If "other" is given, construct a copy. */
  if (self && other)
    self->i_sum = ((struct CmlSum*)other)->i_sum;
  return self;
}

void CmlSum_destruct(void* instance)
{
  free(instance);
}

int32_t CmlSum_process(void* instance, void* arguments)
{
  struct CmlSum* self = (struct CmlSum*)instance;
  struct CmlSum_args* args = (struct CmlSum_args*)arguments;
  /* NOTE: Modifying the "in" structure is not allowed. */
  if (!self->in.negate)
    args->out.sum = (self->i_sum += args->in.a);
  else
    args->out.sum = (self->i_sum -= args->in.a);
  return VA_SUCCESS;
}

void CmlSum_notify(void* instance, va_event_type event, void* data)
{
  (void)data; /* unused */
  struct CmlSum* self = (struct CmlSum*)instance;
  if (event == va_event_stopped)
  {
    printf("Final sum = %d\n", self->i_sum);
    self->i_sum = 0;
  }
}

If a custom constructor is provided, a custom destructor must also be given. This is because it cannot be guaranteed that the memory allocator used in the plugin is compatible with the one used by the VisionAppster platform. They may be using completely separate heaps, for example. In this example, the constructor returns a heap-allocated pointer that can be safely destroyed by free() on the plugin side. (Using the VisionAppster platform's free() would not be safe.)

If member functions are provided, the VisionAppster platform assumes they are for state management and therefore disables multi-threaded execution mode execution mode by default. This can be overridden with the EXEMODE macro. For example, the default mode can be restored by EXEMODE(va_execute_any).

Types

The C language API naturally uses the built-in C types. Structured types such as images and matrices are passed as pointers even though all type names are given without the asterisk (*) in the function declaration. An important thing to note is that all pointers in the auto-generated argument structures are owned by the platform, not the user. This applies to default values of optional and fixed input parameters, to everything that is passed in to the process function and to everything the process function assigns to the output arguments.

For example, the default value for an input argument of type va_imatrix in the OPTIONAL_INPUT macro can be created by calling va_imatrix_calloc(). The ownership of the returned pointer is transferred to the platform. It is also safe to assign a newly allocated pointer to the value parameter in a dynamic meta-data function.

The ownership of pointers in instance variables that are not visible to the platform (not listed in the tool declaration macro) remains at the user.

String encoding

Text strings can contain any data that can be encoded into a double-quoted string in C. If the string contains non-ASCII characters, things can get a bit complicated.

The C standard does not specify source file encoding. Thus, strings containing non-ASCII characters may end up being encoded in an unexpected way if proper care is not taken. The VisionAppster platform has no way of detecting which compiler or encoding was used when building external plugins, and it always assumes UTF-8 encoding.

If a text string appears incorrect, you may want to check that your compiler's idea of the source character set matches the actual encoding of your source file and that UTF-8 encoding is used in the executable. Visual Studio, for example, provides the /source-charset and /utf-8 compiler options for this.

Calling conventions

The VisionAppster Tool API does not explicitly specify the calling convention of functions declared in user-created tool plugins. Instead, it is assumed that the default calling convention for each hardware platform is used. On x86, the default convention is cdecl. On ARM and x64, arguments are passed in registers when possible.

If you encounter strange crashes when loading a tool plugin, make sure you haven't specified a non-default calling convention when compiling the plugin. The potentially conflicting command-line options for the Visual Studio compiler are /Gv (vectorcall), /Gz (stdcall) and /Gr (fastcall).