PROGRAMMING BY DEMONSTRATION ENVIRONMENT

Introduction

The technical specification diagram above represents the overall architecture of the robot controller. The constraint-based robot controller is a reactive controller that needs to run at a higher sample frequency (order of magnitude 100Hz). The Orocos real-time component system RTT is used to write real-time components. Orocos offers lock-free, thread-safe inter-component communication in a single process. The 5C's : computation, communication, coordination, composition and configuration) can be configured from the outside of the components, resulting in a flexible framework for real-time components in robotics.

Hardware (robots, sensors, ...) and other external inputs and outputs are handled using hardware abstraction components such as robot drivers and sensor drivers. Using the Orocos-ROS bridge the controller can communicate with a ROS2 environment (cf. ROS framework).

GUI and visualization (if needed) is done as a node in the ROS environment.

The eTaSL constraint-based controller component can flexibly connect to Orocos communication ports and expose this in the constraint-based task specification language. This allows to write a constraint-based task specification in two mostly independent parts: a part describing the properties and constraints related to the robot system, and a part describing the skill or application.

When used with programming-by-demonstration, a constraint-based robot task specification will consist of two parts:

• a probabilistic description of the demonstrated force/motion interactions describing nominal force/motion and the allowable variations on this nominal force/motion.
• a model-based task constraint-based task description describing the parts of the task that is known before-hand. The constraints are described in a LUA-based language called eTaSL.

The robot controller computes robot motions that satisfy optimally these two parts of the task specification. Low-level coordination of control-subtasks is done using a lightweight state-chart implementation (rFSM)

The HMI module is further detailed in the HI section.

The eTaSL constraint-based controller is very flexible and can be completey configured for different robots and sensors, and different tasks using the eTaSL constraint-based task specification language. To interface a robot or sensor with eTaSL, there are two parts:

• A hardware abstraction component in Orocos that takes care of communicating with the sensor or robot. Typically implemented in C++. Sometimes this communicates directly from within Orocos to the hardware. In other cases it communicates via the Orocos-ROS bridge to a ROS node that knows how to talk to the hardware.
• An URDF-file or piece of eTaSL specification that describes how the robot hardware relates to the task specification.

Technical Specifications

Software and hardware Requirements

A ROS2 Humble environment is assumed and therefore a Ubuntu Linux - Jammy Jellyfish (22.04) Operating System using x86_64 is recommended.

Usage Manual

eTaSL can be divided into 3 layers. The specification layer contains all libraries and tools to specify a task inside eTaSL. These libraries and tools makes it easier to specify a task. All these tools use LUA to specify the task, sometimes with a C++ back-end.

These specifications are used to generate a C++ datastructure Context that contains all information of the data-structure.
This datastructure is then used to compose an optimization problem (the Solver step) at each time-step and then use a numerical solver to find a solution for this optimization problem.

The basic principles behind eTaSL can also be found in the following paper and are summarized below.

Installation

This repository gives instructions on installing eTaSL.

This template application can help to get started with eTaSL and provides support for UR10, Kinova, Franka Emika and KUKA iiwa.

Below you will find a short introduction to what the constraint-based robot controller does, the concepts behind constraint-based task specification, and the syntax of the specification language. A more elaborate documentation can be found here

Constraints

Using our constraint-based task specification and control framework eTaSL, we specify the behavior of some expression, the task-function $e_i(q,t)$ where $q$ refers to the joints of the robot system and $t$ to time. In case of sensor-input or other inputs such as desired trajectories, the task function can be dependent on an additional input vector $u$: $e_i(q,u(t),t)$.

We write specifications of behavior mathematically down in multiple equations of the following form:

$$e_i \overset{K}{\underset{w}\longrightarrow} 0,$$

meaning that we want the expression $e_i$ (i.e. the task function) to evolve to $0$ as a first order linear system with gain $K$. In case of multiple, conflicting constraints, we assign a weight $w_i$ to this behavior. Formulated as a constraint this becomes:

$$\dot{e}_i = - K e_i + \varepsilon_i,$$

where $\varepsilon_i$ is a slack variable weighted with the weight $w_i$.

This leads to an evolution of the task function similar to the figure below, with a time constant $\tau = 1 / K$:

Note:

If we specify a position constraint, we only specify a behavior towards this position. We do not directly impose this position.

The eTaSL controller needs to determine the joint velocities to be sent to the robot to most optimally satisfy constraints written in the above form. We do this by expanding the total time derivative :math:\dot{e}_i into its partial derivatives and applying the chain rule for differentiation:

$$\dot{e}_i = \frac{\mathrm{d}}{\mathrm{d}t} e_i = \frac{\partial e_i}{\partial q} \dot{q} + \frac{\partial e_i}{\partial t} = -K e_i + \varepsilon_i$$

The partial derivatives towards the joints is nothing else than Jacobian, such that the above can be written as:

$$J_i(q,t)~\dot{q} = - K e_i(q,t) - \underbrace{\frac{\partial e_i}{\partial t}(q,t)}_{\text{feedforward}} + \varepsilon_i ~~~~(1)$$

Optimization problem

To deal with multiple, possibly conflicting, constraints, and to control the robot to best satisfy all constraints, eTaSL solves the following optimization problem at each sample period:

$$\min_{\dot{q}} \sum_i w_i \varepsilon_i^2 + \lambda \sum_i \nu_i \dot{q}_i^2$$

subject to:

$$J_i(q,t)~\dot{q} = - K e_i(q,t) - \underbrace{\frac{\partial e_i}{\partial t}(q,t)}_{\text{feedforward}} + \varepsilon_i ~~~~~~~~~ \text{for all}~i $$

Note the additional term $\lambda \sum_i \nu_i \dot{q}_i^2$ which represents regularization in case of rendundant degrees of freedom or singularities ($\lambda$ is small).

Note:: eTaSL does not by itself look ahead into the future. It does only do instantaneous optimization. It only determines the current velocities to be sent to the robot without regards to what happens in the future.

Note:: An eTaSL constraint does not plan a trajectory. If you directly impose a desired pose on your end effector that is far away, the result will not be what you expected, due to the large velocities because of the large errors. eTaSL describes behaviors.

Feature variables

Besides joints $q$ corresponding to a physical robot joints, eTaSL also allows us to define virtual joints, called feature variables. These feature variables do not correspond to a physical robot joint but are part of the task specification.

These feature variables specify additional freedoms that you explicitly want to introduce. Sometimes a behavior is more easily specified by describing the constraints it needs to satisfy, sometimes specifying explitly the freedoms that it should allow is more easy. For example, if we want to specify that the robot needs to be on a path, but we do not care where on the path, this can be achieved using a feature variable:

For the derivation of constraints described above, feature variables are equivalent to the physical joints and are included in the $q$ variables described above.

Task specification language

The eTaSL controller is very flexible and can be configured to use many types of sensor-inputs and robots while executing a large variety of tasks. This configuration is done using the eTaSL task specification language and uses the constraint-based control formalism summarized above. For this purpose, eTaSL uses the LUA language. From this task specification eTaSL will generate a controller that performs the described control strategy.

An editing environment with intelli-sense autocomplete and API-support using Visual Studio code is provided:

. The specification language consists of the following core concepts:

1. Variables

All variables in eTaSL have a name and a type. The type can be a robot variable, meaning that the variable corresponds to a physical joint that can be actuated, or can be a feature variable, meaning that the variable is a virtual variable that does not correspond to a physical joint. All the variables are named and automatically managed when composing the optimization problem. This makes a task specification very modular. It is easy to swap out one robot for another, even when they have different degrees of freedom. Variables are declared as follows:

    fv1 = Variable {
        context  = ... [the context to define the variable in, typically ctx]
        name     = ... [string representing the name of the variable]
        vartype  = ... [optional, can be 'robot' or 'feature', default 'feature']
        initial  = ... [optional number, default 0.0]
        weight   = ... [optional number expression, default constant(1.0)]
    }

The weight of a variable is useful in case of a redundant robot configuration where there are less constraints than degrees of freedom. In such a case, eTaSL minimizes the weighted squared norm of the variable velocities (i.e. both robot- and feature variables).

By default, the variable time is declared automatically and can be used to model explicit time dependencies.

2. Expressions

Expressions are a core concept in eTaSL and are used to represent robot kinematics and aspects of the task specification. Task functions are described using these expressions. Expressions can have different types:

• scalar expressions (expression_double). A scalar expression refering to a constant value can be specified as constant(1.0).
• vector (expresion_vector), a 3-vector representing and expression for positions or position-differences.
• orientations (expression_rotation) a 3x3-matrix representing an expression for an orientations
• frame (expression_frame) a 4x4 matrix representing an expression for a homogeneous transformation matrix.
• twist (expression_twist) a composition of two vectors, with first the velocity vector and then the rotational velocity vector. Representing the velocity of a rigid body.
• wrench (expression_wrench) a composition of two vectors, with first the the force vector and then the torque vector. Representing a force/moment interaction on a rigid body.
• quaternions ("expression_quaternion") (in development, not yet available)

Libraries are provides for common operations such as translations and rotations in SE(3), distances between geometric entities such as lines and planes, and distances between convex objects. Expressions can also be generated from an URDF-description of a robot.

3. Constraints

Constraints in eTaSL are described by the following very flexible command:

   Constraint{
       context = ...      [The context, normally ctx is used]
       name = ...         [optional, default_name<nr>]
       expr = ...         [ compatibility, if expr is used then model==expr and meas==expr ] 
       target = ...       [optional, 0.0] ( can be expression )
       target_lower = ... [optional, 0.0] ( can be expression )
       target_upper = ... [optional, 0.0] ( can be expression )
       K = ...            [optional, 4.0] ( the gain, i.e. inverse of the time constant of the 
                                           first order linear system describing the behavior)
       weight = ...       [optional, defaultweight, >= 0] ( can be expression )
       priority = ...     [optional, defaultpriority, 0..2
   }

A constraint needs to be defined in a context. In normal circumstances, this is always the global ctx variable. You can give the constraint a name, so that you can trace it back in error messages.

expr is used to specify an expression (i.e. task function) to impose a behavior on.
This expression can of type expression_double, expression_vector, expression_rotation or expression_frame.

target is an expression specifying the desired value for the expression. Inequality constraints can be specified by using target_lower or target_upper instead of target. Inequality constraints are only defined for scalar expressions. vector, orientation or frame expressions can only have equality constraints.

weight specifies the weight of the constraint.

Using the notation described at the start of this introduction, the constraint corresponds to the following specification:

$$ ( \text{expr}-\text{target} ) \overset{K}{\underset{\text{weight}}\longrightarrow} 0$$

priority defines the priority of the constraint. Priority equal to 0 corresponds to a hard constraint, i.e. a formulation of the constraint without slack-variable. Priority equal to 2 corresponds to a soft constraint, i.e. a constraint with a slack variable such that deviations from the constraint are possible.

4. Input and output declarations

By default, eTaSl needs to receive as input a position measurement for each of the robot joints that are defined in the specification. At each sample period, eTaSL will send out the desired velocity for each of the robot joints as one of its outputs.

Additional inputs, such as from a 3D joystick, a force/torque sensor or the location of a visual feature on a camera system can be used as input of eTaSL. eTaSL provides a generic mechanism to define such inputs:

    r = ctx:createInputChannelScalar(name, defaultvalue)

name is a name associated with this input. defaultvalue is the value that eTaSL will use when no input has yet been received. This function will return an expression corresponding to the specified input.

Similarly outputs are specified using:

     ctx: setOutputExpression(name, expr)

Again, name specifies an name associated with this output. expr contains an expression that will be evaluated at each sample time in order to generate the output.

5. Monitors

A typical set-up of eTaSL uses a finite state machine to coordinate different control-task for a given application. Typically eTaSL will generate some events that will be received by the finite state machine. This finite state machine in turn switches the task at hand or changes some of the inputs of the eTaSL controller:

In order to generate these events, eTaSL uses monitors. Monitors observe an expression, check whether it stays inside an interval and sends out an event each time the expression goes outside the interval. In other words, monitors are edge-triggered, each time you go from inside to outside the interval an event is generated. If you stay outside the interval, no events are triggered any more:

A monitor is declared with the following statement:

    Monitor {
        context = 
        name    = [optional]
        expr    = ...
        lower   = ... [optional, default -infinity]
        upper   = ... [optional, default +infinity]
        actionname    = ... 
        argument= ...[optional, default value "e_finished"]
    }

As always context refers to the context to add the monitor to (typically ctx). name is an optional name for the monitor. expr is the scalar expression to monitor. lower and upper indicate the interval to monitor. An event will be sent when the expression goes out of the interval. actionname is the eventhandler to be used to signal the event. argument is an optional argument for the event-handlers.

Use Case.

Use Case Diagram

The main objective of the AI-Prism programming by demonstration environment is to quickly deploy robot applications. To do so, generic applications are assembled and configured from a library of skills. These generic applications, also called "robot apps". These generic applications are then further specialized for a specific factory site using programming by demonstration. Different actors are involved in the process of deploying a new application. The proposed approach supports each of these actors in a different way. In some cases, the different actors are roles that can be fulfilled by the same person:

The system developer is a robotics and control expert that develops a set of skills that define how the robot behaves and reacts to inputs. A few examples of skills are given in the mockup example below. The formulation of these skills is aided by using a constraint-based task specification language and corresponding controller and rFSM state machines

The application developer assembles, configures and coordinates the above skills into an application. He is an expert in the application domain and is capable of developing configuration scripts. He decides on what skills will be used and what parts of the application will be obtained using programming-by-demonstration. His objective is to combine model-based programming and programming-by-demonstration such that the application can be deployed with only a minimal number of demonstrations. His work is facilitated by an appropriate set of skills developed by the system developer.

The demonstrator is a human worker that is allowed to perform minor configurations on the robot system and will be trained in how to perform the demonstrations. His work is facilitated by an appropriately defined generic application.

The operator is the day-to-day operator that performs tasks in cooperation with the robot system. The system not only has to react to disturbances or actions of the operator, also environmental or work piece related disturbances can influence the execution. They are indicated by the environmental disturbances actor.

Use Case Mock-ups

This is a backend/hardware development component, hence it does not require front-end interfaces. Instead a mockup-scenario inspired on the KEBA pilot is explained below. Many of the details of this specific example are subject to change:

• There is a library with a set of skills such as a skill to avoid humans in the environment for a force-controlled insertion or a skill to use vision to grasp a printed circuit board(PCB).

• The generic application could be placing a PCB in a holder, where the PCB is located using vision, and where the fitting with narrow tolerances of the PCB is aided using force control, and the human operator in the environment is avoided.

• The demonstrator deploys the application to a specific factory site for a specific holder and PCB-type by using programming by demonstration (PbD) to demonstrate a number of position/force trajectories using one of the HMI modalities. These are then used to learn a statistical distribution of allowable trajectories used for execution

• The generic application and the models and parameters learned from demonstration are then used during execution.
• During execution the operator works in the environment of the robot. Additional constraints on the allowable trajectories are added to avoid the operator and to impose the location determined by vision. These additional constraints are continuously updated to deal with (observed) disturbances.

In the above example, there could be many deployed applications, corresponding to different combinations of PCB's, holders and robot-cells.

Mock-ups and more information related to the component's usage can be found in the online documentation [LINK]

Functional Specifications

Functional Block Diagram

The diagram represents the functional specification of the AI-Prism PbD environment. Considering a library of skills, the application developer makes an analysis of the definition of the application to determine whether any new skills are required to be added to the library. The system developer develops new skills. Such a new skill consists of constraint-based task specifications in the eTaSL language and of finite state machines described in rFSM. The eTaSL controller can interpret these specifications and execute a robot controller with the defined behaviour. These skills describe how the robot should behave and how it should react to disturbances. It can not only contain aspects of reactive control such as force-control or visual servoing, but also basic motion planning. After a validation cycle, newly defined skills are added to the library and used to configure and assemble skills into a "generic" application.

In these application specifications, aspects of some tasks are left open and are handled by a programming-by-demonstration model. These demonstrations do more than just a teach-in of a trajectory, they specify the nominal value and the allowable variations of some degrees of freedom in the application. These can be trajectories, forces or other aspects of the task. At execution this demonstration model will be combined with the previously mentioned constraints in order to determine the concrete motion.

Often, the same generic application, will be used multiple times in different contexts: e.g., another factory site, with different work pieces or in a different work cell. Demonstrations will be performed in order to deploy the generic application to these specific contexts.

Main interfaces

List of main interfaces between functional components shown in the figure.

ID	Component	Name	Description	Sense
1	eTaSL controller during PbD	eTaSL PbD specification	A constraint-based robot controller governing the robot behavior during demonstrations	In
2	eTaSL controller during execution	eTaSL execution specification	A constraint-based robot controller governing the robot behavior during execution	In
3	eTaSL controller during PbD	Force/motion model	Probabilistic model of force/motion interaction and its allowable variations	Out
4	eTaSL controller during execution	Force/motion model	Probabilistic model of force/motion interaction and its allowable variations	In

Interaction with additional sensing systems and the HMI module are described inside the eTaSL specifications and not further detailed in this list.

Sequence Maps

The following shows a sequence diagram illustrating a typical interaction between the subcomponents of the programming-by-demonstration module and the HMI module.

The eTaSL controller is a general purpose constraint-based controller that reads an eTaSL task specification and controls the robot system to implement the specified behavior. In the use case below, two instances of this controller module are used:

• the eTaSL demonstration controller that implements the behavior of the robot during demonstrations. For example, it implements a force/torque controller that allows the demonstrator to take the robot end effector and guide it along a desired path (called kinaesthetic demonstration), while at the same time the robot is limited to a pre-defined workspace.
• the eTaSL application controller that implements the behavior of the robot during execution.

Both instances of the controller module takes a controller specification as input. This specification is a constraint-based task specification described in the eTaSl task specification language. This specification language also allows a probabilistic force/motion model as input.

The sequence diagram shows the following steps:

• The application developer creates a task specification for the controller that is active during kinaesthetic demonstration of the task.
• The application developer creates a task specification that can describes how to control the application at hand. Because some of the details and motions are not known during design time, this specification uses probabilistic force/motion models to describe the unknown parts. Later these models will be trained from demonstration data.
• For a particular deployment of the application at a given site/location, the demonstrator demonstrates the missing force/motion interactions. Often, this demonstrator is the same person as the operator. These interactions use the (continuous) HMI and can be bi-directional. These data are recorded and sent to the probabilistic force/motion model. Optionally, this probabilistic force/motion submodule can compute an intermediate probabilistic model to facilitate the next demonstrations using a shared control where both robot controller and human can influence the demonstrated motion.
• When a sufficient number of demonstrations is performed, the probabilistic force/motion model is constructed and send to the eTaSL application controller
• The eTaSL application controller then executes the application while interacting with the environment (external disturbances) and the operator (typically using a discrete HMI, such as a GUI or buttons/lights). This interaction is typically bi-directional.

Additional Information component-related