AI-BASED AMBIENT LEVEL REASONING MODULES (CR)
Introduction
The AI-based Ambient Level Reasoning Enhancing Modules (CR) are a set of modules that enable the ambient level reasoning, which is the reasoning that involves the coordination of activities performed by agents operating in the ambient environment.The vision is to develop standalone cloud applications (ie modules) specifically designed to solve different reasoning problems. Some examples of the problems in the scope of AI-based ambient level reasoning include: task allocation, task scheduling, task assignment, and optimal routing. Ambient level reasoning also encompasses modules that enable the control of operations at the ambient level by monitoring events and taking responsive actions, such as reallocating tasks or adjusting routing, to ensure optimal performance.
Agent level reasoning and ambient level reasoning are complementary. Agent level reasoning encompasses lower-level reasoning tasks focused on individual robot agents and their immediate environment both in time and space (ie, ensure smooth interaction and the completion of planned tasks for a given collaborative robot). Ambient level reasoning on the other hand involves higher-level reasoning problems that focus on larger-scale time horizons and manufacturing environments. In a generic sense, ambient level reasoning is concerned with the definition and configuration of the manufacturing environment (what resources are available, what tasks are to be performed, with what resources, and when) and the control of operations (monitoring the execution of the manufacturing process and adjusting the allocation of resources to ensure optimal performance). Ambient level reasoning is also concerned with the coordination of agents, distributing the tasks to be performed among the available resources, and dispatching the planned tasks to the agents to ensure the completion of the manufacturing process. Agent level reasoning is concerned with the execution of the tasks that in general are assigned by the ambient level reasoning modules. This separation establishes a hierarchical control structure that facilitates the control of the manufacturing process at different levels, and at the same time, it fosters modularity, scalability, specialization, and reusability of the reasoning modules:
- Modularity: The separation of reasoning tasks into agent level and ambient level reasoning modules enables the development, testing, and deployment of modules independently. This means that improvements or changes in one module do not necessarily require changes in other modules.
- Scalability: As the system grows in size and complexity, the system is able to scale up by adding more agents or new modules without overhauling the entire system.
- Specialization: Modules in each level can be optimized for specific tasks and environments. Agent level reasoning can be optimized for real-time, low-latency reasoning, while ambient level reasoning modules can be optimized for high-level planning and coordination.
- Reusability: Modules can be reused in different environments and scenarios. For example, a task scheduling module can be reused in different manufacturing environments.
- Fault tolerance: If one agent fails, ambient level reasoning can reassign the tasks to other (human or robot) agents to ensure the completion of the manufacturing process. Conversely, if there is a failure in ambient level modules, the agent level modules can continue to operate and complete the assigned tasks based on the last received instructions.
At the time of writing, the following modules are envisioned for ambient level reasoning:
-
Holistic AI-PRISM Planning Module: This module uses a unified planning framework to solve general optimization problems found in any collaboration ambient. The module queries the AI-PRISM Data Platform services to read the input data (according to the AI-PRISM unified data model definitions) and instantiate optimization problems in the ambient. The instances are solved using a generic planning tool, and the results are written back to the Data Platform services, so that they are available to other modules. This module is being developed by FBK and its implementation takes an incremental approach, will take an incremental approach, focusing first on general problems like multi-agent task planning, and then adding others like mobile robot path planning depending on the resources available, the needs of specific pilots, and the research interests of involved partners. The unified planning framework will be used to develop this module.
-
AW Ad-hoc Scheduling/Re-scheduling Modules: These modules are specifically designed to optimize production scheduling in the Andreu World pilot, based on different (and possibly conflicting) optimization objectives, including human / social sustainability objectives. It works in a similar fashion as the AI-PRISM planning module: it will read the data from the Data Platform services using the unified AI-PRISM data model, solve the problem, and write the output back in the AI-PRISM Data Platform. As mentioned in the previous paragraphs, ambient level reasoning modules use the AI-PRISM Data Platform services to read the input data to reason with and write the results of the reasoning process so that they are available to other modules. In this sense, all reasoning modules use the same unified data model to represent the collaborative ambient at different levels, and the AI-PRISM Data Platform services provide the necessary services and interfaces to read and write data in the unified data model.
Technical Specifications
The technical specification diagram represents the architecture of an ambient reasoning module (task scheduler) and its components. It includes a frontend application for user interfaces, a modeling backend for database connectivity, and a northbound communication module for integration with enterprise systems. During production, the operations control component connects to the task scheduler for rescheduling based on real-time information from the collaboration environment.
This second diagram is a detailed representation of the technical specifications of the scheduler module.
Software and hardware Requirements
Each module is a standalone cloud application that can be deployed in any cloud environment, supported on the AI-PRISM CI/CD Framework for AI-based Solutions (CD). Each module is deployed as a Kubernetes application, composed of one or several containers. Each application implements a set of files describing how the application should be deployed in a Kubernetes cluster (Helm chart) and defines the minimum computing resources required to run the application (CPU and memory), together with the dependencies with other reasoning modules or AI-PRISM components.
Deployment and installation
The AI-PRISM CI/CD Framework for AI-based Solutions (CD) supports each module as a standalone cloud application that can be deployed in any cloud environment. As it is illustrated in the diagram below, the CI/CD infrastructure provides a Git Deployment repository, a docker image registry, and potentially an MLFlow model registry. The Git repository will contain Helm charts, which are configuration files for managing the deployment of platform components or sub-components as stand-alone applications in a Kubernetes cluster. A cloud application is a Kubernetes application composed of one or several microservices that are deployed in a Kubernetes cluster from a Helm charts folder containing the files required to deploy and configure an instance of the application in that cluster. Based on this definition, each ambient reasoning module will be a stand-alone cloud application. The cluster management as it can be seen in the Functional Specifications' section provides a visual and simple way to manage one or several Kubernetes clusters that conform to an AI-PRISM installation. This component manages the deployment of the Apps in a cluster or clusters managed by the final user and provides UIs for users to browse the deployment repository and select which cloud applications to install. Furthermore, the user can also define some specifications of the deployed applications, such as the number of replicas, the CPU and memory resources, and the dependencies with other components.
To deploy each module as a Kubernetes application, it is necessary to maintain Helm charts in the deployment repository. It's important to note that it's not mandatory to push all source code to this repository, only the Helm charts, which use docker images in the Docker registry, so partners are free to maintain source code in one or multiple separate git repositories, as long as they push their Docker images to the Docker registry and the Helm charts to the Git deployment repository. A Helm chart is a package that contains all the necessary resources to deploy an application to a Kubernetes cluster. It includes YAML files that describe the Kubernetes resources required to run the application, such as deployments, services, and ingress rules. By organizing the application's components into modular charts that can be easily installed and upgraded, Helm simplifies the process of managing application components. It can reduce the amount of manual work required to maintain an application and helps to avoid errors and inconsistencies that can arise when managing complex systems manually.
Usage Manual
Use Case Diagram
The ambient reasoning module is designed to enhance collaboration among agents in a dynamic environment. In this usage model, three agents have been identified as responsible for managing and administering the modules. These agents are categorized based on their specific action purposes and engage with different tools or interfaces that enable effective interaction with the system. The aim is to ensure that each agent can perform their tasks efficiently and effectively in a dynamic environment.
Following the use case diagram, the users, according to their role, can interact with the system through the following actions:
-Browse and Select. This involves browsing through the catalog of available reasoning modules and performing actions related to navigating through different modules for ambient level reasoning that are ready for deployment and use. Two actors have access to the "Browse and Select" interface: the production manager, who identifies the required applications and requests their installation, and the system administrator (sys. admin), who manages and deploys these modules within the system.
-Configure: Once a tool is installed, certain configuration is necessary to adapt it to the deployed environment, including the Industrial Process Definition. This can be divided into two levels: - Configure System Connections: The sysadmin ensures that the system is correctly configured and connected to the rest of the system and other modules, if necessary (Configure System Connections). This activity may involve configuring connections to other corporate software, such as the Enterprise Resource Planning (ERP) system.
- Configure Reasoning: On the other hand, the production manager ensures that all the inputs required by the reasoning modules are accurate. This entails defining or validating the definition of industrial processes that will be used by the reasoning modules. For example, it involves verifying the correctness of information related to production orders, Bills of Materials (BoMs), resources, worker shifts, and so on. It also includes configuring or validating the setup of triggering events that will guide the execution of the reasoning modules.
-Validate Reasoning: Production managers can request the execution of a reasoning module for a given input or set of inputs. The reasoning module then processes the configured input data and provides an output as a reasoning result, such as an action to be taken. The production manager can review the reasoning results and validate them if they align with the intended goal. For example, the production manager can obtain the result of a planning or scheduling reasoning module for a specific set of production orders to validate the results.
-Monitor Process Execution Performance: Operators and production managers can monitor the execution of the industrial production process within the collaborative ambient. This monitoring allows them to assess the performance of the process and ensure alignment with the targeted goals. The reasoning modules automatically take actions to achieve these goals based on the provided configurations. Additionally, they provide recommendations or suggestions to the individuals involved in the process to facilitate an appropriate response to unexpected events. Operators and production managers can also visualize the performance of previous (historic) executions of the production process.
-Explain Reasoning: In certain cases, the actions performed by the program or the changes applied may require additional explanation for the production manager or operator to fully comprehend or provide more data for decision-making. The interfaces should provide this information when necessary.
Use Case Mock-ups
The following mock-ups show the interfaces through which the actors interact with the system in each use case.
-Browse and Select The first interface shows the deployment and initial configuration of the modules to be implemented to interact with this module. Through this interface, the system administrator can browse the catalog of available modules, check those already deployed and install new ones. It uses Helm Charts to define, install and upgrade Kubernetes applications.
Through this process the system administrator can also monitor all deployed modules, define the minimum computing resources required to run the application or other restrictions, and other configuration aspects such as the dependencies with other reasoning modules or AI-PRISM components.
-Configuration
The previous interface is mainly related to the system administrator role, but the following interface can be also used by the production manager. The configuration wizard allows the user to complete step by step the module configuration process to adapt it to the environment in which it is going to work. This includes establishing all the necessary connections with other deployed modules, ERP, etc. to set up the company's features in a guided and user's friendly way.
Following this initial configuration process, the production manager can configure the reasoning module and check the different industrial processes that are going to be used.
-Validate Reasoning
The project manager can select and request the execution of a reasoning module for a given input or set of inputs. The reasoning module will provide an output, such as an action to be taken, as a reasoning result after processing the configured input data. The production manager can review the reasoning results and validate them if they are aligned with the intended goal. In the mock-up interface the production scheduling obtained after the reasoning process is shown in a gantt format, where the production manager can check all data related to the production orders, resources, etc. before validating the result.
-Reasoning Details
The production manager can also check the details of the reasoning process, including the main variables and constraints used in the reasoning process, the different actions taken, etc. This information can be used to understand the reasoning process and to explain the results obtained.
-Monitor Process Execution Performance
The production manager can monitor the execution of the industrial production process within the collaborative ambient through the Performance interface.
-Available process programs
The specialist can check and select from the programs that are already available in the system to perform an action as set out in the production order included in the planning. In case there is no program for a complete part, the process must be done by the specialist and not by the robot, since it is not taken into account to perform the production planning until there is no program for that action in the system.
The available programs will be shown in tables, and will be classified according to the process (sanding, painting) as can be seen in the mock-up of the interface below the text. Each process will have different programs depending on the product on which it is applied or other characteristics that affect it. In addition, each program may have several options available that will correspond to different trainings that have been done with that program, although there will always be one that is set as the main one. The latter will be the one that will be applied by default until another one is selected. The operator can navigate between each one and see the specifications it presents, such as creation date, duration time, model on which it is applied, etc. When a new delivery program is included, the result must be validated before being available for production, so the operator in charge of the selection during the production process will not be able to choose it until then.
Functional Specifications
Functional Block Diagram
Generic Functional Block Diagram
Each ambient reasoning module will have a different functionality, but they will all share a common technical architecture, common data model and generic functional blocks, which is shown in the figure below.
The main functional blocks identified in the generic functional block diagram are:
- Model Industrial Process: This functional block encompasses the modeling and definition of the industrial processes the reasoning modules will act upon. This implies that the inputs coming either from the users (e.g. through a configuration interface as mentioned in the previous section) or external systems (normally the ERP) will be transformed into a model of the industrial process that the reasoning modules can understand and reason with.
- Define Tasks: These models will also define the tasks (or operations, or units of work) that will be executed by the agents (robots or humans) in the collaboration agent. These definitions of the operations do not necessarily need to include low level descriptions of the robot programming (for instance, path trajectories), but rather, higher-level definitions relevant for operations management. An example of a task can be ‘paint piece with code 09890’. These high-level models will be linked to other lower-level models used for programming or reasoning at the agent level, and will include models of the (time) performance of the execution of the task, which can be used for operations management at the ambient level (e.g. planning, scheduling, operations control). Again, this might imply transforming or processing low-level models to extract the information that is relevant for operations management. The tasks defined should also be approved by production managers, so that they explicitly agree that the performance of the individual task is in agreement with the objectives of the company.
- Compute reasoning result (e.g. Create Task Scheduling and assignment): Based on these definitions, each reasoning module will implement functions to request a reasoning result (for instance to create a task scheduling or assign tasks to robots in the ambient) that will be triggered by the user or an external system through an input, and will generate a reasoning result conveying a set of coordination actions in the collaboration ambient. For instance, the result of a task scheduling module will be a detailed schedule of the execution of tasks coordinating different agents in the ambient. The user may approve the resulting decisions or actions, or request a different result, so as to establish a target overall process performance. For instance, the production manager can request a production scheduling or an assignment of tasks to agents, and approve these results if they provide an acceptable performance according to different criteria, so that they can be used in production. These actions will be also expressed using the industrial process model definitions.
- Control Operations Execution: The reasoning modules may also implement functions to monitor the execution of the industrial process, based on the information collected from the collaboration ambient and the perception modules, and based on this input (e.g. people working close to the robot or position of the part being operated on), assess the actual process performance.
- Take Responsive Actions: Reasoning modules may take actions to ensure that the execution of the tasks is in agreement with the planned schedule. For instance, if a robot is no longer available, the reasoning module can reassign the task to another agent (any other robot or human agent able to perform the task), so this new plan will be sent to the IIoT platform, which will forward it to the new assigned agent.
Concrete Functional Block Diagram (AW Ad-hoc Scheduling/Re-scheduling Modules)
In the AW pilot the Ambient reasoning module is applied to carry out the weekly and daily planning. To do so, the input information that is required in the scheduling reasoning module regarding pending manufacturing orders will be read from the company's ERP, which will include information such as:
- The products that need to be processed, their delivery dates and other characteristics such as color, priority of the production order and quantity of product;
- Additional information will also be read for the creation of the plan, such as the speed of each operation with respect to each product at each stage of the process;
- Changeover times according to pairs of product and operation characteristics, for example the time needed to clean the paint system when changing between two colours, this information must be given for each possible combination of color pairs, part to produce and operation that is applied. * The routes or production programs, that is, the set of operations that is applied to each product according to its characteristics such as color and type of product.
- Existing programs in the robotic cabin. As part of this project, a paint booth will be robotized and the robot will be trained to perform painting operations. In this case, the list of parts (with types of parts and colors) that are already delivered to the robot must be available to be able to schedule production.
- Continuing with the case of the robotic cabin, information regarding the training capacity must also be available, that is, how many pieces can be trained per day. In the event that any of the required data is not available in the company's information system, an initial measurement or an estimate of all these values must be made using the knowledge of the workers. When a robotic cabin does not have the capacity to carry out the process of a part, the solution will propose a date for the training of the robot.
Certain information necessary for daily production scheduling is not present in the ERP and must be stored in an independent data source (although it could also be added to the ERP if possible), such as the hourly availability of workers in the sanding stage. This information will make it possible to modify the production speed of the sanding stage during the working day. The data source can be a database, plain text files or Excel files.
Based on the information about the pending manufacturing orders, the manufacturing capacity of the plant (including the capacity and programs present in the robotic paint cell or cells), the personnel available for sanding, the production programs for each piece, etc. The AI-PRISM solution will propose the manufacturing orders to be carried out each day of the week and the order of their processing within each day in order to maximize the use of all the plant's resources, avoiding unnecessary stops in the paining cell cabins and taking into account the capabilities of each resource involved. In the case of robotic painting cells, the time for robot training will also be proposed when parts to be processed appear for which the robot has not yet been trained.
The solution contemplates the possibility that one or several paint booths are robotized cells, there is even the possibility that the cells are temporarily robotized (the robot arm can be changed to another cell or taken to an outside cell for training). In this case, the robotic arm schedule should be available as input data to the scheduling solution. In the process of creating the weekly schedule and daily rescheduling, the availability of the robotic cabin is taken into account. In the event that the robotic booth is going to be used during the week that production is being scheduled, the AI-PRISM solution will take into account whether the robot has already been trained to process the assigned parts. If the program to paint a work order part is not present on the robot, the solution will move this work order to a later day and modify the proposed robot training plan so that the robot is trained before the work order needs to be processed. As a consequence of the creation of the weekly production plan, a weekly robot training plan will also be created when it is necessary to process work order parts for which the robot has not yet been trained.
At the beginning of each day, the list of working orders to be processed is obtained. This list comes from the weekly planning and contains a list of working orders to be processed on the production line following the proposed order. The daily working order list contains an ordered list of products to be processed that includes the necessary information for its process, such as the operations that comprise it and product characteristics such as the type of product and color that must be applied.
There may be events or states that prevent a manufacturing order from being processed at the time that corresponds to it according to the daily plan, for example, that the robotic painting cell has not been trained following the proposed training plan, that the product to be processed is not in stock due to delays in transport or that there are no stocks of upholstery products necessary for the product. When any of these cases occurs, the personnel in charge of the line may cancel a manufacturing order proposed for that day. When canceling an order you must justify the reason for cancellation (like any of the examples above). At this time, rescheduling will be executed, which consists of removing the indicated manufacturing order, trying to exchange it with some other manufacturing order that must be processed later in the same week. If this is not possible, the manufacturing order will be removed from the planning. weekly and an attempt will be made to enter another pending order obtained from the ERP. This process is interactive and iterative, since several changes to the daily production plan can be proposed on the same day and this will affect both the plan for the current day and the weekly plan also in progress.
Main interfaces
Generic Interfaces
Based on the functional diagram, the following table shows the main generic interfaces in the Ambient reasoning Modules, including the ones between the different components of it and the ones between the system and the user, through which the data is exchanged.
| ID | Component | Name | Description | Sense |
|---|---|---|---|---|
| 1 | Configuration | General Configuration | Interface to configure the deployment characteristics of the system. | In |
| 1.1 | ERP Configuration | Industrial process definition | Through this interface the module is configured to connect to the company's ERP to get different production information. | In |
| 1.2 | Definition of operations | Task definition | The task definition interface allows the user to create instances for every different operations. | In |
| 2 | Reasoning service | Reasoning result | The reasoning module will provide the result of the algorithm based on the input data. | Out |
| 2.1 | Reasoning service | Accept / Rejected result | After getting the result of the reasoning module, the PM can approve or reject it. | In |
| 3 | IIoT Platform | Ambient data | Real-time information related to the ambient data to adapt to the new situation (operator standing or standing in the area, etc.) | Out |
| 3.1 | IIoT Platform | Perception data | Process information that affect the performance, such as part position that might require some adjustment in the schedule. | Out |
| 4 | Reasoning service | Corrective action | After a corrective action, the result is sent to the performance agents. | Out |
CONFIGURATION ERP Connection To model the industrial process, the system needs to connect to the company's ERP to get different production information. It is done through the configuration interface, where the user should complete the different fields in the form to configure the connection with it, and be able to work with the production information, such as product model, colour codes, orders, stock of materials, etc. Some of the information that is required to connect with the ERP includes the ERP details (type, version), the ERP URL, the ERP access credentials (user, password, tokens, certificates), the ERP connection protocols (SOAP, REST, etc.), data transfer frequency (how often should the system exchange information with the ERP), and other fields depending on the ERP type and version.
Task definition The task definition interface allows the user to create instances for every different operations that are part of the industrial process, and the different parameters that are needed for each of them. Through it, the user links this high-level tasks with the low-level operations that are executed by the agents (robots or humans) in the collaboration agent, the cycle time of each operation, the resources that are needed to execute them, the different parameters that are needed for each operation, etc.
REASONING SERVICE After a reasoning process result is provided, the PM can accept or reject it. If the result is accepted, the system will process it and send it to the performance agents. If the result is rejected, the system will try to find another solution following the new specifications provided by the PM.
AMBIENT STATUS INPUT The ambient status input interface provides real-time information related to the ambient data to adapt to the new situation, so when the ambient status changes, the system can adapt to it. For example, if the operator is standing or standing in the area, the system can adapt the schedule to the new situation, as it may lower down the speed or stop for a while until the operator leaves the area.
PERCEPTION MODULES INPUT The perception modules input interface provides information about the process, such as the position of the part that might require a reevaluation of the process.
CORRECTIVE ACTION The corrective action interface provides the result of the corrective action to the performance agents, so the PM can check and follow the new schedule.
Concrete Interfaces (AW Ad-hoc Scheduling/Re-scheduling Modules)
List of main interfaces between functional components shown in the figure.
| ID | Component | Name | Description | Sense |
|---|---|---|---|---|
| 1 | Import and Validation | Validated data | The service Import and Validations connects with different data source and validates and formats data that is needed in other services | Out |
| 2 | Scheduling service | Weekly Schedule | The Scheduling Service module is in charge of generating weekly schedule for the painting (and sanding) plant and a weekly robot training schedule | Out |
| 2.1 | Scheduling service | Weekly robot training plan | The Scheduling Service module is in charge of generating weekly schedule for the painting (and sanding) plant and a weekly robot training schedule | Out |
| 3 | Web interface service | Daily schedule | allows user to accept or reject scheduled working orders, generated a new daily schedule and also update the weekly schedule if necessary | In |
| 3.1 | Web interface service | Rejected orders | the user can reject scheduled working orders this will cause that the Scheduling service generate a new daily schedule and also update the weekly schedule if necessary | Out |
| 4 | Export | Generated data | exports work order schedules and robot training schedules to information systems and storage services | Out |
| 5 | Data storage service | Weekly schedule | stores all data generated by the solution like, weekly schedule, daily schedules and robot training plan | In |
| 5.1 | Data storage service | Weekly robot training plan | stores all data generated by the solution like, weekly schedule, daily schedules and robot training plan | In |
| 5.2 | Data storage service | Daily schedule | each beginning of the day the current day schedule is obtained from the storage service | Out |
| 5.3 | Data storage service | Weekly schedule + robot training plan | Weekly schedules, weekly robot training plan and its updates can be exported to other information systems | Out |
Sequence Maps
The following sequence maps show the general interactions between the functional sub-components. The first one represents the main sequence from the user perspective, starting with the configuration of the system and ending with the export of the first result of a reasoning process.
The second one represents the sequence of the adaptation process when there is a deviation from the reasoning process, requiring a new reasoning process to be executed. It starts with the user creating and validating a new reasoning process (which is sent to the IIoT Platform, so it gets to the right agents). Through the performance process, the data from the ambient and perception modules is checked. If a deviation appears, a new reasoning process is required, which may be rejected and relaunch until the result meets the target and is approved. Then, the result is sent to the performance agents, and the process continues.
The last sequence map shows the interactions between the functional (sub-)components from the AW daily schedule use case.
