SIMULATION ENVIRONMENT (SE)

Introduction

The Simulation Environment is a tool that allows photorealistic 3D representations of a collaborative manufacture environment, including robots, industrial equipment and human agents. Its development is motivated by the idea of helping industrial partners to simulate processes in a more realistic way, enhancing the state of the art of current Digital Twins and considering human operators in the simulation. With the Simulation Environment, users will be able to recreate production plants and lines, taking into consideration interactions between the existing agents in the environment and integrating human avatars. As a main characteristic, the avatars of the SE include the definition of human (workers) skills, how they change dynamically over time, and the simulation results are focused on production results and scheduling.

Technical Specifications

The technical specification diagram below represents the architecture of the Simulation Environment and its main technical components. The Simulation Environment Extension (SE Extension) module have a frontend which provides all the user interfaces needed to complete the actions mentioned in the Use Case and Functional Specifications diagrams. A backend service is used to handle the information provided in the frontend, retrieve, and save data from the database. Also, the backend service, sends all information to the 3D visualization in Isaac Sim frontend and carry out the simulation itself. The Database will register the needs of the production line (i.e, shift production, types of products) and contain the information regarding the capabilities available (i.e, number of machines available, workers, robot specifications).

Software and Hardware Requirements

The Simulation Environment is an application based on the NVIDIA Omniverse Platform, consequently to work with it is necessary to fullfil some requirements.

Software Requirements: - NVIDIA Omniverse Platform: NVIDIA Omniverse Platform: Download and install the NVIDIA Omniverse. There will be available all NVIDIA Omniverse specific apps.

Hardware Requirements: - NVIDIA GPU: Omniverse applications are GPU-intensive and require an NVIDIA RTX GPU with dedicated ray-tracing cores, such as GeForce RTX 5080. Consult the up-to-date GPU requierements in the official documentation - System requierements: Consult the system requierements in the official documentation

Internet Connection: - An active internet connection is necessary to access and use Omniverse services, cloud collaboration features, and updates.

Recommendations: - Most of the specifications before mentioned refers to Windows computers. In case you are planning to use Linux or any specific deployment, please refer to the NVIDIA Omniverse Technical Requirements to get the suggested technical requirements.

Component Deployment

Isaac Sim NVIDIA Omniverse is the foundation for the Simulation Environment (SE). Please refer to the documentation provided above to check the technical requirements of this software and set it up. To get access to the SE, users need to install the given extension as follow.

Open NVIDIA Isaac Sim, from the Omniverse launcher. This first mock-up represents the software open with an empty project.

From the top menu, go to “Window” and click in “Extensions” from the drop-down menu.

A window call Extensions will appear near the Viewport. It includes all Extensions available in the system. The simulation Environment Extension will be there too.

Usage Manual

Use Case Diagram

The simulation platform allows the user to simulate the scenarios of the pilots to test the developed components. This environment provides the user with a tool to optimize their own solutions by applying them in simulations of target scenarios. The following elements describe the main use cases shown in the use case diagram.

• Select Agent: Through the UI, the operator will choose from the catalogue of the developed elements (e.g., robots, cobots, human avatars), all the agents needed to describe the specific layout to simulate.

• Configure: For each selected agent, the operator has to configure a basic setup that includes information related to the real agent to simulate. For example, in a conveyor, this information could include the maximum speed to transport the product.

• Position: After the configuration of the agents, the operator has to position the elements in its spatial coordinates and indicate the sequence between agents.

• Save layout: Not mandatory feature that allows the user to save the current layout with the filled information of the elements.

• Run simulation: Executes the simulation of the pilot use case (Silverline)

Additionally, a second extension was created to evaluate the physical interaction between the assets displayed in the scenario.

Use Case Mock-ups

Keep in mind that two Extensions are available inder this component. The first one, called Assets Simulation allows the user to simulate the human-robot collaboration workflow of Silverline use case (pilot of AI-PRISM project). The second one, called Simulation Environment, permits the user to create their own set ups by adding human workers, robots and cobot to an scenario and, evaluate the assets physical interactions.

Assets Simulation Extension

Once the extension is clicked, some information regarding it will appear in the screen. Install the extension.

Once it is installed, the user can activate it by clicking the toggle button of the Extension.

Click on the extension to open it.

The side panel of the extension will appear. Click on Load extension.

Once the scenario is loaded, press the Run Scenario drop down.

Press on Run scenario, to simulate the human-robot collaboration workflow of Silverline use case

Click Stop to freeze the simulation.

Press Reset button to reset the scenario.

Simulation Environment Extension

Once the extension is clicked, some information regarding it will appear in the screen. Install the extension.

Once it is installed, the user can activate it by clicking the toggle button of the Extension.

Click on the extension to open it.

The side panel of the extension will appear. Click on the Assets configuration drop down.

This drop down will allow the user to configurate all the assets (human, robots and cobots) that are loaded to the scenario. First, select the number of assets to be loaded.

Click Load Assets. Initially, the assets will be loaded in the shape of color coded cylinders (pink for human workers, yellow for cobots and blue for robots).

Once the cylinders appear in the scenario, position them in their final location.

Click Fix Positions.

Once the cylinders are positioned, the user can start to configurate them. In the Asset selection drop down, choose the first cylinder to configure. Users can double-check the cylinders in the Stage panel (right side of the screen)

Click the second drop-down called Asset assignment and choose the asset to substitute the cylinder selected before.

Once selected, click Upload Asset.

The asset will appear in the stage inder a folder called Assets.

Repeat the 3 previous steps until all the assets are in the scenario. Then click Configuration Done.

Click on the Assets interaction drop down.

Group the staged assets in workstations by creating workstations names and selecting the asset belonging to it.

Click Assign Asset to include the selected asset under the workstation. Complete these 2 previous steps for all the assets in the scenario.

Click Show Interaction to see the working envelopes of the assets and the p-HRI KPIs.

Functional Specifications

Functional Block Diagram

The functional specifications in the diagram below represent the flow of the main actions in the Simulation Environment. The activities start with the selection of an agent (e.g. robot, conveyor, human...) from a list of available agents. If an agent is selected, the agent should be configured. The configuration is related to the key features that define the agent previously selected. In this step, the user should introduce the properties that will determine the behavior of the agent. Following, in case the agent has been correctly configured, it will be necessary to position it in the scene. To perform this action, the user must provide the spatial coordinates in which the agent will be located. Once all the agents composing a scene are positioned, the user should decide if the definition of the layout is completed. If the layout is defined, it can be saved, with the option of storing it in the database for later use or directly running the simulation.

Main interfaces

List of main interfaces between functional components.

ID	Component	Name	Description	Sense
1	Isaac Sim FE	Extensions Panel	NVIDIA Isaac Sim's interface to open Omniverse extensions	In
2	SE FE	Layout information	Selection of the target layout in which the simulations will be executed	In
3	SE FE	Configuration information	Starting or loading a configuration. This implies: robots, cobots, human operators and their positions in the layout	In
4	SE FE	Simulation Parameters	Definition of which parameters to simulate	In
5	Simulation Engine	Simulation Results	Results of the simulation	Out
6	Simulation Engine	p-HRI KPIs	Results of the simulation	Out