SIMULATION ENVIRONMENT (SE)

Introduction

Explain what is the component about and its architecture with a small architecture diagram.

The Simulation Environment is a tool that allows photorealistic 3D representations of a collaborative manufacture environment, including robots, industrial equipment and human agents. Its development is motivated by the idea of helping industrial partners to simulate processes in a more realistic way, enhancing the state of the art of current Digital Twins and considering human operators in the simulation. With the Simulation Environment, users will be able to recreate production plants and lines, taking into consideration interactions between the existing agents in the environment and integrating human avatars. As a main characteristic, the avatars of the SE include the definition of human (workers) skills, how they change dynamically over time, and the simulation results are focused on production results and scheduling.

The Simulation Environment is comprised of: - SE Frontend - SE Manager - Simulation Engine - Skills Engine - ECM Engine

Technical Specifications

The technical specification diagram below represents the architecture of the Simulation Environment and its main technical components. The Simulation Environment Extension (SE Extension) module have a frontend which provides all the user interfaces needed to complete the actions mentioned in the Use Case and Functional Specifications diagrams. A backend service is used to handle the information provided in the frontend, retrieve, and save data from the database. Also, the backend service, sends all information to the 3D visualization in Isaac Sim frontend and carry out the simulation itself. The Database will register the needs of the production line (i.e, shift production, types of products) and contain the information regarding the capabilities available (i.e, number of machines available, workers, robot specifications).

Software and Hardware Requirements

Which software and hardware are required to run the component. The Simulation Environment is an application based on the NVIDIA Omniverse Platform, consequently to work with it is necessary to fullfil some requirements.

Software Requirements: - NVIDIA Omniverse Platform: NVIDIA Omniverse Platform: Download and install the NVIDIA Omniverse. There will be available all NVIDIA Omniverse specific apps.

Hardware Requirements: - NVIDIA GPU: Omniverse applications are GPU-intensive and require an NVIDIA RTX GPU with dedicated ray-tracing cores, like RTX 30-series GPUs with minimum 10GB. - System Memory (RAM): Minimum requirement is 32 GB (as the SE uses Isaac Sim application). - Extra information:

Internet Connection: - An active internet connection is necessary to access and use Omniverse services, cloud collaboration features, and updates.

Recommendations: - Most of the specifications before mentioned refers to Windows computers. In case you are planning to use Linux or any specific deployment, please refer to the NVIDIA Omniverse Technical Requirements to get the suggested technical requirements.

Component Deployment

NVIDIA Omniverse is the foundation for the Simulation Environment (SE). Please refer to the documentation provided to check the technical requirements of this software and set it up. To get access to the SE, users need to install the given extension as follow.

Open NVIDIA Isaac SIM, from the Omniverse launcher. This first mock-up represents the software open with an empty project.

From the top menu, go to “Window” and click in “Extensions” from the drop-down menu.

A window call Extensions will appear near the Viewport. It includes all Extensions available in the system. The simulation Environment Extension will be there too.

Once the extension is clicked, some information regarding it will appear in the screen. Install the extension.

Once it is installed, the user can activate it by clicking the toggle button of the Extension.

Usage Manual

Use Case Diagram

The simulation platform allows the user to simulate the scenarios of the pilots to test the developed components. This environment provides the user with a tool to optimize their own solutions by applying them in simulations of target scenarios. The following elements describe the main use cases shown in the use case diagram.

• Select Agent: Through the UI, the operator will choose from the catalogue of the developed elements (e.g., robots, conveyors, human avatars), all the agents needed to describe the specific layout to simulate.

• Configure: For each selected agent, the operator has to configure a basic setup that includes information related to the real agent to simulate. For example, in a conveyor, this information could include the maximum speed to transport the product.

• Position: After the configuration of the agents, the operator has to position the elements in its spatial coordinates and indicate the sequence between agents.

• Save layout: Not mandatory feature that allows the user to save the current layout with the filled information of the elements.

• Run simulation: Executes the simulation and provides the user with production parameters and test the user´s specific tools.

Use Case Mock-ups

The following set of mock-ups show a few of the interfaces that the actors utilize to communicate with the system.

After activating the extension [LINK TO COMPONENT DEPLOYMENT], a new window called “Simulation environment” will appear in the screen. The user can move it and position it where it suits better. This window has a top menu consisting in three tabs: 1. Layout. 2. Configure. 3. Summary.

These tabs compose the needed steps to simulate the scene.

1. Layout: If the user clicks in the “Layout” tab, new information will be displayed. This window is made to select the predefined layout to simulate.
   1. Layout options buttons.
   2. 3D preview of the selection.
   3. “Select layout” button.

For example, by clicking “Fastory line” option, the preview of the line layout will be displayed.

After that, the user can select the chosen layout by clicking in the “Select layout” button.

1. Configuration: Once completed the layout selection, users can click in the configuration button of the top menu bar. The following information will be displayed:
   1. Layout preview with available Agents.
   2. “Start new configuration” button.
   3. “Load saved configuration” button.

If there´s no saved configurations, users will click in “Start new configuration” button.

Once this is selected, a new screen will appear where the user is allowed to choose and characterize the available agents: operators (Ox), cobots (Cx) and robots (Rx). The preview of the layout will tell how many agents are needed, the type of agent and its name in the drop-down menu.

When a position is chosen, for instance C4, it designates a cobot for the workstation (WS) 4. The user will be given the option to select a specific available agent from a list using a new drop-down menu that will display.

A preview of the agent will appear when the user picks any of these choices, providing further details about it.

Click the "Confirm" button to confirm the choice once the appropriate agent has been chosen.

Regarding robots and human operators the selection workflow is similar to the cobots one.

Next, a user might select "Load saved configuration" in a parallel workflow.

A list of different previous saved configurations will appear in the screen for the selected layout.

The user can select his/her preference and apply any of them.

1. Summary: Last tab in the menu is the summary of the layout selection and configuration. If the previous step included the configuration of the agents, the screen would show:
   1. Layout selected name.
   2. Configuration information.
   3. Simulation parameters to be add by the user (i.e, duration of the shift, time of simulation).
   4. Configuration name. Optional parameter in case the user wants to save the configuration.
   5. “Save configuration” button.
   6. “Confirm selection” button.

Optionally, the user can save the preferred configurations. A configuration name is required for this. The configuration could then be saved by clicking the "Save configuration" button.

After checking the introduced data, the user will add the simulation parameters and “Confirm the selection”.

In case the user has chosen a predefined configuration by loading saved parameters, the screen will not include the “Save configuration” button. Apart form that, the workflow remains the same.

Once the selection is confirmed, the user can go back to viewport window to see the extension applied.

Functional Specifications

Functional Block Diagram

The functional specifications in the diagram below represent the flow of the main actions in the Simulation Environment. The activities start with the selection of an agent (e.g. robot, conveyor, human...) from a list of available agents. If an agent is selected, the agent should be configured. The configuration is related to the key features that define the agent previously selected. In this step, the user should introduce the properties that will determine the behavior of the agent. Following, in case the agent has been correctly configured, it will be necessary to position it in the scene. To perform this action, the user must provide the spatial coordinates in which the agent will be located. Once all the agents composing a scene are positioned, the user should decide if the definition of the layout is completed. If the layout is defined, it can be saved, with the option of storing it in the database for later use or directly running the simulation.

Main interfaces

List of main interfaces between functional components.

ID	Component	Name	Description	Sense
1	Isaac Sim FE	Extensions Panel	NVIDIA Isaac Sim's interface to open Omniverse extensions	In
2	SE FE	Layout information	Selection of the target layout in which the simulations will be executed	In
3	SE FE	Configuration information	Starting or loading a configuration. This implies: robots, cobots, human operators and their positions in the layout	In
4	SE FE	Simulation Parameters	Definition of which parameters to simulate	In
5	Simulation Engine	Skills Information	Information about the skills of the human operators	In
6	Simulation Engine	ECM Information	EC profile of the human operators	In
7	Simulation Engine	Simulation Results	Results of the simulation	Out

Sequence Maps

The following sequence diagrams shows first how the user interacts with the NVIDIA Isaac Sim front-end to open the Simulation Environment Extension and next the interactions performed in the SE Extension to execute a simulation.

Additional Information component-related