Overview
What is AI-PRISM
The AI-PRISM project is focused on enhancing smart manufacturing through AI-powered, human-centered robot interactions. It aims to create a collaborative ecosystem where robots and humans work together, particularly in manufacturing scenarios where tasks are complex and require flexibility and speed.
High-level features
The value proposition of AI-PRISM is structured around five high-level features that aim to enhance the capabilities of collaborative robotics in manufacturing environments. Our human-centered solutio while at the same time facilitating the integration of human workers.
- Introduce Collaborative Robotics in Manufacturing Environments: AI-PRISM focuses on simplifying the integration and deployment of hardware and software necessary for collaborative robotics in manufacturing settings. It ensures that setting up robotic systems is efficient, with robust interoperability mechanisms, real-time communication and data exchange capabilities to facilitate synchronized operations and enhance overall productivity.
- Digitalize Collaborative Workplaces: By employing reliable sensors and perception technologies, this feature digitizes the collaborative workplace, enabling dynamic monitoring and understanding of both the environment and human workers. It includes simulations that incorporate both robots and humans to predict, plan, and optimize tasks, ensuring a seamless and efficient collaborative environment.
- Capture Tacit Knowledge from Workers: Programming by demonstration allows robots to learn tasks by observing human workers through programming by demonstration, capturing their tacit knowledge to facilitate intuitive programming, providing algorithms to interpret and accurately translate human actions into robotic actions. It aims to ensure smooth and intuitive collaboration between humans and robots, minimizing the need for extensive training and enhancing productivity.
- Enhance Reasoning Capabilities of Collaborative Robotics: Artificial Intelligence (AI) enhances the robots’ ability to understand, reason, and make informed decisions about their environment. It also aims to advance robot-human interaction capabilities and ensure safety, fostering a more intelligent and secure collaborative environment.
- Provide Continuous Improvement through Experimentation Environments: AI-PRISM promotes the creation of open experimentation environments where continuous improvement of collaborative systems can take place. It allows for iterative testing, refinement, and enhancement of robotic processes and interactions, ensuring the system remains cutting-edge and highly effective.
Architecture
[TBC]
Installation
[TBC]
Components
F1: Collaboration Multiagent ROS based Robotic Framework
| Component Name | Description | Documentation |
|---|---|---|
| ROS Base | Project to build and push ROS Base images | Click Here |
| Cluster Management | Installation scripts for Kubernetes and Rancher | Click Here |
Data Services
| Component Name | Description | Documentation |
|---|---|---|
| IIoT Platform | Data Collection and device management | Click Here |
| Data Platform | Persistence for IIoT Data Platform | Click Here |
| Real Time Wireless Sensor Network | Supported real time communication hardware. SDN, QDSN, WSN sensor network control and management. | Click Here |
| Collaboration Ambient Data Model | Data model providing additional context to sensor data | Click Here |
Hardware Drivers
| Component Name | Description | Documentation |
|---|---|---|
| Universal Robots Driver | Universal Robots Driver | Click Here |
| Sawyer Robot Driver | Sawyer Robot Driver | Click Here |
| Dobot CR10 Driver | Dobot CR10 Robot Driver | Click Here |
| Dobot CR5 Driver | Dobot CR5 Robot Driver | Click Here |
| Ridgeback Driver | Ridgeback Robot Driver | Click Here |
| Comau Driver | Comau Robot Driver | Click Here |
| Oculus Touch Driver | Oculus Touch Driver | Click Here |
| Thorlabs Precision Table Driver | Thorlabs Driver | Click Here |
| MBJ HDF-07-RD-s Driver | MBJ HDF-07-RD-s Driver | Click Here |
| SPS electronic Safety Tester KT1885 Driver | SPS electronic Safety Tester KT1885 Driver | Click Here |
| FTE-AXIA80-DUAL SI-200-8/SI-500-20 Driver | FTE-AXIA80-DUAL SI-200-8/SI-500-20 Driver | Click Here |
| Planistar 32-32-Xled-3-VA-46w Driver | Planistar 32-32-Xled-3-VA-46w Driver | Click Here |
| Schmalz SCG-HSS 1xE100 AR 25 47 Driver | Schmalz SCG-HSS 1xE100 AR 25 47 Driver | Click Here |
| Comau Driver | Comau Robot Driver | Click Here |
F2: Collaborative workplaces digitalization
Ambient Digitalization Modules
| Component Name | Description | Documentation |
|---|---|---|
| Logitech Streamcam Driver | Logitech Driver | Click Here |
| a2A5320-23ucBAS Driver | Basler ace Driver | Click Here |
| Microphone 130F21 ICP® ARRAY Driver | 130F21 ICP® ARRAY Driver | Click Here |
| Wenglor camera Driver | Wenglor camera Driver | Click Here |
| Seeed Studio microphone Driver | Seeed Studio Driver | Click Here |
| Delta Optical Driver | Delta optical Driver | Click Here |
| Humidity and temperature sensor | Humidity and temperature sensor | Click Here |
| Intel RealSense Driver | Intel RealSense Driver | Click Here |
| ZED 2I | ZED 2I Driver | Click Here |
| SEN54-SDN Driver | SEN54-SDN Driver | Click Here |
| Openmote-B Driver | Openmote-B Driver | Click Here |
| HOKUYO UTM-30LX-EW driver | HOKUYO UTM-30LX-EW Driver | Click Here |
| Audio data publisher | Publishes audio streams to ZMQ socket (unix or TCP/IP) | Click Here |
| Real sense video publisher | Publishes video streams from real sense camera using IP protocols | Click Here |
| RealSense data publisher | Publishes video streams to ZMQ socket (unix or TCP/IP) | Click Here |
| RealSense bridge for ROS | Publishes video streams to ROS | Click Here |
| 3D reconstruction | Performs 3D reconstruction by fusing multi-view RGBD sensory data | Click Here |
| None | A software module to get a reliable retrieval of the video flow from the camera | Click Here |
| UWB driver | Driver to integrate UWB sensor data in ROS 2 | Click Here |
AI-based Perception Enhancing Modules
| Component Name | Description | Documentation |
|---|---|---|
| Hood detection and tracking | Detects and tracks hoods | Click Here |
| 2D Defect Detection | detects defects of hood | Click Here |
| Chip object detection | Detects the 2D position of microchips | Click Here |
| 6D pose estimation | Detects 6D pose of arbitrary objects | Click Here |
| 3D Object detection | Detects PCB Boards and determines best grasping position | Click Here |
| Defect detection | Detects defects in microchips | Click Here |
| AI Control | None | Click Here |
| 2D scanning module | None | Click Here |
| Force-based behaviors | Module that provides force-based behaviors for compliant execution | Click Here |
| Speech Recognition | Detect voice and recognize speech into text from real-time audio | Click Here |
| NLP Intent and Slot | Intent and slot recognition from text utterance. | Click Here |
| Person Follower | Detect person and track relative angle for robot following. | Click Here |
| Follow me AMR module - UWB | Detects a person and tracks a person | Click Here |
| None | Computer Vision model to detect persons, forklift and dangerous areas (through geo-fencing). The model is hosted on-edge. After trainging from public data the model has been optimized to be executed on edge modules | Click Here |
| Few-shot Key Point Detection | Few-shot key point detection AI training | Click Here |
| Component Name | Description | Documentation |
|---|---|---|
| Simulation Environment | None | Click Here |
Human Modeling
| Component Name | Description | Documentation |
|---|---|---|
| Human detection and pose estimation. | Human Detection and Pose Estimation | Click Here |
| Human status estimation | Human Status Estimation Module | Click Here |
| Human activity recognition | Human Activity Recognition | Click Here |
| Affective worker condition monitoring | Monitoring of psychophysiological states of human workers | Click Here |
F3: Tacit Knowledge Capture from workers
AI-based Agent level reasoning enhancing modules
| Component Name | Description | Documentation |
|---|---|---|
| Deployable Constraint based reactive Controller | Deployable constraint-based reactive controller | Click Here |
Human Machine Interaction Modules
| Component Name | Description | Documentation |
|---|---|---|
| HRC assembly constraint-based tasks | Task specifications involving HRC in assembly | Click Here |
| Speech Synthesis | Convert text to speech for voice control feedback | Click Here |
| Voice Control | Voice Control system - from real-time audio to JSON output | Click Here |
| Audio-based linguistic module | Conversational module for human-robot interaction | Click Here |
| Robot UI | Robots UI deployed at ETRI. | Click Here |
Programming by Demonstration
| Component Name | Description | Documentation |
|---|---|---|
| Skills for complex tasks | Skills for complex tasks that require force-based robot behaviors (e.g. insertion) or learning from demonstrations | Click Here |
| PbD Module | PbD module | Click Here |
| Programming by Demonstration toolkit | Toolkit to analyze demonstrations data and generate a learned motion model | Click Here |
| Painting PbD | Painting and sanding programming by demonstration | Click Here |
| Multimodal interface Pbd | Multimodal Interface Prototype for programming robotic tasks (PROF) | Click Here |
Agent level reasoning
| Component Name | Description | Documentation |
|---|---|---|
| BetFSM | A behavior tree and finite state machine framework in Python using cooperative multitasking where new state machines or behavior-tree nodes can be easily specified | Click Here |
F4: Enhanced Manufacturing Operations
Ambient Level Reasoning
| Component Name | Description | Documentation |
|---|---|---|
| Unified Planning tool | Unified planning tool | Click Here |
| Follow me AMR path planning | AMR path planning module using UWB data | Click Here |
| HRC painting re/scheduler | HRC Scheduler and Rescheduler for furniture painting | Click Here |
| Collaborative order forming | Collaborative order forming (follow-me) | Click Here |
| Smart pick and place for recycling | Smart pick and place for recycling | Click Here |
| Smart depalletizing | Smart depalletising | Click Here |
| AI Assistant | AI-assistant for easy operation | Click Here |
F5: Experimentation
| Component Name | Description | Documentation |
|---|---|---|
| Network Suite | Network Suite | Click Here |