Human-robot manipulation of soft materials, such as fabrics, composites, and sheets of paper/cardboard, is a challenging operation that presents several relevant industrial applications. Estimating the deformation state of the manipulated material is one of the main challenges. Viable methods provide the indirect measure by calculating the human-robot relative distance. In this paper, we develop a data-driven model to estimate the deformation state of the material from a depth image through a Convolutional Neural Network (CNN). First, we define the deformation state of the material as the relative roto-translation from the current robot pose and a human grasping position. The model estimates the current deformation state through a Convolutional Neural Network, specifically, DenseNet-121 pretrained on ImageNet. The delta between the current and the desired deformation state is fed to the robot controller that outputs twist commands. The paper describes the developed approach to acquire, preprocess the dataset and train the model. The model is compared with the current state-of-the-art method based on a camera skeletal tracker. Results show that the approach achieves better performances and avoids the drawbacks of a skeletal tracker. The model was also validated over three different materials showing its generalization ability. Finally, we also studied the model performance according to different architectures and dataset dimensions to minimize the time required for dataset acquisition.

In Human-Machine interaction, the possibility of increasing the intelligence and adaptability of the controlled plant by imitating human control behavior has been an objective of many research efforts in the last decades. From classical control-theory human control models to modern machine learning, neural networks, and reinforcement learning paradigms, the common denominator is the effort to model complex nonlinear dynamics typical of human activity. However, these approaches are very different, and finding a guiding line is challenging. This review investigates state-of-the-art techniques from the perspective of human control modeling, considering the different physiological districts involved as the starting point. The focus is mainly directed toward nonlinear dynamical system modeling, which constitutes the main challenge in this field. In the end, transport systems are presented as a technological scenario in which the discussed techniques are mainly applied.

The ease of use of robot programming interfaces represents a barrier to robot adoption in several manufacturing sectors because of the need for more expertise from the end-users. Current robot programming methods are mostly the past heritage, with robot programmers reluctant to adopt new programming paradigms. This work aims to evaluate the impact on non-expert users of introducing a new task-oriented programming interface that hides the complexity of a programming framework based on ROS. The paper compares the programming performance of such an interface with a classic robot-oriented programming method based on a state-of-the-art robot teach pendant. An experimental campaign involved 22 non-expert users working on the programming of two industrial tasks. Task-oriented and robot-oriented programming showed comparable learning time, programming time and the number of questions raised during the programming phases, highlighting the possibility of a smooth introduction to task-oriented programming even to non-expert users.

With the growing interest in applications involving humans and robots teaming together, the need to understand each other's intentions and behavior arises. This work presents a method to online identify the time-varying human feedback control law during physical Human-Robot Interaction. The robot motion is implemented as a Cartesian impedance, and the interaction with the human happens by force exchange. The coupled system is modeled with a state-space formulation. The state vector is augmented with the unknown parameters, and an Extended Kalman Filter (EKF) is implemented for online identification. This approach is compared with the Least Squares (LS) and the Recursive Least Squares (RLS) methods. Both simulation and experimental results are provided, showing the presented approach's feasibility in identifying the parameters and reconstructing the control inputs.

Combining symbolic and geometric reasoning in multiagent systems is a challenging task that involves planning, scheduling, and synchronization problems. Existing works overlooked the variability of task duration and geometric feasibility intrinsic to these systems because of the interaction between agents and the environment. We propose a combined task and motion planning approach to optimize the sequencing, assignment, and execution of tasks under temporal and spatial variability. The framework relies on decoupling tasks and actions, where an action is one possible geometric realization of a symbolic task. At the task level, timeline-based planning deals with temporal constraints, duration variability, and synergic assignment of tasks. At the action level, online motion planning plans for the actual movements dealing with environmental changes. We demonstrate the approach's effectiveness in a collaborative manufacturing scenario, in which a robotic arm and a human worker shall assemble a mosaic in the shortest time possible. Compared with existing works, our approach applies to a broader range of applications and reduces the execution time of the process.

In many real-world applications (e.g., human-robot collaboration), the environment changes rapidly, and the intended path may become invalid due to moving obstacles. In these situations, the robot should quickly find a new path to reach the goal, possibly without stopping. Planning from scratch or repairing the current graph can be too expensive and time-consuming. This paper proposes MARS, a sampling-based Multi-pAth Replanning Strategy that enables a robot to move in dynamic environments with unpredictable obstacles. The novelty of the method is the exploitation of a set of precomputed paths to compute a new solution in a few hundred milliseconds when an obstacle obstructs the robot's path. The algorithm enhances the search speed by using informed sampling, builds a directed graph to reuse results from previous replanning iterations, and improves the current solution in an anytime fashion to make the robot responsive to environmental changes. In addition, the paper presents a multithread architecture, applicable to several replanning algorithms, to handle the execution of the robot's trajectory with continuous replanning and the collision checking of the traversed path. The paper compares state-of-the-art sampling-based path-replanning algorithms in complex and high-dimensional scenarios, showing that MARS is superior in terms of success rate and quality of solutions found. An open-source ROS-compatible implementation of the algorithm is also provided.

Dataset used for the paper submitted to RO-MAN 2022 Human-robot co-manipulation of soft materials: enable a robot manual guidance using a depth map feedback

Model-based approaches aiming to characterize human behavior when interacting with a controlled machine have been a matter of research investigation in various domains, from aerospace to semi-autonomous driving and robotics. Human-robot collaboration is one of the most exciting scenarios of application in which a continuous physical interaction between humans and the controlled plant is present. In this context, the human subject can adapt its control behavior to the external sensed dynamics. This capability has a significant observable effect on the control delay, making its characterization and prevision a crucial aspect to understand. This work investigates a linear modeling approach that uniquely describes human and robot control actions and applies to a collaborative robotic task.

The ease of use of robot programming interfaces represents a barrier to robot adoption in several manufacturing sectors because of the lack of expertise of the end-users. Current robot programming methods are mostly the past heritage, with robot programmers reluctant to adopt new programming paradigms. This work aims to evaluate the impact on non-expert users of introducing a new task-oriented programming interface that hides the complexity of a programming framework based on ROS. The paper compares the programming performance of such an interface with a classic robot-oriented programming method based on a state-of-the-art robot teach pendant. An experimental campaign involved 22 non-expert users working on the programming of two industrial tasks demonstrating a high acceptance level of the task-oriented interface with not signicant di_erence in the learning time compared to a standard interface.

Human-robot co-manipulation of large but lightweight elements made by soft materials is a challenging operation that presents several relevant industrial applications. This paper proposes using a 3D camera to track the deformation of soft materials for human-robot co-manipulation. Thanks to a Convolutional Neural Network (CNN), the acquired depth image is processed to estimate the element deformation. The output of the CNN is the feedback for the robot controller to track a given set-point of deformation.

Human-robot collaboration is migrating from lightweight robots in laboratory environments to industrial applications, where heavy tasks and powerful robots are more common. In this scenario, a reliable perception of the humans involved in the process and related intentions and behaviors is fundamental. This paper presents two projects investigating the use of robots in relevant industrial scenarios, providing an overview of how industrial human-robot collaborative tasks can be successfully addressed.

Human-robot co-manipulation of large but lightweight elements made by soft materials, such as fabrics, composites, sheets of paper/cardboard, is a challenging operation that presents several relevant industrial applications. As the primary limit, the force applied on the material must be unidirectional (i.e., the user can only pull the element). Its magnitude needs to be limited to avoid damages to the material itself. This paper proposes using a 3D camera to track the deformation of soft materials for human-robot co-manipulation. Thanks to a Convolutional Neural Network (CNN), the acquired depth image is processed to estimate the element deformation. The output of the CNN is the feedback for the robot controller to track a given set-point of deformation. The set-point tracking will avoid excessive material deformation, enabling a vision-based robot manual guidance.

Human awareness in robot motion planning is crucial for seamless interaction with humans. Many existing techniques slow down, stop, or change the robot's trajectory locally to avoid collisions with humans. Although using the information on the human's state in the path planning phase could reduce future interference with the human's movements and make safety stops less frequent, such an approach is less widespread. This paper proposes a novel approach to embedding a human model in the robot's path planner. The method explicitly addresses the problem of minimizing the path execution time, including slowdowns and stops owed to the proximity of humans. For this purpose, it converts safety speed limits into configuration-space cost functions that drive the path's optimization. The costmap can be updated based on the observed or predicted state of the human. The method can handle deterministic and probabilistic representations of the human state and is independent of the prediction algorithm. Numerical and experimental results on an industrial collaborative cell demonstrate that the proposed approach consistently reduces the robot's execution time and avoids unnecessary safety speed reductions.

Nowadays, many applications involving humans and robots working together require physical interaction. It is known that, during an interaction, the mutual understanding and knowledge of the partner's goal improves and allows natural interaction. For this purpose, this work proposes Inverse Optimal Control (IOC) to recover the cost function of a human performing a reaching task with a robot in passive impedance control. This work presents the potentialities and limitations of the presented IOC method to describe human objectives. This work represents a preparatory study toward smooth and natural physical Human-Robot Interaction (pHRI), intending to understand the basic information on humans' behavior.

Mobile manipulators are often comprised of an extensive kinematic chain resulting from an industrial robot mounted on top of an autonomous mobile robot. In such a way, the system not only retains the parameters embedded in the two sub-systems, hence DH parameters for the industrial robot and odometry parameters for the mobile robot, but also includes the relative transformation between the two parts and an additional transformation for a camera mounted on the kinematic chain.In this complex setup, it is relatively simple to introduce kinematic inaccuracies, or in some cases, to operate the system in such a way that the kinematic parameters vary(e.g., rubber wheels on high payload).Estimating the values of such parameters might be too demanding for the on-board computing system.In this work, we propose a cloud-based visually aided parameter estimation method, which constantly receives data from the mobile manipulator and generates better estimates of the kinematic parameters through an UKF dual estimation.The overall system architecture is presented to the reader, together with the reasons for relying to a cloud based paradigm, for then giving a theoretical analysis and real world experiments and results.

The problem addressed in this work is the arbitration of the role between a robot and a human during physical Human-Robot Interaction, sharing a common task. The system is modeled as a Cartesian impedance, with two separate external forces provided by the human and the robot. The problem is then reformulated as a Cooperative Differential Game, which possibly has multiple solutions on the Pareto frontier. Finally, the bargaining problem is addressed by proposing a solution depending on the interaction force, interpreted as the human will to lead or follow. This defines the arbitration law and assigns the role of leader or follower to the robot. Experiments show the feasibility and capabilities of the proposed control in managing the human-robot arbitration during a shared- trajectory following task.

A good estimation of the actions' cost is key in task planning for human-robot collaboration. The duration of an action depends on agents' capabilities and the correlation between actions performed simultaneously by the human and the robot. This paper proposes an approach to learning actions' costs and coupling between actions executed concurrently by humans and robots. We leverage the information from past executions to learn the average duration of each action and a synergy coefficient representing the effect of an action performed by the human on the duration of the action performed by the robot (and vice versa). We implement the proposed method in a simulated scenario where both agents can access the same area simultaneously. Safety measures require the robot to slow down when the human is close, denoting a bad synergy of tasks operating in the same area. We show that our approach can learn such bad couplings so that a task planner can leverage this information to find better plans.

Industry 4.0 is pushing forward the need for symbiotic interactions between physical and virtual entities of production environments to realize increasingly flexible and customizable production processes. This holds especially for human–robot collaboration in manufacturing, which needs continuous interaction between humans and robots. The coexistence of human and autonomous robotic agents raises several methodological and technological challenges for the design of effective, safe, and reliable control paradigms. This work proposes the integration of novel technologies from Artificial Intelligence, Control and Augmented Reality to enhance the flexibility and adaptability of collaborative systems. We present the basis to advance the classical human-aware control paradigm in favor of a user-aware control paradigm and thus personalize and adapt the synthesis and execution of collaborative processes following a user-centric approach. We leverage a manufacturing case study to show a possible deployment of the proposed framework in a real-world industrial scenario.

The increasing requests for flexible robotic applications involving the rapid relocation of the robot manipulator, possibly mounted on a mobile base, imposes tolerance to imprecise positioning. The high manipulability of the nominally designed poses, i.e., the capacity to change the position and the orientation of a given robot joint configuration's end-effector, is often considered a proxy for robustness to imprecise positioning. This work presents a method for choosing target end-effector poses to manipulate bulky objects in complex environments. The paper proposes a two-layer optimizer connected in cascade to maximize the manipulability and achieve reasonable computational time. First, using a genetic algorithm (GA) allows a global search for a satisfactory solution to the target poses of the task at the same time. Subsequently, the output of the GA becomes the initial guess for the simulated annealing (SA) algorithm, which locally maximizes each pose's manipulability separately. The feasibility of the connecting trajectories and collisions are checked in both layers. Experiments show the method's ability to find excellent solutions within a limited time, considering a complex problem involving manipulating large objects in a cluttered environment. The simulations of three working scenarios allowed testing of the proposed method. The final validation of the algorithm was on two relevant industrial use-cases: the manipulation of sidewalls and the manipulation of cargo panels inside an aircraft fuselage.