Humans learn to be dexterous by interacting with a wide variety of objects in different contexts. Given the description of an object's physical attributes, humans can determine a proper strategy and grasp an object. This paper proposes an approach to determine grasping strategy for a 10 degree-of-freedom anthropomorphic robotic hand simply based on natural-language descriptions of an object. A probabilistic learning-based approach is proposed to help a robotic hand learn suitable grasp poses starting from the natural language description of the object. The solution involves a three-step learning model. In the first step, the information parsed from an object's natural-language descriptions are used to identify/recognize the object by making use of a novel nearestneighbor distance metric. In the second step, the probability distribution of grasp types for the given object is learned using a deep neural net which takes in object features as input. The labels for this grasp learning model is supplied from human grasping trials. The discrete, two-dimensional grasp type/size vector is mapped back to the ten-dimensional robot joint-angles configuration space using linear inverse-kinematics models. The grasping strategy generated by the proposed approach is evaluated both by simulation study and execution of the grasps on an AR10 robotic hand. Index Terms--robotic grasping, human grasp primitives, natural language processing, object features extraction, neural networks classification.

The ability to learning from interaction with environments shapes an intelligent agent. For exploratory robots, they need specific structured action to interact with the physical world efficiently. Geometry-based grasping, which serves as the primary action for many complex manipulation tasks, can be of great help for robot exploration. With a learned grasping strategy, the robot can directly execute object-specific action. This thesis studies the problem of 6-DoF geometric grasping by a parallel gripper captured using a commodity depth sensor from a single viewpoint. We address the problem in a learning-based framework with point cloud input. At the higher level, we rely on a single-shot grasp proposal network built upon the PointNet++ backbone. Our single-shot neural network architecture can predict grasp proposals efficiently and effectively. At the lower level, we proposed a method to generate training data automatically. Our training data synthesis pipeline can generate scenes of complex object configuration and leverage an innovative gripper contact model to create dense and high-quality grasp annotations. Experiments in synthetic and real environments have demonstrated that the proposed approach can outperform the state-of-the-art geometry-based grasping method by a large margin. The grasp proposal network trained in a synthetic scene can work well in real-world scenarios, which also shows the point-based method have high potential to bridge the sim-to-real gap. We hope the work of the geometric grasping algorithm will help future research for more complex robot manipulation skills.

Grasping can be a complicated process for robotics due to the replication of human fine motor skills and typically high degrees of freedom in robotic hands. Robotic hands that are underactuated provide a method by which grasps can be executed without the onerous task of calculating every fingertip placement. The general shape configuration modes available to underactuated hands lend themselves well to an approach of grasping by shape primitives, and especially so when applied to gloveboxes in the nuclear domain due to the finite number of objects anticipated and the safe assumption that objects in the set are rigid. Thus, the object set found in a glovebox can be categorized as a small set of primitives such as cylinders, cubes, and bowls/hemispheres, etc. These same assumptions can also be leveraged for reliable identification and pose estimation within a glovebox. This effort develops and simulates a simple, but robust and effective grasp planning algorithm for a 7DOF industrial robot and three fingered dexterous, but underactuated robotic hand. The proposed grasping algorithm creates a grasp by generating a vector to the object from the base of the robot and manipulating that vector to be in a suitable starting location for a grasp. The grasp preshapes are selected to match shape primitives and are built-in to the Robotiq gripper used for algorithm demonstration purposes. If a grasp is found to be unsuitable via an inverse kinematics solution check, the algorithm procedurally generates additional grasps to try based on object geometry until a solution can be found or all possibilities are exhausted. The algorithm was tested and found capable of generating valid grasps for visually identified objects, and can recalculate grasps if one is found to be incompatible with the current kinematics of the robotic arm.

While humans can grasp and manipulate novel objects with ease, rapid and reliable robot grasping of a wide variety of objects is highly challenging due to sensor noise, partial observability, imprecise control, and hardware limitations. Analytic approaches to robot grasping use models from physics to predict grasp success but require precise knowledge of the robot and objects in the environment, making them well-suited for controlled industrial applications but difficult to scale to many objects. On the other hand, deep neural networks trained on large datasets of grasps labeled with empirical successes and failures can rapidly plan grasps across a diverse set of objects, but data collection is tedious, robot-specific, and prone to mislabeling. To improve the efficiency of learning deep grasping policies, we propose a hybrid method to automate dataset collection by generating millions of synthetic 3D point clouds, robot grasps, and success metrics using analytic models of contact, collision geometry, and image formation. We present the Dexterity-Network (Dex-Net), a framework for generating training datasets by analyzing mechanical models of contact forces and torques under stochastic perturbations across thousands of 3D object CAD models. We describe dataset generation models for training policies to lift and transport novel objects from a tabletop or cluttered bin using a 3D depth sensor and a parallel-jaw (two-finger) or suction cup gripper. To study the effects of learning from massive amounts of training data, we generate datasets containing millions of training examples using distributed Cloud computing, simulations, and parallel GPU processing. We use these datasets to train robust grasping policies based on Grasp Quality Convolutional Neural Networks (GQ-CNNs) that take as input a depth image and a candidate grasp with up to five degrees of freedom and predict the probability of grasp success on an object in the image. To transfer from simulation to reality, we develop novel analytic grasp success metrics based on resisting disturbing forces and torques under stochastic perturbations and bounding an object's mobility under an energy field such as gravity. In addition, we study techniques in algorithmic supervision to guide dataset collection using full knowledge of the object geometry and pose in simulation. We explore extensions to learning policies that sequentially pick novel objects from dense clutter in a bin and that can rapidly decide which gripper hardware is best in a particular scenario. To substantiate the method, we describe thousands of experimental trials on a physical robot which suggest that deep learning on synthetic Dex-Net datasets can be used to rapidly and reliably plan grasps across a diverse set of novel objects for a variety of depth sensors, robot grippers, and robot arms. Results suggest that policies trained on Dex-Net datasets can achieve up to 95% success in picking novel objects from densely cluttered bins at a rate of over 310 mean picks per hour with no additional training or tuning on the physical system.

The world of artificial systems is reaching complexity levels that es cape human understanding. Surface traffic, electricity distribution, air planes, mobile communications, etc. , are examples that demonstrate that we are running into problems that are beyond classical scientific or engi neering knowledge. There is an ongoing world-wide effort to understand these systems and develop models that can capture its behavior. The reason for this work is clear, if our lack of understanding deepens, we will lose our capability to control these systems and make they behave as we want. Researchers from many different fields are trying to understand and develop theories for complex man-made systems. This book presents re search from the perspective of control and systems theory. The book has grown out of activities in the research program Control of Complex Systems (COSY). The program has been sponsored by the Eu ropean Science Foundation (ESF) which for 25 years has been one of the leading players in stimulating scientific research. ESF is a European asso ciation of more than 60 leading national science agencies spanning more than 20 countries. ESF covers has standing committees in Medical Sci ences, Life and Environmental Sciences, Physical and Engineering Sci ences, Humanities and Social Sciences. The COSY program was ESF's first activity in the Engineering Sciences. The program run for a period of five years starting January 1995.