Intelligent Autonomy (IA) is a multi-year program within the Office of Naval Research (ONR) Autonomous Operations (AO) Future Naval Capabilities (FNC) program. The primary goal of the effort is to develop and demonstrate technologies for highly automated and fully autonomous mission planning and dynamic re-tasking of multiple classes of Naval unmanned systems and minimization of human intervention in unmanned vehicle operations. This technology is being applied to both individual and teams of unmanned air, surface, ground, and undersea vehicles for a variety of mission areas including reconnaissance/search, persistent surveillance, tracking, and some limited application to strike. Autonomy technologies will be matured through a series of phased demonstrations to allow low risk transition to current and future Navy and Marine Corps systems. Demonstrations will be done using both real vehicles and simulation. Some of the major simulation demonstrations will be done within the context of a simulated warfare environment at the Naval Air Systems Command based around the Air Combat Environment Test and Evaluation Facility (ACETEF) and the Unmanned System Research and Development Lab (USRDL). The demonstrations at NAVAIR will utilize much of the architecture and many of the assets from the NCW4.0X Virtual Laboratory (V-LAB) project. Metrics for testing of IA software in this environment are currently being developed. This paper will discuss some candidate performance metrics that are currently being considered for evaluation of the Intelligent Autonomy technologies.

To design and develop capable, dependable, and affordable intelligent systems, their performance must be measurable. Scienti?c methodologies for standardization and benchmarking are crucial for quantitatively evaluating the performance of eme- ing robotic and intelligent systems’ technologies. There is currently no accepted standard for quantitatively measuring the performance of these systems against user-de?ned requirements; and furthermore, there is no consensus on what obj- tive evaluation procedures need to be followed to understand the performance of these systems. The lack of reproducible and repeatable test methods has precluded researchers working towards a common goal from exchanging and communic- ing results, inter-comparing system performance, and leveraging previous work that could otherwise avoid duplication and expedite technology transfer. Currently, this lack of cohesion in the community hinders progress in many domains, such as m- ufacturing, service, healthcare, and security. By providing the research community with access to standardized tools, reference data sets, and open source libraries of solutions, researchers and consumers will be able to evaluate the cost and be- ?ts associated with intelligent systems and associated technologies. In this vein, the edited book volume addresses performance evaluation and metrics for intel- gent systems, in general, while emphasizing the need and solutions for standardized methods. To the knowledge of the editors, there is not a single book on the market that is solely dedicated to the subject of performance evaluation and benchmarking of intelligent systems.

This paper describes an autonomous Intelligent Controller (IC) architecture directly applicable to the design of unmanned autonomous vehicles and performance measures associated with intelligent autonomy. The vehicles may operate independently or cooperate to carry out complex missions involving disparate sensors or payload packages. An approach to measure the performance achieved with collaborative control is presented and simulation scenarios are provided to demonstrate how the metrics are applied.

The recently released DoD Unmanned Aerial Vehicles Roadmap discusses advancements in UAV autonomy in terms of autonomous control levels (ACL). The ACL concept was pioneered by researchers in the Air Force Research Laboratory's Air Vehicles Directorate who are charged with developing autonomous air vehicles. In the process of developing intelligent autonomous agents for UAV control systems we were constantly challenged to "tell us how autonomous a UAV is, and how do you think it can be measured?" Usually we hand-waved away the argument and hoped the questioner will go away since this is a very subjective, and complicated, subject, but within the last year we've been directed to develop national intelligent autonomous UAV control metrics - an IQ test for the flyborgs, if you will. The ACL chart is the result. We've done this via intense discussions with other government labs and industry, and this paper covers the agreed metrics (an extension of the OODA - observe, orient, decide, and act - loop) as well as the precursors, "dead-ends", and out-and-out flops investigated to get there.

This book describes the latest research accomplishments, innovations, and visions in the field of robotics as presented at the 13th International Conference on Intelligent Autonomous Systems (IAS), held in Padua in July 2014, by leading researchers, engineers, and practitioners from across the world. The contents amply confirm that robots, machines, and systems are rapidly achieving intelligence and autonomy, mastering more and more capabilities such as mobility and manipulation, sensing and perception, reasoning, and decision making. A wide range of research results and applications are covered, and particular attention is paid to the emerging role of autonomous robots and intelligent systems in industrial production, which reflects their maturity and robustness. The contributions have been selected through a rigorous peer-review process and contain many exciting and visionary ideas that will further galvanize the research community, spurring novel research directions. The series of biennial IAS conferences commenced in 1986 and represents a premiere event in robotics.

This volume explores the intersection of robust intelligence (RI) and trust in autonomous systems across multiple contexts among autonomous hybrid systems, where hybrids are arbitrary combinations of humans, machines and robots. To better understand the relationships between artificial intelligence (AI) and RI in a way that promotes trust between autonomous systems and human users, this book explores the underlying theory, mathematics, computational models, and field applications. It uniquely unifies the fields of RI and trust and frames it in a broader context, namely the effective integration of human-autonomous systems. A description of the current state of the art in RI and trust introduces the research work in this area. With this foundation, the chapters further elaborate on key research areas and gaps that are at the heart of effective human-systems integration, including workload management, human computer interfaces, team integration and performance, advanced analytics, behavior modeling, training, and, lastly, test and evaluation. Written by international leading researchers from across the field of autonomous systems research, Robust Intelligence and Trust in Autonomous Systems dedicates itself to thoroughly examining the challenges and trends of systems that exhibit RI, the fundamental implications of RI in developing trusted relationships with present and future autonomous systems, and the effective human systems integration that must result for trust to be sustained. Contributing authors: David W. Aha, Jenny Burke, Joseph Coyne, M.L. Cummings, Munjal Desai, Michael Drinkwater, Jill L. Drury, Michael W. Floyd, Fei Gao, Vladimir Gontar, Ayanna M. Howard, Mo Jamshidi, W.F. Lawless, Kapil Madathil, Ranjeev Mittu, Arezou Moussavi, Gari Palmer, Paul Robinette, Behzad Sadrfaridpour, Hamed Saeidi, Kristin E. Schaefer, Anne Selwyn, Ciara Sibley, Donald A. Sofge, Erin Solovey, Aaron Steinfeld, Barney Tannahill, Gavin Taylor, Alan R. Wagner, Yue Wang, Holly A. Yanco, Dan Zwillinger.

“IAS has been held every two years since 1986 providing venue for the latest accomplishments and innovations in advanced intelligent autonomous systems. New technologies and application domains continuously pose new challenges to be overcome in order to apply intelligent autonomous systems in a reliable and user-independent way in areas ranging from industrial applications to professional service and household domains. The present book contains the papers presented at the 17th International Conference on Intelligent Autonomous Systems (IAS-17), which was held from June 13–16, 2022, in Zagreb, Croatia. In our view, 62 papers, authored by 196 authors from 19 countries, are a testimony to the appeal of the conference considering travel restrictions imposed by the COVID-19 pandemic. Our special thanks go to the authors and the reviewers for their effort—the results of their joint work are visible in this book. We look forward to seeing you at IAS-18 in 2023 in Suwon, South Korea!”