Are psychometric tests valid for a new reality of artificial intelligence systems, technology-enhanced humans, and hybrids yet to come? Are the Turing Test, the ubiquitous CAPTCHAs, and the various animal cognition tests the best alternatives? In this fascinating and provocative book, José Hernández-Orallo formulates major scientific questions, integrates the most significant research developments, and offers a vision of the universal evaluation of cognition. By replacing the dominant anthropocentric stance with a universal perspective where living organisms are considered as a special case, long-standing questions in the evaluation of behavior can be addressed in a wider landscape. Can we derive task difficulty intrinsically? Is a universal g factor - a common general component for all abilities - theoretically possible? Using algorithmic information theory as a foundation, the book elaborates on the evaluation of perceptual, developmental, social, verbal and collective features and critically analyzes what the future of intelligence might look like.

The general theme of this book is to present the applications of artificial intelligence (AI) in test development. In particular, this book includes research and successful examples of using AI technology in automated item generation, automated test assembly, automated scoring, and computerized adaptive testing. By utilizing artificial intelligence, the efficiency of item development, test form construction, test delivery, and scoring could be dramatically increased. Chapters on automated item generation offer different perspectives related to generating a large number of items with controlled psychometric properties including the latest development of using machine learning methods. Automated scoring is illustrated for different types of assessments such as speaking and writing from both methodological aspects and practical considerations. Further, automated test assembly is elaborated for the conventional linear tests from both classical test theory and item response theory perspectives. Item pool design and assembly for the linear-on-the-fly tests elaborates more complications in practice when test security is a big concern. Finally, several chapters focus on computerized adaptive testing (CAT) at either item or module levels. CAT is further illustrated as an effective approach to increasing test-takers’ engagement in testing. In summary, the book includes both theoretical, methodological, and applied research and practices that serve as the foundation for future development. These chapters provide illustrations of efforts to automate the process of test development. While some of these automation processes have become common practices such as automated test assembly, automated scoring, and computerized adaptive testing, some others such as automated item generation calls for more research and exploration. When new AI methods are emerging and evolving, it is expected that researchers can expand and improve the methods for automating different steps in test development to enhance the automation features and practitioners can adopt quality automation procedures to improve assessment practices.

Artificial Intelligence and Evaluation: Emerging Technologies and Their Implications for Evaluation is a groundbreaking exploration of how the landscape of program evaluation will be redefined by artificial intelligence and other emerging digital technologies. In an era where digital technologies and artificial intelligence (AI) are rapidly evolving, this book presents a pivotal resource for evaluators navigating the transformative intersection of their practice and cutting-edge technology. Addressing the dual dimensions of how evaluations are conducted and what is evaluated, a roster of distinguished contributors illuminate the impact of AI on program evaluation methodologies. Offering a discerning overview of various digital technologies, their promises and perils, they carefully dissect the implications for evaluative processes and debate how evaluators must be equipped with the requisite skills to harness the full potential of AI tools. Further, the book includes a number of compelling use cases, demonstrating the tangible applications of AI in diverse evaluation scenarios. The use cases range from the application of GIS data to advanced text analytics. As such, this book provides evaluators with inspirational cases on how to apply AI in their practice as well as what pitfalls one must look out for. Artificial Intelligence and Evaluation is an indispensable guide for evaluators seeking to not only adapt to but thrive in the dynamic landscape of evaluation practices reshaped by the advent of artificial intelligence.

The authors develop methods for assessing the impact of deploying artificial intelligence (AI) systems to support intelligence missions.

The Complete Beginner’s Guide to Understanding and Building Machine Learning Systems with Python Machine Learning with Python for Everyone will help you master the processes, patterns, and strategies you need to build effective learning systems, even if you’re an absolute beginner. If you can write some Python code, this book is for you, no matter how little college-level math you know. Principal instructor Mark E. Fenner relies on plain-English stories, pictures, and Python examples to communicate the ideas of machine learning. Mark begins by discussing machine learning and what it can do; introducing key mathematical and computational topics in an approachable manner; and walking you through the first steps in building, training, and evaluating learning systems. Step by step, you’ll fill out the components of a practical learning system, broaden your toolbox, and explore some of the field’s most sophisticated and exciting techniques. Whether you’re a student, analyst, scientist, or hobbyist, this guide’s insights will be applicable to every learning system you ever build or use. Understand machine learning algorithms, models, and core machine learning concepts Classify examples with classifiers, and quantify examples with regressors Realistically assess performance of machine learning systems Use feature engineering to smooth rough data into useful forms Chain multiple components into one system and tune its performance Apply machine learning techniques to images and text Connect the core concepts to neural networks and graphical models Leverage the Python scikit-learn library and other powerful tools Register your book for convenient access to downloads, updates, and/or corrections as they become available. See inside book for details.

Machine learning algorithms and artificial intelligence influence many aspects of life today. This report identifies some of their shortcomings and associated policy risks and examines some approaches for combating these problems.