Understanding and emulating human-like intelligence has been a long-standing goal of researchers in various domains leading to the emergence of an inter-disciplinary area called Brain-inspired or Neuromorphic Computing. This research area aims to achieve brain- like intelligence and energy efficiency by understanding and emulating its functionality. In the contemporary world of big data-driven analytics that has fueled ever-increasing demands for computing power, combined with the end of Moore's law scaling, the sheer energy cost of providing exascale-compute capability could soon make it economically and ecologically unsustainable. It, therefore, becomes imperative to explore alternate and more energy-efficient computing paradigms and the human brain, with its 20 W operating power budget, provides the ideal inspiration for building these future computing systems. Spiking Neural Networks (SNNs) are a class of biologically-inspired algorithms designed to mimic natural neural networks found in the brain. Besides playing an important role in biological simulations for neuroscience-related studies, SNNs are recently gaining traction as low- power counterparts of high-precision DNNs. However, in order to build systems with brain-like energy efficiency, we need to capture the functionality of billions of neurons and their communication mechanism in hardware, and this requires innovations at the device/circuit, architecture, algorithm and application levels of the computing stack. Further, efficiently utilizing and incorporating the SNN-led temporal computing paradigm in day-to-day tasks on time-dependent data also requires considerable algorithmic and architectural innovations. With these over-arching princi- ples, this dissertation is aimed at addressing the following architectural and algorithmic issues in SNN inference and training: (i) Investigating the design space of scalable, low- power SNNs by taking a holistic approach spanning the device/circuit levels for designing extremely low power spiking neurons and synapses, architectural solutions for efficient scal- ing of these networks, as well as algorithm-level optimizations for improving the accuracy of SNN models. Further, the SNN characteristics are compared against those of deep/analog neural networks (DNN/ANN), the de-facto drivers of modern AI. Based on this study, a low-power SNN, ANN and hybrid SNN-ANN inference architecture is designed using spintronics-based Magnetic Tunnel Junction (MTJ) devices, while also accounting for the deep interactions between the algorithm and the device. (ii) Training an SNN to solve a problem in a user-level application has so far proved to be challenging due to its discrete and temporal nature. SNNs are, therefore, often converted from high-precision ANNs that can be easily trained using gradient descent-based backpropagation. In this chapter, we study the effectiveness of existing ANN-SNN conversion techniques on sparse event-based data emitted by a neuromorphic camera -- several low-power, hardware-friendly techniques are proposed to boost conversion accuracy and their efficacy is evaluated on a gesture recognition task. (iii) Next, we address the computational challenges involved in train- ing a deep SNN using gradient-descent backpropagation, which is the most effective and scalable technique for training DNNs and SNNs from scratch. By reducing the memory footprint and computational overhead of backpropagation through time-based SNN train- ing, we enable the training and exploration of deeper SNNs on resource-limited hardware platforms including edge devices. Techniques such as re-computation, approximation and a combination thereof, are explored in the context of SNN training. In a nutshell, this dissertation identifies the major compute and memory bottlenecks afflicting SNNs today and proposes efficient algorithm-architecture co-design techniques to alleviate them, with the ultimate goal of facilitating the adaption of energy-efficient Neuromorphic Computing in the mainstream computing paradigm.

Explains current co-design and co-optimization methodologies for building hardware neural networks and algorithms for machine learning applications This book focuses on how to build energy-efficient hardware for neural networks with learning capabilities—and provides co-design and co-optimization methodologies for building hardware neural networks that can learn. Presenting a complete picture from high-level algorithm to low-level implementation details, Learning in Energy-Efficient Neuromorphic Computing: Algorithm and Architecture Co-Design also covers many fundamentals and essentials in neural networks (e.g., deep learning), as well as hardware implementation of neural networks. The book begins with an overview of neural networks. It then discusses algorithms for utilizing and training rate-based artificial neural networks. Next comes an introduction to various options for executing neural networks, ranging from general-purpose processors to specialized hardware, from digital accelerator to analog accelerator. A design example on building energy-efficient accelerator for adaptive dynamic programming with neural networks is also presented. An examination of fundamental concepts and popular learning algorithms for spiking neural networks follows that, along with a look at the hardware for spiking neural networks. Then comes a chapter offering readers three design examples (two of which are based on conventional CMOS, and one on emerging nanotechnology) to implement the learning algorithm found in the previous chapter. The book concludes with an outlook on the future of neural network hardware. Includes cross-layer survey of hardware accelerators for neuromorphic algorithms Covers the co-design of architecture and algorithms with emerging devices for much-improved computing efficiency Focuses on the co-design of algorithms and hardware, which is especially critical for using emerging devices, such as traditional memristors or diffusive memristors, for neuromorphic computing Learning in Energy-Efficient Neuromorphic Computing: Algorithm and Architecture Co-Design is an ideal resource for researchers, scientists, software engineers, and hardware engineers dealing with the ever-increasing requirement on power consumption and response time. It is also excellent for teaching and training undergraduate and graduate students about the latest generation neural networks with powerful learning capabilities.

Explains current co-design and co-optimization methodologies for building hardware neural networks and algorithms for machine learning applications This book focuses on how to build energy-efficient hardware for neural networks with learning capabilities—and provides co-design and co-optimization methodologies for building hardware neural networks that can learn. Presenting a complete picture from high-level algorithm to low-level implementation details, Learning in Energy-Efficient Neuromorphic Computing: Algorithm and Architecture Co-Design also covers many fundamentals and essentials in neural networks (e.g., deep learning), as well as hardware implementation of neural networks. The book begins with an overview of neural networks. It then discusses algorithms for utilizing and training rate-based artificial neural networks. Next comes an introduction to various options for executing neural networks, ranging from general-purpose processors to specialized hardware, from digital accelerator to analog accelerator. A design example on building energy-efficient accelerator for adaptive dynamic programming with neural networks is also presented. An examination of fundamental concepts and popular learning algorithms for spiking neural networks follows that, along with a look at the hardware for spiking neural networks. Then comes a chapter offering readers three design examples (two of which are based on conventional CMOS, and one on emerging nanotechnology) to implement the learning algorithm found in the previous chapter. The book concludes with an outlook on the future of neural network hardware. Includes cross-layer survey of hardware accelerators for neuromorphic algorithms Covers the co-design of architecture and algorithms with emerging devices for much-improved computing efficiency Focuses on the co-design of algorithms and hardware, which is especially critical for using emerging devices, such as traditional memristors or diffusive memristors, for neuromorphic computing Learning in Energy-Efficient Neuromorphic Computing: Algorithm and Architecture Co-Design is an ideal resource for researchers, scientists, software engineers, and hardware engineers dealing with the ever-increasing requirement on power consumption and response time. It is also excellent for teaching and training undergraduate and graduate students about the latest generation neural networks with powerful learning capabilities.

Explore the cutting-edge of neuromorphic technologies with applications in Artificial Intelligence In Neuromorphic Devices for Brain-Inspired Computing: Artificial Intelligence, Perception, and Robotics, a team of expert engineers delivers a comprehensive discussion of all aspects of neuromorphic electronics designed to assist researchers and professionals to understand and apply all manner of brain-inspired computing and perception technologies. The book covers both memristic and neuromorphic devices, including spintronic, multi-terminal, and neuromorphic perceptual applications. Summarizing recent progress made in five distinct configurations of brain-inspired computing, the authors explore this promising technology’s potential applications in two specific areas: neuromorphic computing systems and neuromorphic perceptual systems. The book also includes: A thorough introduction to two-terminal neuromorphic memristors, including memristive devices and resistive switching mechanisms Comprehensive explorations of spintronic neuromorphic devices and multi-terminal neuromorphic devices with cognitive behaviors Practical discussions of neuromorphic devices based on chalcogenide and organic materials In-depth examinations of neuromorphic computing and perceptual systems with emerging devices Perfect for materials scientists, biochemists, and electronics engineers, Neuromorphic Devices for Brain-Inspired Computing: Artificial Intelligence, Perception, and Robotics will also earn a place in the libraries of neurochemists, neurobiologists, and neurophysiologists.

This book provides a structured treatment of the key principles and techniques for enabling efficient processing of deep neural networks (DNNs). DNNs are currently widely used for many artificial intelligence (AI) applications, including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Therefore, techniques that enable efficient processing of deep neural networks to improve key metrics—such as energy-efficiency, throughput, and latency—without sacrificing accuracy or increasing hardware costs are critical to enabling the wide deployment of DNNs in AI systems. The book includes background on DNN processing; a description and taxonomy of hardware architectural approaches for designing DNN accelerators; key metrics for evaluating and comparing different designs; features of DNN processing that are amenable to hardware/algorithm co-design to improve energy efficiency and throughput; and opportunities for applying new technologies. Readers will find a structured introduction to the field as well as formalization and organization of key concepts from contemporary work that provide insights that may spark new ideas.