Hardware-Aware Probabilistic Machine Learning Models

Hardware-Aware Probabilistic Machine Learning Models
Author: Laura Isabel Galindez Olascoaga
Publisher: Springer Nature
Total Pages: 163
Release: 2021-05-19
Genre: Technology & Engineering
ISBN: 3030740420

This book proposes probabilistic machine learning models that represent the hardware properties of the device hosting them. These models can be used to evaluate the impact that a specific device configuration may have on resource consumption and performance of the machine learning task, with the overarching goal of balancing the two optimally. The book first motivates extreme-edge computing in the context of the Internet of Things (IoT) paradigm. Then, it briefly reviews the steps involved in the execution of a machine learning task and identifies the implications associated with implementing this type of workload in resource-constrained devices. The core of this book focuses on augmenting and exploiting the properties of Bayesian Networks and Probabilistic Circuits in order to endow them with hardware-awareness. The proposed models can encode the properties of various device sub-systems that are typically not considered by other resource-aware strategies, bringing about resource-saving opportunities that traditional approaches fail to uncover. The performance of the proposed models and strategies is empirically evaluated for several use cases. All of the considered examples show the potential of attaining significant resource-saving opportunities with minimal accuracy losses at application time. Overall, this book constitutes a novel approach to hardware-algorithm co-optimization that further bridges the fields of Machine Learning and Electrical Engineering.

Hardware-aware Algorithms for Efficient Machine Learning

Hardware-aware Algorithms for Efficient Machine Learning
Author: Tri Dao Phuc Quang
Publisher:
Total Pages: 0
Release: 2023
Genre:
ISBN:

Machine learning (ML) training will continue to grow to consume more cycles, their inference will proliferate on more kinds of devices, and their capabilities will be used in more domains. Some goals central to this future are to make ML models efficient so they remain practical to train and deploy, and to unlock new application domains with new capabilities. We describe some recent developments in hardware-aware algorithms to improve the efficiency-quality tradeoff of ML models and equip them with long context. In Chapter 2, we focus on structured sparsity, a natural approach to mitigate the extensive compute and memory cost of large ML models. We describe a line of work on learnable fast transforms that, thanks to their expressiveness and efficiency, yields some of the first sparse training methods to speed up large models in wall-clock time (2x) without compromising their quality. In Chapter 3, we focus on efficient Transformer training and inference for long sequences. We describe FlashAttention, a fast and memory-efficient algorithm to compute attention with no approximation. By careful accounting of reads/writes between different levels of memory hierarchy, FlashAttention is 2-4x faster and uses 10-20x less memory compared to the best existing attention implementations, allowing us to train higher-quality Transformers with 8x longer context. FlashAttention is now widely used in some of the largest research labs and companies. In Chapter 4, we examine state-space models, a promising architecture designed for long-range memory. As we seek to understand why early state-space models did not perform well on language modeling tasks, we propose simple multiplicative interaction that expands their expressiveness. We also design hardware-friendly algorithms to train them. As a result, we are able to train state-space models to multi-billion parameter scale, demonstrating a new kind of model competitive with the dominant Transformers in language modeling. We conclude with some exciting directions in ML and systems, such as software-hardware co-design, structured sparsity for scientific AI, and long context for new AI workflows and modalities.

IoT Streams for Data-Driven Predictive Maintenance and IoT, Edge, and Mobile for Embedded Machine Learning

IoT Streams for Data-Driven Predictive Maintenance and IoT, Edge, and Mobile for Embedded Machine Learning
Author: Joao Gama
Publisher: Springer Nature
Total Pages: 317
Release: 2021-01-09
Genre: Computers
ISBN: 3030667707

This book constitutes selected papers from the Second International Workshop on IoT Streams for Data-Driven Predictive Maintenance, IoT Streams 2020, and First International Workshop on IoT, Edge, and Mobile for Embedded Machine Learning, ITEM 2020, co-located with ECML/PKDD 2020 and held in September 2020. Due to the COVID-19 pandemic the workshops were held online. The 21 full papers and 3 short papers presented in this volume were thoroughly reviewed and selected from 35 submissions and are organized according to the workshops and their topics: IoT Streams 2020: Stream Learning; Feature Learning; ITEM 2020: Unsupervised Machine Learning; Hardware; Methods; Quantization.

Efficient Execution of Irregular Dataflow Graphs

Efficient Execution of Irregular Dataflow Graphs
Author: Nimish Shah
Publisher: Springer Nature
Total Pages: 155
Release: 2023-08-14
Genre: Technology & Engineering
ISBN: 3031331362

This book focuses on the acceleration of emerging irregular sparse workloads, posed by novel artificial intelligent (AI) models and sparse linear algebra. Specifically, the book outlines several co-optimized hardware-software solutions for a highly promising class of emerging sparse AI models called Probabilistic Circuit (PC) and a similar sparse matrix workload for triangular linear systems (SpTRSV). The authors describe optimizations for the entire stack, targeting applications, compilation, hardware architecture and silicon implementation, resulting in orders of magnitude higher performance and energy-efficiency compared to the existing state-of-the-art solutions. Thus, this book provides important building blocks for the upcoming generation of edge AI platforms.

Advances in Intelligent Data Analysis XVIII

Advances in Intelligent Data Analysis XVIII
Author: Michael R. Berthold
Publisher: Springer Nature
Total Pages: 601
Release: 2020-04-22
Genre: Computers
ISBN: 3030445844

This open access book constitutes the proceedings of the 18th International Conference on Intelligent Data Analysis, IDA 2020, held in Konstanz, Germany, in April 2020. The 45 full papers presented in this volume were carefully reviewed and selected from 114 submissions. Advancing Intelligent Data Analysis requires novel, potentially game-changing ideas. IDA’s mission is to promote ideas over performance: a solid motivation can be as convincing as exhaustive empirical evaluation.

Machine Learning and Probabilistic Graphical Models for Decision Support Systems

Machine Learning and Probabilistic Graphical Models for Decision Support Systems
Author: Kim Phuc Tran
Publisher: CRC Press
Total Pages: 330
Release: 2022-10-13
Genre: Computers
ISBN: 100077144X

This book presents recent advancements in research, a review of new methods and techniques, and applications in decision support systems (DSS) with Machine Learning and Probabilistic Graphical Models, which are very effective techniques in gaining knowledge from Big Data and in interpreting decisions. It explores Bayesian network learning, Control Chart, Reinforcement Learning for multicriteria DSS, Anomaly Detection in Smart Manufacturing with Federated Learning, DSS in healthcare, DSS for supply chain management, etc. Researchers and practitioners alike will benefit from this book to enhance the understanding of machine learning, Probabilistic Graphical Models, and their uses in DSS in the context of decision making with uncertainty. The real-world case studies in various fields with guidance and recommendations for the practical applications of these studies are introduced in each chapter.

Fundamentals

Fundamentals
Author: Katharina Morik
Publisher: Walter de Gruyter GmbH & Co KG
Total Pages: 506
Release: 2022-12-31
Genre: Science
ISBN: 3110785943

Machine learning is part of Artificial Intelligence since its beginning. Certainly, not learning would only allow the perfect being to show intelligent behavior. All others, be it humans or machines, need to learn in order to enhance their capabilities. In the eighties of the last century, learning from examples and modeling human learning strategies have been investigated in concert. The formal statistical basis of many learning methods has been put forward later on and is still an integral part of machine learning. Neural networks have always been in the toolbox of methods. Integrating all the pre-processing, exploitation of kernel functions, and transformation steps of a machine learning process into the architecture of a deep neural network increased the performance of this model type considerably. Modern machine learning is challenged on the one hand by the amount of data and on the other hand by the demand of real-time inference. This leads to an interest in computing architectures and modern processors. For a long time, the machine learning research could take the von-Neumann architecture for granted. All algorithms were designed for the classical CPU. Issues of implementation on a particular architecture have been ignored. This is no longer possible. The time for independently investigating machine learning and computational architecture is over. Computing architecture has experienced a similarly rampant development from mainframe or personal computers in the last century to now very large compute clusters on the one hand and ubiquitous computing of embedded systems in the Internet of Things on the other hand. Cyber-physical systems’ sensors produce a huge amount of streaming data which need to be stored and analyzed. Their actuators need to react in real-time. This clearly establishes a close connection with machine learning. Cyber-physical systems and systems in the Internet of Things consist of diverse components, heterogeneous both in hard- and software. Modern multi-core systems, graphic processors, memory technologies and hardware-software codesign offer opportunities for better implementations of machine learning models. Machine learning and embedded systems together now form a field of research which tackles leading edge problems in machine learning, algorithm engineering, and embedded systems. Machine learning today needs to make the resource demands of learning and inference meet the resource constraints of used computer architecture and platforms. A large variety of algorithms for the same learning method and, moreover, diverse implementations of an algorithm for particular computing architectures optimize learning with respect to resource efficiency while keeping some guarantees of accuracy. The trade-off between a decreased energy consumption and an increased error rate, to just give an example, needs to be theoretically shown for training a model and the model inference. Pruning and quantization are ways of reducing the resource requirements by either compressing or approximating the model. In addition to memory and energy consumption, timeliness is an important issue, since many embedded systems are integrated into large products that interact with the physical world. If the results are delivered too late, they may have become useless. As a result, real-time guarantees are needed for such systems. To efficiently utilize the available resources, e.g., processing power, memory, and accelerators, with respect to response time, energy consumption, and power dissipation, different scheduling algorithms and resource management strategies need to be developed. This book series addresses machine learning under resource constraints as well as the application of the described methods in various domains of science and engineering. Turning big data into smart data requires many steps of data analysis: methods for extracting and selecting features, filtering and cleaning the data, joining heterogeneous sources, aggregating the data, and learning predictions need to scale up. The algorithms are challenged on the one hand by high-throughput data, gigantic data sets like in astrophysics, on the other hand by high dimensions like in genetic data. Resource constraints are given by the relation between the demands for processing the data and the capacity of the computing machinery. The resources are runtime, memory, communication, and energy. Novel machine learning algorithms are optimized with regard to minimal resource consumption. Moreover, learned predictions are applied to program executions in order to save resources. The three books will have the following subtopics: Volume 1: Machine Learning under Resource Constraints - Fundamentals Volume 2: Machine Learning and Physics under Resource Constraints - Discovery Volume 3: Machine Learning under Resource Constraints - Applications Volume 1 establishes the foundations of this new field (Machine Learning under Resource Constraints). It goes through all the steps from data collection, their summary and clustering, to the different aspects of resource-aware learning, i.e., hardware, memory, energy, and communication awareness. Several machine learning methods are inspected with respect to their resource requirements and how to enhance their scalability on diverse computing architectures ranging from embedded systems to large computing clusters.

Probabilistic Machine Learning for Civil Engineers

Probabilistic Machine Learning for Civil Engineers
Author: James-A. Goulet
Publisher: MIT Press
Total Pages: 298
Release: 2020-04-14
Genre: Computers
ISBN: 0262538709

An introduction to key concepts and techniques in probabilistic machine learning for civil engineering students and professionals; with many step-by-step examples, illustrations, and exercises. This book introduces probabilistic machine learning concepts to civil engineering students and professionals, presenting key approaches and techniques in a way that is accessible to readers without a specialized background in statistics or computer science. It presents different methods clearly and directly, through step-by-step examples, illustrations, and exercises. Having mastered the material, readers will be able to understand the more advanced machine learning literature from which this book draws. The book presents key approaches in the three subfields of probabilistic machine learning: supervised learning, unsupervised learning, and reinforcement learning. It first covers the background knowledge required to understand machine learning, including linear algebra and probability theory. It goes on to present Bayesian estimation, which is behind the formulation of both supervised and unsupervised learning methods, and Markov chain Monte Carlo methods, which enable Bayesian estimation in certain complex cases. The book then covers approaches associated with supervised learning, including regression methods and classification methods, and notions associated with unsupervised learning, including clustering, dimensionality reduction, Bayesian networks, state-space models, and model calibration. Finally, the book introduces fundamental concepts of rational decisions in uncertain contexts and rational decision-making in uncertain and sequential contexts. Building on this, the book describes the basics of reinforcement learning, whereby a virtual agent learns how to make optimal decisions through trial and error while interacting with its environment.

Machine learning using approximate inference

Machine learning using approximate inference
Author: Christian Andersson Naesseth
Publisher: Linköping University Electronic Press
Total Pages: 39
Release: 2018-11-27
Genre:
ISBN: 9176851613

Automatic decision making and pattern recognition under uncertainty are difficult tasks that are ubiquitous in our everyday life. The systems we design, and technology we develop, requires us to coherently represent and work with uncertainty in data. Probabilistic models and probabilistic inference gives us a powerful framework for solving this problem. Using this framework, while enticing, results in difficult-to-compute integrals and probabilities when conditioning on the observed data. This means we have a need for approximate inference, methods that solves the problem approximately using a systematic approach. In this thesis we develop new methods for efficient approximate inference in probabilistic models. There are generally two approaches to approximate inference, variational methods and Monte Carlo methods. In Monte Carlo methods we use a large number of random samples to approximate the integral of interest. With variational methods, on the other hand, we turn the integration problem into that of an optimization problem. We develop algorithms of both types and bridge the gap between them. First, we present a self-contained tutorial to the popular sequential Monte Carlo (SMC) class of methods. Next, we propose new algorithms and applications based on SMC for approximate inference in probabilistic graphical models. We derive nested sequential Monte Carlo, a new algorithm particularly well suited for inference in a large class of high-dimensional probabilistic models. Then, inspired by similar ideas we derive interacting particle Markov chain Monte Carlo to make use of parallelization to speed up approximate inference for universal probabilistic programming languages. After that, we show how we can make use of the rejection sampling process when generating gamma distributed random variables to speed up variational inference. Finally, we bridge the gap between SMC and variational methods by developing variational sequential Monte Carlo, a new flexible family of variational approximations.