Continuous Models for Cameras and Inertial Sensors

Continuous Models for Cameras and Inertial Sensors
Author: Hannes Ovrén
Publisher: Linköping University Electronic Press
Total Pages: 67
Release: 2018-07-25
Genre:
ISBN: 917685244X

Using images to reconstruct the world in three dimensions is a classical computer vision task. Some examples of applications where this is useful are autonomous mapping and navigation, urban planning, and special effects in movies. One common approach to 3D reconstruction is ”structure from motion” where a scene is imaged multiple times from different positions, e.g. by moving the camera. However, in a twist of irony, many structure from motion methods work best when the camera is stationary while the image is captured. This is because the motion of the camera can cause distortions in the image that lead to worse image measurements, and thus a worse reconstruction. One such distortion common to all cameras is motion blur, while another is connected to the use of an electronic rolling shutter. Instead of capturing all pixels of the image at once, a camera with a rolling shutter captures the image row by row. If the camera is moving while the image is captured the rolling shutter causes non-rigid distortions in the image that, unless handled, can severely impact the reconstruction quality. This thesis studies methods to robustly perform 3D reconstruction in the case of a moving camera. To do so, the proposed methods make use of an inertial measurement unit (IMU). The IMU measures the angular velocities and linear accelerations of the camera, and these can be used to estimate the trajectory of the camera over time. Knowledge of the camera motion can then be used to correct for the distortions caused by the rolling shutter. Another benefit of an IMU is that it can provide measurements also in situations when a camera can not, e.g. because of excessive motion blur, or absence of scene structure. To use a camera together with an IMU, the camera-IMU system must be jointly calibrated. The relationship between their respective coordinate frames need to be established, and their timings need to be synchronized. This thesis shows how to automatically perform this calibration and synchronization, without requiring e.g. calibration objects or special motion patterns. In standard structure from motion, the camera trajectory is modeled as discrete poses, with one pose per image. Switching instead to a formulation with a continuous-time camera trajectory provides a natural way to handle rolling shutter distortions, and also to incorporate inertial measurements. To model the continuous-time trajectory, many authors have used splines. The ability for a spline-based trajectory to model the real motion depends on the density of its spline knots. Choosing a too smooth spline results in approximation errors. This thesis proposes a method to estimate the spline approximation error, and use it to better balance camera and IMU measurements, when used in a sensor fusion framework. Also proposed is a way to automatically decide how dense the spline needs to be to achieve a good reconstruction. Another approach to reconstruct a 3D scene is to use a camera that directly measures depth. Some depth cameras, like the well-known Microsoft Kinect, are susceptible to the same rolling shutter effects as normal cameras. This thesis quantifies the effect of the rolling shutter distortion on 3D reconstruction, depending on the amount of motion. It is also shown that a better 3D model is obtained if the depth images are corrected using inertial measurements. Att använda bilder för att återskapa världen omkring oss i tre dimensioner är ett klassiskt problem inom datorseende. Några exempel på användningsområden är inom navigering och kartering för autonoma system, stadsplanering och specialeffekter för film och spel. En vanlig metod för 3D-rekonstruktion är det som kallas ”struktur från rörelse”. Namnet kommer sig av att man avbildar (fotograferar) en miljö från flera olika platser, till exempel genom att flytta kameran. Det är därför något ironiskt att många struktur-från-rörelse-algoritmer får problem om kameran inte är stilla när bilderna tas, exempelvis genom att använda sig av ett stativ. Anledningen är att en kamera i rörelse ger upphov till störningar i bilden vilket ger sämre bildmätningar, och därmed en sämre 3D-rekonstruktion. Ett välkänt exempel är rörelseoskärpa, medan ett annat är kopplat till användandet av en elektronisk rullande slutare. I en kamera med rullande slutare avbildas inte alla pixlar i bilden samtidigt, utan istället rad för rad. Om kameran rör på sig medan bilden tas uppstår därför störningar i bilden som måste tas om hand om för att få en bra rekonstruktion. Den här avhandlingen berör robusta metoder för 3D-rekonstruktion med rörliga kameror. En röd tråd inom arbetet är användandet av en tröghetssensor (IMU). En IMU mäter vinkelhastigheter och accelerationer, och dessa mätningar kan användas för att bestämma hur kameran har rört sig över tid. Kunskap om kamerans rörelse ger möjlighet att korrigera för störningar på grund av den rullande slutaren. Ytterligare en fördel med en IMU är att den ger mätningar även i de fall då en kamera inte kan göra det. Exempel på sådana fall är vid extrem rörelseoskärpa, starkt motljus, eller om det saknas struktur i bilden. Om man vill använda en kamera tillsammans med en IMU så måste dessa kalibreras och synkroniseras: relationen mellan deras respektive koordinatsystem måste bestämmas, och de måste vara överens om vad klockan är. I den här avhandlingen presenteras en metod för att automatiskt kalibrera och synkronisera ett kamera-IMU-system utan krav på exempelvis kalibreringsobjekt eller speciella rörelsemönster. I klassisk struktur från rörelse representeras kamerans rörelse av att varje bild beskrivs med en kamera-pose. Om man istället representerar kamerarörelsen som en tidskontinuerlig trajektoria kan man på ett naturligt sätt hantera problematiken kring rullande slutare. Det gör det också enkelt att införa tröghetsmätningar från en IMU. En tidskontinuerlig kameratrajektoria kan skapas på flera sätt, men en vanlig metod är att använda sig av så kallade splines. Förmågan hos en spline att representera den faktiska kamerarörelsen beror på hur tätt dess knutar placeras. Den här avhandlingen presenterar en metod för att uppskatta det approximationsfel som uppkommer vid valet av en för gles spline. Det uppskattade approximationsfelet kan sedan användas för att balansera mätningar från kameran och IMU:n när dessa används för sensorfusion. Avhandlingen innehåller också en metod för att bestämma hur tät en spline behöver vara för att ge ett gott resultat. En annan metod för 3D-rekonstruktion är att använda en kamera som också mäter djup, eller avstånd. Vissa djupkameror, till exempel Microsoft Kinect, har samma problematik med rullande slutare som vanliga kameror. I den här avhandlingen visas hur den rullande slutaren i kombination med olika typer och storlekar av rörelser påverkar den återskapade 3D-modellen. Genom att använda tröghetsmätningar från en IMU kan djupbilderna korrigeras, vilket visar sig ge en bättre 3D-modell.

Action Recognition in Continuous Data Streams Using Fusion of Depth and Inertial Sensing

Action Recognition in Continuous Data Streams Using Fusion of Depth and Inertial Sensing
Author: Neha Dawar
Publisher:
Total Pages:
Release: 2018
Genre: Human activity recognition
ISBN:

Human action or gesture recognition has been extensively studied in the literature spanning a wide variety of human-computer interaction applications including gaming, surveillance, healthcare monitoring, and assistive living. Sensors used for action or gesture recognition are primarily either vision-based sensors or inertial sensors. Compared to the great majority of previous works where a single modality sensor is used for action or gesture recognition, the simultaneous utilization of a depth camera and a wearable inertial sensor is considered in this dissertation. Furthermore, compared to the great majority of previous works in which actions are assumed to be segmented actions, this dissertation addresses a more realistic and practical scenario in which actions of interest occur continuously and randomly amongst arbitrary actions of non-interest. In this dissertation, computationally efficient solutions are presented to recognize actions of interest from continuous data streams captured simultaneously by a depth camera and a wearable inertial sensor. These solutions comprise three main steps of segmentation, detection, and classification. In the segmentation step, all motion segments are extracted from continuous action streams. In the detection step, the segmented actions are separated into actions of interest and actions of non- interest. In the classification step, the detected actions of interest are classified. The features considered include skeleton joint positions, depth motion maps, and statistical attributes of acceleration and angular velocity inertial signals. The classifiers considered include maximum entropy Markov model, support vector data description, collaborative representation classifier, convolutional neural network, and long short-term memory network. These solutions are applied to the two applications of smart TV hand gestures and transition movements for home healthcare monitoring. The results obtained indicate the effectiveness of the developed solutions in detecting and recognizing actions of interest in continuous data streams. It is shown that higher recognition rates are achieved when fusing the decisions from the two sensing modalities as compared to when each sensing modality is used individually. The results also indicate that the deep learning-based solution provides the best outcome among the solutions developed.

An Invitation to 3-D Vision

An Invitation to 3-D Vision
Author: Yi Ma
Publisher: Springer Science & Business Media
Total Pages: 542
Release: 2012-11-06
Genre: Computers
ISBN: 0387217797

This book introduces the geometry of 3-D vision, that is, the reconstruction of 3-D models of objects from a collection of 2-D images. It details the classic theory of two view geometry and shows that a more proper tool for studying the geometry of multiple views is the so-called rank consideration of the multiple view matrix. It also develops practical reconstruction algorithms and discusses possible extensions of the theory.

Risk - A Multidisciplinary Introduction

Risk - A Multidisciplinary Introduction
Author: Claudia Klüppelberg
Publisher: Springer
Total Pages: 472
Release: 2014-06-10
Genre: Mathematics
ISBN: 3319044869

This is a unique book addressing the integration of risk methodology from various fields. It will stimulate intellectual debate and communication across disciplines, promote better risk management practices and contribute to the development of risk management methodologies. Individual chapters explain fundamental risk models and measurement, and address risk and security issues from diverse areas such as finance and insurance, the health sciences, life sciences, engineering and information science. Integrated Risk Sciences is an emerging discipline that considers risks in different fields, aiming at a common language, and at sharing and improving methods developed in different fields. Readers should have a Bachelor degree and have taken at least one basic university course in statistics and probability. The main goal of the book is to provide basic knowledge on risk and security in a common language; the authors have taken particular care to ensure that all content can readily be understood by doctoral students and researchers across disciplines. Each chapter provides simple case studies and examples, open research questions and discussion points, and a selected bibliography inviting readers to further study.

Modern Computer Architecture and Organization

Modern Computer Architecture and Organization
Author: Jim Ledin
Publisher: Packt Publishing Ltd
Total Pages: 667
Release: 2022-05-04
Genre: Computers
ISBN: 1803238232

A no-nonsense, practical guide to current and future processor and computer architectures that enables you to design computer systems and develop better software applications across a variety of domains Key FeaturesUnderstand digital circuitry through the study of transistors, logic gates, and sequential logicLearn the architecture of x86, x64, ARM, and RISC-V processors, iPhones, and high-performance gaming PCsStudy the design principles underlying the domains of cybersecurity, bitcoin, and self-driving carsBook Description Are you a software developer, systems designer, or computer architecture student looking for a methodical introduction to digital device architectures, but are overwhelmed by the complexity of modern systems? This step-by-step guide will teach you how modern computer systems work with the help of practical examples and exercises. You'll gain insights into the internal behavior of processors down to the circuit level and will understand how the hardware executes code developed in high-level languages. This book will teach you the fundamentals of computer systems including transistors, logic gates, sequential logic, and instruction pipelines. You will learn details of modern processor architectures and instruction sets including x86, x64, ARM, and RISC-V. You will see how to implement a RISC-V processor in a low-cost FPGA board and write a quantum computing program and run it on an actual quantum computer. This edition has been updated to cover the architecture and design principles underlying the important domains of cybersecurity, blockchain and bitcoin mining, and self-driving vehicles. By the end of this book, you will have a thorough understanding of modern processors and computer architecture and the future directions these technologies are likely to take. What you will learnUnderstand the fundamentals of transistor technology and digital circuitsExplore the concepts underlying pipelining and superscalar processingImplement a complete RISC-V processor in a low-cost FPGAUnderstand the technology used to implement virtual machinesLearn about security-critical computing applications like financial transaction processingGet up to speed with blockchain and the hardware architectures used in bitcoin miningExplore the capabilities of self-navigating vehicle computing architecturesWrite a quantum computing program and run it on a real quantum computerWho this book is for This book is for software developers, computer engineering students, system designers, reverse engineers, and anyone looking to understand the architecture and design principles underlying modern computer systems: ranging from tiny, embedded devices to warehouse-size cloud server farms. A general understanding of computer processors is helpful but not required.

Proceedings of the 8th International Conference on Computational Science and Technology

Proceedings of the 8th International Conference on Computational Science and Technology
Author: Rayner Alfred
Publisher: Springer Nature
Total Pages: 887
Release: 2022-03-25
Genre: Technology & Engineering
ISBN: 9811685150

This book gathers the proceedings of the Seventh International Conference on Computational Science and Technology (ICCST 2021), held in Labuan, Malaysia, on 28–29 August 2021. The respective contributions offer practitioners and researchers a range of new computational techniques and solutions, identify emerging issues, and outline future research directions, while also showing them how to apply the latest large-scale, high-performance computational methods.

Computer Vision -- ACCV 2010 Workshops

Computer Vision -- ACCV 2010 Workshops
Author: Reinhard Koch
Publisher: Springer
Total Pages: 467
Release: 2011-09-15
Genre: Computers
ISBN: 3642228194

The two-volume set LNCS 6468-6469 contains the carefully selected and reviewed papers presented at the eight workshops that were held in conjunction with the 10th Asian Conference on Computer Vision, in Queenstown, New Zealand, in November 2010. From a total of 167 submissions to all workshops, 89 papers were selected for publication. The contributions are grouped together according to the main workshops topics, which were: computational photography and aesthetics; computer vision in vehicle technology: from Earth to Mars; electronic cultural heritage; subspace based methods; video event categorization, tagging and retrieval; visual surveillance; application of computer vision for mixed and augmented reality.

Road Condition Estimation with Data Mining Methods using Vehicle Based Sensors

Road Condition Estimation with Data Mining Methods using Vehicle Based Sensors
Author: Masino, Johannes
Publisher: KIT Scientific Publishing
Total Pages: 234
Release: 2021-03-31
Genre: Technology & Engineering
ISBN: 3731510049

The work provides novel methods to process inertial sensor and acoustic sensor data for road condition estimation and monitoring with application in vehicles, which serve as sensor platforms. Furthermore, methods are introduced to combine the results from various vehicles for a more reliable estimation.