Streaming Data

Streaming Data
Author: Andrew Psaltis
Publisher: Simon and Schuster
Total Pages: 314
Release: 2017-05-31
Genre: Computers
ISBN: 1638357242

Summary Streaming Data introduces the concepts and requirements of streaming and real-time data systems. The book is an idea-rich tutorial that teaches you to think about how to efficiently interact with fast-flowing data. Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. About the Technology As humans, we're constantly filtering and deciphering the information streaming toward us. In the same way, streaming data applications can accomplish amazing tasks like reading live location data to recommend nearby services, tracking faults with machinery in real time, and sending digital receipts before your customers leave the shop. Recent advances in streaming data technology and techniques make it possible for any developer to build these applications if they have the right mindset. This book will let you join them. About the Book Streaming Data is an idea-rich tutorial that teaches you to think about efficiently interacting with fast-flowing data. Through relevant examples and illustrated use cases, you'll explore designs for applications that read, analyze, share, and store streaming data. Along the way, you'll discover the roles of key technologies like Spark, Storm, Kafka, Flink, RabbitMQ, and more. This book offers the perfect balance between big-picture thinking and implementation details. What's Inside The right way to collect real-time data Architecting a streaming pipeline Analyzing the data Which technologies to use and when About the Reader Written for developers familiar with relational database concepts. No experience with streaming or real-time applications required. About the Author Andrew Psaltis is a software engineer focused on massively scalable real-time analytics. Table of Contents PART 1 - A NEW HOLISTIC APPROACH Introducing streaming data Getting data from clients: data ingestion Transporting the data from collection tier: decoupling the data pipeline Analyzing streaming data Algorithms for data analysis Storing the analyzed or collected data Making the data available Consumer device capabilities and limitations accessing the data PART 2 - TAKING IT REAL WORLD Analyzing Meetup RSVPs in real time

Data Pipelines Pocket Reference

Data Pipelines Pocket Reference
Author: James Densmore
Publisher: "O'Reilly Media, Inc."
Total Pages: 276
Release: 2021-02-10
Genre: Computers
ISBN: 1492087785

Data pipelines are the foundation for success in data analytics. Moving data from numerous diverse sources and transforming it to provide context is the difference between having data and actually gaining value from it. This pocket reference defines data pipelines and explains how they work in today's modern data stack. You'll learn common considerations and key decision points when implementing pipelines, such as batch versus streaming data ingestion and build versus buy. This book addresses the most common decisions made by data professionals and discusses foundational concepts that apply to open source frameworks, commercial products, and homegrown solutions. You'll learn: What a data pipeline is and how it works How data is moved and processed on modern data infrastructure, including cloud platforms Common tools and products used by data engineers to build pipelines How pipelines support analytics and reporting needs Considerations for pipeline maintenance, testing, and alerting

Data Pipelines with Apache Airflow

Data Pipelines with Apache Airflow
Author: Bas P. Harenslak
Publisher: Simon and Schuster
Total Pages: 478
Release: 2021-04-27
Genre: Computers
ISBN: 1617296902

This book teaches you how to build and maintain effective data pipelines. Youll explore the most common usage patterns, including aggregating multiple data sources, connecting to and from data lakes, and cloud deployment. --

Kafka: The Definitive Guide

Kafka: The Definitive Guide
Author: Neha Narkhede
Publisher: "O'Reilly Media, Inc."
Total Pages: 315
Release: 2017-08-31
Genre: Computers
ISBN: 1491936118

Every enterprise application creates data, whether it’s log messages, metrics, user activity, outgoing messages, or something else. And how to move all of this data becomes nearly as important as the data itself. If you’re an application architect, developer, or production engineer new to Apache Kafka, this practical guide shows you how to use this open source streaming platform to handle real-time data feeds. Engineers from Confluent and LinkedIn who are responsible for developing Kafka explain how to deploy production Kafka clusters, write reliable event-driven microservices, and build scalable stream-processing applications with this platform. Through detailed examples, you’ll learn Kafka’s design principles, reliability guarantees, key APIs, and architecture details, including the replication protocol, the controller, and the storage layer. Understand publish-subscribe messaging and how it fits in the big data ecosystem. Explore Kafka producers and consumers for writing and reading messages Understand Kafka patterns and use-case requirements to ensure reliable data delivery Get best practices for building data pipelines and applications with Kafka Manage Kafka in production, and learn to perform monitoring, tuning, and maintenance tasks Learn the most critical metrics among Kafka’s operational measurements Explore how Kafka’s stream delivery capabilities make it a perfect source for stream processing systems

Google BigQuery: The Definitive Guide

Google BigQuery: The Definitive Guide
Author: Valliappa Lakshmanan
Publisher: O'Reilly Media
Total Pages: 522
Release: 2019-10-23
Genre: Computers
ISBN: 1492044431

Work with petabyte-scale datasets while building a collaborative, agile workplace in the process. This practical book is the canonical reference to Google BigQuery, the query engine that lets you conduct interactive analysis of large datasets. BigQuery enables enterprises to efficiently store, query, ingest, and learn from their data in a convenient framework. With this book, you’ll examine how to analyze data at scale to derive insights from large datasets efficiently. Valliappa Lakshmanan, tech lead for Google Cloud Platform, and Jordan Tigani, engineering director for the BigQuery team, provide best practices for modern data warehousing within an autoscaled, serverless public cloud. Whether you want to explore parts of BigQuery you’re not familiar with or prefer to focus on specific tasks, this reference is indispensable.

SQL Server 2019 Administrator's Guide

SQL Server 2019 Administrator's Guide
Author: Marek Chmel
Publisher: Packt Publishing Ltd
Total Pages: 522
Release: 2020-09-11
Genre: Computers
ISBN: 1789950333

Use Microsoft SQL Server 2019 to implement, administer, and secure a robust database solution that is disaster-proof and highly available Key FeaturesExplore new features of SQL Server 2019 to set up, administer, and maintain your database solution successfullyDevelop a dynamic SQL Server environment and streamline big data pipelinesDiscover best practices for fixing performance issues, database access management, replication, and securityBook Description SQL Server is one of the most popular relational database management systems developed by Microsoft. This second edition of the SQL Server Administrator's Guide will not only teach you how to administer an enterprise database, but also help you become proficient at managing and keeping the database available, secure, and stable. You’ll start by learning how to set up your SQL Server and configure new and existing environments for optimal use. The book then takes you through designing aspects and delves into performance tuning by showing you how to use indexes effectively. You’ll understand certain choices that need to be made about backups, implement security policy, and discover how to keep your environment healthy. Tools available for monitoring and managing a SQL Server database, including automating health reviews, performance checks, and much more, will also be discussed in detail. As you advance, the book covers essential topics such as migration, upgrading, and consolidation, along with the techniques that will help you when things go wrong. Once you’ve got to grips with integration with Azure and streamlining big data pipelines, you’ll learn best practices from industry experts for maintaining a highly reliable database solution. Whether you are an administrator or are looking to get started with database administration, this SQL Server book will help you develop the skills you need to successfully create, design, and deploy database solutions. What you will learnDiscover SQL Server 2019’s new features and how to implement themFix performance issues by optimizing queries and making use of indexesDesign and use an optimal database management strategyCombine SQL Server 2019 with Azure and manage your solution using various automation techniquesImplement efficient backup and recovery techniques in line with security policiesGet to grips with migrating, upgrading, and consolidating with SQL ServerSet up an AlwaysOn-enabled stable and fast SQL Server 2019 environmentUnderstand how to work with Big Data on SQL Server environmentsWho this book is for This book is for database administrators, database developers, and anyone who wants to administer large and multiple databases single-handedly using Microsoft's SQL Server 2019. Basic awareness of database concepts and experience with previous SQL Server versions is required.

Continuous Delivery with Docker and Jenkins

Continuous Delivery with Docker and Jenkins
Author: Rafal Leszko
Publisher: Packt Publishing Ltd
Total Pages: 326
Release: 2017-08-24
Genre: Computers
ISBN: 1787126145

Unleash the combination of Docker and Jenkins in order to enhance the DevOps workflow About This Book Build reliable and secure applications using Docker containers. Create a complete Continuous Delivery pipeline using Docker, Jenkins, and Ansible. Deliver your applications directly on the Docker Swarm cluster. Create more complex solutions using multi-containers and database migrations. Who This Book Is For This book is indented to provide a full overview of deep learning. From the beginner in deep learning and artificial intelligence to the data scientist who wants to become familiar with Theano and its supporting libraries, or have an extended understanding of deep neural nets. Some basic skills in Python programming and computer science will help, as well as skills in elementary algebra and calculus. What You Will Learn Get to grips with docker fundamentals and how to dockerize an application for the Continuous Delivery process Configure Jenkins and scale it using Docker-based agents Understand the principles and the technical aspects of a successful Continuous Delivery pipeline Create a complete Continuous Delivery process using modern tools: Docker, Jenkins, and Ansible Write acceptance tests using Cucumber and run them in the Docker ecosystem using Jenkins Create multi-container applications using Docker Compose Managing database changes inside the Continuous Delivery process and understand effective frameworks such as Cucumber and Flyweight Build clustering applications with Jenkins using Docker Swarm Publish a built Docker image to a Docker Registry and deploy cycles of Jenkins pipelines using community best practices In Detail The combination of Docker and Jenkins improves your Continuous Delivery pipeline using fewer resources. It also helps you scale up your builds, automate tasks and speed up Jenkins performance with the benefits of Docker containerization. This book will explain the advantages of combining Jenkins and Docker to improve the continuous integration and delivery process of app development. It will start with setting up a Docker server and configuring Jenkins on it. It will then provide steps to build applications on Docker files and integrate them with Jenkins using continuous delivery processes such as continuous integration, automated acceptance testing, and configuration management. Moving on you will learn how to ensure quick application deployment with Docker containers along with scaling Jenkins using Docker Swarm. Next, you will get to know how to deploy applications using Docker images and testing them with Jenkins. By the end of the book, you will be enhancing the DevOps workflow by integrating the functionalities of Docker and Jenkins. Style and approach The book is aimed at DevOps Engineers, developers and IT Operations who want to enhance the DevOps culture using Docker and Jenkins.

Building an Anonymization Pipeline

Building an Anonymization Pipeline
Author: Luk Arbuckle
Publisher: O'Reilly Media
Total Pages: 167
Release: 2020-04-13
Genre: Computers
ISBN: 1492053406

How can you use data in a way that protects individual privacy but still provides useful and meaningful analytics? With this practical book, data architects and engineers will learn how to establish and integrate secure, repeatable anonymization processes into their data flows and analytics in a sustainable manner. Luk Arbuckle and Khaled El Emam from Privacy Analytics explore end-to-end solutions for anonymizing device and IoT data, based on collection models and use cases that address real business needs. These examples come from some of the most demanding data environments, such as healthcare, using approaches that have withstood the test of time. Create anonymization solutions diverse enough to cover a spectrum of use cases Match your solutions to the data you use, the people you share it with, and your analysis goals Build anonymization pipelines around various data collection models to cover different business needs Generate an anonymized version of original data or use an analytics platform to generate anonymized outputs Examine the ethical issues around the use of anonymized data