Data Analytics with Google Cloud Platform , livre ebook

Data Analytics with Google Cloud Platform

Build Real Time Data Analytics on Google Cloud Platform

by
Murari Ramuka
FIRST EDITION 2020
Copyright © BPB Publications, India
ISBN: 978-93-89423-631
All Rights Reserved. No part of this publication may be reproduced or distributed in any form or by any means or stored in a database or retrieval system, without the prior written permission of the publisher with the exception to the program listings which may be entered, stored and executed in a computer system, but they can not be reproduced by the means of publication.
LIMITS OF LIABILITY AND DISCLAIMER OF WARRANTY
The information contained in this book is true to correct and the best of author’s & publisher’s knowledge. The author has made every effort to ensure the accuracy of these publications, but cannot be held responsible for any loss or damage arising from any information in this book.
All trademarks referred to in the book are acknowledged as properties of their respective owners.
Distributors:
BPB PUBLICATIONS
20, Ansari Road, Darya Ganj
New Delhi-110002
Ph: 23254990/23254991
MICRO MEDIA
Shop No. 5, Mahendra Chambers,
150 DN Rd. Next to Capital Cinema,
V.T. (C.S.T.) Station, MUMBAI-400 001
Ph: 22078296/22078297
DECCAN AGENCIES
4-3-329, Bank Street,
Hyderabad-500195
Ph: 24756967/24756400
BPB BOOK CENTRE
376 Old Lajpat Rai Market,
Delhi-110006
Ph: 23861747
Published by Manish Jain for BPB Publications, 20 Ansari Road, Darya Ganj, New Delhi-110002 and Printed by him at Repro India Ltd, Mumbai
Disclaimer
This book is neither endorsed nor affiliated with Google. Some of the images have been taken from Google
Dedicated to
My Lady Luck Smt Kusum Ramuka, Smt Sunita Ramuka, Miss Sukriti Ramuka
About the Author
Murari Ramuka is a seasoned Data Analytics professional, with 12+ years of experience in enabling data analytics platform using traditional DW/BI and Cloud Technologies (Azure, Google Cloud Platform) to uncover hidden insights and maximize revenue, profitability and ensure efficient operations management. He has worked with several multinational IT giants like Capgemini, Cognizant, Syntel and Icertis.
He is a self-motivated and committed data enthusiast, with expertise in banking and healthcare domains. He has worked for various customers across the world. He has helped them in setting up their end-to-end Data platform, which enabled them to progress to the next level of analytics, which include Real-time Analytics, recommendation engine, sentiment analytics, with his expertise in data analytics, especially the Cloud platform.
His keen interest lies in Cloud data analytics, machine learning and applications of natural language processing in various industry sectors. In his leisure time, he enjoys reading about the latest trends in data space and sharing the knowledge about the same through his LinkedIn page.
He believes in sharing and spreading knowledge and has conducted several meetups and technical events across India in the past. He is also part of AIM Mentor of Change program, a Govt of India Initiative by Niti Ayog.
About the Reviewer
Saurabh Saraff has 6 plus years of experience in data engineering, data modeling and architecture. He has more than 3 Years of experience working on Google cloud platform and is a Google Cloud certified Cloud Architect, Cloud engineer, and a Data engineer. He pursued his B.E. in Information Technology from Pune University. He is a data enthusiast and is working on AI and ML projects.
Acknowledgement
First and foremost, I would like to thank God for giving me the courage to write this book. I would like to thank everyone at BPB Publications for giving me this opportunity to publish my book.
I would also like to thank my loving and caring wife, Mrs. Sunita Ramuka, and my family for their endless support and help in numerous ways.
I would like to thank my mentors, Mr. Sanjay Raj, Mr. Monish Darda, Mr. Chetan Manjrekar, Mr. Ashish Arora, Mrs. Hema Chandrasekhar, all the other seniors, and my friends, Mr. Sunil Upadhyay, Mr. Rakesh Sahay, Mr. Shrish Tripathi for their useful discussions and suggestions-right from deciding the topics, writing the concepts, framing exercises, etc.
Lastly, I would like to thank my critics and reviewers. Without their inputs, I would not have been able to write this book.
—Murari Ramuka
Preface
In the last few years, Cloud Computing has been very popular and has become the first choice for organizations. It is being used across different industries due to its unmatched features and benefits.
This book will help in learning and applying sophisticated knowledge on Cloud Computing. It also explains different types of data services/technology and machine learning algorithm/Pre-Trained API to real-business problems, which are built on the Google Cloud Platform (GCP). With some of the latest business examples and hands-on guide, this book will enable the developers entering the data analytics fields to implement an end-to-end data pipeline, using the GCP Data services. Through the course of the book, you’ll come across multiple industries wise use cases, like Building Datawarehouse using Big Query, a sample real-time data analytics solution on machine learning, and Artificial Intelligence, which helped business decisions by employing a variety of data science approaches on the Google Cloud environment. Whether your business is at an early stage of cloud implementation in its journey or well on its way to digital transformation, Google Cloud’s solutions and technologies will help chart a path to success. This book can be used to develop the GCP concepts in an easy way. It contains many examples showcasing the implementation of a GCP service. It also aids in learning the basic and advanced concepts of Google Cloud Data Platform. This book is divided into 7 chapters and provides a detailed description of the core concepts of each of the Data services offered by Google Cloud.
Through this book you will learn how to: Make different Services available in the Google cloud Platform and when to use what for an application. Build a real-time streaming data pipeline to carry out real-time analytics using Cloud Dataflow, Pub/Sub, and Big Query. Conduct interactive data exploration and discovery with Google BigQuery. Auto Scaling and high availability to run the business. Performing Data Transformation, Data Cleansing, Data Wrangling, and Data Visualization Activities using different Google Cloud Platform Services. Create a high-performing Machine learning prediction model with TensorFlow. Taking advantage of fully trained ML models from Google Cloud Platform.
Errata
We take immense pride in our work at BPB Publications and follow best practices to ensure the accuracy of our content to provide with an indulging reading experience to our subscribers. Our readers are our mirrors, and we use their inputs to reflect and improve upon human errors if any, occurred during the publishing processes involved. To let us maintain the quality and help us reach out to any readers who might be having difficulties due to any unforeseen errors, please write to us at :
errata@bpbonline.com
Your support, suggestions and feedbacks are highly appreciated by the BPB Publications’ Family.
Table of Contents
1. GCP Overview and Architecture
Introduction
Structure
Objectives
Cloud computing history
On-premise versus cloud computing
Benefits of cloud computing
Infrastructure as a Service (IaaS)
Platform as a Service (PaaS)
Software as a Service (SaaS)
The cloud computing architecture
Google Cloud Platform
Why Google Cloud Platform?
Google Cloud Platform regions and zones
Google Cloud Platform Console
Billing
Resource hierarchy
Projects
Command-line interface
Roles and services in GCP
Primitive roles
Predefined roles
Custom roles
Application Engine
Services
Versions
Instances
Application requests
Limits
Compute Engines
Instances
Container Engines
Container technologies that run on Compute Engine
Container-optimized VM images
Cloud Functions
Connect and extend cloud services
Events and triggers
Serverless
Security via IAM
Cloud IAM and policy APIs
Policy hierarchy
Conclusion
Questions
2. Google Cloud Platform Storage
Cloud Storage
Overview of storage classes
Comparison of storage classes
Bucket
Cloud Datastore
Cloud Firestore in Datastore mode
Comparison of Cloud Datastore and Cloud Firestore with ancient databases
Cloud Firestore
Documents
Collections
Cloud SQL
Comparison between Cloud SQL and standard MySQL based on their functionality
Cloud SQL for PostgreSQL
Differences between Cloud SQL and the standard PostgreSQL functionality
Cloud Spanner
Cloud Bigtable
Cloud Bigtable storage model
Cloud Bigtable architecture
Cloud BigQuery
Conclusion
Questions
3. Data Processing and Message with Dataflow and Pub/Sub
Introduction
Structure
Objectives
Cloud Dataflow
Cloud Dataflow templates
Traditional versus templated job execution
Data transformation with Cloud Dataflow
Example of the WordCount Template
Apache Beam
Working of Apache Beam code
Cloud Pub/Sub
Publisher-subscriber relationships
Cloud Pub/Sub message flow
Cloud Pub/Sub integrations
Fundamentals of a Publish/Subscribe service
Judging performance of a messaging service
Cloud Pub/Sub basic architecture
Control plane
Data plane - The lifecycle of a message
Cloud Pub/Sub implementation
Conclusion
Questions
4. Data Processing with Dataproc and Dataprep
Introduction
Structure
Objectives
Cloud Dataproc
Cloud Dataproc usage
Cloud Dataproc parts
Removing/Terminating the Dataproc Cluster
Moving on-premises Hadoop infrastructure to GCP
Best practices of Cloud Dataproc
Cloud Dataprep
Setting up Cloud Dataprep service
Create a flow in Cloud Dataprep
Conclusion
Questions
5. BigQuery and