Master Azure Databricks

Description

Module 1 :

What is Data Pipeline
What is Azure databricks
Azure Databricks Architecture
Azure Account Setup
WorkSpace Setup

Module 2:

Navigate the Workspace
Runtimes
Clusters
Notebooks
Libraries
Repos
Databricks File System (DBFS)
DBUTILS
Widgets
Workflows
Metastore – Setup external Metastore

Module 3 :

What is RDD
Creating RDD
RDD transformations
RDD Actions
RDD Joins
Pair RDD
Broadcast Variables
Accumulators
Convert RDD to DataFrame
Import & Read data
Create a table using the UI
Create a table in a Notebook

Module 4 :

Create DataFrames
Define Schema
Functions
Casting Operations
Filter Transformation
Update, Update ALL & UpdateByName
OrderBy & SortBY
GroupBy
Remove Duplicates
Window Functions
Date and Timestamp Functions
UDF (User Defined Function)
JOIN
Handle corrupt records using the badRecordsPath
File metadata column

Module 5 :

Read Parquet File
Read CSV Files
Read JSON Files
Read XML Files
Read Excel file
SQL databases using JDBC
Azure blob storage

Module 6 :

What is Spark Structure Streaming
Data Source & Sink
Rate & File Source
Kafka Source
Sink : Console, Memory, File & Custom
Build Streaming ETL
Stream ETL 1 : Setup Event Hub
Streaming ETL 2 : Event Hub Producer
Streaming ETL 3 : Integrate Event Hubs with Data Bricks
Streaming ETL 4 : Transformation
Streaming ETL 5 : Ingest into Azure Data storage
Twitter Sentiment Analysis – Introduction
Setup Twitter Developer Account
Twitter Sentiment Analysis – II
Twitter Sentiment Analysis – III

Module 7 :

Components in Databricks SQL
Configuring a SQL Endpoint
Creating a Table from a CSV File
Create Queries
Parameterized Query
Query Profile
Building Visualization (Table, BAR & PIE )
Building Line Chart & Counter Chart
Adding Charts to Dashboards
Defining a Query Alert
Access Control on Databricks SQL Objects
Lab: Data Object Access Control
Transfer Ownership
Access SQL Endpoint from Python
Databricks SQL CLI
Databricks SQL CLI

Who this course is for:

Data Engineering Students & Developers
Bigdata Developer
Python & SQL Developer

Requirements

Basic Python Skills
Basic SQL Skills
Azure Account

Last Updated 10/2022