Data Engineering for Beginners: Learn SQL, Python & Spark
Год выпуска: 2/2025
Производитель: Udemy
Сайт производителя:
https://www.udemy.com/course/data-engineering-essentials-sql-python-and-spark/
Автор: Durga Viswanatha Raju Gadiraju, Vinay Gadiraju
Продолжительность: 55h 56m
Тип раздаваемого материала: Видеоурок
Язык: Английский
Описание: Why Learn Data Engineering?
Data Engineering is one of the fastest-growing fields in the tech industry. Organizations of all sizes rely on Data Engineers to build and maintain the infrastructure that powers big data analytics, reporting, and machine learning. Data Engineers design, implement, and optimize data pipelines to efficiently process and manage data for business intelligence, real-time analytics, and AI applications.
With SQL, Python, and Apache Spark, Data Engineers can handle large-scale data processing efficiently. These skills are highly sought after in finance, healthcare, e-commerce, and every data-driven industry.
If you are looking for an industry-relevant and practical course that teaches you how to work with SQL, Python, Apache Spark (PySpark), and Databricks on Google Cloud Platform (GCP), this course is the perfect place to start.
What You Will Learn in This Course
This course is designed to take you from a beginner to an intermediate level in Data Engineering. You will gain hands-on experience working with SQL, Python, Apache Spark (PySpark), and Databricks by building real-world batch and streaming data pipelines.
SQL for Data Engineering (PostgreSQL)
Install and configure PostgreSQL to practice SQL queries
Learn fundamental SQL concepts such as SELECT, WHERE, JOIN, GROUP BY, HAVING, and ORDER BY
Perform advanced SQL operations including window functions, ranking, cumulative aggregations, and complex joins
Learn how to optimize SQL queries for performance and debugging
Python for Data Engineering
Understand Python fundamentals for data processing
Work with Python Collections to efficiently process structured data
Use Pandas to manipulate, clean, and analyze data
Build real-world Python projects, including a File Format Converter and a Database Loader
Learn how to troubleshoot and debug Python applications
Understand performance tuning strategies for Python-based data pipelines
Apache Spark (PySpark) for Big Data Processing
Learn Spark SQL to process structured data at scale
Work with PySpark DataFrame APIs to manipulate big data
Create and manage Delta Tables and perform CRUD operations (INSERT, UPDATE, DELETE, MERGE)
Perform advanced SQL transformations using window functions, ranking, and aggregations
Learn how to optimize PySpark jobs using Spark Catalyst Optimizer and Explain Plans
Debug, monitor, and optimize Spark jobs using Spark UI
Deploying Data Pipelines on Databricks (Google Cloud Platform - GCP)
Set up and configure Databricks on Google Cloud Platform (GCP)
Learn how to provision and manage Databricks clusters
Develop PySpark applications on Databricks and execute jobs on multi-node clusters
Understand the cost, scalability, and benefits of using Databricks for Data Engineering
Performance Tuning and Optimization in Data Engineering
Learn query performance optimization techniques in SQL and PySpark
Implement partitioning and columnar storage formats to improve efficiency
Explore debugging techniques for troubleshooting SQL and PySpark applications
Analyze Spark execution plans to improve job execution performance
Common Challenges in Learning Data Engineering and How This Course Helps
Many learners struggle with setting up a proper Data Engineering environment, finding structured learning material, and gaining hands-on experience with real-world projects.
This course eliminates these challenges by providing:
A step-by-step guide to setting up PostgreSQL, Python, and Apache Spark
Hands-on exercises that simulate real-world Data Engineering problems
Practical projects that reinforce learning and build confidence
Cloud-based Data Engineering with Databricks on Google Cloud, making it easier to work with large-scale data
Who Should Take This Course?
This course is designed for:
Beginners who want to start a career in Data Engineering
Aspiring Data Engineers who want to learn SQL, Python, Apache Spark (PySpark), and Databricks
Software Developers and Data Analysts who want to transition into Data Engineering
Data Science and Machine Learning Practitioners who need a deeper understanding of data pipelines
Anyone interested in Big Data, ETL processes, and cloud-based Data Engineering
Why Take This Course?
Beginner-Friendly Approach
This course starts with the fundamentals and gradually builds up to advanced topics, making it accessible for beginners.
Hands-On Learning with Real-World Projects
You will work on real-world projects to reinforce your skills and gain practical experience in building Data Pipelines.
Cloud-Based Training on Databricks (GCP)
This course teaches cloud-based Data Engineering using Databricks on Google Cloud, a platform widely used by companies for Big Data processing and machine learning.
Comprehensive Curriculum Covering All Key Data Engineering Skills
This course covers SQL, Python, Apache Spark (PySpark), Databricks, ETL, Big Data Processing, and Performance Optimization—all essential skills for a Data Engineer.
Performance Tuning and Debugging
You will learn how to analyze Spark execution plans, optimize SQL queries, and debug PySpark jobs, which are crucial for real-world Data Engineering projects.
Lifetime Access and Updates
You get lifetime access to the course content, which is regularly updated to keep up with industry trends and new technologies.
Course Features
Step-by-step instructions with detailed explanations
Hands-on exercises to reinforce learning
Real-world projects covering batch and streaming data pipelines
Complete Databricks setup guide for Google Cloud
Performance optimization techniques for SQL and PySpark
Best practices for debugging and tuning Spark jobs
Enroll Today and Start Your Data Engineering Journey
If you are serious about learning Data Engineering and want to master SQL, Python, Apache Spark (PySpark), and Databricks on Google Cloud, this course will provide you with the essential skills and hands-on experience needed to succeed in this field.
Take the first step in your Data Engineering journey today—enroll now!
Содержание
01. Introduction to Data Engineering Essentials using SQL, Python, and PySpark
1. Introduction to Data Engineering Essentials Course (02:41)
2. Overview of our support to Data Engineering Essentials course (04:40)
3. Overview of SQL topics covered in the course (06:57)
4. Overview of Python topics covered in the course (07:02)
5. Overview of Getting Started with GCP related to the course (04:09)
6. Overview of Spark and Databricks Environment related topics (04:46)
7. Detailed outline of Spark SQL Topics in the course (06:35)
8. Detailed outline of Pyspark Topics in the course (05:10)
9. Detailed outline of ELT Data Pipelines on Databricks (02:27)
10. Overview of Performance Tuning of Spark covered in the course (05:14)
02. Getting Started with SQL for Data Engineering
1. Introduction to SQL for Data Engineering (01:36)
2. Overview of Application Architecture and RDBMS (03:58)
3. Overview of Database Technologies and relevance of SQL (06:16)
4. Overview of Purpose Built Databases (06:00)
5. Overview of Data Warehouse and Data Lake (03:13)
6. Usage of RDBMS and Data Warehouse technologies (03:36)
7. Differences and Similarities between RDBMS and Data Warehouse Technologies (03:52)
03. Setup Tools for Data Engineering Essentials
1. Introduction to Setting up Tools for Data Engineering Essentials (02:29)
2. Setup VS Code on Windows (03:22)
3. Setup Python 3.9 on Windows (05:42)
4. Configure Environment Variable PATH for Python on Windows (04:18)
5. Overview of learning Python using Python CLI (03:22)
6. Integrate VSCode with Python on Windows (05:02)
7. Install Postgres 14 on Windows 11 (05:13)
8. Getting Started with pgAdmin on Windows (02:36)
9. Getting Started with pgAdmin on Mac (05:31)
10. Conclusion of Setting up Tools for Data Engineering Essentials (01:45)
04. Setup Application Tables and Data in Postgres Database
1. Overview of Postgres Database Server and pgAdmin (08:07)
2. Overview of Database Connection Details (07:03)
3. Overview of Connecting to External Databases using pgAdmin (04:19)
4. Create Application Database and User in Postgres Database Server (03:47)
5. Clone Data Sets from Git Repository for Database Scripts (02:50)
6. Register Server in pgAdmin using Application Database and User (03:58)
7. Setup Application Tables and Data in Postgres Database (07:19)
8. Overview of pgAdmin to write SQL Queries (11:41)
05. Writing Basic SQL Queries
1. Review Data Model Diagram (04:33)
2. Define Problem Statement for SQL Queries (01:42)
3. Filtering Data using SQL Queries (06:27)
4. Total Aggregations using SQL Queries (05:26)
5. Group By Aggregations using SQL Queries (07:29)
6. Order of Execution of SQL Queries (07:31)
7. Rules and Restrictions to Group and Filter Data in SQL queries (05:19)
8. Filter Data based on Aggregated Results using Group By and Having (04:31)
9. Inner Joins using SQL Queries (05:15)
10. Outer Joins using SQL Queries (07:12)
11. Filter and Aggregate on Join Results using SQL (03:57)
12. Overview of Database Views (06:02)
13. Overview of Common Table Expressions or CTEs (04:03)
14. Outer Join with Additional Conditions in SQL Queries (08:10)
15. Explanation about Fix of SQL Queries with Filtering on Outer Join Results (04:20)
06. Cumulative Aggregations and Ranking in SQL Queries
1. Introduction to Cumulative Aggregations and Ranking in SQL Queries (01:29)
2. Overview of CTAS to create tables based on Query Results (04:26)
3. Create Tables for Cumulative Aggregations and Ranking (02:11)
4. Overview of OVER and PARTITION BY Clause in SQL Queries (06:59)
5. Compute Total Aggregation using OVER and PARTITION BY in SQL Queries (01:47)
6. Overview of Ranking in SQL (03:09)
7. Compute Global Ranks using SQL (03:54)
8. Compute Ranks based on key using SQL (04:15)
9. Rules and Restrictions to Filter Data based on Ranks in SQL (02:45)
10. Filtering based on Global Ranks using Nested Queries and CTEs in SQL (05:04)
11. Filtering based on Ranks per Partition using Nested Queries and CTEs in SQL (04:51)
12. Create Students table with Data for ranking using SQL (03:11)
13. Difference between rank and dense rank using SQL (05:39)
07. SQL Troubleshooting and Debugging Guide
1. Introduction to SQL Troubleshooting and Debugging Guide (01:48)
2. Overview of Database Connectivity Issues (07:17)
3. Validate and Setup Telnet on Mac or PC (04:12)
4. Validate Connectivity to Database Server using telnet (04:27)
5. Troubleshoot Database Connectivity Issue with Correct Host Details (06:12)
6. Current Databases and Users in Postgres Database Server (04:38)
7. Troubleshoot Database Credentials and Permissions Issues (07:55)
8. Overview of Compilation of SQL Queries (05:31)
9. Troubleshooting Syntax Errors in SQL Queries (04:03)
10. Troubleshooting Semantec Errors in SQL Queries (04:56)
11. Overview of Bugs in SQL Queries (02:56)
12. Development Best Practices with tips to troubleshoot SQL bugs (03:47)
13. Develop Initial Solution based on the requirement (04:14)
14. Identify and Troubleshoot Bugs in SQL Queries (08:06)
15. Develop Solution using Development Best Practices (04:39)
08. Performance Tuning of SQL Queries
1. Introduction to Performance Tuning of SQL Queries (03:32)
2. Overview of SQL Compilation Process and Explain Plans (04:26)
3. Generate Explain Plans for SQL Queries (05:59)
4. Review Tables used for Performance Tuning of SQL Queries (07:03)
5. Review Data Storage Internals for Tables and Indexes (05:15)
6. Review key terms used in Explain Plans for SQL Queries (03:42)
7. Interpret Explain Plans for Basic SQL Queries (05:23)
8. Review the Common Application Scenarios for Performance Tuning (03:01)
9. Write SQL Queries for Customer Orders (05:37)
10. Performance Testing of SQL Queries using Stored Procedure (04:05)
11. Add Required Indexes to tune performance of SQL Queries (07:26)
12. Guidelines on adding Indexes on Tables for SQL Queries (03:40)
13. Interpreting the explain plan for SQL Queries using Indexes (05:38)
14. Conclusion of Performance Tuning of SQL Queries (02:42)
09. Exercises for Basic SQL Queries
1. Simple Exercises for Filtering and Aggregations (04:23)
2. Exercises on Joins and Aggregations using SQL (02:50)
10. Solutions for Basic SQL Queries
1. Solutions for Filtering and Aggregations (04:56)
2. Solutions for Filtering and Aggregations (05:31)
3. Validate Data and Review Data Model Diagram (03:25)
4. Solution for Exercise 1 to get Customer Order Count (07:50)
5. Solution for Exercise 2 to get Dormant Customers using Outer Join (10:55)
6. Solution for Exercise 3 to get Revenue Per Customer using Outer Join (07:49)
7. Solution for Exercise 4 to get Revenue Per Category (07:37)
8. Solution for Exercise 5 to get Product Count Per Department (08:29)
11. Getting Started with Python
1. Setup Visual Studio Workspace for Python Application Development (06:04)
2. Setup Notebook Environment in VS Code Workspace (06:48)
3. Overview of VS Code Notebook Environment (04:39)
4. Overview of Cells in VS Code Notebook (06:06)
5. Defining Functions in VS Code Notebooks (05:49)
6. Run the Code in VS Code Notebook Cell by Line (05:18)
7. Constants and Variables in Python (05:43)
8. Overview of Python Data Types (04:11)
9. Getting help on Python Variables and Functions (06:15)
10. Pre-Defined String Manipulation Functions (06:37)
11. Overview of Python Lists (06:42)
12. Loops and Conditions in Python (08:49)
13. User Defined Functions in Python (06:29)
12. Python Collections for Data Engineering
1. Overview of File IO using Python (08:22)
2. Read Data from CSV File into Python List (03:55)
3. Overview of Python Collections (08:26)
4. Getting Started with Processing Python Lists (03:17)
5. Overview of Lambda Functions in Python (06:30)
6. Usage of Lambda Functions (03:31)
7. Filter Data in Python Lists using filter and lambda (05:43)
8. Get unique values from list using map and set (05:23)
9. Sort Python lists using key (07:42)
10. Overview of JSON Strings and Files (03:54)
11. Read JSON Strings to Python dicts or lists (06:34)
12. Read JSON Schemas from file to Python dicts (06:18)
13. Overview of Processing JSON Data using Python (03:04)
14. Extract Details from Complex JSON Arrays using Python (03:39)
15. Sort Data in JSON Arrays using Python (04:17)
16. Create Function to get Column Details from Schemas JSON File (05:28)
13. Data Processing using Pandas Dataframe APIs
1. Overview of Pandas for Data Processing (04:43)
2. Overview of Reading CSV Data using Pandas (04:38)
3. Read Data from CSV Files to Pandas Dataframes (04:43)
4. Filter Data in Pandas Dataframe using query (07:06)
5. Get Count by Status using Pandas Dataframe APIs (05:10)
6. Get count by Month and Status using Pandas Dataframe APIs (04:32)
7. Create Dataframes using dynamic column list on CSV Data (04:47)
8. Performing Inner Join between Pandas Dataframes (04:21)
9. Perform Aggregations on Join results (05:00)
10. Sort Data in Pandas Dataframes (06:45)
11. Overview of Writing Pandas Dataframes to Files (03:10)
12. Write Pandas Dataframes to JSON Files (07:16)
14. Project 1 - File Format Converter using Python
2. Get File Names to be processed using glob (05:53)
3. Get Column Names using Schemas File (04:03)
4. Get Data Set Names from File Names or Paths using regular expressions (05:34)
5. Read CSV Data into Pandas Dataframe with Schema Dynamically (05:04)
6. Generate File Paths for Target JSON Files Dynamically (04:01)
7. Recap of Writing Pandas Dataframe to JSON File (02:24)
8. Write Pandas Dataframe to JSON Files (04:15)
9. Modularize File Format Converter for Dataset (05:18)
10. Wrapper to Process all Data Sets (06:37)
11. Setup Project for File Format Converter using Python (04:17)
12. Install Dependencies for the Python Project using pip (03:30)
13. Add Core Logic to Python Application (04:10)
14. Overview of Run-time Arguments and Environment Variables (02:03)
15. Using Run Time Arguments in Python Applications (05:32)
16. Overview of Environment Variables (04:21)
17. Setting Environment Variables on Windows or Mac or Linux (04:53)
18. Use Environment Variables in Python Applications (05:34)
19. Use Environment Variables in File Format Converter (09:15)
20. Pass JSON Array as argument to Python Applications (05:51)
21. Pass Data Sets as Run Time Arguments to File Format Converter (09:20)
22. Exception Handling in Python Applications (03:35)
23. Raising Exceptions in Python Applications (04:07)
24. Exception Handling in File Format Converter Application (04:38)
15. Project 2 - Files to Database Loader
2. Install Python Dependencies for Pandas and Database Integration (02:11)
3. Run Queries from Notebook using SQL Magic (07:58)
4. Validate Pandas and SQL Integration (05:21)
5. Write CSV Data from File to Database Table (05:43)
6. Write CSV Data from Files to Database Tables in Chunks (07:05)
7. Overview of Deploying File to DB Loader Project (09:12)
16. Troubleshooting and Debugging Python Issues
1. Introduction to Troubleshooting and Debugging Python issues (02:48)
2. Guidelines for Troubleshooting and Debugging Python related Issues (01:47)
3. Overview of Database Connectivity using Python Applications (07:32)
4. Overview of Database Connectivity using Python (04:15)
5. Troubleshoot Network Connectivity to the Database Server using telnet (03:44)
6. Troubleshoot Module Related issues for Database Connectivity using Python (05:08)
7. Troubleshoot Credentials Related issues for Database Connectivity using Python (03:08)
8. Overview of Python process to run Python Applications (02:40)
9. Troubleshooting Compilation Errors in Python (04:55)
10. Troubleshooting Run Time Errors in Python (03:58)
11. Overview of Software Development Life Cycle (03:31)
12. Overview of Unit Testing or Validation of Applications (03:11)
13. Overview of Debugging VS Code Notebooks using Debug Feature (09:14)
14. Debug VS Code Notebooks using Debug Feature (09:46)
15. Getting Started with Debugging of Python Programs using VS Code (02:27)
16. Recap of running File Format Converter application (06:26)
17. Debug Python Application using VS Code with breakpoints (06:46)
18. Managing Breakpoints for Debugging in VS Code (02:56)
19. Conclusion to Troubleshooting and Debugging Python Issues (02:02)
17. Performance Tuning of Python Applications
1. Introduction to Performance of Python Applications (07:00)
2. Setup Database Loader Python Application (05:42)
3. Ensure Postgres Database is setup for file to db loader Python Application (09:22)
4. Cleanup the tables to run file to db loader application (03:08)
5. Run and Validate File to DB Loader Application (06:07)
6. Fix the error message in file to db loader application (02:29)
7. Overview of Execution of file to db loader application (07:16)
8. Performance Tuning using Chunksize in Pandas (06:50)
9. Review Pandas Data Frame API to load data into the target table (08:30)
10. Overview of multi or batch insert into Database Tables (04:53)
11. Develop application for multiprocessing (03:06)
12. Getting Started with Multiprocessing using Python (05:14)
13. Invoking User Defined Functions using multiprocessing in Python (06:21)
14. Refactor File to Database Loader Application for Multiprocessing (04:38)
15. Add Parallel Processing to file to db loader Python Application (04:41)
16. Validate File to DB Loader Application with Multiprocessing (07:07)
17. Understanding the concept of Multiprocessing in Python (08:01)
18. Performance Tuning Scenarios of Python Applications (05:11)
18. Getting Started with GCP
1. Introduction to Getting Started with GCP (01:00)
2. Pre-requisite Skills to Sign up for course on GCP Data Analytics (02:03)
3. Overview of Cloud Platforms (04:00)
4. Overview of Google Cloud Platform or GCP (03:20)
5. Overview of Signing for GCP Account (01:42)
6. Create New Google Account using Non Gmail Id (02:17)
7. Sign up for GCP using Google Account (03:16)
8. Overview of GCP Credits (03:36)
9. Overview of GCP Project and Billing (02:11)
10. Overview of Google Cloud Shell (03:28)
11. Install Google Cloud SDK on Windows (04:38)
12. Initialize gcloud CLI using GCP Project (03:25)
13. Reinitialize Google Cloud Shell with Project id (03:04)
14. Overview of Analytics Services on GCP (02:20)
15. Conclusion to Get Started with GCP for Data Engineering (01:05)
19. Overview of Big Data and Data Lakes
1. Different Types of Databases (01:59)
2. Usecases for Different Types of Databases (04:01)
3. Technologies for Different Types of Databases (03:41)
4. Volumes for Different Types of Databases (03:29)
5. Overview of Big Data (04:19)
6. Evolution of Big Data Technologies (02:33)
7. Overview of Data Lake using Hadoop eco system (06:14)
8. Limitations of Hadoop eco system (05:31)
9. Overview of Modern Data Lakes on Cloud (02:36)
10. Implementation of Modern Data Lakes on Cloud (03:23)
11. Advantages of Modern Data Lakes on Cloud (04:11)
20. Overview of Spark and Spark Architecture
1. Overview of Data Processing (05:42)
2. Overview of Data Processing Libraries (01:56)
3. Setup Environment to explore Pandas, Dask and Pyspark (04:19)
4. Code Examples of Pandas, Dask and Pyspark (06:21)
5. Differences between Pandas, Dask and Pyspark (06:13)
6. Overview of Distributed Computing (05:16)
7. Overview of Official Documentation of Apache Spark (03:37)
8. Overview of Spark Key Features and Platforms (05:01)
9. Overview of Spark Infrastructure (03:58)
10. Overview of Spark Cluster using Databricks (06:14)
11. Overview of Executors in Spark Cluster (03:03)
12. Overview of Spark Glossary (01:44)
13. Understand Spark Key Terms (06:42)
21. Setup Databricks Environment using GCP
1. Overview of Databicks on GCP (01:08)
2. Signing up for Databricks on GCP (04:21)
3. Create Databricks Workspace on GCP (04:46)
4. Getting Started with Databricks Clusters on GCP (03:17)
5. Getting Started with Databricks Notebook (03:51)
6. Overview of Databricks on GCP (07:11)
7. High level architecture of Databricks (05:19)
8. Setup Databricks CLI on Mac or Windows (05:07)
9. Overview of Databricks CLI and other clients (05:20)
10. Configure Databricks CLI on Mac or Windows (05:18)
11. Troubleshoot issues to configure Databricks CLI (05:11)
12. Overview of Databricks CLI Commands (05:20)
13. Setup Data Repository for Data Sets (01:59)
14. Setup Data Sets in DBFS using Databricks CLI Commands (04:50)
22. Basic Transformations using Spark SQL
1. Process Data in DBFS using Databricks Spark SQL (05:03)
2. Getting Started with Spark SQL Example using Databricks (04:44)
3. Create Temporary Views using Spark SQL (06:34)
4. Exercise to create temporary views using Spark SQL (01:27)
5. Spark SQL Query to compute Daily Product Revenue (06:10)
6. Save Query Result to DBFS using Spark SQL (04:25)
7. Overview of Pyspark Examples on Databricks (01:04)
8. Process Schema Details in JSON using Pyspark (07:32)
9. Create Dataframe with Schema from JSON File using Pyspark (06:03)
10. Transform Data using Spark APIs (04:13)
11. Get Schema Details for all Data Sets using Pyspark (04:08)
12. Convert CSV to Parquet with Schema using Pyspark (05:01)
23. Create Delta Tables using Spark SQL
1. Introduction to Creating Delta Tables using Spark SQL (01:34)
2. Overview of Supported Providers for Spark Metastore Tables (02:07)
3. Create Database and Review the Details (05:58)
4. Create and Review Managed Spark Metastore Table using Delta Format (04:31)
5. Copy Data into Spark Metastore Managed Table (04:41)
6. Validate Data in Spark Metastore Managed Table (03:17)
7. Create and Review External Spark Metastore Table using Delta Format (04:08)
8. Insert Data into Spark Metastore External Table (03:13)
9. Validate Data in Spark Metastore External Table (03:00)
10. Overview of Spark Metastore (04:36)
11. Difference Between Managed and External Spark Metastore Tables (06:38)
12. Perform CRUD Operations on Delta Tables in Spark Metastore (06:14)
13. Using Merge to Update and Insert into Delta Tables in Spark Metastore (09:31)
14. Conclusion of Creating Delta Tables using Spark SQL (02:15)
24. Pre-Defined Functions in Spark SQL
1. Overview of Functions in Spark SQL (03:08)
2. Validate Functions in Spark SQL (04:01)
3. Overview of String Manipulation Functions in Spark SQL (01:54)
4. Case Conversion and Length of Strings using Spark SQL (03:51)
5. Extract Substring using substr in Spark SQL (05:19)
6. Extract Substrings from Delimited Strings using split in Spark SQL (04:33)
7. Trimming Characters or Strings using Spark SQL (09:32)
8. Padding Characters to Strings using Spark SQL (07:32)
9. Reverse and Concatenate Strings using Spark SQL (07:06)
10. Overview of Date Manipulation Functions in Spark SQL (01:37)
11. Overview of Standard Date and Timestamp in Spark SQL (03:27)
12. Date Arithmetic using Spark SQL Functions (04:40)
13. Overview of trunc and date_trunc in Spark SQL (05:36)
14. Extract Information from Date or Time using Spark SQL (08:23)
15. Convert Non Standard Dates or Timestamps to Standard Ones using Spark SQL (07:39)
16. Extract Information using Calendar Functions from Date or Timestamp using Spark (02:51)
17. Dealing with Unix Timestamp using Spark SQL (08:54)
18. Overview of Numeric Functions in Spark SQL (12:15)
19. Data Type Conversion using Spark SQL (06:15)
20. Overview of Handling Null Values using Spark SQL (03:51)
21. Replace Null Values with default values using nvl and coalesce in Spark SQL (07:15)
22. Conditional Logic on Null Values using nvl2 and case in Spark SQL (06:08)
23. Overview of Case and When in Spark SQL (03:07)
24. Using CASE and WHEN for conditional logic in Spark SQL (06:09)
25. Aggregate using CASE and WHEN in GROUP BY in Spark SQL (04:00)
26. Word Count Query using Pre-defined Functions in Spark SQL (06:34)
27. Exercises for Pre-defined functions in Spark SQL (02:02)
28. Solutions for Exercises 1 and 2 on Pre-defined Functions in Spark SQL (08:27)
29. Solutions for Exercises 3 and 4 on Pre-defined Functions in Spark SQL (05:49)
30. Solutions for Exercises 5 and 6 on Pre-defined Functions in Spark SQL (09:59)
31. Solutions for Exercises 7 and 8 on Pre-defined Functions in Spark SQL (10:12)
25. Setup Spark Metastore Tables for Basic Transformations
1. Introduction to Basic Transformations using Spark SQL (01:45)
2. Prepare Spark Metastore Tables for Basic Transformations (03:47)
3. Projecting Data using Spark SQL (06:01)
26. Filtering Data using Spark SQL Queries
1. Filtering Data using Equal Condition in Spark SQL (04:56)
2. Using IN, LIKE and BETWEEN in Spark SQL Queries (04:48)
3. Filter Data using Boolean AND in Spark SQL Queries (04:25)
4. Filter Data using Boolean OR in Spark SQL Queries (04:19)
5. Dealing with NULLS while Filtering Data in Spark SQL Queries (03:15)
27. Aggregations using Spark SQL Queries
1. Perform Total Aggregations using Spark SQL Queries (06:45)
2. Overview of Aggregations using GROUP BY in Spark SQL Queries (03:59)
3. GROUP BY Examples using Spark SQL Queries (04:45)
4. Order of Execution of Spark SQL Queries (06:07)
5. Filter Data based on Aggregate Results using HAVING in Spark SQL Queries (06:23)
28. Joins using Spark SQL Queries
1. Overview of Joins in Spark SQL Queries (04:20)
2. Inner Join using Spark SQL Queries (05:47)
3. Concepts Behind Inner Joins in Spark SQL (05:03)
4. Outer Joins using Spark SQL Queries (08:39)
5. Example - Inner Join along with GROUP BY using Spark SQL Queries (03:36)
6. Example - Outer Join along with GROUP BY using Spark SQL Queries (03:29)
7. Example - Filtering and Outer Joins along with GROUP BY in Spark SQL Queries (03:33)
8. Example - Filtering and Outer Joins along with GROUP BY in Spark SQL Queries (03:58)
29. Sorting using Spark SQL Queries
1. Sorting Data using Spark SQL Queries (04:01)
2. Dealing with Nulls while Sorting Data using Spark SQL Queries (03:28)
30. Copy Query Results into Spark Metastore Tables
1. Overview of Copying Query Results into Spark Metastore Tables (02:09)
2. Query to Compute Daily Revenue using Spark SQL (02:16)
3. Copy Query Results into Spark Metastore Tables using CTAS (04:25)
4. Copy Query Results into Spark Metastore Tables using INSERT (03:38)
5. Design Pipeline using CTAS and INSERT in Spark SQL (09:02)
6. Copy Query Results into Spark Meatstore Tables using MERGE (07:14)
31. Ranking using Spark SQL Windowing Functions
1. Ranking using Spark SQL Windowing Functions (01:31)
2. Create Temporary View for ranking using Spark SQL Windowing Functions (01:36)
3. Compute Global Rank using Spark SQL Windowing Functions (05:37)
4. Compute Ranks Per Key using Spark SQL Windowing Functions (03:55)
5. Difference Between rank and dense_rank (04:10)
6. Filter on Ranks using Spark SQL Windowing Functions (06:59)
32. Processing JSON like Data using Spark SQL
1. Overview of JSON (04:41)
2. Creating Spark Metastore Tables with Array Type Columns (04:01)
3. Dealing with Array Type Columns using Spark SQL Queries (04:42)
4. Creating Spark Metastore Tables with Struct Type Columns (03:03)
5. Projecting Data From Struct Type Fields in Spark SQL (03:54)
6. Creating Spark Metastore Tables with Array of Struct Column (03:48)
7. Dealing with Array of Struct Type Columns using Spark SQL Queries (04:30)
8. Overview of Important Functions to Process JSON Data in Spark SQL (03:32)
9. Generate Array Type Columns from Regular Columns in Spark SQL (04:35)
10. Generate Array of Struct Type Columns from Regular Columns in Spark SQL (03:53)
11. Processing Delimited Strings using Spark SQL Queries (04:08)
33. Getting Started with Pyspark Data Frame APIs
1. Overview of Pyspark Examples on Databricks (01:04)
2. Process Schema Details in JSON using Pyspark (07:32)
3. Create Dataframe with Schema from JSON File using Pyspark (06:03)
4. Transform Data using Spark APIs (04:13)
5. Get Schema Details for all Data Sets using Pyspark (04:08)
6. Convert CSV to Parquet with Schema using Pyspark (05:01)
34. Create Spark Data Frames using Pyspark Data Frame APIs
1. Create Spark Data Frame using Pyspark Data Frame APIs (04:26)
2. Introduction to Processing JSON like Data using Spark SQL (01:03)
3. Overview of Data Processing using Conventional loops (06:50)
4. Overview of Data Frame Concepts (05:43)
5. Advantages of Pyspark Data Frames (03:40)
6. Overview of Spark Data Frames and their Characteristics (05:13)
7. Projecting Data in Spark Data Frames using Select (03:57)
8. Using drop to drop columns from Spark Data Frame (02:28)
9. Applying functions on Spark Data Frame Columns (05:55)
10. Using withColumn to apply transformations on Spark Data Frames (03:19)
11. Overview of Writing Data in Data Frame to Delta Files (04:49)
35. Basic Transformations using Pyspark Data Frame APIs
1. Overview of Basic Transformations using Pyspark Data Frame APIs (03:00)
2. Overview of Row Level Transformations (05:15)
3. Apply Row Level transformations using Pyspark Data Frame APIs (07:27)
4. Filtering Data using Pyspark Data Frame APIs (04:39)
5. Filtering Data with Multiple Conditions using Pyspark Data Frame APIs (05:54)
6. Perform Aggregations by Key using Spark Data Frame APIs (04:31)
7. Perform Aggregations by Key using Spark Data Frame APIs (06:03)
8. Perform Aggregations by Key using Spark Data Frame APIs (03:57)
9. Sorting Data using Spark Data Frame APIs (05:20)
10. Composite Sorting using Spark Data Frame APIs (05:56)
11. Develop Spark SQL Queries for Sorting Data (02:50)
12. Review Data Set with Nulls for Sorting using Spark Data Frame APIs (05:26)
13. Dealing with Nulls while Sorting the Data in Spark Data Frames (04:09)
36. Joining Data using Spark Data Frame APIs
1. Introduction to Joining Data using Spark Data Frame APIs (01:41)
2. Create Data Frames to Join using Spark Data Frame APIs (05:04)
3. Review Syntax for join using Spark Data Frame APIs (02:44)
4. Inner Join using Spark Data Frame APIs (05:31)
5. Join and other Spark Data Frame APIs to process the data (06:57)
6. Analyze Data for outer joins using Spark Data Frame APIs (03:26)
7. Left Outer Join using Spark Data Frame APIs (06:08)
8. Right Outer Join using Spark Data Frame APIs (04:19)
9. Equivalent Spark SQL Queries for Joins (05:05)
37. Ranking using Pyspark Data Frame APIs
1. Introduction to Ranking using Spark Data Frame APIs (05:52)
2. Syntax for ranking using Spark Data Frame APIs (02:59)
3. Compute Global Ranks using Spark Data Frame APIs (05:02)
4. Filter Based on Global Ranks using Spark Data Frame APIs (04:19)
5. Compute Ranks per Partition using Spark Data Frame APIs (03:30)
6. Filter Based on Ranks Per Partition using Spark Data Frame APIs (03:46)
7. Difference Between rank and dense_rank (07:27)
38. Integration of Spark SQL and Pyspark Data Frame APIs
1. Introduction to Integration of Spark SQL and Pyspark Data Frame APIs (01:42)
2. Run Spark SQL Queries on Spark Data Frames (05:19)
3. Create Spark Metastore Tables using Data Frames (05:00)
4. Insert into Spark Metastore Tables using Data Frames (04:03)
5. Read Data from Spark Metastore Table to Data Frames (03:33)
6. Process Data in Spark Metastore Tables using Data Frame APIs (05:37)
7. Manage Spark Metastore Database Objects using Spark APIs (05:07)
39. ELT Data Pipelines using Databricks
1. Overview of Databricks Workflows (03:10)
2. Pass Arguments to Databricks Python Notebooks (07:36)
3. Pass Arguments to Databricks SQL Notebooks (03:16)
4. Create and Run First Databricks Job (07:31)
5. Run Databricks Jobs and Tasks with Parameters (05:40)
6. Create and Run Orchestrated Pipeline using Databricks Job (06:54)
7. Import ELT Data Pipeline Applications into Databricks Environment (02:56)
8. Spark SQL Application to Cleanup Database and Datasets (03:52)
9. Review File Format Converter Pyspark Code (05:11)
10. Review Databricks SQL Notebooks for Tables and Final Results (03:57)
11. Validate Applications for ELT Pipeline using Databricks (07:36)
12. Build ELT Pipeline using Databricks Job in Workflows (09:22)
13. Run and Review Execution details of ELT Data Pipeline using Databricks Job (05:00)
40. Performance Tuning of Spark - Catalyst Optimizer
1. Getting Started with Performance Tuning using Spark on Databricks (03:12)
2. Overview of Spark Catalyst Optimizer (04:16)
3. Review Explain Plan for Spark Dataframe logic using Spark UI (03:41)
4. Review Explain Plan for Spark SQL logic using Spark UI (03:04)
5. Generate Explain Plans on Spark Dataframes using explain function (03:48)
6. Generate Explain Plans on Spark SQL Queries using explain command (02:33)
7. Interpreting Explain Plan for Spark SQL Query (04:57)
8. Overview of Spark Architecture (03:27)
9. Understand Filter and Broadcast of Orders Data (05:31)
10. Understand Join and Aggregation for Daily Product Revenue (04:18)
41. Performance Tuning of Spark - Cluster Configuration
1. Introduction to Databricks Cluster Configuration (01:59)
2. Difference Between All Purpose and Jobs Clusters (05:17)
3. Setting up All Purpose Databricks Compute Clusters (04:46)
4. Understand the size of the data using dbutils (06:03)
5. Create Multinode Databricks Cluster with Auto Scaling (02:58)
6. Overview of Auto Scaling of Databricks Clusters (03:21)
7. Performance Tuning of Cluster using Auto Scaling (04:40)
8. Analyze Airlines Data using Spark SQL (02:58)
9. Setup Databricks Job Compute Clusters using Workflows (04:27)
10. Review Running Job Details using Spark UI (03:03)
11. Review Completed Job Details using Spark UI (03:20)
42. Performance Tuning while inferring schema from CSV or JSON files
1. Overview of Inferring Schema using CSV or JSON Files (01:45)
2. Steps to convert CSV or JSON Files to Parquet or Delta Files (03:50)
3. Overview of CSV or JSON Files (05:08)
4. Overview of overhead for inferring schema (05:47)
5. Performance Tuning to infer schema of Spark Dataframe (04:58)
43. Performance Tuning using Columnar File Format and Partitioning Strategy
1. Introduction to Performance Tuning while storing data in Data Lake (01:15)
2. Side effects of using CSV Files in Data Lake (04:28)
3. Review the side effects of using CSV Files in Data Lake (03:02)
4. Restructure CSV Data to Columnar Format using Pyspark (04:55)
5. Compute Size of restrucutured data using Parquet File Format (03:53)
6. Run Operations on Partitioned Parquet Data (04:21)
7. Overview of Performance Assessment of Spark Jobs (03:44)
8. Review Performance Details of Spark Operations on Parquet Files (07:54)
9. Overview of Columnar File Formats in Spark and Databricks (06:49)
10. Overview of Folder Structure for Partitioned Data in Spark (04:48)
11. Evaluate Requirements against Partition Pruning (06:32)
12. Solutions on Airlines Data for Performance Tuning using Partitioning (07:00)
13. Review Execution Details for Performance Tuning using Partitioning (10:17)
14. Get Airlines Data using Date Range without Partition Pruning (06:23)
15. Parameterize Spark SQL Solution for Partition Pruning (03:43)
16. Add Condition for Partition Pruning (04:41)
17. Redesign Partition Strategy to tune the performance (05:21)
18. Recap of Spark Performance Tuning Scenarios (04:52)
44. Setup Hadoop and Spark Cluster using Dataproc
1. Introduction to Setup Hadoop and Spark Cluster using Dataproc (03:18)
2. Overview of different Spark Platforms on Cloud (06:42)
3. Overview of Hadoop and Spark Cluster Types and Architecture (10:46)
4. Setup Single Node Hadoop and Spark Cluster using Dataproc (03:42)
5. Validate SSH Connectivity to the Dataproc Cluster (06:28)
6. Convert IP Address to Static for Dataproc Cluster (03:58)
7. Setup Project using VS Code Remote Development (15:46)
8. Setup Local Data Sets on Hadoop and Spark Cluster (02:18)
9. Getting Started with HDFS Commands to Manage Files (07:30)
10. Getting Started with Spark CLI using Python (04:52)
11. Getting Started with Spark CLI using Scala (03:59)
12. Getting Started with Spark CLI using SQL (05:00)
13. Stopping the Cluster and Understanding the costs (05:36)
14. Download VS Code Workspace and Delete Cluster (04:17)
45. Recap of important Linux Commands for Data Engineering
1. Introduction to Linux Commands and Scripts for Data Engineers (02:56)
2. Overview of SSH to connect to remote Servers (21:33)
3. Overview of Profile in Linux Shell (07:16)
4. Overview of Environment Variables in Linux (06:19)
5. Understanding PATH Environment Variable (18:13)
6. Creating Folders in Linux using mkdir (08:54)
7. Copy Files and Folders in Linux using cp command (09:39)
8. Move Files and Folders in Linux using mv command (06:31)
9. Delete Files and Folders in Linux using rm command (08:37)
10. Listing Files and Folders using ls command (10:11)
11. Searching for files using find command in Linux (06:15)
12. Standard Directories in Linux (08:44)
13. Troubleshooting issues in Linux using grep command (13:46)
14. Overview of Shell Scripts (10:40)
15. Running and Debugging Shell Scripts with Arguments (09:26)
16. Overview of Hadoop and Spark Executables (07:44)
46. Mastering Hadoop HDFS Commands and Concepts
1. Introduction to Mastering Hadoop HDFS Commands and Concepts (02:20)
2. Start the Hadoop and Spark Cluster using Dataproc (02:22)
3. Create Hadoop and Spark Cluster and Setup VS Code Workspace (07:12)
4. Overview of important HDFS Commands (02:51)
5. Create Folder and Copy Files into HDFS using Commands (08:19)
6. Programmatically Copy files into HDFS using Python (11:22)
7. Overview of Multinode Hadoop Cluster (12:19)
8. Review Important Properties of HDFS (06:20)
9. Review HDFS Properties on Dataproc Cluster using VS Code (04:05)
10. Overview of local storage of files (09:35)
11. Setup Data Sets to understand HDFS Concepts (05:51)
12. Overview of Distributed Storage of files in HDFS (06:39)
13. Determining Number of Blocks for each file (09:31)
14. Overview of Blocks related to files in HDFS (09:11)
15. Overview of Replication related to files in HDFS (08:22)
16. Physical Storage of HDFS File Blocks (06:53)
17. Overview of HDFS Namenode for HDFS File Metadata (08:40)
18. Recap of HDFS on Dataproc Cluster (11:32)
47. Build Hive Applications in Hadoop and Spark Clusters
1. Introduction to Building Hive Applications (02:11)
2. Getting Started with Hive (15:02)
3. Overview of Hive Architecture (08:38)
4. Overview of Integrating Hive Commands with Shell Scripts (07:16)
5. Run Hive Commands using Scripts (08:52)
6. Override Run time Hive Configuration Properties and Variables (11:09)
7. Getting Started Data Loader for NYSE Data using HIve (07:24)
8. Design NYSE Data Loader Application (05:52)
9. Create Partitioned Parquet Table for NYSE Data (11:30)
10. Populate Data into Partitioned NYSE table from Stage Table (10:03)
11. Process NYSE Data and load into partitioned table (05:45)
12. Develop Hive QL Script to Load NYSE Data (05:20)
13. Run Hive QL Commands using Script for NYSE Loader (07:33)
14. Redesign the Solution using HDFS to stage files (12:50)
15. Validate Hive Application to Convert NYSE Data (09:13)
16. Deploy Hive Application in HDFS (05:48)
17. Develop Shell Wrapper to run Hive Application (07:50)
18. Overview of Scheduling and Crontab (09:37)
19. Schedule Hive Applications using Cron (13:11)
20. Recap of Application Development Life Cycle using Hive (05:07)
48. Getting Started with Spark SQL on Hadoop and Spark Cluster
1. Getting Started with Spark SQL on Hadoop and Spark Cluster (01:45)
2. Getting Started with Data Sets and Spark SQL CLI (05:17)
3. Spark SQL Metastore Architecture (11:14)
4. Overview of Spark Metastore Warehouse Directory (05:14)
5. Launch Spark SQL CLI with Delta Lake Packages (04:58)
6. Populate Data into Delta Lake Tables using Spark SQL (14:31)
7. Run Individual Spark SQL Commands (10:06)
8. Run Spark SQL Scripts using Spark SQL CLI (08:14)
49. Build Real Time Applications using Spark SQL with Shell Wrapper
1. Introduction to Application Development Life Cycle of Spark SQL Applications (01:11)
2. Review the Requirements and Datasets for NYSE Data (05:33)
3. Design NYSE Converter Application using Spark SQL and Delta (04:22)
4. Copy Files into HDFS for NYSE Converter (04:28)
5. Create External Stage Table for NYSE CSV Files (07:30)
6. Create Target Table for NYSE Data using Delta Format (08:44)
7. Populate Data for Additional Years into Delta NYSE Table (15:30)
8. Develop Spark SQL Application for NYSE Data Conversion (05:15)
9. Validate Spark SQL Application for NYSE Data Conversion (06:56)
10. Develop Shell Wrapper for Spark SQL Application (06:54)
50. Getting Started with Pyspark on Hadoop and Spark Cluster
1. Introduction to Getting Started with Pyspark (01:31)
2. Launching or Getting Started Pyspark CLI (09:03)
3. Overview of Spark Properties Files (04:56)
4. Review Data Sets to explore Pyspark APIs (04:34)
5. Getting Started with Pyspark for Data Processing (06:14)
6. Read Orders and Order Items Data into Spark Data Frames (05:28)
7. Process Data using pyspark Dataframe APIs (10:06)
8. Overview of Spark Submit Command (04:09)
51. Submitting Python based Spark Applications
1. Introduction to Submitting Python based Spark Applications (01:03)
2. Develop Pyspark Application for Daily Revenue (05:52)
3. Run Spark Application using spark-submit (06:05)
4. Specify Paths using Environment Variables in Spark Applications (06:26)
5. Run Spark Application with Environment Variables in Client Mode (04:07)
6. Run Spark Application with Environment Variables in Cluster Mode (05:08)
7. Review Spark Application Details using Spark UI (03:11)
8. Review YARN Logs for Spark Applications in Cluster Mode (06:51)
9. Overview of Execution Process of Spark Applications (07:27)
10. Deep Dive into Spark Deploy Modes (13:29)
11. Submit Spark Applications with dependencies as packages (12:10)
12. Submit Spark Applications with dependencies as jars (16:19)
13. Develop Shell Wrappers to submit Spark Applications (07:49)
52. Logging in Python based Spark Applications
1. Introduction to Logging in Python baesd Spark Applications (01:06)
2. Run Application without logging (06:33)
3. Overview of Logging Concepts such as Log Levels (04:50)
4. Getting Started with logging using Python (06:52)
5. Changing the Log Message Format using logging (06:28)
6. Add logging to Python based Spark Applications (03:56)
7. Validate Logging of Spark Application using Client Mode (03:49)
8. Validate Logging of Spark Application using Cluster Mode (05:16)
53. Performance Tuning of Spark Applications on Hadoop and Spark
1. Introduction to Performance Tuning of Spark Applications on Hadoop and Spark Clu (02:03)
2. Delete Single Node Hadoop and Spark Cluster using Dataproc (02:12)
3. Increase GCP VM Quotas for Mutlinode Hadoop and Spark Cluster (03:42)
4. Review Quotas to setup Multinode Hadoop and Spark Cluster (02:17)
5. Setup Multinode Hadoop and Spark Cluster using GCP Dataproc (03:47)
6. Setup SSH Connectivity and VS Code Workspace using Master Node (07:19)
7. Review Multi Node Hadoop and Spark Clusters using Web Interfaces (05:14)
8. Overview of Multinode Hadoop and Spark Cluster Topology (12:08)
9. Computing Overall Capacity of Multinode Hadoop and Spark Clusters (06:06)
10. Determine Overall YARN Capacity (05:24)
11. Overview of Spark History Server UI (04:53)
12. Generate Test Data for Spark Performance Tuning (04:23)
13. Develop Word Count Application using Spark (11:57)
14. Develop and Validate Shell Script for Word Count (05:05)
15. Overview of Jobs related to Spark Applications using Spark UI (08:14)
16. Review Environment Properties and Disabling Dynamic Allocation (07:42)
17. Overriding Spark Executor Instances to tune the performance (07:14)
18. Determine Maximum Capacity to submit a Spark Application (03:35)
19. Overview of Adaptive Query Execution (07:03)
20. Overview of Shuffling - Part 1 (08:51)
21. Overview of Shuffling - Part 2 (12:21)
22. Overview of Spark Application (04:45)
23. Overview of Lazy Evaluation (08:40)
24. Review the code of Word Count Application (03:22)
25. Run Spark Application with out Adaptive Query Execution (10:12)
26. Run Spark Application using Adaptive Query Execution (06:20)
27. Overview of Spark Dynamic Allocation (09:27)
28. Demo on Spark Dynamic Allocation (10:41)
29. Running Spark Application using Dynamic Allocation (06:46)
30. Overview of number of Spark Partitions (08:50)
31. Delete Multinode Hadoop and Spark Cluster (03:09)
Файлы примеров: не предусмотрены
Формат видео: MP4
Видео: AVC, 1280x720, 16:9, 30fps, 972kbps
Аудио: AAC, 44.1kHz, 62.8kbps, stereo
