Data Engineering Course
Design, build, orchestrate, and deploy production-grade data pipelines, lakes, and warehouses on AWS with Airflow and PySpark.
What you'll master in Level 1
Design, build, orchestrate, and deploy production-grade data pipelines, lakes, and warehouses on AWS with Airflow and PySpark.
Training Objectives & Core Deliverables
Advanced SQL and query performance tuning with indexes and execution plans
Star-schema data warehouse modeling with fact, dimension tables and SCD patterns
Robust Python data pipelines with logging, error handling and incremental loading
Batch pipeline orchestration, scheduling, and error recovery with Apache Airflow
End-to-end cloud data platform deployment using AWS S3, Glue, Athena, Lambda and Redshift
Distributed big data processing, partitioning, and parquet storage with PySpark
Containerized pipeline delivery with Docker and modular transformation testing with dbt
Comprehensive end-to-end multi-tier pipeline capstone project presentation
Day-by-Day Training Schedule & Live Labs
What is data engineering?
Set up all tools. Draw the data flow of a real business case, for example loan applications from a branch app to a management report, naming sources, storage and outputs.
Linux command line and Git essentials
Create a project repository with a clean folder structure and push it to GitHub.
Python for data pipelines
Write a script that pulls data from a public REST API, cleans it with Pandas, logs each step, and saves it as a Parquet file.
Advanced SQL for data engineers
Write analytical queries on a retail transactions dataset: rank top products, calculate running totals and find month-over-month growth.
Query tuning and performance
Take a slow query on a 1-million-row table, read the query plan, add the right index, and measure the speed-up.
Data modelling: relational to dimensional
Design a star schema for an e-commerce or retail business: define the grain, build one fact table and three dimension tables.
Data modelling in practice (Mini Project 1)
Mini Project 1: build the star-schema warehouse from Day 6 in PostgreSQL, load sample data, and answer five business questions with SQL.
Building reliable batch pipelines with Python
Build a Python batch pipeline that loads CSV and API data into PostgreSQL staging tables.
Reliable pipelines and data quality
Add data quality checks and incremental loading to the Day 8 pipeline.
Apache Airflow basics
Create a first DAG that runs a Python task on a daily schedule.
Airflow in practice (Mini Project 2)
Mini Project 2: an automated daily pipeline in Airflow that extracts, validates, loads and notifies.
AWS foundations and S3
Set up a budget alert and an IAM user, then upload data to S3 using the console and Python (boto3).
Data lake with Glue Catalog and Athena
Convert CSV into partitioned Parquet and query it with Athena. Compare data scanned and cost.
Serverless processing: Lambda and Glue
Build a Lambda that validates files when they land in S3, and a Glue job that transforms them.
Data warehousing with Redshift
Load curated data from S3 into Redshift, run analytical queries, then clean up resources.
AWS pipeline project (Mini Project 3)
Mini Project 3: an S3 to Glue to Athena or Redshift pipeline for a business dataset.
Introduction to PySpark and distributed data
Set up PySpark in a notebook, load a large CSV, and run basic transformations and counts.
DataFrame transformations and Spark SQL
Clean a messy dataset with PySpark, join two datasets, and run analytical queries using both DataFrame API and Spark SQL.
Performance, partitioning and the Spark UI
Re-partition a dataset, save it as partitioned Parquet, look at the output folders, and compare query times before and after.
PySpark pipeline project (Mini Project 4)
Mini Project 4: build a PySpark job that reads raw data from S3 (or local), cleans and aggregates it, and writes partitioned Parquet files.
Docker and dbt essentials
Build a Dockerfile for a Python pipeline, then build one dbt model with a test on the warehouse from Mini Project 1.
Capstone kick-off and design
Learners design the pipeline architecture and data model with mentor support.
Capstone build: ingest and transform
Learners build the ingestion and transformation steps with mentor support.
Capstone build: quality, scheduling and documentation
Learners finish the pipeline. Remaining work is completed as practice.
Presentations, portfolio and career prep
Present the capstone and receive feedback.
Detailed Module-wise Curriculum
Navigating files, permissions, grep, pipes and redirection
Scheduling with cron basics
Git essentials: init, commit, branch, push and GitHub
Python virtual environments and requirements files
Reading and writing CSV, JSON and Parquet files
Calling REST APIs with the requests library
Pandas for cleaning and transformation
Logging, error handling, config files and environment variables
Review of joins, aggregations and subqueries
Common table expressions (CTEs) for readable code
Window functions: ROW_NUMBER, RANK, DENSE_RANK
LAG, LEAD and moving averages for trend analysis
How PostgreSQL runs a query: reading query plans with EXPLAIN and EXPLAIN ANALYZE
Indexes: B-tree basics, composite indexes and when to avoid them
Common reasons for slow queries: full table scans, bad joins, missing indexes
Transactions, locks and ACID properties in simple terms
Relational design: entities, relationships and normalisation (1NF, 2NF, 3NF)
Dimensional modelling: star schema vs snowflake schema
Fact tables (events) and dimension tables (context)
Surrogate keys, grain of a fact table and slowly changing dimensions (SCD 1 and 2)
Reviewing learner data models
Writing DDL to create tables with keys and constraints
Loading sample data and running business queries
Code review and presentation
ETL vs ELT: what each means and when to use which
Source to target pipeline structure
Extracting from files, databases and APIs
Layered design: raw, clean and curated
Full loads vs incremental loads
Idempotency: safe reruns without duplicates
Validation checks: nulls, duplicates, ranges, row counts
Retries, error handling and alerting basics
What an orchestrator does and why it is needed
Airflow architecture: scheduler, workers, metadata database
DAGs, tasks, operators and schedules
Using a ready-made Docker setup
Task dependencies and branching
Variables and connections
Retries, failure alerts and backfills
Cloud basics: regions, services, shared responsibility
IAM: users, roles, policies and least privilege
S3: buckets, storage classes and lifecycle rules
Cost control: free tier, budgets and alerts
The data lake concept on S3
CSV vs Parquet, compression and partitioning
Glue Data Catalog and crawlers
Querying S3 data with Athena SQL
Lambda functions and event triggers
AWS Glue ETL jobs (PySpark) overview
Monitoring with CloudWatch logs
Redshift architecture and columnar storage
Distribution and sort keys (concepts)
Loading from S3 with COPY
Cost awareness, and shutting down resources to avoid charges
Designing the pipeline architecture
Orchestration options: Airflow and Step Functions overview
Cost clean-up checklist
When pandas is not enough: why distributed processing matters
Spark architecture: driver, executors, tasks and partitions
SparkSession, DataFrames and lazy evaluation
Reading and writing files: CSV, JSON and Parquet
Filtering, selecting, renaming and creating columns (withColumn)
Aggregations and groupBy
Joins: inner, left, outer and broadcast joins
Spark SQL: register tables and write SQL queries
Narrow vs wide transformations: why shuffles are slow
Partitioning data by date or category when saving
Reading the Spark UI: jobs, stages, tasks and execution time
Running Spark on the cloud: AWS EMR and Databricks concepts
Structuring a PySpark project: scripts, config and functions
Handling missing data and data quality in Spark
Saving curated outputs for analytics
Docker basics: images, containers, Dockerfile and Docker Compose
Why data teams use Docker: consistent environments and easy deployment
What dbt is: transformation inside the warehouse using SQL and Jinja
dbt project structure: models, tests, sources and documentation
Project brief and raw dataset shared by the instructor
Architecture design and data model
Project plan and folder structure
Build ingestion from files and an API
Transformations in Python, SQL or Spark
Loading into the warehouse or data lake
Data quality checks and error handling
Scheduling the pipeline with Airflow
README and architecture diagram
Capstone demo: architecture, code walkthrough and quality checks
GitHub portfolio and resume points for data engineers
Interview preparation: SQL, Python, Spark, AWS and pipeline design questions
Next steps: AWS data certification, streaming, data architecture
Certificate of Technical Competency — Data Engineering Course
Complete all live hands-on labs to earn this verifiable competency badge. This credential unlocks progression to Data Analyst - Advanced with Python.
Enroll in Data Engineering Course
Reserve your seat for upcoming live batches, syllabus consultation & fee structure.Frequently Asked Questions
Frequently Asked Questions: Data Engineering Course
Find clear answers to common questions about our live interactive classes, cohort schedules, project mentoring, and career assistance.
Beginners should start with 'Data Analyst - Job Ready' to master business metrics, foundational SQL queries, and Power BI dashboards. Next, advance to 'Data Analyst - Advanced with Python' for automated analytics, and finally graduate to the 50-hour 'Data Engineering Course'.
No. Cloud basics are taught from the ground up starting with AWS IAM and S3, with step-by-step mentor guidance in every live session.
Most tasks use the AWS free tier. We configure budget alerts, and every cloud lab ends with a clean-up step to avoid unexpected bills. Learners should expect minimal or zero extra cost if instructions are followed.
Yes. All tools work on Windows, macOS, and Linux. Windows users will install WSL2 (Windows Subsystem for Linux) with mentor support, which makes the developer experience identical to Linux.
Every session is recorded in HD and shared within 24 hours, along with code repositories and daily lab tasks. You can catch up and ask questions in the next session or on the batch discussion channel.
You will finish with production-grade portfolio projects documented and pushed to your personal GitHub repository, verifiable certificates of completion, a professionally reviewed resume, and rigorous technical interview preparation.
Ready to Start Your Tech Journey?
Connect with our academic training counsellors. We analyze your background, career goals, and recommend the best live online track for your success.
