Interactive Mentor-Led Cohorts

(02) 8902 0900contact@aumtrend.comWhatsApp / Viber
AUM Training
HomeCourses
Who We Serve
About UsContact
LMS Portal
DATA ENGINEERING
50 Hours (25 Days × 2 Hours)
View Full Curriculum

Data Engineering Course

Design, build, orchestrate, and deploy production-grade data pipelines, lakes, and warehouses on AWS with Airflow and PySpark.

Live Interactive Syllabus
50 Hours (25 Days × 2 Hours)
•
Instructor-Led Online
•
25 Hands-on Coding Labs
What you'll master in Level 1
Target Competency Outcome

Design, build, orchestrate, and deploy production-grade data pipelines, lakes, and warehouses on AWS with Airflow and PySpark.

Training Objectives & Core Deliverables

Advanced SQL and query performance tuning with indexes and execution plans

Star-schema data warehouse modeling with fact, dimension tables and SCD patterns

Robust Python data pipelines with logging, error handling and incremental loading

Batch pipeline orchestration, scheduling, and error recovery with Apache Airflow

End-to-end cloud data platform deployment using AWS S3, Glue, Athena, Lambda and Redshift

Distributed big data processing, partitioning, and parquet storage with PySpark

Containerized pipeline delivery with Docker and modular transformation testing with dbt

Comprehensive end-to-end multi-tier pipeline capstone project presentation

Day-by-Day Training Schedule & Live Labs
25 Live Sessions • 100% Hands-on
Day 1
What is data engineering?
Live Session
Topics Explored:
Data engineer vs data analyst vs data scientist
The data lifecycle: source, ingest, store, transform, serve
Batch vs streaming data, OLTP vs OLAP
Data lake vs data warehouse vs lakehouse
Tool setup: Python environment, VS Code, Git, PostgreSQL, Docker Desktop, AWS free-tier account
Hands-on Lab Exercise

Set up all tools. Draw the data flow of a real business case, for example loan applications from a branch app to a management report, naming sources, storage and outputs.

Day 2
Linux command line and Git essentials
Live Session
Topics Explored:
Navigating files, permissions, grep, pipes and redirection
Scheduling with cron basics
Git essentials: init, commit, branch, push and GitHub
Python virtual environments and requirements files
Hands-on Lab Exercise

Create a project repository with a clean folder structure and push it to GitHub.

Day 3
Python for data pipelines
Live Session
Topics Explored:
Reading and writing CSV, JSON and Parquet files
Calling REST APIs with the requests library
Pandas for cleaning and transformation
Logging, error handling, config files and environment variables
Hands-on Lab Exercise

Write a script that pulls data from a public REST API, cleans it with Pandas, logs each step, and saves it as a Parquet file.

Day 4
Advanced SQL for data engineers
Live Session
Topics Explored:
Review of joins, aggregations and subqueries
Common table expressions (CTEs) for readable code
Window functions: ROW_NUMBER, RANK, DENSE_RANK
LAG, LEAD and moving averages for trend analysis
Hands-on Lab Exercise

Write analytical queries on a retail transactions dataset: rank top products, calculate running totals and find month-over-month growth.

Day 5
Query tuning and performance
Live Session
Topics Explored:
How PostgreSQL runs a query: reading query plans with EXPLAIN and EXPLAIN ANALYZE
Indexes: B-tree basics, composite indexes and when to avoid them
Common reasons for slow queries: full table scans, bad joins, missing indexes
Transactions, locks and ACID properties in simple terms
Hands-on Lab Exercise

Take a slow query on a 1-million-row table, read the query plan, add the right index, and measure the speed-up.

Day 6
Data modelling: relational to dimensional
Live Session
Topics Explored:
Relational design: entities, relationships and normalisation (1NF, 2NF, 3NF)
Dimensional modelling: star schema vs snowflake schema
Fact tables (events) and dimension tables (context)
Surrogate keys, grain of a fact table and slowly changing dimensions (SCD 1 and 2)
Hands-on Lab Exercise

Design a star schema for an e-commerce or retail business: define the grain, build one fact table and three dimension tables.

Day 7
Data modelling in practice (Mini Project 1)
Live Session
Topics Explored:
Reviewing learner data models
Writing DDL to create tables with keys and constraints
Loading sample data and running business queries
Code review and presentation
Hands-on Lab Exercise

Mini Project 1: build the star-schema warehouse from Day 6 in PostgreSQL, load sample data, and answer five business questions with SQL.

Day 8
Building reliable batch pipelines with Python
Live Session
Topics Explored:
ETL vs ELT: what each means and when to use which
Source to target pipeline structure
Extracting from files, databases and APIs
Layered design: raw, clean and curated
Hands-on Lab Exercise

Build a Python batch pipeline that loads CSV and API data into PostgreSQL staging tables.

Day 9
Reliable pipelines and data quality
Live Session
Topics Explored:
Full loads vs incremental loads
Idempotency: safe reruns without duplicates
Validation checks: nulls, duplicates, ranges, row counts
Retries, error handling and alerting basics
Hands-on Lab Exercise

Add data quality checks and incremental loading to the Day 8 pipeline.

Day 10
Apache Airflow basics
Live Session
Topics Explored:
What an orchestrator does and why it is needed
Airflow architecture: scheduler, workers, metadata database
DAGs, tasks, operators and schedules
Using a ready-made Docker setup
Hands-on Lab Exercise

Create a first DAG that runs a Python task on a daily schedule.

Day 11
Airflow in practice (Mini Project 2)
Live Session
Topics Explored:
Task dependencies and branching
Variables and connections
Retries, failure alerts and backfills
Hands-on Lab Exercise

Mini Project 2: an automated daily pipeline in Airflow that extracts, validates, loads and notifies.

Day 12
AWS foundations and S3
Live Session
Topics Explored:
Cloud basics: regions, services, shared responsibility
IAM: users, roles, policies and least privilege
S3: buckets, storage classes and lifecycle rules
Cost control: free tier, budgets and alerts
Hands-on Lab Exercise

Set up a budget alert and an IAM user, then upload data to S3 using the console and Python (boto3).

Day 13
Data lake with Glue Catalog and Athena
Live Session
Topics Explored:
The data lake concept on S3
CSV vs Parquet, compression and partitioning
Glue Data Catalog and crawlers
Querying S3 data with Athena SQL
Hands-on Lab Exercise

Convert CSV into partitioned Parquet and query it with Athena. Compare data scanned and cost.

Day 14
Serverless processing: Lambda and Glue
Live Session
Topics Explored:
Lambda functions and event triggers
AWS Glue ETL jobs (PySpark) overview
Monitoring with CloudWatch logs
Hands-on Lab Exercise

Build a Lambda that validates files when they land in S3, and a Glue job that transforms them.

Day 15
Data warehousing with Redshift
Live Session
Topics Explored:
Redshift architecture and columnar storage
Distribution and sort keys (concepts)
Loading from S3 with COPY
Cost awareness, and shutting down resources to avoid charges
Hands-on Lab Exercise

Load curated data from S3 into Redshift, run analytical queries, then clean up resources.

Day 16
AWS pipeline project (Mini Project 3)
Live Session
Topics Explored:
Designing the pipeline architecture
Orchestration options: Airflow and Step Functions overview
Cost clean-up checklist
Hands-on Lab Exercise

Mini Project 3: an S3 to Glue to Athena or Redshift pipeline for a business dataset.

Day 17
Introduction to PySpark and distributed data
Live Session
Topics Explored:
When pandas is not enough: why distributed processing matters
Spark architecture: driver, executors, tasks and partitions
SparkSession, DataFrames and lazy evaluation
Reading and writing files: CSV, JSON and Parquet
Hands-on Lab Exercise

Set up PySpark in a notebook, load a large CSV, and run basic transformations and counts.

Day 18
DataFrame transformations and Spark SQL
Live Session
Topics Explored:
Filtering, selecting, renaming and creating columns (withColumn)
Aggregations and groupBy
Joins: inner, left, outer and broadcast joins
Spark SQL: register tables and write SQL queries
Hands-on Lab Exercise

Clean a messy dataset with PySpark, join two datasets, and run analytical queries using both DataFrame API and Spark SQL.

Day 19
Performance, partitioning and the Spark UI
Live Session
Topics Explored:
Narrow vs wide transformations: why shuffles are slow
Partitioning data by date or category when saving
Reading the Spark UI: jobs, stages, tasks and execution time
Running Spark on the cloud: AWS EMR and Databricks concepts
Hands-on Lab Exercise

Re-partition a dataset, save it as partitioned Parquet, look at the output folders, and compare query times before and after.

Day 20
PySpark pipeline project (Mini Project 4)
Live Session
Topics Explored:
Structuring a PySpark project: scripts, config and functions
Handling missing data and data quality in Spark
Saving curated outputs for analytics
Hands-on Lab Exercise

Mini Project 4: build a PySpark job that reads raw data from S3 (or local), cleans and aggregates it, and writes partitioned Parquet files.

Day 21
Docker and dbt essentials
Live Session
Topics Explored:
Docker basics: images, containers, Dockerfile and Docker Compose
Why data teams use Docker: consistent environments and easy deployment
What dbt is: transformation inside the warehouse using SQL and Jinja
dbt project structure: models, tests, sources and documentation
Hands-on Lab Exercise

Build a Dockerfile for a Python pipeline, then build one dbt model with a test on the warehouse from Mini Project 1.

Day 22
Capstone kick-off and design
Live Session
Topics Explored:
Project brief and raw dataset shared by the instructor
Architecture design and data model
Project plan and folder structure
Hands-on Lab Exercise

Learners design the pipeline architecture and data model with mentor support.

Day 23
Capstone build: ingest and transform
Live Session
Topics Explored:
Build ingestion from files and an API
Transformations in Python, SQL or Spark
Loading into the warehouse or data lake
Hands-on Lab Exercise

Learners build the ingestion and transformation steps with mentor support.

Day 24
Capstone build: quality, scheduling and documentation
Live Session
Topics Explored:
Data quality checks and error handling
Scheduling the pipeline with Airflow
README and architecture diagram
Hands-on Lab Exercise

Learners finish the pipeline. Remaining work is completed as practice.

Day 25
Presentations, portfolio and career prep
Live Session
Topics Explored:
Capstone demo: architecture, code walkthrough and quality checks
GitHub portfolio and resume points for data engineers
Interview preparation: SQL, Python, Spark, AWS and pipeline design questions
Next steps: AWS data certification, streaming, data architecture
Hands-on Lab Exercise

Present the capstone and receive feedback.

Detailed Module-wise Curriculum
25 Modules

Data engineer vs data analyst vs data scientist

The data lifecycle: source, ingest, store, transform, serve

Batch vs streaming data, OLTP vs OLAP

Data lake vs data warehouse vs lakehouse

Tool setup: Python environment, VS Code, Git, PostgreSQL, Docker Desktop, AWS free-tier account

Lab Practical: Set up all tools. Draw the data flow of a real business case, for example loan applications from a branch app to a management report, naming sources, storage and outputs.

Navigating files, permissions, grep, pipes and redirection

Scheduling with cron basics

Git essentials: init, commit, branch, push and GitHub

Python virtual environments and requirements files

Lab Practical: Create a project repository with a clean folder structure and push it to GitHub.

Reading and writing CSV, JSON and Parquet files

Calling REST APIs with the requests library

Pandas for cleaning and transformation

Logging, error handling, config files and environment variables

Lab Practical: Write a script that pulls data from a public REST API, cleans it with Pandas, logs each step, and saves it as a Parquet file.

Review of joins, aggregations and subqueries

Common table expressions (CTEs) for readable code

Window functions: ROW_NUMBER, RANK, DENSE_RANK

LAG, LEAD and moving averages for trend analysis

Lab Practical: Write analytical queries on a retail transactions dataset: rank top products, calculate running totals and find month-over-month growth.

How PostgreSQL runs a query: reading query plans with EXPLAIN and EXPLAIN ANALYZE

Indexes: B-tree basics, composite indexes and when to avoid them

Common reasons for slow queries: full table scans, bad joins, missing indexes

Transactions, locks and ACID properties in simple terms

Lab Practical: Take a slow query on a 1-million-row table, read the query plan, add the right index, and measure the speed-up.

Relational design: entities, relationships and normalisation (1NF, 2NF, 3NF)

Dimensional modelling: star schema vs snowflake schema

Fact tables (events) and dimension tables (context)

Surrogate keys, grain of a fact table and slowly changing dimensions (SCD 1 and 2)

Lab Practical: Design a star schema for an e-commerce or retail business: define the grain, build one fact table and three dimension tables.

Reviewing learner data models

Writing DDL to create tables with keys and constraints

Loading sample data and running business queries

Code review and presentation

Lab Practical: Mini Project 1: build the star-schema warehouse from Day 6 in PostgreSQL, load sample data, and answer five business questions with SQL.

ETL vs ELT: what each means and when to use which

Source to target pipeline structure

Extracting from files, databases and APIs

Layered design: raw, clean and curated

Lab Practical: Build a Python batch pipeline that loads CSV and API data into PostgreSQL staging tables.

Full loads vs incremental loads

Idempotency: safe reruns without duplicates

Validation checks: nulls, duplicates, ranges, row counts

Retries, error handling and alerting basics

Lab Practical: Add data quality checks and incremental loading to the Day 8 pipeline.

What an orchestrator does and why it is needed

Airflow architecture: scheduler, workers, metadata database

DAGs, tasks, operators and schedules

Using a ready-made Docker setup

Lab Practical: Create a first DAG that runs a Python task on a daily schedule.

Task dependencies and branching

Variables and connections

Retries, failure alerts and backfills

Lab Practical: Mini Project 2: an automated daily pipeline in Airflow that extracts, validates, loads and notifies.

Cloud basics: regions, services, shared responsibility

IAM: users, roles, policies and least privilege

S3: buckets, storage classes and lifecycle rules

Cost control: free tier, budgets and alerts

Lab Practical: Set up a budget alert and an IAM user, then upload data to S3 using the console and Python (boto3).

The data lake concept on S3

CSV vs Parquet, compression and partitioning

Glue Data Catalog and crawlers

Querying S3 data with Athena SQL

Lab Practical: Convert CSV into partitioned Parquet and query it with Athena. Compare data scanned and cost.

Lambda functions and event triggers

AWS Glue ETL jobs (PySpark) overview

Monitoring with CloudWatch logs

Lab Practical: Build a Lambda that validates files when they land in S3, and a Glue job that transforms them.

Redshift architecture and columnar storage

Distribution and sort keys (concepts)

Loading from S3 with COPY

Cost awareness, and shutting down resources to avoid charges

Lab Practical: Load curated data from S3 into Redshift, run analytical queries, then clean up resources.

Designing the pipeline architecture

Orchestration options: Airflow and Step Functions overview

Cost clean-up checklist

Lab Practical: Mini Project 3: an S3 to Glue to Athena or Redshift pipeline for a business dataset.

When pandas is not enough: why distributed processing matters

Spark architecture: driver, executors, tasks and partitions

SparkSession, DataFrames and lazy evaluation

Reading and writing files: CSV, JSON and Parquet

Lab Practical: Set up PySpark in a notebook, load a large CSV, and run basic transformations and counts.

Filtering, selecting, renaming and creating columns (withColumn)

Aggregations and groupBy

Joins: inner, left, outer and broadcast joins

Spark SQL: register tables and write SQL queries

Lab Practical: Clean a messy dataset with PySpark, join two datasets, and run analytical queries using both DataFrame API and Spark SQL.

Narrow vs wide transformations: why shuffles are slow

Partitioning data by date or category when saving

Reading the Spark UI: jobs, stages, tasks and execution time

Running Spark on the cloud: AWS EMR and Databricks concepts

Lab Practical: Re-partition a dataset, save it as partitioned Parquet, look at the output folders, and compare query times before and after.

Structuring a PySpark project: scripts, config and functions

Handling missing data and data quality in Spark

Saving curated outputs for analytics

Lab Practical: Mini Project 4: build a PySpark job that reads raw data from S3 (or local), cleans and aggregates it, and writes partitioned Parquet files.

Docker basics: images, containers, Dockerfile and Docker Compose

Why data teams use Docker: consistent environments and easy deployment

What dbt is: transformation inside the warehouse using SQL and Jinja

dbt project structure: models, tests, sources and documentation

Lab Practical: Build a Dockerfile for a Python pipeline, then build one dbt model with a test on the warehouse from Mini Project 1.

Project brief and raw dataset shared by the instructor

Architecture design and data model

Project plan and folder structure

Lab Practical: Learners design the pipeline architecture and data model with mentor support.

Build ingestion from files and an API

Transformations in Python, SQL or Spark

Loading into the warehouse or data lake

Lab Practical: Learners build the ingestion and transformation steps with mentor support.

Data quality checks and error handling

Scheduling the pipeline with Airflow

README and architecture diagram

Lab Practical: Learners finish the pipeline. Remaining work is completed as practice.

Capstone demo: architecture, code walkthrough and quality checks

GitHub portfolio and resume points for data engineers

Interview preparation: SQL, Python, Spark, AWS and pipeline design questions

Next steps: AWS data certification, streaming, data architecture

Lab Practical: Present the capstone and receive feedback.
DATA ENGINEERING COMPETENCY BADGE
Certificate of Technical Competency — Data Engineering Course

Complete all live hands-on labs to earn this verifiable competency badge. This credential unlocks progression to Data Analyst - Advanced with Python.

LIVE ONLINE COHORT
Admissions Open
Enroll in Data Engineering Course
Reserve your seat for upcoming live batches, syllabus consultation & fee structure.
Selected Course Program *
100% Privacy. Official Admissions Desk.

Frequently Asked Questions

Frequently Asked Questions: Data Engineering Course

Find clear answers to common questions about our live interactive classes, cohort schedules, project mentoring, and career assistance.

The program offers 3 comprehensive, career-focused courses: (1) Data Engineering Course (50 Hours), (2) Data Analyst - Advanced with Python (30 Hours), and (3) Data Analyst - Job Ready (30 Hours). Learners can enroll in individual courses or the complete data career master pathway.

Beginners should start with 'Data Analyst - Job Ready' to master business metrics, foundational SQL queries, and Power BI dashboards. Next, advance to 'Data Analyst - Advanced with Python' for automated analytics, and finally graduate to the 50-hour 'Data Engineering Course'.

No. Cloud basics are taught from the ground up starting with AWS IAM and S3, with step-by-step mentor guidance in every live session.

Most tasks use the AWS free tier. We configure budget alerts, and every cloud lab ends with a clean-up step to avoid unexpected bills. Learners should expect minimal or zero extra cost if instructions are followed.

Yes. All tools work on Windows, macOS, and Linux. Windows users will install WSL2 (Windows Subsystem for Linux) with mentor support, which makes the developer experience identical to Linux.

Every session is recorded in HD and shared within 24 hours, along with code repositories and daily lab tasks. You can catch up and ask questions in the next session or on the batch discussion channel.

You will finish with production-grade portfolio projects documented and pushed to your personal GitHub repository, verifiable certificates of completion, a professionally reviewed resume, and rigorous technical interview preparation.

Still have questions? We're here to help!

Speak directly with our technical advisors to discuss cohorts, curriculum, and career assistance.

FREE 1-ON-1 ACADEMIC GUIDANCE

Ready to Start Your Tech Journey?

Connect with our academic training counsellors. We analyze your background, career goals, and recommend the best live online track for your success.

Call Helpline: (02) 8902 0900WhatsApp or Viber Support