Data Engineering

01

Python Programming

▼

• Python from ScratchVariables, data types, control flow, functions, and writing clean, readable Python code
• Data Structures & CollectionsLists, dictionaries, sets, tuples, list comprehensions, and generators — the building blocks of data manipulation
• File I/O & APIsReading and writing CSV, JSON, and text files, making HTTP requests, and working with REST APIs to fetch real-world data
• Libraries for Data Engineeringpandas for data manipulation, requests for API calls, os/pathlib for file handling, and logging for production scripts
• Scripting & AutomationWriting reusable scripts, scheduling tasks, error handling with try/except, and building CLI tools with argparse

02

SQL & Relational Databases

▼

• SQL from ScratchSELECT, WHERE, JOINs, GROUP BY, subqueries — mastering the language of data through real datasets
• Database Design & NormalizationER diagrams, primary/foreign keys, 1NF-3NF, and designing schemas that scale
• PostgreSQL Deep DiveSetting up PostgreSQL locally, indexes, views, CTEs, window functions, and query optimization
• Data Modeling for AnalyticsStar schema, snowflake schema, slowly changing dimensions — designing warehouse-ready data models

03

Foundations of Data Engineering

▼

• What is Data Engineering?Role of a data engineer, how it differs from data science/analytics, and the modern data stack overview
• Data Formats & StorageCSV, JSON, Parquet, Avro — understanding when to use which format and why columnar storage matters
• Linux & Command Line EssentialsNavigating the terminal, file operations, piping, cron jobs — the toolkit every data engineer needs
• Version Control with Gitinit, add, commit, push, pull, branching — collaborating on data projects using Git and GitHub

04

ETL Pipelines & Orchestration

▼

• ETL vs ELTUnderstanding Extract-Transform-Load patterns, batch vs streaming, and when to use each approach
• Building Pipelines with PythonReading from APIs, transforming with pandas, loading into databases — building end-to-end pipelines
• Apache AirflowDAGs, operators, scheduling, monitoring — orchestrating complex data workflows like production teams
• Data Quality & TestingSchema validation, data contracts, Great Expectations, and building trust in your data

05

Cloud Data Warehousing

▼

• Introduction to Cloud PlatformsAWS, GCP basics — understanding cloud storage (S3, GCS), compute, and managed services
• Data WarehousesBigQuery, Snowflake, Redshift — loading data, running analytical queries, and understanding partitioning & clustering
• dbt (Data Build Tool)Transforming data in the warehouse, writing modular SQL models, testing, and documentation
• Infrastructure as CodeDocker basics for data engineering, containerizing pipelines, and reproducible environments

06

Capstone Project & Real-World Systems

▼

• Streaming FundamentalsIntroduction to Apache Kafka, real-time data ingestion, and understanding event-driven architectures
• End-to-End ProjectDesign and build a complete data pipeline — from raw source to analytics-ready tables in a cloud warehouse
• Monitoring & ObservabilityLogging, alerting, data freshness checks, and building dashboards to track pipeline health
• Portfolio & Career ShowcaseDocumenting your project on GitHub, writing technical blog posts, and presenting your work to peers