1. Why This Matters
2. Core Concept — What is Containerization?
Analogy: A Fully Equipped House
The Mental Model
3. Docker Basics
Key Components
Let’s Make It Real
Build and Run
4. Why Docker is Useful in Data Engineering
5. Docker Compose — Managing Multiple Containers
Docker vs Docker Compose
The Key Insight
Example: Multi-Service Setup
8. Common Mistakes
9. Best Practices
10. Conclusion You write your code.
You test it locally.Everything works perfectly. Then it goes to production… and breaks. You spend hours debugging, only to realize:nothing is wrong with your code — the environment is the problem. In data engineering, this happens all the time: At its core, the issue is simple: Your environment is not consistent. Containerization solves this by packaging everything your application needs into a single, portable unit that runs the same way anywhere. Let’s simplify it with an analogy. Imagine being placed in an empty field with nothing around you. No food.No water.No electricity.
No shelter. You might survive for a while, but functioning properly would be difficult. Now imagine being placed inside a fully equipped house. Everything you need is already there: No matter where that house is moved, you can still live comfortably because your essentials move with you. Applications work the same way. An application needs certain things to function: Without them, the application breaks. Containerization solves this problem by packaging the application together with everything it needs to run. Think of a container as: a fully equipped house for your application. Inside the container, the app already has: So whether the container runs on: …the application still behaves the same way. Containerization gives your application its own portable environment with everything it needs to survive and run consistently. Here’s the smallest possible Docker setup for a Python app. Notice what we didn’t do: The environment is fully defined in the Dockerfile. In real-world data systems, you work with tools like: Without Docker, they often conflict. each tool runs in its own isolated environment — no conflicts, no surprises. This is especially useful in batch data pipelines because the entire workflow can be reproduced across different machines and environments. Real systems are never just one container. A Dockerized data engineering pipeline may include: Running each service manually quickly becomes painful. Without Docker Compose: one command starts everything. A simplified Docker Compose setup for a batch pipeline may include Airflow and PostgreSQL. This breaks almost everyone at first. localhost refers to the container itself, not your machine. Missing configs often cause silent failures. Containers are temporary. Without volumes, your data disappears. Poor Dockerfile structure can slow builds significantly. Use fixed versions to keep builds predictable. They have different requirements. It helps simulate real systems easily. This simplifies networking and debugging. Containerization changes how you think about environments. The real shift is this: You stop debugging environments — and start defining them as code. And once you reach that point: You’re no longer just writing code — you’re building systems. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse
print("Hello from Docker!")
print("Hello from Docker!")
print("Hello from Docker!")
FROM python:3.10-slim WORKDIR /app
COPY app.py . CMD ["python", "app.py"]
FROM python:3.10-slim WORKDIR /app
COPY app.py . CMD ["python", "app.py"]
FROM python:3.10-slim WORKDIR /app
COPY app.py . CMD ["python", "app.py"]
docker build -t my-python-app .
docker run my-python-app
docker build -t my-python-app .
docker run my-python-app
docker build -t my-python-app .
docker run my-python-app
services: airflow-webserver: image: apache/airflow:3.2.1 container_name: airflow_webserver command: airflow webserver ports: - "8080:8080" environment: AIRFLOW__CORE__EXECUTOR: LocalExecutor AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow volumes: - ./dags:/opt/airflow/dags - ./logs:/opt/airflow/logs - ./jobs:/opt/airflow/jobs depends_on: - postgres airflow-scheduler: image: apache/airflow:3.2.1 container_name: airflow_scheduler command: airflow scheduler environment: AIRFLOW__CORE__EXECUTOR: LocalExecutor AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow volumes: - ./dags:/opt/airflow/dags - ./logs:/opt/airflow/logs - ./jobs:/opt/airflow/jobs depends_on: - postgres postgres: image: postgres:16 container_name: postgres_db environment: POSTGRES_USER: airflow POSTGRES_PASSWORD: airflow POSTGRES_DB: airflow ports: - "5433:5432" volumes: - postgres_data:/var/lib/postgresql/data volumes: postgres_data:
services: airflow-webserver: image: apache/airflow:3.2.1 container_name: airflow_webserver command: airflow webserver ports: - "8080:8080" environment: AIRFLOW__CORE__EXECUTOR: LocalExecutor AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow volumes: - ./dags:/opt/airflow/dags - ./logs:/opt/airflow/logs - ./jobs:/opt/airflow/jobs depends_on: - postgres airflow-scheduler: image: apache/airflow:3.2.1 container_name: airflow_scheduler command: airflow scheduler environment: AIRFLOW__CORE__EXECUTOR: LocalExecutor AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow volumes: - ./dags:/opt/airflow/dags - ./logs:/opt/airflow/logs - ./jobs:/opt/airflow/jobs depends_on: - postgres postgres: image: postgres:16 container_name: postgres_db environment: POSTGRES_USER: airflow POSTGRES_PASSWORD: airflow POSTGRES_DB: airflow ports: - "5433:5432" volumes: - postgres_data:/var/lib/postgresql/data volumes: postgres_data:
services: airflow-webserver: image: apache/airflow:3.2.1 container_name: airflow_webserver command: airflow webserver ports: - "8080:8080" environment: AIRFLOW__CORE__EXECUTOR: LocalExecutor AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow volumes: - ./dags:/opt/airflow/dags - ./logs:/opt/airflow/logs - ./jobs:/opt/airflow/jobs depends_on: - postgres airflow-scheduler: image: apache/airflow:3.2.1 container_name: airflow_scheduler command: airflow scheduler environment: AIRFLOW__CORE__EXECUTOR: LocalExecutor AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow volumes: - ./dags:/opt/airflow/dags - ./logs:/opt/airflow/logs - ./jobs:/opt/airflow/jobs depends_on: - postgres postgres: image: postgres:16 container_name: postgres_db environment: POSTGRES_USER: airflow POSTGRES_PASSWORD: airflow POSTGRES_DB: airflow ports: - "5433:5432" volumes: - postgres_data:/var/lib/postgresql/data volumes: postgres_data:
volumes: - postgres_data:/var/lib/postgresql/data
volumes: - postgres_data:/var/lib/postgresql/data
volumes: - postgres_data:/var/lib/postgresql/data
FROM python:3.10-slim
FROM python:3.10-slim
FROM python:3.10-slim
node_modules .git .env
node_modules .git .env
node_modules .git .env - A Spark job runs locally but fails in production
- Airflow works on Ubuntu but breaks on macOS
- Kafka pipelines behave differently across environments - electricity - runtime versions
- system tools
- environment variables
- dependencies - its dependencies
- configurations
- runtime environment
- required tools - your laptop
- a cloud server
- a teammate’s machine - Image - A blueprint/template
- Container - A running instance of that image
- Dockerfile - Instructions to build the image - Install Python manually
- Manage versions
- Configure anything - Apache Airflow
- Spark / PySpark
- PostgreSQL or another data warehouse
- Reporting tools or dashboards - Different dependencies
- Different configurations
- Different runtime requirements
- Different ports
- Different environment variables - Airflow may require specific Python packages
- PySpark may need Java and Spark installed
- PostgreSQL may need database credentials and storage
- Dashboard tools may need access to the processed data - An Airflow webserver
- An Airflow scheduler
- A PostgreSQL database
- A Spark / PySpark processing service
- Shared folders for DAGs, logs, scripts, and data - Docker - runs one container
- Docker Compose - runs an entire system made up of multiple containers - Multiple terminals
- Manual startup order
- Constant configuration issues
- Harder networking between services - Using localhost inside containers - Forgetting environment variables - Not persisting data - Rebuilding unnecessarily - Use lightweight images - Add a .dockerignore - Avoid latest in production - Separate dev and production setups - Use Docker Compose for local development - Use clear service names - Docker packages your application into a portable unit.
- Docker Compose runs entire systems with one command.
- Your pipelines become reproducible and consistent.
