Compute: Where Does Your Pipeline Actually Run?
Here's something beginners sometimes mix up:
Storage is not compute. S3 can store your data, but it isn't where you normally think of your Python application or Spark job executing.
1
3
124
IAM: AWS Doesn't Just Ask "Can You Run This?"
Imagine you've written a Python script that uploads a file to S3.
◉ The code looks correct.
◉ Your bucket exists.
◉ The file exists.
◉ You run the script.
◉ And AWS responds: AccessDenied
1
1
125
Why Data Engineers Care About S3
One of the first questions you should ask when designing a data platform is:
Where is the data going to live?
When you're working locally, the answer is usually simple. Your files are somewhere on your laptop, an external drive, or perhaps
1
11
359
CI/CD for Airflow Pipelines
Your Airflow DAG is code. So treat it like code.
📌Instead of changing production DAGs manually, you can have a workflow like:
Code
↓
Git
↓
Pull Request
↓
Tests
↓
Review
↓
Merge
↓
Deploy
Retries Are Not a Magic Fix
Your pipeline fails.
Airflow says:
>>> Retrying...
Great. But should every failure be retried? Consider an API timeout. That's potentially a temporary problem, so retrying could make sense. Now consider invalid SQL.
2
2
132
Your Pipeline Failed. Now What?
Your pipeline has been running successfully for two weeks.
📌Then one morning: FAILED.
What do you do? First, don't immediately rerun everything. Find the failed task and check its logs.
📌You might discover:
◉ An API timeout
1
1
174
Airflow vs Prefect vs Dagster
Once you start learning orchestration, you'll quickly discover that Airflow isn't the only option. You'll come across tools like Prefect and Dagster too.
📌So what's the difference?
Writing Your First Airflow DAG
Enough theory. Let's build something. Imagine we're pulling data from an API, transforming it with Python and loading it into PostgreSQL.
📌Our DAG needs to handle three main tasks:
◉ Extract → Transform → Load
1
1
125










