Data Engineering Community (DEC) (@data_dec) header
Data Engineering Community (DEC) (@data_dec) profile picture

A non-profit organisation providing Data Engineers a supportive and collaborative platform.

Nigeria
Joined April 2024
Compute: Where Does Your Pipeline Actually Run? Here's something beginners sometimes mix up: Storage is not compute. S3 can store your data, but it isn't where you normally think of your Python application or Spark job executing.
1
3
124
IAM: AWS Doesn't Just Ask "Can You Run This?" Imagine you've written a Python script that uploads a file to S3. ◉ The code looks correct. ◉ Your bucket exists. ◉ The file exists. ◉ You run the script. ◉ And AWS responds: AccessDenied
1
1
125
Why Data Engineers Care About S3 One of the first questions you should ask when designing a data platform is: Where is the data going to live? When you're working locally, the answer is usually simple. Your files are somewhere on your laptop, an external drive, or perhaps
1
11
359
CI/CD for Airflow Pipelines Your Airflow DAG is code. So treat it like code. 📌Instead of changing production DAGs manually, you can have a workflow like: Code ↓ Git ↓ Pull Request ↓ Tests ↓ Review ↓ Merge ↓ Deploy
1
1
8
245
Scheduling Is More Than @daily 📌One of the easiest things to do in Airflow is schedule a DAG: >>> schedule="@daily" But production scheduling is more complicated than that.
2
1
6
202
Retries Are Not a Magic Fix Your pipeline fails. Airflow says: >>> Retrying... Great. But should every failure be retried? Consider an API timeout. That's potentially a temporary problem, so retrying could make sense. Now consider invalid SQL.
2
2
132
Your Pipeline Failed. Now What? Your pipeline has been running successfully for two weeks. 📌Then one morning: FAILED. What do you do? First, don't immediately rerun everything. Find the failed task and check its logs. 📌You might discover: ◉ An API timeout
1
1
174
Airflow vs Prefect vs Dagster Once you start learning orchestration, you'll quickly discover that Airflow isn't the only option. You'll come across tools like Prefect and Dagster too. 📌So what's the difference?
1
3
4
164
Writing Your First Airflow DAG Enough theory. Let's build something. Imagine we're pulling data from an API, transforming it with Python and loading it into PostgreSQL. 📌Our DAG needs to handle three main tasks: ◉ Extract → Transform → Load
1
1
125
Airflow Tasks and Operators Now that we understand DAGs, let's talk about the actual work happening inside them. A task represents a unit of work that Airflow needs to execute. 📌That work could be: ◉ Running a Python function ◉ Executing SQL ◉ Running a Bash command
1
2
5
157