Pinned Tweet
A little difficult to write this, but here goes.
Things haven't been great financially for me lately. And honestly, with Diwali and the festive season around, it feels a little harder than usual.
I'm not sharing this for sympathy. I'm trying to find a way forward,
1/5
Nobody tells you how much of Data Engineering is actually detective work.
You start your day thinking you'll build a new pipeline.
Then you notice yesterday's record count doesn't match today's.
You check the source. Then the target. Then the transformation logic.
You open the logs, check the watermark, compare timestamps, and start questioning decisions you made three weeks ago.
Two hours later, you finally find the issue.
And just when you think you're done, another pipeline fails.
Sometimes I feel Data Engineers are just detectives who happen to know SQL and Python.
And honestly, that's one of the things I enjoy about this field.
Every problem has a story. You just have to find it.
27
One thing I wish more Data Engineers understood about interview preparation:
Knowing the answer and being able to explain the answer are two very different skills.
You might know how a Spark join works. You might understand partitioning, broadcast joins, caching and data skew.
But when an interviewer asks, "Your Spark job is taking 90 minutes to finish. How would you investigate it?"
That's where things get interesting.
Can you explain where you'd start? What metrics would you check? How would you identify the bottleneck? And why would you choose one solution over another?
The same applies to SQL, Python, ADF and Databricks.
I've realised that preparing for interviews isn't just about covering more questions. It's about developing the ability to think through a problem, explain your decisions and defend your approach.
Don't just prepare to answer questions. Prepare to have engineering conversations.
That's a skill worth building, whether you're interviewing next week or not.
2
121
I've been working in Data Engineering for several years, and one thing I've learned is that this field never really lets you get comfortable.
You learn SQL, then Python. You get comfortable with PySpark, and suddenly there's something new to figure out in Databricks. You build pipelines, troubleshoot failures, investigate missing records, and learn something new almost every week.
But I don't want this account to be just another place for technical tips and interview questions.
I want to share the real side of working in tech too — things I've learned, career lessons, engineering opinions, mistakes worth learning from, and the occasional frustration that every Data Engineer will understand.
If you're into Data Engineering, technology or simply figuring out how to grow in your career, let's connect.
There's plenty to learn from each other.
2
3
189
Being a Data Engineer means spending 3 hours debugging a pipeline, fixing it in 2 minutes, and telling everyone it was a small change.
Incremental loading sounds simple until pipelines fail, records get duplicated, or data goes missing. Sharing practical lessons on watermarks, failure handling, idempotency, and building reliable data pipelines using ADF, SQL, and Databricks.
3
3
165
One thing I've noticed about Data Engineering interviews:
Having 5–7 years of experience doesn't automatically make you good at interviews.
You might have spent years working with SQL, Python, PySpark, Azure Data Factory and Databricks. You might have built pipelines, worked on migrations, handled production issues and delivered multiple projects.
But when an interviewer asks you to explain your technical decisions, things can get surprisingly difficult.
For example:
1. SQL: You have a query that takes 20 minutes to execute. How would you identify the bottleneck and optimise it?
2. PySpark: Your job works perfectly with 10 million records but becomes extremely slow with 500 million. What would you investigate?
3. Azure Data Factory: You need to ingest 100 tables from different source systems. How would you design a reusable ingestion framework?
4. Databricks: Your Delta table receives multiple updates and deletes every day. How would you maintain data consistency and handle schema changes?
5. Data Architecture: Your organisation wants to move from a traditional data warehouse to a Lakehouse architecture. How would you approach the migration?
These questions don't simply test whether you've used a particular technology.
They test whether you understand the underlying concepts, can explain trade-offs and know how to solve problems when things don't go according to plan.
I genuinely believe that one of the biggest mistakes experienced Data Engineers make is preparing for interviews only when they start looking for a new job.
Interview preparation should be an ongoing habit, not a last-minute activity.
1
2
113
Nobody:
Absolutely nobody:
Data Engineer at 2 AM:
"Let me just check why this pipeline failed."
1
56
Your SQL query takes 30 seconds to execute.
Don't immediately add an index.
First check:
- Execution plan
- Table scans
- Join strategy
- Index usage
- Statistics
- Unnecessary columns
- Expensive sorts
Optimisation starts with understanding the problem, not randomly changing the query.
#DataEngineering #SQL #SQLPerformance #DataEngineer #InterviewPreparation
1
50
I've spent the last 7 years learning, building, breaking things and figuring out my way around technology.
From software development to Data Engineering, it's been quite a journey.
Today, I work with Azure, Databricks, PySpark and SQL.
I want to share more of what I learn along the way:
→ Practical Data Engineering
→ Technology & AI
→ Career growth
→ Productivity and continuous learning
No gurus. No overnight success stories.
Just practical lessons, honest experiences and conversations with people who love technology.
If you're on a similar journey, let's connect.
What are you currently learning?
5
2
169
Azure Data Factory interview question:
You need to ingest 100 SQL Server tables into ADLS Gen2.
Would you create 100 pipelines?
I'd build a metadata-driven framework using:
- Configuration tables
- Lookup activity
- ForEach activity
- Reusable child pipelines
- Dynamic source and sink paths
- Audit logging
But how would you handle the 3 tables that fail while the other 97 succeed?
1
51
My Engineering grades
1st -> 5.96
2nd -> 6.78
3rd -> 6.77
4th -> 7.57
5th->7.93
6th-> 7.48
7th -> 6.74
8th -> 8.0
Not sure if things went well
My Engineering grades
1st -> 8.7
2nd -> 8.6
3rd -> 8.52
4th -> 8.4 (realised this shit is a scam)
5th->8.2
6th->7.8
7th ->7.6
8th ->7.7
I ended up pretty alright post college tbh!
4
221
One SQL concept every Data Engineer should understand properly:
ROW_NUMBER() vs RANK() vs DENSE_RANK().
They might look similar, but they behave differently when duplicate values exist.
Interviewers love testing this with real-world ranking problems.
Don't just memorise the syntax. Understand the difference.
1
150
SQL interview question:
You have an Orders table with customer_id, order_id, order_date and amount.
How would you retrieve the latest order placed by each customer?
Would you use:
1. ROW_NUMBER()
2. RANK()
3. A JOIN with MAX(order_date)
What's your approach?
2
166
PySpark interview question:
You have two DataFrames.
One contains 10 million records. The other contains only 5,000 records.
You need to join them.
Would you use a broadcast join?
Why or why not?
The answer isn't just about row count. Think about the actual size in memory, join strategy and executor capacity.
2
192
Your PySpark job takes 40 minutes instead of 5.
What do you do?
Most people immediately think about increasing cluster size.
But before spending more on compute, I'd investigate these 7 things.
A practical Spark performance checklist 🧵
1
3
274
7/ Execution plan
Use explain() and inspect the physical plan.
Understand what Spark is actually doing instead of guessing.
1
15


