
Freelance Data Engineer and instructor, enjoy solving data problems with #ApacheSpark #AWS #GCP #Azure 👨🏭 | [email protected]
remote
Joined January 2011
- Tweets860
- Following79
- Followers1.8K
- Likes759
Bartosz Konieczny retweeted
⏰ Final Reminder – Delta Lake Webinar Tomorrow!
Wondering if data engineering design patterns can unlock new insights into Delta Lake? Or how Delta Lake can become a key part of your streaming data architecture?
Join @newfront (@bufbuild) and @waitingforcode as they tackle these questions head-on!
🗓️ Oct 14 @ 9AM PT
🎥 Live on LinkedIn, YouTube & X
📍 Reserve your spot today: luma.com/delta-1014
#opensource #oss #deltalake #streaming #dataengineering
Bartosz Konieczny retweeted
Why don’t Iceberg or Delta Lake have secondary indexes?
Because analytics workloads and OLTP workloads optimize for opposite I/O patterns.
See my dive into data layout, pruning, and what “indexing” really means in open table formats: jack-vanlightly.com/blog/202…
Bartosz Konieczny retweeted
Are you wondering if general concepts like data engineering design patterns can help you learn about #DeltaLake? Or, if it's possible to leverage Delta Lake within your streaming data architecture?
In this webinar, Scott Haines and Bartosz Konieczny will answer these two questions. Scott, who gained streaming expertise at Yahoo, Twilio, and Nike, will share with you best practices for leveraging Delta Lake as a component of your streaming architecture. ✅
Bartosz, who recently published Data Engineering Design Patterns, will reverse-engineer a few of these design patterns to explain which Delta Lake features make everything tick.
🗓️ Tuesday, Oct 14
🕝 9AM PT
Don't miss it! 🔗 Register today: luma.com/delta-1014
#opensource #oss #dataarchitecture #dataengineering @waitingforcode
Bartosz Konieczny retweeted
Releasing Soon! Pre-order now shroffpublishers.com/books/9…
Data Engineering Design Patterns
By Bartosz Konieczny @waitingforcode. with @OReillyMedia
Focusing on various aspects of data engineering, including data ingestion, data quality, idempotency, and more. #dataengineering
Bartosz Konieczny retweeted
If you want to understand the consistency models of the mentioned table formats of the paper, I've written about it extensively and written formal models.
* jack-vanlightly.com/analyses…
* jack-vanlightly.com/analyses…
* jack-vanlightly.com/analyses…
* github.com/Vanlightly/table-…
Bartosz Konieczny retweeted
Data Engineering patterns on the cloud by Bartosz Konieczny is on sale on Leanpub! Its suggested price is $39.00; get it for $24.65 with this coupon: leanpub.com/sh/ygsnqbRD @waitingforcode #CloudComputing #AmazonWebServices #GoogleCloudPlatform #MicrosoftAzure
Bartosz Konieczny retweeted
Join @newfront and @waitingforcode and learn all about streaming Delta Lake tables with Apache Spark Structured Streaming! 🦀
🗓 March 21st
🕝 9:00AM PT / 12:00PM ET
💻 Join this webinar via LinkedIn, YouTube, or Zoom!
Learn more: linkedin.com/events/streamin…
#deltalake #streaming
Bartosz Konieczny retweeted
I have been busy the last few months writing a book for O'Reilly about how to build ML systems (batch, real-time, and LLMs), distilling much of what I have learnt from both working with customers as well as students. Why could the book interest you?
* Data Scientists - transition from training models to building ML systems
* ML Engineers - learn about how to build batch, real-time, LLM systems in modular parts that you compose into a ML system
* Data Engineers - learn about the data transformation taxonomy for ML and how badly structured DAGs prevent reuse in ML systems
* Architects - divide et impera - learn how modularity helps you build faster and better ML systems.
Early access to the first chapter (52 pages) is available here:
hopsworks.ai/lp/oreilly-book…
Bartosz Konieczny retweeted
I don't want to start a flame war here, but IMO it is a mistake to jump straight to distributed databases (and 90% of the content below is distributed databases) without first learning fundamentals on single node databases.
Here's my 10 things to understand about databases:
1. Relational model. Primary keys, foreign keys, normal form.
2. SQL language. Ideally with advanced SQL (CTE, analytics)
3. ACID and how transactions work
4. Write-ahead log (or binlog) and how it is used. Especially around restarts, recovery and replication.
5. Buffer cache, disk storage layout and how they interact
6. What happens when databases start? when they shut down?
7. Indexes, cluster tables, partitions and other types of database structures.
8. Query parsing, planning and optimizing.
9. MVCC and how to deal with its quirks in your DB of choice
10. Security - authentication, authorization, encryption on wire and at rest.
11. (Bonus) Investigating performance issues and making sense of benchmarks.
Entire world, stuff that 99% of developers use daily. You can be a deep expert without ever looking at distributed databases. And this also serves as strong foundation once you do.
And if you use Postgres, I found this free book super helpful in making sense of things: interdb.jp/pg/
Ten things to understand about your database:
1) High level Architecture
2) How writes work? (Replication, data distribution, internal organisation etc)
3) How reads work? (Consistency guarantees, tuning options, etc)
4) CAP theorem, ex. CP or AP
5) Transactions and Concurrency models
6) How does it scale?
7) How are failures handled?
8) Best practices on Querying data
9) How is geo-distribution supported, so you can plan ahead in time?
10) How to optimise cost?
Episodes to watch to understand the above for different databases:
DynamoDB:
youtu.be/ifSckJlatWE
Cassandra:
youtu.be/V1EO_0i3RNA
CockroachDB:
youtu.be/1NuvxQEoVHU
General database internals: Part-1
youtu.be/DiLA0Ri6RfY and Part-2
youtu.be/IW4cpnpVg7E
Realtime Analytics with Apache Pinot:
youtu.be/cGTffWg2EFs
Geo Distribution of databases:
youtu.be/JQfnMp0OeTA
CDC and Debezium:
youtu.be/VGH6TlhEJpM
Twisp - A ledger database:
youtu.be/VGb54yNQrHM
Kafka internals
youtu.be/d89W_GzWnRw
YugaByteDB Internals:
youtu.be/cXIPIA7e220
Write ahead logging:
youtu.be/yV_Zp0Mi3xs and
youtu.be/2MqY_mT1vw8
B-Trees on Disk:
youtu.be/dTfR0S_rBGg
Graph Database Internals:
youtu.be/iihJXKAQZkA
ScyllaDb internals:
youtu.be/AqY13RjWwJg
Duckdb Internals:
youtu.be/f9QlkXW4H9A
RisingWave Streaming Database:
youtu.be/nckuW02gI3Y
Clickhouse Internals:
youtu.be/sh5EBqrrwEU
Bartosz Konieczny retweeted
Data Engineering patterns on the cloud by Bartosz Konieczny is on sale on Leanpub! Its suggested price is $39.00; get it for $26.10 with this coupon: leanpub.com/sh/1T4q5Z81 @waitingforcode #CloudComputing #AmazonWebServices #GoogleCloudPlatform #MicrosoftAzure
Bartosz Konieczny retweeted
Chapter 4 of The Architecture of Serverless Data Systems: CockroachDB (serverless).
jack-vanlightly.com/analyses…
Bartosz Konieczny retweeted
The early release of Delta Lake: The Definitive Guide is here! 🎉 The latest edition includes the addition of Chapter 12: Performance Tuning.
Download here ➡️ bit.ly/472DVY7
Authors @dennylee, Prashanth Babu, Tristen Wentling, & @newfront
#opensource #deltalake #oss
Bartosz Konieczny retweeted
Data Engineering patterns on the cloud: How to solve common data engineering problems with cloud services? leanpub.com/data-engineering… by Bartosz Konieczny is the featured book on the Leanpub homepage! leanpub.com @waitingforcode #CloudComputing #AmazonWebServices
Last week I spent some time to understand the #PySpark applyInPandasWithState. This week I'm refactoring the code, hoping to still understand it 2 months later ;) 👉 waitingforcode.com/apache-sp…
In the previous release #PySpark has got an interesting streaming feature -> the arbitrary stateful processing. It has a different API than the Scala version but is more adapted to the Python world.
More 👉 waitingforcode.com/apache-sp…
Bartosz Konieczny retweeted
A list of articles I share again and again when developers ask me about Kafka 🧵
Bartosz Konieczny retweeted
[ANNOUNCEMENT] Congrats to the Apache Spark community and all the contributors! The Apache Spark 3.5.0 release is here. Try it out! spark.apache.org/releases/sp…
It's not a rebranding but more a regrouping 😉 All my additional #dataengineering content is now available from there waitingforcode.com/better (planning to add some stream processing materials soon)
If Delta Lake implemented the commits only, I could stop exploring this transactional part after the previous article. But as for RDBMS, #DeltaLake implements other ACID-related concepts, such as isolation levels 👉 waitingforcode.com/delta-lak…
One of the great features of table file formats is the ability to handle write conflicts. It wouldn't be possible without commits that are the topic of my #DeltaLake blog post. waitingforcode.com/delta-lak…















