
0 to 1 Builder of data platforms and data products. Lately you can find me at the lake chilling with Apache Hudi, Apache Iceberg, and Delta Lake $LIOR
Joined July 2011
- Tweets608
- Following496
- Followers676
- Likes1K
Kyle Weller retweeted
🏆 Onehouse earned an honorable mention in Gartner’s 2026 Magic Quadrant for data lakehouse platforms. And we’re the only full-featured lakehouse on that elite list that can deploy anywhere.
It’s great to see Gartner recognize the full breadth of today’s Onehouse platform.
We started small. Now Onehouse is a fast, full-stack lakehouse: ingestion, Spark/SQL pipelines, cataloging, table management, compute, and low-latency serving.
But we’ve stayed stubborn about one thing: customers should own a single open copy of their data, use the engines they want, and run the platform where they want.
That flexibility, paired with a fully managed lakehouse, is exactly the vision we’ve been building towards for years.
Gartner captured it in a few crisp sentences.
Excited for where this goes. More to come.
Kyle Weller retweeted
More than 200,000 CPU cores power Uber’s data lake ingestion. At that scale, every efficiency gain counts.
Making better use of that compute means examining how Spark executes queries, moves data, and writes to growing tables. Accelerating one stage can leave other bottlenecks untouched.
Uber engineers are improving production workloads with native vectorized execution and evaluating new ways to accelerate write-heavy pipelines.
Explore the engineering choices behind those efforts, where the benchmarks showed gains, and what remains to be optimized.
Kyle Weller retweeted
You don't need Spark to read a Hudi table.
hudi-rs is the native Rust implementation with Python and C++ bindings — built for the AI/ML toolchains that already live outside the JVM.
Kyle Weller retweeted
Rare: a table-formats article that doesn't tell you to rip-and-replace everything for Iceberg. If you work near a lakehouse, it's worth a read.
Key points, regardless of what you run:
💸 The format itself costs $0. Iceberg, Delta, and Hudi are Apache-2.0. You pay for compute, storage, and the platform—so the real question is engine economics for your workload.
Kyle Weller retweeted
🚨 The "sync dilemma": metadata in a lakehouse, embeddings in a vector DB, images in object storage, semi-structured data in a doc store — all copies of the same asset, all drifting out of sync.
Apache Hudi 1.2.0 unifies multimodal data into a single transactional lakehouse table 🎉
🧠 Native VECTOR, BLOB & VARIANT types 🔍 Built-in vector search via hudi_vector_search() 🗂️ Lance file format alongside Parquet, ORC, HFile ⚡ Record Level Index + FLIP-27 source for Flink 🔧 Storage-based locking natively on Azure
600+ commits. One table. No silos.
Dive in: hudi.apache.org/blog/2026/06…
#ApacheHudi #DataEngineering #AI
Kyle Weller retweeted
Spark teams always trade off: Control via self-run on k8s or perf from managed services. Rarely both. ⚖️
Quanton Kubernetes Operator (QKO) flips that—drop-in for k8s Spark Operator, runs jobs 3-4x faster on your infra. 🚀
Why? Compute is standardizing on k8s, but OSS Spark on k8s is slow. QKO: Same jobs/setup, huge perf boost. ⚡
Get: 3-4x speed, 40-60% cost cuts, zero changes. 💰🔁
Early access: github.com/onehouseinc/quant…
Kyle Weller retweeted
Most table formats treat keys as optional semantics.
In Hudi, record keys are crucial operationally 🔑.
They drive:
- Updates finding prior records 🔄
- Deletes application 🗑️
- Deduplication 🔍
- Record mapping to file groups 📁
- Bounded concurrency & merging ⚖️
This lets Hudi handle true granular updates, not vague file rewrites.
Stable record identity makes tables act like databases, where records are compacted/clustered transparently.
Kyle Weller retweeted
Ruiyang Wang works on pretraining infrastructure at Anthropic. If he says PDFs belong in your threat model, that should get your attention. 🛡️⚠️
At #OpenXData, Ruiyang is laying out a rasterize-first architecture for safe PDF processing at scale: sandbox the file, render it to images, then OCR. He previously spent years on Ray at Anyscale and now builds large-scale data pipelines at Anthropic. That is a strong setup for a talk about handling hostile documents without losing sight of quality.
If PDFs touch a training pipeline, RAG stack, or document ingestion flow in your org, register here: openxdata.ai/
#OpenXData #DataEngineering #MLInfra #AIPipelines #AISafety
Kyle Weller retweeted
Replying to @J_
@J_ co-created Apache Parquet, Apache Arrow, and OpenLineage. Three projects. Three industry standards.
Parquet at Twitter in 2013. Arrow at Dremio. OpenLineage at Datakin, acquired as part of Astronomer's $213M Series C. He is now Principal Engineer at Datadog and an officer of the Apache Software Foundation. That is an unusual track record of picking the right abstraction at the right time.
His OpenXData talk argues that the current wave of challengers -- Lance, Vortex, Nimble, FastLanes, BtrBlocks, F3 -- are solving real problems but misreading what made Parquet succeed in the first place. The core contribution was not the encoding choices. It was the community consensus mechanism those choices were built inside. His case: use established open source communities to absorb these innovations rather than fragment the ecosystem across six competing formats.
He published the written version of this argument at sympathetic.ink in December 2025. OpenXData is where you can push back live.
👉 Register here: openxdata.ai
Kyle Weller retweeted
🚀 Why companies resist moving compute out of their own accounts as they scale? e.g Moving from Databricks Classic to SQL Serverless
Because Bring-Your-Own-Cloud (BYOC) isn't just smart architecture—it's the economics that make sense.
Here's why it wins:
💰 1) Spot and reserved discounts work for you, not the vendor.**
🔐 2) Data stays secured inside your own compliance perimeter.**
📊 3) Full transparency into the real $/GB cost of your workloads.**
With AI driving massive data growth, BYOC becomes essential once costs start ramping.
Kyle Weller retweeted
Vendors push 'serverless' (EMR to EMR Serverless, Databricks Classic to SQL Serverless)—ease of use? Sure, but it's economic control. 💰
Compute outside your account means they keep:
📉 Hyperscaler discounts
🔍 Hidden $/GB pricing
🏦 Sticky margins
🤝 Leverage on commits
Enterprises need cost transparency and data sovereignty. 🧭
BYOC platforms deliver control + convenience. ⚡
#Serverless #DataPlatform #CloudInfra
Kyle Weller retweeted
Peloton started with Copy-on-Write for simplicity 🔄, but frequent updates across hundreds of partitions made writes too expensive 💸—some runs hit nearly an hour, with high storage amplification from commit history ⏱️.
Switched to Merge-on-Read for:
📥 More frequent ingestion
⚡ Lower write latency
💾 Better storage efficiency
🔧 Fit for mutable workloads
Table type is a workload call, and workloads evolve 📈.
Kyle Weller retweeted
Everyone assumes usage-based pricing in cloud data is fair and efficient. ⚖️
But it has a real problem: It can stop vendors for building faster engines.
Traditional models priced on value—Oracle earned more for standout features.
Now, with EMR or Databricks, bills hinge on compute usage. Customers win from compute efficiency (lower costs), but vendors lose revenue, pushing them to own the compute layer for pricing control.
Sure, usage models offer flexibility, but they misalign incentives long-term.
What's better? We need outcome-based pricing that rewards real value, like queries executed or data processed. 🚀📊
Kyle Weller retweeted
Spark is still a $15B+ annual spend category 💰
Yet most enterprises treat Spark like a black box. 🧠
TLDR: pip install spark-analyzer
Apache Spark still powers the backbone of lakehouse workloads 🏗️
Yet inside most companies, no one can clearly answer:
❓ Where does the spend actually go?
❓ Why don’t optimizations translate into real savings?
❓ Why is Spark cost so unpredictable?
A huge share of this spend runs on
⚠️ slow runtimes that waste compute cycles (e.g. default EMR setups)
💸 premium platforms charging 2–3× markups for engines like Photon
If you now want to do something about it : pypi.org/project/spark-analy…
Kyle Weller retweeted
1/
✨ Azure just made the list.
Not the list you’re thinking of.
The list of clouds that Onehouse runs on.
With our launch on Microsoft Azure, the only truly modular data lakehouse platform now runs across AWS, GCP, and Azure.
Kyle Weller retweeted
1/
🔥 Today we’re announcing Onehouse’s low-latency interactive query engine.
Because if AI generates most of your SQL queries, your current engine won’t scale. 🧵👇
Kyle Weller retweeted
Onehouse LakeBase™ - The first lakehouse serving layer with database capabilities like indexing and caching. Built for machines + humans. Handling high-QPS, low-latency queries from AI agents and heavy analytics.
onehouse.ai/blog/announcing-…
Kyle Weller retweeted
1/ 🤖 AI is coming for jobs.
💻 Software is dead.
📉 Software stocks are getting wrecked.
That’s the narrative this week.
But it’s… incomplete.
🧵⬇️























