Founder @Onehousehq — Data Infra for Agents. Creator of @apachehudi, built the first #DataLakehouse @Uber. Databases, Dist. Systems. LinkedIn, Confluent.
Joined April 2009
- Tweets1.5K
- Following235
- Followers1.9K
- Likes841
It's similar to how Snowflake or Databricks buy/reserve compute for years on hyperscalers, then resell as platform credits, right? As long as demand sticks, it works out?
Snowflake and Databricks arguing about AI revenue. Finally just something inexpensive from both companies: free entertainment. 🍿
Spark on Kubernetes is hard to run well. Our live session on the these hard parts had a great turnout. If you missed it, here's the full written guide.
Running Spark on Kubernetes looks deceptively easy. Most teams moving Spark from YARN to Kubernetes underestimate what YARN gave them for free. In reality, it's full of interesting problem solving on resource management and failure-handling.
Here's what the guide covers:
⚙️ Basics around how the driver, executors, and the K8s scheduler work together, pod mapping
🏛️ Why teams are moving from YARN: unifying all compute on k8s for deploys, access controls, chargebacks, and autoscaling
🧩 spark-submit vs. operators, and where adding YuniKorn or Volcano can help
💥 Production lessons from running this at Onehouse: OOM kills, shuffle storage without an external shuffle service, and driver failures
It starts small with SparkPi on a local kind cluster and builds up to what actually breaks at scale.
We hope you find it useful. Guide link in the comments 👇
While we are at it, can you all please also publish your numbers on closed vs open formats.
release all the numbers.
does 80% yoy become 65% when you remove passthru inference?
what does it become when you remove ai natives?
and how much of it is from one time software license --> service transition?
fcf, RPO, cRPO?
op margin non-gaap, gaap? dilution?
curious ppls want to know
ps: @alighodsi happy for your success. the world is big and life long.
teapaw.org/bhorowitz/status/21150…
If you want spare EC2 capacity, look at your Apache Spark clusters and then go give quanton.dev a shot.
sign up link: onehouse.ai/events/webinars/…
Spark on Kubernetes at scale is hard. Onehouse runs a lot of it, and tomorrow we're sharing what we've learned.
In most places, the knowledge required to get all of this right lives across the data team and the Kubernetes/infra team.
We've collected a list of things that looked simple at first, but hard in production:
- OOM kills
- Shuffle storage
- Spot interruptions
- Autoscaling
- Long-running drivers
- Cluster upgrades
- Keeping streaming jobs alive while the infra underneath changes
Most of it lives in the gap between the data team and the infra team.
Webinar tomorrow. Bring your worst production problem.
Link in comments.
Last week, I posted on why warehouses still default to closed formats. What I learned: they won't flip the default. Open formats are opt-in.
So the fix is with the users. Give every open format (Delta, Hudi, Lance, Vortex, ClickHouse, DuckLake) the same consideration on performance and flexibility that you give the closed ones.
Your data will thank you.
Back to building on open formats.
ALT A man in a field checks his watch repeatedly, while a skeleton lies nearby, humorously illustrating waiting for open file formats.
Does the technical problem simply boil down to a very large table on object storage with fast writes and fairly fast analytical and point lookups?
I like that the ChatGPT Mac app has a built-in markdown editor. It definitely reduces friction in review cycles.
Not at all. Customers are stuck. We speak to dozens every week. They have to pay hefty egress costs to do the migration.
The question I raised is more about first principle and forward-looking. Why not make open formats the default for any new table created from this point forward? That costs the vendor nothing.
Btw the LinkedIn counterpart for this x post has a lot more thoughts if you are curious about this topic .
My thoughts and receipts are in the blog onehouse.ai/blog/snowflake-b…
Last week, Databricks CEO Ali Ghodsi and I debated Iceberg and open table formats on a post. Then I caught myself.
Why are we debating each other while Snowflake, BigQuery and Redshift still default customers to closed formats?
Two years of Iceberg announcements. Still no public retirement date for their closed formats.
Open-source projects get scrutinized for interoperability on every release. The largest warehouses should meet the same bar.
We’ve spent a decade building open formats and data lakes. I think it’s time we asked.
ALT Anakin and Padmé meme. Snowflake, BigQuery and Redshift in 2024: “The future is open. Use Iceberg.” Data community: “So you’ll retire your closed formats?” The same warehouses in 2026 say nothing, and the data community asks: “You’re still planning to retire them... right?