The next-gen stream processing engine. Now part of @CloudflareDev.
Berkeley, CA
Joined October 2022
- Tweets68
- Following2
- Followers236
- Likes3
Built on Arroyo!
The Cloudflare Data Platform has been in beta for the past year and has helped amazing companies like @anomalyco ingest, store, and query their analytical data. Today, for @Cloudflare Birthday Week, we're going GA. And we have a new name: Cloudflare Basin.
Arroyo retweeted
For Arroyo 0.16 (coming very soon!) we rebuilt the control plane, in order to scale to the thousands of jobs we're running for Cloudflare Pipelines. Along the way we had to design a new distributed fencing protocol on r2/s3 to achieve exactly once commits: arroyo.dev/blog/worker-leade…
Arroyo retweeted
I'm very excited to announce that after 8 months of work, Arroyo 0.15.0 is now available! This is a huge release that brings all of the improvements, fixes, and learnings from running Arroyo at scale to power @CloudflareDev Pipelines, our fully-managed ingestion service.
Arroyo retweeted
The news is finally out! Cloudflare has a Data Platform! We're starting with serverless streaming pipelines (powered by @ArroyoSystems), a managed Iceberg Catalog, and a new distributed SQL engine built on @ApacheDataFusio.
Arroyo 0.14.1 is now available! This is a patch release with a number of fixes over 0.14.0, and is a drop-in upgrade: github.com/ArroyoSystems/arr…
Arroyo retweeted
The third SF DataFusion meetup is today at 6! We have a bunch of great speakers including project leader @andrewlamb1111, @camuelg, @lakesail , and @Greptime. I'll be giving a talk about how Arroyo does streaming incremental view maintenance to support unbounded SQL queries.
We couldn't be more excited to be building the future of real-time process with Cloudflare! See all of the details on the blog: arroyo.dev/blog/arroyo-is-jo…
1
1,078
We are very excited to announce that we are joining @Cloudflare to bring stream processing to everybody! Arroyo is coming to the Cloudflare Developer Platform as a serverless stream processing system, and will also remain open-source and self-hostable.
Arroyo retweeted
Two big things:
1. Pipelines, our brand new streaming ingestion service, is now in public beta.
2. We've acquired @ArroyoSystems so we can bring SQL-based stateful transforms to Pipelines!
Ingest tens-of-thousands of records per second right now: blog.cloudflare.com/cloudfla…
Arroyo 0.14.0 is now available! New features in this release include:
🔎 Lookup joins
🪆 Nested updating aggregates
{} Struct types
📖 Streaming SQL syntax
🔀 Sink shuffles
See the full release notes at arroyo.dev/blog/arroyo-0-14-…
Arroyo retweeted
You'd think that the key to being a fast streaming engine is like clever join algorithms, but it's mostly just being really good at JSON. @ArroyoSystems uses Arrow and the arrow-rs JSON decoder. I think it's pretty cool, so I wrote up how it works: arroyo.dev/blog/fast-arrow-j…
Building a near-real-time analytics stack used to take a team months or years. Thanks to modern tech like S3, @redpandadata, @duckdb, and Arroyo, any company can build a sub-minute data lake in under an hour: arroyo.dev/blog/building-a-r…
Arroyo 0.13.1 is now available: github.com/ArroyoSystems/arr…
This patch release includes several fixes over 0.13.0 and is a drop-in upgrade.
1
210
Arroyo 0.13.0 is now available, with
📘 Source metadata support
🐰 RabbitMQ streams connector
⚛ Atomic updating outputs
🔐 IAM auth for Kafka
⛓️Operator chaining
See all of the details in the release post: arroyo.dev/blog/arroyo-0-13-…
Arroyo 0.12.1 is now available: github.com/ArroyoSystems/arr…
This patch release contains several bug fixes and is a drop-in upgrade for clusters running 0.12.0.
241
Arroyo retweeted
If you missed #p99conf last week, talks are now available to stream on YouTube. I spoke about the design decisions that went into Arroyo's incredible performance: youtube.com/watch?v=7H4CIKZ4…
Come for the Rust hot takes, stay for my terrible hand-drawn architecture diagrams 😅
Arroyo 0.12.0 is now available, with support for 🐍 Python UDFS 🐍, plus Protobuf, faster JSON functions, custom state TTLs, and more.
See all the details in the release notes: arroyo.dev/blog/arroyo-0-12-…
Arroyo is coming to #Current24! Come chat with the team and learn about the future of stream processing.
Arroyo retweeted
Looking forward to talking about how we dynamically-load DataFusion UDFs in @ArroyoSystems! Because Rust lacks a stable ABI, this is harder than it sounds!
Excited to announce the next SF DataFusion meetup will take place September 25th hosted by @trychroma!
We'll have talks by @asc89 @mwylde and @0xLukeKim
Event link in comments 👇