Jared Sulzdorf retweeted
Any git power users in my mutuals/ timeline?
We have a new and faster git experience coming up on Hugging Face and we'd love to get feedback from you!
Comment or DM and I'll hook you up!
Jared Sulzdorf retweeted
The Hugging Face Hub team is on a tear recently:
> You can create custom apps with domains on spaces
> Edit GGUF metadata on the Fly
> 100% of the Hub is powered by Xet - faster, efficient
> Responses API support for ALL Inference Providers
> MCP-UI support for HF MCP Server
> Search papers based on the Org
> Showcase repository size on the UI
and a lot more - excited for the coming weeks/ months as we continue to improve the overall UX! 🤗
Jared Sulzdorf retweeted
HuggingFace just shipped in-browser GGUF editing
It allows you to edit GGUF metadata in the comfort of your browser, without having to even download the full model. This feature is enabled via the Xet technology that makes partial file updates possible.
The Hub is on 100% on Xet. 🚀
A little over a year ago, @huggingface acquired @xetdata to unlock the next phase of growth in models and datasets. huggingface.co/blog/xethub-j…
In April, there were 1,000 Hugging Face repos on Xet. Now every repo (over 6M) on the Hub is on Xet.
Today, we've finalized this first phase of migrating the Hub to a new, modern storage system. One that's built to scale with AI builders of today and tomorrow. huggingface.co/blog/from-fil…
There's still a lot of work to do, but we're excited for what's next. 💪
1
63
Jared Sulzdorf retweeted
Let me explain why Hugging Face Datasets storage is faster than S3 + why today's release changes everything 🧵
Jared Sulzdorf retweeted
New blog post 🚨 Every data engineer should read it
@kszucs_ (@ApacheArrow PMC) announces how to drastically speed up Parquet files uploads and downloads.
Yes, it can easily outspeed S3.
Best part: the feature enabling this is open source
Link in 🧵
Jared Sulzdorf retweeted
A new Pandas feature landed 3 days ago and no one noticed.
Upload ONLY THE NEW DATA to dedupe-based storage like @huggingface (Xet). Data that already exist in other files don't need to be uploaded.
Possible thanks to the recent addition of Content Defined Chunking for Parquet.
We've moved the first 20PB from Git LFS to Xet on @huggingface without any interruptions, now we're migrating the rest of the Hub. We got this far by focusing on the community first.
Here's a deep dive on the infra making this possible and what's next: huggingface.co/blog/migratin…
You can see over the past few months some of the biggest migrations show up in our cluster throughput.
Each spike corresponds to a significant migration (where we download from LFS and upload to Xet) with the baseline steadily increasing to just shy of 100 Gb/s
1
2
93
It's been a bit since I took a step back and looked at our progress to migrate @huggingface from Git LFS to Xet, but every time I do it's mind boggling.
A month ago there were 5,500 users/orgs on Xet with 150K repos and 4PB. Today?
🤗 700,000 users/orgs
📈 350,000 repos
🚀 15PB
Meanwhile, our migrations have pushed throughput to numbers that are bonkers.
In June, we hit upload speeds of 577Gb/s (crossing 500Gb/s for the first time).
1
1
49
These are hard numbers to put into context, but let's try.
The latest run of Common Crawl from @CommonCrawl was 471 TB.
We now have ~32 crawls stored in Xet. At peak upload speed we could move the latest crawl into Xet in about two hours.
🤯🤯🤯
38
Jared Sulzdorf retweeted
Xet is now the default storage for new builders on @huggingface !
What it means for 🤗Datasets:
- Deduplicated downloads and uploads for speed⚡
- Works with the new Parquet CDC writer, robust to insert/delete/edits 💪
@ApacheParquet has a bright future on HF :)
New users and organizations can say goodbye to LFS on @huggingface; Xet is now the default storage for new builders on the Hub 🚀🚀🚀
Just sign up for an account, create a new repo, pip install huggingface_hub and you're off!
huggingface.co/changelog/xet…
To migrate your existing repos to Xet, sign up here huggingface.co/join/xet
And we'll take care of the rest 🤗
3
108
Continuing to move all the LFS bytes into Xet storage on Hugging Face! Currently up to:
🤗 5,500 users and orgs with Xet access
🚀 150,000 Xet-backed models and datasets
🤯 4+ PB managed by Xet
How much more to go? If the Hub's top storage users are any indication: many bytes
So many folks to thank, but a special 🤗♥️ to @bartowski1182 @UnslothAI @PrunaAI @nomic_ai @Alibaba_Qwen @tomaarsen @ngxson and everyone that has jumped on the Xet waitlist organically.
If you want to join the fun, the waitlist is over here huggingface.co/join/xet
1
32














