MagicLex retweeted
My O'Reilly book on building ML systems with a feature store is still available for download free for only 1 more month. Get it now.
hopsworks.ai/lp/full-book-or…
MagicLex retweeted
The top 10 fallacies of MLOps
hopsworks.ai/post/the-10-fal…
@Aurimas_Gr @paulabartabajo_ @iusztinpaul @moteropedrido
MagicLex retweeted
The 𝐅𝐓𝐈 (𝐟𝐞𝐚𝐭𝐮𝐫𝐞, 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠, 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞) architecture.
A mental map to build ML Systems and not get lost in the process ...
#MachineLearning #MLOps #DataScience
MagicLex retweeted
Training ML models is easy.
Transforming the data these models need is the hard part... until you learn this ↓↓↓
𝗧𝗵𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺
Building a real-world ML system is
> 𝟭𝟬% about training and deploying ML models,
and
> 𝟵𝟬% about transforming the data these models needs to work.
And the thing is, 𝗻𝗼𝘁 all data transformations are the same.
𝗧𝗵𝗲 𝘁𝗮𝘅𝗼𝗻𝗼𝗺𝘆 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻𝘀
In every ML system we can have up to 3 different types of data transformations
1️⃣ 𝗠𝗼𝗱𝗲𝗹-𝗜𝗻𝗱𝗲𝗽𝗲𝗻𝗱𝗲𝗻𝘁 Transformations, for example rolling averages.
> Reusable across models
> Stored in the feature store
2️⃣ 𝗠𝗼𝗱𝗲𝗹-𝗗𝗲𝗽𝗲𝗻𝗱𝗲𝗻𝘁 Transformations, for example feature normalization
> Specific to one model
> Applied in both training and inference
3️⃣ 𝗢𝗻-𝗗𝗲𝗺𝗮𝗻𝗱 Transformations
> Require real-time data
> Used in online inference
Once you understand how and 𝗪𝗛𝗘𝗥𝗘 your data transformation happens, you are in a good position to start building ML software that works.
If you want to learn more about the taxonomy of data transformations read this excellent blog post by the great @jim_dowling
> 🔗 hopsworks.ai/post/a-taxonomy…
----
Hi there! It's Pau Labarta Bajo 👋
Every day I share free, hands-on content, on production-grade ML, to help you build real-world ML products.
𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 so you don't miss what's coming next
MagicLex retweeted
Why is building AI systems hard?
Because even with the simplest feature that you can imagine, you need to consider where it will be computed based on whether it's a batch AI system or a real-time AI system.
Luckily, there are some principles to guide you.
A thread 🧵👇
MagicLex retweeted
From our SIGMOD'24 paper, an easier read on the work we have done and are doing on making #rondb the database for real-time AI applications. If you thought #redis is good enough for real-time AI, please read this and tell us if we have changed your mind. hopsworks.ai/post/rondb-a-re…
MagicLex retweeted
Did you know that you can generate and manage your training data right from the Hopsworks UI? @TheMagicLex show how you can simplify your data workflow and enhance traceability with just a few clicks. 🔄
bit.ly/4eNRyyx
MagicLex retweeted
Introducing a query engine for AI on table formats (@apachehudi , @DeltaLakeOSS ) . Benchmarks from @sigmod show @hopsworks outperforms up to 45X Databricks, AWS, GCP. A step towards making Python a 1st class client for Lakehouse data.
hopsworks.ai/post/from-lakeh…
MagicLex retweeted
We released a new blog post made jointly with the team at @hopsworks! Read it to find out how Feldera and Hopsworks.ai work together to do real-time feature engineering with Feldera computing ML features and Hopsworks handling the storage layer: feldera.com/blog/hopsworks
MagicLex retweeted
It was a long journey for us to be the first feature store to add support for the Snowflake Schema Data Model. This means you can add more precomputed features to your online models with minimal additional latency.
hopsworks.ai/post/the-journe…
MagicLex retweeted
The taxonomy for data transformations for AI helps when you create features that are reused by many models: (1) in which ML pipeline to place this data transformation, (2) why creating training data takes hours instead of minutes?
hopsworks.ai/post/a-taxonomy…
MagicLex retweeted
How to architect AI systems is not a very sexy topic, but is crucial to the success of AI projects. Modularity is a key mechanism for decomposing a system into independent components. Composability, is also necessary to avoid microservices-hell.
hopsworks.ai/post/modularity…
MagicLex retweeted
Now that Redis open-source is dead, look no further than rondb.com for an open-source in-memory database. But one that scales .... hopsworks.ai/post/ai-ml-need….
MagicLex retweeted
Introducing the general availability of Hopsworks 3.7 - the GenAI release! This new release adds additional capabilities to support LLM/GenAI use cases and includes updates such as:
📈 Feature monitoring
🛠️ Delta Lake support
🔔 Notification service
bit.ly/3TB3wCV
MagicLex retweeted
Hopsworks has officially added support for #DeltaLake! 🚀🦀
"We have decided to support Delta Lake to accelerate our mission to build the Python-Native Data for AI platform."
🔎 Check out this comprehensive blog post by @hopsworks to learn more: hopsworks.ai/post/delta-lake…
#oss
MagicLex retweeted
I have been busy the last few months writing a book for O'Reilly about how to build ML systems (batch, real-time, and LLMs), distilling much of what I have learnt from both working with customers as well as students. Why could the book interest you?
* Data Scientists - transition from training models to building ML systems
* ML Engineers - learn about how to build batch, real-time, LLM systems in modular parts that you compose into a ML system
* Data Engineers - learn about the data transformation taxonomy for ML and how badly structured DAGs prevent reuse in ML systems
* Architects - divide et impera - learn how modularity helps you build faster and better ML systems.
Early access to the first chapter (52 pages) is available here:
hopsworks.ai/lp/oreilly-book…
MagicLex retweeted
Imagine you had to pick a novel data source and identify a prediction problem for it. What ML system could you build in a few weeks?
Here's what KTH students built (mostly with @hopsworks).
id2223kth.github.io/assignme…
Examples in the 🧵below














