̷̨̨̛̛̲͎̣̞̘͈̯͚͂̈́̂̄̽̎́̽̔͑̄̏̽̏͒̾́̅̐̈́̾̎̆͆̽́͌̽̀̚̕̚̕͠͝͝ (@Quantum_Stat) header
̷̨̨̛̛̲͎̣̞̘͈̯͚͂̈́̂̄̽̎́̽̔͑̄̏̽̏͒̾́̅̐̈́̾̎̆͆̽́͌̽̀̚̕̚̕͠͝͝ (@Quantum_Stat) profile picture

҈̿ ̶̢̢̧̡̼̜̝̬͍̜̘̙͉̘͎͓͍̣̰͖̹͖͚̭̘̖̟͕̬̠̬͇̹̮͎̣̱̎̾͌̊́̇͛̅͂̀̀͑̃̈̀̓̏͌͌͋̐̾̒͋͋̏͋̽̈́͐͑̐̂͆̊̈́̾͌̓͌̕̚͝͝͠͠͝ͅ Repos: https://teapaw.org/t.co/3x0DctKuld

Joined March 2018
🚀🚀 Super excited to share the latest benchmark results for our quantized BGE models. A few weeks ago, these models were introduced with the aim of enhancing performance and efficiency for generating embeddings. And we've now conducted thorough comparisons between running PyTorch SentenceTransformers vs. our DeepSparse-optimized models on both a 10-core laptop and a 16-core AWS instance. The benchmarks have yielded significant improvements in processing speed. For example, running the bge-small quantized model on the 10-core laptop, achieves up to a 3X increase in speedup. What's even better, is that when tested on a 16-core AWS instance, these models achieved up to a 5X improvement. 🤗 Updated model cards: bge-small-quant: huggingface.co/zeroshot/bge-… 6K+ downloads bge-base-quant: huggingface.co/zeroshot/bge-…  2K+ downloads bge-large-quant: huggingface.co/zeroshot/bge-… 2K+ downloads (#1 model for STS datasets on the MTEB leaderboard) Don't forget to check out the DeepSparse repo github.com/neuralmagic/deeps… for more information on benchmarking and running these models on the MTEB leaderboard. 💥 cc @neuralmagic
2
10
1,636
⚡Getting to Know the NPZ file format to Compress BGE Embedding Models ⚡ For One-Shot Quantization (INT8), Sparsify relies on the .npz format for data storage, a file format rooted in the mighty NumPy library.
2
1
4
441
⚡IT HAPPENED!⚡ There's a new state-of-the-art sentence embeddings model for the semantic textual similarity task on Hugging Face's MTEB leaderboard 🤗! Bge-large-en-v1.5-quant was the model I quantized in less than an hour using a single CLI command using Neural Magic's open
2
6
44
4,254
🚀🚀 Explore Sparsify's One-Shot Experiment Guide! Discover how to quickly optimize your models with post-training algorithms for a 3-5x speedup. Perfect for when you need to sparsify your model with limited time and improved inference speedups.🔥 **FYI, this is what I used to compress the bge-small-en-v1.5model for sentence embeddings . ** 1️⃣ **Experiment Overview**: Learn about the benefits of One-Shot Experiments and how they work for transformers and soon LLMs. 2️⃣ **CLI Quickstart**: Get started with a step-by-step guide on running One-Shot Experiments using the Sparsify CLI with a single command :) 3️⃣ **Data Prep**: Guide for understanding how to turn samples from your calibration dataset into NPZ files from your tokenizer's output. 4️⃣ **Next Steps**: Explore other Sparsify pathways, including Sparse Transfer and Training Aware Experiments. 5️⃣ **Link**: github.com/neuralmagic/spars… 💪 #AI #MachineLearning @neuralmagic
1
5
218
🚀✨ Run CodeGen on CPUs with this detailed Colab notebook! 📝 Explore how to sparsify and perform Large Language Model (LLM) inference using Neural Magic's stack, featuring Salesforce/codegen-350M-mono as an example. Dive into these key steps: 1️⃣ **Installation**: Quickly set up Sparsify and DeepSparse. 2️⃣ **ONNX Export**: Download and export the model to ONNX for optimization. 3️⃣ **Apply One-Shot Pruning and Quantization**: Optimize the model with Sparsify's FastOBCQ algorithm. 4️⃣ **Evaluate Accuracy**: Assess model accuracy using deepsparse.transformers.eval_downstream CLI for perplexity calculation. 5️⃣ **Inject KV Cache**: Improve inference speed with KV-caching in the ONNX graph. 6️⃣ **Run Inference With DeepSparse**: Execute text generation using DeepSparse. notebook: github.com/neuralmagic/docs/…
1
3
240
🚀 Text Generation User Guide 🚀 🔮 This DeepSparse user guide unveils the seamless process of running LLM models. Whether you're a seasoned developer or an AI enthusiast, you can generate output with just a few lines of code. ⭐Here's a sneak peek into what this guide covers: 1️⃣ Installation: Learn how to set up DeepSparse for text generation. Discover how to install DeepSparse's nightly build via pip, ensuring you're equipped with the latest AI capabilities. 2️⃣ System Requirements: Understand the hardware and operating system prerequisites for DeepSparse. From x86 AVX2 to ARM v8.2+, DeepSparse is designed to work on various platforms, including Linux. 3️⃣ Basic Usage: Unleash the power of text generation with DeepSparse's user-friendly interface. Construct text generation pipelines effortlessly, and generate text from prompts. Whether you're writing a story or crafting responses, DeepSparse makes it a breeze. 4️⃣ Model Format: Explore the different ways to use models with DeepSparse. Whether it's ONNX format, SparseZoo stubs, or local directories, you have options to suit your needs. 5️⃣ Input and Output Formats: Delve into the input and output formats of DeepSparse. Understand how to structure your input data and interpret the generated text. Explore options like streaming for real-time results. 6️⃣ Decoding Parameters: Customize your text generation experience with GenerationConfig. Control output length, sampling methods, temperature, top-k, top-p, and repetition penalties to shape the generated text to your liking. Link: github.com/neuralmagic/deeps… #llms #textgeneration @neuralmagic
6
284
😁 Happy to introduce one of the most comprehesive ChatGPT cheat sheets: a 30 pg. paper highlighting various prompts to manage ChatGPT for generating text. The document not only highlights what ChatGPT can generate but also how it can generate it! drive.google.com/file/d/1UOf…
1
6
459
Replying to @RedHat_AI
@neuralmagic is at the Intel Vision Conference... Check out YOLOv5 in action! 😍 And it turns out Michael Goin is human 😭😭😭 LET'S GO 🚀🚀 #IntelON