Despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. Galactica is also significantly less toxic than other language models based on evaluations.

We train for over four epochs and experience improving performance with use of repeated tokens. For the largest 120B model, we trained for four epochs without overfitting.

Galactica performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%.

We release our initial paper below. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. Includes scientific text and also scientific modalities such as proteins, compounds and more. galactica.org/paper.pdf

🪐 Introducing Galactica. A large language model for science. Can summarize academic literature, solve math problems, generate Wiki articles, write scientific code, annotate molecules and proteins, and more. Explore and get weights: galactica.org

SoTA in the LRA benchmark is achieved by Mega (Ma et al. 2022), a single-head gated attention mechanism equipped with exponential moving average to incorporate inductive bias of position-aware local dependencies into the position-agnostic attention mechanism. 8/10

There’s also interest in assessing speed and memory footprint of ML models. Tay et al. (2020) reported efficiency results for a set of models. Performer and Linear Transformer make a better trade-off in terms of speed and performance with reasonable memory usage. 7/10

There exists many solutions that have been built and designed to perform well at long-range modeling. Some notable transformer-based models include BigBird, Performer, Longformer, Synthesizer, among others. Check performance of these models here: paperswithcode.com/sota/long… 5/10

The LRA benchmark includes examples of sequences that range from 1K to 16K tokens. It consists of different data types and modalities such as text, synthetic images, mathematical expressions and so on. (Figure below shows ListOPs task included in the LRA benchmark) 4/10

One of the most recent challenging benchmarks to measure capabilities of language models is called BIG-bench. It already includes 204 tasks! It includes tasks like detecting misconceptions, dark humor detection, high school physics/biology, and more. paperswithcode.com/dataset/b… 2/6

Below are examples that illustrate the difficulty of the tasks in MMLU. The problems are framed as multiple choice questions where a model is trained to select the right answer from the list of possible answers. 3/8

MMLU aims to evaluate a model on world knowledge and expert-level problem solving. It covers 57 subjects across STEM, humanities, social sciences, and more. Tasks include abstract algebra, astronomy, high school biology/physics, medical genetics, virology and more. 2/8

📈MMLU: a benchmark to evaluate the problem solving capability of language models. In this thread, let's take a closer look at the latest progress on the MMLU benchmark. ↓ 1/8 paperswithcode.com/sota/mult…

10) Deep Patch Visual Odometry - a new deep learning system for monocular visual odometry that achieves 2x-5x real-time speeds; outperforms previous works on several benchmarks in terms of accuracy and speed. paperswithcode.com/paper/dee…

9) YOLOPv2 - an effective and efficient multi-task learning network for performing faster and better on tasks such as traffic object detection and lane detection. paperswithcode.com/paper/yol…

8) PeRFception - leverages NeRF variant to create large-scale implicit representation datasets for perception tasks. paperswithcode.com/paper/per…

7) MinVIS - a minimal video instance segmentation framework, without video-based training, that produces state-of-the-art performance and is comparable to fully-supervised approaches. paperswithcode.com/paper/min…

6) Flow-Guided Transformer - proposes a Transformer-base model leveraging motion discrepancy from optical flows; this approach helps to instruct attention retrieval in transformers for video inpainting. paperswithcode.com/paper/flo…

5) LLM.int8() - a new quantization procedure that allows large scale model checkpoints (16/32-bit) to be loaded and converted to Int8. This allows access to large language models (LLMs) that could not be accessed due to limited GPU memory. paperswithcode.com/paper/llm…