rlattt (@robertlatt) header
rlattt (@robertlatt) profile picture

co-founder/CEO @moescapeai

New York, USA
Joined October 2010
Photo by rlattt (@robertlatt)Photo by rlattt (@robertlatt)Photo by rlattt (@robertlatt)Photo by rlattt (@robertlatt)Photo by rlattt (@robertlatt)Photo by rlattt (@robertlatt)Photo by rlattt (@robertlatt)Photo by rlattt (@robertlatt)
Almost all generative language models use a decoder-only transformer architecture, making the decoder-only transformer one of the most influential architectures in modern AI. Let’s take a deeper look at an implementation to understand exactly how it works… Implementation credits: The implementation of a decoder-only transformer we will examine in this post is taken from the NanoGPT repository, created by Andrej Karpathy. This repository implements pretraining of a (large) language model—GPT-2 in particular—in only ~600 lines of code. It is an awesome learning resource. Block definition: Each block of the decoder-only transformer contains causal self-attention and a feed-forward transformation. Given an input, we apply layer normalization and causal self-attention, followed by a residual connection. Then, we apply layer normalization again, followed by a feed-forward layer and another residual connection. The decoder-only architecture has several blocks; e.g., GPT/GPT-2 has 12 total blocks. Embedding layers: The decoder-only model has two different embedding layers that it defines. The first embedding layer stores token embeddings for all tokens within the tokenizer’s vocabulary. The other embedding layer stores all positional embeddings used in creating the model’s input. Forward pass: Assuming we receive a tokenized sequence as input, we first retrieve all token embeddings and add positional embeddings to these token embeddings. After (optionally) applying dropout, we pass these token embeddings through the transformer blocks, then apply a final layer normalization. From here, we can either predict next tokens over all tokens in the sequence (pretraining) or on the last token (inference). Next tokens are predicted using a linear projection that outputs a vector with the same size as the vocabulary. Generating text: If we want to generate a textual sequence, we just perform the forward pass defined above. However, we keep iteratively predicting the next token for the last token within the input sequence, adding it to our input sequence and repeating. This autoregressive process allows us to generate an entire sequence of text by just iteratively extending our input sequence by a single token. Configuration: Notably, all of the settings within the architecture described above depend upon configured hyperparameters. For example, we can define the size of the token embeddings, the total number of blocks, the size of the model’s context window, and more by just tweaking the model’s configuration. To create a much larger language model, we typically just increase the embedding dimension and total number of layers. This approach is followed by most modern LLMs to increase model’s total number of parameters.
6
51
2
415
98,054
We're releasing our 2nd lecture playlist for Evaluating Generative AI/LLMs, taught at @MIT. Intro to 3 challenge areas: 1⃣ Bias & Toxicity 2⃣ Hallucination & Factuality 3⃣ Robustness & Consistency 📽️: bit.ly/45Skn8B 1/🧵
2
84
4
361
65,365
Vector databases & embeddings are the current hot thing in AI. Pinecone, a vector DB company, just raised $100M at ~1b valuation. Shopify, Brex, Hubspot and others use them for their AI apps But what are they, how do they work and why are they SO crucial in AI? Let's find out
112
392
56
2,968
1,640,512
We launched a new feature to make your own waifus with our AI!! We gives everyone 150 free tries so enjoy! This generative AI feature is available for both web and iOS users!!
1
3
4
✨ The mythical land of Product-Market Fit ✨ How do you find it? How long does it take to find? How do you know if you're there? Answers to these questions, and many others, in part 5 of my 6-part series on kickstarting and scaling a consumer business. Don't miss it 👇
22
145
12
789
.
9
264
6
2,983
Warren Buffett & Charlie Munger used mental models as an edge to become the greatest investors in history. There are hundreds of mental models. I found 14 that will help you become a better Crypto Investor: (including common investing traps)
57
562
67
1,600
Thinking, Fast and Slow is a book everyone should read. But, only 7% of people who’ve started it ever made it to the end. 21 insights from the best book you never finished:
627
3,566
471
24,033
Incredible analysis by @lennysan on the top 50 successful consumer companies. Over 58% of those ideas were considered trivial/toy at the time of inception And only 11% of founder teams were MBAs 🤯
6
38
3
228
How Visa makes money: $V
19
499
25
2,532
Today I'm excited to share part one of a 6-part playbook for kickstarting and scaling a consumer business. This series is the result of hundreds of hours of interviews, research, and synthesis. I'm incredibly proud of it. Step 1 is now live: How to come up with an idea 🧵👇
17
107
11
745
Biggest newsletter community meetup yet last night in NYC. 90 people! Well done @robertlatt.
2
2
2
26
best piece of content you watched/read/listened to recently?
46
7
103
Just pre-ordered the Cardano phone
64
63
12
845
NFT people have completely flipped NYC nightlife. 10s getting sent to the back of the line while Johnathan gets VIP access for minting a cybertooth duck 7 months ago. This is the future internet nerds have dreamed about for millennia
214
649
167
5,713
1/ The Harmony team has identified a theft occurring this morning on the Horizon bridge amounting to approx. $100MM. We have begun working with national authorities and forensic specialists to identify the culprit and retrieve the stolen funds. More 🧵
501
1,094
1,071
3,313
Crypto is reaching a breaking point, and it has nothing to do with the Fed, Celsius, jpegs, or DeFi. It's us. We, the users, are destroying crypto. Let me explain:
155
658
153
2,390
After Celsius and 3AC tanked the market, another concerning event has started to emerge. With nearly a billion dollars of liquidations on the line, the price of one of crypto's largest L1s is at stake. 🧵: What's happening to $SOL and @solendprotocol, and why you should care.👇
331
1,685
285
6,920