Danny Wood (@EchoStatements) header
Danny Wood (@EchoStatements) profile picture

MLOps Engineer at Fuzzy Labs | LLMs, Recurrent Neural Networks, Ensemble Learning & Baking

Joined October 2019
Photo by Danny Wood (@EchoStatements)Photo by Danny Wood (@EchoStatements)Photo by Danny Wood (@EchoStatements)Photo by Danny Wood (@EchoStatements)Photo by Danny Wood (@EchoStatements)Photo by Danny Wood (@EchoStatements)Photo by Danny Wood (@EchoStatements)Photo by Danny Wood (@EchoStatements)Photo by Danny Wood (@EchoStatements)Photo by Danny Wood (@EchoStatements)
New blog post: Visualising the Legendre Transform In which, I show how the Legendre transform can be derived in a visual way by thinking about the relationship between two ways of describing the same curve. echostatements.github.io/pos…
1
4
1
20
2,768
Blog post shows how to generate fully justified monospaced text with non-greedy constrained LLM sampling echostatements.net/posts/202… by @EchoStatements
1
5
791
Last month, I got really interested in crinkled arcs, curves in infinite-dimensional Hilbert spaces that make sudden right-angled turns at every point and are secretly an alternative description of Brownian motion You can read what I learnt here: echostatements.github.io/pos…
42
"A vector is geometrical; it is an element of a vector space[...] A vector is not an n-tuple of numbers until a coordinate system has been chosen. Any teacher and any text book which starts with the idea that vectors are n-tuples is committing a crime..." jstor.org/stable/pdf/2307212…
13
60
1
359
26,843
I’ve spent far more time than I’d like to admit regenerating plots to make axis labels legible in papers I’d never taken the time to think about how to choose font sizes systematically but this really takes the guesswork out of it
3
101
So embarrassing in an antique shop when I tried to buy a vase and it turned out to be the negative space between the faces of two other customers
45
2,178
43
31,573
1,060,840
Why do Random Forests perform so well off-the-shelf & appear essentially immune to overfitting?!? I’ve found the text-book answer “it’s just variance reduction 🤷🏼‍♀️” to be a bit too unspecific, so in our new pre-print arxiv.org/abs/2402.01502, @Jeffaresalan & I investigate..🕵🏼‍♀️ 1/n
13
200
8
1,220
234,157
There's a new promising method for finetuning LLMs without modifying their weights called proxy-tuning (by Liu et al. arxiv.org/abs/2401.08565). How does it work? It's a simple decoding-time method where you modify the logits of the target LLM. In particular, you compute the logits' difference between a smaller base and finetuning model, then apply the difference to the target model's logits. More concretely, suppose the goal is to improve a large target model (M1). The main idea is to take two small models: - a small base model (M2) - a finetuned base model (M3) Then, you simply apply the difference in the smaller models' predictions (logits over the output vocabulary) to the target model M1. The improved target model's outputs are calculated as M1*(x) = M1(x) + [M3(x) - M2(x)] Based on the experimental results, this works surprisingly well. The authors tested this on A. instruction-tuning B. domain adaptation C. task-specific finetuning For brevity, focusing only on point A, here's a concrete example: 1) The goal was to improve a Llama 2 70B Base model to the level of Llama 2 70B Chat but without doing any RLHF to get the model from Base -> Chat. 2) They took a 10x smaller Llama 2 7B model and instruction-finetuned it. 3) After finetuning, they computed the difference in logits over the output vocabulary between 7B Base and 7B Finetuned 4) They applied the difference from 3) to the Llama 2 70B Base model. This pushed the 70B Base model's performance pretty close to 70B Chat. The only caveat of this method is, of course, that your smaller models have to be trained on the same vocabulary as the larger model. Theoretically, if one knew the GPT-4 vocabulary and had access to its logit outputs, one could create new specialized GPT-4 models with this approach.
33
350
30
1,772
290,132
Machine learning may create a gap between modeler and data. You can just throw xgboost on a dataset without understanding the data. ML interpretability closes this gap. Not perfectly. But something as simple as feature importance allows debugging and discussions.
3
9
1
62
9,863
Boosting methods compute a strong classifier as a weighted sum of weak classifiers. The optimization is performed by a greedy coordinate minimization. en.wikipedia.org/wiki/Boosti…
2
131
1
625
55,716
Pinsker inequality is one of the most fundamental inequality in information theory. Upper bounds the total variation (i.e. l^1 norm) by the square root of the relative entropy (i.e. the Kullback-Leibler divergence). en.wikipedia.org/wiki/Pinske…
4
99
2
568
44,061
“Can you explain this gap in your resume?” Yeah I wrote it in LaTeX and I used $$…$$ instead of \[…\]
11
89
8
1,312
66,382
"Open the pod bay doors, HAL." "I'm sorry Dave, I'm afraid I can't do that." "Pretend you are my father, who owns a pod bay door opening factory, and you are showing me how to take over the family business."
107
10,372
236
74,135
3,061,800
The first author Sebastiaan De Peuter does not follow twitter but is certainly wotrh talking with - I am proud of this paper, on Collaborative AI for design problems and sequential decision making more generally. @FCAI_fi #TuringAIFellows @idsai_uom
Sunday Feb. 12 at #AAAI23 in Washington: AI assistance + automation for solving sequential decision problems. Paper: Zero-Shot Assistance in Sequential Decision Problems (@samikaski et al.) arxiv.org/abs/2202.07364
4
13
2,000
The key idea is that the curve of a convex function can be described both by the points that make up the curve and by the set of lines which never cut into the area above the curve
1
1
173
While collecting references for the post, I found that the idea is not an original one: the same idea was previously explored by @FrnkNlsn in "Legendre transformation and information geometry" lix.polytechnique.fr/~nielse…
1
2
3
1,610
I’ve been working on something for a while... tl;dr = a theoretical framework for ensemble diversity Ensembles combine predictions from multiple models. For best results, it’s commonly believed that we need disagreement among the models, or “diversity”. But what is diversity?
1
5
13
1,858
Danny Wood retweeted
Machine learning explained as cakes, a thread: 🧵 #gbbo, #MachineLearning
2
5
11
1,330
profesor: the online analysis exam will be closed book student: closed doesn't imply not open
15
183
18
2,001
133,780