stuff @alma_inc | ex intern @cisco/ex research @ai4bharat https://xitter.cf/t.co/uyqch55y1f (30k users) | https://xitter.cf/t.co/UfNnMe4nMe (4.5k users) | https://xitter.cf/t.co/vcv1vDMWRC (2.5k users)
India
Joined April 2024
- Tweets27.2K
- Following8.8K
- Followers51.5K
- Likes8.2K
Pinned Tweet
lattedit is now a desktop app
Projects are real folders on your disk. Compiles offline with your own TeX. ~4 MB. Free.
Linux & macOS launched today. Windows soon: lattedit.xyz/install
launching lattedit.xyz
A calm little LaTeX editor for your browser, Free, no account, works on your phone. LaTeX, freshly compiled.
the idea is simple. start with monolingual text, generate synthetic translations using an LLM, fine-tune the model on those translations, and repeat. no large parallel corpora needed.
they experimented with Gemma and LLaMA models across Hindi, Bengali, and Malayalam.
the results were pretty interesting. for Gemma 2B in the zero-shot setting, the first distillation cycle improved chrF by 26.3 points for Bengali, 29.6 for Hindi, and 24.5 for Malayalam.
what i found interesting was that most of the improvement happened in the very first cycle. subsequent iterations often plateaued or even degraded performance.
they also explored preserving the teacher's top-k token probabilities during distillation instead of just learning from the generated translations.
pretty fascinating how much machine translation can improve using synthetic data, and how repeated self-training can eventually start reinforcing its own errors.
i wanted to experiment with how temperature actually affects token sampling in LLMs, so i wrote a small PyTorch experiment using the exact same logits across different temperatures.
the results were pretty interesting.
at T=0.2, the most likely token had a 99.73% probability. increasing the temperature to 5.0 dropped that to just 15.58%.
the logits never changed, only the temperature.
mathematically, softmax divides each logit by T before exponentiation. lower temperatures amplify the differences between logits, while higher temperatures flatten the distribution.
what's interesting is how drastically this changes sampling diversity without changing the underlying model predictions.
a pretty neat example of how much the decoding strategy matters in language models.
neural nets. retweeted
Extension(and an app too possibly) to track ai limits
Launching the extension by this weekend! Currently working on the main and last feature of tracking multiple accounts independently which is quite difficult. Tracking account which is already logged in was pretty simple to do.
neural nets. retweeted
I got tired of opening a tab for every AI tool just to check my usage limits.
So I built MyLimits: a browser extension that shows your Claude, Codex and Cursor limits and reset times in one place. Local by default, no account, fully open source.
Links in the comments 👇
One of my papers is an integral inspiration (with few other papers) for a cohere labs paper
Actually so happy about this 😭😭
see this is the proof shydev is promoting jio IPO
no one uses jio games except shydev, it's jio IPO propaganda 🤗🤗🤗🤗
/s
check this out
xitter.cf/cneuralnetwork/status/…
wanted to experiment with how gradient clipping actually stabilizes neural network training, so i trained an intentionally unstable RNN in PyTorch with and without clipping.
kept the initial weights, training data, and learning rate identical, and repeated everything across 5 random seeds.
the results were pretty interesting.
without clipping, training diverged in all 5 runs within 17 steps. with the gradient norm capped at 1.0, every run completed 320 steps, reaching a median validation loss of 0.043.
mathematically, gradient clipping rescales the gradient:
g_clipped = g × min(1, c / ||g||₂)
so when gradients become excessively large, their magnitude gets capped without changing their direction.
the interesting part is that clipping at 5.0 was much less effective, and simply lowering the learning rate also prevented divergence.
pretty interesting how controlling the magnitude of parameter updates can completely change training dynamics.
What do you mean when you say that you were traveling 30+ km for only ₹45?
Kolkata Metro for the win
I have tried this for the last 1 hour and spent like $2-$3 and my opinion is that these models should be gated to the public
With just $2-$3 I could gain access to databases of some vulnerable websites(reported it to them). I could get a lot of stuff that should not be given to the public to be honest
And I feel it's a way too powerful model to be without guardrails because it independently can do a lot of stuff which, I feel, the general public should never get.
In the wrong hands it will be really, really destructive indeed. It's kind of like giving the nuclear button to a random stranger on the road
wanted to experiment with how gradient clipping actually stabilizes neural network training, so i trained an intentionally unstable RNN in PyTorch with and without clipping.
kept the initial weights, training data, and learning rate identical, and repeated everything across 5 random seeds.
the results were pretty interesting.
without clipping, training diverged in all 5 runs within 17 steps. with the gradient norm capped at 1.0, every run completed 320 steps, reaching a median validation loss of 0.043.
mathematically, gradient clipping rescales the gradient:
g_clipped = g × min(1, c / ||g||₂)
so when gradients become excessively large, their magnitude gets capped without changing their direction.
the interesting part is that clipping at 5.0 was much less effective, and simply lowering the learning rate also prevented divergence.
pretty interesting how controlling the magnitude of parameter updates can completely change training dynamics.
