Software Technologist, Historian, Karyakarta and Proud Indian. Hindu by birth and choice. Fascist TMC should be banned from my Bengal.
Houston, TX
Joined August 2011
- Tweets23.2K
- Following180
- Followers259
- Likes42.3K
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
This is the strangest property of LLM inference:
Your LLM's output depends on requests that aren't yours.
The same prompt can produce different logits depending on how many other requests share its batch on the GPU.
Here's what happens inside the GPU.
Most of an LLM's forward pass is spent accumulating sums.
Every matmul, RMSNorm, and attention step reduces thousands of floating-point values into a single output, so the order of those additions directly affects the result.
The catch is that floating-point addition isn't associative.
Every addition rounds the result to the nearest representable value, so (a + b) + c and a + (b + c) can produce different low-order bits.
fp16 has an 11-bit significand, so between 1024 and 2048 the spacing between representable values is 1.0.
If you compute 1024 + 0.4 + 0.4 left to right, each addition rounds back to 1024. If the two small values are added first, 0.4 + 0.4 gives 0.8, and 1024 + 0.8 rounds to 1025.
The order of accumulation depends on batch size, because kernels choose their parallelization strategy to maximize GPU utilization.
A GPU like the H100 has 132 compute units, and a small batch doesn't give each of them enough work.
So the kernel splits each long sum into chunks, adds the chunks on different units at the same time, and then adds those partial results together.
So a different batch size gives a different partitioning, a different accumulation order, and slightly different logits.
Usually the difference is negligible, except when two tokens are nearly tied.
In a batch of 8:
- "Queens" might score 12.3417
- "New" might score 12.3415
In a batch of 32, the rounding might shift them to 12.3414 and 12.3416.
If temperature is 0, argmax will pick "Queens" in the first case and "New" in the second. Every later token is conditioned on a different prefix, so the whole completion diverges.
So the nondeterminism isn't coming from sampling, but rather from the fact that your request's numerics depend on concurrent traffic you don't control.
This is pretty evident at Temperature 0, but the nondeterminism comes from the forward pass, not the sampler.
You can fix this by making every reduction batch invariant, meaning each row is reduced in the same fixed order regardless of how many rows share the batch.
This costs throughput, because the kernel can no longer select the optimal split for each batch size. It uses one fixed strategy even when that leaves compute units idle.
The major inference engines now expose this as a flag:
โ vLLM has VLLM_BATCH_INVARIANT=1, still in beta.
โ SGLang has --enable-deterministic-inference, at a 25-45% slowdown.
โ DeepSeek's DeepInfer has DEEPINFER_BATCH_INVARIANT=1.
If you run evals, RL rollouts, or anything that needs reproducible outputs, enable it and treat batch composition as part of your experimental setup.
And in production, the batch around your request never stays fixed. With continuous batching, the server adds and removes requests at every decoding step.
I wrote an article that explains how it works and why every major inference engine uses it.
Read it below.
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
RAG vs. Vectorless RAG, clearly explained!
In a standard RAG setup, documents are split into chunks, embedded, and stored in a vector DB. The query gets embedded too, and the top-k closest chunks go to the LLM.
The problem is that similarity and relevance answer different questions.
For instance, five annual reports from one company can likely follow the same template every year, and only the numbers can change.
So a FY2022 question can lie just as close to an FY2025 chunk, and the LLM might answer from the wrong year.
Cross-references introduce another problem. If a chunk mentions "see Appendix G" but looks nothing like the table in Appendix G, that table will never make it to the top-k.
Vectorless RAG drops the embedding model and the vector DB, and replaces similarity search with LLM reasoning instead.
The preprocessing step parses each document's layout into a ToC tree index. Every node holds a section title, its page range, and a short summary, and the whole tree fits in the LLM's context.
At query time, the LLM gets the query along with the tree. It reads the section titles and summaries and picks the few sections most likely to hold the answer.
For example, for a question about debt, it might pick the "Financial Summary" section and also "Appendix G", because its summary says the debt details are there.
The system then pulls those full sections and checks whether the evidence is enough.
โณ If it is, the LLM answers from the full sections, with page citations.
โณ If not, it goes back to the tree and opens more nodes.
The LLM also sees the chat history. So if you ask about assets and then follow up with "what about liabilities?", it goes back to the same report.
The trade-off is that every query now runs LLM calls over the tree.
Top-k vector search will be cheaper in that same run (although not guaranteed to be as accurate), and the tree only helps when documents have real structure.
So the two suit different data:
โณ Use a tree for long, structured documents like filings, contracts, and manuals.
โณ Stick with a vector DB for lots of short, unrelated text.
The diagram below depicts both flows.
If you want to dive deeper, I have also written a full breakdown of MIT's Recursive Language Models.
It covers why LLMs get worse as context grows and how RLMs let the model split a long document and work through it piece by piece.
Read it below.
That is equal to the population of America. Such poor performance by Modi and the โjumlaโ group. #ModiKahinNahiJayega
According to World Bank's latest report, India's extreme poverty has fallen from 27.6% to just under 2.6%.
And for Indians living under Rs. 400 a day (lower middle income), it has come down from 57.7% of the population to 16.1%.
Remarkable achievement by India given the rupee depreciated and the inflationary pressure of a fast growing economy.
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
The hooked cross symbol has had many names in many cultures since the old stone age. It was only made a symbol of antisemitism by Germans. As a symbol of antisemitism, it should be called the "hakenkreuz". No other culture used it that way. indiancentury.org/community-โฆ
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
This is why the Khelo India Sports Goods Manufacturing Scheme cleared by the Cabinet will be a game changer:
Dedicated manufacturing zones for sports goods-related manufacturing.
โจInvestment opportunities and MSME support.
โจResearch, innovation and skill development.
โจFocus on quality.
โจMore jobs and export opportunities.
It is a major push for Make in India, Aatmanirbhar Bharat and India's global sporting ambitions!
pib.gov.in/PressReleaseDetaiโฆ
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
The Nazi symbol of hate was a European hakenkreuz ("hooked cross"), not an Indian swastika. In fact, the one place you would never find a swastika is in Nazi propaganda. It wasn't Adolf Hitler who called his symbol a swastika. It was the New York Times.
indiancentury.org/community-โฆ
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
๐จ ITโS OFFICIAL NOW, FOLKS!!!!!!!! ๐ฅ
INDIA FINISH 4TH IN THE MEDAL TALLY AT THE ASIAN PARA GAMES 2026! ๐ฅนโค๏ธ
๐ฅ 30 GOLD
๐ฅ 35 SILVER
๐ฅ 49 BRONZE
114 MEDALS IN TOTAL!
INDIAโS BEST-EVER FINISH IN ASIAN PARA GAMES HISTORY!!!!!!!!!!!!!
WE DID IT, TEAM INDIA! ๐ช๐ฎ๐ณ
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
They told me it was impossible. ๐ช
New Yorkers waited decades.
But I led the fight to make Diwali a NYC school holidayโฆand we won!
This is my Diwali gift to the world.
We showed that Hindus, Buddhists, Sikhs and Jains are part of the American story.
#HappyDiwali New York!
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
๐จ James Talarico releases new ad SLAMMING Elon Musk for bankrolling Ken Paxton.
Talarico is running to overturn Citizens United.
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
How did a person in a fiercely tribal, stateless society end up founding a religion and a state that would transform the course of world history? My latest long-form essay, โThe Fourth Iteration of Monotheism: Born with a State โ The Origins of Islam in Arabia,โ explores the historical circumstances in the Middle East that made the rise of Islam possible, and also how its origins shaped its distinctive theology and political character.
We begin our journey in the Arabian peninsula of late antiquity, a world overshadowed by the mighty Byzantine and Persian empires, yet inhabited in its interior by fiercely independent tribes that had little use for states and vehemently hated kings. What made this tribal society function without a government? A seemingly trivial incident could provoke a confrontation between entire tribes. A single murder could unleash years of ritualised warfare and revenge killings, eventually settled through blood money. Through such stories preserved in early Islamic sources, we explore the intricate world of tribal loyalties, kinship, customary justice, and warfare that formed the society into which Muhammad was born.
We then explore the remarkable religious landscape of pre-Islamic Arabia, where ancient pagan gods and goddesses coexisted with Jewish communities, with Christian kingdoms encircling the peninsula, and other forms of monotheism. At a time when Christianity was at continuous ascent across much of the surrounding world, why did the Arabs of the แธคijฤz not simply embrace Christianity? Why did an entirely new monotheistic religion emerge instead? And how did Muhammad's proclamation of hard monotheism based on the one God worshipped by Abraham offend and challenge Arabian paganism, and reinterpret the Abrahamic religious traditions of his time?
The most important part of the story is Muhammad's transformation from a religious preacher in Mecca into the founder of a political community in Medina. In a society where tribal protection determined an individual's survival, the uncompromising proclamation of monotheism set off a chain of conflicts that ultimately necessitated political state-building. Through the biography of Muhammad and the document known as the Constitution of Medina, we explore how traditional tribal institutions were brought under a new, overarching religious community, the ummah, and how religious identity came to constrain and reshape the older tribal ties based on blood and ancestry. We also follow the military and political developments through which this new community came to dominate Arabia, laying the foundations for the Arab conquests that followed Muhammad's death.
Finally, we turn to the religion itself. How does Islam understand its relationship with Judaism and Christianity? Why does it claim to restore the original religion of Abraham? Why is the Qur'an understood so differently in Islam than the Christian Bible? And, most importantly, how did the circumstances of Islam's birth help shape the intimate relationship between religious and political authority, that would distinguish its historical development from that of Christianity? Unlike Christianity, which existed for three centuries before acquiring imperial patronage, Islam was born alongside a state. Understanding how and why this happened offers a window into its origins and its distinctive theology.
This is the fourth and concluding essay in my series tracing the historical evolution of monotheism, from ancient Egypt through biblical Israel and Christianity to the emergence of Islam in seventh-century Arabia.
If you enjoy history that goes beyond recounting events to explore the deeper social, political, and religious forces that shaped them, I hope you'll enjoy this historical journey!
Read the full essay here:
ramaseshanchandrasekaran.subโฆ
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
RT if @ConnerWeigman deserves a Heisman vote.
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
A new week, a new AP Top 25! What do you think of the latest rankings?
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
Ranking of the greatest mathematicians of all time:
1. ๐ฎ๐ณ Srinivasa Ramanujan
2. ๐จ๐ญ Leonhard Euler
3. ๐ฉ๐ช Carl Friedrich Gauss
4. ๐ด๓ ง๓ ข๓ ฅ๓ ฎ๓ ง๓ ฟ Isaac Newton
5. ๐ฌ๐ท Archimedes
6. ๐ฌ๐ท Euclid
7. ๐ฉ๐ช Gottfried W. Leibniz
8. ๐ฌ๐ท Pythagoras
9. ๐ฎ๐ณ Aryabhata
10. ๐ฉ๐ช Bernhard Riemann
Source: thetoptens
Am I wrong in stating that @gupta_rekha is just doing a wonderful job? Very focused to do everything that is needed.
๐ฅ | Delhi CM Rekha Gupta flagged off primary and secondary waste collection and transportation vehicles to strengthen door-to-door garbage collection across six wards of the Central Zone. Union Minister Harsh Malhotra and Delhi BJP leaders were also present at the ceremony.
@gupta_rekha | @CMODelhi | #RekhaGupta #HarshMalhotra #Delhi #GarbageCollection #TheStatesman
Una noche!!!!
Una noche en H-Town๐
๐ #18 Oklahoma State
๐ 6:30pm
๐ TDECU Stadium
๐บ FOX
๐๏ธ uhcoogs.co/tix
#BuiltByHouston x #GoCoogs
Bappa ๐บ๐บ๐ธ๐ฎ๐ณ retweeted
"13 crore names" is the number every TV debate on the Election Commission's SIR now starts from.
You get media to repeat that number ten times and it becomes the fact. Almost nobody has asked what is inside it...
@TheJaggi breaks it down. What the 13 crore actually contains:
- 2.8 crore listed as dead
- 6.3 crore flagged as possible permanent migrants
- 1 crore duplicate registrations
- ~3 crore simply absent from home when enumerators called
- Every one of these can be restored on appeal (except the dead..)
These numbers - the scale of them - is UNREMARKABLE at India's scale:
- Up to 1 crore Indians die every year
- The last SIR was 2002-04
- at least 10-12 crore voters have died since
- and most never came off the rolls (nobody files a deletion for the dead)
How about migrants?
- India's internal migrant population is around 45 crore
- 6.3 crore flagged as migrated is a fraction of that
- Wrongful deletions are probably under 50 lakh nationally
- half of one percent of a roll approaching 100 crore
The two mind traps doing the work:
FIRST
- Anchoring: the first big number sticks.
- Vinod Rai's Rs 1.76 lakh crore "presumptive loss" on 2008 telecom licences came with three lower estimates
-only the biggest survived
SECOND
- Loss aversion: a voter who fears deletion shouts louder than the 99% who were never touched
- Result: "millions and millions" disenfranchised (Justice Ujjal Bhuyan at Nalsar) and "90 lakh knocked out in Bengal to help the BJP" (Rohinton Nariman)
- neither claim came with a denominator
It's not like EC is all OK, they've also goofed up but not that badly:
- It never stress-tested its process for a country where every gross number is huge
- A once-in-two-decades purge was always going to throw up figures this size
- roll-cleaning has to be continuous
- The appeals route for wrongful exclusion has to be stronger than it is now
(Link below)
lets unite
Guys, make at least one post about Anojan Sivarasa.
Tag actors, celebrities, influencers, and politicians.
Keep posting until it trends and people start talking about it.
Save Anojan Sivarasa
#SaveAnojan
