yaatehr retweeted
Today we establish that AI is as good as expert human tutors for immediate GRE learning gains (p=.015, n=2,383 students, study conducted July-Sep 2026).
One hour with an expert GRE tutor: $75.
One equivalent hour with an AI tutor: 7 cents .
918x cheaper, equivalent learning (p=.044, n = 140).
In 5 of 7 academic topics, the top AI tutor beat the expert human tutors on average.
How we measured it: pre-test -> 1 hour with (AI or human or no) tutor -> post-test.
There are no LLM judges. Learning gains are post-pre. Everything is real students.
Why we did this:
> 1 trillion USD is being spent per year on machines making machines better. StudentBench shows us how to shift these resources so machines also make humans better.
Why I did this:
I grew up in rural Kentucky. My dad, grandad, and great grandad were all mailmen. Getting into a good college changed my life. Now we have done a rigorous study to help AI labs evaluate LLMs and help show how bright AI can be for students of all incomes.
I believe in good science that leads to the democratization of opportunity.
The research paper behind StudentBench: arxiv.org/abs/2609.28470
Data, code, details: in this thread
If you're wondering what I'm selling, sorry to disappoint: we released all study data for free on Hugging Face, and made studentbench.org free to use. Enjoy.
lots more in the video / thread!!
Alright after digging seems like this is largely right. How I’d think about this new Jev model:
It is a huge speed up, but it’s not verifiable
It’s an on the fly converter from natural language to semantic categories
Use it where you would have used embeddings: rag, etc
I think that It is a gliner2 with slight modification github.com/fastino-ai/GLiNER…
Finally someone willing to post a paper on Jev. So far all I’ve seen is intuition and hand waving from @CompleteSkeptic. I’ll believe it when I understand it
Jev is a classifier with a twist: you can freeform declare N categories as a parameter-input, and then your proper tokenised input is classified accordingly. Using arxiv.org/pdf/2407.02423, finding plausible variant attn mechs to achieve this was fast.
yaatehr retweeted
im not asking for the benefits of a classifier AI
im asking how is this different from a classifier AI
which does the same input -> probability score on choices
main difference i see: we normally purpose train classifiers, and this is adapted on the fly
yaatehr retweeted
Jev is a classifier with a twist: you can freeform declare N categories as a parameter-input, and then your proper tokenised input is classified accordingly. Using arxiv.org/pdf/2407.02423, finding plausible variant attn mechs to achieve this was fast.
yaatehr retweeted
last lineup of tonight is @solvibestation @MateoCorrea88 - they run a robust RPC & Validator business.
We’ve now had 11 different founders use the Wall st venue for interviews, podcasts, live demo sessions.
If you are a builder on @solana - come to the venue & tell us what you’re building.
Last date Nov 5th.
yaatehr retweeted
Humans invented software to free ourselves from manual work.
Somehow, we still waste billions of hours clicking across websites.
Today, @joinkaizen launches to fulfill that original promise.
Announcing our $4M raise from @8VC and the public launch of the Kaizen platform.
yaatehr retweeted
Goodbye Hallucinations! Today, Cleanlab launches the Trustworthy Language Model (TLM 1.0), addressing the biggest problem in Generative AI: reliability.
technologyreview.com/2024/04…
