ml • search • abstraction • cog sci
Texas
Joined December 2021
- Tweets679
- Following629
- Followers138
- Likes4.8K
Fun thread
Ok folks, informal poll: use the little browser app at the link below to say how you think the sets of physical systems, living beings, and minds fit together in a Venn diagram:
three-circles.space.fast/
One of the best things about AI code gen is not just that I can make local tools quickly, but that they can also be tailored and made to look nice. It's such an improvement to my daily work.
Behind the phrase vibe code is a masked problem. There is a continuum of good to bad engineering and adding AI to bad engineering will mire a team in problems and slow everything down.
This is a nice analysis, as expected, from Hamel and Antaripa. Good illustration of more traditional analysis in the newer domains of LLM evals and decision models.
For critique I would say it's pushing the boundaries of decision models by using a very broad prompt and single decision vs breaking the trace eval into several smaller decisions.
Really appreciate the ongoing work from @HamelHusain and @doesdatmaksense. They are so helpful for advancing this eval field.
hamel.dev/blog/posts/decisio…
Patterns in System One space:
. Zero shot general classifier
. Zero shot general slot filling
...
What's next?
one thing that made this click for me — with decision models, you have to define categories and it returns probabilities that the input matches those categories.
but so much of the data we interact with is completely new to us. how can you define categories for something you've never read?
with clairic, you can extract values directly from the inputs, without any premeditated categories. the pipeline becomes clairic to extract values → clef to make decisions with those values → LLMs to take actions on those decisions. all of these models run on Workers AI.
This space of possible models was alluded to by @CompleteSkeptic on the @latentspacepod recently. It's been on my mind a lot. It's going to grow faster than the community is able to orient and integrate the ideas. Very interesting area of research right now.
Using the word Taste when talking about AI seems like sidestepping the hard work of describing more specific skills which are a value add.
Phrases that need to see increased use given some techniques growing in popularity:
. Black box optimization
. Bayesian optimization
. Derivative free optimization
. Stochastic hill climbing
A huge part of leveraging AI is being able to evaluate and encapsulate risk.
High risk? Slow down and be deliberate.
Low risk? Just rip and lfg
Building custom local tools is a crazy unlock. It's so useful for analysis, research, and development in data centered domains.
I was doing this at a startup before code gen with AI took off. It was a useful tactic for me at the time. Looking back, it was so slow and primitive compared to what you can do now.
Codex with training wheels?
Cursor in learning mode?
In coding agents, I would like to see a mode that only allows creating one function at a time.
The early days of coding agents were only that good. Guide it a function at a time. Carefully arranging modules and then populating them. It was an interesting experience. The cursor tab model was similar.
Having a new mode could orient engineers to learn good composition practices while also still getting some advantages of code gen.
Imagine putting junior engineers, or any engineer entering a new domain, on codex with training wheels and encouraging more learning and synthesis.