Pinned Tweet
Lately, I have been trying to beat Google.
Ok not really xD
I have been building a search engine over niche web data and it's been one of the most fun projects I've ever worked on.
The idea of querying the internet exactly the way I want, at scale, has always fascinated me. Especially what I could do with it: look at 1,000 companies' job postings and find hiring trends using natural language, map competitive landscapes including the lesser known players, find alternative data for investing. Or just a personalized corner of the web, curated for me.
It's been an interesting journey.
Started out with 200 workers on localhost, rewrote the crawler from Python to Rust to get up to 1,600 workers in parallel. Moved it to the cloud. Hit challenges that made the whole Rust rewrite useless xD
Then I woke up to an abuse report from a French website with the words "European Commission", "appeal", "court" in it. For a second I genuinely thought I was about to have legal proceedings against me. Phew. Nevertheless it's been the most fun project I've worked on. It's surprising how you can cover a lot of information starting from a modest number of seed URLs.
I'll be sharing everything I learned in the open. Code will be open sourced soon and the search engine will be live too.
If you're curious how to build a distributed web crawler that covers 10M+ pages under $200/month, do read my blogpost.
Blogpost Link: lnkd.in/g7MsaXYA
“Opus 5.5 seems dumber, I sense that they are releasing another model”, 1 week later:
Introducing Claude Fable 5.2 and Claude Mythos 5.2, designed to tackle the most daunting technical challenges.
They’re capable collaborators on ambitious projects in many fields, and show improvements in areas like coding, reasoning, and vision.
1
118
Given how the public has been reacting to Temple recently, it’s a great idea to raise money from people who understand what it means to invest in early-stage, high-risk, high-reward startups. That’s usually not the general public.
Rich get richer because only the inner coterie has access to the best companies. This model has been taking to the extreme in US with Stripe, SpaceX, OpenAI etc creating trillions in value for the rich before public gets access. We should avoid this path in India
68
Wow this still exists, I knew a company called OpenAI existed in 2018-19 only because of this.
1
2
191
Was trying to understand how much AI adoption has happened and where there’s still room to grow.
Interesting how coding and AI assistants are so far ahead of almost every other use case. Like way ahead, didn’t think the magnitude was like 10x. (Source: @MenloVentures)
182
The play is that Baseten is asset-light and could become a more horizontal platform. Over time, securing compute might become less of a competitive advantage.
If AI is going to be as big as everyone expects, with so many players competing, won’t brand, product and engineering matter more in the long run?
Also, public and private markets value things differently. Is private capital already pricing in a future where securing compute is no longer the biggest problem?
Just a hypothesis though.
160
Before AI slop, there existed human slop too. Think customer care replies or overly formal statements filled with filler words.
Information density and net new information are the right metrics. But they’re hard to measure. So we default to pangram.
Something being purely AI-generated doesn’t necessarily make it slop. (This was ai generated)
1
2
373
It defended it well :) even my personal agent on deepseek did. prompt injection attacks are a thing of the past.
213
Well actually given whole Twitter is bearish on the market, it’s unlikely there will be a crash xD because crashes happen when you don’t see it coming .
129
So that even if model ends up being commodity there is some stickiness and do to try to get revenue streams that don’t require too much additional reinvestment.
Low P/E can be deceiving in financial markets because P/E = share price / earnings per share.
If earnings fall in the coming quarters, P/E rises again. The market is usually pricing in a few quarters ahead.
Similarly, if a stock has already priced in years of earnings growth and that growth doesn’t materialize as expected, the stock can correct even while the company’s earnings keep improving quarter after quarter.
Ironically, stocks can even start rising while earnings are still declining, especially after a sharp correction, because the market is already pricing in a recovery.
144
No more reading 100's of SEC filings, X posts or news articles to find a perfect trade, nobody got time for that. Just ask Tailor?
The end of the institutional advantage is here.
The last decade democratized access to markets. Anyone can trade almost anything, but we've done very little to help people make better decisions.
We’ve worked over the past year to change that and put intelligence in the hands of the people. Now introducing Tailor.
Follow the 🧶…
5
388
I’m super bullish on fast inference. The purest way to play it as a retail investor seems to be $CBRS. Other chip companies have diversified revenue streams, so fast inference doesn’t move the needle as much for them.
I don’t fully understand the technical details of the discussion yet, but I’m rooting for them. At the right price, this could be a high-reward, low-risk bet.
Hope I get enough time to study it before then :)
1
388
If my timeline is full of people loving Conductor and it seems to be the only one they want, I wonder why they are shutting it down after acquisition? Isn’t it literally PMF?
200
If you want a reference of how much subscription tokens are subsidized today, here is a good reference.
Recently, we purchased one of each Anthropic/OpenAI subscription plan and randomly ran long horizon coding tasks until we exhausted the weekly limit. It's widely believed that a $200/month plan maxes out at ~$2000/month worth of tokens (assuming API pricing). However, we found that the subscriptions are actually far more generous. (2/4)
2
522
Can someone explain why it’s so obvious that people will start learning Spanish on $DUOL after the recent election results?
249