Michał Stradczuk (@smerftv) header
Michał Stradczuk (@smerftv) profile picture

Building something new @humalike | ex SWE @Revolut Core Platform | Co-Founder of FiveCity | CS Graduate @ MIMUW

Barcelona, Spain
Joined September 2019
Photo by Michał Stradczuk (@smerftv)Photo by Michał Stradczuk (@smerftv)Photo by Michał Stradczuk (@smerftv)Photo by Michał Stradczuk (@smerftv)Photo by Michał Stradczuk (@smerftv)Photo by Michał Stradczuk (@smerftv)Photo by Michał Stradczuk (@smerftv)Photo by Michał Stradczuk (@smerftv)Photo by Michał Stradczuk (@smerftv)Photo by Michał Stradczuk (@smerftv)
Michał Stradczuk retweeted
I'm currently blocked by three things: - ~80% of runs with Ant's models fail: it's either issue with the output limit / thinking never finishing or refusal to continue playing because they think the game is impossible to solve - there are a few design decisions that I'm unsure about and I'd love to discuss them with people that are more experienced than me in building solid benchmarks - grant/API credits would be nice; currently paying for everything out of my own pocket, but I'd love to benchmark non-frontier models as well If you can help (or know someone that can) with any of these, feel free to contact me. My last contact at Epoch ghosted me🫠 I'd love to wrap this up asap, since I already have 5 more ideas for reasoning benchmarks that should be relatively easy to build
Update on the puzzlescript benchmark: - game list is finalized: 40 diverse games selected out of 500+ games tested - it's massive, models take 150K actions on average per complete run; all games are at least 1h long (for experienced puzzle solvers), some of them are much longer - all levels within all games have proven solutions - score is a mix of completion% and relative scoring for action efficiency; it's subject to change, so I won't get into details; it's scaled so the maximum is 100, but if a new model shows up that's more action efficient, previous scores will drop down - my guess is that it needs another 4-6 months to be near-fully saturated by frontier labs - everything is run with harness with no web access and no code execution allowed; models are free to write notes if they want to I've tested all of the OpenAI models so far. And yes, Sol 6.2 is significantly better than Astra 6.2, while also being much cheaper. All of the runs are on xhigh. Why only OpenAI models you ask. Good question. Testing Anthropic's & Google's models has been beyond miserable so far. I get so many refusals, various errors and malformed answers that if I scored them under the same rules, they would all be below terra 6.1. I'll probably make another post about this with the hope that someone more experienced with benchmarking might give me some hints.
7
5
1
87
5,535
Michał Stradczuk retweeted
next month we'll hand over 100 samples of our new device a few slots covered already. unicorn founders & S tier funds included we'll record on camera every single device handed over (that I'll personally give) so far, we'll go to: - SF - NYC - London - Madrid - Barcelona - Warsaw this device is the endgame of what we've been working towards for over a year. extremely high-quality human-AI experiences we'll post some parts on my yt channel and more epic videos on Humalike youtube.com/@marti_carmona
8
5
1
33
1,577
I merged almost 1,000 pull requests in the last 2 weeks. -> We’re a team of 6 at @humalike -> We work 7 days a week, usually 10-12 hours a day. Since I joined few months ago, we’ve rebuilt our setup many times. Here are some thoughts on building with AI now. 🧵
15
5
79
12,487
What still sucks and how we deal with it: -> CI/CD queues. Agents push way faster than CI can run. We fight it with metrics and optimizing CI times, running the full CI only once, right before the last push, not on every push, and simply adding more runners. Of course that costs more infra, but if it lets us ship more, it's worth it. Velocity matters more than costs for now. -> Agents going down rabbit holes or looping on the same thing, sometimes for hours at night. We have hooks that wake them up, and the leader checks if they're actually making progress. And you need a really clear definition of done, or they'll decide on their own that they're done.
2
5
736
-> And of course, human review and actually understanding the code. Obv we're the bottleneck. Right now reviews are mostly there so that at least 2 people know what's going on in the code. PR summaries in simplified technical English with diagrams help a lot. But so much gets shipped that reviews mean constant context switching, and deep focus is super valuable now and there's less and less of it. Also people put less effort into reviewing AI-generated code. Your brain just doesn't want to go deep there. So we literally have a skill where AI quizzes us on how the code works. Tbh I don't have a good answer for how this should look. Nobody knows exactly how every line works anymore, and that's normal. You need to understand the high-level architecture, but also know where you have to dig in and where you can skip it and not waste time. Very few people have S-tier judgement here (more have good judgement, but I mean S/S+ tier), and right now it matters more than ever. A strong theoretical background helps, but it's still really hard.
1
7
682
Michał Stradczuk retweeted
Replying to @huma
@huma I’ll follow you until you sell your acc
1
1
11
608
Missing GTA VI launch to build our product. We better be worth $1B by the end of 2027 😠
1
3
9
1,744
Michał Stradczuk retweeted
3
4
29
1,049
POLAND MOUNTAIN 🇵🇱 @humalike
1
2
22
1,322
Michał Stradczuk retweeted
Replying to @renald_vie
there is no replacing a founder like Jensen. There never has been, and there can never be, a true succession for a founder like him. Nvidia ends today, and is replaced with a thing cosplaying nvidia. It will be run by custodians trying and failing to preserve the flame he ignited
299
1,146
194
14,884
1,379,736
pro tip: bring a clean bathroom towel on your trial day and you're basically hired
We’re hiring a Founding Ops Lead in Barcelona Our team (6 people in our 20s): six-figure side projects before 20, ex-pro esports, ex-special forces, Informatics Olympiad medals > Working at the frontier of human / AI interactions @humalike No playbook. RevOps, Recruiting, Contracts, Operations, etc. Chaotic and extreme. If it sounds exciting, DM me :))
1
2
17
4,215