@System64fumo

🇺🇸🇮🇱 Icy fairy goofing around with embedded systems for fun!

 Israel
github.com/System64fumo
Joined May 2016
  • Tweets545
  • Following115
  • Followers93
66Media
Hello everyone! I'm pleased to publicly announce a project of mine that I've been working on for the past 2 months. github.com/System64fumo/kapp…

GitHub - System64fumo/kappai: High performance inference engine

High performance inference engine. Contribute to System64fumo/kappai development by creating an account on GitHub.

github.com
3
3
1
30
2,348
Did some work on kappai today and i think i did well? Still very W.I.P, Chat did i cook? 👀
1
22
I'm not pulling off any special tricks here or sabotaging llama.cpp by having some load in the background, Both engines are running with the same parameters same conditions, etc.. Unfortunately (Or rather fortunately) this isn't the end, There's still plenty of headroom left.
10
Compute is expensive.. Let's make it cheaper, And local!
1
3
95
I cooked hard, Here are some things that landed in kappai recently: GLM 5.3 Flash support OpenGL ES 3.1 backend Several new IQ quants TUI setup/run screen & many more There's also a new matrix and discord for this project, Come join us! github.com/System64fumo/kapp…

GitHub - System64fumo/kappai: High performance inference engine

High performance inference engine. Contribute to System64fumo/kappai development by creating an account on GitHub.

github.com
1
32
I don't actually use discord btw so if you want to contact me do so on matrix.
11
Holidays are over, Time to cook again. Where are all my cracked C devs at? 👀
21
Ok so i didn’t expect to get this much attention, But damn those are a lot of eyes. 👀 I’m out for holidays right now so sorry for the lack of updates. Thank you Bee fumo! 🐝
2
62
I have some pretty telling evidence that Union Alpha is some sort of GLM model. @Zai_org Can we please "pace the frontier"? /s And while you're at it could you kindly fix the glaring security hole? I wanted to work not dive down this rabbit hole, k? thx.
1
115
Happy #CirnoDay!
2
102
4
83
771
10,357
I don't know what others think but i find it quite amusing that i made a 12 CPU (CD8180) go toe to toe with a 20 core iGPU. (M4 Pro mac mini) Wish i had proof for this but the proper comparison with GLM 5.2 gave me: 0.38PP and 0.32TG Numbers from colibri:
1
149
Also please don't get me wrong, I'm not throwing shade at colibri or llama.cpp. They're great projects, I just can't benchmark properly since i don't have disk space nor memory for either. Sorry if that came off wrong. 😅
78
Humm maybe claiming to have a performance oriented inference engine followed by showing sub token performance on a pretty banger aarch64 system doesn't quite showcase the improvements.. Fine here are, Some direct 1:1 comparisons with llama.cpp on rk3588. 🧵
1
1
133
more replies
I also have an NPU backend that does indeed run here but unfortunately i can't publicly ship that code since my license is M.I.T and the NPU code is based on GPL.
1
33
Note: Just like my last post i had to re-create this post too, Again sorry!
27
Before i conclude i want to be crystal clear about some things. 1. I am not an AI/ML engineer, I am just a dude who likes to optimize things. 2. The codebase is primarily generated using @Zai_org's amazing models and i couldn't have done it without them.
1
1
128
PS: Had to re-upload the images and re-create the post because twitter's drafts made them super low resolution, Sorry about that! 😭
1
113
Load more