Feiteng (@FeitengLi) header
Feiteng (@FeitengLi) profile picture

Independent Researcher Speech · Image · Video · LLM multimodal generative models Research→Infrastructure→Built AI algo serving 250M+ users 公众号: Generative AI

Singapore
Joined November 2016
Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)
Qwen3.8-27B below 3 bpw: we’ve released two GGUF versions at 2.96 / 2.48 bpw, weighing in at just 9.71 / 8.20 GiB, both with MTP included. Both achieved 100% on AIME 2026 in our evaluation. We’ve also slimmed down the vision encoder and resolved its GGUF quantization compatibility issues—no modifications to llama.cpp required. The Q4_K version is just 314.46 MiB, 64.59% smaller than BF16, while retaining 0.99537 cosine similarity to the original HF FP32 vision outputs.
3
5
1
39
6,699
最近看到好多的全双工 speech-to-speech 模型是参考 moshi 思路构建的 moshi 于去年 2025 年开源了 finetune 代码 github.com/kyutai-labs/moshi… 不禁回想起 2024 年构建全双工语音模型也是参考 moshi 做的设计,而负责训练的同学三个月不提交训练代码且一直训练不出合理智力的模型; 不可忍受 强烈要求必须提交代码,一个周末我通过 review 抓出来了两个错误 1)数据导出错误 2)训练 RQ-Transformer 的数据排列错误 导致 loss 计算不合理 训练推理不一致 改完这两个错误 infra 同学随便就训练出能对话的模型了
11
29
276
12,078
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
68
292
99
2,535
524,904
how should multimodal models encode visual input? → pretrained encoder? → encoder-free? → before the LLM? published models bundle these choices together. we trained 200+ models from scratch to see which of these choices matters only one of them does 🧵 [1/n]
23
136
27
850
125,459
今天 六个月半年整,头一次经历这事🤢 前司是平薪入职(实际降薪 没补充公积金 +补贴)🤯
大家拿大礼包或离职后都gap多久了?
1
509
DeepSeek V4.1 Flash 量化 100G 第一个版本在 benchmark 上表现不佳 我做了一次深入的分析: 1. 四个 benchmark 有两个竟然在校准数据中几乎没有覆盖(也是表现不佳的两个) 2. 如果不做专家裁剪常规的量化格式完全做不到 100G尺寸,为此我在社区已有的工作上发展了一个新型的量化格式做到了不裁剪专家也能压缩到 100G,重建误差和更高 bpw 的水平相当 看起来进展不错
1
4
972
还是要写开源项目呀,最近几份工作都不用写简历就直接 offer 了 今年我得再写一个 回报社会
17
1
162
19,726
🧟‍♂️🤖😆😵
Replying to @wholemars
Haven’t even had sex in ages (sigh)
807
如果这个身份之前被封过号就别再用了,换个新证就行,上次十一大封特封后开的新号目前活着
我的Claude突然要求做身份验证,用中国身份证,会马上封号吗? 在线等,挺急的。
584
哈哈中年搞机 我这半年买了 PRO6000 和 Mac Studio 256G,花费约 18w,还是发挥了重要作用的; 后悔想买 NV Spark 的时候犹豫了下,觉得没空搞这么多机器,没珍惜 5 万多的 PRO6000 最终八万五拿下 🤢
중년의 위기를 극복하기 위해 로컬LLM장비 구매 2개월 후기 1. 역시 사기 전이 가장 재밌다. 2. 가성비 따져가며 샀다가 사자마자 후회함, 관심을 끌기 위해서는 사람들이 가장 많이 쓰는 구성이 좋다는것을 깨닳음(5090, Dgx spark, Rtx pro 6000) 로컬LLM을 안하는 사람은 아예 관심이 없고 하는 사람은 자기 장비에서 도느냐가 중요하기 때문, 심지어 한국어권은 진짜 한줌단 3. 서빙하는걸 최적화 하느라 서빙을 할 수가 없음 4. 5천만원 이내로 haiku 5.5보다 좋기 어려움 5. 좋아요 많이 받기 위해서는 별거 아닌 무가치한 숫자를 굉장하다며 과장광고해야되는데 그렇게 그나마 없는 인망을 잃고싶지 않음 6. 이미지/영상 같은 디퓨전 모델은 평소에 쓸일이 없었는데 장비 아까워서라도 써보게 됨 이건 꽤 좋음 결론: 중고로 사서 몇개월 재미와 지식을 얻고 살짝 감가상각하여 판매한다면 같은 금액을 주식에 넣은 기대수익보다 나은것 같다. 중년의 위기 극복에는 꽤 좋았다.
1
931
A\ 前几天送的 Fable 重置还是很有意思的,正好赶上 Fable 5.2 发布,Fable 用量占比高就适合先用 Fable 重置
374
Feiteng retweeted
Megakernel blog is out now We implement a full layer (MoE + Attention) megakernel, achieving 1100 toks/s for Muse Spark, 2.1x faster than baseline. Nvidia's Instruction cache makes it really hard to write bigger kernels. We ended up tuning both instruction and data caches at all levels for extreme latency hiding! With this, some warps of next operation can start running before previous operation has finished - redefining what GPU utilization means. Full blog post: lmsys.org/blog/2026-11-05-mu…
Muse Spark Open: 517 → 1,109 tokens/s on 8× GB300 with SGLang megakernels. One kernel launch per layer, overlapping compute, memory loads and communication. Total launches per decode step drop from 1,450 to 97. lmsys.org/blog/2026-11-05-mu… In our new technical post, Pranjal Shankhdhar explains what makes this work: - Prefetching weights into shared memory and L2 before they’re needed. - Warming the instruction cache so large kernels don’t stall fetching their own code. - Using data itself as a readiness signal for cross-GPU communication. - Fusing the draft model to make drafting 2.6× faster. The post also includes measured timelines, CUDA/PTX examples, and the tradeoffs behind stopping at one megakernel per layer.
13
62
10
600
34,080
Feiteng retweeted
The main reason H3 Image struggles on detail is that it has a highly compressed latent space. Here is a round trip comparison of the most used VAEs. The H3 Image VAE scores worse for this image on every metric, even worse than the SD 1.5 VAE.
36
24
3
388
25,563
Feiteng retweeted
VoxMel Wavelet-Factorized Autoregressive Mel-Spectrogram Modeling for Zero-Shot Speech Synthesis models: huggingface.co/VoxDance-Lab
16
19
4
206
19,080
好啦不要吵了,来 AI 观鸟吧
Our strongest frontier model, Gemini 4 Carbon, is now available. Built for deeply complex, long-horizon workflows that push a frontier model’s intelligence, Carbon delivers frontier performance in agentic workflows across complex knowledge work, software engineering and web development, cybersecurity, and scientific reasoning. While Gemini 4 Argon remains the ideal model for the majority of software engineering and knowledge work, Carbon steps in for your hardest tasks that push a model to its limits — from complex, multi-step debugging to open-ended scientific research. Available starting today: — Consumers: use it in the @GeminiApp and AI Mode in Search for Google AI Ultra subscribers — Developers and Enterprises: build via the API in @GoogleAIStudio and @AndroidStudio, Google @Antigravity for Google AI Ultra subscribers, and Gemini for business — Scientists and Researchers: apply for a chance to get 6 months free via @Antigravity — Cybersecurity Defenders: apply to join the Fairwind Program for access to Carbon for cyber Watch how Carbon brings reasoning and coding together, turning avian research into an interactive 3D simulation of a songbird’s vocal organ that synthesizes the sounds of six bird species live.
1
400
NV J 教主跟下属 xx 实属不该呀
318