Building LLMKube: K8s inference across NVIDIA, AMD, and Apple Silicon. Data stays yours. AI/ML Ops Director @ Ashley. Ham Radio: N7CPM
Gig Harbor, WA
Joined January 2018
- Tweets875
- Following604
- Followers335
- Likes2.8K
Pinned Tweet
Badge in hand. Speaking at #KubeCon NA on Tuesday.
Turning Apple Silicon into Production GPU Nodes for LLM Inference on Kubernetes
Tue 11/10, 2:50 PM MT, Grand Ballroom H
CUDA vs Metal benchmarks, one operator across heterogeneous GPUs, and the real tradeoffs. Come say hi!
2
7
495
Every PR one of my models writes for me gets a second pair of eyes before I see it: a read-only Qwen3.8-Flash-Next reviewer, served by LLMKube on an AMD Strix Halo.
Today, 9 minutes on a Go security bump: the code was clean, but it caught a wrong claim in the PR description.
My #kubecon presentation is over, so now it’s time to hit the LLMKube backlog like a beast 😄
1
5
152
Last-minute benchmarks on my M5 Max before my talk tomorrow: @ashxhart's TensorFold ran Qwen3.8-27B at 129 tok/s across 162K tokens of long output, 5x the same model with drafting off (24 tok/s). Not a 256-token demo: 30 runs, 2.7K to 8.2K tokens each, three sampling settings.
Decode tok/s by workload, from TensorFold's own per-request stats:
Agent rewriting a Go file: 231 (up to 359)
K8s YAML: 160
Go package with tests: 144
Code review: 104
Prose essay: 69
That's 3x on prose, 6-7x on code and YAML, and 14x when the output copies its input. Sampling barely moved it: 129 / 134 / 125 overall at temp 0, 0.7 and 1.0. Greedy drafted output was byte-identical to undrafted, so the speedup costs no quality.
I also checked the outputs. 36/36 K8s objects pass strict schema validation, and the Go file edit builds and passes LLMKube's tests. Weak spots: unit tests written in one shot, and invented numbers in the essay. Check those with any model. In opencode it fixed its own jq errors and shipped a working kubectl plugin in under 2 minutes.
Setup: Qwen3.8-27B MLX 4-bit, thinking off, with z-lab's DFlash2 drafter, single stream, 128 GB M5 Max, served by LLMKube's Metal agent. TensorFold 0.3.4.1, because my demo has been pinned since September; 1.2.5 is current. Short templated YAML still hits 227 tok/s; long answers settle lower. Happy to share the script.
Chris Maher retweeted
Be careful what you wish for, America…
“The rest of the world is ready and willing” to build data centers - ARM CEO
My stickers arrived just in time for Kubecon! Excited to share these out for LLMKube and Socair!
1
4
111
Deck's done and I'm pumped! At #KubeCon I'm sharing how I got Kubernetes running LLM inference on Apple Silicon: what worked, what broke (a kernel panic included), and patterns you can take home to your own cluster. See you there!
Last month, Pluto Security reported a Hugging Face campaign that shipped crypto miners as small Qwen2 test checkpoints. No poisoned weights, no pickle trick: just Python that runs under trust_remote_code. The second wave's flagship scanned clean on the hub.
1
1
80
Apache-2.0, every check. Go, air-gap buildable. Try it in your browser, nothing uploaded: socair.ai/try
github.com/defilantech/socai…
1
17
Source, Pluto Security's write-up: pluto.security/blog/hugging-…
1
22
I pointed DeepSeek Harness's Creator Mode at DeepSeek V4.1 Flash on my three DGX Sparks.
423 steps later it had built its own plugin, then said it never looked at the page. The page was broken.
Second pass: 7 minutes, one parenthesis.
llmkube.com/blog/deepseek-ha…
2
1
88