gbxk

And here is the deep-dive blogpost series: https://beci.me/blog

Amazing project. Have you guys thought about running it or adapting it for FPGA-like approaches like this one below that reach Taalas-speed like 20k tok/sec on cheap hardware?

https://www.mikeayles.com/blog/on-chip-llm-kv260/

500 tok/sec is great but this would be 40x and it changes the kind of tasks that it could even be used for if you link it to fast tools too.

Amazing project, I love it.

How about using these Cactus models?

Would it make sense for you to collab with those guys (1) for you to design a cheap but improved, commercialisable version of your $250 chip and (2) for them to tailor their runtime and quantizations to such FPGA hardware?

https://github.com/cactus-compute/cactus

Also have you thought about using a Alveo V80? Still not crazy expensive and could fit bigger models with same approach

That does not look planned.

Nobody said that’s the only safeguard. When the attack surface is all of language you better have a defense-in-depth philosophy or as close as you can to that.

Update: Katakate now supports ARM architecture (on Linux) thanks to a PR merged from Katakate's first external GitHub contributor: @spullara. Thank you!

Test passed, PR merged

As promised: https://github.com/Katakate/k7/tree/fix/no-dns-res-in-lockdo...

Will merge that in after it passes all network tests on a clean/wiped instance.

Thanks! I'll review Arrakis and come back. E2B is often considered harder to setup and less AI engineers friendly for direct stack contributions, as Katakate is the only alternative fully implemented in Python (core modules, Typer CLI, FastAPI, Python SDK).

Our native K8s support and exposition of K8s API also makes it friendly to devops.

Finally, our deploy/infra stack is lean and tightly fits in a single Ansible playbook, which makes it easy to understand and contribute to, letting you rapidly gain full understanding and ownership of the stack.

Thanks! Yes: Katakate provides much stronger isolation, since it uses hardware virtualization (via Kata Containers and Firecracker) while gVisor relies purely on software sandboxing in user space.

gVisor isolates containers by intercepting system calls in a user-space kernel, so it can still be vulnerable to sandbox escape via gVisor bugs, though not directly through Linux kernel exploits (since gVisor doesn’t expose the host kernel to the container).

Katakate also provides more than isolation: it offers orchestration through Kubernetes (K3s)

You could create a gVisor RuntimeClass in Kubernetes to orchestrate gVisor sandboxes, but that would require extra setup.

Thanks for sharing, adding it to my list.

Very cool! Apple containers run on Apple ARM so it's complimentary to my stack which doesn't support ARM yet (but soon will when extending to Qemu which supports ARM). Thanks for sharing!

This is an excellent point. I moved this to #1 on the TODO list. I'll deny all DNS resolution by default until Cilium is integrated, if that passes the basic functionality tests.

I'll also add to the roadmap whilelist/deny for container pulling.

Thanks!

Thanks, will study that one too!

Interesting, thanks for sharing!

Very cool one. That's dedicated to Apple ARM which I don't currently support so the two are complimentary. Apple containers shares some primitives with Kata. I'll investigate if it's possible to use Apple containers as a VMM inside Kata, or creating an Apple Containers runtime class in Kubernetes. If either is possible, we could then potentially use Apple containers as a backend in Katakate. I need more time to study that.

Thanks, I'll review that one too and compare.

Lucky you! And lucky me for sharing the info :)

Katakate is built on top of Kata, and sets up a stack combining Kubernetes (K3s), Kata, Firecracker, and devmapper snapshotter for thin pool provisioning. Combining these tools together is highly non-trivial and can be a headache for many, especially for AI engs who are often more comfortable with Python workflows. The stack gets deployed with an Ansible playbook. It implements a CLI, API and Python SDK to make it super easy to use. A lot of defense in depth settings are also baked-in so that you don't need to understand those systems at a low level to get a secure setup.

That's a config example.

Yes, blocking DNS exfiltration requires DNS filtering at cluster level. This is what will be added with the Cilium integration which is top-3 on the roadmap (top of readme).

DNS resolution is required for basic Kubernetes functionality and hostname resolution within the cluster.

That's said explicitly in several places in the docs: "DNS to CoreDNS allowed"

One thing I could do is make it exposed in config, to allow the user to block all DNS resolutions until Cilium is integrated. LMK if desired!

Thanks everyone for the amazing feedback and discussion!

For anyone curious:

– Docs: https://docs.katakate.org

- LangChain Agent tutorial: https://docs.katakate.org/guides/langchain-agent

It's getting late where I am, so I'm heading to bed — looking forward to replying to any new comments tomorrow!

It uses Kata with Firecracker which gives you as light of a boot as it gets. Subsecond booting for instance is accessible with a lighter rootfs, which is also on the roadmap (one of the easiest items, actually). The k8s layer doesn't add overhead either compared to any other VM. If you want to compare to bare containers, depending on workload, you could see a 5% overhead due to virtualization. Exact overhead would depend on workload.

That's an interesting direction! TEE support would be relatively straightforward with current stack (and it's on my roadmap), so that could be a first step forward.

It is well known that containers do not provide you safe isolation. It is not their purpose. They share kernel and page cache with the host. Any kernel exploit gives to someone in a container potential root control of the host (see DirtyPipe, DirtyCow). That's why you need VM-level isolation.

Sure one day if it really kicks off I could think of offering additionally a SaaS solution with paid enterprise features like SOC 2 compliance, RBAC, multiple clouds supported, etc. Why not. But I strongly believe that for it to be successful, it needs a strong open-source base. Then, billing huge companies for compliance features or huge usage makes sense. That would support development of the open-source part too.

I like the Docker model, for instance: free for companies under 250 employees or $10m/y revenue.

In any case, it will always be open-source.

Those paid enterprise features wouldn't come from closed-source: they would come from compliance of a particular SaaS-offered infra setup, that anybody else could reproduce. Just like HuggingFace.

If you have any suggestion on how I can make this more friendly UX-wise to your personal usage, I am most interested to hear! And this will shape my roadmap.

Actually you can! After you run "k7 install" you'll have a k3s cluster up and running, with Kata as a runtime class, and Firecracker specified in Kata config. So nothing prevents you from hitting the Kubernetes API! kubectl will work.

Note: I use k3s' internal kubectl and containerd, to avoid messing with your own if you have some already installed. That means you can run commands like "k3s kubectl ..."

And thank you for the compliments on the stack.

True! GCP does. I haven't tested it yet. I didn't know D.O does. If anyone knows others, I'm interested too!

No business model short-term. My goal is broad adoption, 100% open-source.

By multi-node I mean so far I only support 1 k8s node, i.e. 1 machine, but soon adding support for multiple. Still, on 20 CPUs I can run +50 VM pods with fractional vCPU limits.

For GPU passthrough: not possible today because I use Firecracker as VMM. On roadmap: Add support for Qemu, then GPU passthrough possible.

Inter-VM networking: it's already possible on single-node: 1 VM = 1 pod. Can have multiple pods per node (have a look at utils/stress-test.sh). Right now I default deny-all ingress for safety (because by default k8s allows inter pod communication), but can make ingress configurable.

Startup time: a second, or a few seconds, depending on which base image (alpine, ubuntu, etc...) and whether you use a before_script or not (what I execute before the network lockdown)

Large artifacts: you can configure resource allocated to a VM pod in the sandbox config and it basically uses k8s resource limits.

Let me know if any other question! Happy to help

wow that's super new! Thanks for that, will look deeply into it and compare