HNHacker News
TopNewBestAskShowJobs

darkolorin

73 karma · joined August 5, 2014

prev CEO & co-founder Prisma & Capture, LFG, now doing on-device inference at Mirai
submissionscomments
darkolorin··on Show HN: We made our own inference engine for Apple Silicon
Basically “faster” means better performance e.g. tokens/s without loosing quality (benchmarks scores for models). So when we say faster we provide more tokens per second than llama cpp. That means we effectively utilize hardware API available (for example we wrote our own kernels) to perform better.
darkolorin··on 90T/s on my iPhone llama3.2-1B-fp16
I made it! 90 t/s on my iPhone with llama1b fp16

We completely rewrite the inference engine and did some tricks. This is a summarization with llama 3.2 1b float16. So most of the times we do much faster than MLX. lmk in comments if you wanna test the inference and I’ll post a link.

darkolorin··on Quantum product
It's my first experience on Medium. I hope community can help me to improve my skills.
darkolorin··on Morning Mail – one email at a time
really great idea, but why morning? %)