zacksiri

Based on my testing the larger the model the better it is at handling larger context.

I tested with 8B model, 14B model and 32B model.

I wanted it to create structured json, and the context was quite large like 60k tokens.

the 8B model failed miserably despite supporting 128k context, the 14b did better the 32B one almost got everything correct. However when jumping to a really large model like grok-3-mini it got it all perfect.

The 8B, 14B, 32B models I tried were Qwen 3. All the models I tested I disabled thinking.

Now for my agent workflows I use small models for most workflow (it works quite nicely) and only use larger models when the problem is harder.

LLMs are relatively new technology. I think it's important to recognize the tool for what it is and how it works for you. Everyone is going to get different usage from these tools.

What I personally find is. It's great for helping me solve mundane things. For example I'm recently working on an agentic system and I'm using LLMs to help me generate elasticsearch mappings.

There is no part of me that enjoy making json mappings, it's not fun nor does it engage my curiosity as a programmer, I'm also not going to learn much from generating elasticsearch mappings over and over again. For problems like this, I'm happy to just let the LLM do the job. I throw some json at it and I've got a prompt that's good enough that it will spit out results deterministically and reliably.

However if I'm exploring / coding something new, I may try letting the LLM generate something. Most of the time though in these cases I end up hitting 'Reject All' after I've seen what the LLM produces, then I go about it in my own way, because I can do better.

It all really depends on what the problem you are trying to solve. I think for mundane tasks LLMs are just wonderful and helps get out of the way.

If I put myself into the shoes of a beginner programmer LLMs are amazing. There is so much I could learn from them. Ultimately what I find is LLMs will help lower the barrier of entry to programming but does not mitigate the need to learn to read / understand / reason about the code. Beginners will be able to go much further on their own before seeking out help.

If you are more experienced you will probably also get some benefits but ultimately you'd probably want to do it your own way since there is no way LLMs will replace experienced programmer (not yet anyway).

I don't think it's wise to completely dismiss LLMs in your workflow, at the same time I would not rely on it 100% either, any code generated needs to be reviewed and understood like the post mentioned.

You could say that! hahaha! I'm happy to see someone understand what it is.

Have a look here, it's an early preview

https://x.com/zacksiri/status/1922500206127349958

You can see it's going from introduction, asking me for my name, and then able to answer question about some topic. There is also another example in the thread you can see.

Behind the scenes, the system prompt is being modified dynamically based on the user's request.

All the information about movies is also being loaded into context dynamically. I'm also working on some technique to unload stuff from context when the subject matter of a given thread has changed dramatically. Imagine having a long thread of conversation with your friend, and along the way you 'context switch' multiple times as time progresses, you probably don't even remember what you said to your friend 4 years ago.

There is a concept of 'main thread' and 'sub threads' involved as well that I'm exploring.

I will be releasing the code base in the coming months. I need to take this demo further than just a few prompt replies.

I publish my findings on my youtube channel and my blog, you're welcome to have a look. Both links are in my profile.

I've been working on solving this with quite a bit of success, I'll be sharing more on this soon. It involves having 2 systems 1st system is the LLM itself and another system which acts like a 'curator' of thoughts you could say.

It dynamically swaps in / out portions of the context. This system is also not based on explicit definitions it relies on LLMs 'filling the gaps'. The system helps the llm break down problems into small tasks which then eventually aggregate into the full task.

Here is a list of inference engines i've tried:

- SGLang

- vLLM

- TGI (Huggingface's)

- llama.cpp

- infinity (great for embedding / reranking models not for LLMs)

My personal feeling is SGLang / vLLM have issues that make me not want to use it. Sure it's fast, but there are reliability issues, you need lots of flags and tinkering to make it work. Also there is the problem of 100% cpu usage on idle which the core contributors say is 'normal' and 'expected'. You can do a search in the respective repositories on this topic if you don't believe me. People even submitted PRs to solve these issues which they have not merged. The mindset of these software is just to get it to 'work' but not really on polish and ease of use.

TGI on the other hand is in a class of it's own. You can just feel the polish that went into it. Things tend to 'just work'. It's the only engine I tried that was able to run a model I wanted in a single try. Then I added the flags to make it fit with my hardware (like sharding and max prefill tokens). TGI uses flashinfer by default which is SOTA when it comes to flash attention backend.

llama.cpp has widest model support, however it does not perform as well as TGI / vLLM / SGLang. So if you can accept that you are losing performance (based on my testing about 30% slower) tt's great for testing, development purposes but if you want to do production grade stuff I would recommend TGI.

This is fantastic work! They look beautiful. I hope you make lots of money from them because I know it's a lot of work. I'll be a customer in the near future for sure.

- React - check

- Framer Motion - check

This hit the spot for me.

I recently used Gemini and Google search (with overview) to confirm whether a snack i bought from japan has expired. Used gemini to take a picture of the label written in japanese

One item said 25/7/25 the other one said 25/7/24 as you can imagine I was sure the first one was safe but the second one was confusing.

It told me that it's safe to eat because japanese date format is Year / Month / Date.

I looked up japanese date format in google (with overview) just to confirm. I guess we'll find out. Will report back soon.

I use mistral-small-24b as my daily companion, and my AI project. It is truly amazing. The model is fast and does everything I need. I often feel the output of mistral's models are better than most of the other open models out there including the recently released gemma-3-27b.

I believe mistral just released Mistral Small 3.1, I can't wait to try it out.

There is also a reasoning variant from Nous Research which I've recently got up and running.

Mistral's model is in a class of their own. I'm inclined to try out their larger models via API. I wish they had a reranking model as well.

You should try Mistral Small 24b it’s been my daily companion for a while and have continued to impress me daily. I’ve heard good things about QwQ 32b that just came out too.

We're not throwing out Prometheus stack that's for sure.

I've had experience using Elastic for a long time I understand it's capabilities and limitations, I'm familiar with it's querying capabilities, and things have always played out nicely with elastic for me. I guess it's all about what tools you are familiar with.

If you are a paying customer and say you want prometheus metrics today we would show you how to access the already available LXD /metrics endpoint securely. You can learn more about this here https://documentation.ubuntu.com/lxd/en/latest/metrics/. With Opsmaru you get full access to the underlying LXD cluster by default out of the box.

Ideally we would want to build an uplink version of the /metrics endpoint to ensure more smooth / automated / seamless integration but the above will work perfectly well if you want it today. The uplink /metrics endpoint would also provide additional metrics regarding application specific metrics not available from the LXD endpoint. LXD only provides infrastructural metrics not application metrics.

In the future we want to integrate APM by default since we handle the routing for the apps we can bring together all these metrics like response times, SIEM and provide more metrics than infrastructural level metrics.

Thank you for your feedback, you have a valid point about the /metrics endpoint. We're planning on providing a /metrics endpoint in the future. I mentioned this previously as well in another reply.

This isn't a custom system at all we're simply removing the need to install / configure / manage another external package by implementing a data shipper into uplink using elixir broadway. The end goal is still that Ops / SREs can still use their existing favorite monitoring pipeline whether that's Grafana / Prometheus / Loki or Elastic / OpenSearch stack. There are several advantages, it means less things to install / maintain / patch / secure as mentioned in the post. We believe doings things this way leads to a more robust / secure system in the long term.

As for the 15 seconds scrapes we can tune that and provide that as an option for customers as well. These are things we can improve and provide to our customers as options. For now for MVP we're shipping data to the elastic stack certain decisions are made to help simplify and reduce the amount of things we have to do to get the product to an MVP.

We can provide the /metrics endpoint in the future, it's just a matter of time and priorities.

There are reasons why we're shipping data into elastic that will be clearer once things mature a little more. There are things Elastic can do that we need at a base level for our internal product plans, there will be a follow up post about this later.

Will provide more blog post articles giving you more details as to the decisions we've made and why we made them. Always happy to read feedback.

We were planning on supporting incus. However incus dropped support for fan networking in favor of OVN for cross node networking.

OVN is very heavy and requires a lot of management when it comes to provisioning and maintenance. So for now we didn't want to go there just yet.

We're sticking with LXD, it's been receiving a lot of updates from canonical, the team is responsive on the forum and has been a pleasure to work with.

Once we have some breathing room, we definitely want to explore incus and see what networking options are out there.

Maybe we'll just adopt wireguard and make that work out of the box with incus in a future iteration.

This was not an easy decision. I believe both are great products and you wouldn't go wrong either way. I was on the fence for a long time before making the decision.

I think it's a combination of a lot of things. I've been a long time Elasticsearch user, I think I used it since version 0.17.

Elastic just seems to have a lot more built in and seem to be the leader on this front when it comes to innovation. They did start the whole project and are leading when it comes to new stuff being built in. Their business / survival depends on building the best search product in whatever circumstances they find themselves in.

OpenSearch is funded by Amazon, and it's not their sole focus.

Things just feel more polished, well integrated than OpenSearch. The Kibana UI, Observability stack and their AI search stuff some of which is unique to elasticsearch and I'm sure there are more things yet to be uncovered.

Taking a long term view just seems like the offering of Elastic is more fitting to our product requirements.

That said whatever we've implemented in this blog post would also work with OpenSearch. In the future we will enable customers to bring their own timeseries DB and it would work with OpenSearch.

Happy to help! Yes we handle Elixir / Phoenix as a first class citizen. The whole thing is built in Elixir / Phoenix. We can also handle astrojs apps, our docs is built in astro hosted using Opsmaru.

Thank you! Yes! That's the main focus moving forward. Parts of it is still being built out, essentially we want to enable an App Store like experience for Web Applications. Open source developers should be able to monetize their applications by selling instances of their app to people who are non-technical but need their apps. Developers get paid via stripe connect.

I chose LXD for several reasons. There is much less overhead cost when it comes to managing an LXD cluster:

- It's more vertically integrated for example networking across node is built in, you get it out of the box.

- It supports stateful workload out of the box with no fuss. Running DBs doing snapshots, deletion protection etc... is very simple with LXD.

- LXD supports running Docker inside containers which means in the future we will be enabling docker containers. Each LXD containers can be treated as a 'pod' that can run multiple docker containers inside. But it's just a simple system container you can treat like a VM.

- Working with GPUs is very simple and straight forward. This is going to be key as we start to enable AI work loads.

- LXD doesn't require a Master Node which means each instance I provision I can use it to run my work load. It also supports redundancy as I grow my cluster because it handles distribution through raft. Which means in terms of overhead it's much lower than K8s

- Overall LXD feels like a batteries included container hypervisor.

This solution doesn't replace things like prometheus. In fact LXD has native support for prometheus, we would also be able to extend the solution to pushing data to a prometheus instance or expose a /metrics endpoint for prometheus to consume.

For our MVP we just chose Elastic but it will be easy to extend to support prometheus as well. We're shipping data using open telemetry format. OpenTelemetry is a specification when we ship data we try to keep it as close to what open telemetry does as much as possible. Elastic's observability supports this out of the box.

All this solution does is it queries the underlying infrastructure metrics and ship it to a destination. The only scaling it needs to handle is ship the data and handle back-pressure incase the destination cannot handle the load. Broadway does this out of the box.

Hey there! Founder of Opsmaru here. I didn't expect the post to get to the first page after it didn't get upvoted when I first posted it. Happy to answer any questions about the product and this post!

Most developers avoid GPU because of pricing. It’s simply too expensive to run 24/7 and there is the overhead of managing / bootstrapping instances loading large models to do intermittent instances. That’s the gist of it I think.

Unless you have constant load that justify 24/7 deployments most devs will just use an API. Or find solutions that doesn’t need you to pay > $1 / hour.

I was thinking this exact thing as I was reading the article.

Actual AI here.

I've been a redis / elasticsearch user since the early days. Seeing these companies grow has been sources of inspiration for me.

I'm glad you're back and look forward to redis's future. There is still a lot of work to be done.

In terms of licensing there is no right answer, large companies evidently abuse opensource, at the same time community is required to make projects like redis / elasticsearch grow.

This is a good move and I think the future is long with redis. So much to explore and so much more that can be done.

I use s6 in my product. It’s the standard for every app deployed using our platform.

Couldn’t be happier with the decision.

You can see how we provision s6 files here https://github.com/upmaru/pakman

I wonder if the same sentiment applies to products using .ai TLDs.

The thing about AI is it reminds me of what Steve Jobs said about speeds and fees.

People care about "1000 songs in your pocket" not "30GB hdd". AI seems to be the "30GB hdd" and people don't always relate AI to how it's going to help them.

I think it's important to realize, each level of abstraction has it's pros and cons. Concluding to have to go all the way down the stack and run your own data center and fiber is extreme.

In life it's always going to be a balance of what's feasible that you can do to keep control and giving up control because something is not feasible.

In this case obviously the author is having a problem high up the stack. The best solution would be to have more control in order to keep his business going.

The thing about fly.io / heroku / vercel is you are gaining convenience by giving up control over the underlying infrastructure.

When you give up control of the infrastructure you also give up the ability to decide what's right and wrong.

If you want to work with something that allows you retain that control but still have the convenience of a nice deployment workflow check out my profile.

It makes sense then for AMD to buy them out.

If they’ve trained LLMs with lumi which has a lot of instinct GPUs there is a high chance they’ve had to work through and solve a lot of the gaps in software support from AMD.

They may have already figured out a lot of stuff and kept it all proprietary and AMD buying them out is a quick way to get access to all the solutions.

I suspect AMD is trying to fast track their software stack and this acquisition allows them to do just that.

Thank you for your explanation. It's helpful to see another perspective on 'understanding'.