ThunderSizzle

I'll give 27B-MTP a try. I think I can tolerate 45 tps if the results are technically better. 35B is pretty good, but definitely shows it's inabilities at times (probably either due to the heavy caching quantization I'm doing, or the heavy model quantization vs what 2 GPUs could run).

My biggest gripe is that both pi and opencode seem to have trouble parsing the thinking blocks at times, and the model sometimes cuts-off mid-thinking or prints out weird character tokens at times. I don't know if that's because of llamacpp, pi/opencode, or qwen3.6, or some weird combination of them all, as I haven't investigated that problem fully yet.

Agreed. I have a single 9700 and I'm able to fit Q6 27B at 30tps or Q5 35B at 100tps very easily via llamacpp running vulkan.

The results are impressive considering the amount of people trashing AMD and still trying to recommend 3090s. I hope to buy a 2nd one at some point, but I also hate the version hell of vLLM, the R9700, the ROCM version, and Qwen3.6 all not agreeing with each other. I haven't gotten vLLM to run properly for Qwen3.6, since the version that runs on a 9700 doesn't support 3.6 yet.

I'm trying to quickly hack out a optimized path for just Qwen3.6 to run against rocm natively (e.g. my own inference server for 9700s basically) and see if it can perform better than llamacpp vulkan's results.

Word of caution - the last llamacpp with good performance was b9209 from a month ago. After that, for some reason, vulkan performance dropped by 10x, which has made me lose confidence in llamacpp in the long run.

Having said all that, 3x is 96GB for 4k and peak 900 watts. A 96GB Blackwell is $12k and peak 600 watss. And they will have a similar memory throughput (minor negative to the AMD cards for split processing). It's crazy how price efficient the r9700 is compared to the Nvidia cards.

The cheapest 3090s I could find with any sort of guarantee were pushing $1500.

An AMD AI Pro R9700 32GB brand new is $1350 right now.

After some tweaking, I had it running faster than the models the 3090 could run, and it could obviously run with higher context limits and bigger models due to the extra vram.

Obamacare, but for AI, where every American has to now pay a penalty to not use AI or something like that?

That was the last major thing the Democrats did, and healthcare has gotten substantially worse...but at least it's well regulated now.

The CEO won't get in trouble, but the employee who can't justify a bad result/prompt?

There will always be a competitor that can undercut the inference market. There is no "moat" given that you can self host decently capable LLM agents like Qwen3.6 on not super expensive hardware, like an AMD R9700, and still get competitive speeds to most cloud interfaces.

If you can self host it that easily, any Joe can scale it out much like shared web hosting, and shared web hosting or even dedicated rented boxes has always been cheaper than the big cloud providers.

I don't think OpenAI or Anthropic can reasonable compete in the long term if they can't achieve "AGI", and they won't, no matter what shareholders desire.

An R9700 is $1350 and can get 100 TPS running Qwen3.6-35B-A3B Q5 with 130k context window (with room to spare) with a bit of fine tuning llamacpp-vulkan, but llamacpp's repository instability and lack of real versioning frustrates me.

In terms of electricity, if you aren't using it, even with all the vram loaded, at most your wasting about 30 watts or so.

Prompt processing a large uncached context is annoying, which is why I forced a lower context window, but I don't know if it's any worse in performance than the cloud models I've used.

There's a niceness, to me, knowing I don't have to rent it anymore. If you rent it, the terms can change regularly.

Like a compliment sandwich or something.

The coding isn't the hard part. It's the people and networking. Facebook's only moat is HOA boards that think private communication behind Facebook groups somehow equates to public messaging a community...

In other words, once people got on it, it was too late.

I hope this is sarcasm, or "half" my job doesn't exist or something. Or you talking about full time non-dev scrum masters?

My 980 is currently unpowered. I have not tried to integrate them yet. It's on my to-do list, but the first time they were both powered on, the system had booting issues, and I didn't want to care at the time, since the 980 was probably going to be idle 99% of the time anyway.

I'll probably try to figure that problem out in about a month. Worst case is I move it to another even older desktop to replace the 9800 GTX+ inside of that one.

GitHub Copilot. It was one of the best values around in terms of cheap LLM access since each prompt was basically 4 cents (more or less), no matter how much it would do or how many tokens it used. A simple "Proceed" prompt that was telling the agent to execute a sophisticated plan could burn a lot of time without needing any direct intervention by the user, but as of June 1st, they switched to metered billing, meaning each token in/out has a cost now.

It was suspected to come soon enough, but it was a nice cheap road for my small hobby stuff. When they announced the price changes, I started to explore alternatives, and with the news of Qwen3.6 35B being both and having quality, I figured it was worth a try out, and self-hosting made the most sense to me, since that meant I was free from being a forever-renter.

I added an R9700 32GB to my 10+ year old desktop that had a 980 4GB card in it, for a grand total of $1350 or so. The payoff compared to what I was using with GHCP was 33 months, but when GHCP announced their price increase, it basically became a 3 month payoff at minimum (so yes, GHCP did a 10x price increase for non-parallel agentic workflows)

I can easily run Qwen3.6 35B-A3B with Q5_K_M with a 260k+ context window with some vram to spare. It easily runs probably 80tps. It took me quite a while to find the

Compared to GHCP Claude Sonnet 4.5 or 4.6, I have full parity. The wall clock time is faster for agentic workflows, and rule following is about on par.

With either, doing something kind of novel or obscure takes more hand holding compared to just generate a GUI or crud app. For example, trying to build an actual program that performs a complicated process correctly requires quite a bit of hand holding to get it to properly help.

Sure, it isn't Opus or something, but I think with the right harness, it probably can get close. I think most of the issues these days is the harnesses are lacking.

I wonder if you paired a few different types of AI together, an LLM agent might be good at strategizing -. E.g. building a strategy on how to handle a scenario. But, it would need to know the entire game manual basically. Then it would pass the stratrgy to a better AI in some way. But it might not be needed if the better gaming AI can just do that part too already.

I admit I know nothing about this though.

My instructions and steering is to force the LLM agent to focus on mocking only system boundaries and outside in unit testing. I've found these two make verbose but good tests that actually prove behavior pretty well.

But that was my testing strategy already, but writing one of those tests could take legitimately hours with how many columns and crazy rules our area has.

There's a lot of reasons, but it's probably how Google makes most of their money - brand names end up being high value ad targets - at least 6+ years ago when I worked in that sphere of things (ecommerce, primarily). It's legit extortion.

I guess that's why - because extortion is immoral. Eventually, those paying the protection fees will eventually find someone better to pay or will stop paying when the protection doesn't show up.

Right, but that's because Google permits that. OP is saying Google should simply stop permitting advertising over the real answer.

Fine. Then auto-closing tags? If you have a schema doc, you could do that. If you are writing free-form, then I'd definitely be complicated...?

I still see that better than YAML's indentation problem. Relying solely on indentation is a nightmare.

If you do manual edits, I find it best to start a new conversation. But if your instructions and documentation is good enough, the new conversations won't have any problems picking up where it needs to be.

Having said that, I fear what June 1st brings for copilot It might suddenly be very useless for me.

It wouldn't need to be if closing tags were allowed to be unnamed. For most cases, we can tell the closing tags easily enough for simpler files:

  <project>MAML</>
  <tags>
    <item>minimal</>
    <item>readable</>
  </>
  
  <!-- A simple nested object -->
  <spec>
    <version>1</>
    <author>Anton Medvedev</>
  </>

  <!-- Array of objects -->
  <examples>
    <item>
      <name>JSON</>
      <born>2001</>
    </>
    <item>
      <name>MAML</>
      <born>2025</>
    </>
  </>

  <notes>
  This is a multiline raw strings.
  Keeps formatting as-is.
  </>

Freetaxusa isn't. I've done complicated schedule K, etc forms on it

But what's your argument? That phone-based extensions are more vulnerable somehow than desktop extensions?

If anything, wouldn't a phone extension be more sandboxed than most desktop environments?

What if you split it into less complex tasks? E.g. use the model to help decompose the task into parts, then help it iterate through it.

Gives you more control over the outcome and more steering anyway.

A Qwen3.6-35B-A3B or whatever it's full name is, when on a 3090, can at the very least, with very little fine tuning, compete with Haiku and blows away GPT4.1 (aka, the cheap models).

It might keep up with Sonnet 4.5 with some tinkering.

But long story short: it seems to have better performance and similar quality for a payoff of a year or so compared to cloud models. In the same way you can self host faster/easier/cheaper than cloud hosting, if you are okay with the negatives.

I'm returning my 3090 soon for a R9700 after some more basic benchmarking, since the higher RAM should improve my observations more.

I've heard this before.

Then COVID happened and employers didn't have 2-4 months of savings built up, and ended up shuttering due to lack of money immediately.

Also, since post COVID, we've had hyper inflation and a locked up housing market. $150 townhomes that a $50 family salary could afford are now going for $300. And rent has gone up to match.

I don't think the numbers not matching is because of anyone's personal financial responsibility. It's more from the Fed's and Congress's horrible actions over the past 2 decades and their financial irresponsibility.

Granted, grumbling about the powers that be doesn't solve the problem, which is why I fear civil turmoil will be here very soon.

Many American roads don't have lines. Residential roads, parking lots, many business driveways have limited markings.

Then there's roads with just the center line markers with no road should markings.

Then there's a whole class of roads of lines over "demarked" old lines that weren't demarked well, or lines fading that should've been painted a long time ago.

I'm surprised you've never seen a non-perfect road?

Just a bunch of sophisticated if statements, I guess.

I just bought a 3090 off of Amazon. It was expensive ($1500ish), but covered under prime and an included any reason 90 return policy.

But that gives me a good while to determine if it's worth it or not. I've heard good and bad, so here's hoping for good or close to it.

I wasn't going to fork out $1000 on a chance it might be enough with a rough return strategy.

It reminds of early smart phones when the cell providers pulled away from unlimited data...and then they brought it back in s few years.

I think competition will get fierce. We see many people are attracted to the price stability of GHCP - it became clear what a request could do - the problem is that they didn't match results with cost. It's not clear what a 5 hour usage window in Claude Code can do.

There's no reason the harness couldn't provide a quote on the next request, aside from it takes effort and it would be upfront to the user, creating expectations.

They can engineer the harness to limit the amount it does. When pressing enter, it's be nice to have a "budget" per prompt, much like the model multiplier. When the harness used up the budget, it cleans up and cuts off the work.

But that would entail actual work and effort...and care for user's time and money.