It absolutely does and a very good effort of compatibility with GitHub actions. It’s not perfect but migrating is far less of a pain than I experienced moving to others
I've heard stories of bills like that and is wild to me.. I built a SaaS that had just over a PB in data and our monthly was low 5 digit and largest part was S3, and co-location was already on the table. I can't imagine getting to 6-7 digit a month.. I understand how it happens with rapid growth, but even 6 months of that I would be scrambling for other hosting options.
Honestly surprised to hear that GPT OSS 20B runs slow on mac hardware. It's absolutely one of the fastest models I've run on local GPUs for its size, but only tried Nvidia cards.
Edit: TIL it is MoE and only has 3.6B active, explains a lot.
I might be reading the pricing wrong but you have to pay per hour for the port plus per GB transfer? And looks like the cheapest is $0.02 per GB? Is that really the 'cheap' option? That looks fine for a TB or two, but still crazy when getting closer to PBs.
Very impressive. One thing that seems odd to me is that is at like 4 minutes before it starts a response for large input? I don't use mac hardware for LLMs, but that is quite surprising and would seem to be a pretty large stumbling block for practical usage.
Edit: Caching story makes a lot more sense for regular usage:
> Claude Code may send a large initial prompt, often around 25k tokens, before it starts doing useful work. Keep --kv-disk-dir enabled: after the first expensive prefill, the disk KV cache lets later continuations or restarted sessions reuse the saved prefix instead of processing the whole prompt again.
I agree, the mechanical refactoring of modern IDE tooling, especially with typed languages is so much faster and safer, it's not even close. These tools can be useful for sure, but I think in general they are being wayy over prescribed to different tasks.
This is very true, I've found these tools that I am highly encouraged to use very hit and miss, which they are by nature. After using Matt Pocock's skills, I've come around to the idea that LLM's main utility is to act as the ultimate rubber ducky. The `grill-me` feature is honestly the most useful, not for guiding the follow up writing of code, but to make me write down and explore the idea I have more quickly. It's guesses of questions to ask are generally pretty good. I don't believe there is any 'understanding', so I feel the rubber ducky analogy works quite well. This isn't anything you couldn't do before with some discipline, but at least I find it helpful to be more consistent.
A 90s Camry, Corolla, or Civic seems to have become the peak minimalist car. Shame we will never likely see an EV equivalent focused on utility and cost efficiency without all the bloat. I don’t think there is a good option sadly, any ICE car will eventually just become unmaintainable, and I can’t see a path to EVs that are just cars and don’t come with all this tracking.. hope to be proved wrong..
I've found MiniMax 2.7 pretty decent and even pay-as-you-go on OpenRouter, it's $0.30/mt in, and $1.20/mt out you can get some pretty heavy usage for between $5-$10. Their token subscription is heavily subsidized, but even if it goes up or away, its pretty decent. I'm pretty hopeful for these openweight models to become affordable at good enough performance.
I'm trying out codex for first time as well cause something up with Claude for sure, 4.7 has been super frustrating. For other models, highly recommend trying MiniMax 2.7, using it with Hermes is actually pretty good, and their token subscription plans include a lot of usage for $10.
Worked on the software side of increasing the rate of solar penetration in electricity networks between 2016-2020 via global solar radiation forecasting. The uptake of the software was slow the first year but then rapid once more electricity networks were struggling with knowing how much solar was in the network. Once it is easier to predict, the network becomes easier to manage, and more can be safely added, and make it economically profitable. Sucks this was a commercial operation, but excited to see all the hard work across various industries is solving problems to get more renewable energy into networks.
Building websites, I agree has little value, but using it as a way to explain basics of how the web works I think is pretty valuable. Web likely isn't going anywhere for a long time, having some basic knowledge of how it works I think very useful for a lot of people. I hate the idea of any more MS apps like Excel being regularly incorporated, but basic usage of something similar definitely can help know of how to use a useful tool/computer skill. Even in the early 90's we had computer labs for learning computer skills which I think there is value. But forcing tech everywhere into teaching is an issue IMO.
"side loading", I know this term is the one used but I think should be pushed back against with just using the standard "installing"/"install". It makes the control point clearer and (should be) unsettling when you can't "install" software on hardware you own.
I think it would assist in exploiting exploring multiple solution spaces in parallel, and can see with the right user in the loop + tools like compilers, static analysis, tests, etc wrapped harness, be able to iterate very quickly on multiple solutions. An example might be, "I need to optimize this SQL query" pointed to a locally running postgres. Multiple changes could be tested, combined, and explain plan to validate performance vs a test for correct results. Then only valid solutions could be presented to developer for review. I don't personally care about the models 'opinion' or recommendations, using them for architectural choices IMO is a flawed use as a coding tool.
It doesn't change the fact that the most important thing is verification/validation of their output either from tools, developer reviewing/making decisions. But even if don't want that approach, diffusion models are just a lot more efficient it seems. I'm interested to see if they are just a better match common developer tasks to assist with validation/verification systems, not just writing (likely wrong) code faster.
I'm quite surprised the A100 is not much better since the power levels for the Ampere cards I believe is a lot lower.
Does this mean even for a model that fits on a single server that trains for a few weeks will absolutely need a recovery process? Interested in peoples experiences around this.
Right, a market is a small tool of larger systems. That’s fine, hard to get right but can make systems better. Type two just seems to be the cargo culted everywhere..
Exactly, these markets exist in the real world, so as their size and use increases, the more likely the odds will influence real world events. Look at sports betting for a much smaller example. Match fixing is known. Electricity markets are gamed for individual profits at the detriment to everyone and the stability of the system, even with regulators trying to keep things stable. Enough "Market for all the things" already..
This happens when you get worse and worse inequality when it comes to buying power. The most accurate prediction into how this all plays out I think is what Gary Stevenson calls "The Squeeze Out" -> https://www.youtube.com/watch?v=pUKaB4P5Qns
Currently we are still at the stage of extraction from the upper/middle class retail investors and pension funds being sucked up by all the major tech companies that are only focused on their stock price. They have no incentive to compete, because if they do, it will ruin the game for everyone. This gets worse, and the theory (and somewhat historically) says it can lead to war.
Agree with the analysis or not, I personally think it is quite compelling to what is happening with AI, worth a watch.
Totally true. I have a trusty old (like 2016 era) X99 setup that I use for 1.2TB of time series data hosted in a timescaledb PostGIS database. I can fetch all the data I need quickly to crunch on another local machine, and max out my aging network gear to experiment with different model training scenarios. It cost me ~$500 to build the machine, and it stays off when I'm not using it.
Much easier obviously dealing with a dataset that doesn't change, but doing the same in the cloud would just be throwing money away.
> The second you turn your head though, your fellow teammates will conspire to replatform onto Go or Rust or NodeJS or GitHub Actions and make everything miserable again.
Curious how would you use use Smalltalk in replace of GitHub Actions assuming you need a GitHub integrated CI runner?
I have a hard time trying to conceptualize lossy text compression, but I've recently started to think about the "reasoning"/output as just a by product of lossy compression, and weights tending towards an average of the information "around" the main topic of prompt. What I've found easier is thinking about it like lossy image compression, generating more output tokens via "reasoning" is like subdividing nearby pixels and filling in the gaps with values that they've seen there before. Taking the analogy a bit too far, you can also think of the vocabulary as the pixel bit depth.
I definitely agree replacing AI or LLMs with "X driven by compressed training data" starts to make a lot more sense, and a useful shortcut.
> In general once you start thinking about scaling data to larger capacities is when you start considering the cloud
What kind of capacities as a rule of thumb would you use? You can fit an awful lot of storage and compute on a single rack, and the cost for large DBs on AWS and others is extremely high, so savings are larger as well.
Same setup here, one game setup I've hit but this will be a rare problem, is StarCraft Remastered. Wine has an issue with audio processing which I can't seem to configure my way out of. It pegs all 32 threads and still stutters. Thankfully this game can likely run on an actual potato, so I have a separate mini PC running windows for this when I want to get my ass kicked on battle.net.
Working at IT places in the late 2000s, it was still pretty common place for there to be a server rooms. Even for a large org with multiple sites 100s of kms a part, you could manage it with a pretty small team. And it is a lot easier to build resilient applications now than it was back then from what I remember.
Cloud costs are getting large enough that I know I’ve got one foot out the door and a long term plan to move back to having our own servers and spend the money we save on people. I can only see cloud getting even more expensive, not less.