Hacker News Reader

Comment by travisgriggs | Hacker News Reader

I had my formative years in programming when memory usage was something you still worried about as a programmer. And then memory expanded so much that all kinds of “optimal” patterns for programming just become nearly irrelevant. Will we start to actually consider this in software solutions again as a result?

View on HN

You're right in terms of fitting your program to memory, so that it can run in the first place.

But in performance work, the relative speed of RAM relative to computation has dropped such that it's a common wisdom to treat today's cache as RAM of old (and today's RAM as disk of old, etc).

In software performance work it's been all about hitting the cache for a long time. LLMs aren't too amenable to caching though.

LLMs need memory bandwidth to stream lots of data through quickly, not so much caching. Well, this is basically the same way that a GPU uses memory.

OTOH, LLM inference tends to have very predictable memory access patterns. So well-placed prefetch instructions that can execute predictable memory fetches in parallel with expensive compute might help CPU performance quite a bit. I assume that this is done already as part of optimized numerical primitives such as GEMM, since that's where most of the gain would be.

AFAIK, you can't explicitly allocate cache like you allocate RAM however. A bit like if you could only work on files and ram was used for cache. Maybe I am mistaken ? (Edit: typo)

You can't explicitly allocate cache, but you can lay things out in memory to minimize cache misses.

Out of curiosity, why has there not been a slight paradigm shift in modern system programming languages to expose more control over the caches?

There are cache control instructions already. The reason why it goes no further than prefetch/invalidate hints is probably because exposing a fuller api on the chip level to control the cache would overcomplicate designs, not be backwards compatible/stable api. Treating the cache as ram would also require a controller, which then also needs to receive instructions, or the cpu has to suddenly manage the cache itself.

I can understand why they just decide to bake the cache algorithms into hardware, validate it and be done with it. Id love if a hardware engineer or more well-read fellow could chime in.

Another reason for doing cache algorithms in hardware is that cache access (especially for level 1 caches) has to be low latency to be useful.

There has! Intel has Cache Acceleration Technology, and I was very peripherally involved in reviewing research projects at Boston University into this. One that I remember was allowing the operating system to divide up cache and memory bandwidth for better prioritization.

https://www.intel.com/content/www/us/en/developer/articles/t...

Because programmers are in general worse at managing them than the basic LRU algorithm.

And because the abstraction is simple and easy enough to understand that when you do need close control, it's easy to achieve by just writing to the abstraction. Careful control of data layout and nontemporal instructions are almost always all you need.

Same as the failure of Itanium VLIW instructions: you don't actually want to force the decision of what is in the cache back to compile time, when the relevant information is better available at runtime.

Also, additional information on instructions costs instruction bandwidth and I-cache.

> you don't actually want to force the decision of what is in the cache back to compile time, when the relevant information is better available at runtime

That is very context-dependent. In high-performance code having explicit control over caches can be very beneficial. CUDA and similar give you that ability and it is used extensively.

Now, for general "I wrote some code and want the hardware to run it fast with little effort from my side", I agree that transparent caches are the way.

x86 provides this control with non-temporal load/store instructions.

that solves the pollution problem, but it doesn't pin cache lines. it also doesn't cover the case that ppc does where you want to assert a line is valid without actually fetching.

thats a strange statement. its certainly not black and white, but the compiler has explicit lifetime information, while the cache infrastructure is using heuristics. I worked on a project which supported region tags in the cache for compiler-directed allocation and it showed some decent gains (in simulation).

I guess this is one place where it seems possible to allow for compiler annotations without disabling the default heuristics so you could maybe get the best of both.

That seems correct, but it also doesn’t account for managed languages with runtimes like JavaScript or Java or .NET, which probably have a lot of interesting runtime info they could use to influence caching behavior. There’s an amount of “who caches the cacher” if you go down this path (who manages cache lines for the V8 native code that is in turn managing cache lines for jitted JavaScript code), but it still seems like there is opportunity there?

This is not applicable to most programming scenarios since the cache gets trashed unpredictably during context switches (including the user-level task switches involved in cooperative async patterns). It's not a true scratchpad storage, and turning it into one would slow down context switches a lot since the scratchpad would be processor state. Maybe this can be revisited once even low-end computers have so many hardware cores/threads that context switches become so rare that the overhead is not a big deal. But we are very far from anything of the sort.

I would say this is the main benefit of cuda programming on gpu. You get to control local memory. Maybe nvidia will bring it to the cpu now that the make CPU’s

A fun fact for the people who like to go on rabbit holes. There is an x86 technique called cache-as-RAM (CAR) that allows you to explicitly allocate a range of memory to be stored directly in cache, avoiding the DRAM entirely.

CAR is often used in early boot before the DRAM is initialized. It works because the x86 disable cache bit actually only decouples the cache from the memory, but the CPU will still use the cache if you primed it with valid cache lines before setting the cache disable bit.

So the technique is to mark a particular range of memory as write-back cacheable, prime the cache with valid cache lines for the entire region, and then set the bit to decouple the cache from memory. Now every access to this memory region is a cache hit that doesn't write back to DRAM.

The one downside is that when CAR is on, any cache you don't allocate as memory is wasted. You could allocate only half the cache as RAM to a particular memory region, but the disable bit is global, so the other half would just sit idle.

Thanks – I was wondering how code that initializes DRAM actually runs

You can in CUDA. You can have shared memory which is basically L1 cache you have full control over. It's called shared memory because all threads within a block (which reside on a common SM) have fast access to it. The downside: you now have less regular L1 cache.

Reminds me somehow on: https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...

;-)

I've actively started to use outlook and teams through chrome to free up some of my ram, easily saves 3-4gb. It's gotten ridiculous how much ram basic tools are using, leaving nothing for doing actually real work

People get on me all the time about not installing programs on my computer. I run everything in the browser, if I can. Partly so I can kill it properly without it misbehaving, and partly because I don't trust their software at all. Zoom, Slack, Gmail, etc-- if I can run it in the browser, then that's the only way I'll run it.

Same for me on mobile. I don’t install the Amazon app I just use the browser where I can limit tracking and only log in when actually buying something.

ive found the web versions use a similar amount of memory and have fewer features

my issue is that my company won't issue laptops with more than 16 gbs of ram

guess i'm not virtualizing anything...

Every app ships with its own isolated web browser now. That idea needs to die.

Back to native apps without bloated toolkits!

Or at least improving the shared browser ui / chromeless experience for "app" installs. I think that Tauri is pretty reasonable as well, weak link being Linux currently.

No fuck the browser. It's just layers of shit on shit on shit.

Mail.app is sitting here using 137Mb of RAM. Outlook 1270Mb.

And the likes of Zed save so much ram over VS Code... oh, wait...

I use vim for everything so I have no idea.

My main machine has 16Gb of RAM and I don't think I've ever seen it go over 4Gb and that was when I had a 200gb mmap'ed sparse array.

On my personal desktop, I have 96gb... I've never gone over 70 or so.. but that was with a lot of services running a fairly complex system with data loaded locally. I generally don't five a f*ck about the ram I'm using day to day. I'll run various updates and reboot between once a month and once a quarter.

I doubt it. I predict in a few years, maybe sooner, one/some of the AI companies buying up the supply will either have achieved their goal or collapsed, and then the market will be flooded with a glut of memory driving prices low again. Or, conversely, the demand stays high for a sustained period of time and the suppliers just increase supply. There's no hard bill of materials/technical reasons for the memory prices to be this high, unlike 20+ years ago.

And in the meantime, major buyers (government, big orgs) adjust by extending the planned lifespan of their computers, and upping the IT wage budget a bit to support that. That adjustment probably won't go away after supply returns.

That's honestly a good thing. Computers aren't really getting faster for end users doing mundane tasks the past couple of years.

Will help reduce E-Waste, and to the end user there won't be a different. A machine from 5 years ago feels just as fast as a brand new machine.

Im always shocked how much good IT equipment is shoved into the trashbin: At a lot of companies I could make a great deal - either for using it on my own or selling it on Ebay later on.

Big Corporations offen trash IT equipment thats only 3 - 4 years old. And there is no recycling etc. Very sad.

Big corporations tend to send old hardware through the surplus marketplace. There's lots of 3-4 year old corporate computers for sale. Often, the company leases the computers and then the lessor will sell them when they're returned.

Apple themselves make pretty good business leasing their own products that way.

I got a pretty nice 120Hz ultrawide dell for cheap at one of those surplus shops awhile back.

Meanwhile I’m at faang and they’re reusing 7 year old monitors still. Still getting equipment from pre-covid era.

Where are these luxurious big corporations that give their employees nice new equipment? :(

As long as it's working (and not gross), why do you need a new monitor? My current monitor is a 2010 model, I think I got it around 2013. I don't know what a new monitor would do for me, other than have a worse aspect ratio, cause Dell stopped making 30" 16:10 monitors.

In theory, yes. However a lot of these monitors are still 2560x1440 and are 30”+. The ppi is quite low. I’m looking for 4k and something that looks similar enough to the M4 MBP I’m working on. A lot of these just don’t look good as they used to.

Probably your new corporate monitor is also going to have relatively low ppi though. The problem isn't really the age then.

1440p is good enough that you aren't going to see individual pixels - just sit far back enough from the screen and use reasonable font hinting (Mac users are sadly out of luck here, but even then 2160p/4K is overkill).

What Im doing wrong? My 24" with 1920x1200 from 2010 is doing quite fine until today?

> A machine from 5 years ago feels just as fast as a brand new machine.

Except you can't install Windows 11 on it, and the org has to trash it anyway to keep up with security requirement (I know people on that line of work, they're all angry about it)

Well, at least then there would be enough RAM to run run Windows7 and Crysis from a RAMdisk, Id guess?

Also RAMsan will have a renaissance then? :-D

AI companies aren't buying RAM, they are buying the Wafers themselves. Then they are making special AI stuff. So the RAM never exists, and there will be no glut memory coming. Maybe some DDR5 will dribble out, but HBM isn't something we can use (at the moment).

> And then memory expanded so much that all kinds of “optimal” patterns for programming just become nearly irrelevant.

I don't think that ever happened. Using relatively sparse amount of memory turns into better cache management which in turn usually improves performance drastically.

And in embedded stuff being good with memory management can make the difference between 'works' and 'fail'.

The need to use optimal patterns didn't go away, but the techniques certainly did. Just as a quick example, it's usually a bad idea now to use lookup tables to accelerate small math workloads. The lookup table creates memory pressure on the cache, which ends up degrading performance on modern systems. Back in the 1980s, lookup tables were by far the dominant technique because math was *slow.*

The way to approach this is to benchmark and then pick the best solution.

> Back in the 1980s, lookup tables were by far the dominant technique because math was slow.

This actually generalizes in a rather clean way: compared to the 1980s, you now want to cheaply compress data in memory and use succinct representations as much as practicable, since the extra compute involved in translating a more succinct representation into real data is practically free compared to even one extra cacheline fetch from RAM (which is now hundreds of cycles latency, and in parallel code often has surprisingly low throughput).

It’s a mad word where ultimate performance in one problem can require compressing data in ram and in another storing it uncompressed on disc.

The same atmosphere that makes bread hard makes crackers soft.

When was the last time you used mergesort because you had to?

Coincidentially, last night, and I'm not pulling your leg! But to be fair that's the first time in much more than a decade. I don't normally work with such huge files and this was one very rare exception. I also nearly crashed my machine by triggering the OOM killer after naively typing 'vi file' without first checking how large it had become. I'm working on a project that I probably should run on a more serious machine but I don't feel like moving my whole work environment from the laptop that I normally use.

It obviously never became completely irrelevant. But I think programmers spend a lot less time thinking about memory than they used to. People used to do a lot of gymnastics and crazy optimizations to fit stuff into memory. I do quite a bit of embedded programming and most of the time it seems easier for me to simply upgrade the MCU and spend 10cents more (or whatever) than to make any crazy optimimzations. But of course there are still cases where it makes sense.

While thinking less about memory optimizations is possible since we have more memory, it was enabled by the languages and libraries we use. Fourty years ago, you were probably implementing your own data structures. Sure, there were plenty of languages that offered them back then (LISP was based on linked lists, and that language is from the 1960's). Chances are you weren't using such languages unless you were using big computers or writing software that didn't handle much data. These days, pretty much any language will provide at least some data structures and their related algorithms. Even systems programming languages like C++ and Rust. Of course, there are an absurd number of libraries if you need anything more specialized.

I never really bought in to the anti-Leetcode crowd’s sentiment that it’s irrelevant. It has always mattered as a competitive edge, against other job candidates if you’re an employee or the competition of you’re a company. It only looked irrelevant because opportunities were everywhere during ZIRP, but good times never last.

Most developers work at banks, insurance companies and other “enterprise” jobs. Even most developers at BigTech and who are working “at scale” are building on top of scalable infrastructure and aren’t worrying about reversing a btree on a whiteboard.

Agree that the whiteboard thing is often not applicable but it's so nice when a developer has efficient code if only because it indicates that they know what's going on and also that there are fewer bugs and other bottlenecks in the system.

Those bugs don’t come from using the wrong algorithm, they come from not understanding the business case of what you’re writing. Most performance issues in the real world for most cases don’t have anything to do with the code. It’s networking, databases, etc.

Your login isn’t slow because the developer couldn’t do leetcode

No, it's because 50k reads of settings are happening with a SQL Table in memory that's queried via SQL statement instead of a key/value hashtable. (real world experience, I think it was close to 28k reads, but the point stands)

It mattered to pass through the interview, but not for the job itself. With all leetcode geniuses in Microsoft why Teams and Windows are so shitty?

Because they are only allowed to review what the LLM has come up with.

It's not like most developers are wasting memory for fun by using Electron etc. It's just the simplest way to deploy applications that require frequent multiplatform changes. Until you get Apple to approve native app changes faster and Linux users to agree on framework, app distribution, etc., it's the most optimal way to ship a product and not just a program.

> for fun

Not for fun but for convenience (laziness occasionally?). Someone needed to "pay" for the app being available on all platforms. Either the programmer by coding and optimizing multiple times, or the user by using a bloated unoptimized piece of software. The choice was made to have the user pay. It's been so long I doubt recent generations of coders could even do it differently.

> applications that require frequent multiplatform changes

Maybe a bit of engineering and planning could help here. Shipping always half finished products is it usually not a recipe to success.

RAM didn't get more expensive to produce. It just got more desirable. The prices will come down again when supply responds. It may take some time, but it will happen eventually.

RAM actually got more expensive to produce in the medium term because production is bottlenecked. It takes years to expand production.

RAM production is highly inelastic and controlled by an oligopoly. They have little desire to increase production considering the lead time and the risk that the AI demand might be transient.

They actively prefer keeping confortable margins than competing between each other. They have already been condemned for active collusion in the past.

New actors from China could shake things up a bit but the geopolitical situation makes that complicated. The market can stay broken for a long time.

They are increasing production as fast as they can (which is not fast at all, it's more like slowly steering a huge ship towards the correct direction) because current prices are too high even when accounting for the historical oligopoly dynamics. They can easily increase their collective profits by making more.

As far as I know, they are merely shifting capacities from the customer market towards the data center market with minimal retooling. I am unaware of any of the three actively investing in new capacity. Some modest increase are planned but nowhere near what you would expect given current demand.

RAM manufacturers don't increase production as fast as possible, because they've been through enough boom and bust.

Rapid increase in capacity leads to oversupply which leads to negative margins. They've been there before, and they don't want to go there again.

RAM manufacturers do routinely setup new fabs and decommision old fabs. Maybe they're trying to hurry up new fab construction in times like these, and they would likely defer shutting down old fabs or restart them where possible. But they're less likely to build new fabs that weren't already part of their long term plans.

They've actually not seen such prices before. DRAM now costs as much per Gb as it did around 2006-2007 - despite around 20 years of real technical progress since then! That's genuinely unprecedented.

We would have, if the expensive memory was a long term trend. It is not - eventually the supply will expand to match demand. There is no fundamental lack of raw materials underlying the issues, it is just a demand shock.

Also, it's not like we have regressed in the process itself either, which was historically the limiting factor. As you said this is purely an economics thing resulting from a greedy shift in business focus by e.g. Micron.

I just heard in a podcast, they talked about how powerful our devices are today but do not feel faster than they did 15 years ago and that it's because of what you write here.

A lot of that is on the OS vendors (and security requirements drive some inefficiencies that didn't used to be needed either).

I have a 2020 Intel Mac (quad core, 16gb RAM) and it feels as slow as the Packard Bell from 2000 when I was a kid. The launchpad takes 1-2 seconds to show a bunch of icons. Absolutely insane!

When I train some leetcode problems, I remember the best solution was the one that optimised cpu (time) instead of memory. Meaning adding data index in memory instead of iterating on the main data structure. I thought, ok, thats fine, it's normal, you can (could) always buy more RAM, but you can't buy more time.

But well, I think there is no right answer and there always be a trade off case by case depending on the context.

> I had my formative years in programming when memory usage was something you still worried about as a programmer.

As 'just' a user in the 1990s and MS-DOS, fiddling with QEMM was a bit of a craft to get what you wanted to run in the memory you had.

* https://en.wikipedia.org/wiki/QEMM

(Also, DESQview was awesome.)

I've recently started a side project for the N64, and this is very relatable! Working within such tight constraints is most of the fun.

Depends on the machine you’re targeting.

I do embedded Linux and ram usage is a major concern, same for other embedded applications.

I’m partying like it’s the 90s, on a 32-bit processor and a couple hundred MB of ram.

Android's investing significantly in reducing the memory usage of the next release simply because the BOM cost of RAM for their low-end partners is becoming prohibitive.

But if that new or different because of this event? No it's not, Android has had several initiatives to enable low end devices, from optimizing full fatter Android, to inventing new versions of Android.

In this case it is explicitly because of the RAMpocalypse. The initiatives have existed forever but they've gotten a lot more funding and a lot more exec attention because of the situation in the hardware market.

Android has been talking about these kinds of things for a long time. But if they're actually meaningfully making progress on them, it's most likely because of real pressure. (He types on his phone with 6GB of ram)

most likely in a couple years this bubble will pop, just like 8 years and 16 years ago

it's just a cartel cycle of gaining profits while soon eliminating all investments into competitors when flood of cheap ram "suddenly" appears

This is coming from an insane demand spike, not some nefarious plot by the RAM manufacturers.

Yes, it's a nefarious plot of AI producers to attempt a monopoly with a product that no one seems capable of demonstrating has the exponential value they're betting on.

Once everybody has a decent amount of VRAM they can just run local AIs and the need to mess with Ad-laden search results will fizzle. So of course they are desperate to grab a new monopoly. People haven't realised yet, that local AIs are fast and produce good results - on pretty average hardware. If they don't manage to grab a new monopoly Google will be history.

But it doesn't really need a nefarious plot for the price spikes. There is a serious lack of VRAM deployed out there. Filling that gap will take quite some time. Add to that the nefarious plot and the situation will most likely get even worse....

LLM inference is mostly read only, so high-bandwidth flash looks like it could provide huge cost savings over VRAM. It's not yet in commercial products but there are working prototypes already. Previous HN discussion:

https://news.ycombinator.com/item?id=46700384

Are you saying that intel’s optane product was just ahead of its time? Is optane the answer to LLM’s ever increasing appetite?

AI companies yes, RAM manufacturers no.

> This is coming from an insane demand spike, not some nefarious plot by the RAM manufacturers.

Something something, 2000 dot-com bubble, something

These somethings are doing so much work I can't tell if you're agreeing with them or not tbh.

I can never understand why so many people resort conspiracy theories when the obvious answer is supply and demand. I know well educated people, who do this when they talk about the resential property market. (Including an accountant).

Supply and demand can be caused by a conspiracy. OpenAI secretly bought 40% of the world's RAM on purpose. It's only a conspiracy if Anthropic and Google did something similar, though.

Which is in large part due to hoarding by OpenAI.

Although their stated reason for hoarding is that they "really need it", I think it was a strategic move to make their competitors' lives more difficult with little regard for the collateral consequences to non-competitors, such as regular people or companies needing new computers.

Eventually new capacity will come online, and the money the DRAM companies are making are going to accelerate even ,ore new capacity. If you can get your new capacity going before your competitors, maybe you can avoid a bubble burst. If you don’t build new capacity, your competitors will, etc, etc…

They're not building any new manufacturing capacity though. They assume this is a demand bubble and they don't want supply to exceed demand after it pops.

multiple major DRAM factories are currently being built or planned, driven by AI demand and government incentives. Micron is constructing a massive $100 billion "megafab" complex in New York, with groundbreaking occurring in January 2026, and is building new facilities in Idaho. Other projects include expansion in Singapore and Japan.

Key DRAM Factory Construction Projects:

Micron Technology (USA): Building a $100 billion, 4-fab complex in Clay, New York (first production expected around 2030) and a new $15 billion, 2-fab project in Boise, Idaho.

Micron (Global): Investing in expanding capacity in Singapore and Taiwan.

Nanya Technology (Taiwan): Previously initiated a $10.69 billion DRAM facility in New Taipei, Taiwan.

A quick search tells me the megafab in New York was announced years ago, the Singapore fab is for NAND flash, and the Taiwan fab already exists and they're buying it. So none of those are in response to the AI demand for RAM, are they?

I get that you are an AI skeptic but you can do better than that with a quick search these days. HBM for high end (commercial) GPUs.

SK Hynix

The current HBM market leader is fast-tracking multiple "megafabs" and packaging centers. Cheongju, South Korea (P&T7): A new $13 billion advanced packaging and testing plant dedicated to stacking and testing HBM chips. Construction is set to begin in April 2026, with completion by late 2027.

Cheongju, South Korea (M15X): This fab is being fast-tracked for HBM4 mass production, with the first cleanroom now expected to open in February 2027.

Yongin, South Korea: SK Hynix is investing roughly $22 billion in the first fab of a massive new semiconductor cluster. Operations are planned to start in February 2027.

West Lafayette, Indiana, USA: A $3.87 billion advanced packaging site that will integrate HBM directly onto GPUs. Construction fencing was installed in February 2026, with production targeted for late 2028.

Samsung Electronics

Samsung is accelerating its "Shell First" strategy to secure production space ahead of competitors.

Pyeongtaek, South Korea (P4 & P5): Samsung has advanced the construction of the P5 cleanroom by several months, with a new operational target of late 2027. The P4 line is expected to come online even earlier, likely during 2026.

Taylor, Texas, USA: This $17 billion "megafab" is designed for advanced logic and HBM packaging. While hit by delays, it is now targeting a late 2026 opening.

Micron Technology

Micron is diversifying its HBM production across the U.S. and Asia to grow its market share.

Boise, Idaho, USA (ID1 & ID2): The ID1 fab reached a key milestone in June 2025 and is expected to start wafer output in the second half of 2027. ID2 is planned to follow shortly after.

Onondaga County, New York, USA: Micron officially broke ground in January 2026 on a $100 billion "megafab" complex, though significant supply is not expected until near 2030.

Hiroshima, Japan: A planned $9.6 billion HBM-focused fab is expected to come online between 2027 and 2028.

Singapore & Taiwan: Micron began construction on a $24 billion wafer facility in Singapore in January 2026 and acquired a fab in Taiwan for $1.8 billion to rapidly expand DRAM capacity by late 2027.

For lower end GPUs, like what goes into Apple machines.

New LPDDR Production Facilities

Samsung (Pyeongtaek P4 & P5): Samsung is converting several NAND flash lines to DRAM and accelerating the P4 and P5 fabs in South Korea. While these fabs support HBM, they are also designed for mass-producing 6th-generation 1c DRAM, which will form the basis of the next-gen LPDDR6 modules expected to debut in 2026.

SK Hynix (Icheon & M15X): SK Hynix is planning an 8-fold increase in 1c DRAM production by the end of 2026. This capacity will be split between HBM and "general-purpose" DRAM, which includes the LPDDR variants used in mobile and laptop chips.

Micron (Boise, Idaho - ID1): Micron's new ID1 fab in Boise is currently under construction, with structural steel completion reached in late 2025. It is scheduled to begin wafer output in the second half of 2027, focusing on leading-edge DRAM that includes LPDDR for the U.S. market.

The "Memory Wall" for Apple

The primary challenge is that HBM production requires significantly more wafer area than standard LPDDR. Consequently, even as these new factories open, the shortage of commodity DRAM (LPDDR5X/LPDDR6) is expected to persist through 2028 because manufacturers find HBM far more profitable.

Reply on news.ycombinator.com