LUMI, Europe’s most powerful supercomputer

I don’t think that is realistic, and your word “every” is doing some very heavy lifting to give an overly simplistic answer. Renewable electricity is over 50% in NZ[0]. From a pure economic argument, green demand would need to approach green supply before before hydro could gain much of an excess price. Paradoxically, it seems plausible to me that electricity generation companies have enough moat to be able to just squeeze profits out, and not increase green generation at all (regardless of your hypothetical financial incentives). Most renewable requires huge capital investments, and a few percent extra profit can easily not move the profitability needle enough to change a project from a no to a go.

I would like to see a good economic and ecological comparison of the regulatory/tax choices the government could make, because I am somewhat suspicious about the recent electric-car subsidy. I have an acquaintance who has been part of a very well planned ($x million invested) solar project for NZ with an expected capital cost in the hundreds of millions, and anecdotally the government doesn’t seem to give a shit about it. Maybe they have the wrong political connections? Hydro is the same as a battery, so solar appears to be very sensible in NZ.

> aluminium smelter [claims] it is powered by green energy

I already answered that one in the comment you are replying to. More specifically: our NZ government has decided for decades to give Tiwai Smelter cheap electricity contracts (maybe for the sake of 2500 jobs), and finally decided to pull the plug on them. NZ consumers have had to buy marginally expensive and mostly non-renewable electricity over many decades due to that government choice. I haven’t looked at the country economics for the smelter, but I strongly suspect our government has been making uneconomic political decisions that have cost the country dearly. I presume they will put in a link from Manapouri to Benmore (connect to the Benmore to Wellington DC link, which isn’t 100% capacity constrained?[3]). I would also guess that the government was politically pressured to have more green generation for NZ, and shutting down Tiwai finally made political sense on the world stage to achieve “more” renewable generation.

Who wants another 1992 due to mis-planning for our security of supply? 1992 was perhaps historical blindness given that: “the power crises of the early 50s when serious shortfalls in supply created shutdowns and blackouts throughout the country. These were a regular occurrence in the years between 1954 and 1957”[1]. Although I guess I would need to see a past analysis of the risks versus the costs to judge whether the 1992 economic GDP reduction was just an acceptable cost versus the costs of upgrading generation. Governments generally are motivated by quick wins, and are sometimes not motivated to use regulation to encourage long term risk reduction against fat-tail events.

[0] https://www.mbie.govt.nz/assets/Data-Files/Energy/nz-energy-...

[1] https://www.engineeringnz.org/programmes/heritage/heritage-r...

[2] 2020: “Transpower has indicated it will cost $600 million to put the transmission structures in place to take the power from Southland to Auckland. That work was due to be completed over the next five to seven years.” The majority of Auckland electricity consumption is residential, the remainder of usage mostly commercial, and Glenbrook mill is less than 10%. https://www.stuff.co.nz/national/122113863/the-power-game-wh...

[3] https://www.transpower.co.nz/power-system-live-data

Unless the grid network connection is constrained, it certainly doesn’t “come” from nearby hydro because 1kWh is mostly fungible. Transmission costs are likely low. That is the point - although I admit marginal economics are confusing and most people don’t try to understand it (Disclaimer: I don’t have a good grasp of it either).

The grid connection is not likely to be constrained, because that would likely mean no redundant power. I assume even government data centers like reliable power supply: commercial data centres usually plan to be located where there is power redundancy for availability uptime. Although constraints are often only in one direction, so hard to predict just from a redundancy argument. Details of the network matter, however I am not motivated to find that exact information.

A hypothetical analogy would be to think of a lake in Lakeland. Currently the lake water is supplied 100 units by a clean green river and 100 units by a desalination plant (producing equivalently clean water, but using wads of dirty fossil fueled power to produce). 200 units of water is currently taken out of the lake by the residents of Lakeland. A data center is added that takes 50 units of water from the lake, and they sign up with the local Lakeland government that contractually agrees that the water the data centre gets from the lake is clean green water from the river. The desalination plant now needs to produce 150 units of water. Analogy: the lake is the power grid, the river is hydropower generation, and the desalination plant is fossil fuel power generation. You are saying the data center is getting water from the river (because contracts), and I am saying the extra marginal demand causes greenwashed fossil fuel generation.

> Finland uses very little fossil fuels for electricity

But nearly 100% of added power usage is supplied by non-renewable resources. That is the core of the likely greenwashing, and the point I am making about knowing where 1kWh of extra generation comes from, if 1kWh of load is marginally added.

I know I am repeating what I said, but that is because you are not arguing against the points I made.

There is a sort of argument that Finland will have 100% renewable power in the future. But if Finland has cross-connections to countries with dirty power, then the marginal argument still holds, just as though the lake in Lakeland is connected with lakes in other countries (European grid?)

To be called green:

1: add new green generation that wouldn’t otherwise be added. BUT if solar and wind generation installation is being installed at a maximum rate around the world due to manufacturing capacity limits, then you can’t claim “your” green generation unless you also increase manufacturing capacity limits. There’s a parallel to the lake analogy.

2: or reduce power usage that wouldn’t otherwise have been reduced. Usually hard to actually account for (the reduction needs is difficult to get right).

3: AND while doing the above, don’t spend a bunch of uneconomic money, because spending money is usually wasteful and indirectly generates carbon.

It is hard to do the above. However it is almost free to just label your power as “green”. Most projects choose a green label. Very very few new projects marginally reduce carbon dioxide - because new projects are usually marginal increases in power usage.

> 2000: Dell announces that it is now the No. 2 provider of Linux-based systems worldwide and the first major manufacturer to offer Linux across its full product line

https://en.wikipedia.org/wiki/History_of_Linux

2019 => https://newsroom.ibm.com/2019-07-09-IBM-Closes-Landmark-Acqu...

Linux never had any issues becoming yet another UNIX clone, hence why its strong points are CLI and headless applications.

I bet LUMI researchers won't be doing data analysis on their laptops from Linux distributions.

See also: Mitsubishi Pajero, Mazda Laputa.

Doesn't look like in a quick glance

In addition, the performance engineers tend to be employed by the facilities, not the computational scientists. They're the ones who do a bunch of legwork of profiling the existing code on their new platform, and figuring out how to squeeze any machine-specific performance out of the code.

A lot of these codes are time-marching PDE solvers that do a bunch of matrix math to advance the simulation, so the kernel of the code is responsible for a vast majority of the time spent during a job. So it's not necessarily a huge chunk of code that needs to be tuned to wring better performance out of the machine.

The parallel communication they do is also to an API, not an ABI - the supercomputing vendors drop in the optimizations in the build of the library for their machine, to take advantage of network-specific optimizations for various communications patterns. If you express your code in the most-specific function (doing a collective all-to-all explicitly, say, rather than building your own all-to-all out of the point-to-point primitive) the MPI build can insert optimized code for those cases.

There's some misalignment because the facility will be in the top 500 for a few years, while the code lives on and on and on. If your supercomputer architecture is really out of left field (https://en.wikipedia.org/wiki/Roadrunner_(supercomputer)) it's not going to be super worth it for people to try to run on it without porting support from the facility.

The end result mostly depends on the balance between scientists and engineers in the development team, it will oscillates between "this is python because the scientists working on the code know only that but we are using MPI to at least use several cores" and "we have a direct line with the hardware vendors in order to help us write the best software possible for this thing".

For something like climate simulations where a project is running big long jobs repeatedly I imagine they spend quite a bit of time on making it fast.

For something like detector development where you run the hardware simulation production once and then spend three years trying to find the best way to reconstruct events less effort is put into making it fast. Saving two months from a six months job you run once isn't worth it if you have to spend more than a few weeks optimising it, and as these type of jobs need to write a lot to disk there's a limit to how much you'll get from optimising the hot loop.

What's interesting is that over time, the datacenter folks ended up adding supercomputers to their datacenters, with very large and fast database/blob storage/data warehousing systems connected up to "ML supercomputers" (like supercomputers, but typically only do single precision floating point). The two work well together so long as you scale the bandwidth between them. At the end of the day, any interesting data center has obscenely complex networking technology. For example, TPUs are PCI-attached devices in Google data centers; they plug into server machines just like GPUs. The TPUs themselves have networking between TPUs, that allows them to move important data, like gradients, between TPUs, as needed to do gradient descent and other operations, but the hosts that the TPUs are plugged into have their own networks. The TPUs form a mesh- the latest TPUs form a 3D mesh, but physically implemented through a complex optical switch, while the hosts they are attached to multiple switches which themselves from complex graphs of networking elements. When running ML, part of your job might be using the host CPU to read in training data and transform it, keeping the network busy, keeping some remote disk servers busy, while pushing the transformed data into the TPUs, which then communicate internal data between themselves and other TPUs, over an entirely distinct network. Crazy stuff.

Why do not DCs appear? Because they have not submitted benchmarks and power measurements.

The rest kind of follows from that, like how a supercomputer that consists of multiple computers needs a fast, low-latency interconnect between them to coordinate and exchange results, while computers in a DC care a lot less about each other.

On the other hand the distinction is fluid. Google could call the indexers that power their search engine a supercomputer, but they prefer to talk about datacenters