Hacker News Reader

I interned at LLNL on three separate occasions during school. Everything about what y'all do in high-performance computing is levels beyond anything I've gone on to see in the "real world". I cannot drop enough praise.

thanks!

In many similar contexts, buying hardware is different than buying services (e.g., AWS time), in terms of how you interact with funding agencies. Because of those realities, you might sometimes be unable to "go cloud" even if it's actually cheaper.

Also, licensing can be a big problem. Some software packages understand licensing for a "real" cluster, but have no concept of how to license for the cloud.

Finally, think about all of your interactions over the last ten years with companies behind the major cloud services. Which of them would you really trust, if it's your money/job/reputation on the line?

I'm a former LLNL lattice QCD postdoc and current user of the LLNL machines (and others). I have to say, I was wildly spoiled.

Every other computing center where I have an account cannot hold a candle to the helpfulness of the LLNL support, the ease of use (except for the 2-factor requirement, which I understand), and absolutely insane power.

> All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud.

It's weird that these networking technologies are not used more in "plain" datacentre settings, since networking latency and throughput has to be a significant challenge to scaling up non-trivial workloads and achieving true datacentre-scale computing. We hear a lot about how to "scale out", but that's only really feasible for relatively simple workloads where you just seek to do away with the whole issue of keeping different nodes in sync on a real-time basis, and accept the resulting compromises. In many cases, that's just not going to be enough.

There are a lot of people from the National Lab super computer world who end up in High Frequency Trading for just the reason you describe.

Specifically, how do you optimize a large cluster of computers to operate at the lowest possible latency. For the National Labs, those computers could be in the lab or with other labs around the world. For the HFT folks, the machines could be in an exchange or spread across multiple exchanges around the world.

Source: I used to be head of Global Latency Monitoring for a HFT.

Hey I’d love to pick your brain about what it takes to do this kind of work coming from a web/systems engineering background. Shoot me an email at samheutmaker at gmail if you’re open to it.

I'm curious why you moved to LLNL from HFT?

Money is a safe guess. Research pay scales aren't even close to private sector, especially not finance.

It sounds like the opposite direction happened here.

Most business workloads simply aren't CPU bound, nor amenable to parallel computation. The public clouds would offer it if they had demand for it.

Surely network latency (and perhaps throughput) is just as relevant to IO-bound and transactional workloads though? At least if they need to be served by a multiple-node cluster.

Umm, most workloads aren't so compute intense that cpu interconnect speed would have a huge impact. The vast majority of rented computer tasks will be for things that need to communicate with external services (payments, region synchronizing, general api state comms, etc). In these cases, I doubt that your code is tuned tightly enough to reveal interconnect latency (as opposed to cache misses and swaps). Besides, hooking up interconnect runs gets ludicrously complicated and expensive fast. Like, $50k in cables and a week of experimenting with numa zones and kernel settings for a small cluster. I can only imagine the headaches of orchestration on national lab clusters.

Source: I have set up, and currently manage a small cluster which is always pegged. I also collaborate with folks at a couple of the national labs for scientific research.

Infiniband and friends aren't cpu interconnects, they're alternatives to ethernet. Network latency is important and it's not that hard to write code whose performance is bounded by io latency/throughput.

Well they can be both, so I can see how parent might have gotten confused.

True enough.

> it's not that hard to write code whose performance is bounded by io latency/throughput

I suspect that most people don't, though. How many applications are built in Ruby, Python, or PHP, using some very productive but inefficient framework? Those are all very fine tools in their own way, and are the right tool for the job in many contexts. But they're unlikely to saturate IO.

Those technologies, or mostly IB, were more important when Ethernet speeds lagged behind. IB was first to 100G. Now that Ethernet has caught up, IB has almost no advantage besides a tiny latency decrease. Even mellanox makes more Ethernet than IB products now.

I thought process-to-process, Ethernet has still several times larger latency than IB?

We're talking about a few hundred nanoseconds usually. If that matters, then IB is still better.

> it's our job to stay on the bleeding edge, and I'm not sure how we would do that without working with vendors through these procurements and running our own systems.

I think this is key. Also the fact that you guys have a land grant so your datacenter costs are cheaper than most. :)

I grew up across the street from the the Lab on (Cheryl Dr.). Now early 30s, I’m a AWS architect and software engineer at a finance company in Bozeman, Montana, but I would kill to be an engineer for pursuit of knowledge ya know? I dream of working there some day.. and I miss home

Hey, I’m in Bozeman too. We should get lunch sometime.

Sorry about a slightly off-topic comment. Working on a deep tech startup, with a significant segment of potential customers being national labs, I'm very interested in learning various aspects [beyond the basics] about the process(es) that national labs use to procure application software (not the system one [OS distributions, monitoring etc.], but rather domain-specific one, e.g., scientific data management, modeling and simulations). I have looked at relevant sections on some of the labs' websites and my initial impression is quite fuzzy. If you are familiar with this subject, I would very much appreciate if you could clarify that. If you prefer, you can reach me at aleksandr <dot> blekh <at> Google's e-mail service.

P.S. I do have connections at some national labs, but thought that you might shed some additional light on this.

I used to work at Pacific Northwest National Laboratory. Their flagship computational chemistry application. NWChem, is developed mostly by their own employees. They also collaborate with researchers from other national labs and academia. Development started in the 1990s. When I was at the lab, it was free-as-in-beer but required onerous licensing agreements to get a copy of the source code. It has been open source since 2007. It has started using github to coordinate development in the last few years.

http://www.nwchem-sw.org/index.php/Main_Page

https://github.com/nwchemgit/nwchem/

I appreciate your comment and links (BTW, I'm slightly familiar with NWChem and similar software :-). Having said that, I'm not sure how your comment addresses my question. You are talking about internally-developed software transfer from the lab to external users, whereas my use case is delivering externally-developed commercial software to national labs' users, a value transfer in the opposite direction. Care to comment on this aspect, if you're familiar with relevant procurement processes?

I can speak at least a little bit about my field (computational chemistry). A lot of academic software is free and/or open source (like NWChem). These are installed by the cluster support for use by anyone. This is helpful because the software can be optimized beyond what most end users can do.

Beyond that, a lot of academics run custom software or uncommon packages. By definition, many academics are doing something no one else has really done, so they are writing their own software and then running that on the clusters.

There is one large commercial package that takes up a significant amount of time on many clusters -- VASP. But no idea the details of how that ends up on HPC systems :)

My impression from running on lots of clusters is that the software installed is more by 'pull' (user request) than 'push' (cluster dictating what is used). Clusters have to be very accommodating of user software.

If you are working in computational chemistry, I would be interested in hearing more details (if you are willing).

I appreciate your feedback. My question above is not about the mechanics of deploying commercial scientific software (it certainly somewhat varies between fields and organizations, though common themes and methods are pretty standard). It is about the procurement processes within national labs ecosystem. That is, about governmental gating / filters "in front of" vendors' go-to-market strategy and processes (RFPs, pilots etc.).

Re: more details - I'm not working in computational chemistry per se, but in the adjacent and closely related, as you understand, field of materials science and engineering. Will be happy to discuss my plans (within limits - currently very early and in stealth) through direct channels. You can connect with me on LinkedIn or shoot me an e-mail (see above).

We do visual GPU graph tech close to core national lab missions (sec, fraud, misinfo, genetics, social, ...) so are in a similar boat.

While I love the folks doing related work at PNNL (and am inspired!), I also recognize, done wrong, we can be existentially threatening to internal groups building related things.

Instead of aggressively persuing contracts and spending effort getting into nasty politics, we let the users come to us: we are there when the internal tools groups rather do other stuff vs do the multi-year investment. We are now focusing a lot more on enabling researchers to find and try us on public cloud / the web -- this road takes literally years longer for reaching those teams, but these are government labs, so that's the default assumption to begin with. (Likewise, not the cornerstone of our revenue.) Years later, folks are starting to write RFPs using us.

As a scientist, I love the community, and respect that we should assist it appropriately.

Thank you for sharing your thoughts. If I understood you correctly (the text's meaning appears somewhat fuzzy), your go-to-market strategy is to first collaborate (and/or consult?) - perhaps, through your "PoCs and Rapid Prototyping" services - and then allow users to "pull" your capabilities and core service into their labs. Right?

Having said that, I haven't seen any science-focused channels, resources or offers on your website ...

I do not advocate a go to market that prioritizes these markets. We started more with enterprise/federal data architects, which involved a different go to market - not self serve on a website.

We (and our broader market) are now at point where it is easier for us to launch self-serve and help adjacent roles. Stay tuned :) This fits better with national labs market too afaict. We already quite heavily discount on-prem for edu etc, and I expect we will see that carry through here: bottom-up early cloud use -> discounted on-prem. As budgets are mostly hw or small, and follow slow grant/annual cycles, hard to see going another way.

Two market trends relevant: -- Federal: without quite special circumstances, can easily take 3+ years for a new fed sales team to pay for itself -- Edu: small budgets to begin with Better to target fast happy-to-spend teams and as tech/sales/product gets easy, easier to help adjacent markets. Got burned whenever we didn't :)

I appreciate your additional insights. Will take time to parse it in detail. Just to clarify, my planned go-to-market strategy is hybrid (white-glove, typical for complex B2B solutions, and self-serve), since my target users - researchers and engineers - are from several major sectors (in the order of current priority): large-to-medium enterprises, national labs, small companies, academia (both research and education) - the last two are of ~ equal priority. Currently, I'm not too concerned about structuring the exact mix as well as tactical details of this strategy implementation - first, I need to validate my core ideas (from scientific and technological perspectives, not from the market validation perspective) and build a decent enough (for B2B enterprise) MVP. Easier said than done ... Oh well. :-)

I misunderstood your original question. When you asked how they procure domain specific application software I thought you meant "how do they get it?," which includes writing it themselves. (Not just procure as in "procurement" processes.)

No problem, we are on the same page now. Thank you, again, for your feedback.

didn't they EOL omnipath last year?

Yes but we still have many OP systems on the floor (see the link above).

Does it run Fortran?

Some, but LLNL codes are mostly C++, especially now that they need to make effective use of GPUs.