kkielhofner

One of the things driving this crazy spend on AI infrastructure by big tech is excess capital.

Big tech is getting increasing scrutiny by regulators, etc for monopolistic/anticompetitive practices/etc[0]. From S&P:

"Regulatory scrutiny is likely to translate into fewer mega deals, however. Adobe Inc. recently abandoned its $20 billion acquisition of Figma Inc. in the face of ongoing opposition from the UK's Competition and Markets Authority and the EU's European Commission. While Microsoft Corp. was able to close its $68.7 billion Activision Blizzard buy, it took nearly two years to complete the deal amid an intense fight with the regulators.

The harsh regulatory environment will continue to have a chilling effect on large tech M&A, but strategic buyers are likely to focus on smaller tuck-ins that do not tempt regulators to intervene."

So instead of growing further by spend on acquiring/rolling out new platforms to draw further scrutiny, they're shoveling piles of cash towards other capital expenses (with accounting tricks) that don't get further attention/scrutiny from regulators.

Meta doesn't use this infrastructure just for LLaMA - they get to use these hundreds of millions of dollars of spend to extract more value (deeper, not wider) from the users and platforms they already have.

Case in point: AI generated ads[1]. There's a not-too-distant future where the ads displayed on Facebook are an AI generated video of you in a new Toyota (or whatever). In terms of even open models Meta is much more than LLaMA. They have done a lot with speech (seamless), vision (segment anything), and much more.

That model training, inference, etc is running on these Nvidia GPUs as well.

There is also a take from Reid Hoffman saying that buying 100,000 GPUs or whatever is basically "table stakes" at this point[2]. Boards, analysts, the market, etc says "What I know for sure is this AI stuff needs Nvidia GPUs. Elon has 100k. How many do we have?".

So it's also a Cold War arms race/missile gap scenario. Plus, don't discount ego/"male anatomy measuring contest" that goes on with these guys. Consider this story where a bunch of centi-billionaires got together at Nobu to fight over Nvidia GPU shipments[3].

[0] - https://www.spglobal.com/marketintelligence/en/news-insights...

[1] - https://www.theverge.com/2024/10/8/24265065/meta-ai-edited-v...

[2] - https://www.theinformation.com/articles/openai-investor-hoff...

[3] - https://finance.yahoo.com/news/elon-musk-oracles-larry-ellis...

CPU, yes, but more importantly memory bandwidth.

An RTX 3090 (as one example) has nearly 1TB/s of memory bandwidth. You'd need at least 12 channels of the fastest proof-of-concept DDR5 on the planet to equal that.

If you have a discrete GPU, use an implementation that utilizes it because it's a completely different story.

Apple Silicon boasts impressive numbers on LLM inference because it has a unified CPU-GPU high-bandwidth (400GB/s IIRC) memory architecture.

Make sure to check out the existing Triton client libraries:

https://github.com/triton-inference-server/client

> I don't think NCCL + Deepspeed/FSDP are really an alternative to Scuda

I put "remote" in quotes because they're not direct equivalents but from a practical standpoint it's the alternate current approach.

> they all require the models in question to be designed for distributed training. They also require a lot of support in the libraries being used.

IME this has changed quite a bit. Between improved support for torch FSDP, Deepspeed, and especially HF Accelerate wrapping of each with transformer models it's been a while since I've had to put much (if any) work in.

That said if you're running random training scripts it likely won't "just work" but given larger models becoming more common I see a lot more torchrun, accelerate, deepspeed, etc in READMEs and code.

> This has been a struggle for data scientists for a while now. I haven't seen a good solution to allow a data scientist to work locally, but utilize GPUs remotely, without basically just developing remotely (through a VM or Jupyter)

Remotely, as in over the internet? 400gb ethernet is already too slow vs PCIe5 x16 (forget SXM). A 10gb internet connection is 40x slower (plus latency impacts).

Remote development via the internet with scuda would be impossibly and completely uselessly slow.

This is very interesting but many of the motivations listed are far better served with alternate approaches.

For "remote" model training there is NCCL + Deepspeed/FSDP/etc. For remote inferencing there are solutions like Triton Inference Server[0] that can do very high-performance hosting of any model for inference. For LLMs specifically there are nearly countless implementations.

That said, the ability to use this for testing is interesting but I wonder about GPU contention and as others have noted the performance of such a solution will be terrible even with relatively high speed interconnect (100/400gb ethernet, etc).

NCCL has been optimized to support DMA directly between network interfaces and GPUs which is of course considerably faster than solutions like this. Triton can also make use of shared memory, mmap, NCCL, MPI, etc which is one of the many tricks it uses for very performant inference - even across multiple chassis over another network layer.

[0] - https://github.com/triton-inference-server/server

Hah, it actually gets worse. What I was describing was the Triton ONNX backend with the OpenVINO execution accelerator[0] (not the OpenVINO backend itself). Clear as mud, right?

Your issue here is model performance with the additional challenge of offering it over a network socket across multiple requests and doing so in a performant manner.

Triton does things like dynamic batching[1] where throughput is increased significantly by aggregating disparate requests into one pass through the GPU.

A docker container for torch, ONNX, OpenVINO, etc isn't even natively going to offer a network socket. This is where people try to do things like rolling their own FastAPI API implementation (or something) only to discover it completely falls apart at any kind of load. That's development effort as well but it's a waste of time.

[0] - https://github.com/triton-inference-server/onnxruntime_backe...

[1] - https://docs.nvidia.com/deeplearning/triton-inference-server...

The model you linked is not an LLM either by architecture or size.

A few thoughts:

1) TensorRT anything isn’t an option because it requires Nvidia GPUs.

2) The serving frameworks you linked likely don’t support the architecture of this model, and even if they did they have varying levels of support for CPU.

3) I’m not terribly familiar with Hetzner but those instance types seem very low-end.

The model you linked has already been converted to ONNX. Your best bet (probably) is to take the ONNX model and load it in Triton Inference Server. Of course Triton is focused on Nvidia/CUDA but if it doesn’t find an Nvidia GPU it will load the model(s) to CPU. You can then do some performance testing in terms of requests/s but prepare to not be impressed…

Then you could look at (probably) int8 quantization of the model via the variety of available approaches (ONNX itself, Intel Neural Compressor, etc). With Triton specifically you should also look at Openvino CPU execution accelerator support. You will need to see if any of these dramatically impact the quality of the model.

Overall I think “good, fast, cheap: pick two” definitely applies here and even implementing what I’ve described is a fairly significant amount of development effort.

Do you feel like the overall volume level changes based on the general genre of film?

As I understand it:

Action movies, for example, have to factor in audio for everything from whispers to big explosions. In order to do that properly (avoiding clipping, etc) you have to look at the dynamic range. Then the problem becomes trying to make sure most people can hear the whispers while not blowing them away with the explosion at the other end of amplitude.

If the film is just dialog at relatively consistent volumes the necessary dynamic range gets narrower and you can (pretty much) settle on a much more reasonable "sweet spot" output volume.

Of course there are normalizers/compressors/etc and other forms of processing you can get heavy handed with but they're not really that popular when it comes to art because they arguably modify the art itself. There's even a specific Academy Award for sound[0].

When it comes to music it's a different story as we are well into the loudness war[1].

Anyway, to answer your question: probably not.

[0] - https://en.wikipedia.org/wiki/Academy_Award_for_Best_Sound

[1] - https://en.wikipedia.org/wiki/Loudness_war

> burning GPUs and pushing CO2 in the atmosphere

My startup develops AI for the nuclear power industry to drive process, documentation, and regulatory efficiency. We like to say "AI needs nuclear and nuclear needs AI".

Big tech has finally realized/gone public that casually saying things like "we're building our next 1GW datacenter" is uhh, problematic[0].

For some time now there has been significant interest/activity in wiring up entire datacenters to nuclear reactors (existing Gen 2, SMRs, etc):

https://finance.yahoo.com/news/nvidia-huang-says-nuclear-pow...

https://www.ans.org/news/article-5842/amazon-buys-nuclearpow...

https://www.yahoo.com/news/microsoft-signs-groundbreaking-en...

https://www.cnbc.com/2024/09/10/oracle-is-designing-a-data-c...

https://thehill.com/policy/technology/4913714-google-ceo-eye...

[0] - https://www.npr.org/2024/07/12/g-s1-9545/ai-brings-soaring-e...

Synthetic data (data from some kind of generative AI) has been used in some form or another for quite some time[0]. The license for LLaMA 3.1 has been updated to specifically allow its use for generation of synthetic training data. Famously, there is a ToS clause from OpenAI in terms of using them for data generation for other models but it's not enforced ATM. It's pretty common/typical to look through a model card, paper, etc and see the use of an LLM or other generative AI for some form of synthetic data generation in the development process - various stages of data prep, training, evaluation, etc.

Phi is another really good example but that's already covered from the article.

[0] - https://www.latent.space/i/146879553/synthetic-data-is-all-y...

Additionally, they've been on both sides.

If they are looking to invest in a company when they do technical due diligence and bring in a source code auditing company like Synopsis Black Duck any AGPL you're using is so problematic for them it can be a deal breaker. At a minimum it's such a major sticking point it can be one of the most significant things to hold up a transaction as you try to explain why it isn't as problematic as they think.

Having been through that process a couple of times I won't touch AGPL because it's such a PIA - your poison pill.

On the flip side, if they have or are investing in you and and you've made some aspect of your solution open source under AGPL they know any competitor using it is going to have challenges getting VC investment (see point above).

> just like bitcoin

The problem with this comparison is Bitcoin has basically just been SHA256 for 15 years and likely will continue to be for some time.

Transformers have been mostly dominant for at least several years but there are still other archs (CNN, RNN, etc) in various use-cases and we're already seeing nearly-fundamental changes in Transformers and "emerging" approaches like Mamba, RWKV, hybrids, etc. Transformers have shown remarkable versatility and adaptability (that's their whole thing) but it's already creaking and showing its age.

Startups building Transformer-specific silicon are playing a very risky game that is already somewhat problematic now and almost certainly won't end well.

AI is much newer, much more vast, and moving much more quickly. The ASIC design, tape out, manufacture, software ecosystem, actually getting to market, etc cycle is fundamentally too long and I suspect even the Transformer-specific silicon we see now will be viewed as a major blunder in the relatively near future:

"Oh yeah, remember those graveyard companies that did transformer silicon back in the first AI hype round?"

I cannot see how anything other than GPGPU, TPU, NPU, etc (or similar "generic" approaches) will have legs.

Growing up my dad was a very well known PhD in his field (Occupational Therapy).

There were quite a few people who insisted on using Doctor when referring to others, calling themselves Doctor, etc.

He never did but I experienced it quite a bit.

Hah, that’s amazing!

Now I get to tell him this story next time I see him :).

I have a project on Frontier - happy to answer any questions!

Funny story about Bronson Messer (quoted in the article):

On my first trip to Oak Ridge we went on a tour of “The Machine”. Afterwards we were hanging out on the observation deck and got introduced to something like 10 people.

Everyone at Oak Ridge is just Tom, Bob, etc. No titles or any of that stuff - I’m not sure I’ve ever heard anyone refer to themselves or anyone else as “Doctor”.

Anyway, the guy to my right asks me a question about ML frameworks or something (don’t even remember it specifically). Then he says “Sorry, I’m sure that seems like a really basic question, I’m still learning this stuff. I’m a nuclear astrophysicist by training”.

Then someone yells out “AND a three-time Jeopardy champion”! Everyone laughs.

You guessed it, guy was Bronson.

Place is wild.

> There is nothing about going to MIT that makes me think

That makes you think (no shade)... Question is: does it bother you enough to not proceed in engaging? Likely not. The upside is other audiences that love that kind of stuff - investors, other customers, individuals, etc.

Mostly neutral to positive, little negative.

> I want to know what the current power requirements are, as well as the cost for the machine.

"If you care and have to ask it's not for you".

In all seriousness I've worked with and am familiar with Cerebras, Groq, etc. Let's just say GPUs still reign supreme in terms of practicality outside of usage of their hardware via cloud for nearly all use-cases.

Groq, for example, has essentially stopped selling their "real" HW directly because the borderline absurd amount of floor space, etc was found to be challenging once they hit the market. There's enough demand and more (recurring) money to be made anyway hosting services on your chips.

Similar to the Bitcoin mining ASIC game in the heyday - sure we could sell these or we could just use them to mine, develop next gen, sell previous gen, repeat.

Hard disagree. A solid chunk of the value in attending prestigious universities is being able to to drop names and logos in this exact case.

Like it or not touting that your team has pedigreed educational backgrounds is something you put up front and center.

Omi has a pretty interesting architecture that has been completely open-sourced:

- The device itself is a Nordic Semiconductor BLE chip with GPIO, mic, and BLE.

- Audio is streamed (as OPUS) via BLE to the associated mobile device where some initial processing is handled.

- The audio is then passed to a FastAPI-based backend API service that handles integrations with Deepgram, etc.

Overall I think it's a clever way to handle this: you get to use very cheap hardware that sips power (battery) while utilizing the connectivity of the associated mobile device whether it be WiFi or cellular.

Human medical errors have been one of the leading causes of death[0] since we started tracking it (at least decades).

The healthcare system has always killed plenty of people because humans are notoriously unreliable, fallible, etc.

It is such a stubborn, critical, and well-known issue in healthcare I welcome AI to be deployed slowly and responsibly to see what happens because the situation hasn’t been significantly improved with everything else we’ve thrown at it.

[0] - https://www.ncbi.nlm.nih.gov/books/NBK225187/

Absolutely.

For some reason inference seems to be overlooked. A lot of “ink” has been spilled over GPUs for training tasks but at the end of the day if you can’t do inference you can’t serve users and you can’t make money.

I remember seeing my first of these in a small beachside hotel (pousada) in Brasil.

In this case the wiring was over-stripped and copper was exposed at the shower head. I used it but I was very, very careful not to splash…

> You could be out of a job with as little control of the whole situation as those rustbeltians you have such disregard for.

I'll admit I was a bit angry and offended when I first read this. Did you miss the part about how horribly this has affected my friends, family, and neighbors? I say this expressly out of concern and literal, actual love.

Where do you live?

> NAFTA and globalism killed the rustbelt.

As I said in my other reply the generation before this one also outright dismissed the ability for Mexico, China, etc to take their jobs. In any case it certainly wasn't prevented.

> The refusal of auto plant Larry to learn Java didn’t.

Again, as I said in my other reply what little manufacturing is left is a fraction of the jobs with a completely different skill set - CNC programming, etc. While you're off on Java because (frankly) you don't know what you're talking about it's pretty much exactly this.

> Or they have developed a good filter for BS marketing.

Maybe. Or (maybe) yelling at the TV is a lot easier (and more visceral) than thinking "Hmm, this whole AI thing just might work out. Maybe I should treat the threat seriously this time just like my dad should have with the Chinese, robots, etc". As I said in my other reply it's generally not a good idea to outright dismiss what is even in the early stages a pretty clear potential threat to your livelihood.

> The markets can adapt also.

This can mean nearly anything and I'm not sure what you're trying to communicate in this context.

> It's good that people scrutinize and resist corporate greed.

I agree. Problem is it didn't do anything when they were saying the exact same things that led to the closure or significant job loss of the manufacturing that was the core of these communities:

1) "Hah, my factory job is complicated. The Mexicans, Chinese, Japanese, etc will never figure it out." Racist/xenophobic but said at the time. Even more wrong.

2) Look at robotics. It also took some time for the technology to develop and many of these same people looked at early implementations and thought the exact same thing: "Oh those robots don't really work and they never will". Then their factory jobs became a fraction of the head count with a completely different skill set...

Throughout history this has played out over and over again.

> People have the power to vote with their wallets (and their actual votes), and they should do so.

Key word being /should/. Yet it has been proven time and time again that people regularly vote against their own best economic interests in the ballot box and with their wallets. The same people who laughed at and dismissed the Chinese are going to Wal-Mart everyday and buying goods from (you guessed it) - China. They're also voting for the same corporate-funded politicians.

For many of these people their best option now is working at the large Amazon warehouse down the road where they pick and pack goods largely from (you guessed it again) - China. But even then there just aren't that many of those jobs and as the horror stories reported in the media show they're routinely abused there.

> Now, I'm not American and not an expert on the Rust Belt, but I'm pretty certain this is not the reason why the Rust Belt exists.

Next time you're in the US visiting New York, California, Florida, etc come visit the rust belt in "flyover country" and see just how devastating the effects of this are. I see and live it (to some extent) everyday.

> Probably rather something to do with corporate greed and greedy politicians.

A huge factor and the ultimate root of the problem but back to voting, wallets, etc: in the real world this isn't going anywhere and from what I see it's just getting worse.

My concern comes from seeing the blight and literal deaths in MY community. My concern now is AI is coming for the white collar workforce as well.

Another old adage:

"Don't underestimate your enemy/competition".

I grew up and live in small town Wisconsin. Sounds very believable.

I say this with the utmost respect and concern for my friends, family, and neighbors:

This attitude is part of the reason why the rust belt exists - filled with towns and cities that are a shell of their former selves. Areas of constant economic hardship, endemic drug use, etc.

The two big factories in the town I went to high school in are closed. Needless to say things aren’t going well… Yelling at the TV didn’t do anything to prevent that and it’s not going to do anything to prevent the next wave of it.

It’s harsh, but an important lesson since the beginning of time:

Adapt or die.

Naming anything is hard and I don’t have better suggestions but when you’re doing something that’s already poking at something a big corp holds dearly hitting on trademark while you’re at it makes it really easy for them.

“Weirdo” is an interesting take.

Richard Heart has been charged by the SEC for the usual crypto scam stuff and spent many months dodging subpoenas.

Charges and subpoenas need their day in court of course but it reads like the quintessential crypto story of embezzlement, fraud, etc for a guy to flash Veblen goods on social media.

ChatGPT hit 100 million users in two months (previous record was TikTok in nine). It’s currently estimated to be at least double that…

In less than two years.

Coinbase has been around since 2012 and has roughly 100 million users.

Yes ChatGPT is free but let’s not kid ourselves - most Coinbase users are on the platform essentially gambling with the hopes of making money. Get rich quick is an extremely powerful motivator…

In terms of actual usage, do some basic analysis of on chain activity for the top X crypto whatever. You’ll quickly discover usage is comically low outside of what is clearly trading and a tiny number of hardcore cryptocurrency nerds swapping around whatever for who knows what use cases.

People are generally good about figuring out what is useful to them, what improves their lives, and what they enjoy. 15 years after the advent of bitcoin the use cases for mass adoption (and therefore users) just aren’t there.

Crypto tech can be interesting but it’s just that - interesting. I have at least a dozen wallets on various chains, I’ve written smart contracts, etc. I can honestly say the entire thing could disappear tomorrow and I wouldn’t know the difference. Other than one friend at a Cardano-related startup I don’t know anyone else who would notice either.

ChatGPT alone, meanwhile, would be a loss I and everyone I know would feel immediately - and that’s just ChatGPT.

“AI” is much more than LLMs…

Not only have all three of these companies released LLM model weights, Google and Microsoft especially have released nearly countless model architectures, weights, evaluation/training/inference/etc frameworks, toolkits, etc across an extremely wide spectrum.