cavisne

I guess their argument is the reason those things suck is because they are hooking in some c++ monstrosity (tensorflow) or making rpcs to an external daemon (redis, horovod).

So rather than writing another Python wrapper over c++ they are making a new performant language that can call Python.

To me it makes sense as torch is great and hard to compete with, but everything feeding into it is a mess today (Data loading, distribution logic).

Yes, but to move the needle you need to use another single vendor Mellanox (which nvidia owns).

So the (slight) differentiations are.

AWS - gpudirect and high bandwidth (but also high latency)

GCP - 16 gpus on one machine

Azure/OCI - mellanox

GPU's in cloud is just a bad business unfortunately.

There is only one vendor (for anything scientific/ML anyway) and you have to buy specific (overpriced) models from them to be compliant with the license. So zero negotiating power, and everything is proprietary so you cant build any software/support edge.

You cant oversell a GPU like you can CPU/memory, and most of the computation happens on the chip so no one cares how good the rest of your stack is.

And finally you are in a neverending battle with crypto miners, which would never be profitable on cloud so they have no intention of paying you.

Maybe there was something strategic about Dall-E, used for some sort of dataset collection/tuning for GPT4.

Like how Whisper unlocks a lot more text data for GPT4.

The founder gave an interesting interview about this.

In short when they did user testing with a Web UI, people would write a basic prompt "dog", "cat" and then get stuck.

Discord makes it social by default, you can see what people are prompting and get inspiration.

I also found it a bit weird, but now that ive onboarded its great having a multi platform native experience (ie even chatgpt still has no mobile apps).

I meant more broadly the train crashes in the US, although the light rail specifically has also had problems

https://www.seattletimes.com/seattle-news/transportation/sou...

This sort of stuff combined with the train situation really does feel like the start of an Ayn Rand novel.

I will say this is a remarkably small overspend for this sort of project though "Originally earmarked to cost no more than $968 million, the Port authority was forced to pump an additional $18 million into the project in 2021, but now there are fears an additional $78 million will need to be found to address the gate layout design."

Normally these projects at least double.

Hilarious, that last graph really shouldve been in this puff piece article.

I don't see a lot of traditional "HPC engineers" in the ml infrastructure space though.

I do wonder if in hindsight using a Slurm cluster for scheduling, Lustre for data, MPI for any connectivity thats not covered by NCCL, would have been better than trying to make object storage, grpc, kubernetes, ray etc work.

LLM training (at least the ones we have research details on) sync weights every step so they have very high networking and latency needs. That’s why every cloud vendor competes on interconnect speeds for GPU machines.

So they are basically classic supercomputing workloads.

It’s going to be so much easier to push these models out to the edge if they are open source (see ggml, diffusion bee etc).

This won’t be possible without accelerator compute for training. Open source developers can’t afford this compute.

Apple benefits because they can deploy on iOS devices. Amazon benefits because they can tarball the models and sell it as a managed service.

I don’t think either wants to be in the business of figuring out which researchers is going to use the compute for deep fakes or for the next big model.

So something like Stability should exist, Amazon and Apple should figure out how to make that happen.

There is a minimum cluster size to get good utilization of the GPU’s. $1 an hour per chip might get you one A100 but it won’t get you hundreds clustered together.

Is it weird that this wasn’t in a TPM somewhere? I don’t know how host keys work, are they rotated based on some root cert regularly or is it just the same keypair since the creation of GitHub?

For whisper whisper.cpp is probably a better bet, it also doesn’t use the NPU but it performs much better than the pytorch implementation.

Nvidia's business model encourages this for starters. They charge a huge markup for their datacenter GPU's through some clever licensing restrictions. So it is cheaper per FLOP to run inference on a personal device.

Centralization of compute has not always won (even if that compute is mostly controlled by a single company). The failure of cloud gaming vs consoles, and the success of Apple (which is very centralized but pushes a lot of ML compute out to the edge) for example.

No, they banked there because SVB would throw money at VC's and founders (ski trips, free meals, lines of credit) who in turn would recommend/force their companies to use the bank. For all the talk of being "friendly" and "relationships" thats basically it.

The average american now has to cover the loss, either through eventual taxes, or through passthrough costs from their bank (who will not give them a free meal or ski trip).

so silvergate assets exceeded/matched its deposits? I know it was a pretty niche bank just curious as I thought it was in a way worse position.

Doesnt this just push the can down the road? Interest rates will likely be higher not lower in 1 year so their bond portfolio will be even more underwater.

Curious there was nothing about silvergate?

SVB saw more than 25 percent of their deposits withdrawn. No bank could handle that. Ultimately the money has to go somewhere (it won’t fit under everyone’s mattress) so a big bank would be unlikely to fail.

Companies have to guess what is PII and what is not, the EU have no idea (other than they know which companies they want to punish)

And the 2 “closed” in seattle have been boarded up for some time, they are in an area that became even higher crime during Covid and never really recovered.

There is a large administrative load for all healthcare which seems to serve no one except the hospitals (to avoid paying any local taxes their non-profit status does not get them out of), and the insurers (who justify their margins by negotiating through the bureaucracy).

The three companies all self fund their employees insurance, so the goal was to cut out some of this bloat by providing services directly to employees.

The Amazon Care implementation was amazing, it's unclear why it failed. I would speculate its because costs are dominated by services they didn't provide (emergency rooms etc).

Does openai actually specify the size of the model?

InstructGPT 2B outperformed gpt 3 175B, and chatgpt has a huge corpus of distilled prompt -> response data now.

I’m assuming most of these requests are being served from a much smaller model to justify the price.

OpenAI is fundamentally about training larger models, I doubt they want to be in the business of selling A100 capacity at cost when it could be used for training

This has been downvoted but I have to say, living atlas seems like an awesome resource

Google Books does seem to still be digitizing books (albeit with the various copyright lawsuits they arent displaying many results) [1]

So they would be in a unique situation to build this, as they have 40 million books many of which no one else would be able to scan.

[1] https://www.blog.google/products/search/15-years-google-book...

It went viral in Montana and was not shot down until it left the country in North Carolina.

If this was a planned media push it would have been shot down before it entered the united states.

I feel like this is getting really close. GPT is create at writing sql queries from text and turning a blob of semi structured data into an sql schema.

We just need to somehow tie it together so anyone can explain their use case, and show an example of the data in plain english, then lock in a schema and feed everything in.

I do a daily walk for ~30 minutes.

For me the key to enjoying exercise was losing weight. When you are overfat you tend to have quite strong legs from carrying the excess weight around. Even with minimal strength training losing fat will leave you quite capable of doing long walks/hikes etc.

Cycling is also great, but its easier to make weather/time excuses about it.

Linkedin and Salesforce PM's tend to be the stereotypical answer (products that haven't changed for years).