Hacker News Reader

Comment by simonw | Hacker News Reader

This is a fantastic notebook, thanks very much for sharing.

"If you want to run the full notebook on a full size model, expect training the tokenizer to take ~15 hours, pretraining with the MLM objective to take ~17 hours (on a 3070 RTX, adjust expectations for your own system), and finetuning to take about an hour"

I wonder how hard it would be to modify this code to run on a 64GB M2 Mac.

It's frustrating how much potential that platform has for this kind of thing (given the way the GPU shares memory with the CPU) that isn't yet harnessed because most of the ecosystem is built around NVIDIA and CUDA.

View on HN

> I wonder how hard it would be to modify this code to run on a 64GB M2 Mac.

It isn't that hard, I was able to run in on M1. The changes are:

remove or modify multiprocessing - it doesn't work on Mac the same way as in the code;

replace `device = "cuda"` with `device = "mps"`

In this line ` att_idxs = (torch.clamp(torch.arange(context_size)[None, :] - torch.arange(context_size)[:, None], -pos_emb_radius, pos_emb_radius-1) % pos_emb_size).to("cuda")` replace cuda with "mps"

in `optim.AdamW` remove `fused=True` - we can't do it without CUDA

Replace ```with autocast(device_type='cuda', dtype=torch.float16): _, loss = mlm_head(bert(batch_data_torch_xs[mb_start_idx:mb_end_idx]), batch_data_torch_ys[mb_start_idx:mb_end_idx]) ```

with simply `_, loss = mlm_head(bert(batch_data_torch_xs[mb_start_idx:mb_end_idx]), batch_data_torch_ys[mb_start_idx:mb_end_idx])`

replace `scaler.scale(corrected_loss).backward()` with `corrected_loss.backward()`

replace ``` scaler.unscale_(optimizer) scaler.step(optimizer) scaler.update() ``` with `optimizer.step()`

It should work.

> It's frustrating how much potential that platform has for this kind of thing (given the way the GPU shares memory with the CPU) that isn't yet harnessed because most of the ecosystem is built around NVIDIA and CUDA.

I'm sure it's frustrating from a consumer perspective, but it should be no surprise why Nvidia won here. CUDA shipped unified memory addressing ten years before the M1 hit shelves. On top of that, their architecture and OS support is top-notch, you can ship your CUDA code on anything from a $250 Jetson to a $300,000 DGX system, and their hardware is relatively ubiquitous.

The frustrating thing is how companies like Apple and Nvidia insist on being each other's enemies. Only consumers feel the pain when researchers discover cool stuff like this and want to share.

You can add an Nvidia card to basically every kind of hardware, desktop, server or most importantly rent in the cloud. You must buy a Mac to use a M1/M2. Given the cost of some cards it could make sense but then everybody using that software will have to buy a Mac too.

A lot of people have M1s or M2s already, who who would have to pay for access to an NVidia card. I think that's the disconnect. It's more about making use of what a lot of people (and let's not forget a lot of developers) already have.

I've personally got an 8GB M1 Macbook as my work development machine, and while I'm having a lot of fun with llama.cpp it does feel somewhat disconnected from the bulk of the ML ecosystem.

Reply on news.ycombinator.com