visionscaper

Author of collabmem here! Let me add some detail on how the memory stays manageable as it grows.

Two mechanisms keep it sustainable, neither deletes anything, and both are discussed with you before being applied:

- Upward consolidation — when the episodic index grows large, mature stable knowledge from old episodes is extracted into the world model. Consolidated index entries move to a searchable archive; the original notes stay put. The active index keeps focused on recent work while the world model absorbs what's been learned.

- Downward compaction — when a world model file approaches its size cap, it's rewritten to stay compact. Removed knowledge is preserved in an episodic note so it remains iscoverable.

Caveat: these two mechanisms are designed but not yet tested; this is one of my high-priority todos. Feedback especially welcome here.

Happy to answer questions — looking forward to feedback!

Isn’t to core of MacOS derived from Unix?

> Instead of modifying spacetime, the theory – dubbed a “postquantum theory of classical gravity” – modifies quantum theory and predicts an intrinsic breakdown in predictability that is mediated by spacetime itself. This results in random and violent fluctuations in spacetime that are larger than envisaged under quantum theory, rendering the apparent weight of objects unpredictable if measured precisely enough.

I have so many questions about this paragraph; how did they change quantum theory? Are they saying that breakdown in predictability at the quantum level is actually caused by space-time? Why does this result in violent fluctuations in space-time? And why are the space-time fluctuations larger than in the case of quantum gravity?

Related question, does anyone have experience with using the AMD MI100 for deep learning? With 32GB and a second hand price of ~1100 USD, it could be a good choice.

I’m wondering about the performance of such a cluster of repurposed mining GPUs. Mining doesn’t need high bandwidth interconnects nor large per GPU memory, on the other hand training deep learning models such as LLMs does need that.

I’m not saying it’s impossible to get good performance with custom training solutions; that would actually be very valuable as it would unlock the computing power of lower grade setups.

Just read your comment, I made a similar comment here: https://news.ycombinator.com/item?id=36543226

Like others mentioned in this thread, I don't think this is Apple's final vision for a Apple Silicon Mac Pro. To me, the only way forward is for Apple to, either use PCIe, or another proprietary high bandwidth bus solution, to connect together multiple Apple Silicon boards. This would result in a Mac Pro in which you can slide in multiple Apple Silicon modules, i.e. Mx Ultra modules.

For distributed Machine Learning workloads it also make sense to combine a CPU and a GPU per pluggable module. In such workloads, data that needs to be processed by a specific GPU (i.e. for training), is usually preprocessed by a CPU. In Intel machines with multiple Nvidia cards, this implies that you need a single beefy CPU with high core count. Having a CPU and GPU in each module makes sense from this perspective.

I'm wondering to what extend these benchmarks, especially the GPU benchmarks, are really optimised for the specific Apple M2 architecture? And to what extent the high GPU-GPU bandwidth is benchmarked, this also influences real-world performance.

While this answer spooks me, the LLM is literally following your brief; it is explicitly unethical and immoral, just like you asked.

This is the result of a system without any value judgment or morals, that’s the scary part. If these items are from existing lists it picked lists from authoritarian and totalitarian playbooks.

I’m a bit skeptical. ChatGPT can still hallucinate, generating information that seems correct, but is in fact nonsense. I’m wondering how they are going to deal with that.

Only tangentially related; why is Teams removing space indentations in chats on MacOS? It’s so annoying when you type a piece of (pseudo) code, all leading spaces are removed! It drives me nuts, why would anyone implement this!

I’m not using an M1 machine yet, but I think is extremely useful to be able to do some small scale experiments for testing and debugging purposes locally before starting a full-scale training job in the cloud.

Further, in general, I’m really charmed by the potential of having unified memory; the idea you can test some batch training iterations with a batch size that fills almost all 128MB is a unique capability!

Last, a large amount of unified memory allows to do inference and “prompt engineering” with very large models, locally. E.g. Using GPT-J like models (6B parameters).

I don’t know much about the black hole information loss problem and I do like Sabine’s skepticism. However, saying it has no use to work on mathematical solutions in this area is taking it one step too far. Maybe, based on the solutions other (indirect) experiments can be devised, maybe it will help solve other Physics, mathematics or engineering problems in the future.

I’m wondering what this all means for the upcoming Mac Pro.

Apple mentioned that the M1 Ultra is the last member of the M1 family. So how is the Mac Pro going to scale?

Will Apple enable “traditional” scaling by allowing multiple M1 Ultra chips to be combined in one system? Or what?

Further, how will an Apple Silicon based Mac Pro be made expandable; something that has been a corner stone feature of Mac Pros in the past?

Interesting, thanks for sharing this.

Apparently they are going to use “HIP” to convert CUDA applications to be able to run on AMD:

> The OLCF plans to make HIP available on Summit so that users can begin using it prior to its availability on Frontier. HIP is a C++ runtime API that allows developers to write portable code to run on AMD and NVIDIA GPUs. It is essentially a wrapper that uses the underlying CUDA or ROCm platform that is installed on a system. The API is very similar to CUDA so transitioning existing codes from CUDA to HIP should be fairly straightforward in most cases. In addition, HIP provides porting tools which can be used to help port CUDA codes to the HIP layer, with no loss of performance as compared to the original CUDA application. HIP is not intended to be a drop-in replacement for CUDA, and developers should expect to do some manual coding and performance tuning work to complete the port.

Would this one day lead to a HIP based PyTorch? I hope so!

What software will they use to, for instance, train large deep learning models? Nvidia has CUDA, AMD has what? Are they writing new software from scratch? Maybe they have a lot of frameworks to solve problems in the “traditional” HPC space (eg weather forecasts), but in the ML space I only heard of ROCm which seems to be poorly supported.

AMD seems such an odd choice for “AI supercomputers”.

I don't know too much about it myself, but you could checkout https://rocm.github.io/dl.html

Edit : directly linked to the ROCm deep learning page.

I decided a few weeks ago to transition to PyTorch (was using Keras before) and I must say that I really love it! How PyTorch is structured gives me the right balance between ease of use and the ability to make customisations. Further, using DistributedDataParallel, dividing the work over multiple processes, where each process uses one GPU, is very fast and GPU memory efficient.

Before my switch I tried out Keras for Tensorflow, and even got a lot of support from Google in my endeavours to resolve the issues I encountered (kudos to Google for that!). In the end I felt it was still not mature enough. Further, although I do believe TF and Keras are moving in the right direction, I still felt that in some cases the way the software was set up just didn't sit well with me.

Maybe it is worth to try again in a year or so, or by then I will tryout Swift for Tensorflow, which I think has a great future ahead.

> Configure two MPX Modules with up to four GPUs

That's total, right? Not 4 GPUs per MPX module?

Why is my comment downvoted? If you have any beef with it, please reply.

The 2019 Mac Pro is what you get when you let "Pro's", the users of the intended product, influence the vision of the product. Something I don't think has Apple has done ever, and rightly so. Users aren't product designers, they should not be involved in the vision of the product; they must be involved in getting the details right for them.

I would have liked to see a much more innovative product where not only the internals were replaceable and extendable, but that also allowed users to string multiple Mac Pro's together, making it's compute and memory almost infinitely expandable. Maybe even in some sort of lattice network form.

In this way a single Mac Pro could be cheaper and appeal to a broader range of Pro users (e.g. 4000 USD with 2 high-end GPUs). If you'd really wanted computational power, like the hollywood studios need to have, you could string three or four Mac Pro's together. If you are a heavy machine learning Pro, maybe you would put 16 Mac Pro's in a lattice setup, who knows.

I would have expected that the Mac Pro would have some sort of high-bandwidth interface, almost directly exposing the PCI bus, making it possible for software to see all resources (GPUs, CPUs, memory) of all connected Macs as a single machine. Apple, with their software and hardware design capabilities could have pulled that off.

[edit] PS: The Mac mini is not a good replacement for the suggestion I made to string multiple Mac Pro's together, because it lacks to possibility to install one or more powerful GPUs.

Is the used dataset with conversations available somewhere? What about the model architecture?

From experience I know that models using RNNs have trouble training with FP16 precision. The common solution is to do training in FP32 and inference in FP16. To make this happen you often have to implement custom code (e.g. using Tensorflow or Keras as a meta framework)

Although the RTX 2080 Ti performs significantly better than the 1080 Ti, I'm still drawn towards the 1080 Ti; I can buy two second-hand 1080 Ti's for the price of one new 2080 Ti, providing me the double amount of memory, plus, the computing performance of 2x 1080 Ti is much better in FP32 than one 2080 Ti.

I'm using my GPUs to train large sequence to sequence models (with long sequences) that need FP32 for training and can use FP16 for inference (mixed-precision training), so I can't even use the FP16 performance of the Tensorcores for training.

The only disadvantage is that the energy costs are higher using two 1080 Ti's compared to one 2080 Ti.

I got this response from Nvidia customer care concerning these questions:

"Hello Freddy,

Thus far we do not have any information from PR / Marketing on GeForce GTX 20 series Tensor core nor float16 compute performance.

If or when such information is available from them we would be glad to discuss it.

Best Regards, Troy NVIDIA Customer Care L2"

> It may well be that NVidia will make them available for inference via DirectX so they can do AI based denoising and anti aliasing in games but artificially limit support in frameworks for training so as not to eat into the Quadro / Tesla markets. I'm holding off on ordering one until I see more info on that.

Good point, that would not be cool. I'll wait with ordering until more is known about this.

Is anything known about the number of tensor cores in the new 2080 Ti? What about the float16 performance using the Tensor cores? I couldn't find this on the spec page [1].

[1] https://www.nvidia.com/en-us/geforce/graphics-cards/rtx-2080...

There had been a lot of research on this topic (just Google it), with good results.

Thanks, I also read the article better now. I updated my comment.