pavelstoev

Very nice. Can we chat briefly ?

Often you are better off running certain workloads on lesser GPUs. But this requires certain tricky compiler-level optimizations. For example, can run certain LLM inference with comparable latency on cheaper A40s vs running on A100s. Could also run on 3090s (sometimes even faster). This helps with operating costs but may also resolve availability constraints.

please consider CoreWeave - great experience so far.

May also suggest the suite of open source DeepView tools and which are part of PyPi. Profile and predict your specific model training performance on a variety of GPUs. I wrote a linked in post with usage GIF here: https://www.linkedin.com/posts/activity-7057419660312371200-...

And links to PyPi:

=> https://pypi.org/project/deepview-profile/ => https://pypi.org/project/deepview-predict/

And you can actually do it in browser for several foundational models (more to come): => https://centml.ai/calculator/

Note: I have personal interests in this startup.

AI/ML is a disruptive tech with huge financial benefits. History has shown us that government regulation of disruptive technologies can often have unintended consequences and push those technologies into the shadows, where they are harder to monitor and control. For example, during the manufacturing revolution in the 19th century, many governments attempted to regulate the new factories and their working conditions, but these regulations often resulted in factories moving to countries with fewer regulations and lower costs.

Similarly, during the Prohibition era in the United States, the ban on alcohol only fueled a thriving dark market and increased criminal activity. In the case of AI, any government regulation could limit the positive financial benefits of AI technology, so there will be actors who will take advantage of that. Furthermore, regulation is unlikely to prevent malicious actors from using AI in harmful ways. Regulation could drive the development and use of AI underground, making it even harder to monitor and control. As we have seen with other emerging technologies, such as biological cloning, government regulation often lags behind the technology itself, and by the time regulations are in place, the technology has already advanced beyond their reach. The same is likely to be true for AI.

Instead of relying on government regulation, the development and use of AI should be guided by ethical principles and best practices established by the AI industry itself. This approach has been successful in other industries, such as engineering, architecture, finance and medicine, and can help ensure that AI is developed and used responsibly while still allowing for innovation and progress.

"No man's life, liberty, or property are safe while the legislature is in session." - Mark Twain

Training and inference on GPUs significantly underutilize …the GPUs. So tuning and various tricks need to be applied to achieve dramatic performance gains. If I am not good at cooking, giving me a larger kitchen will not make me faster or better.

I have been using The quick brown fox jumps over the lazy dog

Or you can apply GPU optimizations for such ML workloads. By optimizing the way these models run on GPUs, significantly improve efficiency and slash costs by a factor of 10 or even more. These techniques include kernel fusion, memory access optimization, and efficient use of GPU resources, which can lead to substantial improvements in both training and inference speed. This allows AI models to run on more affordable hardware and still deliver exceptional performance. For example, LLMs running on A100 can also run on 3090s with no change in accuracy and comparable inference latency.

Generally, Hidet outperforms other inference compilers - PyTorch Eager, ORT, TRT, TVM. For example, PyTorch Eager - too much framework overhead. ORT -doesn't do operator fusion. TRT - close-sourced and hard to fix if a model can not run. TVM - tuning time is too long, also limited expressiveness in optimization.

Additionally Hidet comes with Hidet Script, a brand new domain-specific language to write tensor programs in Python with high flexibility to express optimizations that can only be done in C++ CUDA C code. Hidet Script also supports operator tuning and automatic fusion.

Generally, Hidet outperforms other inference compilers - PyTorch Eager, ORT, TRT, TVM. For example, PyTorch Eager - too much framework overhead. ORT -doesn't do operator fusion. TRT - close-sourced and hard to fix if a model can not run. TVM - tuning time is too long, also limited expressiveness in optimization.

Additionally this comes with Hidet Script, a brand new domain-specific language to write tensor programs in Python with high flexibility to express optimizations that can only be done in C++ CUDA C code. Hidet Script also supports operator tuning and automatic fusion.

How much they pay for GPU use ?

DeepView.Predict - FOSS tool developed by CentML that predicts a deep neural network's training iteration execution time on a variety of GPUs. Know your model training costs before running it !

model-depending, you can train on lesser (cheaper) GPUs but system-level optimizations are needed. Which is what we provide at centml.ai

Training a ChatGPT-beating model for much less than $85,000is entirely feasible. At CentML, we're actively working on model training and inference optimization without affecting accuracy, which can help reduce costs and make such ambitious projects realistic. By maximizing (>90%) GPU and platform hardware utilization, we aim to bring down the expenses associated with large-scale models, making them more accessible for various applications. Additionally, our solutions also have a positive environmental impact, addressing the excess CO2 concerns. If you're interested in learning more about how we are doing it, please reach out via our website: https://centml.ai

Hi Bea and thank you for sharing your creation. Looking good ! Couple of questions:

1) could you please describe your data privacy considerations. Like what happens to my documents after they are uploaded ? Are they stored somewhere (encrypted or not) or deleted ?

2) could you please share more details on how this works “under the hood”. Specifically how do you ingest and digest the knowledge contained in my documents ?

Thanks !

We have these in Florida, USA and I have seen these types in Ontario, Canada where I am from originally.

As the world marvels at the astonishing capabilities of OpenAI's GPT-4, I find myself contemplating the rapid acceleration of AI and machine learning, and the evolutionary impact it is having on our lives. Naturally, I turned to GPT-4 to assist me in these thoughts.

GPT-4's human-level performance on professional and academic benchmarks - such as the 88th percentile on the LSAT and the 89th on SAT Math - is a testament to the leaps we've made in artificial intelligence. Yet, these achievements also raise pressing questions about our future. Just as Homo Sapiens once outperformed and eventually displaced their Neanderthal cousins, could a new breed of humans - enhanced with GPT-X-like capabilities - arise to dominate those who remain unequipped with such powers?

What will it mean for our species, our societies, and our collective story when the lines between natural intelligence and intelligence assisted by AI/ML become ever more blurred?

As we ponder the remarkable rise of GPT-4 and the future of humanity, let us consider not only the implications of this technology but also our roles in shaping its trajectory. We are already over the cusp of this new chapter in the story of humankind, will we become merely a footnote in the annals of our own creation?

At CentML, we profiled GPU utilization on a larger AI/ML research institute cluster. 10% to 45% range, mostly in 10% utilization range. We then offered them software optimizers (which do not affect model accuracy) to get to the 90% utilization for GPUs

wait till you visit eastern europe where drivers believed strings or single CD handing on the rear-view mirror somehow makes the car stealth to a police radar.

It does not have to be. We have optimizations for all kinds of workloads - https://CentML.ai

And we can make Ray more efficient by optimizing GPU hardware utilization https://centml.ai/

Before spending $$$ on the hardware accelerators, there is still a lot to achieve using software optimizations for inference and trainings workloads. We optimized GPT-J, and others, for real. https://centml.ai/benchmarks/

We do GPU-specific training and inference speedups, at CentML.

It is a loosing uphill battle to reduce the operational costs as deep learning models get larger and more complex. Nvidia CEO says hardware alone will not be able to address the growing compute demand. The solution is computational optimization which is what we do.

We have done the same and published back in early 2020. THis paper does reference our work as [11]. Blog post with our human detection video here (eerie !):

https://futurebnd.com/human-detection-classification-and-vis...

and the paper here:

https://arxiv.org/pdf/2001.05842.pdf