Grid: AI platform from the makers of PyTorch Lightning | Hacker News Reader

Grid: AI platform from the makers of PyTorch Lightning

grid.ai

107 pointsshcheklein61 comments

I dislike that pytorch advertises TPU support. Pytorch doesn’t support TPUs. Pytorch supports a gimped version of TPUs that have no access to the TPU CPU, a massive 300GB memory store that handles infeed. No infeed means you have to feed the TPUs manually, on demand, like a gpu. And TPUs are not GPUs. When you try to do that, you’re talking at least a 40x slowdown, no exaggeration. The TPU CPU is the heart of the TPU’s power and advantage over GPUs, and neither pytorch nor Jax support it at all yet. No MLPerf benchmark will ever use pytorch in its current form on TPUs.

Luckily, that form is changing. There are interesting plans. But they are still just plans.

It’s better to go the other direction, I think. I ported pytorch to tensorflow: https://twitter.com/theshawwn/status/1311925180126511104?s=2...

Pytorch is mostly just an api. And that api is mostly python. When people say they “like pytorch”, they’re expressing a preference for how to organize ML code, not for the set of operations available to you when you use pytorch.

We currently have google engineers training on TPU pods with PyTorch Lightning.

TPU support is VERY real... but yes, sometimes it breaks but PyTorch and Google are working very hard to bridge that gap.

But we have dedicated partners at Google on the TPU team working to get Lightning working seamlessly on pods.

Check out the discussions here: https://github.com/PyTorchLightning/pytorch-lightning/issues...

No, you do not support the TPU infeed, and this is a crucial distinction. Saying that you do support this has caused endless confusion and much surprise. It’s almost not an exaggeration to say that you’re lying (sorry for phrasing this so bluntly, but I’ve seriously spent dozens of hours trying to break this misconception due to hype like this).

TPU support is real. Pytorch does in fact run on TPUs. But you don’t support TPU CPU memory, the staging area that you’re supposed to fill with training data. That staging area is why a TPU v3-512 pod can train an imagenet resnet classifier in 3 minutes at around 1M examples per second.

You will not get anywhere near that performance with pytorch on TPUs. In fact, you’re expected to create a separate VM for every 8 TPU cores. The VMs are in charge of feeding the cores. That’s insane; I’ve driven TPU pods from a single n1-standard-2 using tensorflow.

Repeat after me: if you are required to create more than one VM, you do not (yet!) support TPU pods. I wish I could triple underline this and put it in bold. People need to understand the limitations of this technique. Creating 256 VMs to feed a v3-2048 is not sustainable.

There's a difference between "supporting TPUs" and "supporting TPUs at 100% potential". Although the distinction is important, I don't think the marketing here is misleading.

Not only is it misleading, it even somehow tricked you. :)

We’re not talking about a small 10% reduction in performance here. We’re talking like 40x differences.

If it seems unbelievable, and like it can’t possibly be true, well: now you understand my frustration here, and why I’m trying to break the myth.

Notice not a single benchmark has ever gone head to head in MLPerf using pytorch on TPUs. And that’s because using pytorch on TPUs requires you to feed each image manually to the TPU on demand, from your VM. Meaning the TPU is always infeed bound.

Engineers should be wincing at the sound of that. Especially anyone with graphics experience. Being infeed bound means you have lots of horsepower sitting around doing nothing. And that’s exactly the situation you’ll end up in with this technique.

There’s a way to settle this decisively: train a resnet classifier on imagenet, as quickly as possible. If you get anywhere near the MLPerf v0.6 benchmarks for tensorflow on TPUs, I will instantly pivot the other direction and sing the praises of pytorch on TPUs far and wide.

Like I said... pytorch and tensorflow team are working very hard to make this work. And yes, it's not a 1:1 with tensorflow, but we're making progress very aggressively.

I love what you guys are doing, and I love improving the ML ecosystem, but you’ve godda understand, people see this and think “oh, ok, it’s a small difference, no big deal.” In fact it’s a huge difference.

Picture a person with one arm and without legs. Would you say they aren’t “1:1 in terms of features”? They certainly won’t be winning any races.

And unlike real people, you can’t graft on a prosthetic limb to help this situation. The issue I’m describing here is a fundamental one that everyone keeps trying to sweep under the rug and pretend isn’t an issue. And then everyone wonders what’s going on.

I 100% agree. We don't want to misrepresent TPU support. In fact, we explicitly warn users in our docs. Open to suggestions about how we can communicate this much better to our users.

We just need to be a part of the effort to help bridge the big gap and barriers keeping users from TPU adoption.

https://pytorch-lightning.readthedocs.io/en/latest/tpu.html#...

> In fact, we explicitly warn users in our docs

This was mentioned above, but nowhere on that page does it talk about any limitations whatsoever.

I want to signal-boost this. TPU support on PyTorch is partial. You can run modeling computation on the TPU with PyTorch, but not the data-loading. And without the TPU's data-loading, you're significantly, significantly bottle-necked, to the point where you are often better off using GPUs. The reason why TensorFlow and TPUs synergize so well is that the TPUs themselves can consume data for training, allowing for massive scalability.

I have great respect for the PyTorch-TPU team, but I would recommend not heavily advertising PyTorch-TPU support until this major feature disparity is made up.

we highlight these issues in our docs explicitly.

https://pytorch-lightning.readthedocs.io/en/latest/tpu.html#...

Can you point out where exactly in those docs you highlight the issue?

I just read the linked page and found no references to data loading limitations or performance limitations. Is it only in the video which isn't search indexed and few people would bother watching?

edit: The page literally advertises the speed of TPUs with "In general, a single TPU is about as fast as 5 V100 GPUs!" which is the exact opposite of warning people.

> When people say they “like pytorch”, they’re expressing a preference for how to organize ML code

Maybe that's the way you feel, but for me that's very different. Pytorch is much more than just an API (which is also nothing to scoff at).

It's also much cleaner documentation, a very different ecosystem of libraries (mostly better, but sometimes lacking depending on the niche), less magic (which makes it easier to debug). It also has the benefit of less ecosystem churn, while the transitions of TF1->2 as well as the external Keras->internal Keras are a shitshow that's almost as bad as Python2->3.

What niche libraries do you think PyTorch is lacking? Do you have some examples of ones that exist in Tensorflow with no PyTorch equivalent?

I meant model libraries/implementations with that like e.g. Stardist[0]. Obviously there will be some research niches that are mostly implemented in TF over PyTorch or the other way around.

[0]: https://github.com/mpicbg-csbd/stardist

We both know (and correct me if I'm wrong) that this is an issue with the GCP cloud architecture. https://github.com/pytorch/xla/issues/1858

So you can't blame the PyTorch team. If there's anyone to blame, it's Google Cloud. In the meantime, I don't think there's any harm advertising PyTorch with TPU support if running on TPUs with PyTorch is often much faster than running on GPUs with PyTorch.

In the above-linked GitHub issue, the Google TPU team is now giving an ETA of early 2021. At that point PyTorch TPU training (including on TPU pods) should be equivalent to TF TPU training. But I think my point still stands that as long as PyTorch TPU training is faster than GPU training, even in its current state, there's nothing wrong advertising TPU support now.

I am extremely pessimistic for ML ops startup like this. At the end of the day, cloud service providers have too much of an incentive to provide these tools for free as a cloud value add.

The other thing is that stitching together other open source tools like this is simply not enough value. Who will be incentivised to buy?

Saying this as FAANG ML org person where I see the push to open source ops tooling like this.

Sounds like a great reason to acquire a startup like this for $XX million. Might be a great outcome if the team is lean with minimal investment.

Well that ship seems to have sailed with a ~$19mil investment and a 20 person (+$4mil/year burn in NYC?) team.

On paper that's the case.

Google certainly has made a push toward scalable AI training and deployment. However, it is not fun to use in practice, speaking from experience.

Startups beating an incumbent with substantially better UX is always a good story. Improving productivity is an easy winner for potential customers.

Well... we're not really an ML ops startup haha. I am ALSO pessimistic about ML Ops startups.

But calling Grid an ML Ops startup is like calling Lightning Keras... maybe at a quick blink it looks like that, but that's where the similarities end.

For what it's worth, a lot of what we're building comes from my experience at FAIR.

If it is not ML Ops, how would you describe it?

it's honestly just a different approach. ML ops is adjusting your code to work with the cloud and managing all that.

For us is basically integrating clouds directly into your code so the barrier disappears and the cloud providers become an extension of your laptop.

Might not be your target audience, but high energy physics has been operating an infrastructure like that for years: https://www.etp.physik.uni-muenchen.de/research/grid-computi... You can basically use these frameworks to run your analysis jobs on any of the connected HPC centers and interactively move workloads around. The data never has to touch your hard drive either but gets moved to the compute on demand. This is how thousands of physicists do statistical analysis on petabytes of data.

super cool!

One of the professors at my lab at NYU CILVR (Kyle Cranmer) i believed was super involved with this. Will definitely sync up with him!

Thanks for the heads up!

Congratulations to the Grid team on the fundraise and the announcement! Exciting stuff.

It seems like there is an emerging consensus that (a) DL development requires access to massive compute, but (b) if you’re only using off-the-shelf PyTorch or TensorFlow, moving your model from your personal development environment to a cluster or cloud setting is too difficult — it is easy to spend most of your time managing infrastructure rather than developing models. At Determined AI, we’ve spent the last few years building an open source DL training platform that tries to make that process a lot simpler (https://github.com/determined-ai/determined), but I think it's fair to say that this is still very much an open space and an important problem. Curious to take a look at Grid AI and see how it compares to other tools in the space -- some other alternatives include Kubeflow, Polyaxon, and Spell AI.

maybe determined should come up with its own API instead of copying Lightning's :)

Not a nice move for the opensource spirit. Also, pretty sure it's a violation of our patent and 100% copyright infringement.

Hi William -- we have absolutely not copied any of Lightning's APIs.

In fact, our PyTorch API makes some significantly different design choices than Lightning does -- e.g., we require users to step optimizers and run the backward pass explicitly, which is a bit lower-level but allows for more flexibility when using the API.

For instance, here is an example of a GAN using our PyTorch API: https://github.com/determined-ai/determined/blob/master/exam...

This is a port of this PyTorch Lightning example: https://github.com/PyTorchLightning/pytorch-lightning/blob/m...

Despite the former being a port of the latter, there are significant differences between the two APIs.

More broadly, we welcome competition in this space and think there's a lot that we can all learn from one another.

Projects copying from each other is exactly in the open source spirit. Don't release something with an Apache License if you don't want people copying it with compatible licenses. Also, patents of open source software (to limit uses that the license would otherwise allow) and API copyrights are pretty strongly frowned upon in the open source community. As a note, the Apache license you released Lightning under grants patent lawsuit protection to anyone using your code under the license so claiming copyright and patent infringement on another Apache licensed project seems amazingly skeevy.

If this is the philosophical stance that Grid and Lightning are taking then it's definitely a project I'm going to advice people to stay well clear off. It's the worst flavor of commercialized open source software and potentially a legal liability to touch in any way as you seem way too lawsuit trigger happy.

No! we want people to build on Lightning and we want companies to deliver value and products for their users.

The Apache License you released Lightning under explicitly allows people to clone your framework if they adhere to the license and provides them patent protection if they do so.

Just to be 100% clear.

We want people to build on Lightning and we want companies to deliver value and products for their users.

We place no limitations on how Lightning is use or what products people will build.

Our in-process patent portfolio is not to limit the use of lightning in any way but for defensibility purposes only.

If you have any other inquiries please email legal@pytorchlightning.ai for more details.

No! we want people to build on Lightning and we want companies to deliver value and products for their users.

The Apache License you released Lightning under explicitly allows people to clone your framework if they adhere to the license and provides them patent protection if they do so.

As I said in my previous comment, using patents to try and get around an open source license is skeevy as hell.

Just to be 100% clear.

We want people to build on Lightning and we want companies to deliver value and products for their users.

We place no limitations on how Lightning is use or what products people will build.

Our in-process patent portfolio is not to limit the use of lightning in any way but for defensibility purposes only.

If you have any other inquiries please email legal@pytorchlightning.ai for more details.

So this is the endgame of pytorch-lightning, which was always a mystery to me. (if you haven't used it, it's strongly recommended if you use PyTorch: https://github.com/PyTorchLightning/pytorch-lightning )

IMO, open source is at its best when it's supported by a SaaS as it provides a strong incentive to keep the project up-to-date, and the devs of PL have been very proactive.

yes! our goal is to completely remove any engineering from the AI research -> production lifecycle.

Not just a marginal improvement on that experience but a 10x completely different approach.

Dead end. Train researchers to be engineers. Not the other way around.

I wish you luck though. We’ll see in ten years whether programmers are as effective as researchers, or whether researchers are as effective as programmers. In 60 some years of computing, no one has achieved the latter, despite many attempts.

Also, I was surprised that this webpage is basically a waitlist and nothing else. No discussion of technique, no docs, no substance. Just a “you like pytorch? Pytorch rules!” type hype.

I do like pytorch, but I also like knowing one or two substantive points about what the proposal here is. If you want to train a model from your laptop, it’s a matter of applying to TFRC and kicking off a TPU.

The whole ecosystem is in need of massive overhaul. I like the ambition. But I dislike trying to pretend we aren’t programmers. ML is programming, and pretending otherwise will always cause massive, avoidable delays.

I would pick engineers over researchers any day as you can teach an engineer how to do research but the opposite is rarely a case.

It is much harder to do research then it is to do engineering. I do both at work where we have about 15 ML engineers and only 2-3 of them are capable of doing something novel. Maybe you mean something different by “research” but it is very hard to teach someone how to come up with an idea or a novel method publishable in a top conference. Teaching engineering on the other hand usually goes like this: “Here’s how this works. Here are your options, tradeoffs, pitfalls.” After you’ve done it a few times you mastered it.

I forgot who said it: “In engineering, if you don’t know what you’re doing you shouldn’t be doing it. In science, if you know what you’re doing you shouldn’t be doing it.“

>Research is "creative and systematic work undertaken to increase the stock of knowledge"

I meant this. Academia-related BS is just a one way to do that.

Very little of the work done in machine learning is truly novel versus simple extensions of what has been done before. There's probably the same amount of novel work in machine learning and engineering except the ML crowd publishes it while the engineering version stays in closed source code bases. There's also a distinction between doing novel machine learning work and publishing successfully on novel machine learning work.

I agree with you. However very few "ML engineers" do any novel work, let alone "truly novel" work. Typically they just take the existing code and apply it to their specific problems with some tweaking. Sometimes they would try to reproduce papers when no code is available. Very few engineers are capable of the NeurIPS quality research - it does not matter if it get published or stays in a private repo. That's much harder to do and to teach than what 90% of typical ML engineers do. That was my point.

There's also a ton of bad research being published in second rate conferences, but I don't consider that "research". You won't get "simple extensions of what has been done before" published in a top level conference, the acceptance rates have been extremely low recently.

I would not have touched TensorFlow (or AI in general) at all if it weren't for Keras, and I wouldn't be happy with PyTorch if it weren't for PyTorch Lightning.

Easy onboarding for ML tooling is very valuable for the industry as a whole.

not sure where is "you like pytorch" coming from. Grid.ai is not limited to pytorch :)

Lightning is built for researchers by researchers... we've already taken a much different approach.

100% agree with you that going the other way is likely not the best approach.

Lightning + Grid elevates and turns non experts closer to researchers... ie: focus on building the products and doing science and not the engineering.

That's what lightning excels at today. That's the experience Grid will 10x.

As a scientist: The amount I struggled with getting distributed training on a HPC cluster to work vs. how easy it was with Lightning was eye opening. Almost no code change and finally I can run across 20 nodes with 4 V-100 each :). Plus the automatic SLURM checkpoints and restarts <3.

How do you handle the security of training data? If the data is super sensitive how do you deal with it?

I know the same could be said about Azure and AWS, but the big name cloud providers stake their prestige on having tight security, while a startup has much less to lose.

On the opposite, startup has everything to lose. No other business lines

A large company may lose more than the value of the impacted business line while a startup is capped at the value of the business line (or rather the cash on hand which is even less). So the maximum loss is much higher in absolute terms for a large company and so they're able to invest more absolute money into security. Plus, philosophically, startups already have such massive risk that another X% due to a potential security breach isn't even worth bothering with versus spending the money on lowering risk in other areas.

Can't answer to 'Proven' on account of comment depth, but I am not concern trolling. I work as a ML engineer and many of our projects and datasets are sensitive. The company has signed contracts of confidentiality for this data and we are really concerned about where it sits and who gets access to it.

Another major concern is GDPR - can you reliably track and delete data related to a specific person? You got to think about this right from the start to keep the necessary metadata in the tagging pipeline.

The name is unfortunately close to “The Grid”, an AI website builder that had a lot of buzz then scammed a lot of people out of money then disappeared https://medium.com/@seibelj/the-grid-over-promise-under-deli...

The grid is how you harness and distribute power and electricity.... like that coming from lightning :)

Second, electricity was a great new technology (ie: AI), but you needed the power grid to make it usable - that's grid AI.

Also related to TRON

More on this is here https://techcrunch.com/2020/10/08/grid-ai-raises-18-6m-serie...

What do you think folks?

Seems like Pytorch lightening is the only first-class citizen in your offering. Is that true? Or are there value-added features for TensorFlow and other non-DL libraries such as scikit-learn?

Also, is there support for distributed training for large datasets that don't fit into single instance memory? or just distributed grid-search/hyper-parameter optimization?

No, we support other frameworks too!

Just that if you use lightning you'll have zero friction. Well as with the others... you might run into issues inherent in the other framework's hard to work with designs.

Sadly Julia's Flux.jl have fallen behind to the point that I am switching to using pytorch. It's just faster.

Every 3-6 months I try a new project in Flux and it’s definitely catching up quickly! The neural ODE integration is super cool for example. There are also other applications where you might need to calculate higher order gradients or even do forward-mode autodiff, and Flux is first rate there. But I too have had problems porting state of art models in Flux, due to speed and memory issues. The developers are doing an amazing job but only have a fraction of the resources of pytorch or tensorflow. But that will never change unless people adopt and contribute.

i used pytorch lightning back in may when i was working on pretraining gpt2 on TPUs (https://bkkaggle.github.io/blog/nlp-research-part-2/). it was really impressive how stable it was especially given how a lot of features were still being added at a very fast pace.

also, this was probably the first (and maybe still is?) high-level pytorch library that let you train on tpus without a lot of refactoring and bugs which was a really nice thing to be able to do given how the pytorch-xla api was still unstable at that point. <3

Reply on news.ycombinator.com