Two_hands

Hey I have a blog site where I write about implementing different papers and doing deep dives on the papers!

https://ym2132.github.io

I try to go for those things you're looking for. It's hard to find good resources nowadays with real people behind them.

I hope you enjoy the posts. Feel free to reach out about anything on there

> You're not as far behind as you might think.

Thanks!

> you don't even require machine learning code

Interesting, I'll check it out for sure

> ConvNext

I certainly won't its on my papers to read list.

> seek out a mentor

Finding one seems to be the hard part.

> because progress is made in small steps,

This seems easily forgotten by a large number of people. I try to remind myself to step back from the hype and explore the lesser travelled paths.

> I'll give some examples that are easier to read[0-2]

I need to reach ResNet strikes back, it was one the first networks I implemented and it is cool to see it still being worked on.

I'll check out [3]. I've wondered recently how you could get a GAN to generate things out of distribution but that still look like the training data, if that even makes sense.

> the StyleGAN code is not the easiest to read lol

Yup, even the official PGGAN code was quite hard to understand. I'll try out the PyTorch compile I've heard a lot about it recently. I had thought TensorRT was for LLMs I suppose it's applicable in other areas too?

> so recognize this as a hyper-parameter

Okay that makes sense. I'll reread this after exploring Diffusion models too in the future.

> carefully study Goodfellow's original paper

This is something I have not done, my current workflow is just to understand how best to implement what is written. I think deep exploration is the next step, no matter how many "I know nothings" I will experience. This side of GANs I had not considered (the theoretical, it looked interesting but very complex).

> I hope this can help provide direction

It certainly will, I imagine I'll come back to this comment many times. Thanks for taking the time to read my posts and provide so much material for further study.

> if unfortunately hard to gain

I agree it is rewarding and I hope I can purvey some of this knowledge in my blog for others too! That was why I started it, so much knowledge is locked away and hard to access or understand without some guidance.

> you'll likely fall in love with these types of models

Sounds like a fun rabbit hole to fall into.

Thanks for the insights I wasn't aware that GANs are still so prevalent. And I haven't heard of a lot of these methods, I'll check them our for sure.

> the self fulfilling prophesy of considering things dead

This is quite sad, GANs are an amazing piece of tech and it doesn't seem like they are finished yet. The rule in ML is that it's never over for a method, so maybe someone somewhere will get GANs fashionable again. There's many things like this in ML though...

On the FFHQ point, are you saying currently GANs are better at benchmarks like FFHQ where the target is realistic looking images? Or better at representing the training data?

> Karras wrote custom cuda kernels for StyleGAN

I didnt know they wrote custom kernels, perhaps for my StyleGAN post I can try triton and write a custom kernel for the operations. However, I've never looked into this.

What does it mean to have a backbone? Does it just mean the underlying architecture used in the method? Also, on the decoder only vs encoder-decoder point, taken that way it's very difficult (almost impossible) to have diffusion models have a better efficiency than GANs?

Thanks for the detailed comment, you've given me a lot to think about.

I haven't looked into latent diffusion yet. But what are you saying the output is converted to images/audio using GANs?

I didn’t know that GANs were still in use, that’s pretty cool.

As a technique I think it’s quite stunning, from an ML perspective. Hence why I’ve decided to write these blog posts. The GAN just has something about which makes it riveting to work with.

I’ve realised that Tero Karras made major contributions, I can across the PGGAN from the StyleGAN2. What did you mean by your last sentence, what is the limiting compute factor for GANs?

Thank you, I appreciate the kind comments!

My thoughts had been related to the ordering, but it makes sense that it doesn’t matter. I have read that it is actually better to train the model in separate batches with generated and real images in their own batches before the gradient step.

Is it better to train without the shuffling or shuffling has negligible effects?

It'd be cool to run some tests where you train a model with data and then supplement the training data with generated stuff.

Thank you

Right, even though the paper is almost 10 years old I still found it fascinating. I hope you enjoyed the post!

I think diffusion models are useful too, I’m currently working on a project to use them to generate medical type data. It seems they'd both be useful as they are both targeted towards generation of data, especially in areas where data is hard to come by. Doing this blog made me wonder of the application in finance too.

Which open source project?