Comment by threeducks | Hacker News Reader

Comment by threeducks | Hacker News Reader

Parent Full thread

ConvNeXT's architecture contains an AdaptiveAvgPool2d layer: https://github.com/pytorch/vision/blob/5f03dc524bdb7529bb4f2...

This means that you can split your image into tiles, process each tile individually, average the results, apply a final classification layer to the average and get exactly the same result. For reference, see the demonstration below.

You could of course do exactly the same thing with a vision transformer instead of a convolutional neural network.

That being said, architecture is wildly overemphasized in my opinion. Data is everything.

    import torch, torchvision.models

    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = torchvision.models.convnext_small()
    model.to(device)
    tile_size, image_size = 32, 224 # note that 32 divides 224 evenly
    image = torch.randn((1, 3, image_size, image_size), device=device)

    # Process image as usual
    x_expected = model(image)

    # Process image as tiles (using for-loops for educational purposes; should use .view and .permute instead for performance)
    features = [
        model.features(image[:, :, y:y + tile_size, x:x + tile_size])
        for y in range(0, image_size, tile_size)
        for x in range(0, image_size, tile_size)]
    x = model.classifier(sum(features) / len(features))

    print(f"Mean squared error: {(x - x_expected).pow(2).mean().item():.20f}")

Slicing up images to analyze them is definitely something people do - in many cases, such as satellite imagery, there is not much alternative. But it should be done mindfully, especially if there are differences between the training and testing steps. Depending on the architecture and the application, it's not the same as processing the whole image at once. Some differences are more or less obvious (for example, you might have border artifacts), but others are more subtle. For example, contrary to the expected positional equivariance of convolutional nets, they can implicitly encode positional information based on where they see border padding during training. For some types of normalization such as instance normalization, the statistics of the normalization may vary significantly when applied across patches or whole images.

As someone who's done a fair bit of architecture work -- both are important! Making it either or is a very silly thing, both are the limiting factor for the other and there are no two ways about it.

Also, for classification, MaxPooling is often far superior, you can learn an average smoothing filter in your convolutions beforehand in a data-dependent manner so that Nyquist sampling stuff is properly preserved.

Also, please do smoothed crossentropy for image class stuff (generally speaking, unless maybe data is hilariously large), MSE won't nearly cut it!

But that being said, adaptive stuff certainly is great when doing classification. Something to note is that batching does become an issue at a certain point -- as well as certain other fine-grained details if you're simply going to average it all down to one single vector (IIUC).

> Also, please do smoothed crossentropy for image class stuff (generally speaking, unless maybe data is hilariously large), MSE won't nearly cut it!

Of course. The MSE here is not intended to be a training loss, but as a means to demonstrate that both approaches lead to almost the same result except for some rounding error. The MSE is somewhere in the order of 10^-9.

> Also, for classification, MaxPooling is often far superior, you can learn an average smoothing filter in your convolutions beforehand in a data-dependent manner so that Nyquist sampling stuff is properly preserved.

I don't think that max pooling the last feature maps would be a good idea here, because it would cut off about 98 % of the gradients and training would take much longer. (The shape of the input feature layer is (1, 768, 7, 7), pooled to (1, 768, 1, 1).)

> Something to note is that batching does become an issue at a certain point

Could you elaborate on that?

> The MSE here is not intended to be a training loss, but as a means to demonstrate that both approaches lead to almost the same result except for some rounding error.

Ah, gotcha

> I don't think that max pooling the last feature maps would be a good idea here, because it would cut off about 98 % of the gradients and training would take much longer. (The shape of the input feature layer is (1, 768, 7, 7), pooled to (1, 768, 1, 1).)

MaxPooling is generally only useful if you're training your network for it, but in most cases it ends up performing better. That sparsity actually ends up being a good thing -- you generally need to suppress all of those unused activations! It ends up being quite a wide gap in practice (and, if you have convolutions beforehand -- using avgpooling2d is a bit of extra wasted extra computation blurring the input)

> Could you elaborate on that?

Variable-sized inputs don't batch easily as the input dims need to match, you can go down the padding route but that has its own particularly hellacious costs with it that end up taking away from compute that you could be using for other useful things.

Reply on news.ycombinator.com