Foreword

The author of the technical paper involved in this article is currently under some controversy due to academic misconduct, but the paper itself has not been confirmed to be involved in the controversial content. This article is for learning and record-keeping purposes only. If the paper or its technology is later proven to be involved in academic issues, this article will be permanently archived or deleted.

Preliminary Introduction

In machine learning, Convolution is one of the most classic algorithms, especially in the field of Computer Vision (CV). We can briefly review the operation of 2D convolution, which extracts features by sliding a kernel over 2D features. Before delving into Involution, we need to understand some characteristics of traditional convolution. Convolutional operations slide a shared convolutional kernel across spatial locations. This design has two core features: spatial-agnosticism and channel-specificity.

Spatial-agnosticism means that the same convolutional kernel is applied to all spatial positions of the feature map; the weights used are identical regardless of the position. This design is highly efficient in terms of parameter usage but also limits the model's ability to adaptively model different spatial positions. Channel-specificity is reflected in convolution learning independent convolutional kernels for each output channel, which introduces redundancy in the kernels.

The Involution operator, on the other hand, is designed based on characteristics opposite to those of convolution: it possesses spatial-specificity and channel-agnosticism. This means that Involution generates a unique kernel for each spatial position, and these kernels are shared across the channel dimension, offering the following advantages:

1. Enhanced adaptability: Each pixel location has its own processing kernel, allowing it to better capture local contextual information;
2. Improved parameter efficiency: The channel-sharing mechanism significantly reduces the number of parameters, especially in deep networks with a high number of channels;
3. Long-range dependency modeling: By increasing the kernel size, Involution can efficiently model long-range spatial relationships while reducing the loss of short-range information, balancing the ability to extract data at both close and distant scales.

Simply put, this operator produces adaptive kernels at each position, while in the channels, grouping determines which channels share a core. This allows each element to use a different kernel while avoiding redundancy produced by different channels. Next, I will briefly explain its principle.

Principle Overview

Taking 2D data as an example, for an input feature map \(\mathbf{X} \in \mathbb{R}^{H \times W \times C}\), the output of Involution at position \((i,j)\) can be expressed as:

\[ Y_{i,j}= \sum_{(u,v)\in\mathcal{N}(i,j)} \mathcal{H}_{i,j,u,v} \cdot \mathbf{X}_{u,v} \]

where \(\mathcal{N}(i,j)\) is the neighborhood centered at \((i,j)\), and \(\mathcal{H}_{i,j} \in \mathbb{R}^{K \times K \times C}\) is the involution kernel for that position. Its kernel mapping function is defined as: \[ \phi = \mathbb{R}^C \mapsto \mathbb{R}^{K\times K\times G} \] This expression means that the kernel mapping, with channel as the dimension, takes a three-dimensional form of kernel size multiplied by the number of groups. It can be understood that the kernel mapping operation is performed based on the channel vector corresponding to the element's position. The following function is its specific expression: \[ \mathcal{H}_{i,j} = \phi(\mathbf{X}_{i,j})=W_1\sigma(W_0\mathbf{X}_{i,j}) \] Here, \(W_0\) and \(W_1\) are two linear transformations, and \(\sigma\) refers to the activation function. After obtaining the kernel parameters through the above mapping, they are then applied to the original data to complete the kernel operation. Below is the schematic diagram used in the original paper: My understanding is this: Involution generates kernels based on single-position channel information. This way, the kernel parameters themselves contain channel features. Then, through channel sharing, it avoids independent channel operations and the multi-channel integration process during output, thereby reducing computational complexity.

Basic Implementation Analysis (Code)

The Involution kernel generation function mentioned above uses two linear transformations to generate kernel weights. How exactly is this done? The original paper provides the implementation code; let's observe the specific methods of these two transformations from a code perspective.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# B: batch size, H: height, W: width
# C: channel number, G: group number
# K: kernel size, s: stride, r: reduction ratio
################### initialization ###################
o = nn.AvgPool2d(s, s) if s > 1 else nn.Identity()
reduce = nn.Conv2d(C, C//r, 1)
span = nn.Conv2d(C//r, K*K*G, 1)
unfold = nn.Unfold(K, dilation, padding, s) # unfolds the spatial neighborhood into columns for easier calculation with kernel parameters later
#################### forward pass ####################
x_unfolded = unfold(x) # B,CxKxK,HxW
x_unfolded = x_unfolded.view(B, G, C//G, K*K, H, W)
# kernel generation, Eqn.(6)
kernel = span(reduce(o(x))) # B,KxKxG,H,W
kernel = kernel.view(B, G, K*K, H, W).unsqueeze(2)
# Multiply-Add operation, Eqn.(4)
out = mul(kernel, x_unfolded).sum(dim=3) # B,G,C/G,H,W
out = out.view(B, C, H, W)
return out