1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# B: batch size, H: height, W: width
# C: channel number, G: group number
# K: kernel size, s: stride, r: reduction ratio
################### initialization ###################
o = nn.AvgPool2d(s, s) if s > 1 else nn.Identity()
reduce = nn.Conv2d(C, C//r, 1)
span = nn.Conv2d(C//r, K*K*G, 1)
unfold = nn.Unfold(K, dilation, padding, s) # Extract spatial neighborhoods and unfold into columns for subsequent computation with kernel parameters
#################### forward pass ####################
x_unfolded = unfold(x) # B,CxKxK,HxW
x_unfolded = x_unfolded.view(B, G, C//G, K*K, H, W)
# kernel generation, Eqn.(6)
kernel = span(reduce(o(x))) # B,KxKxG,H,W
kernel = kernel.view(B, G, K*K, H, W).unsqueeze(2)
# Multiply-Add operation, Eqn.(4)
out = mul(kernel, x_unfolded).sum(dim=3) # B,G,C/G,H,W
out = out.view(B, C, H, W)
return out

In the initialization section, we can see that two Conv2d functions are used, and the convolution kernel size specified in both is 1. As we know, a \(1\times 1\) convolution kernel calculation only involves the channel dimension of the data, without touching other dimensional information. In other words, this function performs a linear combination across channels corresponding to each position. Connecting this with the formulas above, these two 2D convolution functions with a kernel size of 1 are essentially the linear transformations \(W_0\) and \(W_1\) mentioned earlier. Their forms can be written as: \[ \begin{align*} &W_0:\mathbb{R}^C \mapsto \mathbb{R}^{C//r}\\ &W_1:\mathbb{R}^{C//r} \mapsto \mathbb{R}^{K\times K\times G}\\ \end{align*} \] The first transformation \(W_0\) performs dimensionality reduction to reduce the parameter count for computing the kernel; the second transformation \(W_1\) performs mapping, projecting the compressed features into the required number of weight values.

These two transformations are the core steps of kernel generation. However, up to this point, the core parameters have not yet assumed the theoretical 3D shape. Therefore, in the forward pass section, the array is reshaped. The neighborhood of the output is unfolded into columns, which are then grouped according to channels, rearranging the dimensions to obtain the desired kernel shape \((B, G, K\times K, H, W)\). Here, \(H\) and \(W\) refer to the spatial output positions, and it is finally applied across the complete channel dimension to yield the final feature map.

Of course, the above is only a brief analysis of the basic code implementation. Understanding up to this point is sufficient to adjust the operator according to project requirements.

Simple Project Application

In the author's project, the Involution operator was used to build a neural network to predict disease probability, utilizing a dataset based on patient information and clinical data manifestations as features (refer to the UCI Heart Disease Dataset). This dataset does not consist of sequential information, yet convolution can still be used to extract relational features among attributes within a local range. As mentioned earlier, the characteristics of the Involution operator allow it to retain local feature extraction capabilities across broad data spans while demanding lower computational cost, making it suitable for non-sequential datasets as well. However, because this dataset is one-dimensional, the original code cannot be directly applied, and the operator needs to be reduced in dimension.

Once the basic implementation method is understood, the logic for dimensional reduction is straightforward: simply modify the original 2D convolution operations to 1D convolutions. The logic of a 1D involution operator can be referenced in the example below:

The kernel generation operation is carried out across channels and does not have much to do with the spatial dimensionality of the data itself. It can also be generalized to higher-dimensional datasets; however, existing libraries cannot be directly invoked, and high-dimensional convolution operations must be implemented manually.

Of course, as seen in the example above, the author also modified the activation function in the operator's kernel generation step. The original code used ReLU, while the author chose Swish, which is more sensitive to small negative values. This can be selectively adjusted according to the project. If the network constructed using the involution operator does not perform as expected, you can try adjusting the activation function. In the author's project, the improvement brought by this adjustment was relatively minor.

Thoughts on Model Acceleration Schemes

An idea occurred to the author while completing another course on parallel computing: since kernel generation in involution is computed based on fixed data, it could potentially be parallelized for acceleration. In fact, a CUDA-accelerated version is already provided in the original authors' source code. According to conventional parallelization thinking, the kernel generation operation at each position can be parallelized. The author experimented with a step-by-step parallelization method: parallelizing the matrix operations of the two convolutions separately, with a wait state at the intermediate layer. This parallelization approach achieved good speedup even when running on a CPU. However, this was merely a simple experiment, as model training is rarely conducted on CPUs in most scenarios. Perhaps there are other approaches...

References

[1] Li D, Hu J, Wang C, et al. Involution: Inverting the inherence of convolution for visual recognition[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 12321-12330.

[2] Involution: Spatial Non-sharing? A High-Performance Operator Fully Replacing Convolution | CVPR 2021