RKNN toolkit support - 1d pytorch conversion

General
#1

Not sure this is really appropriate to this board, but can’t really find anywhere else offering support, and I assume there are at least a few people here also working with the RKNN toolkit so thought I’d give it a go.

Having some trouble with pytorch model conversion, targeting the TB-96AIoT RK1808 board. I should preface this by saying we are trying to use layers which are definitely NOT listed in the supported OPs document (https://github.com/rockchip-linux/rknn-toolkit/blob/master/doc/RKNN_OP_Support_V1.6.0.md), so this may ultimately be an unsolvable problem. To confirm my environment is correctly configured, the example conversion script runs fine:

--> config model
done
--> Loading model
./resnet18.pt ********************
done
--> Building model
W The target_platform is not set in config, using default target platform rk1808.
done
--> Export RKNN model
done
--> Init runtime environment
done
--> Running model

-----TOP 5-----
[812]: 0.9993759989738464
[404]: 0.0004593815538100898
[657 833]: 3.2843185181263834e-05
[657 833]: 3.2843185181263834e-05
[895]: 2.075813608826138e-05

done

Running the same script with a deliberately broken input size still ‘works’, or rather the load step still works. It complains later about wrong size and gets a wrong answer, but the model does get loaded, converted, and executed.

Trying to convert my own model, I only get as far as the load step. Here you see the output when trying to load a pytorch model with a MaxPool1D layer, which is not supported.

E Catch exception when loading pytorch model: /home/colin/VirtualboxShare/xyzronin_v0-5_maxpool1d.pt!
E Traceback (most recent call last):
E   File "rknn/api/rknn_base.py", line 339, in rknn.api.rknn_base.RKNNBase.load_pytorch
E   File "rknn/base/RKNNlib/RK_nn.py", line 146, in rknn.base.RKNNlib.RK_nn.RKnn.load_pytorch
E   File "rknn/base/RKNNlib/app/importer/import_pytorch.py", line 128, in rknn.base.RKNNlib.app.importer.import_pytorch.ImportPytorch.run
E   File "rknn/base/RKNNlib/converter/convert_pytorch_new.py", line 2250, in rknn.base.RKNNlib.converter.convert_pytorch_new.convert_pytorch.load
E   File "rknn/base/RKNNlib/converter/convert_pytorch_new.py", line 2055, in rknn.base.RKNNlib.converter.convert_pytorch_new.PyTorchOpConverter.report_missing_conversion
E NotImplementedError: The following operators are not implemented: ['aten::max_pool1d']

Note that it explicitly states that MaxPool1D is unsupported. Here you see the essentially the same model with an unsqueeze-maxpool2d-squeeze replacing the MaxPool1D.

W Channels(200) of input node: input.1 > 128, mean/std values will be set to default 0/1.
W Please do pre-processing manually before inference.
E Catch exception when loading pytorch model: /home/colin/VirtualboxShare/xyzronin_v0-5_maxpool2d.pt!
E Traceback (most recent call last):
E   File "rknn/api/rknn_base.py", line 339, in rknn.api.rknn_base.RKNNBase.load_pytorch
E   File "rknn/base/RKNNlib/RK_nn.py", line 146, in rknn.base.RKNNlib.RK_nn.RKnn.load_pytorch
E   File "rknn/base/RKNNlib/app/importer/import_pytorch.py", line 128, in rknn.base.RKNNlib.app.importer.import_pytorch.ImportPytorch.run
E   File "rknn/base/RKNNlib/converter/convert_pytorch_new.py", line 2255, in rknn.base.RKNNlib.converter.convert_pytorch_new.convert_pytorch.load
E   File "rknn/base/RKNNlib/converter/convert_pytorch_new.py", line 2370, in rknn.base.RKNNlib.converter.convert_pytorch_new.convert_pytorch.parse_nets
E   File "rknn/base/RKNNlib/converter/convert_pytorch_new.py", line 2059, in rknn.base.RKNNlib.converter.convert_pytorch_new.PyTorchOpConverter.convert_operators
E   File "rknn/base/RKNNlib/converter/convert_pytorch_new.py", line 643, in rknn.base.RKNNlib.converter.convert_pytorch_new.PyTorchOpConverter.convolution
E IndexError: tuple index out of range

As you can see it no longer complains about the MaxPool, but seems to have a problem with a convolution. We are using 1D convolutions, which again are not explicitly supported, but it doesn’t explicitly complain about an unsupported layer. The index error is also seemingly not related to the input size; this was established by my attempt to feed the example a wrong input size earlier; it got past this step without any issue. I have also tried all plausible input sizes and some completely nonsense sizes, with no change.

My guess is that it’s attempting to work through the convolution layer, but assuming it’s a 2D convolution, and going out of bounds looking for a non-existent dimension.

Can anyone shed any light on this, or offer suggestions? One of our team is currently working on switching out the Conv1D with an unsqueeze-Conv2D-squeeze, to confirm whether that is the issue. However we have many other 1D layers so it would be a pain to do that everywhere.

It’s really frustrating that every NPU and associated software suite seems to be heavily engineered towards image recognition, to the detriment of other layer types.

Please ask for more information if I haven’t provided enough. Thanks.

#2

For anyone who might be interested, it seems like our guess was right. Converting all the 1D layers to unsqueeze-2Dvariant-squeeze gives a successful conversion and successful inference. Unfortunately this has made our model a lot bigger so we’ll have to explore further optimizations, or possibly custom operators.

#3

One of our next steps is to attempt a quantized conversion for performance reasons, but the RKNN API appears to make the assumption that sample data can be supplied as image files. The example code looks like this:

rknn.build(do_quantization=True, dataset="dataset.txt")

This is asking for a text file containing image paths. I have to assume the image loading is done with opencv, since everything else in the examples is based on opencv. Since I can’t realistically save a standard-compliant 6-channel image, this isn’t going to work when reading. Any tips here, or any way to pass the sample data directly rather than through a file?

Thanks

#4

We’re pressing forward without quantization for now, and might revisit it later.

Without quantization we’re having a lot of trouble getting the rknn output to match the pytorch output. It doesn’t seem to care much whether the means and stds are supplied or whether they are realistic values; output differs in a seemingly random way from the original model. Can anyone confirm they’ve had a decent model conversion outside of the example models/scripts? And if so, would you willing to share any code samples?