Struggling with OV5640 camera scaling using D3 camera mazzanine

PiBorg · July 10, 2017, 3:57pm

Hi everyone,

I have been trying to get the pipeline scaling as mentioned here working with an OV5640 camera without much luck.

So far the best I have managed is to use the camera at a higher resolution of 1280x720 and rescale the image in software using OpenCV, but this has a large performance impact.

The working commands for setting this up are:

sudo media-ctl -d /dev/media1 -l '"msm_csiphy0":1->"msm_csid0":0[1],"msm_csid0":1->"msm_ispif0":0[1],"msm_ispif0":1->"msm_vfe0_rdi0":0[1]'
sudo media-ctl -d /dev/media1 -V '"ov5640 1-0078":0[fmt:UYVY8_2X8/1280x720 field:none],"msm_csiphy0":0[fmt:UYVY8_2X8/1280x720 field:none],"msm_csid0":0[fmt:UYVY8_2X8/1280x720 field:none],"msm_ispif0":0[fmt:UYVY8_2X8/1280x720 field:none],"msm_vfe0_rdi0":0[fmt:UYVY8_2X8/1280x720 field:none]'

This allows me to grab the camera images and shrink them myself.

What I am trying to get working is using the pipeline to scale the image down much smaller, such as 480x270, to reduce the CPU workload for processing the images.

My current commands look like this:

sudo media-ctl -d /dev/media1 -l '"msm_csiphy0":1->"msm_csid0":0[1],"msm_csid0":1->"msm_ispif0":0[1],"msm_ispif0":1->"msm_vfe0_pix":0[1]'
sudo media-ctl -d /dev/media1 -V '"ov5640 1-0078":0[fmt:UYVY8_2X8/1920x1080 field:none],"msm_csiphy0":0[fmt:UYVY8_2X8/1920x1080 field:none],"msm_csid0":0[fmt:UYVY8_2X8/1920x1080 field:none],"msm_ispif0":0[fmt:UYVY8_2X8/1920x1080 field:none],"msm_vfe0_pix":0[fmt:UYVY8_2X8/1920x1080 field:none compose:(0,0)/480x270],"msm_vfe0_pix":1[fmt:UYVY8_2X8/480x270 field:none]'

This fails with the error Unable to setup formats: Inappropriate ioctl for device (25), so I presume I have made a mistake somewhere.

I am currently running the Release v1.1.1 Linux Kernel from the D3Engineering GitHub, full version Linux linaro-alip 4.9-camera-lt-qcom #1 SMP PREEMPT Fri Jun 30 16:14:43 UTC 2017 aarch64 GNU/Linux

I have also tried the latest standard Linaro Debian build, dragonboard410c_sdcard_install_debian-252.zip, but it does not show /dev/media1 at all so I presume the necessary driver(s) for the camera are missing.

Any help pointing me in the right direction would be appreciated as I am rather stuck at this stage. In particular it would be helpful to know if this is likely to be a mistake with the commands or a problem with the software setup / drivers.

Thanks,
Arron.

ljking · July 10, 2017, 6:02pm

Hi @PiBorg:

The issues you are seeing are:

the default 17.06 image doesn’t have cameras enabled in the device tree, hence you don’t see /dev/media1
the D3 image doesn’t have the video scaler support (I am sure they will add it soon)
the default gstreamer doesn’t have support for all of the features, you need to patch and rebuild gstreamer

The solution is to use the 17.06 image, enable the cameras, and patch gstreamer, then it will work.

I will provide detailed step-by-step instructions in a day or two, I have done this before but I want to go through my procedure from top to bottom one more time to make sure I didn’t miss any steps before I post it.

Full Disclosure: I am an employee of Qualcomm Canada, any opinions I may have expressed in this or any other post may not reflect the opinions of my employer.

ljking · July 11, 2017, 8:08pm

Sorry for passing on the information a bit at a time instead of all in one big post.

Instructions to rebuild gstreamer are part way down this page: http://builds.96boards.org/releases/dragonboard410c/linaro/debian/latest/ Keep in kind when you get to the quilt import/quilt push steps you need to run it 12 times, once for each of the 12 patches. I built a script to do it all:

mkdir gstreamer
cd gstreamer
sudo apt-get install -y build-essential fakeroot devscripts quilt
sudo apt-get build-dep gstreamer1.0-plugins-good
apt-get source gstreamer1.0-plugins-good
cd gst-plugins-good1.0-1.10.4/
quilt import ../../V4L2_Patches/01.patch
quilt push
quilt import ../../V4L2_Patches/02.patch
quilt push
quilt import ../../V4L2_Patches/03.patch
quilt push
quilt import ../../V4L2_Patches/04.patch
quilt push
quilt import ../../V4L2_Patches/05.patch
quilt push
quilt import ../../V4L2_Patches/06.patch
quilt push
quilt import ../../V4L2_Patches/07.patch
quilt push
quilt import ../../V4L2_Patches/08.patch
quilt push
quilt import ../../V4L2_Patches/09.patch
quilt push
quilt import ../../V4L2_Patches/10.patch
quilt push
quilt import ../../V4L2_Patches/11.patch
quilt push
quilt import ../../V4L2_Patches/12.patch
quilt push
# edit Rules
sed -i '166a \\t--enable-v4l2-probe \\' debian/rules
sed -i '167a \\t--without-libv4l2 \\' debian/rules
#
debuild -b -uc -us
sudo dpkg -i ../gstreamer1.0-plugins-good_1.10.4-1_arm64.deb

of course I had copied the 12 patches to the directory V4L2_Patches. The script would be better if it had taken the patches directly from https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/log/?h=video-enc instead of expecting me to have downloaded the patches into a directory.

Full Disclosure: I am an employee of Qualcomm Canada, any opinions I may have expressed in this or any other post may not reflect the opinions of my employer.

PiBorg · July 14, 2017, 11:36am

I am now running the 17.06 image and have gstreamer patched

For anyone else following these are the commands I used to get all twelve patches from the repository:

mkdir V4L2_Patches
cd V4L2_Patches
wget -O 01.patch https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/patch/?id=76a6b804edc54cf78f4a6250e79108445ff61b82
wget -O 02.patch https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/patch/?id=b130deff607998cc1b3736d017e9e87e2945f1a5
wget -O 03.patch https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/patch/?id=ba3c90b58e17367318f08edc5d7ae6fc78a4f1ad
wget -O 04.patch https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/patch/?id=069dec4eee12c45d443235215799fbf158e71137
wget -O 05.patch https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/patch/?id=c6fb89ba0366e436334d0371ddad0b252f3d3065
wget -O 06.patch https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/patch/?id=c1f2bfd2f37bdad368004229424a3acbd6d4992c
wget -O 07.patch https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/patch/?id=d5261fc95b8dcdbae2ae96f2a43876b474479cca
wget -O 08.patch https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/patch/?id=45b039632a49219c954a5e6d7f28fa3d85adbd33
wget -O 09.patch https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/patch/?id=89957315847c027f6adab179e6929af22b6f2c54
wget -O 10.patch https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/patch/?id=1a4b231fdde7e744f9d918769d516f21e2498959
wget -O 11.patch https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/patch/?id=848cb3a988412ac2b56063fe65654224ef17b022
wget -O 12.patch https://git.linaro.org/people/todor.tomov/gst-plugins-good.git/patch/?id=3825320c95740d57501638563cdf2546dc016846

I guess that I just need the changes for the device tree and then I can try the scaling code again.