GPU Accelerated Image Processing with Dragon Board!

Hi every body;
I’ve developed an application for dragon board 410c to evaluate the board and specifically it’s gpu
so I’ve paste the first page of my github wiki page for that project, please take a look at and try it.
I believe there are good tip in the wiki
the app works with both D3 camera board and AIStarvision camera board.
The source code originally forked from d3 engineering github.

the github address is: https://github.com/gholamlooali/410c_camera_support

Introduction:

It’s a while that I’m evaluating the dragon board 410c for a computer vision application so I developed the source code originally forked from D3 Engineering website to test the csi camera on the db410c. the board has been tested in 2 mode: 1- direct dump mode 2- nv12 format mode In this journey I have encountered different subject that I’m trying to share which might help anybody who wants to use this board.

First thing first:

In this project I used the debian release 17.06 build #252 ,as it’s mentioned in the 96boards website in the debian image the camera is disabled and all I had to do was to make a small correction on the Device Tree File (enable camera) and re complile and rebuild the image . (every step is clearly in the 96boards web site . piece of cake :slight_smile: ). Forgot to say I used 2 camera board one from D3 and another from AIStar vision, both working fine and did not notice any problem.

How to Run the application:

you can run the app in 2 mode :

1- Direct Dump - In this mode there is no image processing by snapdragon Soc and any frame from camera directly goes to memory and can be used, in this mode the format of the image is UYVY and res. =1920 x 1080 (which is a cropped image from original image)

2- NV12 format - Here the image is processed by CMSS (qualcomm camera module sub system) inside the soc and the final frame is in the NV12 format and can be cropped or scaled by the topology defined in the media-ctl command (in this app topology is res=1280x960 , no scale, no crop, format=NV12)

Applicatiion can be launched by :

./capture -d /dev/video0 -s /dev/v4l-subdev10 -v 3

for direct dump

./capture -d /dev/video3 -s /dev/v4l-subdev10 -v 3

for NV12 format (the only difference is video0 or video3)

Image Processing Commands:

while application is running,

press y to show only the y portion (luminance)

press u to show only the u portion (cb-chrominance)

press v to show only the v portion (cr-chrominance)

press r to show only the red color

press g to show only the green color

press b to show only the blue color

press c to show all colors

press s to apply soble filter for edge detection

press q to quit

Hey,

Thanks for putting the time and effort into putting together this project. I’ve been to trying to get the code working on my setup:

  • DragonBoard410c with Linux linaro-alip 4.9-camera-lt-qcom #1 SMP PREEMPT Fri Jun 30 16:14:43 UTC 2017 aarch64 GNU/Linux
  • D3 Engineering mezzanine camera module Rev 1

I flashed on the latest debian image along with the D3E mezzanine boot and rootfs images onto the dragonboard.

After cloning your repo and running ./capture -d /dev/video3 -s /dev/v4l-subdev10 -v 3 in the opengles_capture dir, I get the following output:

linaro@linaro-alip:~/410c_camera_support/opengles_capture$ cat out.txt
Unable to setup formats: Invalid argument (22)
111111111111/dev/video3
[1522246276.416816] DEBUG: cap->v4l2_fd= 3  [capture.c capture_setup:555]
[1522246276.416857] DEBUG: request count=8  [capture.c capture_setup:574]
[1522246276.425830] DEBUG: request count=8  [capture.c capture_setup:587]
[1522246276.425856] DEBUG: cap->num_buf=8  [capture.c map_buffers:175]
[1522246276.425861] DEBUG: cap->num_planes=1  [capture.c map_buffers:176]
[1522246276.425880] DEBUG: Buffer #0  [capture.c print_v4l2_buffer:68]
[1522246276.425885] DEBUG:      type:   9  [capture.c print_v4l2_buffer:70]
[1522246276.425889] DEBUG:      memory: 1  [capture.c print_v4l2_buffer:71]
[1522246276.425894] DEBUG:       - Plane#: 0  [capture.c print_v4l2_buffer:91]
[1522246276.425899] DEBUG:       - used:   0  [capture.c print_v4l2_buffer:92]
[1522246276.425903] DEBUG:       - length: 3110400  [capture.c print_v4l2_buffer:93]
[1522246276.425908] DEBUG: buf->m.planes[p].length=3110400   cap->v4l2_fd=3  [capture.c map_buffers:213]
[1522246276.426444] DEBUG: Buffer #1  [capture.c print_v4l2_buffer:68]
[1522246276.426452] DEBUG:      type:   9  [capture.c print_v4l2_buffer:70]
[1522246276.426456] DEBUG:      memory: 1  [capture.c print_v4l2_buffer:71]
[1522246276.426461] DEBUG:       - Plane#: 0  [capture.c print_v4l2_buffer:91]
[1522246276.426465] DEBUG:       - used:   0  [capture.c print_v4l2_buffer:92]
[1522246276.426469] DEBUG:       - length: 3110400  [capture.c print_v4l2_buffer:93]
[1522246276.426473] DEBUG: buf->m.planes[p].length=3110400   cap->v4l2_fd=3  [capture.c map_buffers:213]
[1522246276.426993] DEBUG: Buffer #2  [capture.c print_v4l2_buffer:68]
[1522246276.427000] DEBUG:      type:   9  [capture.c print_v4l2_buffer:70]
[1522246276.427005] DEBUG:      memory: 1  [capture.c print_v4l2_buffer:71]
[1522246276.427009] DEBUG:       - Plane#: 0  [capture.c print_v4l2_buffer:91]
[1522246276.427013] DEBUG:       - used:   0  [capture.c print_v4l2_buffer:92]
[1522246276.427017] DEBUG:       - length: 3110400  [capture.c print_v4l2_buffer:93]
[1522246276.427021] DEBUG: buf->m.planes[p].length=3110400   cap->v4l2_fd=3  [capture.c map_buffers:213]
[1522246276.427492] DEBUG: Buffer #3  [capture.c print_v4l2_buffer:68]
[1522246276.427498] DEBUG:      type:   9  [capture.c print_v4l2_buffer:70]
[1522246276.427502] DEBUG:      memory: 1  [capture.c print_v4l2_buffer:71]
[1522246276.427506] DEBUG:       - Plane#: 0  [capture.c print_v4l2_buffer:91]
[1522246276.427510] DEBUG:       - used:   0  [capture.c print_v4l2_buffer:92]
[1522246276.427514] DEBUG:       - length: 3110400  [capture.c print_v4l2_buffer:93]
[1522246276.427518] DEBUG: buf->m.planes[p].length=3110400   cap->v4l2_fd=3  [capture.c map_buffers:213]
[1522246276.427984] DEBUG: Buffer #4  [capture.c print_v4l2_buffer:68]
[1522246276.427989] DEBUG:      type:   9  [capture.c print_v4l2_buffer:70]
[1522246276.427993] DEBUG:      memory: 1  [capture.c print_v4l2_buffer:71]
[1522246276.427997] DEBUG:       - Plane#: 0  [capture.c print_v4l2_buffer:91]
[1522246276.428001] DEBUG:       - used:   0  [capture.c print_v4l2_buffer:92]
[1522246276.428005] DEBUG:       - length: 3110400  [capture.c print_v4l2_buffer:93]
[1522246276.428010] DEBUG: buf->m.planes[p].length=3110400   cap->v4l2_fd=3  [capture.c map_buffers:213]
[1522246276.428458] DEBUG: Buffer #5  [capture.c print_v4l2_buffer:68]
[1522246276.428463] DEBUG:      type:   9  [capture.c print_v4l2_buffer:70]
[1522246276.428467] DEBUG:      memory: 1  [capture.c print_v4l2_buffer:71]
[1522246276.428472] DEBUG:       - Plane#: 0  [capture.c print_v4l2_buffer:91]
[1522246276.428476] DEBUG:       - used:   0  [capture.c print_v4l2_buffer:92]
[1522246276.428480] DEBUG:       - length: 3110400  [capture.c print_v4l2_buffer:93]
[1522246276.428484] DEBUG: buf->m.planes[p].length=3110400   cap->v4l2_fd=3  [capture.c map_buffers:213]
[1522246276.428941] DEBUG: Buffer #6  [capture.c print_v4l2_buffer:68]
[1522246276.428946] DEBUG:      type:   9  [capture.c print_v4l2_buffer:70]
[1522246276.428950] DEBUG:      memory: 1  [capture.c print_v4l2_buffer:71]
[1522246276.428954] DEBUG:       - Plane#: 0  [capture.c print_v4l2_buffer:91]
[1522246276.428958] DEBUG:       - used:   0  [capture.c print_v4l2_buffer:92]
[1522246276.428962] DEBUG:       - length: 3110400  [capture.c print_v4l2_buffer:93]
[1522246276.429079] DEBUG: buf->m.planes[p].length=3110400   cap->v4l2_fd=3  [capture.c map_buffers:213]
[1522246276.429556] DEBUG: Buffer #7  [capture.c print_v4l2_buffer:68]
[1522246276.429562] DEBUG:      type:   9  [capture.c print_v4l2_buffer:70]
[1522246276.429566] DEBUG:      memory: 1  [capture.c print_v4l2_buffer:71]
[1522246276.429570] DEBUG:       - Plane#: 0  [capture.c print_v4l2_buffer:91]
[1522246276.429575] DEBUG:       - used:   0  [capture.c print_v4l2_buffer:92]
[1522246276.429579] DEBUG:       - length: 3110400  [capture.c print_v4l2_buffer:93]
[1522246276.429583] DEBUG: buf->m.planes[p].length=3110400   cap->v4l2_fd=3  [capture.c map_buffers:213]
[1522246278.592597] INFO:  EGL API is EGL_OPENGL_ES_API  [display.c egl_init:270]
[1522246278.602009] DEBUG: Surface Size 640 x 480  [display.c egl_init:340]
>>> disp->program[0] = gles_load_program(nv12_vertex_code, "nv12frag.glsl") (camera_nv12m_setup:711)
size of file=1633
[1522246278.649689] ERROR: Unable to link programerror: all shaders must use same shading language version
  [gles_egl_util.c gles_load_program:212]
<<< disp->program[0] = gles_load_program(nv12_vertex_code, "nv12frag.glsl"): succeeded (camera_nv12m_setup:711)
>>> disp->location[1] = glGetUniformLocation(disp->program[0], "s_chroma_texture") (camera_nv12m_setup:713)
ERROR: <<< disp->location[1] = glGetUniformLocation(disp->program[0], "s_chroma_texture"): failed: 0x0501 (UNKNOWN) (camera_nv12m_setup:713)

Hello,

What do you exactly mean by

Did you use custom boot and RFS images from D3 Engineering or the latest debian release from linaro?

Thanks,
Mani

Hi @ali,

Thanks a lot for sharing your findings, much appreciated :slight_smile:

Btw, where exactly you have used GPU acceleration? I can see only CAMSS doing the format conversion for NV12!

Thanks,
Mani

Hi Mani,

I first loaded the full debian (build 359) image onto the DragonBoard then flashed on the boot and rootfs images provided by D3 Engineering using fastboot

Hi @mashah,

You should use either one of those. If you are using OV5640 camera sensor, then you can use custom images provided by D3 Engineering. But for the case of OV5645, you can follow the instructions in below guide:

https://github.com/96boards/documentation/blob/master/consumer/dragonboard410c/guides/camera-module.md

Hope this helps!

Thanks,
Mani

Hi Mani,

I tried following the D3 guide to flash the debian image using fastboot but I still get the same error, mentioned in my first post, when running the capture code provided by Ali

Hi Abbas @mashah

I suspect you have a “version” problem. The Images provided by D3 are based on the 17.06 builds and infrastructure, build 359 is a later build. in between 17.06, and 18.01 there were changes to the low-level boot loaders, hence the boot and root partitions are not interchangeable between versions.

You will need to go back to the 17.06 SDCard install (which also reloads the low-level boot loaders), then after it is up and running, then you can use fastboot to flash the D3 boot and root images and they will work.

Full disclosure: I am an employee of Qualcomm Canada, any opinions I have expressed in this or any other post may not reflect the opinions of my employer.

Hi @ljking,

I did as you suggested but I still get the same error