HW format conversion YUY2 to NV12 inside gst pipeline when using USB webcam (uvcvideo)

Products Support DragonBoard410c
#1

I have a Dragonboard 410c and I created a pipeline to save h.264 encoded video from an USB camera (uvcvideo) to sdcard. So far, I managed to use v4l2h264enc to do hardware encoding and everything works like a charm.

The main issue with this pipeline is that my camera (like many USB cameras) is only capable to output image in only two formats:

  • YUY2 [raw]
  • MJPEG [compressed]

The encoder is capable to take only NV12 format as input. So, to create the pipeline, I need to do a video format conversion which sadly consumes CPU (and increases the overall power consumption).

Right now I need to find a way to do this format conversion by taking advantage of the DB410’s hardware blocks. I think I have four options (which may lead to a lower power consumption):

  • glcolorconvert: plugin for gstreamer which will use the Ardeno GPU via opengl to offload the conversion
  • libyuv, which can do this conversion in two steps, and uses NEON SIMD to optimize the conversion.
  • mediactl: configure a pipeline that takes advantage of the Camera Subsistem Hardware (CAMSS) which has a built-in converter
  • Hexagon SDK: create a gstreamer plugin which uses the dsp to offload the conversion operation.

In my development workflow I use YP - warrior, and linux-linaro-qcomlt version 4.14 (I also tried YP zeus + linux v5.4, but had some issues with gstreamer, and gave up very quickly - will return to the latest version as soon as I solve my pipeline issues).

Any advice would be greatly appreciated.

Thanks,
Stefan

#2

How much cpu? can you attach the pipeline you’re using? I would first try to add queue element before and after videoconvert to decouple from other elements, and tune n-threads to split software convertion over the four cpus. example:

… queue ! videoconvert n-threads=4 ! queue…

AFAIK it’s currently not possible to convert YUV to NV12 with this method, I’ve only seen it working with NV12 to RGBA…

AFAIK, videoconvert is already optimized, and should rely on SIMD/Neon.

It’s possible, but no sure about the final gain.

#3

Hi Loic,

With my current gstreamer pipeline, it uses 50% of my CPU, and the current draw is at 130 mA (at 12V) on average. I now tried tunning the videoconvert element like you suggested (ie. increasing the n-threads count), and the CPU load distributes across the 4 cores, but the power draw remains the same.
chart
Here is a chart that describes the pipeline power consumption in two cases:

  • 0s - 50s: represents the pipeline with the videoconvert with n-threads set to 1
  • 60s - end: the pipeline with the videoconvert with n-threads set to 4 (max)

Here is the pipeline I’m using now:

gst-launch-1.0 \
v4l2src device=/dev/video0 io-mode=userptr \
! capsfilter caps=video/x-raw,format=YUY2,width=640,height=480,framerate=30/1 \
! queue \
! videoconvert n-threads=4 \
! queue \
! capsfilter caps=video/x-raw,format=NV12,width=640,height=480,pixel-aspect-ratio=1/1,framerate=30/1 \
! queue \
! v4l2h264enc extra-controls="controls,h264_level=12,h264_profile=1" capture-io-mode=dmabuf \
! h264parse \
! splitmuxsink \
      location=video%02d.mkv \
      max-size-time=10000000000 \
      max-files=10 \
      muxer-factory=matroskamux

According to the gstreamer docs, it’s possible to use glcolorconvert in this case. But in my case it didn’t worked.

Dragonboard’s Camera Subsystem would be a real help if it would be able to process software buffers. According to the camera docs, PIX interface of VFE does exactly what I need, but it’s only been used for CSI, not USB.