Latest Build 252 - Possible MIPI Video Encode Regression

Products Support DragonBoard410c
,
#1

Hello All,

I just moved over to the latest 252 release and noticed a bit of a regression in terms of video encode support -
The published raw video output format has changed in the latest version from NV12 to NV16 for some reason. This is an issue because AFAIK the Venus video encoder can only accept the NV12 raw format. Using NV16 in place of NV12 results in a pipeline not negotiated error.

Reverting to #227 resolves this issue.

I am using the AISTARVISION MIPI Adaptor v2.0 and the OV5645 MIPI-CSI Camera

Here is a log of this occurring:

Linaro Linux 17.06 releases
#2

hi,

have you checked the release notes, the instructions are slightly different now what we have added support for NV16 and NV12, you need to configure for NV12 output, see https://github.com/96boards/documentation/blob/master/ConsumerEdition/DragonBoard-410c/Guides/CameraModule.md#format-conversion.

#3

I read through the release notes and tried the following pipelines for nv12 1280x960@30FPS:

sudo media-ctl -d /dev/media1 -l '"msm_csiphy0":1->"msm_csid0":0[1],"msm_csid0":1->"msm_ispif0":0[1],"msm_ispif0":1->"msm_vfe0_rdi0":0[1]'
sudo media-ctl -d /dev/media1 -V '"ov5645 1-0076":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_csiphy0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_csid0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_ispif0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_vfe0_pix":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_vfe0_pix":1[fmt:UYVY8_1_5X8/1280x960 field:none]'

Using the above pipelines appears to work when running the gstreamer video record example, but it does not produce a usable video:
gst-launch-1.0 -e v4l2src device=/dev/video3 ! video/x-raw,format=NV12,width=1280,height=960,framerate=30/1 ! v4l2video5h264enc extra-controls="controls,h264_profile=4,video_bitrate=2000000;" ! h264parse ! mp4mux ! filesink location=video.mp4 Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Setting pipeline to PLAYING ... New clock: GstSystemClock ^Chandling interrupt. Interrupt: Stopping pipeline ... EOS on shutdown enabled -- Forcing EOS on the pipeline Waiting for EOS... Got EOS from element "pipeline0". EOS received - stopping pipeline... Execution ended after 0:02:03.047455430 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ...

When stopping the video record, and checking the file size it appears that we didn’t capture anything with a filesize of only 4KB with no video data, only the mp4 header.

linaro@linaro-developer:~$ ls -lash | grep video.mp4
4.0K -rw-r--r--  1 linaro linaro  595 Jul  6 17:02 video.mp4
linaro@linaro-developer:~$ cat video.mp4 
 ftypmp42mp42mp41isomisofremdat#moovlmvhdՄHՄ@rtrak\tkhdՄHՄH@�mdia mdhdՄHՄH!hdlr�minf$dinfdref
                                                                                                               url \stblstsdsttsstscstszstco=udta5meta!hdlrmhlrmdiilst=udta5meta!hdlrmhlrmdiilst

At this point I’m a little stumped, I have the proper format selected and gstreamer doesn’t error out but instead just sits there not recording anything. I would like to run at the full 1280x960 as I did on previous builds, so I’m not exactly sure how to configure scaling or cropping to make this work like this.

Any ideas? Is scaling/cropping required to align the frames to 32 lines? This seemed to be taken care of when using earlier builds.

#4

Tried the pipelines for format conversion to nv12 once again and still get the same errors as above, I cannot even generate a JPEG using the example pipeline in the format conversion section here.

This results in the following errors in my kernel log:
[ 370.484764] 1b0c000.qcom,cci supply qcom,gdscr-vdd not found, using dummy regulator
[ 370.485128] msm_cci_init:1015: hw_version = 0x10000008
[ 370.882935] 1b0c000.qcom,cci supply qcom,gdscr-vdd not found, using dummy regulator
[ 370.883333] msm_cci_init:1015: hw_version = 0x10000008
[ 384.181912] qcom-camss 1b0ac00.camss: VFE sof timeout
[ 384.693934] qcom-camss 1b0ac00.camss: VFE reg update timeout

The strange thing is that when using the direct dump to memory pipelines found here, I can generate an image from the camera just fine.

EDIT: Wait a minute, are the PIX interfaces even supported in this build? I went looking on the linux-media mailing list and saw that Todor had posted this message stating that only the RDI interface is supported with the new CAMSS driver and that the PIX interface is not. If the PIX interface is not supported then does that mean that we cannot use the video encoder with this version? I am under the impression that the video encoder depends on the NV12 format that isn’t provided by the RDI interface. Can you confirm this?
Source: http://www.spinics.net/lists/linux-media/msg117279.html

#5

Is anyone else having these issues with #252? Can anyone confirm the state of the PIX interface in the latest build?

#6

Hello,
I’m also having trouble trying to capture a jpeg image. (with build #252).
I’m not a Linux Guru so if you need precise information, don’t hesitate to be as exhaustive as possible in your request :slight_smile:
At the end of the day I need h264 video encoding too.

My error is:
Warning: erroneous pipeline: could not link v4l2src0 to jpegenc0

#7

Hello,

There was a small (but bad) error in the instructions on the wiki page. The media controller pipeline for the format conversion must link the “pix” entity, not “rdi0” entity as it was listed. So the correct one is:

sudo media-ctl -d /dev/media1 -l '"msm_csiphy0":1->"msm_csid0":0[1],"msm_csid0":1->"msm_ispif0":0[1],"msm_ispif0":1->"msm_vfe0_pix":0[1]'

Now it is fixed on the wiki too.

Could you please try this?

Yes. And then patch 11/19 adds the format conversion support using the PIX interface. So it is supported as per the information on the wiki page.

#8

Problem was somwhere else (not fully identified yet).
We are able to do video encoding.

We had quite some trouble making the pipeline work and at the end of the day we are using following pipe config:
sudo media-ctl -d /dev/media1 -l ‘“msm_csiphy0”:1->“msm_csid0”:0[1],“msm_csid0”:1->“msm_ispif0”:0[1],“msm_ispif0”:1->“msm_vfe0_rdi0”:0[1]’
sudo media-ctl -d /dev/media1 -V ‘“ov5640 3-0078”:0[fmt:UYVY8_2X8/1920x1080 field:none],“msm_csiphy0”:0[fmt:UYVY8_2X8/1920x1080 field:none],“msm_csid0”:0[fmt:UYVY8_2X8/1920x1080 field:none],“msm_ispif0”:0[fmt:UYVY8_2X8/1920x1080 field:none],“msm_vfe0_rdi0”:0[fmt:UYVY8_2X8/1920x1080 field:none]’

So we are using rdi, not pix.
I will have a look after application of todortomov patch. Keep you posted.

PS: we are using a different hardware than 96 board and an OV5640 sensor (instead of OV5645).
PS2: encoded video is 1088 lines, following “multiple of 32 lines” constraint

#10

Hello,

I checked and patch 11/19 is already applied in build #252.

So I confirm that I don’t have any frame out of the pipeline with /dev/video0 (Pipeline live but never returning a frame) or an error (/dev/video3):
sudo media-ctl -d /dev/media1 -l '"msm_csiphy0":1->"msm_csid0":0[1],"msm_csid0":1->"msm_ispif0":0[1],"msm_ispif0":1->"msm_vfe0_pix":0[1]'
and with following gstreamer test:
gst-launch-1.0 v4l2src device=/dev/video3 num-buffers=1 ! ‘video/x-raw,format=UYVY,width=1920,height=1080,framerate=30/1’ ! jpegenc ! filesink location=image01.jpg

Error with /dev/video3 is:

Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(2950): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn’t want to preroll.
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …

Pipeline is working with (and /dev/video0):
sudo media-ctl -d /dev/media1 -l '"msm_csiphy0":1->"msm_csid0":0[1],"msm_csid0":1->"msm_ispif0":0[1],"msm_ispif0":1->"msm_vfe0_rdi0":0[1]'

When using msm_vfe0_pix, here is my topology (sudo media-ctl -d /dev/media1 -p):

Media controller API version 0.1.0

Media device information

driver qcom-camss
model Qualcomm Camera Subsystem
serial
bus info
hw revision 0x0
driver version 0.0.0

Device topology

  • entity 1: msm_csiphy0 (2 pads, 3 links)
    type Node subtype V4L flags 0
    device node name /dev/v4l-subdev0
    pad0: Sink
    ← “ov5640 3-0078”:0 [ENABLED,IMMUTABLE]
    pad1: Source
    → “msm_csid0”:0 [ENABLED]
    → “msm_csid1”:0 []
  • entity 4: msm_csiphy1 (2 pads, 2 links)
    type Node subtype V4L flags 0
    device node name /dev/v4l-subdev1
    pad0: Sink
    pad1: Source
    → “msm_csid0”:0 []
    → “msm_csid1”:0 []
  • entity 7: msm_csid0 (2 pads, 4 links)
    type Node subtype V4L flags 0
    device node name /dev/v4l-subdev2
    pad0: Sink
    ← “msm_csiphy0”:1 [ENABLED]
    ← “msm_csiphy1”:1 []
    pad1: Source
    → “msm_ispif0”:0 [ENABLED]
    → “msm_ispif1”:0 []
  • entity 10: msm_csid1 (2 pads, 4 links)
    type Node subtype V4L flags 0
    device node name /dev/v4l-subdev3
    pad0: Sink
    ← “msm_csiphy0”:1 []
    ← “msm_csiphy1”:1 []
    pad1: Source
    → “msm_ispif0”:0 []
    → “msm_ispif1”:0 []
  • entity 13: msm_ispif0 (2 pads, 6 links)
    type Node subtype V4L flags 0
    device node name /dev/v4l-subdev4
    pad0: Sink
    ← “msm_csid0”:1 [ENABLED]
    ← “msm_csid1”:1 []
    pad1: Source
    → “msm_vfe0_rdi0”:0 []
    → “msm_vfe0_rdi1”:0 []
    → “msm_vfe0_rdi2”:0 []
    → “msm_vfe0_pix”:0 [ENABLED]
  • entity 16: msm_ispif1 (2 pads, 6 links)
    type Node subtype V4L flags 0
    device node name /dev/v4l-subdev5
    pad0: Sink
    ← “msm_csid0”:1 []
    ← “msm_csid1”:1 []
    pad1: Source
    → “msm_vfe0_rdi0”:0 []
    → “msm_vfe0_rdi1”:0 []
    → “msm_vfe0_rdi2”:0 []
    → “msm_vfe0_pix”:0 []
  • entity 19: msm_vfe0_rdi0 (2 pads, 3 links)
    type V4L2 subdev subtype Unknown flags 0
    device node name /dev/v4l-subdev6
    pad0: Sink
    [fmt:UYVY8_2X8/1920x1080 field:none]
    ← “msm_ispif0”:1 []
    ← “msm_ispif1”:1 []
    pad1: Source
    [fmt:UYVY8_2X8/1920x1080 field:none]
    → “msm_vfe0_video0”:0 [ENABLED,IMMUTABLE]
  • entity 22: msm_vfe0_video0 (1 pad, 1 link)
    type Node subtype V4L flags 0
    device node name /dev/video0
    pad0: Sink
    ← “msm_vfe0_rdi0”:1 [ENABLED,IMMUTABLE]
  • entity 28: msm_vfe0_rdi1 (2 pads, 3 links)
    type V4L2 subdev subtype Unknown flags 0
    device node name /dev/v4l-subdev7
    pad0: Sink
    [fmt:UYVY8_2X8/1920x1080 field:none]
    ← “msm_ispif0”:1 []
    ← “msm_ispif1”:1 []
    pad1: Source
    [fmt:UYVY8_2X8/1920x1080 field:none]
    → “msm_vfe0_video1”:0 [ENABLED,IMMUTABLE]
  • entity 31: msm_vfe0_video1 (1 pad, 1 link)
    type Node subtype V4L flags 0
    device node name /dev/video1
    pad0: Sink
    ← “msm_vfe0_rdi1”:1 [ENABLED,IMMUTABLE]
  • entity 37: msm_vfe0_rdi2 (2 pads, 3 links)
    type V4L2 subdev subtype Unknown flags 0
    device node name /dev/v4l-subdev8
    pad0: Sink
    [fmt:UYVY8_2X8/1920x1080 field:none]
    ← “msm_ispif0”:1 []
    ← “msm_ispif1”:1 []
    pad1: Source
    [fmt:UYVY8_2X8/1920x1080 field:none]
    → “msm_vfe0_video2”:0 [ENABLED,IMMUTABLE]
  • entity 40: msm_vfe0_video2 (1 pad, 1 link)
    type Node subtype V4L flags 0
    device node name /dev/video2
    pad0: Sink
    ← “msm_vfe0_rdi2”:1 [ENABLED,IMMUTABLE]
  • entity 46: msm_vfe0_pix (2 pads, 3 links)
    type V4L2 subdev subtype Unknown flags 0
    device node name /dev/v4l-subdev9
    pad0: Sink
    [fmt:UYVY8_2X8/1920x1080 field:none
    compose.bounds:(0,0)/1920x1080
    compose:(0,0)/1920x1080]
    ← “msm_ispif0”:1 [ENABLED]
    ← “msm_ispif1”:1 []
    pad1: Source
    [fmt:UYVY8_2X8/1920x1080 field:none
    crop.bounds:(0,0)/1920x1080
    crop:(0,0)/1920x1080]
    → “msm_vfe0_video3”:0 [ENABLED,IMMUTABLE]
  • entity 49: msm_vfe0_video3 (1 pad, 1 link)
    type Node subtype V4L flags 0
    device node name /dev/video3
    pad0: Sink
    ← “msm_vfe0_pix”:1 [ENABLED,IMMUTABLE]
  • entity 87: ov5640 3-0078 (1 pad, 1 link)
    type V4L2 subdev subtype Unknown flags 0
    device node name /dev/v4l-subdev10
    pad0: Source
    [fmt:UYVY8_2X8/1920x1080 field:none
    crop:(0,0)/1920x1080]
    → “msm_csiphy0”:0 [ENABLED,IMMUTABLE]

Let’s hope it will help understand the issue here.

#11

Using the following pipelines I was able to capture H.264 encoded video from the Venus video encoder @1280x960:
sudo media-ctl -d /dev/media1 -l ‘“msm_csiphy0”:1->“msm_csid0”:0[1],“msm_csid0”:1->“msm_ispif0”:0[1],“msm_ispif0”:1->“msm_vfe0_pix”:0[1]’
sudo media-ctl -d /dev/media1 -V ‘“ov5645 1-0076”:0[fmt:UYVY8_2X8/1280x960 field:none],“msm_csiphy0”:0[fmt:UYVY8_2X8/1280x960 field:none],“msm_csid0”:0[fmt:UYVY8_2X8/1280x960 field:none],“msm_ispif0”:0[fmt:UYVY8_2X8/1280x960 field:none],“msm_vfe0_pix”:0[fmt:UYVY8_2X8/1280x960 field:none],“msm_vfe0_pix”:1[fmt:UYVY8_1_5X8/1280x960 field:none]’

These two pipelines are in the wiki.

One question though - The video encode gstreamer pipeline references the hardware H.264 encoder element as “v4l2h264enc” but in my build of gstreamer1.0-plugins-good that was compiled using the 12 or so patches from the fnoop repo for the gstreamer plugins reports it as “v4l2video5h264enc”. Is there a newer patch that makes this encoder element generic? If there is then I’m really interested in that since it is kind of a pain when you need to write extra code to make sure that you’re using the correct pipeline elements.

For reference here’s the pipeline from the wiki:
gst-launch-1.0 -e v4l2src device=/dev/video3 ! video/x-raw,format=NV12,width=1280,height=960,framerate=30/1 ! v4l2h264enc extra-controls=“controls,h264_profile=4,video_bitrate=2000000;” ! h264parse ! mp4mux ! filesink location=video.mp4

#12

It appears that the latest versions of Gstreamer from the 1.11.x series supports our encoder natively without any additional patches, on top of that it also fixes the issue of our video encoder changing between /dev/videoX interfaces.
It does this by creating a new element for each codec such as v4l2h264enc for our qualcomm venus video encoder.

You can follow the steps required to build the latest and greatest gstreamer base and plugins by using the gst-build environment from here. simply git clone the repo to get started: https://github.com/GStreamer/gst-build

To obtain the necessary prerequisites for gstreamer simply run:
sudo apt-get build-dep gstreamer1.0 gstreamer1.0-plugins-good

You’ll also need to install the python-pip3 package manager, ninja, and meson using the commands below:
sudo apt-get install python3-pip python3-dev ninja-build libges-1.0 pip3 install --user meson export PATH=$PATH:~/.local/bin/

Finally you can build and install gstreamer using the gst-build environment by firing off the final commands:
cd gst-build/ mkdir build/ && meson build && ninja -C build/ sudo ninja -C build/ install

I’d recommend getting some coffee or lunch during the build/install process, it takes quite a while.

#13

yes, this is correct that the current dev branch for Gstreamer has these enhancements. The ‘stable’ element name was also implemented for the v4l2 decoder (Bug 784908 – v4l2videodec: Implement stable naming scheme for elements).

Note that these changes will be in gst 1.14 when it is released and they are current in 1.13 version (not 1.11).

We are actually bringing 1.12 version into our Debian images , that should be there next week. ‘even’ version numbers of Gstreamer are ‘stable’ version meant to be used by everyone, ‘odd’ number are devel version. So we won’t include devel version in our builds, but your instructions are correct to rebuild gst from scratch…