Gstreamer issue of DragonBoard 410c

Products Support DragonBoard410c
#1

Hi,This is a big issue of Gstreamer.

The playback speed is slower.
The playback speed of Full HD MPEG4 is slower( x0.5)on Linaro LInux 16.09(debian)
And also HD MPEG4 is slower( x0.9).

The command

gst-launch-1.0 filesrc location=DSC_2177.mp4 ! progressreport ! \
qtdemux name=demux
demux.video_0 ! queue ! h264parse ! v4l2dec ! glimagesink

I suppose the reason is socket communication to Xserver
Because the data is processed at the same speed by the following command.

gst-launch-1.0 filesrc location=DSC_2177.mp4 ! progressreport ! \
qtdemux name=demux
demux.video_0 ! queue ! h264parse ! v4l2dec ! fakesink sync=true

I suppose the solution is directly drawing MPEG4data on GPU.
Linaro has roadmap like this development?

#2

hi,

can you please clarify what you mean by ‘slower (x0.5) and slower (x0.9)’? Slower than what? there is at least one limitation in the 16.09 release regarding video playback that we are aware, but I would like to better understand your post first.

#3

Hi,ndec. Thank you for reply.

I made a Full HD Video(MPEG4) with a length 12 sec by Android phone.

The playback speed is 24sec on DB410 when I use first command of previous comment.
So ×0.5 is 12/24sec.

Next,I made a HD Video(MPEG4) with a length 81 sec by Android phone.

The playback speed is 91sec on DB410 when I use first command of previous comment.
So ×0.9 is 81/91sec

When I playback it on Windws PC and Android smartphone,each times are 12sec and 81 sec.

Thank you!

#4

Hi,

Slowly playback is here

Please look at timer counter.
DB410(Right display) timer is slower than iPhone timer.
*After a few days,I will remove it.

Thank you!

#5

hi,

thanks for the interesting details… in 16.09 the video playback pipeline was not fully optimized and i remember we had performance issues (frames dropped) especially with 1080p videos. The pipeline has gotten much better since 16.09, but we haven’t made a release yet.

I am travelling this week, and can’t spend a lot of time on this topic, but I will try to provide you a test build , hopefully next week with all the improvements. With the most recent kernel driver and Gstreamer plugins we can play 1080p content with ~10% cpu usage, most the overhead is gone.

#6

Hi,

Thank you for detail information and your support.

I have understood current status and next step of this issue.
Please provide test build when you finish preparation.If possible,please tell me about provide date.If it is next week first,it’s great help.

And please tell me about release plan of most resent kernel driver and Gstreamer plugins.
We want to develop final products with its.
At that time,we would like to use release version.

Kind regards.

#7

The reason is that initially the customer was going to finish the evaluation to decide using DragonBoard/Snapdragon for their product or not at the end of this week but we would like to negotiate with the customer to prevent to lose the Qualcomm IoT business in Japan, if the media playback speed is going to be improved.
Is it OK to teach me when in next week might be the test build ready with all the improvements?
Thank you for your help.

#8

hi @hase

So we have been looking into this issue, finally. I don’t have a perfect answer yet… but I wanted to share some updates.

First of all I can reproduce the issue. It is definitely a performance issue in the video pipeline, related to extraneous buffers copy in the pipeline. We did some improvements in the video pipeline since the 16.09 release and with a recent build I can play the videos quite well, which is a good news.

The video (v4l2dec) and the X11/GPU rendering (glimagesink) plugins are capable to handle dmabuf import/export (e.g. full zero copy of the buffers), however Gstreamer playbin/decodebin won’t be able to use dmabuf automatically, as of yet. So any pipeline (or Gst apps) that just relies on playbin will suffer from this performance hit.

If I use a fully optimized pipeline, i was able to play (under X11) the video you provided (off line).

I have been using the following pipeline:

GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 filesrc location=/tmp/sa0026.mp4 ! qtdemux name=m m.video_0 ! h264parse ! v4l2video0dec capture-io-mode=dmabuf ! glimagesink

This is the most opimized pipeline for our platform at this point.

If you omit catpure-io-mode on the v4l2dec plugin , it won’t use dmabuf between the output of the video decoder and the x11 sink. The EGL/GLES backend in glimagesink is the only path where dmabuf is implemented. With the fully optimized pipeline i can play your video with ~4% cpu utilization which looks correct.

I have been doing all my experiments with this build:

http://builds.96boards.org/snapshots/dragonboard410c/linaro/debian/202/

It is a recent build using a more recent version of Debian, which we will release soon. It is using 4.4 based kernel, but it has a more recent version of the video driver compared to the one you had in 16.09 release.

A few more things to be aware:

  • once you have flashed this build, please do not upgrade. It uses Gstreamer 1.10.2 and currently in Debian we have 1.10.4 which does not work right now (this is a separate issue, which we are looking into).
  • Gst 1.10.2 has a bug (which was fixed in 1.10.3), the workaround is to set
    export GST_GL_XINITTHREADS=1
    before running any gstreamer pipeline

Note that using the complete Debian desktop will not give you the ‘best overall performance’ since the desktop itself is using CPU and GPU resources. While this might be ok for demonstration/evaluation, using a plain vanilla is often not what you want for your product. We don’t know what your product really is… but if you want to get into a minimal graphical environment and get rid of any desktop visual effects…

In the coming days/weeks , we will publish builds based on 4.9 kernel, and I expect it will have the same level of performance as in this 202 build, but all issues mentioned above will be fixed.

Let us know how things go with this new build and set of instructions.

cheers

#10

Hi @anon91830841

Thank you detail information and the way of solution.

I check fix of this issue.
Some time,playback function lose frames.
But I think this issue is fixed on next release. Isn’t it?

And I have a question.
Does Gstreamer 1.10.2 have encode function?
I can’t look “v4l2enc”

If it have encode function like a “v4l2”,could you please teach me about it.

Note that using the complete Debian desktop will not give you the ‘best overall performance’ since the desktop itself is using CPU and GPU resources. While this might be ok for demonstration/evaluation, using a plain vanilla is often not what you want for your product. We don’t know what your product really is… but if you want to get into a minimal graphical environment and get rid of any desktop visual effects…

Thank you for your valuable comments
I think it more.

Best regards

#11

And if I can clear this issue,I will try transcoding.

#12

hi,

no, there is no v4l2 ‘enc’ in Gstreamer yet. There are patches available on the Gstreamer bugzilla right now, as the plugin is being developed, but it’s not ready/finished yet.

#13

Hi @anon91830841

There are patches available on the Gstreamer bugzilla right now, as the plugin is being developed.

Could you please provide link its patches?
I can’t search its…

Best regards,

#14

All the information is here:

https://bugzilla.gnome.org/show_bug.cgi?id=728438

Please note that this is work in progress, and we will not provide any support until this is finalized.

cheers

#15

Hi @anon91830841

Thank you for quick reply.
I understand that it is no support.
I hope encode function work on DB410…

Best regards,

#16

Hi @anon91830841

I have read bugzilla and I have question.

In bugzilla,Todor said,

I can report using successfully this video encoder plugin too on Dragonboard 410C (Qualcomm APQ8016). I’m running Debian with Gstreamer 1.10.2, I have applied the 12 patches from:
https://github.com/fnoop/gst-plugins-good/commits/master

If we can applied the 12 patches,is there any possibility that we will make encorder plugin successful?
If it is so,could you please teach me about how to the 12 patches.

Best regards,
Hiroshi

#17

Hi,

I am grateful for all your help and support till now.

I could not work encode and trancecode function well.

As reference I explain my task.
Using build image is build 202 of Linaro Linux.

MPEG4(H264) decode test

  1. MPEG4 file decode(display,work HW accelerator)
    To set the environment variable
    export GST_GL_PLATFORM=egl
    export GST_GL_API=gles2
    export GST_GL_XINITTHREADS=1
    Gstreamer pipeline
    gst-launch-1.0 filesrc location=test.mp4 ! progressreport ! \
    qtdemux name=demux
    demux.video_0 ! queue ! h264parse ! v4l2video0videodec capture-io-mode=dmabuf ! glimagesink

  2. MPEG4 file decode(benchmark,work HW accelerator)
    gst-launch-1.0 filesrc location=DSC_2177.mp4 ! progressreport ! \
    qtdemux name=demux
    demux.video_0 ! queue ! h264parse ! v4l2video0videodec capture-io-mode=dmabuf ! fakesink sync=false

  3. MPEG4 file decode(benchmark,work HW accelerator)
    gst-launch-1.0 filesrc location=DSC_2177.mp4 ! progressreport ! \
    qtdemux name=demux
    demux.video_0 ! queue ! avdec_h264 ! fakesink sync=false

In 2 case,performance is 70fps and CPU 10% at FHD(1080p)
In 3 case,performance is 37fps and CPU 400% at FHD(1080p)

This performance is so good to me.

MPEG4(H264) trancecode test
*We use patch of following link to this test.
https://bugzilla.gnome.org/show_bug.cgi?id=778974

Input MPEG4file->change picture size and output other H.264 file.

gst-launch-1.0 -v filesrc location=DSC_2177.mp4 ! progressreport ! qtdemux name=stream
stream.video_0 ! queue ! h264parse ! avdec_h264 !
videoscale ! “video/x-raw,width=640,height=480” !
videoconvert ! “video/x-raw,format=NV12” !
v4l2video1h264enc extra-controls=“controls,h264_profile=4,video_bitrate=4000000;” !
filesink location=trans-scale.h264 sync=false

I realy tried to work HW encoder and HW decoder.
But now,its function is not work(Gstreamer’s bug?).So i tried to work SW decoder and HW encoder.

As this result,it is still slow.But i think it has improved at 10% than current build image.

If I mistake the way of this test,could you please teach me the right way.

We think this issue is “videoconvert” of Gstreamer plugin.Isn’t it?

If you have the schedule of fixed this issue,could you please teach me it.

Best regards,

#18

hi @hase

we haven’t tried any video transcode use case yet, so I cannot confirm whether or not it is working…

In your pipeline, videoscale and videoconvert are definitely going to need a lot of CPU. You can try simple pipeline using videotestsrc instead encoder/decoder to verify the ‘cost’ of videoscale and videconvert alone.

Once we enable hw video encoder and verify transcoding use cases, you should get good performances if you only change bitrate, codecs, … but as soon as you try to do video ‘processing’ (such as scaling) performance will be terrible since it will be done in s/w.

#19

@anon91830841

I am also trying to get video transcoding working on the DB410c. I have built gstreamer-1.11.91(base, good plugins, bad plugins, and rtspserver).

I have a plugin for v4l2video33h264enc but when I try to use with this pipeline:
gst-launch-1.0 -e v4l2src device=/dev/video0 ! video/x-raw,width=640,height=480,framerate=30/1 ! videoconvert ! video/x-raw,format=NV12,width=640,height=480,framerate=30/1 ! x264enc ! h264parse ! mp4mux ! queue ! filesink location=raw.mp4 sync=true

It fails to encode with the following errors:
Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Setting pipeline to PLAYING ... New clock: GstSystemClock ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error. Additional debug info: gstbasesrc.c(2939): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: streaming stopped, reason not-negotiated (-4) EOS on shutdown enabled -- waiting for EOS after Error Waiting for EOS... ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error. Additional debug info: gstbasesrc.c(2939): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: streaming stopped, reason not-negotiated (-4) ^Chandling interrupt. Interrupt: Stopping pipeline ... Interrupt while waiting for EOS - stopping pipeline... Execution ended after 0:00:07.391124201 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ...

Any ideas that I can try?

#20

hi,

this is not really ‘transcoding’. What I meant by transcoding is mp4 → decoder → encoder → h264, for example. Or same codec with a different quality, …

What if /dev/video0 in our pipeline, is that a CSI camera or USB camera?

Is your pipeline/application working with ‘just’ the camera (e.g. if you replace the video encoder with a rendering sink, does it work)?

#21

@anon91830841

Sorry for using the wrong terminology, this is really better described as “encoding” rather than “transcoding”.

In my pipeline /dev/video0 is indeed a MIPI-CSI camera and it seems to be working since this pipeline produces a valid JPEG image from the camera:
gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=1 ! 'video/x-raw,format=UYVY,width=1920,height=1080,framerate=30/1' ! jpegenc ! filesink location=image01.jpg

I have yet to try the software encoder, but I will as soon as I can. I am currently recompiling the gstreamer-plugins-good set of plugins since the version I used was a bit older than the other gstreamer plugins and base.

I was using the code off of this repo, as it had been mentioned by Todor in an earlier post: