Venus encoder I-frame periodicity

[danielq]

Hi,

I cannot change IDR/I-frame periodicity through extra-controls parameters (gop and i_frame_period).

Related:

Setting codec parameters on DB820C

Having done some more digging, it appears that the video_gop_size parameter is actually getting through to the Venus driver in the kernel. The Venus encoder is generating I-frames at the requested interval but these are not IDR frames. /* IDR periodicity, n: n = 0 - only the first I-frame is IDR frame n = 1 - all I-frames will be IDR frames n > 1 - every n-th I-frame will be IDR frame */ ptype = HFI_PROPERTY_CONFIG_VENC_IDR_PERIOD; idrp.idr_period = 0; Setting this to 1 makes all I-fram…

I’ve been checking the upstream venus driver and looks more up-to-date than the one available in landing-teams (linaro git). But I couldn’t find any fix for this.

release/qcomlt-4.14:
https://git.linaro.org/landing-teams/working/qualcomm/kernel.git/tree/drivers/media/platform/qcom/venus/venc.c?h=release/qcomlt-4.14#n715

Mainline:
https://elixir.bootlin.com/linux/v5.3-rc8/source/drivers/media/platform/qcom/venus/venc.c#L702

The issue is about getting key_frames with V4L2 controls: _GOP_SIZE and _I_FRAME_PERIOD

• encoder pipeline snippet:

v4l2h264enc \
extra-controls="controls,h264_profile=4,video_bitrate_mode=0,video_peak_bitrate=25000000,h264_i_frame_period=3,h264_i_frame_period=3,video_gop_size=3;" ! \

Is the pipeline correct? Why I’m only getting the first key_frame as I-frame while I’m setting i_frame_period to 3? Not sure if I’m missing an extra control or need a fix for it.

[Loic]

@danielq

I cannot change IDR/I-frame periodicity through extra-controls parameters ( gop and i_frame_period ).

AFAIK, there is no v4l2 control to set IDR periodicity, gop and I are both used to configure I-frame interval, the venus driver use only gop interval to calculate B/P/I intervals. The venus encoder is statically configured to generate only one IDR frame (the first one):

/* IDR periodicity, n:
 * n = 0 - only the first I-frame is IDR frame
 * n = 1 - all I-frames will be IDR frames
 * n > 1 - every n-th I-frame will be IDR frame
 */
ptype = HFI_PROPERTY_CONFIG_VENC_IDR_PERIOD;
idrp.idr_period = 0;
ret = hfi_session_set_property(inst, ptype, &idrp);
if (ret)
	return ret;

So you can change n to something else and even create the missing v4l2 control…

[danielq]

Thanks for your replay @Loic,

I’ve been testing the driver with idrp.idr_period = N (N>=1) which makes an IDR period of N and I-frame interval is then controlled by video_gop_size.

But according to the linuxtv documentation this value V4L2_CID_MPEG_VIDEO_H264_I_PERIOD stands for the ‘period between I-frames in the open GOP for H264’. Which in our case is controlled by video_gop_size. In addition, the description says: The period between IDR (Instantaneous Decoding Refresh) frames is taken from the GOP_SIZE control which in our case is controlled statically with idrp.idr_period.

My assumption then, is that the driver needs to be modified to be able to control IDR period through this V4L2 ID control: V4L2_CID_MPEG_VIDEO_GOP_SIZE (video_gop_size) and it also needs to control I-frame period through V4L2_CID_MPEG_VIDEO_H264_I_PERIOD (h264_i_frame_period).

If that is correct, I’d glad to help with the code/testing or whatever is needed to do.

[Loic]

You’re correct, a patch is needed, I 'll let you know once I have something to test.

[Loic]

You can try this patch: loic.poulain/linux.git - [no description]

It fixes gop size (IDR period) and I frame interval.

[danielq]

Thanks @Loic, I’ll test it later and come back with some results.

[svarbanov]

It’d be nice if you can test also num_b_frames and gob size, because @Loic changed the calculations. I haven’t go deeply but there is a reason venc_calc_bpframes() to be so complicated (the firmware will not accept or not doing what you are expecting when set number of B-frames and GOP size).

[Loic]

@svarbanov, you’re right, It was mainly to dix the IDR/I frames, but the b_frame support does not work as expected (only b_frames=1 works), so the patch is incomplete. For now I have not been able to find what the firmware expects.

[svarbanov]

@Loic ok it is fine it that case When you said that it works only bframes=1, is that with your fix or without it?

[danielq]

Hi @Loic and @svarbanov,

I’m sorry for the delay. I’ve tested the driver and this is what happens:

• h264_i_frame_period=2 and video_gop_size=10

When I set h264_i_frame_period=2 and video_gop_size=10, I have a key_frame every 10 frames while I think it should produce every 5 according to the @Loic patch. Actually, I’ve added some debug info and this is what it says:

qcom-venus-encoder c00000.video-codec:video-encoder: 742: idrp.idr_period: 5
qcom-venus-encoder c00000.video-codec:video-encoder: 742: idrp.idr_period: 5
qcom-venus-encoder c00000.video-codec:video-encoder: 742: idrp.idr_period: 5

• h264_i_frame_period=1 and video_gop_size=10

Then, when I set h264_i_frame_period=1 and video_gop_size=10, I have again a key_frame every 10 frames. But in this case, the driver is setting and idr period of 10:

qcom-venus-encoder c00000.video-codec:video-encoder: 742: idrp.idr_period: 10
qcom-venus-encoder c00000.video-codec:video-encoder: 742: idrp.idr_period: 10
qcom-venus-encoder c00000.video-codec:video-encoder: 742: idrp.idr_period: 10

So, I guess the idr_period value is not having and impact on the I-frame periodicity.

In addition, I think this was also happening before the patch. I mean, when you had to set the idr period in the driver, the only values working where 0 (for having only the first frame as key frame) and 1.

[Loic]

Well, with h264_i_frame_period=2 and video_gop_size=10:

gst-launch-1.0 -ve videotestsrc ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 ! v4l2h264enc extra-controls="controls,h264_profile=4,video_bitrate=8000000,h264_i_frame_period=2,video_gop_size=10;" ! h264parse ! mp4mux ! filesink location=test.mp4

I get:

GOP: IP 2 CLOSED
GOP: iP 2 OPEN
GOP: iP 2 OPEN
GOP: iP 2 OPEN
GOP: iP 2 OPEN
GOP: IP 2 CLOSED
GOP: iP 2 OPEN
GOP: iP 2 OPEN
GOP: iP 2 OPEN
GOP: iP 2 OPEN
GOP: IP 2 CLOSED
GOP: iP 2 OPEN

Which give a IDR frame every ten frame (closed gop) and a standard I frame every 2 frames (open gop)., is it not what you expect?

[danielq]

Hi @Loic,

You are right. I was looking just for the key_frames and not the type of frame:

So, this is how I’m debugging now for h264_i_frame_period=2 and video_gop_size=10:

ffprobe -show_frames test.mp4 | grep "pict_type\|key_frame\|coded_picture_numb"

key_frame=1
pict_type=I
coded_picture_number=0
key_frame=0
pict_type=P
coded_picture_number=1
key_frame=0
pict_type=I
coded_picture_number=2
key_frame=0
pict_type=P
coded_picture_number=3
key_frame=0
pict_type=I
coded_picture_number=4
key_frame=0
pict_type=P
coded_picture_number=5
key_frame=0
pict_type=I
coded_picture_number=6
key_frame=0
pict_type=P
coded_picture_number=7
key_frame=0
pict_type=I
coded_picture_number=8
key_frame=0
pict_type=P
coded_picture_number=9
key_frame=1
pict_type=I
coded_picture_number=10

BTW, how did you produce the above output?

Thanks!

[Loic]

@danielq

BTW, how did you produce the above output?

I run this script: https://gist.github.com/use-sparingly/7041ee993adb5c911f90