How to capture YVYU, YUYV and RGB888

shivashankar.k · December 23, 2020, 9:53am

Hi Team,

We have a carrier card with SD820 SOM, we have OV5645 camera sensor connected to the
csi-phy0 interface.

I am able to successfully capture UYVY format from the sensor and the media-ctl configuration is as below :

sudo media-ctl -d /dev/media0 -l ‘“msm_csiphy0”:1->“msm_csid0”:0[1],“msm_csid0”:1->“msm_ispif0”:0[1],“msm_ispif0”:1->“msm_vfe0_rdi0”:0[1]’

sudo media-ctl -d /dev/media0 ‘“ov5645 4-003c”:0[fmt:UYVY8_2X8/1920x1080 field:none],“msm_csiphy0”:0[fmt:UYVY8_2X8/1920x1080 field:none],“msm_csid0”:0[fmt:UYVY8_2X8/1920x1080 field:none],“msm_ispif0”:0[fmt:UYVY8_2X8/1920x1080 field:none],“msm_vfe0_rdi0”:0[fmt:UYVY8_2X8/1920x1080 field:none]’

gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=1 ! ‘video/x-raw,format=UYVY,width=1920,height=1080’ ! jpegenc ! filesink location=image01.jpg

Again, When I try to capture YUYV format image using the commands below, it is failing :

sudo media-ctl -d /dev/media0 -l ‘“msm_csiphy0”:1->“msm_csid0”:0[1],“msm_csid0”:1->“msm_ispif0”:0[1],“msm_ispif0”:1->“msm_vfe0_rdi0”:0[1]’

sudo media-ctl -d /dev/media0 ‘“ov5645 4-003c”:0[fmt:YUYV8_2X8/1920x1080 field:none],“msm_csiphy0”:0[fmt:YUYV8_2X8/1920x1080 field:none],“msm_csid0”:0[fmt:YUYV8_2X8/1920x1080 field:none],“msm_ispif0”:0[fmt:YUYV8_2X8/1920x1080 field:none],“msm_vfe0_rdi0”:0[fmt:YUYV8_2X8/1920x1080 field:none]’

gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=1 ! ‘video/x-raw,format=YUYV,width=1920,height=1080’ ! jpegenc ! filesink location=image01.jpg

Similarly for the YVYU format, it is failing with the below configurations and command :

sudo media-ctl -d /dev/media0 -l ‘“msm_csiphy0”:1->“msm_csid0”:0[1],“msm_csid0”:1->“msm_ispif0”:0[1],“msm_ispif0”:1->“msm_vfe0_rdi0”:0[1]’

sudo media-ctl -d /dev/media0 ‘“ov5645 4-003c”:0[fmt:YVYU8_2X8/1920x1080 field:none],“msm_csiphy0”:0[fmt:YVYU8_2X8/1920x1080 field:none],“msm_csid0”:0[fmt:YVYU8_2X8/1920x1080 field:none],“msm_ispif0”:0[fmt:YVYU8_2X8/1920x1080 field:none],“msm_vfe0_rdi0”:0[fmt:YVYU8_2X8/1920x1080 field:none]’

gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=1 ! ‘video/x-raw,format=YVYU,width=1920,height=1080’ ! jpegenc ! filesink location=image01.jpg

Kindly let me know if anything has to be changed in the commands to capture the image in
YUYV format and YVYU format. Also would be great if u tell how to capture RGB888 format too.

Regards,
Shiva Shankar K.

@Loic
@danielt

Loic · December 29, 2020, 5:08pm

Is the ov5645 supports YVYU8_2X8? the RDI interface (raw) only forward what is output from the sensor. You should be howver able to convert UYVY8_2X8 to YVYU8_2X8 in hardware going through the ‘pix’ interface.

AFAIR, this is what I’ve used with an ov5640:

media-ctl --reset 

media-ctl -d /dev/media0 -l '"msm_csiphy0":1->"msm_csid0":0[1],"msm_csid0":1->"msm_ispif0":0[1],"msm_ispif0":1->"msm_vfe0_pix":0[1]'

sudo media-ctl -d /dev/media0 -l '"msm_csiphy0":1->"msm_csid0":0[1],"msm_csid0":1->"msm_ispif0":0[1],"msm_ispif0":1->"msm_vfe0_pix":0[1]'
sudo media-ctl -d /dev/media0 -V '"ov5640 4-0076":0[fmt:UYVY8_2X8/640x480 field:none]'
sudo media-ctl -d /dev/media0 -V '"msm_csiphy0":0[fmt:UYVY8_2X8/640x480 field:none]'
sudo media-ctl -d /dev/media0 -V '"msm_csid0":0[fmt:UYVY8_2X8/640x480 field:none]'
sudo media-ctl -d /dev/media0 -V '"msm_ispif0":0[fmt:UYVY8_2X8/640x480 field:none]'
sudo media-ctl -d /dev/media0 -V '"msm_vfe0_pix":0[fmt:UYVY8_2X8/640x480 field:none]'
sudo media-ctl -d /dev/media0 -V '"msm_vfe0_pix":1[fmt:YUYV8_2X8/640x480 field:none]'

gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,format=NV16,width=640,height=480 ! videoconvert ! jpegenc ! filesink location=image_640x480.jpg

shivashankar.k · December 30, 2020, 10:41am

Yes, I have referred the same ov5640 data sheet for ov5645 sensor, I am not sure about the device ov5640 supporting the YVYU format although it is mentioned in the ov5640 datsheet page no. 86 .

Despite changing the sensor to generate YVYU by changing the 0x4300 register value to 0x31, I am still using the UYVY format in the media-ctl configuration to capture the image, still I can capture the image but with more green component. The capture fails only when I change the media-ctl configuration to the exact format YVYU.

Kindly let me know at least to capture RGB888 format.

Regards,
Shiva Shankar K.

shivashankar.k · January 4, 2021, 9:59am

Hi Team,

Kindly let me know how to capture the RGB888 format.

Regards,
Shiva Shankar K.

Loic · January 5, 2021, 4:54pm

have you tried similar pipeline to me with hardware conversion using the pix interface?

shivashankar.k · January 5, 2021, 6:43pm

Yes, I did try your suggested commands above but it failed, I will share the log tomorrow. Our requirement now is to capture RGB888 so I am focusing on RGB88 capture.

shivashankar.k · January 6, 2021, 8:00am

Please find the log when I tried for hardware conversion using the pix interface.
It is getting stuck after “New clock: GstSystemClock”

root@820-devkit:/home/mistral/Desktop/uyvy# media-ctl --reset
root@820-devkit:/home/mistral/Desktop/uyvy# media-ctl -d /dev/media0 -l ‘“msm_csiphy0”:1->“msm_csid0”:0[1],“msm_csid0”:1->“msm_ispif0”:0[1],“msm_ispif0”:1->“msm_vfe0_pix”:0[1]’
root@820-devkit:/home/mistral/Desktop/uyvy# sudo media-ctl -d /dev/media0 -l ‘“msm_csiphy0”:1->“msm_csid0”:0[1],“msm_csid0”:1->“msm_ispif0”:0[1],“msm_ispif0”:1->“msm_vfe0_pix”:0[1]’

root@820-devkit:/home/mistral/Desktop/uyvy# sudo media-ctl -d /dev/media0 -V ‘“ov5645 5-003c”:0[fmt:UYVY8_2X8/1920x1080 field:none]’
root@820-devkit:/home/mistral/Desktop/uyvy# sudo media-ctl -d /dev/media0 -V ‘“msm_csiphy0”:0[fmt:UYVY8_2X8/1920x1080 field:none]’
root@820-devkit:/home/mistral/Desktop/uyvy# sudo media-ctl -d /dev/media0 -V ‘“msm_csid0”:0[fmt:UYVY8_2X8/1920x1080 field:none]’
root@820-devkit:/home/mistral/Desktop/uyvy# sudo media-ctl -d /dev/media0 -V ‘“msm_ispif0”:0[fmt:UYVY8_2X8/1920x1080 field:none]’
root@820-devkit:/home/mistral/Desktop/uyvy# sudo media-ctl -d /dev/media0 -V ‘“msm_vfe0_pix”:0[fmt:UYVY8_2X8/1920x1080 field:none]’
root@820-devkit:/home/mistral/Desktop/uyvy# [ 1036.078829] IPv6: ADDRCONF(NETDEV_UP): wlp1s0: link is not ready
sudo media-ctl -d /dev/media0 -V ‘“msm_vfe0_pix”:1[fmt:YUYV8_2X8/1920x1080 field:none]’
root@820-devkit:/home/mistral/Desktop/uyvy# gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,format=NV16,width=1920,height=1080 ! videoconvert ! jpegenc ! filesink location=image_640x480.jpg
[ 1044.699941] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1044.702471] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1044.708532] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1044.868069] qcom-camss a34000.camss: CSIPHY 3PH HW Version = 0x10000000
[ 1044.868709] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1044.876793] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1044.880418] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1044.886286] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1044.892253] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1045.216267] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1045.217648] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1045.223940] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1045.385340] qcom-camss a34000.camss: CSIPHY 3PH HW Version = 0x10000000
[ 1045.385845] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1045.393813] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1045.397593] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1045.403340] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1045.409496] qcom-camss a34000.camss: VFE HW Version = 0x70020000
Setting pipeline to PAUSED …
[ 1045.610154] qcom-camss a34000.camss: CSIPHY 3PH HW Version = 0x10000000
[ 1045.610616] qcom-camss a34000.camss: VFE HW Version = 0x70020000
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock

shivashankar.k · January 6, 2021, 10:07pm

Hi Loic,

Please let me know if you need any thing further.

Regards,
Shiva Shankar K.

@Loic
@danielt

Loic · January 13, 2021, 3:59pm

The sensor does not seem to support this format, and the driver only support UYVY8_2X8 so you’ll have to convert in software.

shivashankar.k · January 15, 2021, 9:53am

Hi Loic,

Thanks for your response.

I am a bit surprised to know this, anyway please tell me if we can capture images in YUYV or YVYU format from ov5645 sensor, because when I tried these two formats, I have gotten more of greenish images without any pipeline errors.

I could not attach the images here for your reference.

Regards,
Shiva Shankar K.

shivashankar.k · February 3, 2021, 5:34am

Hi Loic,

At least you can tell us whether the same driver can be used to capture YUYV or YVYU format images, or not?

Regards,
Shiva Shankar K.

Loic · February 3, 2021, 8:53am

The ov5645 driver support UYVY only:

github.com

torvalds/linux/blob/master/drivers/media/i2c/ov5645.c#L846

    
      
          	.s_ctrl = ov5645_s_ctrl,
          };
          
          
static int ov5645_enum_mbus_code(struct v4l2_subdev *sd,
          				 struct v4l2_subdev_state *sd_state,
          				 struct v4l2_subdev_mbus_code_enum *code)
          {
          	if (code->index > 0)
          		return -EINVAL;
          
          
	code->code = MEDIA_BUS_FMT_UYVY8_2X8;
          
          
	return 0;
          }
          
          
static int ov5645_enum_frame_size(struct v4l2_subdev *subdev,
          				  struct v4l2_subdev_state *sd_state,
          				  struct v4l2_subdev_frame_size_enum *fse)
          {
          	if (fse->code != MEDIA_BUS_FMT_UYVY8_2X8)
          		return -EINVAL;

AFAIK via the pix interface, you should be able to convert that to YVYU. But I’ve tested that long time ago, so maybe @robertfoss knows more about this.

shivashankar.k · February 3, 2021, 12:14pm

Robert,

Earlier, I tried to change the driver code by replacing MEDIA_BUS_FMT_UYVY8_2X8 macro by
MEDIA_BUS_FMT_YVYU8_2X8 macro wherever it is present but still it fails.

Could you please tell us what are the changes required to make it working for YVYU format?

@Loic
@robertfoss

robertfoss · February 4, 2021, 10:46am

Hey Loic & Shankar,

I haven’t exercised the format conversion paths personally, and don’t have a lot of advice to give about it.

But updating the ov5645 driver to produce MEDIA_BUS_FMT_YVYU8_2X8 seems like it should work. And it not working seems to indicate some sort of a bug.

I don’t have access to the ov5645 datasheet, but from the product brief it seems to not support RGB888.

ov5645 product brief

The only reason you want to use YVYU8_2X8 as opposed to UYVY8_2X8 is the green tint? If so maybe we can focus on that. Once that you can do the YUV->RGB888 conversion however you like (either by configuring camss correctly or doing it in userspace).

As for the green tint, that does indeed sound like some color channels have been flipped.

shivashankar.k · February 4, 2021, 11:25am

Robert,

Let me rephrase my question, what it takes to add the support for YVYU or YUYV format in general in any existing driver but in our case if we take ov5645 driver to add this support.

Then again, green tint was present in the captured images, when I made the below changes :
a) register settings in sensor is modified for YUYV
b) all the macros in the driver code was retained for UYVY
c) the media-ctl configs are also for UYVY
d) gst-launch command executed for UYVY and then we captured greenish image

If I try to modify all the above things (steps) for YUYV, I get some error. It was during my experiments I found that way to capture greenish image, it may seem a bit awkward and absurd by the way I experimented.

Regards,
Shiva Shankar K.

@robertfoss
@Loic

robertfoss · February 4, 2021, 6:10pm

Thanks for clearing up the current status. Seeing a green tint after A&B&C&D is to be expected, as you are misrepresenting the data the sensor is outputting. So I’d say this is 100% functional.

Additionally the pipeline full UYVY pipeline from the first post seem to also be 100% functional.

If the ISP is configured for YVYU / YUYV and fails for those modes only, I’d say we’re having some issue with camss driver.

But maybe we’re losing track of the goal here. The ISP of the db820c doesn’t support outputting a plain RGB format, only Bayer RGB formats which I suspect isn’t what you want, because then you would have to do the debayering yourself which is pretty terrible stuff.

github.com

torvalds/linux/blob/master/drivers/media/platform/qcom/camss/camss-ispif.c#L128

    
      
          	MEDIA_BUS_FMT_SGBRG10_1X10,
          	MEDIA_BUS_FMT_SGRBG10_1X10,
          	MEDIA_BUS_FMT_SRGGB10_1X10,
          	MEDIA_BUS_FMT_SBGGR12_1X12,
          	MEDIA_BUS_FMT_SGBRG12_1X12,
          	MEDIA_BUS_FMT_SGRBG12_1X12,
          	MEDIA_BUS_FMT_SRGGB12_1X12,
          	MEDIA_BUS_FMT_Y10_1X10,
          };
          
          
static const u32 ispif_formats_8x96[] = {
          	MEDIA_BUS_FMT_UYVY8_2X8,
          	MEDIA_BUS_FMT_VYUY8_2X8,
          	MEDIA_BUS_FMT_YUYV8_2X8,
          	MEDIA_BUS_FMT_YVYU8_2X8,
          	MEDIA_BUS_FMT_SBGGR8_1X8,
          	MEDIA_BUS_FMT_SGBRG8_1X8,
          	MEDIA_BUS_FMT_SGRBG8_1X8,
          	MEDIA_BUS_FMT_SRGGB8_1X8,
          	MEDIA_BUS_FMT_SBGGR10_1X10,
          	MEDIA_BUS_FMT_SGBRG10_1X10,

With that in mind, is there any point to pick an output format other than UYVY?

shivashankar.k · February 5, 2021, 7:05am

Now I feel we both are in the same line of understanding.

We are already trying with the format other than UYVY i.e., YUYV and YVYU formats.
But, I think you are just trying to ask us to pick an output format other than YUV422( UYVY, YVYU, YUYV) i.e., SBGGR8, SGBRG8, and SGRBG8 etc.

Regards,
Shiva Shankar K.