IMX412 driver troubleshooting

Hello,

I’m trying to connect Sony IMX412 camera to the board using MIPI interface and following driver with slight adjustments for it to work with older linux kernel version (4.14.69).

dts
i2c@075ba000{
/* On Low speed expansion */
        label = "LS-I2C3";
        status = "okay";

        camera@1a {
                compatible = "sony,imx412";
                reg = <0x1a>;
                clocks = <&mmcc CAMSS_MCLK0_CLK>;

                assigned-clocks = <&mmcc CAMSS_MCLK0_CLK>;
                assigned-clocks-parent = <&mmcc>;
                assigned-clocks-rate = <24000000>;

                enable-gpios = <&msmgpio 26 GPIO_ACTIVE_HIGH>;
                reset-gpios = <&msmgpio 25 GPIO_ACTIVE_LOW>;
                pinctrl-names = "default";
                pinctrl-0 = <&camera_rear_default>;

                vdddo-supply = <&camera_vdddo_1v8>;
                vdda-supply = <&camera_vdda_2v8>;
                vddd-supply = <&camera_vddd_1v5>;

                status = "okay";

                port {
                        imx412_ep: endpoint {
                                remote-endpoint = <&csiphy0_ep>;
                                data-lanes = <1 2 3 4>;
                                link-frequencies = /bits/ 64 <600000000>;
                        };
                };
        };
};

cci@a0c000 {
        status = "ok";

        camss@a00000 {
                status = "ok";

                ports {
                        #address-cells = <1>;
                        #size-cells = <0>;
                        port@0 {
                                reg = <0>;
                                csiphy0_ep: endpoint {
                                        data-lanes = <1 2 3 4>;
                                        remote-endpoint = <&imx412_ep>;
                                        status = "okay";
                                };
                        };
                };
        };
};

Things that work:

  • i2cdetect sees camera on the bus (however i2cdump shows all zeros)
  • media device appears and I can build a pipeline
  • driver can succesfully probe camera (albeit it reads different device id than specified by default)

What I want to achieve: capture data from camera using either gstreamer of v4l2-ctl

Currently v4l2-compliance -s fails blocking wait and MMAP tests.

v4l2-compliance -s
v4l2-compliance SHA: not available, 64 bits

Compliance test for device /dev/video0:

Driver Info:
	Driver name      : qcom-camss
	Card type        : Qualcomm Camera Subsystem
	Bus info         : platform:a34000.camss
	Driver version   : 4.14.69
	Capabilities     : 0x85201000
		Video Capture Multiplanar
		Read/Write
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x05201000
		Video Capture Multiplanar
		Read/Write
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : qcom-camss
	Model            : Qualcomm Camera Subsystem
	Serial           : 
	Bus info         : 
	Media version    : 4.14.69
	Hardware revision: 0x00000000 (0)
	Driver version   : 4.14.69
Interface Info:
	ID               : 0x03000027
	Type             : V4L Video
Entity Info:
	ID               : 0x00000025 (37)
	Name             : msm_vfe0_video0
	Function         : V4L2 I/O
	Pad 0x01000026   : Sink
	  Link 0x02000029: from remote pad 0x1000024 of entity 'msm_vfe0_rdi0': Data, Enabled, Immutable

Required ioctls:
	test MC information (see 'Media Driver Info' above): OK
	test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
	test second /dev/video0 open: OK
	test VIDIOC_QUERYCAP: OK
	test VIDIOC_G/S_PRIORITY: OK
	test for unlimited opens: OK

Debug ioctls:
	test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
	test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
	test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
	test VIDIOC_ENUMAUDIO: OK (Not Supported)
	test VIDIOC_G/S/ENUMINPUT: OK
	test VIDIOC_G/S_AUDIO: OK (Not Supported)
	Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
	test VIDIOC_G/S_MODULATOR: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_ENUMAUDOUT: OK (Not Supported)
	test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
	test VIDIOC_G/S_AUDOUT: OK (Not Supported)
	Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
	test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
	test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
	test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
	test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
	test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
	test VIDIOC_QUERYCTRL: OK (Not Supported)
	test VIDIOC_G/S_CTRL: OK (Not Supported)
	test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
	test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
	test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
	Standard Controls: 0 Private Controls: 0

Format ioctls (Input 0):
	test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
	test VIDIOC_G/S_PARM: OK (Not Supported)
	test VIDIOC_G_FBUF: OK (Not Supported)
	test VIDIOC_G_FMT: OK
	test VIDIOC_TRY_FMT: OK
	test VIDIOC_S_FMT: OK
	test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
	test Cropping: OK (Not Supported)
	test Composing: OK (Not Supported)
	test Scaling: OK

Codec ioctls (Input 0):
	test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
	test VIDIOC_G_ENC_INDEX: OK (Not Supported)
	test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
	test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
	test VIDIOC_EXPBUF: OK

Test input 0:

Streaming ioctls:
	test read/write: OK
		fail: v4l2-test-buffers.cpp(1245): pid != pid_streamoff
		fail: v4l2-test-buffers.cpp(1278): testBlockingDQBuf(node, q)
	test blocking wait: FAIL
		fail: v4l2-test-buffers.cpp(885): can_stream
	test MMAP: FAIL
	test USERPTR: OK (Not Supported)
	test DMABUF: Cannot test, specify --expbuf-device

Total: 48, Succeeded: 46, Failed: 2, Warnings: 0

I’ve also noticed that driver’s s_stream operation is never called, which means (from my understanding) that for some reason it never starts streaming. Trying to use something like v4l2-ctl -d 0 --stream-mmap --set-fmt-video=width=4056,height=3040,pixelformat=pBAA --stream-to=frame.raw --stream-count=1 just produces file with zero length (and streaming function still not called).

dmesg log
[   19.460072] imx412 3-001a: imx412_probe
[   19.463750] imx412 3-001a: imx412_parse_hw_config
[   19.463826] imx412 3-001a: imx412_power_on
[   19.465061] imx412 3-001a: imx412_detect
[   19.465065] imx412 3-001a: imx412_read_reg
[   19.466313] imx412 3-001a: imx412_init_controls
[   19.470104] imx412 3-001a: imx412_power_off
[   19.541224] imx412 3-001a: imx412_init_pad_cfg
[   19.541230] imx412 3-001a: imx412_fill_pad_format
[   19.541232] imx412 3-001a: imx412_set_pad_format
[   19.541234] imx412 3-001a: imx412_fill_pad_format

I’m not sure where to start, is my device tree flawed or does the driver fail to implement some functionality? I’ve also tried to put the same structure (contained in i2c@075ba000) inside cci@a0c000 block but it made no difference. Is it even theoretically possible to make this camera work?

Hi @xilsqyyx,

After you build your pipeline, can you issue the command “media-ctl -p -d /dev/mediaX” (X is your media device)? Also, when you try to run v4l2-ctl on the /dev/videoX, can you send the dmesg log, after you stop the v4l2-ctl?

Thanks,
Kim

Hi @kimbo,

media-ctl -d /dev/media0 -p (cleaned up)
Media controller API version 4.14.96

Media device information
------------------------
driver          qcom-camss
model           Qualcomm Camera Subsystem
serial
bus info
hw revision     0x0
driver version  4.14.96

Device topology
- entity 1: msm_csiphy0 (2 pads, 5 links)
            type Node subtype V4L flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                <- "imx412 3-001a":0 [ENABLED,IMMUTABLE]
        pad1: Source
                -> "msm_csid0":0 [ENABLED]
                -> "msm_csid1":0 []
                -> "msm_csid2":0 []
                -> "msm_csid3":0 []

- entity 10: msm_csid0 (2 pads, 7 links)
             type Node subtype V4L flags 0
             device node name /dev/v4l-subdev3
        pad0: Sink
                <- "msm_csiphy0":1 [ENABLED]
                <- "msm_csiphy1":1 []
                <- "msm_csiphy2":1 []
        pad1: Source
                -> "msm_ispif0":0 [ENABLED]
                -> "msm_ispif1":0 []
                -> "msm_ispif2":0 []
                -> "msm_ispif3":0 []

- entity 22: msm_ispif0 (2 pads, 12 links)
             type Node subtype V4L flags 0
             device node name /dev/v4l-subdev7
        pad0: Sink
                <- "msm_csid0":1 [ENABLED]
                <- "msm_csid1":1 []
                <- "msm_csid2":1 []
                <- "msm_csid3":1 []
        pad1: Source
                -> "msm_vfe0_rdi0":0 [ENABLED]
                -> "msm_vfe0_rdi1":0 []
                -> "msm_vfe0_rdi2":0 []
                -> "msm_vfe0_pix":0 []
                -> "msm_vfe1_rdi0":0 []
                -> "msm_vfe1_rdi1":0 []
                -> "msm_vfe1_rdi2":0 []
                -> "msm_vfe1_pix":0 []

- entity 34: msm_vfe0_rdi0 (2 pads, 5 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev11
        pad0: Sink
                [fmt:SRGGB10_1X10/4056x3040 field:none colorspace:srgb]
                <- "msm_ispif0":1 [ENABLED]
                <- "msm_ispif1":1 []
                <- "msm_ispif2":1 []
                <- "msm_ispif3":1 []
        pad1: Source
                [fmt:SRGGB10_1X10/4056x3040 field:none colorspace:srgb]
                -> "msm_vfe0_video0":0 [ENABLED,IMMUTABLE]

- entity 37: msm_vfe0_video0 (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video0
        pad0: Sink
                <- "msm_vfe0_rdi0":1 [ENABLED,IMMUTABLE]

- entity 226: imx412 3-001a (1 pad, 1 link)
              type V4L2 subdev subtype Sensor flags 0
              device node name /dev/v4l-subdev19
        pad0: Source
                [fmt:SRGGB10_1X10/4056x3040 field:none colorspace:unknown xfer:none]
                -> "msm_csiphy0":0 [ENABLED,IMMUTABLE]

The thing that really bothers me here is colorspace: unknown in imx412 3-001a block, however I’ve tried different values (in driver colorspace is set to V4L2_COLORSPACE_RAW) it didn’t make much of a difference.

v4l2-ctl --set-fmt-video=width=4056,height=3040,pixelformat=RG10 --stream-mmap --stream-count=1 -d /dev/video0 --stream-to=imx412.raw (stops immediately, imx412.raw file is zero length):

dmesg
[  721.797201] qcom-camss a34000.camss: CSIPHY 3PH HW Version = 0x10000000
[  721.797501] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[  721.828804] imx412 3-001a: imx412_get_pad_format
[  721.828809] imx412 3-001a: imx412_fill_pad_format

And similar story when I try to run v4l2-compliance -d /dev/videoX -s:

dmesg
[  800.949944] qcom-camss a34000.camss: CSIPHY 3PH HW Version = 0x10000000
[  800.950546] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[  800.996341] imx412 3-001a: imx412_get_pad_format
[  800.996348] imx412 3-001a: imx412_fill_pad_format
[  801.005306] imx412 3-001a: imx412_get_pad_format
[  801.005312] imx412 3-001a: imx412_fill_pad_format
[  801.061168] imx412 3-001a: imx412_get_pad_format
[  801.061175] imx412 3-001a: imx412_fill_pad_format

As I said, it just never gets to function assigned to s_stream video operator.

If I enable debug info in videobuf2-core via echo 0x01 > /sys/module/videobuf2_core/parameters/debug I see the following picture:

dmesg
[  858.892435] qcom-camss a34000.camss: CSIPHY 3PH HW Version = 0x10000000
[  858.892591] qcom-camss a34000.camss: CSID HW Version = 0x30050000
[  858.892744] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[  858.903288] vb2-core: vb2_verify_memory_type: USERPTR for current setup unsupported
[  858.905299] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.913696] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.919967] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.926132] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.933194] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.941101] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.948553] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.954400] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.961680] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.968708] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.975818] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.982950] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.990060] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  858.997222] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  859.005878] vb2-core: __vb2_queue_alloc: allocated 2 buffers, 1 plane(s) each
[  859.011334] vb2-core: vb2_core_expbuf: invalid buffer type
[  859.021936] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  859.024280] imx412 3-001a: imx412_get_pad_format
[  859.024282] imx412 3-001a: imx412_fill_pad_format
[  859.024293] vb2-core: vb2_start_streaming: driver refused to start streaming
[  859.034012] vb2-core: __vb2_queue_alloc: allocated 2 buffers, 1 plane(s) each
[  859.038162] imx412 3-001a: imx412_get_pad_format
[  859.038164] imx412 3-001a: imx412_fill_pad_format
[  859.038172] vb2-core: vb2_start_streaming: driver refused to start streaming
[  859.047579] vb2-core: vb2_core_streamon: no buffers have been allocated
[  859.053812] vb2-core: __vb2_queue_alloc: allocated 2 buffers, 1 plane(s) each
[  859.063564] vb2-core: __vb2_queue_alloc: allocated 2 buffers, 1 plane(s) each
[  859.067002] vb2-core: __vb2_queue_alloc: allocated 1 buffers, 1 plane(s) each
[  859.078433] vb2-core: __vb2_queue_alloc: allocated 2 buffers, 1 plane(s) each
[  859.080156] vb2-core: __vb2_wait_for_done_vb: streaming off, will not wait for buffers
[  859.087441] vb2-core: vb2_core_prepare_buf: invalid buffer state 2
[  859.095087] vb2-core: vb2_core_qbuf: invalid buffer state 3
[  859.101257] vb2-core: vb2_core_prepare_buf: invalid buffer state 3
[  859.106724] vb2-core: __vb2_wait_for_done_vb: streaming off, will not wait for buffers
[  859.112976] vb2-core: __vb2_wait_for_done_vb: streaming off, will not wait for buffers
[  859.121007] vb2-core: vb2_core_prepare_buf: invalid buffer state 2
[  859.128768] vb2-core: vb2_core_qbuf: invalid buffer state 3
[  859.134936] vb2-core: vb2_core_prepare_buf: invalid buffer state 3
[  859.140402] vb2-core: __vb2_wait_for_done_vb: streaming off, will not wait for buffers
[  859.147343] imx412 3-001a: imx412_get_pad_format
[  859.147346] imx412 3-001a: imx412_fill_pad_format
[  859.147357] vb2-core: vb2_start_streaming: driver refused to start streaming
[  859.154898] vb2-core: vb2_verify_memory_type: USERPTR for current setup unsupported

I was able to get s_stream operation function called. It wasn’t called because I had inconsistent parameters using v4l2-ctl -d /dev/videoX --set-fmt-video="..." and parameters that driver actually set.
Now it seems like my timings are messed up (v4l2-ctl just hangs until stopped):

dmesg
[  628.640035] qcom-camss a34000.camss: CSIPHY 3PH HW Version = 0x10000000
[  628.640569] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[  628.655099] vb2-core: __vb2_queue_alloc: allocated 4 buffers, 1 plane(s) each
[  628.664806] imx412 3-001a: imx412_get_pad_format
[  628.664812] imx412 3-001a: imx412_fill_pad_format
[  628.716339] imx412 3-001a: imx412_set_stream
[  628.716346] imx412 3-001a: imx412_power_on
[  628.717580] imx412 3-001a: imx412_start_streaming
[  628.846411] imx412 3-001a: imx412_set_ctrl
[  628.846421] imx412 3-001a: imx412_update_exp_gain
[  628.849565] imx412 3-001a: imx412_set_ctrl
[  629.770482] vb2-core: __vb2_wait_for_done_vb: sleep was interrupted
[  630.296008] qcom-camss a34000.camss: VFE sof timeout
[  630.808080] qcom-camss a34000.camss: VFE reg update timeout
[  630.808138] imx412 3-001a: imx412_set_stream
[  630.808142] imx412 3-001a: imx412_stop_streaming
[  630.820856] vb2: counters for queue ffff8000ec178350:
[  630.820881] vb2:     setup: 1 start_streaming: 1 stop_streaming: 1
[  630.825370] imx412 3-001a: imx412_power_off
[  630.825490] vb2:     wait_prepare: 1 wait_finish: 1
[  630.831403] vb2:   counters for queue ffff8000ec178350, buffer 0:
[  630.835866] vb2:     buf_init: 1 buf_cleanup: 1 buf_prepare: 1 buf_finish: 1
[  630.842034] vb2:     buf_queue: 1 buf_done: 1
[  630.849143] vb2:     alloc: 1 put: 1 prepare: 1 finish: 1 mmap: 1
[  630.853389] vb2:     get_userptr: 0 put_userptr: 0
[  630.859444] vb2:     attach_dmabuf: 0 detach_dmabuf: 0 map_dmabuf: 0 unmap_dmabuf: 0
[  630.864134] vb2:     get_dmabuf: 0 num_users: 0 vaddr: 0 cookie: 1
[  630.872033] vb2:   counters for queue ffff8000ec178350, buffer 1:
[  630.878033] vb2:     buf_init: 1 buf_cleanup: 1 buf_prepare: 1 buf_finish: 1
[  630.884192] vb2:     buf_queue: 1 buf_done: 1
[  630.891312] vb2:     alloc: 1 put: 1 prepare: 1 finish: 1 mmap: 1
[  630.895561] vb2:     get_userptr: 0 put_userptr: 0
[  630.901642] vb2:     attach_dmabuf: 0 detach_dmabuf: 0 map_dmabuf: 0 unmap_dmabuf: 0
[  630.906322] vb2:     get_dmabuf: 0 num_users: 0 vaddr: 0 cookie: 1
[  630.914229] vb2:   counters for queue ffff8000ec178350, buffer 2:
[  630.920215] vb2:     buf_init: 1 buf_cleanup: 1 buf_prepare: 1 buf_finish: 1
[  630.926493] vb2:     buf_queue: 1 buf_done: 1
[  630.933493] vb2:     alloc: 1 put: 1 prepare: 1 finish: 1 mmap: 1
[  630.937761] vb2:     get_userptr: 0 put_userptr: 0
[  630.943819] vb2:     attach_dmabuf: 0 detach_dmabuf: 0 map_dmabuf: 0 unmap_dmabuf: 0
[  630.948527] vb2:     get_dmabuf: 0 num_users: 0 vaddr: 0 cookie: 1
[  630.956409] vb2:   counters for queue ffff8000ec178350, buffer 3:
[  630.962409] vb2:     buf_init: 1 buf_cleanup: 1 buf_prepare: 1 buf_finish: 1
[  630.968561] vb2:     buf_queue: 1 buf_done: 1
[  630.975688] vb2:     alloc: 1 put: 1 prepare: 1 finish: 1 mmap: 1
[  630.979931] vb2:     get_userptr: 0 put_userptr: 0
[  630.986019] vb2:     attach_dmabuf: 0 detach_dmabuf: 0 map_dmabuf: 0 unmap_dmabuf: 0
[  630.990696] vb2:     get_dmabuf: 0 num_users: 0 vaddr: 0 cookie: 1

Hi xilsqyyx,

I was going to suggest looking at the v4l2-ctl get-fmt-video (I think that’s what it is) but it seems likes that’s what you did!

I think the colorspace error is odd. We have a camera module with an IMX412 sensor. In the driver supplied, it shows fmt set with these options:

priv->fmt.code = MEDIA_BUS_FMT_SRGGB10_1X10;
priv->fmt.field = V4L2_FIELD_NONE;
priv->fmt.colorspace = V4L2_COLORSPACE_SRGB;

I will try to hook up the camera module today and try using v4l2-ctl to stream and see what I get so that you can compare output.

Thanks,
Kim

Hi @xilsqyyx,

Full disclosure, we are using a custom board with Snapdragon820c, but we are running linux 4.14.

I ran some commands on our board with the IMX-412 mipi camera connected. These are the results:

#slroot @ ~/sl/bin
SD> v4l2-ctl -d /dev/video0 --get-fmt-video
Format Video Capture Multiplanar:
Width/Height : 4056/3040
Pixel Format : ‘pRAA’ (10-bit Bayer RGRG/GBGB Packed)
Field : None
Number of planes : 1
Flags :
Colorspace : sRGB
Transfer Function : sRGB
YCbCr/HSV Encoding: ITU-R 601
Quantization : Full Range
Plane 0 :
Bytes per Line : 5072
Size Image : 15418880

#slroot @ ~/sl/bin
SD> v4l2-ctl -d /dev/video0 --stream-mmap=4 --stream-count=1 --stream-to=imx412.raw --verbose
VIDIOC_QUERYCAP: ok
idx: 0 seq: 0 bytesused: 15418880 ts: 920.725200

#slroot @ ~/sl/bin
SD> ls -l *.raw
-rw-r–r-- 1 slroot linaro 15418880 Jul 29 07:58 imx412.raw

#slroot @ ~/sl/bin
SD>

The IMX-412 is set to 4 mipi data lanes, but we have another driver for our customers that use 2 mipi data lanes. When I run the v4l2-ctl command to stream, this is what is in the dmesg log:

[ 1250.920098] Setting csi frequency=200000000 pixel_clock=492000000
[ 1250.926413] qcom-camss a34000.camss: CSIPHY 3PH HW Version = 0x10000000
[ 1250.926744] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1250.967116] pixel_clock=492000000 settle_cnt = 18 lane=0x8f
[ 1250.969579] imx412_4lane 5-001a: imx412 streaming=1
[ 1251.587272] imx412_4lane 5-001a: VC Sensor MODE=1 PowerOn STATUS=0x80 i=1
[ 1251.587298] imx412_4lane 5-001a: imx412 streaming=0

We have had some issues with the V4L2_CID_PIXEL_RATE settings, so we added some print statements to the qcom_camss driver to show that “pixel_clock”=492000000. With our two lane mipi data lanes we set this to 1/2 that value in the imx412 driver code. I only mention this because of the SOF timeout message:

We set the V4L2_CID_PIXEL_RATE and V4L2_CID_LINK_FREQ in the set_fmt function of our imx412 driver. It “looks” like your default in your driver for the V4L2_CID_PIXEL_RATE is this:
.pclk = 480000000
This seems close to the value we set for our four mipi data lanes. Can you confirm that this is what you set the device tree to use (4 mipi data lanes)?

Thanks,
Kim

Hi @kimbo,

Yes, my device tree is set up for 4 lanes (there’s a small excerpt from it in my first post). I’ve tried to replicate your settings exactly + adjusting pixel rate in my driver and using exact same v4l2-ctcl command you have used, however it didn’t make much of a difference. I get VIDIOC_QUERYCAP: ok and it hangs there indefinitely afterwards, interestingly enough my call stack seems to stop at my s_ctrl function, probably should inspect it a little closer.

EDIT:
s_ctrl function doesn’t hang, in reality it’s videobuf2 that “hangs”

dmesg
[  480.823145] imx412 4-001a: imx412_set_stream
[  480.823159] imx412 4-001a: imx412_power_on
[  480.824403] imx412 4-001a: imx412_start_streaming
[  480.920263] imx412 4-001a: imx412_set_ctrl: V4L2_CID_EXPOSURE
[  480.920273] imx412 4-001a: imx412_update_exp_gain
[  480.922704] imx412 4-001a: imx412_set_ctrl: V4L2_CID_VBLANK
[  480.931222] vb2-core: vb2_core_streamon: successful
[  480.931367] vb2-core: __vb2_wait_for_done_vb: call_qop(ffff8000e9c78350, wait_prepare)
[  480.935027] vb2-core: __vb2_wait_for_done_vb: will sleep waiting for buffers

It looks like my driver doesn’t return data.

EDIT2:
Using oscilloscope I confirmed that data flows from sensor, there’s a clock signal and mipi signals on lanes, signals disappear when I stop trying to stream and appear when I use v4l2-ctl (start streaming), thus data must be “lost” somewhere inside qcom_camss pipeline. Is there a way to find out where, why and how this happens?

Hi @xilsqyyx,

I would start with putting a print statement for the settle-cnt and frequency in the csiphy_lanes_enable() in camss-csiphy-3ph-1-0.c file under \drivers\media\platform\qcom\camss. I put a print statement right after the calculation of settle-cnt:

settle_cnt = csiphy_settle_cnt_calc(pixel_clock, bpp, c->num_data,
csiphy->timer_clk_rate);

printk(“pixel_clock=%d settle_cnt = %d lane=0x%x\n”, pixel_clock, settle_cnt, lane_mask);

If you could start with this print statement and let me know what you get, maybe this will provide some clue by itself.

Then, in the same folder you can find the individual camss pipeline elements (camss-csid.c, camss-vfe.c camss-ispif.c). I would put a print statement in each of the isr functions in these .c files. The camss-vfe.c has an isr for sof and done. I would put a print statement in the vfe_isr_wm_done and vfe_isr_sof functions in this file.

Thanks,
Kim

Hi @kimbo,

camera configuration
Driver Info:
        Driver name      : qcom-camss
        Card type        : Qualcomm Camera Subsystem
        Bus info         : platform:a34000.camss
        Driver version   : 4.14.96
        Capabilities     : 0x85201000
                Video Capture Multiplanar
                Read/Write
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x05201000
                Video Capture Multiplanar
                Read/Write
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : qcom-camss
        Model            : Qualcomm Camera Subsystem
        Serial           :
        Bus info         :
        Media version    : 4.14.96
        Hardware revision: 0x00000000 (0)
        Driver version   : 4.14.96
Interface Info:
        ID               : 0x03000027
        Type             : V4L Video
Entity Info:
        ID               : 0x00000025 (37)
        Name             : msm_vfe0_video0
        Function         : V4L2 I/O
        Pad 0x01000026   : Sink
          Link 0x02000029: from remote pad 0x1000024 of entity 'msm_vfe0_rdi0': Data, Enabled, Immutable
Priority: 2
Video input : 0 (camera: ok)
Format Video Capture Multiplanar:
        Width/Height      : 4056/3040
        Pixel Format      : 'pRAA' (10-bit Bayer RGRG/GBGB Packed)
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : sRGB
        Transfer Function : sRGB
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Full Range
        Plane 0           :
           Bytes per Line : 5072
           Size Image     : 15418880

Apparently something is not right with settle_cnt:

pixel_clock, settle_cnt and lane_mask
[   93.848823] pixel_clock=492000000 settle_cnt=6 lane=0x8f

Before and after I hit Ctrl-C to stop v4l2-ctl nothing appears regarding vfe_isr_wm_done or vfe_isr_sof (I’ve added prints as you suggested), so it must be that these isr’s were never called.

csiphy_settle_cnt_calc()
[   80.361466] pixel_clock = 492000000 bpp = 10 num_lanes = 4 timer_clk_rate = 100000000
[   80.364833] pixel_clock * bpp / (2 * num_lanes) = 78129088
[   80.372645] ui = 12799
[   80.378019] 85000 + 6 * ui = 123394
[   80.380357] timer_period = 10000
[   80.383836] pixel_clock=492000000 settle_cnt=6 lane0x8f

Hi @xilsqyyx,

Yes, that is the issue we run into a lot…settle cnt is not correct. Can you put in another print statement (around line 140 in camss-phy.c, csiphy_set_clock_rates() function)? I think you are close, but we need to check what the mipi receiver frequency is set to so add the print to that function like this:

/* if sensor pixel clock is not available /
/
set highest possible CSIPHY clock rate */
if (min_rate == 0)
j = clock->nfreqs - 1;
printk(“Setting csi frequency=%d pixel_clock=%d\n”, clock->freq[j], pixel_clock);

round_rate = clk_round_rate(clock->clk, clock->freq[j]);
if (round_rate < 0) {

report back what prints.

Thanks,
Kim

Hi @kimbo,

As can be seen from my previous post and now confirmed again

printk()

[ 230.515709] Setting csi frequency=100000000 pixel_clock=492000000

my CSI frequency is wrong, but I’m not quite sure why yet. My driver just sets it using __v4l2_ctrl_s() nothing fancy there. Will try to fix it.

EDIT:
I’ve applied a fix from this post of yours and now I’m getting proper 200 MHz link frequency and proper settle count, VFE errors disappeared, however I’m still unable to get data.

dmesg
[  251.246196] Setting csi frequency=200000000 pixel_clock=492000000
[  251.254285] qcom-camss a34000.camss: CSIPHY 3PH HW Version = 0x10000000
[  251.254581] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[  251.278833] imx412 3-001a: imx412_get_pad_format
[  251.278839] imx412 3-001a: imx412_fill_pad_format
[  251.284524] pixel_clock = 492000000 bpp = 10 num_lanes = 4 timer_clk_rate = 200000000
[  251.306860] pixel_clock=492000000 settle_cnt=18 lane0x8f
[  251.323134] imx412 3-001a: imx412_set_stream
[  251.323141] imx412 3-001a: imx412_power_on
[  251.324368] imx412 3-001a: imx412_start_streaming
[  251.453852] imx412 3-001a: imx412_set_ctrl
[  251.457030] imx412 3-001a: imx412_set_ctrl
[  251.484668] entering vfe_isr_sof
[  251.484714] exiting vfe_isr_sof
[  251.487074] qcom-camss a34000.camss: VFE0 rdi0 overflow
[  251.514261] entering vfe_isr_sof
[  251.514278] exiting vfe_isr_sof
[  251.516594] qcom-camss a34000.camss: VFE0 rdi0 overflow
[  251.543893] entering vfe_isr_sof
[  251.543922] exiting vfe_isr_sof
[  251.546217] qcom-camss a34000.camss: VFE0 rdi0 overflow
[  251.573519] entering vfe_isr_sof
[  251.573535] exiting vfe_isr_sof
[  251.575831] qcom-camss a34000.camss: VFE0 rdi0 overflow
[  251.594158] imx412 3-001a: imx412_set_stream
[  251.594162] imx412 3-001a: imx412_stop_streaming
[  251.594737] imx412 3-001a: imx412_power_off

EDIT2:
Tried setting pixel_clock to 207,360,000 and got an image, quite distorted though.
Debayered using gst-launch-1.0 -v filesrc location=imx412.raw blocksize=42923008 ! "video/x-bayer,format=bggr,width=4056,height=3040,framerate=1/1" ! bayer2rgb ! videoconvert ! avenc_mjpeg ! filesink location=frame.jpeg -v

Hi @xilsqyyx,

Looking better! When you set the pixel clock to 207,306,000, what “csi frequency” do you get? I think you need to make sure that the csi frequency is 200,000,000.

Thanks,
Kim

Hi @kimbo,

Yeah, looks like settle_cnt and link frequency are off again, but I’ve tried 492000000 as pixel_rate and in that case I have 200 MHz link frequency and 18 as settle_cnt, but no picture.

dmesg
[   72.922905] Setting csi frequency=100000000 pixel_clock=207360000
[   72.928442] qcom-camss a34000.camss: CSIPHY 3PH HW Version = 0x10000000
[   72.928752] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[   72.953172] pixel_clock=207360000 settle_cnt=3 lane0x8f

It feels like link frequency hardly depends on my choice, camss just sets whatever frequency it feels like setting depending on pixel rate.

Hi @xilsqyyx,

I would try increasing the pixel_rate from 207,360,000 in 50Meg increments and increase to beyond 492,000,000. I would keep a table of pixel_rate/settle_cnt/csi frequency. Kind of a pain, but I really think you are getting close!

Thanks,
Kim

Hi @kimbo,

Here’s data I got:

pixel_rate settle_cnt csi frequency image?
80 000 000 5 100 000 000 Y
100 000 000 4 100 000 000 Y
150 000 000 4 100 000 000 Y
200 000 000 3 100 000 000 Y
250 000 000 3 100 000 000 Y
300 000 000 3 100 000 000 Y
350 000 000 12 200 000 000 Y
400 000 000 12 200 000 000 Y
450 000 000 34 200 000 000 N
500 000 000 17 200 000 000 N
550 000 000 14 200 000 000 N
600 000 000 13 200 000 000 N
650 000 000 19 266 666 667 N
700 000 000 19 266 666 667 Y
750 000 000 18 266 666 667 Y
800 000 000 18 266 666 667 Y

Still in none of these cases I could get anything resembling proper image.

EDIT:
It must be problem with conversion. I’ve tried using this tool and SRGGB10 format and now it looks like something, though SRGGB10P (which I assume I actually need) is not implemented, so it looks like I will need to write my own debayering algorithm.

Hi @xilsqyyx,

Yes, that makes sense! Let me know when you get a good image.

Thanks,
Kim

Hi @kimbo,

sorry to bother you again, but I believe that I still face issues with frequencies and stuff.
I was able to confirm that my debayering algorithm somewhat works on a test image (not much color, but at least I see contours and have proper geometry, that’s enough for now), however my image still looks skewed. It also bothers me that I cannot use the same pixel_rate as you do, could it be that this frequency can be different (drastically it seems) from yours?

My test object

Somewhat debayered image

Parts of my test object can be recognized (artifact lines are due conversion to jpeg), but for some reason image is tilted and streched.

I’ve tried slightly altering resolution to ‘de-tilt’ the image, but I believe I shouldn’t need to use such tricks.

Hi @xilsqyyx,

I can not say why we would use different values for pixel_rate. Your original driver uses 480000000 and I use 492000000 to get these values for csi frequency and settle-cnt:
[ 1250.920098] Setting csi frequency=200000000 pixel_clock=492000000
[ 1250.926413] qcom-camss a34000.camss: CSIPHY 3PH HW Version = 0x10000000
[ 1250.926744] qcom-camss a34000.camss: VFE HW Version = 0x70020000
[ 1250.967116] pixel_clock=492000000 settle_cnt = 18 lane=0x8f

In my experience with pixel_rate is that we can set a range of values that will still get us a good image. I can try setting the pixel_rate to 480000000, but my guess is that I will still get a decent image.

It could be the manufacturer of the sensor module. We are use a Vision Components IMX412 module, and our own customer hardware that has the Snapdragon 820c. We are also using 4.14 Linux though.

What numbers do you read for settle-cnt and csi frequency with the above image? Are they close to my numbers?

Thanks,
Kim

Hi @kimbo,

For above image I’ve used pixel_clock = 350 MHz, settle_cnt = 12, csi_freq = 200 MHz.
After fair amount of trials I believe there’s no such pixel_clock value for which I get settle_cnt = 18 and csi_freq = 200 MHz.

Could it be that I have something wrong with hblank and/or vblank parameters thus image is tilted?
Or maybe there’s something wrong with sensor configuration through i2c bus?

We’ve somewhat improved our debayering algorithm and noticed that our raw images are slightly bigger than they should be for this resolution and format. We get images of 15418880 bytes in size instead of (4056 * 3040 * 10) / 8 = 15412800 bytes. Why could that be the case?

750 MHz (other frequencies very similar) image using new debayering algorithm

Hi @xilsqyyx,

I would set the IMX-412 sensor register to use a testpattern. You can then grab the raw data and see if there is any extra data. It should be more clear.

If you set the V4L2_CID_PIXEL_RATE to 750,000,000 to get a CSI freq of 266Mhz and debayer the image you capture, does the image look better or worse?

The settle-cnt isn’t an exact science. Can you hook up a scope and get a rough estimate of the settle-cnt as shown in the scope image here: MIPI CSI interface support for interfacing Image Sensor - #59 by kimbo

There could certainly be something wrong with the sensor configuration. Can you contact the vendor of the IMX-412 sensor that you purchased?

Thanks,
Kim