Android Automotive

doitright · June 7, 2018, 1:55pm

Yeah, I noticed the missing sound as well. Seems that I failed to test it again after the previous resync back in march, and since I’ve just gotten back to work this week (new baby at home, so I was off work for a while), I didn’t actually receive a call in my car until just a couple of days ago. And because I’m getting used to a new routine this week, I haven’t even had a chance to dump the logs.

What I’m seeing in your log there that stands out, is “BTCARD: -1”.

This means that it has not picked up the alsa device corresponding to the bluetooth i2s. There are two possible places for the problem; the HAL, or the KERNEL. Both of these possibilities are tied into audio related changes in the kernel.

What I mean by that is this;
Either the code in the HAL that detects the path of the alsa devices was unable to account for the changes in the kernel, or the kernel is no longer presenting the alsa device corresponding to the i2s.

Neither possibility is particularly difficult to fix.

The code that detects the sound card;

gitlab.com

HiKey960-Car/android_device_linaro_hikey/blob/master/usbaudio/audio_hal.c#L311

    
      
          
          
/*
           * Find the primary and bluetooth cards
           *   card    A pointer to a variable to receive the primary card number.
           *   device  A pointer to a variable to receive the primary card device number.
           *   scocard A pointer to a variable to receive the bluetooth card number.
           * NOTE: The variables pointed to by card, device, and scocard return -1 (undefined) if the
           *  associated card/device is not found in the provided string.
           *  Return true if the cards/devices are all found, false otherwise.
           */
          static bool find_cards_devices(int *card, int *device, int *scocard)
          {
              struct dirent **namelist;
              int n, t;
              int cards[10];
              for (n=0; n<10; n++) cards[n] = 0;
          
          
    // initialize to "undefined" state.
              *card = -1;
              *device = -1;
              *scocard = -1;

The relevant kernel bits are here;

gitlab.com

HiKey960-Car/android_kernel_linaro_hikey/blob/android-hikey-linaro-4.9/arch/arm64/boot/dts/hisilicon/hi3660.dtsi#L1363

    
      
                                          frame-master = <&hdmi_master>;
          
          
                                hdmi_master: cpu {
                                                  sound-dai = <&i2s2>;
                                          };
          
          
                                codec {
                                                  sound-dai = <&adv7533>;
                                          };
          			};
          			simple-audio-card,dai-link@1 { /* BT SCO */
          				format = "i2s";
          
          
				bitclock-master = <&sco_master>;
          				frame-master = <&sco_master>;
          
          
				sco_master: cpu {
          					sound-dai = <&i2s2>;
          				};
          
          
				codec {

The way to tell if its kernel or HAL:
You’re looking for a sound device that supports BOTH capture AND playback, that is NOT the USB card.

This is the actual selection criteria I’ve used, which I’ll admit is not the strongest;

gitlab.com

HiKey960-Car/android_device_linaro_hikey/blob/master/usbaudio/audio_hal.c#L292

    
      
          
          
static void adev_remove_stream_from_list(
              struct audio_device* adev, struct listnode* stream_node) {
              device_lock(adev);
          
          
    list_remove(stream_node);
          
          
    device_unlock(adev);
          }
          
          
int sco_card_select(const struct dirent *entry)
          {
             return ((strstr(entry->d_name, "pcmC") != NULL) && (strstr(entry->d_name, "D1c") != NULL));
          }
          
          
int primary_card_select(const struct dirent *entry)
          {
             return ((strstr(entry->d_name, "pcmC") != NULL) && (strstr(entry->d_name, "D0") != NULL));
          }
          
          
/*

If you look at it right now, what you can do is run “ls -l /dev/snd” and “ls -l /sys/class/sound” and post theie outputs here.

Also, if you could get a log of usbaudio from higher up in the logcat “logcat | grep usbaudio”, and a dmesg, those would also be very helpful.

Note that contrary to my previous statement that I wasn’t going to work on software until I’m through with the hardware, this is a small enough bug and significant enough annoyance to me that I’ll definitely work on it asap.

And finally, you will note that I’ve changed over to gitlab. I’m not comfortable with the ownership changes at github, so I’m not going to be using them any further.

doitright · June 7, 2018, 10:07pm

Problem is definitely in the kernel, it isn’t creating the sound device.

doitright · June 8, 2018, 1:15am

I think I found the problem… my fault. I’m a bonehead.

Edit: YEP, found and fixed. You’ll need this;

AhmedX6 · June 8, 2018, 9:51am

Hello @doitright

I already had the CONFIG_DRM_I2C_ADV7511_AUDIO=y.
Here is a picture with sounds driver connected :
Screenshot%20from%202018-06-08%2010-28-58

Screenshot%20from%202018-06-07%2017-19-08

Here is the full logcat when I simulating a call on my phone which is connected to the Hikey.
https://gist.github.com/AhmedX6/dea7b8a79220e00b976291f9b4b48039

I have no sound output from my USB headphones.

EDIT:
Just so you know I reactivated the i2S kernel modules. And the sound seems to work on music player or anything else but NOT on phone call.

Yesterday, I noticed that I have an error -12 coming from usb_audio_hal on the start_output_stream coming sometimes and this error breaks definitely sound’s output. I have to unplug and plug again the usb headphone to make it work again.
Here is the logcat when I have the error :
StreamHAL: Error from HAL stream in function get_presentation_position: Operation not permitted
alsa_device_proxy: proxy_open() pcm_is_ready() failed: cannot open device ‘/dev/snd/pcmC1D0p’: No such file or directory
modules.usbaudio_hal.hikey: out_write: start_output_stream returned error -12

doitright · June 8, 2018, 12:13pm

Ok, so I just realized that you’re working off the MARCH branches. I guess I lost track of when I did things, I actually have a newer branch from APRIL. That explains why you have that config set, but doesn’t explain the log you gave me on the previous message with the “BTCARD: -1”.

On your more recent log, it actually looks like the call was set up correctly. The only thing that stands out in that log is this line;
06-08 08:35:02.873 2374 2387 D modules.usbaudio_hal.hikey: adev_set_parameters: kvpairs: hfp_volume=1

hfp_volume is on a scale of 1 to 15, where 15 is the loudest.
Try raising the volume. If you haven’t set up volume buttons on the SWI, then you should be able to raise the volume using the volume buttons on the side of your cell phone.

StreamHAL errors can be safely ignored.
The open device error says that the device is missing. Probably caused by “unplug and plug again the usb headphone” – don’t do that, the HAL isn’t designed to handle that. The USB sound card has to be plugged in at boot time and left in, or strange things can happen. That includes doing things like plugging in a USB-C cable (the APRIL kernel has USB device mode re-enabled).

One thing to note about the APRIL branch… you will need to update the firmware on the hikey960 to make it work. Not my changes, they’re upstream changes.

In future, please do not use screenshots for conveying logs or terminal output. Its much better to copy/paste the text.

AhmedX6 · June 8, 2018, 4:25pm

Ok. I just realigned to the APRIL branch, I recompiled the kernel etc and copied every files.

It seems that now the OS is not booting anymore. Here’s the output log on boot.
https://gist.github.com/AhmedX6/0dd0a83f66fcd85ee83568510b6a4744

He’s always trying to start services needed by Android but everything getting killed by signal 1.

doitright · June 11, 2018, 8:23am

Did you update the firmware? Or just flash the newly built images?

AhmedX6 · June 11, 2018, 10:23am

I only flashed the newly built image.
What’s new in that new firmware ?

#EDIT
It seems that it works now with the new firmware.

It still does not fix the problem I had with the sound on call event (phone is connected on bluetooth to hikey)

doitright · June 11, 2018, 2:56pm

Have you actually tried to increase the volume as I suggested?
And are you using the same USB sound card that I recommended (Vantec NBA-200U)?

AhmedX6 · June 11, 2018, 4:26pm

The sound is at max.
I’m not using the Vantex NBA-200U. I’m using this one.

doitright · June 11, 2018, 6:12pm

I can’t even find an english language description for that, nor a proper and complete specifications sheet. There are at least a dozen reasons why it won’t work if you’re using the wrong sound card, the most obvious subset are the following;

If it doesn’t support 48 kHz sample rate, it will not work.
If it doesn’t have an integrated mixer, it won’t work.
If it does have an integrated mixer, but its controls are named differently, it won’t work.

That is not to say that you can’t make that one work, just that I have worked specifically with ONE device and I have NOT written the audio HAL to be universal. If you want to try to make that card work, then you will have to read and understand the audio HAL, and adjust where needed.

In particular;
Start SCO:

gitlab.com

HiKey960-Car/android_device_linaro_hikey/blob/master/usbaudio/audio_hal.c#L1749

    
      
          struct str_parms *parms;
          
          
parms = str_parms_create_str(kvpairs);
          
          
ret = str_parms_get_str(parms, AUDIO_PARAMETER_HFP_SET_SAMPLING_RATE, value, sizeof(value));
          if (ret >= 0) {
              val = atoi(value);
              adev->sco_samplerate = val;
          }
          
          
ret = str_parms_get_str(parms, AUDIO_PARAMETER_HFP_ENABLE, value, sizeof(value));
          if (ret >= 0) {
              pthread_mutex_lock(&adev->sco_thread_lock);
              if (strcmp(value, "true") == 0){
                  if (adev->sco_thread == 0) {
                      adev_set_mic_volume(hw_dev, MICROPHONE_VOLUME_DEFAULT);
                      adev->terminate_sco = false;
                      pthread_create(&adev->sco_thread, NULL, &runsco, adev);
                      set_line_in(hw_dev);
                  }
              } else {

Set microphone volume;

gitlab.com

HiKey960-Car/android_device_linaro_hikey/blob/master/usbaudio/audio_hal.c#L1714

    
      
              for (i=0; i<size; i++)
                  mixer_ctl_set_value(vol_ctl, i, (int)((float) max * adev->vol_balance[i] * volume));
          
          
    mixer_close(adev->hw_mixer);
              adev->hw_mixer = 0;
          
          
    pthread_mutex_unlock(&adev->mixer_lock);
              return 0; // any return value other than 0 means that master volume becomes software emulated.
          }
          
          
void adev_set_mic_volume(struct audio_hw_device *hw_dev, int percent)
          {
              struct audio_device * adev = (struct audio_device *)hw_dev;
              if (adev->usbcard < 0) return;
              pthread_mutex_lock(&adev->mixer_lock);
              if (adev->hw_mixer == 0) adev->hw_mixer = mixer_open(adev->usbcard);
              if (adev->hw_mixer == 0) return;
              struct mixer_ctl *vol_ctl = mixer_get_ctl_by_name(adev->hw_mixer, "Mic Capture Volume");
              int max = mixer_ctl_get_range_max(vol_ctl);
          
          
    mixer_ctl_set_value(vol_ctl, 0, max * percent / 100);

Set HFP volume:

gitlab.com

HiKey960-Car/android_device_linaro_hikey/blob/master/usbaudio/audio_hal.c#L1629

    
      
              adev->sco_pcm_near_out = 0;
              adev->sco_pcm_far_in = 0;
              adev->sco_pcm_far_out = 0;
          
          
    adev->sco_thread = 0;
          
          
    return NULL;
          
          
}
          
          
void set_hfp_volume(struct audio_hw_device *hw_dev, int volume)
          {
              // value of volume will be between 1 and 15 inclusive
          
          
    struct audio_device * adev = (struct audio_device *)hw_dev;
              if (adev->usbcard < 0) return;
              pthread_mutex_lock(&adev->mixer_lock);
              if (adev->hw_mixer == 0) adev->hw_mixer = mixer_open(adev->usbcard);
              if (adev->hw_mixer == 0) return;
              struct mixer_ctl *vol_ctl = mixer_get_ctl_by_name(adev->hw_mixer, "Speaker Playback Volume");
              int max = mixer_ctl_get_range_max(vol_ctl);

USB Alsa configuration:

gitlab.com

HiKey960-Car/android_device_linaro_hikey/blob/master/usbaudio/audio_hal.c#L1384

    
      
              .channels = 2,
              .rate = adev->sco_samplerate,
              .format = PCM_FORMAT_S16_LE,
              .period_size = 1024,
              .period_count = 4,
              .start_threshold = 0, // 0's mean default
              .silence_threshold = 0,
              .stop_threshold = 0,
          };
          
          
struct pcm_config usb_config = {
              .channels = 2,
              .rate = 48000,
              .format = PCM_FORMAT_S16_LE,
              .period_size = 1024,
              .period_count = 4,
              .start_threshold = 0,
              .silence_threshold = 0,
              .stop_threshold = 0,
          };

Open USB output:

gitlab.com

HiKey960-Car/android_device_linaro_hikey/blob/master/usbaudio/audio_hal.c#L1409

    
      
          // not opening the pcm's.
          struct listnode* node;
          
          
i = 0;
          do {
              list_for_each(node, &adev->output_stream_list) {
                  struct audio_stream* stream = (struct audio_stream *)node_to_item(node, struct stream_out, list_node);
                  out_standby((struct audio_stream_out *)stream);
              }
          
          
    adev->sco_pcm_near_out = pcm_open(adev->usbcard, 0, PCM_OUT, &usb_config);
              i++;
          } while (i < 10 && (adev->sco_pcm_near_out == 0 || !pcm_is_ready(adev->sco_pcm_near_out)));
          
          
if (adev->sco_pcm_near_out == 0) return NULL;
          if (!pcm_is_ready(adev->sco_pcm_near_out)){
              pcm_close(adev->sco_pcm_near_out);
              return NULL;
          }
          
          
i = 0;

Open USB input:

gitlab.com

HiKey960-Car/android_device_linaro_hikey/blob/master/usbaudio/audio_hal.c#L1426

    
      
              return NULL;
          }
          
          
i = 0;
          do {
              list_for_each(node, &adev->input_stream_list) {
                  struct audio_stream* stream = (struct audio_stream *)node_to_item(node, struct stream_in, list_node);
                  in_standby((struct audio_stream_in *)stream);
              }
          
          
    adev->sco_pcm_near_in = pcm_open(adev->usbcard, 0, PCM_IN, &usb_config);
              i++;
          } while (i < 10 && (adev->sco_pcm_near_in == 0 || !pcm_is_ready(adev->sco_pcm_near_in)));
          
          
if (adev->sco_pcm_near_in == 0){
              pcm_close(adev->sco_pcm_near_out);
              return NULL;
          }
          if (!pcm_is_ready(adev->sco_pcm_near_in)){
              pcm_close(adev->sco_pcm_near_out);
              pcm_close(adev->sco_pcm_near_in);

etc.

Further: When you are testing things, make sure that you DO NOT USE THE USB-C PORT AT ALL. If you want to check logs, you need to use either ADB-over-IP or UART.

ONE LAST THING:
In your “phone call log” outputs, I can see the debug output from this line;

gitlab.com

HiKey960-Car/android_device_linaro_hikey/blob/master/usbaudio/audio_hal.c#L1395

    
      
              .channels = 2,
              .rate = 48000,
              .format = PCM_FORMAT_S16_LE,
              .period_size = 1024,
              .period_count = 4,
              .start_threshold = 0,
              .silence_threshold = 0,
              .stop_threshold = 0,
          };
          
          
ALOGD("%s: USBCARD: %d, BTCARD: %d", __func__, adev->usbcard, adev->btcard);
          
          
// Put all existing streams into standby (closed). Note that when sco thread is running
          // out_write and in_read functions will bail immediately while pretending to work and
          // not opening the pcm's.
          struct listnode* node;
          
          
i = 0;
          do {
              list_for_each(node, &adev->output_stream_list) {
                  struct audio_stream* stream = (struct audio_stream *)node_to_item(node, struct stream_out, list_node);

BUT I do NOT see the debug output from THIS line;

gitlab.com

HiKey960-Car/android_device_linaro_hikey/blob/master/usbaudio/audio_hal.c#L1529

    
      
          audioproc_hpf_en(apm, 1);
          audioproc_aec_drift_comp_en(apm, 0);
          audioproc_aec_en(apm, 1);
          audioproc_aec_delayag_en(apm);
          audioproc_ns_set_level(apm, 2); // 0 = low, 1 = moderate, 2 = high, 3 = veryhigh
          audioproc_ns_en(apm, 1);
          audioproc_agc_set_level_limits(apm, 0, 255);
          audioproc_agc_set_mode(apm, 1); // 0 = Adaptive Analog, 1 = Adaptive Digital, 2 = Fixed Digital
          audioproc_agc_en(apm, 1);
          
          
ALOGD("%s: PCM loop starting", __func__);
          
          
memset(framebuf_far_stereo, 0, block_len_bytes_far_stereo);
          while (!adev->terminate_sco && pcm_read(adev->sco_pcm_far_in, framebuf_far_stereo, block_len_bytes_far_stereo) == 0){
          
          
    memset(framebuf_far_mono, 0, block_len_bytes_far_mono);
              stereo_to_mono(framebuf_far_stereo, framebuf_far_mono, frames_per_block_far);
          
          
    if (loopcounter < 1000){ // Broken input detection
          
          
        temp_c = rejig((uint8_t*)framebuf_far_mono, (uint8_t*)framebuf_far_stereo, block_len_bytes_far_mono, temp_c);

That means that it failed somewhere in between those two lines, probably bailed out, maybe here;

gitlab.com

HiKey960-Car/android_device_linaro_hikey/blob/master/usbaudio/audio_hal.c#L1416

    
      
                  out_standby((struct audio_stream_out *)stream);
              }
          
          
    adev->sco_pcm_near_out = pcm_open(adev->usbcard, 0, PCM_OUT, &usb_config);
              i++;
          } while (i < 10 && (adev->sco_pcm_near_out == 0 || !pcm_is_ready(adev->sco_pcm_near_out)));
          
          
if (adev->sco_pcm_near_out == 0) return NULL;
          if (!pcm_is_ready(adev->sco_pcm_near_out)){
              pcm_close(adev->sco_pcm_near_out);
              return NULL;
          }
          
          
i = 0;
          do {
              list_for_each(node, &adev->input_stream_list) {
                  struct audio_stream* stream = (struct audio_stream *)node_to_item(node, struct stream_in, list_node);
                  in_standby((struct audio_stream_in *)stream);
              }
          
          
    adev->sco_pcm_near_in = pcm_open(adev->usbcard, 0, PCM_IN, &usb_config);

That would happen if your card is not compatible with the configurations being used, which most likely suggests that your card is incapable of 48 kHz sample rate, or is incapable of STEREO input.

AhmedX6 · June 12, 2018, 11:46am

Thank you for this very detailed answer.
I will investigate in that way. I will check where exactly the audio hal fails with this setup.

I’ll let you know.

AhmedX6 · June 18, 2018, 12:02pm

Hello,

I found a solution to hear sound on phone call (your answer helped a lot)

I modified the number of channels for input and now it seems to work correctly.

Now I would like to be able to play an other sound (system sound) during phone call.
I noticed that you put all streams in standby when you run the SCO thread. Could we just let one open (like for example the SYSTEM stream) ?

i = 0;
    do {
        list_for_each(node, &adev->output_stream_list) {
            struct audio_stream* stream = (struct audio_stream *)node_to_item(node, struct stream_out, list_node);
            out_standby((struct audio_stream_out *)stream);
        }
	ALOGD("i = %d", i);
        adev->sco_pcm_near_out = pcm_open(adev->usbcard, 0, PCM_OUT, &usb_config);
        i++;
    } while (i < 10 && (adev->sco_pcm_near_out == 0 || !pcm_is_ready(adev->sco_pcm_near_out)));

How to customize this part of code to handle this case ? Or just be able to mix 2 sounds ?

doitright · June 18, 2018, 1:15pm

Ok, so I take it then that your sound card does not support stereo microphones. This is what makes it difficult/impossible to support multiple USB sound cards – yours doesn’t support stereo microphones. Mine does not support MONO microphones.

Of that segment of code you quoted, look carefully at the last and 3rd from last lines. The 3rd from last attempts to open the stream. The last checks if it was successful. If it was unsuccessful, it tries to close all the system streams and then open the SCO stream again. It does this up to 10 times. You see, the 3rd from last line WILL NOT SUCCEED if ANY of the system streams are open.

So you would have to implement a sound mixer.

I didn’t implement that because in my opinion, that is an anti-feature. It seems good initially, but in real world use, having your phone call interrupted by system sounds is incredibly annoying because now the voices on the phone call are competing with navigation / system sounds which randomly jump in and speak over everyone.

It also significantly increases the complexity of echo cancellation, because now you don’t just have to filter the far side sound from the near side microphone, you will also have to filter the local system sounds from it as well. This will ultimately add audible artifacts to the outbound sound stream. If you really want to do this, my suggestion would be to perform the mixing in the SCO thread’s main loop, BEFORE feeding the reverse stream into WebRTC: usbaudio/audio_hal.c · master_apr2018 · HiKey960-Car / android_device_linaro_hikey · GitLab

If I were doing this, the way I would do it is this:

Create an alsa LOOPBACK device (you do this in the kernel config). Don’t forget to check the card identification routines after this is implemented, because it may misidentify the USB device once the loopback device is present!
When the SCO thread is running, redirect the out_write functions to the loopback device.
In the SCO thread, each read from the BLUETOOTH device should also read from the loopback device, and mix the data read from the loopback device into the bluetooth stream. Make absolutely certain that your read is non-blocking or that you check how much data is available on the loopback device before reading from it, or it will stall your bluetooth stream and either cause your audio to drift, or halt altogether.
After the SCO main loop terminates, don’t forget to reset the system stream.

Be aware that the stream is 16 kHz 1-channel at this point. Its better to keep it this way because it means less load created by the audio processing as compared to 48 kHz or stereo. Android, of course, will expect 48 kHz stereo stream, so when you read from the loopback device, you will then have to (a) downconvert from 2->1 channel, and (b) resample at 16 kHz. You can see examples of both in the SCO main loop, but be aware that my 2->1 channel conversion isn’t mixing the channels together, its just deleting the second channel.

AhmedX6 · June 19, 2018, 8:52am

That’s very interesting.

You’re right about that. I think it’s better to develop an independent system that handle sounds of the car.

doitright · June 19, 2018, 12:23pm

I’m not sure what you mean by “independent”.
The point is that when there is a phone call running, you would typically not want to hear any of the system sounds at all.

AhmedX6 · June 19, 2018, 1:51pm

I mean an other hardware that can handle sounds.

Yes, the user can be in a phone call but imagine he activates the turn signal and there is no sound playing. It’s a bit weird.

doitright · June 19, 2018, 4:03pm

Other hardware is redundant. Can’t understand what you mean by ‘a bit weird’ when activating turn signal. Turn signal sound is caused by a mechanical relay in the dashboard making a clicking sound, has nothing to do with car radio.

AhmedX6 · June 20, 2018, 8:33am

This is on old vehicles. But now there are some other signals (like Belt detached, low battery/fuel level…). It’s often .wav files that are played.

doitright · June 20, 2018, 12:10pm

New cars still use mechanical relays for things like that. If you find a car that fakes the relay clicking with speakers, you probably should avoid buying it, because that vehicle will not be serviceable at all.

But regardless of that, if you did want to implement such “features”, I don’t see why it would need a second sound card, and I still would want to mute it while on a call.

“Belt detached”? What the heck kind of a car would have its belt come off frequently enough to need a special warning for it?