Dragonboard 410c Reboot problem of Android

raymondman · March 17, 2017, 10:00am

Android system is 6.0.1. After operation of ~10hours, the system restarts. From the logcat, global reference table of system server is full.
I reduce the global reference table size to 2560 in order to make it happen faster (~10mins) without any active apps running. Below is the logcat.

01-01 01:22:31.645403 756 768 F art 01-01 01:22:31.645478 756 768 F art 01-01 01:22:31.645504 756 768 F art 01-01 01:22:31.645522 756 768 F art 01-01 01:22:31.645539 756 768 F art 01-01 01:22:31.645555 756 768 F art 01-01 01:22:31.645574 756 768 F art 01-01 01:22:31.645591 756 768 F art 01-01 01:22:31.645610 756 768 F art 01-01 01:22:31.645630 756 768 F art 01-01 01:22:31.645646 756 768 F art 01-01 01:22:31.645663 756 768 F art 01-01 01:22:31.645678 756 768 F art 01-01 01:22:31.645695 756 768 F art 01-01 01:22:31.645710 756 768 F art 01-01 01:22:31.645724 756 768 F art 01-01 01:22:31.645742 756 768 F art …
01-01 01:22:31.647663 756 768 F art 01-01 01:22:31.647677 756 768 F art 01-01 01:22:31.647697 756 768 F art … : art/runtime/indirect_reference_table.cc:115] JNI ERROR (app bug): global reference table overflow (max=2560)
: art/runtime/indirect_reference_table.cc:115] global reference table dump:
: art/runtime/indirect_reference_table.cc:115] Last 10 entries (of 2559):
: art/runtime/indirect_reference_table.cc:115] 2558: 0x13726920 java.lang.ref.WeakReference
: art/runtime/indirect_reference_table.cc:115] 2557: 0x13468570 com.android.internal.util.AsyncChannel$DeathMonitor
: art/runtime/indirect_reference_table.cc:115] 2556: 0x13742460 java.lang.ref.WeakReference
: art/runtime/indirect_reference_table.cc:115] 2555: 0x136dc8c0 java.lang.ref.WeakReference
: art/runtime/indirect_reference_table.cc:115] 2554: 0x1372c280 java.lang.ref.WeakReference
: art/runtime/indirect_reference_table.cc:115] 2553: 0x136b4ba0 java.lang.ref.WeakReference
: art/runtime/indirect_reference_table.cc:115] 2552: 0x1371a300 java.lang.ref.WeakReference
: art/runtime/indirect_reference_table.cc:115] 2551: 0x136c0c40 java.lang.ref.WeakReference
: art/runtime/indirect_reference_table.cc:115] 2550: 0x137287e0 java.lang.ref.WeakReference
: art/runtime/indirect_reference_table.cc:115] 2549: 0x136cef20 java.lang.ref.WeakReference
: art/runtime/indirect_reference_table.cc:115] Summary:
: art/runtime/indirect_reference_table.cc:115] 1 of com.android.server.SerialService
: art/runtime/indirect_reference_table.cc:115] 1 of com.android.server.VibratorService
: art/runtime/indirect_reference_table.cc:115] 1 of com.android.server.location.FlpHardwareProvider
: art/runtime/indirect_reference_table.cc:115] 2 of java.lang.ThreadGroup (2 unique instances)
: art/runtime/indirect_reference_table.cc:115] 1633 of java.lang.ref.WeakReference (1633 unique instances)
: art/runtime/indirect_reference_table.cc:115] 2 of android.opengl.EGLDisplay (1 unique instances)

01-01 01:22:35.119216 756 768 F art : art/runtime/barrier.cc:90] Check failed: count_ == 0 (count_=-1, 0=0) Attempted to destroy barrier with non zero count
01-01 01:22:35.119320 756 768 F art : art/runtime/runtime.cc:366] Runtime aborting — recursively, so no thread-specific detail!
01-01 01:22:35.119352 756 768 F art : art/runtime/runtime.cc:366]
--------- switch to crash
01-01 01:22:35.119735 756 768 F libc : Fatal signal 6 (SIGABRT), code -6 in tid 768 (Binder_2)
--------- switch to crash
01-01 01:22:35.361678 301 301 F DEBUG : Abort message: ‘art/runtime/indirect_reference_table.cc:115] JNI ERROR (app bug): global reference table overflow (max=2560)’
01-01 01:22:35.361808 301 301 F DEBUG : x0 0000000000000000 x1 0000000000000300 x2 0000000000000006 x3 0000000000000000
01-01 01:22:35.361887 301 301 F DEBUG : x4 0000000000000000 x5 0000000000000001 x6 0000000000000000 x7 0000000000000000
01-01 01:22:35.361953 301 301 F DEBUG : x8 0000000000000083 x9 0000007fa17aca70 x10 0000000000000003 x11 0000000000000170
01-01 01:22:35.362012 301 301 F DEBUG : x12 000000000000000e x13 0000000000004000 x14 0000000000000000 x15 0000000000000000
01-01 01:22:35.362073 301 301 F DEBUG : x16 0000007fa17a56a8 x17 0000007fa1767b6c x18 8000000000000000 x19 0000007f8bceb510
01-01 01:22:35.362130 301 301 F DEBUG : x20 0000007f8bceb450 x21 0000000000000000 x22 0000000000000006 x23 0000007f9e43b000
01-01 01:22:35.362183 301 301 F DEBUG : x24 000000558e61a000 x25 000000558e123c20 x26 000000558dff5960 x27 0000007f9e36e000
01-01 01:22:35.362233 301 301 F DEBUG : x28 000000558e5e8630 x29 0000007f8bce9ad0 x30 0000007fa1765308
01-01 01:22:35.362286 301 301 F DEBUG : sp 0000007f8bce9ad0 pc 0000007fa1767b74 pstate 0000000020000000
01-01 01:22:35.371922 301 301 F DEBUG :
01-01 01:22:35.371922 301 301 F DEBUG : backtrace:
01-01 01:22:35.372078 301 301 F DEBUG : #00 pc 000000000006ab74 /system/lib64/libc.so (tgkill+8)
01-01 01:22:35.372184 301 301 F DEBUG : #01 pc 0000000000068304 /system/lib64/libc.so (pthread_kill+68)
01-01 01:22:35.372254 301 301 F DEBUG : #02 pc 00000000000212f8 /system/lib64/libc.so (raise+28)
01-01 01:22:35.372319 301 301 F DEBUG : #03 pc 000000000001ba98 /system/lib64/libc.so (abort+60)
01-01 01:22:35.372389 301 301 F DEBUG : #04 pc 0000000000436508 /system/lib64/libart.so (art::Runtime::Abort()+324)
01-01 01:22:35.372448 301 301 F DEBUG : #05 pc 0000000000139264 /system/lib64/libart.so (art::LogMessage::~LogMessage()+3136)
01-01 01:22:35.372514 301 301 F DEBUG : #06 pc 0000000000133d08 /system/lib64/libart.so (art::Barrier::~Barrier()+296)
01-01 01:22:35.372571 301 301 F DEBUG : #07 pc 000000000046b0f4 /system/lib64/libart.so (art::ThreadList::Dump(std::__1::basic_ostream<char, std::__1::char_traits >&)+212)
01-01 01:22:35.372628 301 301 F DEBUG : #08 pc 00000000004365f0 /system/lib64/libart.so (art::Runtime::Abort()+556)
01-01 01:22:35.372691 301 301 F DEBUG : #09 pc 0000000000139264 /system/lib64/libart.so (art::LogMessage::~LogMessage()+3136)
01-01 01:22:35.372750 301 301 F DEBUG : #10 pc 0000000000276508 /system/lib64/libart.so (art::IndirectReferenceTable::Add(unsigned int, art::mirror::Object*)+372)
01-01 01:22:35.372804 301 301 F DEBUG : #11 pc 000000000030d4ac /system/lib64/libart.so (art::JavaVMExt::AddGlobalRef(art::Thread*, art::mirror::Object*)+56)
01-01 01:22:35.372861 301 301 F DEBUG : #12 pc 00000000003432cc /system/lib64/libart.so (art::JNI::NewGlobalRef(_JNIEnv*, _jobject*)+320)
01-01 01:22:35.372917 301 301 F DEBUG : #13 pc 0000000000462050 /system/lib64/libart.so (art::CreateNativeThread(_JNIEnv*, _jobject*, unsigned long, bool)+180)
01-01 01:22:35.372982 301 301 F DEBUG : #14 pc 000000007253d2b4 /data/dalvik-cache/arm64/system@framework@boot.oat (offset 0x1f08000)

It seems java.lang.ref.WeakReference (1633) are the main cause but I have no idea where they are from. Could anyone advise how I can debug it?

Thanks in advance!

raymondman · March 20, 2017, 7:57am

Hi,

I run SD card Android Image build #118 with a downloaded app “Wifi analyzer” running over the last weekend and the reboot problem also happens. It seems the root cause is inside the Android system. Do anyone have hints to debug it?

Thanks in advance!

danielt · March 20, 2017, 2:03pm

I’m not going to claim much expertise in diagnosing Android problems but I’d be interested to know if the new #128 snapshot build (http://builds.96boards.org/snapshots/dragonboard410c/qualcomm/android/128/ ) exhibits the same problem.

raymondman · March 21, 2017, 11:39am

Hi danielt,

I reduce the global reference table size for #128 and Android can run for 1 hours without reboot, comparing with ~10mins with #118. The problem seems gone. Anyway, I will test the original #128 overnight and tell the result tomorrow.

I used DB410C_build.sh to download AOSP last time. Could you please advise the command details how I download the branch/tag of #128?

Thanks a lot!

raymondman · March 22, 2017, 8:59am

Hi danielt,

The dragonboard has run for over 18 hours without reboot problem. It seems the problem has gone with #128.
I download the following BSP from codeaurora.org but the reboot problem still persists.
March 02, 2017 LA.BR.1.2.7-05610-8x16.0 msm8916_64 LA.BR.1.2.7-05610-8x16.0.xml 06.00.01

Could you please tell which one is #128?

Thanks in advance!

danielt · March 22, 2017, 11:09am

I believe it is built from vLA.BR.1.2.7-01010-8x16.0-4 (available here: https://developer.qualcomm.com/hardware/dragonboard-410c/tools ).

raymondman · March 22, 2017, 11:21am

Hi danielt,

I find that the source-manifest.xml in build #128 is exactly the same as the one I am using which is downloaded from la.br.1.2.7-01010-8x16.0. Could you please tell what the difference is between build #128 and la.br.1.2.7-01010-8x16.0?

Thanks a lot!

danielt · March 22, 2017, 11:33am

The exact versions that contribute to the build should be in the pinned-manifest.xml.

raymondman · March 23, 2017, 11:30am

Hi danielt,

The pinned-manifest.xml shows that revision number is different in 10 folders. e.g. in pinned-manifest.xml,

in my manifest.xml,

When I run “git log --oneline” under device/qcom/common, the output is as follow,
3edfb88 USB camera patch-permissions changes
3eb4cdf device:qcom:common USB to Ethernet patch
1d4c2f7 Merge “IMS-VT: Handle buttons display in InCall Screen”

I do not see the revision “6ef0f58274a9cdf1c4af9bd395e5e39957353ec4”.

I also run “git fetch” but see no difference. Am I missing something or doing wrong in git? Could you please advise how I find the target revision and pull it to my local repository?

Many thanks!