Real Time Processing on Android for Drones and Automation

Products Support DragonBoard410c
#1

Hey folks,

I’ve been looking into controlling a drone with my Dragonboard, remotely to begin with but looking to be fully autonomous down the line. This brought me to discover Real Time Processing, and the differences between an OS like Android and embedded processes Arduinos.

Stock Android cannot run as a Real Time process, and I was on the point of giving up on it, but found this framework called RTAndroid to save the day.

RTAndroid is developed by the Embedded Software Laboratory at RWTH Aachen University and currently supports a few Nexus devices, Odroid and the Raspberry Pi 3, and I was wondering if 96boards could get this ported over to the Dragonboard and other 96board spec boards. With the biggest feature of the boards being the GPIOs and other inputs and outputs, it seems like one of the greatest features would be to enable Real Time processing to take full advantage of them.

I have no idea on the process or collaboration needed for this but I thought I’d bring it up here before sending them a port request.

#2

Hi @furiousben,

The drone with RTAndroid would be extremely advantage for 96Boards which has high performance SoCs.

There are people in Linaro who is good at real time knowledge,
Our resource is pretty much stretched out and I strongly appreciate your posts or any kind of collaboration on the forum for making it happen.

#3

One of the issues with realtime-android is that the patches required to enable real-time support at the OS layer (whether PREEMPT_RT or Xenomai) are not in mainline and therefore an upfront effort is required to do the port and subsequent maintenance tasks.

I can’t comment based on experience on RTAndroid (PREEMPT_RT based) but something that a number of companies are doing and something that I have done myself a number of times is patching the Android kernels with Xenomai and installing gblic+xenomai on the filesystem: that is a relatively simple process that permits realtime applications to execute on Android platforms but of course doesn’t give full realtime support to the whole Android stack (only to those glibc based applications using the Xenomai libraries).

For most use cases this is enough.

#4

Hey guys, thanks for your input. I’ve contacted one of the creators of RTAndroid and while he doesn’t have the time at the moment to port it he is definitely interested in it and wants to support it in the future. I’ve asked him if there’s anything I can do to get it started, and waiting for his reply.

@jro Thanks for sharing your experiences with RT. I’ve had a quick look into Xenomai and I think trying the Cobalt extension might be a good place to start. RT is very new to me, I only learned of it recently, so I’m kinda starting at the beginning. I understand the situation and setup, but have never messed with a kernel before.

I’m currently reading through Android Real Time for the Rest of Us and the papers linked on the RTAndroid Publications page.

I’ll try to keep posting my progress here.

1 Like
#5

Hey folks, it might be easier for me to join this discussion and avoid additional e-mail overhead.

The main problem with plain PREEMPT_RT and Xenomai (and all the others) is that you can only get real-time support (after porting / patching the device-specific kernel) for native code. It’s definitely fine if the target hardware is running Linux and you are writing software for a small dedicated project. But I see no point in doing so with Android if there is no way to benefit from it on the application level and make use the huge world of existing Java software. This is basically the main reason behind RTAndroid.

Unfortunately, Android/Java combination brings in a lot of additional complexity like enforcing SELinux policy, process isolation, additional process monitoring & management, automatic memory management (like blocking GC), Android’s decoupled architecture, …, and finally Bionic instead of glibc. We spent a lot of time doing research and proposing solutions to a number of these things, but with every new Android release we had either to discard our old changes or automate their integration into the new codebase.
Today we had a nighlty release of a new RTAndroid image for Raspberry Pi 3 based on the latest Android N Preview 4. RT IPC or RT GC are not included as we don’t have enough resources at our lab to implement it over and over again. But the good thing is that a whole bunch of other features is being merged, compiled and released fully automatically for every supported hardware platform. This is what I hope we can also do for Dragonboards. It seems to be an amazing hardware to work with and even this “basic” version of RTAndroid with a generic interface for different tasks could highly increase its field of application.

#6

how are you achieving real-time in Android without PREEMPT_RT or Xenomai?

#7

Hey @Igor, great to see you here!

Igor is one (the main?) creator of RTAndroid that I got in contact with for those who don’t know, seems he’s pretty excited about the Dragonboard x)

@Igor From what you’ve said by email it seems you need to setup the toolchain and download the Android OS source code to get started porting RTAndroid. With Xenomai you only need to modify the kernel, but for RTAndroid the entire OS needs to be modified correct? What kind of hardware do you need for that? Is a laptop gonna cut it or does it always take hours to complete? Some of the large Android app projects I’ve worked on take a good 5 minutes to build, how long does the OS take on average?

Also from what I’ve understood a lot of the changes for Xenomai are just merging branches of the Android/Linux tree, what kind of work is needed for RTAndroid?

#8

@jro, we do use PREEMPT_RT. I was just trying to say that there is more than that.

#9

good that is what I thought - as a matter of fact, Xenomai 3 can operate over PREEMPT_RT or the Cobalt microkernel so they are not exclusive when it comes to RTOS abstraction.

When deciding whether to use PREEMPT_RT or other solutions the question often becomes: do I really need to make the whole system realtime or only those parts of the system - or those libraries or binaries or maybe HALs- that do require it?

The later is the Xenomai approach and it is applicable to Android xenomai/glibc based applications (I have done a few - recently for ELC 2016 I wrote Android demo on HiKey to generate a pwm wave using gpios to control a servo motor).

But I do understand the complexity of what you are trying to achieve and wish you luck with it because I think you have a huge maintenance task ahead: just as maintaining Xenomai’s Cobalt over kernel releases for all the CPU architectures is not straightforward, I’d assume maintaining AndroidRT wont be simple either.

#10

@Ben, kernel needs to be modified for Xenomai and for PREEMPT_RT. There is no difference. The only question is whether you need any of the Android API. Then you have to start modifying the platform itself to make the changes in the kernel visible on the application layer. This costs time, but gives you the power of real-time processing in any Android app.

Besides, there is a difference between porting and compiling. The first process requires a lot of work and can be done an any kind of hardware. To compile the result, you better have something powerful. The first compilation of Android N takes about 4 hours and 50 GB space (plus 50 GB more for caching) on a quad-core i7 with 16 GB RAM and SSD. Subsequent compilations don’t need that much time, but it depends on the changes to the source code.

I never tried to port Xenomai for Android and I’m not familiar with its architecture. But every Android device typically has its own (device-specific) Linux kernel with device-specific drivers. For having a solid real-time performance you usually have to modify them too. Merging branches on a high level sounds to be a little bit too easy for me.

#11

@jro, Even if a part of the system should become real-time capable, the Linux kernel has to provide some kind of real-time support. Or you can go with co-kernel scenarios (which Xenomai seemed to be used for). Thats why I currently don’t fully understand why it should be used on the top of PREEMPT_RT. If your demo for ELC 2016 is publicly available I would love to learn more about it. I assume the main logic was in some JNI parts / native code? Did you deploy PREEMPT_RT or a second kernel? How good was your PWM performance? What about interrupt/scheduling latencies?

You are right, that typical scenarios don’t require a hard real-time system. But they usually do require a lot of expert knowledge about Linux kernels / real-time systems / scheduling / interrupts / kernel modules / memory issues and specific tools / solutions which must be used to get there. It’s not a simple task to create a general purpose real-time platform with a simple API, that’s true. But I hope it will allow a lot of people to start working on cool hardware projects if they know how to create an Android app. Even if they never heard the word “real-time” before.

#12

So, here’s my noobs takeaway from everything so far (Please correct me of I’m wrong):

  • To enable any kind of Real Time processing, the (device specific) kernel must be modified. Either A. With a secondary RT kernel companion (Xenomai cobalt) or B. By integrating RT into the kernel (Xenomai mercury/PREEMPT_RT - the basis of RTAndroid).

  • The benefit of RTAndroid is to allow any (or most) Android APIs to run with RT priority. Without RTAndroid RT applications are run in a separate runtime as C programs that are called by an Android app using the JNI.

  • RTAndroid is a modified Android OS that sits on top of a PREEMPT_RT modified kernel. As such a decent knowledge of RT processes and kernels is an absolute must before tackling a port attempt.

Assuming all of that is correct I’m thinking getting the Xenomai cobalt dual kernel setup would be the easiest first step, and documenting it as best as possible for anyone who wants minimal effort RT. Then modifying the Dragonboard kernel more extensively to use PREEMPT_RT, first off with the separate RT runtime and glibc setup, then attempt to port RTAndroid. Is that a logical way to try this?

It feels like I’m going to be the guinea pig for making this available to developers with little or no knowledge of RT development. If I can do it then anyone can lol. Please bear with me in the mean time.

@jro I’d also love to see the source code for the demo you made, coming from an Android dev perpespective it would be great to see a working app example.

#13

@Ben, in general that’s correct. Patching the kernel is preferable, but isn’t required in the first step. The core of RTAndroid (the Android part) already provides a significant scheduling improvement. Here is an example of a board we are currently working on (kernel is not patched yet). We have a preinstalled app for latency evaluation (smaller values are better):

Latency 1
Latency 2

You also get shorter (and more predictable) calculation times, which is crucial if you need reliable control:

Calculation 1

With a full Android source tree, integration into RTAndroid doesn’t take too long. I think I could do it in a couple of days. I just don’t have free time for reading tutorials.

Starting with Xenomai would be one option. I can’t tell how difficult that is and how much you will gain in terms of reliability. And it’s still unclear why you should use Android if you don’t use any of its interfaces and the main logic is compiled to a native binary.

#14

@Igor Wow, so just to be clear “basic RT support” in those benchmarks is RTAndroid but no modified kernel, and that offers 80-90% of the latency benefits? In your experience would you say porting RTAndroid alone would be an easier first step than patching a kernel?

As an Android developer I’d much rather use RTAndroid and develop all my RT code inside the Android framework. If it’s beyond me to port RTAndroid then a native code library would be the next best solution, but the biggest advantage I saw in RTAndroid is to be able to leverage my Android experience instead of getting back up to speed in C and learning the low level APIs.

My laptop runs a quad core i7 with 16GB of ram, though no SSD, so maybe building the OS would be time consuming but not impossible. Have to see if I have the storage space free though.

Would you have the time to guide me through the port? Assuming I can get the OS source code downloaded and building.

#15

@furiousben

The HiKey 4.1 kernel I used for the Android demo

The Xenomai tree (userspace)

The userspace has a gpiopwm executable that can be accessed via adb or via UDP (if you wish to control your motors from a server somewhere instead of the Android screen).

You then need to install glibc on the Android target making sure the dynamic linker can be found (there are a number of documents on the web on how to do that); you will also have to install the Xenomai libaries on the Android rootfs.

I wrote some notes back then for private consumption which I’ll paste here (maybe things can be done differently in a better way, I dont know)

[...]

$ cd aosp
$ mkdir rootfs-glibc
$ mkdir rootfs-glibc/rootfs
$ cd  rootfs-glibc

1 Add makefile:
===============

[jramirez@titan rootfs-glibc]$ cat Android.mk 
LOCAL_PATH:= $(call my-dir)
include $(CLEAR_VARS)

# This part is a hack, we're doing "addprefix" because if we don't,
# this dependency will be stripped out by the build system
GLIBC_ROOTFS := $(addprefix $(TARGET_OUT)/, rootfs-glibc)

$(GLIBC_ROOTFS):
        mkdir -p $(TARGET_OUT)/glibc
        cp -af $(TOPDIR)rootfs-glibc/rootfs/* $(TARGET_OUT)/glibc

ALL_PREBUILT += $(GLIBC_ROOTFS)

2. populate rootfs (can add detail later)
===========================================

- the aarch64 dynamic linker is ld-linux-aarch64.so.1 (as per the aarch64 elf files)
- copy the libs from linaros 16.03 release (or generate your own with debian tools)
- install busybox if you want..

3. modify the build so it steps into rootfs-glibc
=================================================

[jramirez@titan build ((37aa85e...) *)]$ pwd
/home/jramirez/fssd/hikey/android/build

[jramirez@titan build ((37aa85e...) *)]$ git diff
diff --git a/core/legacy_prebuilts.mk b/core/legacy_prebuilts.mk
index f4633d0..8e9bfed 100644
--- a/core/legacy_prebuilts.mk
+++ b/core/legacy_prebuilts.mk
@@ -26,6 +26,7 @@ GRANDFATHERED_ALL_PREBUILT := \
        ime \
        input \
        monkey \
+ rootfs-glibc \
        pm \
        RFFspeed_501.bmd \
        RFFstd_501.bmd \

4. modify system/core to add executable permissions and create symlinks
=======================================================================

[jramirez@titan core ((6c03e31...) +)]$ git diff HEAD
diff --git a/libcutils/fs_config.c b/libcutils/fs_config.c
index 6d50adc..415edcc 100644
--- a/libcutils/fs_config.c
+++ b/libcutils/fs_config.c
@@ -151,6 +151,18 @@ static const struct fs_path_config android_files[] = {
     { 00755, AID_ROOT,      AID_SHELL,     0, "vendor/bin/*" },
     { 00750, AID_ROOT,      AID_SHELL,     0, "sbin/*" },
     { 00755, AID_ROOT,      AID_ROOT,      0, "bin/*" },
+    { 00755, AID_ROOT,      AID_ROOT,      0, "system/glibc/lib/*" },
+    { 00755, AID_ROOT,      AID_ROOT,      0, "system/glibc/bin/*" },
+    { 00755, AID_ROOT,      AID_ROOT,      0, "system/glibc/sbin/*" },
+    { 00755, AID_ROOT,      AID_ROOT,      0, "system/glibc/usr/bin/*" },
+    { 00755, AID_ROOT,      AID_ROOT,      0, "system/glibc/usr/sbin/*" },
+    { 00755, AID_ROOT,      AID_ROOT,      0, "system/glibc/usr/bin/*" },
+    { 00755, AID_ROOT,      AID_ROOT,      0, "system/glibc/usr/sbin/*" },
+    { 00755, AID_ROOT,      AID_ROOT,      0, "system/glibc/usr/lib/*" },
+    { 00755, AID_ROOT,      AID_ROOT,      0, "system/glibc/usr/xenomai-cobalt/bin/*" },
+    { 00755, AID_ROOT,      AID_ROOT,      0, "system/glibc/usr/xenomai-cobalt/sbin/*" },
+    { 00755, AID_ROOT,      AID_ROOT,      0, "system/glibc/usr/xenomai-cobalt/lib/*" },
+    { 00755, AID_ROOT,      AID_ROOT,      0, "system/glibc/usr/xenomai-cobalt/demo/*" },
     { 00750, AID_ROOT,      AID_SHELL,     0, "init*" },
     { 00750, AID_ROOT,      AID_SHELL,     0, "sbin/fs_mgr" },
     { 00640, AID_ROOT,      AID_SHELL,     0, "fstab.*" },
diff --git a/rootdir/init.rc b/rootdir/init.rc
index aa32343..d631299 100644
--- a/rootdir/init.rc
+++ b/rootdir/init.rc
@@ -38,6 +38,12 @@ on init
     # Backward compatibility.
     symlink /system/etc /etc
     symlink /sys/kernel/debug /d
+    symlink /system/glibc/bin /bin
+    symlink /system/glibc/lib /lib
+    symlink /system/glibc/lib /lib64
+    symlink /system/glibc/usr/xenomai-cobalt/lib /xeno.lib
+    symlink /system/glibc/usr/xenomai-cobalt/bin /xeno.bin
+    symlink /system/glibx/usr/xenomai-cobalt/demo /xeno.demo
 
     # Link /vendor to /system/vendor for devices without a vendor partition.
     symlink /system/vendor /vendor

5. TODO
=======

- devtmpfs is not being created and mounting it after boot seems to cause problems
  (xenomai RTDM workaround:
    - cd /dev
    - ln -s /dev/ /dev/rtdm
  
- LD_LIBRARY_PATH needs to setup manually

[...]

I hope that helps.

#16

@Ben

Both setups (“basic real-time” and “advanced real-time”) are evaluated on the same unmodified kernel. Separate test cases are executed in the same run, but with different parameters (thread priority, cpu control, …). Our benchmark application is btw open source and is available on GitHub.

RTAndroid provides you a benefit in the scheduling latency, but without a patched kernel it cannot be guaranteed during a long term run. This basically means that if you start doing other things like playing games or watching HD videos on youtube in the same time, your real-time process will experience latency peaks. But you already gain a significant improvement in process behavior (on average) and those problems are not relevant for your application anyways.

Looking at jro’s example it seems that you will have to modify the Android platform in any case. As platform components of RTAndroid are not open source, I will help you with that. The first step would be setting up the build environment, compiling your own (unmodified) Android for Dragonboard, flashing and booting it. This is the most time consuming part for me, as it differs from device to device. After this I can jump in and replace/patch required components, compile it based on your documentation and provide you a ready-to-go platform image. RTAndroid is free for personal use and for research / evaluation purposes.

@jro

Thanks, I bet a lot of developers will appriciate this tutorial. It seems to require solid knowledge about Android / Linux / Xenomai / native development. And it is probably the right way for people who know what they are doing, as it gives you the full control of the system. I just hope RTAndroid can open a door into this magic world (or parts of it) for “normal” Android apps.

#17

Thanks @jro, that code will certainly be a help to look through, gonna take some time for me to make sense of it but certainly helps to have you guys who have already made it work. Did you modify the OS source code or are all the commands you have there on the target device and kernel build?

@Igor I definitely think you’re right, if we can get me through this it should help a lot of developers make sense of this and take away some of the anxiety and fear of modifying the OS and kernel.

I have the free space on my laptop, so fingers crossed it’s fast enough and I’ll get to setting up the tools and source code to build the OS.

#18

Well… shit. No android build support for Windows. That’s gonna be a pain. Looking into Builduntu to run in a VirtualBox, gonna slow things down some more but I don’t want to dual boot if I can help it. Might mean it’s a more one-fits-all solution for others if it does work though.

#19

@Ben Never heard of Builduntu before. But you won’t be the first compiling Android in a VirtualBox. If you give it enough resources it shouldn’t take much longer. Let us know about your progress.

#20

Hey guys,

I’ve got builduntu installed and running, and it looks to be a quick and easy way to get a build environment ready. These are the exact commands from this video to pull the AOSP source code. The CodeAurora wiki referenced in the video is also very helpful explaining the process.

The main ones to care about are:
Initialize the repo in the current (or home?) directory
repo init -u git://codeaurora.org/platform/manifest.git -b release -m {Manifest} --repo-url=git://codeaurora.org/tools/repo.git

Pull the source code
repo sync -j{download threads to run simultaneously}

(I found a nice explanation of the specific commands here)

I used LA.BR.1.1.2-01420-8x16.0.xml as the {Manifest} file specific to the Dragonboard, which you can find in this list from the video linked above. What is confusing though is that msm8916_64 is the chipset listed for that manifest, yet in the video the Qualcomm engineer refers to the Dragonboard chipset as apq8618, not sure why they’re different.

I have had an issue trying to download the Dragonboard only projects though. At the moment the entire repo is downloading, which takes an age (currently at 8% after about 7 hours or so over the last few days), but I’ve seen you can download specific projects only using this:
repo sync {project}

In the codeaurora repos I’ve found (what I imagine is) the dragonboard project; device/qcom/dragonboard, but adding that path in place of {project} fails with a no project exists error.

Any ideas how to download the dragonboard specific project, rather than the whole source code? Also what other projects are also needed as surely that one alone isn’t enough. Would be great to show people to download the minimum code needed rather than the whole repo.

I also found this wiki to build the boot image, thought I’d post it here for down the road: