Kayak Point Official 1.0 Release

persistence-300x261

image source: Amanda

Persistence.  Making data persistant in your mobile applications without writing a custom SQLite query engine has been a goal of mine for a while.  I finally got around to writing a library that will take care of that work for you:

Kayak Point

This is an ORM Library for Android written from scratch. This library has the following goals:

  • be built for Android primarily
  • be fast and lightweight
  • support SQLite at first, and other data stores in the future.
  • stick to a simple architecture and design
  • use as little code as possible.
  • experiment with different approaches to persistence on Android.

I was originally using ORMLite, and I quickly realized that library is built to support multiple platforms, Android being one of them.  That means the library is big.

So I decided to take a stab at writing an ORM library for Android that is more optimized for persisting Java objects to SQLite on that OS.  But I also wanted to create a library that would let me add on additional kinds of data storage later (e.g. noSQL).  The end result is Kayak Point.

Pend Oreille Official 1.0 Release

Celebration

Happy to announce that version 1.0 of the Pend Oreille library has now been released.  You can check it out here:

Pend Oreille Library on Google Code

This project contains some simple utilities to perform operations Java primitives, like data marshalling. It also has some utility functions to work on the Java class model using reflection. This is compiled against the Android 1.5 platform so it will work on Android as well as J2SE.
The Pend Oreille library is very simple and small (under 20kb in size) and has a few basic features that I have always wanted to have:

  • serialize Java primitives and primitive arrays to byte arrays.
  • deserialize java primitives and primitive arrays from byte arrays.
  • box and unbox primitive arrays
  • serialize an array of boolean values into a packed bit field.

See the file READE.md in the source directory for a developer’s guide on how to use the library.

Pre-Release Announcement: Pend Oreille

gema_01_img0091

Pend Oreille, WA was named after a French Canadian word. source: everyculture.com

I’m happy at long last to announce the pre-release version of a new Android library I wrote:

http://code.google.com/p/pend-oreille/

The release candidate is on the branch called “release-candidate“.

Being a release candidate means the code is far enough along to give it a preliminary try, but not yet ready to be used in production applications. It has passed some basic development smoke testing.  Being pre-release also means that the code and API are subject to change until it’s merged into the master branch for the official release.  You can also expect some things like javadocs and a guide to be missing. In other words – your mileage will vary.

The project is called Pend Oreille (pronounced pond-o-ray), in honor of a little area of Washington State by the same name.

The Pend Oreille library is very simple and small (under 20kb in size) and has a few basic features that I have always wanted to have handy, without always having to use DataOutputStream objects:

  • serialize Java primitives and primitive arrays to byte arrays.
  • deserialize java primitives and primitive arrays from byte arrays.
  • box and unbox primitive arrays
  • serialize an array of boolean values into a packed bit field.

Yep. That’s it.

Gradle Schmadel

The new Android IDE is moving to Gradle as it’s build system. I’ve been using and designing build systems for 20+ years.  If you can name a build system I’ve probably used it.  If not, most likely some derivation of it.

Here’s the thing about build systems:

  1. at the end of the day, you’re running command line operations. Compilers, linkers, etc.
  2. all other aspects of the build system must support #1.
  3. if it’s not #1 or #2 it’s either a waste of time or will cause confusion.

There are many things you can do to support how efficiently you can do #1.  For example, you can use ANT to copy files around.  No matter what, you will always come back to : compile -> link -> package -> install.  And those are command line operations.

With that in mind I peek at Gradle.

The First Build Script

Chapter 6 of the Gradle manual has the basic build script.  It’s very telling:

build.gradle

task hello {
    doLast {
        println 'Hello world!'
    }
}

A few things jump out at me right away:

  1. defining a task means people will be defining task dependencies.  In a build system dependencies exist between files.  Period.  So it’s already clear that Gradle is addressing this by some round-about way.
  2. the notion of doLast tells you build commands are executed in some arbitrary order, but not from top to bottom of the build script.
  3. doLast also means that there are function dependencies. You’ll be building up a mental map of function calls in a build system.
  4. the build script language is seems as lexically complicated as a source code language. There’s not much different between that and the Java Hello World application, for example.
  5. it takes five lines of code to output of “Hello World.”  That’s four lines more than a a UNIX script takes:
echo Hello World!

Will Gradle compensate for these five things in some way?

Time will tell.

Reading Properties Files in Android

This yet-to-be-self-identified blogger from Sweeden knows a thing or two about reading files from Android:

Reading Properties Files on Android (Investigation for microlog4android)

For reference, I’m reproducing a snippet of code he posted.  This will read from the /assets directory:

Resources resources = this.getResources();
AssetManager assetManager = resources.getAssets();

// Read from the /assets directory
try {
    InputStream inputStream = assetManager.open("microlog.properties");
    Properties properties = new Properties();
    properties.load(inputStream);
    System.out.println("The properties are now loaded");
    System.out.println("properties: " + properties);
} catch (IOException e) {
    System.err.println("Failed to open microlog property file");
    e.printStackTrace();
}

And this will  read a properties file from the /raw directory:

// Read from the /res/raw directory
try {
    InputStream rawResource = resources.openRawResource(R.raw.micrologv2);
    Properties properties = new Properties();
    properties.load(rawResource);
    System.out.println("The properties are now loaded");
    System.out.println("properties: " + properties);
} catch (NotFoundException e) {
    System.err.println("Did not find raw resource: "+e);
} catch (IOException e) {
    System.err.println("Failed to open microlog property file");
}

Adding shared libraries to your Android NDK Build: Insanity

Stressful Guy

The Android NDK build system may make you want to choke something. (source: the eating disorder blog)

I’ve been pulling my hair out for the past couple days dealing with Android NDK build sensitivities.  I hope this blog posting saves you some time and stress. If it does, you can send me flowers …. or money.  Or both.  It’s all good.

So, let’s say you have an existing Android NDK project and it works file.  It’s complete.  You want to add a shared library into the application, but it’s not required to build your application. That is, you need it during runtime, but you don’t need it to build your C/C++ code using NDK.

It should be simple, right, but the documentation for the Android NDK seems to suggest that you have to use LOCAL_MODULE and friends to accomplish this.  But you’ll find that can be recipe for disaster.  I sure did.   It’s been tormenting me for the past several days and I think I finally solved the problem.

This blog presents one way to accomplish this.  This is something I just discovered (by spending hours ruling out everything that didn’t work – modifying module definitions in Android.mk files, using APP_MODULES, and setting up module dependencies – you name it, I’ve tried it) which I have not seen explicitly discussed about in the existing android documentation.

The Goal

The goal is simple. You have an existing NDK project and you want to:

  1. add a shared library to your Android NDK build that gets packed up into your .apk file. The library is one that is not needed to compile any other C/C++ module in your project. But it’s one that your application will rely upon indirectly.  You just need it to be packed with the build.
  2. Load that library at runtime with a call to
System.loadLibrary("foo-user");

The Problem

If you read the documentation, you’re going to realize two things: a) the Android NDK build system is built on top of gmake, and b) it relies very heavy on makefile fragments. You’ll see things like this, from PREBUILTS.html (from the Android NDK docs directory):

II. Referencing the prebuilt library in other modules:
------------------------------------------------------

Simply list your prebuilt module's name in the 
LOCAL_STATIC_LIBRARIES or LOCAL_SHARED_LIBRARIES declaration 
in the Android.mk of any module that depends on them.

For example, a naive example of a module using libfoo.so would be:

    include $(CLEAR_VARS)
    LOCAL_MODULE := foo-user
    LOCAL_SRC_FILES := foo-user.c
    LOCAL_SHARED_LIBRARIES := foo-prebuilt
    include $(BUILD_SHARED_LIBRARY)

You might be thinking that your shared library will just be pulled into the build.  But then when you run the build you suddenly get linking errors for your application that was working properly before.

You find that not only is your build failing due to linking errors, your shared library is not getting incorporated into the .apk file.  That’s what happened in my case.

Here’s Something That Works

I was able to accomplish this without build errors, after several days of fighting with the build, by using the following approach:

1. Do NOT add a module to your Android.mk make files.  Leave that out.  Apparently (and this is just a guess), you only should define prebuilt shared modules in your build file that your native application actually needs to be linked against.

2. Build your shared library in its own project.  Make sure the code is cross-compiled for the platform of Android you want the library to run on.  At this point you have two separate, independent projects: Your Android project with native code, and your shared library somewhere on the filesystem.

3. With your native project built, you’re going to see a libs directory at the top level of your project.  In that directory, you’ll see something like this:

MyProject/
     libs/
         armeabi-v7a/
             libfromyourndkproject.so

4. Copy your shared library into the directory that has the name of the architecture you’ve compiled it for.  So, if you’ve built libfoo-user.so for armeabi, you’ll want something like this;

MyProject
     libs
         armeabi-v7a
            libfromyourndkproject.so
            libfoo-user.so

With that setup you should see your shared library as part of your apk build file.  You can use

jar xvf YourApplicaton.apk

to verify it’s there.

There should be no mysteries, unanswered questions, or undocumented use cases when it comes to something as critical as a build system. Without that, your project can come to an untimely screeching halt.

Facebook Patch for Dalvik

If you can solve a problem by re-engineering your app or modifying Android, why would you modify Android?  That’s the burning question at the heart of this blog entry.

Picard to Warf: "Android 2.3 was written that way for a reason, Mr. Warf." (credit: appsgeyser.com)

Picard to Warf: “Android 2.3 was written that way for a reason, Mr. Warf.” (photo credit: CBS/startrek.com; blog source: Paramount)

Engineering mobile applications is embedded engineering.  It’s the art of using less code to accomplish something rather than more code.  And in this case, it’s also the art of avoiding a change that risks impacting every Android user that uses Android 2.3, and every carrier with Android 2.3 phones.

Recently, David Reiss, a Facebook engineer, posted on the Facebook blog information about a Dalvik “patch” that they were working on.  Interesting. My Android Spidey Sense tells me that the problem is more of an application issue, as opposed to an issue within Android OS itself.  So, I decided to find out with some quick analysis.

I’ve been working on Dalvik the past year, so you could say I’m a Dalvik “virtual machinist.”  This engineering problem is pretty juicy, and I couldn’t pass up looking at it.  Here’s the Facebook blog posting, and a quick summary of the issue David is seeing.

Under the Hood: Dalvik patch for Facebook for Android

The suggested fix does not consider that if Android 2.3 is updated from 5MB to 8MB, it may force all mobile phones using Android 2.3 to go through a round of regression testing.  That’s a lot of work for the carriers.  Depending on the hardware configuration on Android phones that run 2.3, the potential side effects far outweigh a more straightforward solution: Facebook simply re-engineers its application.  But that aside …

The issue that is reported by Facebook and others is not a “bug” in the sense that dexopt is crashing.  Nor is it the case that the failure is created by “deep” interface hierarchies, as the Google bug suggests.  In this case, dexopt is simply complaining that the application that Android is trying to load is too big: the code size is too big for the class loader to handle.  This kind of limit is nothing new to Android or any other mobile app.  It’s been there ever since mobile phones could run applications.  So, it could be said that this is an application level problem, as opposed to an OS problem.  This could be resolved using two approaches:

  1. re-engineer the application, especially in cases where you are trying to port code from one form (Javascript) to another (Java).
  2. modify the operating system to utilize more resources – which will affect all applications

For reasons I explain below, using #2 is a slippery slope.  It’s important to remember that 5MB is a lot of memory to  chew on for a device that was designed primarily to answer phone calls. And, as you’ll see, it’s not just 5MB versus 8MB anyway. The difference is potentially 10MB versus 16MB.  And, that my coder roadies, is a whole other ballgame.

Other Factors

The Facebook blog mentions that this limit was bumped into because code was ported from Javascript to Java.  So, this occurred when Facebook decided to add more features to their native Android application.

The Quick Conclusion/Upshot

The basic problem being reported is that when the Android phone loads your program to run, there are limits placed on the amount of code you can have in your application.  That’s a good thing.  If you exceed your limits, it’s much wiser to re-think your application design as opposed to tweaking the OS.

First of all, let me dis-abuse you of the notion that if dexopt fails in this way, it must be a “bug” in the Android platform.  Android was pretty thoroughly tested, as far as mobile operating systems go, so it’s just important to consider that your application design might just need to be re-worked.  In this case, while it appears that a Google engineer has signed on to address this “issue”, it doesn’t mean that Google thinks it’s a bug either.

The limit that is relevant here is a 5MB limit placed on all Android applications for Android 2.3.  These kinds of limits are there for a reason.  This is especially important to respect in the case of Android 2.3 because devices that run that version of the OS tend to be more limited on resources than Android 4.0 devices.

Bumping up the amount of memory that LinearAlloc uses will increase the amount of memory that ALL android applications consume on the outset when loading your classes. Each application that starts will have this amount of memory allocated to the class loader.

So, if you have 20 apps on your phone, each of those apps are going to allocate that amount of memory (5MB on Android 2.3, and 8MB on Android 4.0).  This is very important to consider because Android allows any of its applications to start background services.  For planning purposes, you must take into account the fact that every one of the apps on your phone may be running simultaneously because they may be running services.

More Details and Analysis

Here are the basic working of how LinearAllocHdr (LinearAlloc) is managed by Android:

LinearAllocHdr is a data structure in Android that so far appears to be used only by the class loader.  But, nothing says it can’t be used for other things in the future.  Here’s the structure:

28/*
29 * Linear allocation state.  We could tuck this into the start of the
30 * allocated region, but that would prevent us from sharing the rest of
31 * that first page.
32 */
33typedef struct LinearAllocHdr {
34    int     curOffset;          /* offset where next data goes */
35    pthread_mutex_t lock;       /* controls updates to this struct */
36
37    char*   mapAddr;            /* start of mmap()ed region */
38    int     mapLength;          /* length of region */
39    int     firstOffset;        /* for chasing through */
40
41    short*  writeRefCount;      /* for ENFORCE_READ_ONLY */
42} LinearAllocHdr;

mapAddr points to the block of memory that is allocated.

This structure is instantiated by the function dvmLinearAllocCreate.   The 5MB that David’s post talks about is actually a 5MB file that gets memory mapped. The length of the file is defined by DEFAULT_MAX_LENGTH:

/* default length of memory segment (worst case is probably "dexopt") */
72 #define DEFAULT_MAX_LENGTH  (5*1024*1024)

Memory mapping files is commonly done in Android.  In fact Dalvik maps your entire application code into memory this way.  This means two things for the problem at hand:

  1. 5MB of disk space is used to store the underlying data, and
  2. 5MB of memory is taken up to store the file’s contents in active memory.

So, the impact on the operating system is actually 10MB (worst case). So, we’re not just talking about 5MB here, we’re talking about potentially using twice that.  If you increase that to 8MB, you’re impacting OS with potentially a 16MB memory allocation.  Now, we’re getting into some serious memory for a mobile device to manage.  Remember, it’s an embedded system on a slow processor – especially in the case of Android 2.3 and OMAP.

dvmLinearAllocCreate is called by dvmClassStartup (Class.c), so that confirms that the only place that this is used is by the class loader (for now).  But, this is a very critical part of memory. The more memory used by the class loader, the more overhead you cause for Linux, and the slower your applications might boot up.  Again, perhaps not noticeable on Android 4.0, but it might be noticed on an Android 2.3 device – especially a cheap one that uses lower end hardware.  Especially when that is applied to all applications that run on the device.