Cool Monday: Exploration of dynamic db acces from scala

I use scala on Android and I don't like the integrated database API. It's very verbose and very stateful. I had written my own ORM(DAO would be a more appropriate tag) a while back, before I used scala but it's not enough anymore. So now I'm on a quest for a better database API. My dream is something small that handles schema for me and is type-safe. A nice DSL that is converted to SQL at compile time  and does code generation. So it's fast like hand writing everything. But reduces code footprint by an order of magnitude(at least). Scala SLICK looks promising. It fits most requirements. But it's kinda big for android projects(you need scala library too!) and has not yet hit a stable version so I wouldn't be comfortable shipping it. Will definitely give it a thorough test when scala 2.10 is stable and SLICK is released. Oh, and it needs a third party JDBC  driver for Android. This is another level of abstraction and therefore another source of slowness. I contemplated writing my own clone targeted at Android but   never came around to actually doing it(yet!). It seems like a herculean task for single developer working in spare time.

Meanwhile

Yesterday I stared thinking how dynamic languages handle databases. And I got an idea. Scala has type Dynamic that does compilation magic to provide syntactic sugar for working with dynamic languages or objects. Here's an idea: do queries in plain SQL and perform extraction of data in a dynamic way. 

And how to do this? Just wrap up Cursor to provide necessary methods. 

class WrappedCursor(cursor: Cursor) implements Cursor{
  //delegated methods go here
}

Why I need this? Cake pattern of course, Dynamic cursor get's mixed in.

trait DynamicCursor extends Dynamic{ this: Cursor =>

  def selectDynamic(name: String) =
    getColumn(getColumnIndex(name))

  def getColumn(index: Int) =
    getType(index) match {
    case Cursor.FIELD_TYPE_BLOB => getBlob(index)
    case Cursor.FIELD_TYPE_FLOAT => getDouble(index)
    case Cursor.FIELD_TYPE_INTEGER => getLong(index)
    case Cursor.FIELD_TYPE_NULL => null
    case Cursor.FIELD_TYPE_STRING => getString(index)
  }

  def toSeq = (0 until getColumnCount) map getColumn
}

I targeted API level 14(Ice Cream Sandwich) since getType(method on Cursor) is available from 11 on.    Key method here is getColumn that abstracts over types. So you can read a column and  do pattern matching on it. Or you are evil and use implicit conversions from Any to String, Long etc... Or use implicit conversion to "converter"

implicit class Converter(val value: Any) extends AnyVal{
  def blob = value.asInstanceOf[Array[Byte]]
  def double = value.asInstanceOf[Double]
  def long = value.asInstanceOf[Long]
  def string = value.asInstanceOf[String]
}

But the real deal is selectDynamic. This allows you to write code like this

val c = new WrappedCursor(result) with DynamicCursor
c.someColumn.long

This compiles down to selectDynamic("someColumn") that calls getColumn and finally implicit conversion is inserted that allows for terse cast to Long.
And I threw in a conversion from row to Seq that does a snapshot of current row. This allows pattern matching on rows. Any you can now construct a Stream that will handle Cursor state and lazily evaluate and store these snapshots. Therefore you can abstract away all mutability and handle cursor as immutable collection.

Said conversion to stream

def CursorStream(cursor: DynamicCursorRaw with Cursor) = {
  def loop(): Stream[Seq[Any]] = {
    if(cursor.isAfterLast)
      Stream.empty[Seq[Any]]
    else {
      val snapshot = cursor.toSeq
      cursor.moveToNext()
      snapshot #:: loop()
    }
  }
  cursor.moveToFirst()
  loop()
}

And some more implicits to help

implicit class RichCursorRaw(cursor: Cursor) extends AnyVal{
  def dynamicRaw = new WrappedCursor(cursor) with DynamicCursorRaw
  def toStream = CursorStream(dynamicRaw)
}

All the source is in the project on github https://github.com/edofic/dynamic-db-android (work in progress).

Related articles

• Hunt for a web framework that works
• Status update

Setting up for scala development on Android

Image via CrunchBase

Scala (programming language) (Photo credit: Wikipedia)

I've been developing for android more than a year now and a few months in scala. So naturally I wanted to combine the two. But it's not dead simple. This is kinda a tutorial an a reference if I ever forget how to do this. It took me a few days to figure it all out. I tried maven, ant with special config and sbt(I need to learn more about this one) but in the end I just wanted fast solution integrated into my IDE.

So I use IntelliJ IDEA community edition for my IDE.  You should check it out, it's totally awesome. It's primary a Java IDE but scala plugin rocks. It offers some more advanced text editing capabilities, not like vim or emacs but enough for me. It also brings up coloring and editing features that are language aware. So you have a shortcut(Ctrl-W) to select semantically valid block. And press it again to expand to next bigger valid piece of code. And stuff like that. Real-time structure view is nice and there are some cool refactorings. But scala REPL is where fun begins. You get your module classpath pre-set and you get full editor capabilities in REPL. Enough with advertisement(they didn't pay me to do this) and let's get to work. 

Prerequisites

• JDK...duh!  I use OpenJDK 7, IDEA gives some warnings but it works like a charm
• Android SDK and at least one platform
• IntelliJ IDEA
• scala distribution. I recommend you use latest stable release from here 

Setting up

First install scala plugin. It's quite straightforward. Plugin Manager->Browse repos->search for scala->select->ok.

Now actual setting up. I use global libraries for all my projects, you can also put these into just Libraries and to that on per-project basis.

Open project structure(no project open) and go to Global Libraries. You need to create two libraries containing jars from <path to scala>/lib/.

First scala-compiler with scala-compiler.jar and scala-library.jar and then scala-library with scala-library.jar and anything else you might need. Reason for scala library in compiler is that compiler also relies on scala lib. I needed quite some time to figure this out.

This whole process can be automated if you add scala to your project when creating it but it's not possible with android so you need to know how to do it by hand. 

Creating a project

• Project from scratch
• add android module and configure it
• now go to project structure. add scala facet to this module and go to its settings and set the compiler jar.
• back to module and add dependency to global scala-library
• set dependency to provided. This is important. Else it will try to dex whole library and you'll end up with "too  many methods error".

Now your project should compile. But not run.

Running

Obviously not including scala library in the build means you need to provide it in another way. For developing on emulator I customized it to provide predexed scala library. Excellent tutorial.

In a nutshell

$ git clone git://github.com/jberkel/android-sdk-scala.git
$ cd android-sdk-scala
$ ./bin/createdexlibs

$ bin/createramdisks

$ emulator -avd ... -ramdisk /path/to/custom.img
$ adb shell mkdir -p /data/framework
$ for i in configs/framework/*.jar; do adb push $i /data/framework/; done

And reboot.

There is also (not so trivial) part about pathcing a device. However you can try Scala Installer from Play to do this. I had some success and some failures.

Now the app should run on your device.

Deploying

Well it doesn't work on other devices right now. For export you need to change scala-library dependency back to compile to include it into the build. Trick now is to enable ProGuard to remove unnecessary methods and classed to fit the jar through dexer. You do this in Tools->Android->Export. Select ProGuard and your config. I got mine from jberkel's repo. That's it. Sadly this export takes quite some time. Scala's standard library is not a piece of cake afterall(actually it is a cake). Minute and a half on my machine for small apps. So I only to this for testing on other phones and deployment.

Faster compilation

Compiling with scala-libray set to provided is much faster but not fast enough for me. I want to be doing stuff not waiting for it to compile. 

Turns out compiler is the big time sucker(and I'm being Capt. Obvious). Afterall scalac is not known for it's speed.

Enter FSC or Fast Scala Compiler. This is a scala compiler running in the background having everything preloaded and just does incremental compilation. It even comes with standard scala distribution and is supported by IntelliJ IDEA. Great. 

To set it up just head over to Project Structure->Scala facet and select Use FSC. And then immediately click Setting to access Project Settings and set compiler jar for the compiler.

Success. Scala builds are now on par(or even faster!) than java ones. 

No more fencing for me.

Related articles

• Alternative JVM languages
• Functional Programming Principles in Scala: Setting up IntelliJ
• Easily Cross-compile and Release Scala Projects With sbt
• Pretty function composition in scala and asynchronous function composition on android

Pretty function composition in scala and asynchronous function composition on android

Composition

Surjective composition: the first function need not be surjective. (Photo credit: Wikipedia)

Function composition is a nice way to sequence transformations on data. For example in a compiler you take your source, parse, check, optimize and generate code(in a nutshell). It's a linear series of transformations -> perfect for function composition
In Haskell you can use this beautiful syntax

compile = parse . check . optimize . codegen

leaving out the parameter(as it can be infered) and noting composition as "." which kinda looks like ° used in math(if you squint a bit).
In scala you could do something like

val compile = parse compose check compose optimize compose codegen

Nearly there, I just want it to look a bit prettier. (Compose is a method defined on Function1 trait)
So I define an implicit conversion from function to "composable"

implicit def function2composable[A,B](f: A=>B) = new AnyRef{
  def -->[C](g: B=>C) = v => g(f(c))
}

This creates an object and reimplements "compose" but I really like the syntax:

compile = parse --> check --> optimize --> codegen

Functional programming can be imagined as a waterfall of data, flowing from one function into the next, and the --> operator represents this nicely.

Let's go a step further. If the composition is one-off, and this is a waterfall of DATA it could be nice to represent that.
Something like

result = source --> parse --> check --> optimize --> codegen

So now I'm taking a value and sending it through black boxes. Very nice. Apart from the fact it doesn't work(yet!).

implicit def any2waterfall[A](a: A) = new AnyRef{
  def -->[B](f: A=>B) = f(a)
}

Scala's awesome compiler can handle two implicit conversions with same method names. Nice.
You can even mix and match

result = source --> (parse --> check --> optimize) --> codegen

This does the composition of parse, check and optimize into an anonymous function, applies it to the source and then applies codegen to it's result.

Goin async

Image via CrunchBase

What about asynchronous calls? Can I compose those too? I think it's possible with Lift actors(or with scalaz's?), but I needed to integrate that into Android's activities quite recently. Well I did not *need* to do it, but it was quite a nice solution.

The usual way of doing things async in Android is with the conveniently named AsyncTask. The problem is - you can't subclass it in scala because of some compiler bug regarding varargs parameters. Silly. 

So let's do a lightweight(in terms of code) substitution. We can "spawn a thread" using scala's actors. And activity can receive messages through a Handler. 

import android.os.{Message, Handler}

trait MessageHandler {
  private val handler = new Handler {
   override def handleMessage(msg: Message) {
     react(msg.obj)
   }
  }

  def react(msg: AnyRef)

  def !(message: AnyRef) {
   val msg = new Message
   msg.obj = message
   handler.sendMessage(msg)
  }
}

So in an activity that mixes in MessageHandler I can post messages to my self. An I can do it async since Handler is thread safe

def react(msg: AnyRef) = msg match {
  case s: String = Toast(this, s, Toast.LENGTH_LONG).makeText()
  case _ => ()
}

...
//somewhere inside UI thread
import actors.Actor.actor
val that = this
actor{
  val msg = doExpensiveWork()
  that ! msg
}
...

Not the most concise way to write it, but I believe the most clear one. Method doExpensiveWork is done in the background so it doesn't block UI and it posts the result back as a message.

Async composition - finally

What I want to do now is use function composition to do something like

(input --> expensiveOne --> expensiveTwo) -!> displayResults

In other words, do "waterfall composition" in background using some input I have now and post the result back into the UI thread to method displayResults. That should be the magic of the -!> operator. Do the left side in bg and post it to the right side async.
I need a new trait for that

trait MessageHandlerExec extends MessageHandler{ outer =>
  override protected val handler = new Handler {
    override def handleMessage(msg: Message) = msg.obj match {
      case r: Runnable => r.run()
      case other: AnyRef => react(other)
    }
  }

  implicit def any2asyncComposable[A](a: => A) = new AnyRef{
    def -!>[B](f: A=>B) = outer ! new Runnable{
      def run() = f(a)
    }
  }
}

The trick here is using by-name parameters in the implicit conversion. This delays the execution of a(which in example above would be a waterfall and moves it into a worker thread.

Related articles

• 3 Tips for Writing Performant Scala
• Making a programming language: Part 1 - how to start
• Does Category Theory Make You a Better Programmer ?
• Why Java developers should check out Scala
• Ten Things You Should Know About Haskell Syntax