Providing an error-free API for a heavily developed project is not an easy task. Likely, the first things that come to mind are tests. For a mid-sized API, you may write hundreds or even thousands of end-to-end tests. These tests significantly prolong build times.

In this post, we will explain how we solved long build times with CircleCI test parallelism and Gradle/Grails for Amio main service.

CircleCI setup

CircleCI’s documentation does a decent job of explaining how their command line interface (CLI) tool should be used to enable test parallelism. When I started looking into it for the first time, it wasn’t entirely obvious what the returned result would look like. I was asking myself, “So, I’ll run this command and it will magically start splitting my tests?” Well, of course not! The result is a list of test files that should be executed on a particular container.

Does that sound complicated? Let me explain in an example.

The first thing we have to do is to set the parallelism key in the .circleci/config.yml file. From the CircleCI docs:

The _parallelism_ key specifies how many independent executors will be set up to run the steps of a job.

Any value greater than one will enable parallel execution, but for the sake of this example, let’s go with two. This way, every time a CircleCI job is started, it will spawn two containers which will both do the same tasks.

If we were to use the parallelism key with no additional configuration, it would just run all of our tests twice. That is not what we want. We want to split our tests between the containers.

That’s where the CircleCI CLI comes in. It offers two commands which, when used together, split our tests into equal portions across our two containers.

Let’s say these are the test files in our project:

src/integration-test/groovy/com/package1/Test1.groovysrc/integration-test/groovy/com/package1/Test2.groovysrc/integration-test/groovy/com/package2/Test3.groovysrc/integration-test/groovy/com/package2/Test4.groovysrc/integration-test/groovy/com/package2/Test5.groovy

Naturally, we will have other source files in our project; not just our tests. They may be located in the same src/integration-test/… directory. To achieve our goal of test splitting, we need to select only the test files for the project. That is done by using the glob command:

circleci tests glob "src/integration-test/**/*.groovy"

This command will output the list of our tests (all 5 of them). ? Now we use the split command to, well, split them between containers:

circleci tests glob "src/integration-test/**/*.groovy" | circleci tests split --split-by=timings

The split command offers several strategies to split the tests but timings is my favorite. It uses the timings data that is collected by CircleCI (this has to be enabled via the store_test_results key) to split the tests into portions that take a similar time to execute. Container indexing is automatic. We can run the same command on every container. In our example, running the command on Container 0 might output:

src/integration-test/groovy/com/package1/Test1.groovysrc/integration-test/groovy/com/package2/Test3.groovy

And on Container 1:

src/integration-test/groovy/com/package1/Test2.groovysrc/integration-test/groovy/com/package2/Test4.groovysrc/integration-test/groovy/com/package2/Test5.groovy

I say “might” because the real result would depend on the timings data. As you can see, every container got its half of the tests.

Gradle setup

Splitting the tests in CircleCI was the easy part. The hard part is getting Gradle to execute just the tests that are in the result of the split command. If we were using JavaScript and Mocha, it would be much easier. Mocha accepts a list of files which should be executed. With Gradle 3, I had been using this command to run tests: ./gradlew check -i

Gradle’s documentation isn’t really helpful. Just figuring out what the check task does is a pain. Thankfully, it is possible to pass our test list as a parameter to the Gradle task.

./gradlew check -i -PtestFilter="`circleci tests glob "src/integration-test/**/*.groovy" | circleci tests split --split-by=timings`"

Now, when the check task is started, it has access to the testFilter parameter. To make everything work, we also need to add some code that can handle the parameter in our build.gradle:

integrationTest {  if (project.hasProperty("testFilter")) {    List<String> props = project.getProperties().get("testFilter").split("\\s+")    props.each {      include(it.replace("src/integration-test/groovy/com/", "**/").replace(".groovy", ".class"))    }  }}

Note that the parameter was passed to the task as a single string. In the code block above, Line 3 contains logic to split it back into rows. Calling include will tell Gradle to execute only the tests that we include. Now we can include all the rows, and we’re good, right?

Nope. Gradle doesn’t know how to work with source files. It only understands classes. We need to pass the compiled class files to it.

There are two problems with that. First, the compiled classes are not in the same directory. Second, the suffix is not .groovy but .class.

To overcome the first problem, we replaced the common prefix with **/. This says, “Look in the root directory and all its subdirectories.” Of course, you could replace it with something like build/classes/integrationTest/com. That is cleaner, but not necessary. This should be safe as long as the test classes names are unique. Line 5 in the code block above includes logic that solves both of these problems.

In the end, your .circleci/config.yml should look something like this (just the relevant part):

- run:    # This is just for debugging purposes, you can omit this step    name: test splitting output    command: circleci tests glob “src/integration-test/**/*.groovy” | circleci tests split --split-by=timings | xargs -n 1 echo

- run:    name: test    command: ./gradlew check -i -PtestFilter="`circleci tests glob “src/integration-test/**/*.groovy” | circleci tests split --split-by=timings`"

Conclusion

And that’s it! Easy, right? Well, it was a bit more work than it should have been. Having our test times cut nearly in half was definitely worth it! Applying test parallelism, we’ve decreased the build time from around 15 minutes to 9 minutes.

Developers will continue to add new features throughout an application’s lifetime. More code means not only longer build times — it means longer incremental build time.

For teams with big projects, waiting for new builds could eventually take up 10-15% of their workday. This not only wastes precious developer time — it also makes test-driven development extremely tedious, which hurts overall code quality.

Splitting an application into modules could be a solution to this problem. I was tempted to just split our codebase by feature, by layer, or in some other quick and obvious way. But first, I decided to create an experiment and collect some data, so I could make a better-informed decision. This article will explore the results I collected and my findings.

Before we dive into my experiment, I’d like to talk about the theory behind all this, and explain how we can decrease incremental build time.

A bit of theory

When you create an Android application you have to have at least one application module, it is a module that applied the Gradle application plugin in his build.gradle file:

apply plugin: 'com.android.application'

As a result of building this module you will get an .APK file.

One application module cannot depend on another. It can only depend on a library. It is the module that applied the Gradle library plugin:

apply plugin: 'com.android.library'

As a result of publishing such a module you will get an .AAR file (Android Archive Library). Comparing to .JAR file .AAR has some Android-related things, for example, resources and Manifest.

Building process for any library or application module can be roughly split into five phases, that could be represented with a certain Gradle tasks:

1. Preparation of dependencies. During this phase Gradle check that all libraries this module depends on are ready. If this module depends on another one, that module would be built as well.
2. Merging resources and processing Manifest. After this phase resources and Manifest are ready to be packaged in the result file.
3. Compiling. This phase started with Annotation Processors, in case you use them. Then source code is compiled into byte code. If you are using AspectJ, weaving also happens here.
4. Postprocessing. All Gradle tasks with a “transform” prefix are part of this phase. Most important ones are: transformClassesWithMultidexlist and transformClassesWithDex. They produce .DEX files.
5. Packaging and publishing. For libraries this stage means creating an .AAR file in the end, for applications — .APK.

As I mentioned, our goal is to minimize incremental build time. It’s difficult to speed up all the phases in one experiment, so I decided to focus on the longest ones. For a project with a single module, those were the compiling and postprocessing phases. The merging resources and processing manifest phase can also sometimes be time-consuming, but if the Java code is untouched, than incremental build is quite fast.

We all know, that Gradle runs task in consequent builds only if input isn’t the same. It also doesn’t rebuild a module if it wasn’t changed. That leads to the following hypothesis:

“The Incremental build time for a project with multiple modules is faster than for a single-module project, because only modified modules are recompiled.”

All right, lets find out whether this is true!

Experiment Setup

Project uses Gradle plugins v2.2.2. Minimal Android SDK is 15, that covers most of the devices according to API level dashboard. All modules have a dependency on Butterknife, to make project a bit more “alive”, since all projects has external dependencies.

In all variants application module is called “app”, while library modules are called “app2”, “app3” and so on.

Each variant has in total around 100 packages 15,000 classes 90,000 methods. Assembled that is two DEX files, almost three. Generated code is dummy, to keep all the methods in the .APK file, minifying and shrinking were disabled.

All measurements was done with Gradle build-in profiler. To use it you just need to add “--profile” in the end of the command, like this:

./gradlew assembleDebug --profile

As a result you will get a .HTML file with measurements.

For each setup I repeated each measurement 4 to 15 times, to make sure that my results were reproducible.

Java Code Generator

Writing all 15,000 classes by hand is time consuming, so I wrote a simple code generator in Python. It is available in Gist. Below you can find a schema of the code it generates.

As you can see, every next generated method called the previous one, and every first method of the next class called the last method of the previous class. That makes code more coupled, and increased compilation time.

Pure Java modules

I didn’t make a special experiments for pure Java modules. But I played with them a bit, and I can say that usually they are faster then Android ones. That happens because during build less tasks are executed, for example, there is no resources to merge.

If you’re interested in results for pure Java modules, please write a comment. Or you can clone the project from GitHub and modify it according to your needs. But don’t forget that actual results depend on hardware. To be able to compare your results, please repeat some experiments in your environment as well.

Small losses here and there

It’s no surprise that doing something in parallel does slow down the build. Even having a second project open in Android Studio can make build 5–10% slower. Listening to music, watching YouTube, or browsing the internet increases the build time a lot! I personally saw a speeding up of a build process by up to 30% after just closing everything except Android Studio.

All experiments were done with only necessary browser tabs and a single Android Studio project open.

Let’s do the experiment

Initial State

As a starting point I took a project with a single module that contains all 15 000 classes. For this setup incremental build time is 1m 10s.

3 Modules

First step is splitting one module into three: one application module and two library ones. Application module depends on libraries, but library modules are independent from one another. Each module has about 5 000 classes and 30 000 methods.

If changes are made only in application module, then build time is about 35 seconds. Almost 30 seconds win compare to initial state. But when one of library module is changed, even if application module is untouched, incremental build time grows to 1m 50s. 40 seconds longer!

Let’s take a look into profile report and see what took so long:

Profile report of an incremental build with changes in library module (“:app2”). Since application module (“:app”) depends on the library one, it is also recompiled.

In the screenshots above, you can see that most of the time was spent on building the library module. You might also notice that for library modules, both debug and release tasks were executed. Gradle wasted time executing two sets of tasks instead of one! This is why it took an extra 40 seconds longer than the single-module project.

We can avoid this and make this build even faster than our initial 1m 10s by splitting our code into modules.

But that’s not the only problem. Let’s look into how our application module depends on library modules:

dependencies {    compile project(path: ':app2')    compile project(path: ':app3')}

There’s an important problem in the code above: if a library is added like this, then the application always depends on the release variant, independent of it’s own build type. It also doesn’t matter which variant is chosen in Android Studio. Here’s what the Gradle Plugin User Guide says about all this:

By default a library only publishes its release variant. This variant will be used by all projects referencing the library, no matter which variant they build themselves. This is a temporary limitation due to Gradle limitations that we are working towards removing.

Luckily, it’s possible to change the variant our application depends on.

First, add the following code to libraries’ build.gradle file. It will allow library to publish debug variant as well:

android {    defaultConfig {        defaultPublishConfig 'release'        publishNonDefault true    }}

Second, application module should depend on library ones like this:

dependencies {    debugCompile project(path: ':app2', configuration: "debug")    releaseCompile project(path: ':app2', configuration: "release")    debugCompile project(path: ':app3', configuration: "debug")    releaseCompile project(path: ':app3', configuration: "release")}

Now the debug variant of our application depends on the debug variant of the libraries, and its release depends on their release. So lets make some changes to the module app2 and rebuild it. After these changes, we can check our profile report again:

The most significant difference is the absence of :app2:packageReleaseJarArtifact, which saves us about 15 seconds. In addition, the time is reshuffled a bit between rest of the tasks, and we end up with 1m 32s. That is 18 seconds faster then before, but still 22 seconds slower then our initial configuration. For changes only in the application module, build time stays almost the same — 36 seconds vs 35 seconds in our previous configuration.

Unfortunately I didn’t found a proper explanation for why it builds both flavors. I hope that once this Gradle limitation is resolved, then this problem will just go away as well. The same issue is discussed in AOSP Issue Tracker.

I’m also planning to spend some time experimenting with Gradle tasks to find a workaround for this problem. One possible way is to exclude all release tasks for debug builds.

5 Modules

Obviously, build time depends on the amount of code. If you reduce amount of code by half then the build should also be roughly two times faster. If instead of 3 modules project is split in 5, then build time should be approximately 40% faster.

It is almost true.

If changes are made only in application module then incremental build time is approximately 24s. For changes in a library module incremental build takes 50s. Compare to initial 1m 10s that is already a win. But I have a few more tricks at my disposal.

Reduce Size of The Application Module

Independently of what module is changed, application module will be recompiled every time. So reducing it’s size makes total sense. Ideally, it should just assemble whole application from a separate modules and might also provide a splash screen, because splash screen often depends on lots of features.

That is how idea about 3+1 and 5+1 configurations was born. In both cases project has a small application module that depends on 3 and 5 library modules accordingly. All library modules are independent from one another and has equal sizes. Let’s see what that gives us:

We can observe further drop in incremental build time. Even with changes in library module 5+1 configuration is build almost twice as fast as an initial single-module project. This is a decent progress.

Why does project actually depends on Butterknife?

This is a point where I have to make a confession. There was one very strong reason to add a dependency on Butterknife.

In the initial configuration incremental compilation takes 45s out of 1m 10s, but if Butterknife is removed then project is compiled in 15s only! Three times faster! Whole incremental build without Butterknife is 40s.

Is it a problem in the library?

As it appeared — no. Without Butterknife project compiles so fast because of actual Java incremental compilation, which is disabled for projects that use Annotation processors. You can find related issues in Gradle Jira, in AOSP Issue Tracker, it is also tracked in Gradle design docs. If you will take a closer look into the issue from ASOP Issue Tracker, one of the comment says:

“Annotation processors is not yet supported with incremental java compilation. It will depend on changes in Gradle.

We disabled it for project that apply com.neenbedankt.android-apt, so it's no longer a significant issue.”

That is why build just becomes slower without a notification.

Personally I won’t remove Annotation processors from the whole project. I find libraries like Dagger and Butterknife useful. But having a couple of modules without them could be a good idea, that would make their builds so much faster!

One More Trick — Rump Up API Level

Compilation is not the only thing that slows down build process. Producing .DEX files could also be time consuming. Especially if an application is above DEX Limit. Using multidex configuration increases build time, build system need to decided which classes go to which .DEX file. With introduction of Android Runtime the way how Android OS works with application that has multiple DEX files has changed. This is what Android Studio documentation says about it:

“Android 5.0 (API level 21) and higher uses a runtime called ART which natively supports loading multiple DEX files from APK files. ART performs pre-compilation at app install time which scans for classesN.dex files and compiles them into a single .oat file for execution by the Android device.”

That leads to decrease in build time. The reason is that each module produces its own DEX files, that are included into APK without modification. If you take a look into the tasks that run during build, you will notice that transformClassesWithMultidexlist is no longer executed. Also compilation itself became faster. You can find more information and instructions how to make a flavor that uses API 21 here.

Fastest Build Configuration Achieved.

Using API 21 for debug is an easy gain for every project. I tried it for 5+1 configuration and results were amazing:

Even for changes in library module incremental build time was only 17 seconds! But take into account that all modules are independent from one another. Once dependency between modules is introduced, build time increases from 17s to 42s (last row in the table above)!

Developing Library Module in a Test Driven Way

One of the main reasons why test-driven development (TDD) is difficult for single-module project is build time. TDD encourage to run test often. Running tests multiple times in a minute is a normal practice. But when build takes a minute or two, working in Test Driven way couldn’t be very productive.

With introduction of modules that problem is automagically resolved. Building a single module in the last configuration took only 9s! It make possible to run tests as often as needed.

Conclusion

First and most important, the hypothesis was correct, modularizing project can significantly speed up build process, but not for all configurations.

Second, if splitting is done in a wrong way, then build time will be drastically increased, because Gradle build both, release and debug version of library modules.

Third, working in test-driven way is much easier for a project with multiple modules, because building a small library module is way faster then the whole project.

Forth, doing many things in parallel slows down the build. So having more powerful hardware is a good idea.

Below you can find results of all experiments described in this article:

Please vote for mentioned issues. Resolving any of those would be a huge step towards faster builds. Below you can find all links:

• https://code.google.com/p/android/issues/detail?id=52962
• https://issues.gradle.org/browse/GRADLE-3259
• https://code.google.com/p/android/issues/detail?id=200043

All code is published on GitHub.

Nikita Kozlov (@Nikita_E_Kozlov) | Twitter
_The latest Tweets from Nikita Kozlov (@Nikita_E_Kozlov): “https://t.co/wmGSJ7snW1"_twitter.com

Thank you for you time reading this article. If you like it, don’t forget to click the ? below.