At Layer we are developing a platform to power the next generation of communications-enabled applications. Platform engineering is uniquely challenging — it demands that the engineering organization execute on a number of component projects concurrently to produce a coherent whole. This results in the production of a relatively high number of code respositories, test suites, and supporting infrastructure to coordinate all of the pieces. Managing this complexity requires great tooling, which isn’t always readily available. This article introduces XCTasks, a small Ruby library we’ve developed that provides flexible, commandline test automation for Xcode projects.

On the Applications team, we have atomized LayerKit (our iOS client SDK) into a number of small, standalone libraries. Each of these libraries serves a specific role in the framework and is tested independently of its peers and parent framework. All of these libraries are under the watchful eye of the Jenkins continuous integration server. This setup served us well — except that we found ourselves maintaining strikingly similar, but slightly different sets of build and test scripts across all of these projects. After thoughtful consideration of the problem, we determined that what we really wanted was a flexible DSL that would allow us to succinctly configure test automation tasks and eliminate the repetitive boilerplate.

XCTasks is a Ruby library that adds Xcode specific tasks definitions to the Rake build system. Ruby and Rake were selected for this implementation because our iOS projects have already embraced Cocoapods, which is itself implemented in Ruby. Ruby also has a clean, expressive syntax that is ideal for implementing lightweight DSL’s. XCTasks currently supports the following features:

  • Running tests under iOS or OS X
  • Executing logic and application tests
  • Handles OCTest and XCTest test bundles
  • Testing multiple SDK and simulator destinations
  • Population of shared Schemes into the Workspace
  • Testing via xcodebuild, xcpretty, or xctool
  • Flexible task namespacing and pre-requisites

The XCTasks::TestTask class is used to define a Rake namespace, which defines a group of subtasks. The project under test is expected to be organized into an Xcode Workspace containing one or more Schemes that execute a Test Bundle. Each Scheme is wrapped into a subtask and can then be tested on an arbitrary number of OS versions and Simulator/Device destinations. Let’s take a look at a real-world example from the BZipCompression project (see https://github.com/blakewatters/BZipCompression/blob/master/Rakefile):

require 'rubygems'  
require 'bundler'  
Bundler.setup  
require 'xctasks/test_task'

XCTasks::TestTask.new(:test) do |t|  
  t.workspace = 'BZipCompression.xcworkspace'
  t.schemes_dir = 'Tests/Schemes'
  t.runner = :xcpretty
  t.actions = %w{test}

  t.subtask(ios: 'iOS Tests') do |s|
    s.sdk = :iphonesimulator
  end

  t.subtask(osx: 'OS X Tests') do |s|
    s.sdk = :macosx
  end
end

task default: 'test'  

Let’s invoke Rake quickly to see what tasks we’ve created:

$ rake -T
rake test      # Run all tests (ios, osx)  
rake test:ios  # Run ios tests  
rake test:osx  # Run osx tests  

We can now run our test suite from the commandline by executing rake testor individually run the iOS or OS X tests by invoking rake test:ios or rake test:osx, respectively.

Now let’s quickly walk through this configuration in detail. In this Rakefile, we have imported XCTasks via RubyGems and Bundler. We then require xtasks/test_task to pull the task DSL into our Rake environment. Now the fun starts. Within the task we set up the following configuration:

  1. We initialize a new instance of XCTasks::TestTask providing the symbol :test to the initializer. This specifies that we wish to namespace our tasks under the name test. Note that test is the name of our aggregate task and the prefix for all subtasks.
  2. We configure the Workspace under test. This is a path relative to the Rakefile itself.
  3. We specify a schemes_dir of ‘Tests/Schemes’. This instructs XCTasks to copy a set of Xcode Schemes from the specified directly into the xcshareddata directory within the Workspace. Shared Schemes are required to run tests on the commandline under Xcode 5 because xcodebuild cannot create them. We recommend putting xcshareddata in .gitignore and copying them on each test run such that developers don’t see dirty changes on the schemes during day to day development.
  4. We configure a runner of :xcpretty. The runner is the tool that will be used to test the particular task. Supported runners are :xcodebuild, :xcpretty, or :xctool. The xcodebuild runner does not require any additional tools, but has very verbose output. xcpretty and xctool provide nice, easily to read test output.
  5. We then specify a set of actions to take when executing the subtasks. Supported actions are clean, build, and test. Multiple actions may be specified using an array syntax.

The remaining configuration specifies one or more subtasks, each of which is bound to a specific test Scheme in Xcode. The BZipCompression workspace contains two testing Targets and Schemes: ‘iOS Tests’ and ‘OS X Tests’. We define one subtask for each of these, using the Hash syntax of ios: 'iOS Tests' and osx: 'OS X Tests'. This specifies that we wish to define a task named ‘ios’ that is bound to the Xcode Scheme named ‘iOS Tests’ and another task named ‘osx’ that is bound to the Xcode Scheme ‘OS X Tests’. Each invocation of subtaskoptionally accepts a block, which is yielded an XCTasks::Subtask object. This subtask object can be used to change the configuration of any property inherited from the parent task. In the BZipCompression example above, a subtask instance is used to configure the sdk of the Scheme.

Testing Multiple iOS Versions and Destinations

XCTasks really begins to shine when you’d like to run your test suite under multiple versions of the iOS Simulator. Consider the following task definitions:

XCTasks::TestTask.new do |t|  
  t.workspace = 'LayerKit.xcworkspace'
  t.runner = :xctool

  t.subtask(unit: 'Unit Tests') do |s|
    s.ios_versions = %w{7.0 7.1}
  end

  t.subtask :functional do |s|
    s.runner = :xcodebuild
    s.scheme = 'Functional Tests'
  end
end  

And Rake task output:

$ rake -T
rake test               # Run the unit, functional tests  
rake test:unit          # Run unit tests against iOS Simulator 7.0, 7.1  
rake test:unit:7.0      # Run unit tests against iOS Simulator 7.0  
rake test:unit:7.1      # Run unit tests against iOS Simulator 7.1  
rake test:functional    # Run functional tests  

You can also specify particular destinations that you want the tests to execute under (please read the xcodebuild manual for clarity on the specifics):

XCTasks::TestTask.new(:spec) do |t|  
  t.workspace = 'LayerKit.xcworkspace'
  t.runner = :xctool

  t.subtask :functional do |s|
    s.runner = :xcodebuild
    s.scheme = 'Functional Tests'

    # Run on iOS Simulator, iPad, latest iOS
    s.destination do |d|
      d.platform = :iossimulator
      d.name = 'iPad'
      d.os = :latest
    end

    # Specify a complete destination as a string
    s.destination('platform=iOS Simulator,OS=7.1,name=iPhone Retina (4-inch)')

    # Quickly specify a physical device destination
    s.destination platform: :ios, id: '437750527b43cff55a46f42ae86dbf870c7591b1'
  end
end  

In Closing

XCTasks has greatly simplified our testing workflow at Layer across many iOS and OS X codebases. We expect to expand the functionality to include additional automation tasks as use-cases arise. Questions, comments, and pull requests are always welcome on the XCTasks home on Github.

If you love distributed systems, communications technology, or developing tools and frameworks such as XCTasks then swing by the Layer Jobs Page. We’re hiring engineering roles across the organization to join our mission to deliver the next generation communications Layer for the Internet.