Converting Xcode Test Runs to JUnit, the Fast Way

Testing is very important at PSPDFKit. We're building an SDK. When you give your API to other developers, there are many more things that can go wrong. For a long time, we've been using either a combination of xcodebuild with xcpretty(opens in a new tab) or Facebook's xctool(opens in a new tab). No tool is perfect though. xctool(opens in a new tab) is an alternative to xcodebuild but uses much of it under the hood and often breaks when Xcode is updated. At some point Facebook deprecated it, and currently it doesn't work with Xcode 8 anymore. xcpretty(opens in a new tab) is really good. It parses the very verbose output of xcodebuild and prints it into something sane that a human can read. A big shout-out to Marin Usalj(opens in a new tab), who maintains the project.

There are a few inherent problems with parsing an undocumented log output with regular expressions. Things might get lost if no regex matches, or parsing becomes extremely slow(opens in a new tab) if you log too much. All of this, and much more, has happened to us. It is not really xcpretty's fault — it tries hard. It's more an inherent problem and changing output between Xcode versions doesn't help either. (Oh, you think xcodebuild can't be any more verbose? Did you know about the undocumented log level 5?(opens in a new tab) Have fun!)

I was discussing this with "Mr. Fastlane" Felix(opens in a new tab) on his last visit to Vienna over a (fake)(opens in a new tab) Club Mate(opens in a new tab).

We were talking about potentially rewriting xcpretty. However, after some more <s>drinking</s> talking, we figured that Xcode Server is a thing and that there must be a way how xcodebuild could pass the test results to it. JSON maybe? Of course not, but there is a plist! We were not the first to discover this(opens in a new tab), just the first who had the idea of writing a converter that takes the plist output and converts it into JUnit format.

Why JUnit? Because that's what Jenkins eats for breakfast. So naturally, trainer was born(opens in a new tab). PSPDFKit is a huge project (PDF is hard, and turing-complete(opens in a new tab)) and we have a few thousand tests across about 10 sub-projects. So it turned out to be a good test case to get trainer up and running. Since Xcode didn't close its log pipe (we reported this here as rdar://27447948(opens in a new tab) and it has sinced been fixed in beta 4) and our sad attempts on working around this issue(opens in a new tab) were not very successful, trainer was our best way to get Jenkins <s>green</s> blue again.

Before it was even publicly announced, Twitter had already adopted it and saw a 10x performance increase(opens in a new tab) in JUnit report generation.

It also made our CI much, much less flaky, since random weird log messages no longer confuse the test converter. You can even run this in combination with xcpretty and get the best of both worlds. A flawed, but great version a human can read and a separate converter that translates test results from a (so far) stable-looking plist format.

So far trainer seems to be the only converter, though there are other tools that try to solve this differently, such as the recently open-sourced fbxctest(opens in a new tab) by Facebook.

trainer(opens in a new tab) is currently provided as standalone tool, with a built-in fastlane(opens in a new tab) plugin, so it's easy to get started if you're already using fastlane(opens in a new tab). It focuses on one thing, generating a JUnit file that is supported by all major Continuous Integration systems like Jenkins, Circle and Travis. Combining this with danger(opens in a new tab) and the danger-junit plugin(opens in a new tab) by @orta(opens in a new tab), you can show the exact test failures right in your pull request. This solves the common phenomenon of "PR and Run", which happens on many major open source projects. Typically, a developer submits a pull request on GitHub (often a simple one), which causes a test failure. By the time the test failures are shown on GitHub, the author has already closed the page. After a "PR and Run" occurs, the maintainer then has to manually ping the author to fix the tests. By using danger(opens in a new tab), the failed tests will automatically be posted as a comment on the PR, triggering an email notification to be sent to the author. (See example PR(opens in a new tab))

A big thanks to Felix(opens in a new tab) for building trainer(opens in a new tab)! We hope to see this under the fastlane umbrella(opens in a new tab) soon.