Eyefodder: Code Coverage with Flex - a headless agent for CI builds

In my last blog post I gave details of how I user the modified coverage viewer in an automated build to follow the trend of code coverage over time. The trouble with this approach was that there was a problem either with the localConnection or the code that uses it and there was a wide variance of the values being reported. This post shows you how I fixed it by creating a headless coverage reporter that you can drop into your test harness and remove the need for a second application altogether.

In order to try and do this I decided to use as much of the code as possible that the FlexCover guys had written, and only change what I needed to get it working.Once I had it up and running I could worry about making it faster / leaner / more neatly coded...

The approach

So here's the basic plan:

Swap out the default localConnection reporting mechanism for one that will pick and choose depending on environment the app is running in
Create a headless agent that will pull the coverage data collection and reporting side of the coverageViewer into the app
Use commandline options to swap between the headless agent and local connection agent
Default to using the localConnnection agent

How it works

When your app compiles using the instrumented SDK created for using FlexCover, your application makes use of a class called CoverageManager. This manager is a patch that allows you to plug in a custom coverage agent for use in your app. By default it uses a class called LocalConnectionCoverageAgent which broadcasts coverage metadata to the coverage viewer application. What my patch does is get in and allow you to use a different, headless agent. To do this simply call during the preinitialize event of your main application:

private function injectAgent():void{
	CoverageManager.agent = new CoverageAgentSwitch();
}

As a quick test, run your app as you had previously and confirm it still works by sending data over a localConnection to the CoverageViewer. This is expected as by default, the switch will create a LocalConnectionCoverageAgent. In order to use the headless agent you need to set a few commandline properties. First you need to tell the switch that you want to use the headless agent. Then the headless agent needs to know where it's getting its metadata from, where it should output the coverage report. The commandline options are:

-coverage-agent-type=headless Headless is only option, anything else will default to localConnectionAgent -coverage-metadata='/full/path/to/coverage/metadata.cvm' -coverage-output='/full/path/to/coverage/reportInOriginalFormat.cvr' -emma-report='/full/path/to/coverage/reportInEmmaFormat.xml'

Note that if you don't specify the metadata path and at least one of the report formats, the headless agent will log errors but otherwise fail silently.

In terms of the edits I made to get it working, that is pretty much it. The headlessCoverageAgent swc is probably larger than it needs to be and is definitely grossly inefficient. I will update this soon to improve this, but right now I only have time to get this post up.

Obviously you will have to change your buildscript to pass these new commandline parameters in to the testharness when you run it. If there is any interest, I'll post my updated build script and testharness modifications that dispense with the log parser altogether and make for a more repeatable build script.

The Result

My build process is now much quicker because I don't have to wait to be sure that the coverage viewer has initialized. It's also completely stable and the coverage trend has been a great motivator for the team.

Coverage trend without crazy variance found using the external viewer

Download headless agent library

protected function onPreInitialize(event:Event):void {
CoverageManager.agent = new CoverageAgentSwitch();
}
protected function onCreationComplete(event:Event):void {
testRunner.addEventListener(TestRunner.TESTS_COMPLETE, onTestsComplete);
}
protected function onTestsComplete(event:Event):void {
CoverageManager.exit();
}

]]>

Comments

I am trying use your headlessCoverageAgent in our build system on a flex4 project. I get the following error when I try to compile test application.

Error: Type was not found or was not a compile-time constant: InvokeEvent

I suspect it because your swc was compiled with flex 3.2 or flex 3.4.2. Is there any chance that I could get hold of the source or have you compile me a flex4 swc?

Posted by: Abhinav Vohra | August 13, 2009 03:00 AM

Hi there,
I could post the source, but I'm not sure it will get you where you want to go. The flexcover instrumented sdk has not yet been ported to flex 4 (although this is being done currently). If you want to help on the porting of the compiler I could put you in touch with the guys who are working on that. So - bottom line: my hacking wont get you code coverage in a flex 4 app yet :(

Posted by: PBH | August 13, 2009 09:46 AM

When I try to use this headlessCoverageAgent.swc within my Flex project I get errors that AIR classes like File or InvokeEvent are missing. Am I doing something wrong or can it only be used in AIR projects?

Posted by: anonymous | September 14, 2009 03:59 AM

Hi there,
as the headless coverage agent needs to be able to read command line arguments and write files to the users filesystem, it will only support an AIR project. What you can do however is have an AIR project for your test harness and have the flex project classpath referenced into it. This is what I do on library projects and it works just fine

Posted by: PBH | September 15, 2009 07:43 AM

This is awesome! Thank you for taking the time to post this!

I am trying to get this working and am running into some difficulties. First I should mention that I am using Flex 4 via the port now available on the FlexCover site: http://code.google.com/p/flexcover/issues/detail?id=35 - Using this I am able to get coverage results by using the standard coverage agent.

When I run with the command line parameters, I am getting cvr and emma files that are reporting zero coverage.

When using the swapped agent and not passing any command line parameters, I am also unable to get coverage reported. The only clue I have there are these trace messages:

LocalConnectionCoverageAgent.initializeACKConnection: ackConnectionName null
LocalConnectionCoverageAgent.addPendingMapAndAttempSend: 1
LocalConnectionCoverageAgent.sendMaps: sent: 1 remaining:0
Using method coverageData
LocalConnectionCoverageAgent.initializeACKConnection: ackConnectionName null
LocalConnectionCoverageAgent - Coverage Registration Error. Another program has already opened a LocalConnection with id '_flexcover_ack1'. Try another connection name...
LocalConnectionCoverageAgent.initializeACKConnection: ackConnectionName null

After this, it looks like it's transmitting data to the viewer, but the viewer numbers never change. It says it receiving new elements and updates happen, but the coverage numbers never change.

The constant trace messages that come up when running without command line parameters do not happen when running with the command line parameters.

Any ideas?

Posted by: Brian Thomas | September 21, 2009 04:43 PM

Hey Brian,
off the top of my head, I'm not sure what's going on here. I'm pretty much heads down at the moment, but I'll fire you over a copy of the source code and you can have a shot at stepping through the code.

Posted by: PBH | September 21, 2009 04:49 PM

Quick update on this: Brian managed to get the headless coverage agent working perfectly by simply recompiling using the Flex 4 SDK he had (4.0.0.10118). For now I won't post a new headless agent as the plan is to incorporate the code into FlexCover in due course. If anyone has a desperate need to use the headless agent in a stable Flex 4 environment, let me know and I'll work out a way to get the code to you so you can build a swc for your SDK.

Posted by: PBH | September 22, 2009 03:23 PM

Hello - Thanks for the great work here.

I'm trying to incorporate the headlessAgent into Fluint's AirTestRunner.

I've modified the AirTestRunner.mxml, adding a preinitialize which injects CoverageManager.agent = new CoverageAgentSwitch(); and have added parameters to the launch as appropriate.

It looks like the headless agent is being created in lieu of LocalConnectionCoverageAgent, and Fluint manages to dump it's output when the application exits, but I am not seeing any of the generated reports from the headless agent, either in the original format, or in emma format.

I'm seeing this in the debug console:
[SWF] AIRTestRunner.swf - 1,502,252 bytes after decompression
11/17/2009 11:30:37.760 [DEBUG] com.eyefodder.flexcover.agents.AgentFactory creating agent of type com.eyefodder.flexcover.agents::HeadlessCoverageAgent
11/17/2009 11:30:37.775 [INFO] com.eyefodder.flexcover.agents.HeadlessCoverageAgent headless agent invoked
11/17/2009 11:30:38.432 [DEBUG] com.allurent.coverage.model.application.Recorder startCoverageRecording

My commandline arguments look like this:

-debug -headless -reportDir='C:\reports\coverage' -fileSet="C:\work\coverage\TestModule.swf" -coverage-agent-type=headless -coverage-metadata='C:\work\coverage\main-app-cov.cvm' -emma-report='C:\reports\coverage\main-app-Emma.xml' -coverage-output='c:\reports\coverage\main-app-Original.cvr'

Posted by: Rob | November 17, 2009 02:39 PM

For those who might be interested, getting this running with Fluint was pretty painless, but a matter of cobbling bits of info from various places. Here's an AIRTestRunner.mxml you can drop into a pull of the fluint code; export a release build with these changes, and your airtestrunner will be "headless enabled".

Posted by: Rob | November 18, 2009 12:37 PM

Hey Rob,
congrats on getting this working - sorry I didn't reply yesterday as I was travelling

cheers!

Paul

Posted by: PBH | November 18, 2009 01:31 PM

Eyefodder

Code Coverage with Flex - a headless agent for CI builds

The approach

How it works

The Result

Comments

Post a comment

Search

Tags

Archives

Recent Posts

Projects

Syndicate

Holiday Fund