Code Coverage with Flex – a headless agent for CI builds

In my last blog post I gave details of how I user the modified code coverage viewer for flex in an automated build to follow the trend of code coverage over time. The trouble with this approach was that there was a problem either with the localConnection in flex or the code that uses it and there was a wide variance of the values being reported. This post shows you how I fixed it by creating a headless code coverage reporter that you can drop into your test harness and remove the need for a second application altogether.


In order to try and do this I decided to use as much of the code as possible that the FlexCover guys had written, and only change what I needed to get it working.Once I had it up and running I could worry about making it faster / leaner / more neatly coded…

The approach

So here’s the basic plan:

  1. Swap out the default localConnection reporting mechanism for one that will pick and choose depending on environment the app is running in
  2. Create a headless agent that will pull the code coverage data collection and reporting side of the coverageViewer into the app
  3. Use commandline options to swap between the headless and localConnnection agent
  4. Default to using the localConnnection

How it works

When your app compiles using the instrumented SDK created for using FlexCover, your application makes use of a class called CoverageManager. This manager is a patch that allows you to plug in a custom code coverage agent for use in your app. By default it uses a class called LocalConnectionCoverageAgent which broadcasts code coverage metadata to the coverage viewer application. What my patch does is get in and allow you to use a different, headless agent. To do this simply call during the preinitialize event of your main application:

private function injectAgent():void{
CoverageManager.agent = new CoverageAgentSwitch();
}

As a quick test, run your app as you had previously and confirm it still works by sending data over a localConnection to the CoverageViewer. This is expected as by default, the switch will create a LocalConnectionCoverageAgent. In order to use the headless agent you need to set a few commandline properties. First you need to tell the switch that you want to go headless. Then the headless agent needs to know where it’s getting its metadata from, where it should output the code coverage report. The commandline options are:

-coverage-agent-type=headless <em>Headless is only option, anything else will default to localConnectionAgent</em>
-coverage-metadata='/full/path/to/coverage/metadata.cvm'
-coverage-output='/full/path/to/coverage/reportInOriginalFormat.cvr'
-emma-report='/full/path/to/coverage/reportInEmmaFormat.xml'

Note that if you don’t specify the metadata path and at least one of the report formats, the headless agent will log errors but otherwise fail silently.
In terms of the edits I made to get it working, that is pretty much it. The headlessCoverageAgent swc is probably larger than it needs to be and is definitely grossly inefficient. I will update this soon to improve this, but right now I only have time to get this post up.
Obviously you will have to change your build script to pass these new commandline parameters in to the test harness when you run it. If there is any interest, I’ll post my updated build script and test harness modifications that dispense with the log parser altogether and make for a more repeatable build script.

The Result

My build process is now much quicker because I don’t have to wait to be sure that the coverage viewer has initialized. It’s also completely stable and the code coverage trend has been a great motivator for the team.
better coverage
Code Coverage trend without crazy variance found using the external viewer
Download headless library

Update

A few of you have asked for a copy of the source code so you can play around with it and undoubtedly make it all work better for your needs. Attached here—with no warranty or support—is the headless agent project. In order to get it to work, you will need to have the CoverageAgent and CoverageUtilities projects in your workspace, as well as the modified CoverageViewer project from previous posts. Good luck with it – It’s code I haven’t looked at for a while so don’t hate on me for it :/

This entry was posted in Agile Software Development, Continuous Integration, Flash & Actionscript. Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

22 Comments

  1. Abhinav Vohra
    Posted August 13, 2009 at 3:00 am | Permalink

    I am trying use your headlessCoverageAgent in our build system on a flex4 project. I get the following error when I try to compile test application.
    Error: Type was not found or was not a compile-time constant: InvokeEvent
    I suspect it because your swc was compiled with flex 3.2 or flex 3.4.2. Is there any chance that I could get hold of the source or have you compile me a flex4 swc?

  2. Posted August 13, 2009 at 9:46 am | Permalink

    Hi there,
    I could post the source, but I’m not sure it will get you where you want to go. The flexcover instrumented sdk has not yet been ported to flex 4 (although this is being done currently). If you want to help on the porting of the compiler I could put you in touch with the guys who are working on that. So – bottom line: my hacking wont get you code coverage in a flex 4 app yet :(

  3. Abhinav Vohra
    Posted August 13, 2009 at 10:21 am | Permalink

    Thanks for writing back. Well I have successfully used the flex4 sdk mod available at the following link
    http://code.google.com/p/flexcover/issues/detail?id=35#c2
    to produce a cvm and run the coverageviewer to see results. In that respect, I believe the coverage mod is working. Now, I would like to integrate this into my hudson build.
    Could you please post the source or share with me a flex4 flavored swc?

  4. Abhinav Vohra
    Posted August 13, 2009 at 6:10 pm | Permalink

    Hello again
    Not sure if my previous message made it to you but I have the flex4 working with flexcover using the coverageviewer. All i need now is a flex4 flavored swc of headlessCoverageAgent.
    THanks in adavance

  5. Serioga Serii
    Posted August 28, 2009 at 7:35 am | Permalink

    Hi,
    I have the same problem as Abhinav noted, however i’m using flex sdk 3. I’ve get rid of this error at compile time by adding air libs to lib path. Unfortunately when i am trying to run the resulting swf file i’m getting the following:
    “VerifyError: Error #1014: Class flash.events::InvokeEvent could not be found.
    at global$init()
    at FlexUnitRunner/injectAgent()[C:\work\ASUTCover\src\test\FlexUnitRunner.mxml:58]
    at FlexUnitRunner/___FlexUnitRunner_Application1_preinitialize()[C:\work\ASUTCover\src\test\FlexUnitRunner.mxml:6]
    at flash.events::EventDispatcher/dispatchEventFunction()”
    Any ideas?
    Thanks

  6. anonymous
    Posted September 14, 2009 at 3:59 am | Permalink

    When I try to use this headlessCoverageAgent.swc within my Flex project I get errors that AIR classes like File or InvokeEvent are missing. Am I doing something wrong or can it only be used in AIR projects?

  7. PBH
    Posted September 15, 2009 at 7:43 am | Permalink

    Hi there,
    as the headless coverage agent needs to be able to read command line arguments and write files to the users filesystem, it will only support an AIR project. What you can do however is have an AIR project for your test harness and have the flex project classpath referenced into it. This is what I do on library projects and it works just fine

  8. Brian Thomas
    Posted September 21, 2009 at 4:43 pm | Permalink

    This is awesome! Thank you for taking the time to post this!
    I am trying to get this working and am running into some difficulties. First I should mention that I am using Flex 4 via the port now available on the FlexCover site: http://code.google.com/p/flexcover/issues/detail?id=35 – Using this I am able to get coverage results by using the standard coverage agent.
    When I run with the command line parameters, I am getting cvr and emma files that are reporting zero coverage.
    When using the swapped agent and not passing any command line parameters, I am also unable to get coverage reported. The only clue I have there are these trace messages:
    LocalConnectionCoverageAgent.initializeACKConnection: ackConnectionName null
    LocalConnectionCoverageAgent.addPendingMapAndAttempSend: 1
    LocalConnectionCoverageAgent.sendMaps: sent: 1 remaining:0
    Using method coverageData
    LocalConnectionCoverageAgent.initializeACKConnection: ackConnectionName null
    LocalConnectionCoverageAgent – Coverage Registration Error. Another program has already opened a LocalConnection with id ‘_flexcover_ack1′. Try another connection name…
    LocalConnectionCoverageAgent.initializeACKConnection: ackConnectionName null
    After this, it looks like it’s transmitting data to the viewer, but the viewer numbers never change. It says it receiving new elements and updates happen, but the coverage numbers never change.
    The constant trace messages that come up when running without command line parameters do not happen when running with the command line parameters.
    Any ideas?

  9. Posted September 21, 2009 at 4:49 pm | Permalink

    Hey Brian,
    off the top of my head, I’m not sure what’s going on here. I’m pretty much heads down at the moment, but I’ll fire you over a copy of the source code and you can have a shot at stepping through the code.

  10. Posted September 22, 2009 at 3:23 pm | Permalink

    Quick update on this: Brian managed to get the headless coverage agent working perfectly by simply recompiling using the Flex 4 SDK he had (4.0.0.10118). For now I won’t post a new headless agent as the plan is to incorporate the code into FlexCover in due course. If anyone has a desperate need to use the headless agent in a stable Flex 4 environment, let me know and I’ll work out a way to get the code to you so you can build a swc for your SDK.

  11. Posted October 9, 2009 at 5:07 am | Permalink

    Hi,
    I’m reading this article and I’m really inetrested in getting this working on Flex4, I’d like to integrate flexcover in our CI (hudson).
    Could you, please, send me the source to compile in flex 4?
    Thanks in advance for your great job!
    Nico

  12. Rob
    Posted November 17, 2009 at 2:39 pm | Permalink

    Hello – Thanks for the great work here.
    I’m trying to incorporate the headlessAgent into Fluint’s AirTestRunner.
    I’ve modified the AirTestRunner.mxml, adding a preinitialize which injects CoverageManager.agent = new CoverageAgentSwitch(); and have added parameters to the launch as appropriate.
    It looks like the headless agent is being created in lieu of LocalConnectionCoverageAgent, and Fluint manages to dump it’s output when the application exits, but I am not seeing any of the generated reports from the headless agent, either in the original format, or in emma format.
    I’m seeing this in the debug console:
    [SWF] AIRTestRunner.swf – 1,502,252 bytes after decompression
    11/17/2009 11:30:37.760 [DEBUG] com.eyefodder.flexcover.agents.AgentFactory creating agent of type com.eyefodder.flexcover.agents::HeadlessCoverageAgent
    11/17/2009 11:30:37.775 [INFO] com.eyefodder.flexcover.agents.HeadlessCoverageAgent headless agent invoked
    11/17/2009 11:30:38.432 [DEBUG] com.allurent.coverage.model.application.Recorder startCoverageRecording
    My commandline arguments look like this:
    -debug -headless -reportDir=’C:\reports\coverage’ -fileSet=”C:\work\coverage\TestModule.swf” -coverage-agent-type=headless -coverage-metadata=’C:\work\coverage\main-app-cov.cvm’ -emma-report=’C:\reports\coverage\main-app-Emma.xml’ -coverage-output=’c:\reports\coverage\main-app-Original.cvr’

  13. Rob
    Posted November 18, 2009 at 12:37 pm | Permalink

    For those who might be interested, getting this running with Fluint was pretty painless, but a matter of cobbling bits of info from various places. Here’s an AIRTestRunner.mxml you can drop into a pull of the fluint code; export a release build with these changes, and your airtestrunner will be “headless enabled”.

    
        		
  14. Posted November 18, 2009 at 1:31 pm | Permalink

    Hey Rob,
    congrats on getting this working – sorry I didn’t reply yesterday as I was travelling
    cheers!
    Paul

  15. Rob
    Posted February 18, 2010 at 3:08 pm | Permalink

    http://wiki.github.com/wulfen/flexunit/
    I’ve pulled a fork off of the FlexUnit project on github, and committed a FlexUnitRunner project, which incorporates the headlessCoverageAgent.swc
    Details available at the above link.
    Posting this as a track-back.

  16. Posted June 2, 2010 at 11:11 am | Permalink

    Did this code make it in to the flexcover 0.90 release? I’m attempting to set this up with flex4. thanks, Lloyd

  17. Santos Derek
    Posted August 9, 2010 at 10:10 pm | Permalink

    Great Post! I am using the flexunit task but cannot get the command line parameters to be passed in properly. Can you post your modified build script? Or point me in the right direction?
    Thanks!

  18. Nikhil Sharma
    Posted April 25, 2011 at 7:37 am | Permalink

    Hi, I want to use the flexcover so that if i put teh instrumented swf on a central server so that many users can use it and code coverage will increase accordingly.??
    Can i do so ??
    Is there any way that the instrumented swf is being used by many users??
    PLease let me know as early as possible
    Thanks in advance

  19. Richard Lee
    Posted June 5, 2011 at 4:10 am | Permalink

    Hi there-

    The download links for the emma reporting patch and the headless coverage agent swc seem busted. I was able to pull the emma patch out of google’s cache, but no such luck for the coverage agent swc. BTW- do you have source code for that?

    Richard

    • Paul Barnes-Hoggett
      Posted June 5, 2011 at 4:14 pm | Permalink

      Hi Richard,
      sorry about that! I’m in the process of moving my blog to a new host & new platform, so it looks like I missed a few links…
      re: source code: I’ll see if I can dig that up too – also moving my svn repo so it’s like I’ve moved digital house and everything is in boxes…

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

  • Stay in touch!

    If you want me to let you know when I add new posts, drop your email here

    * = required field