Eyefodder

So by now I had my Coverage Viewer happily spitting out EMMA formatted coverage reports that my build machine was just dying to consume. All I had to do was get the running of the coverage test and the outputting of the report to be an integral part of my build. Here is the sequence of events for my build process:

1. clean my output folders (delete and recreate them)
2. Compile the main application using the instrumented SDK. This will generate the .cvr metadatafile that the coverageViewer needs
3. Compile the test harness using the instrumented SDK. This is so that when it runs, the hooks injected by the SDK report coverage data.
4. Launch the Coverage Viewer Application (specifying EMMA output file)
5. Run the test harness. When it is finished, call CoverageManager.exit() which will close the coverage viewer
6. Wait for the test results and coverage report to be available. By implication this means that the test harness and coverage viewer have quit.
7. Compile the main app using the standard SDK
8. (AIR Apps only) remove test harness and any test data from the output folder
9. (AIR Apps only) package up the application

In theory this is all well and good. In practice there are a few problems:

* The coverage viewer has to launch and parse the coverage metadata file. There isnt a simple way to feed this back to the ant script, so you have to get the script to wait. I set mine to 30 seconds, which should be enough. Nevertheless, this sort of thing frankly just irks me - either at some point it will fail, and up until that point, the build is taking longer han it needs to.

* There are two applications being launched by the build process. In order for this to work, you need to launch the coverageViewer with spawn='true' If something goes wrong with the build, the script no longer has control of this process (I might be wrong on this one - correct me if so..)

* The coverage agent (that gets injected into our test harness by the instrumented SDK) and the coverage viewere communicate via LocalConnection. This allows two flash applications on the same machine to talk to one another. Unfortunately, LocalConnections can be kind of flaky and you have to build a separate local connection for each direction of communication.

The first two I could live with, but when I put this together, I found that there was quite a wide variance of about 10% from build to build of coverage values:

Coverage trend showing wide variance of reporting results


I was left with one of two conclusions:

1. There was a bug/ fragility in the localConnection and it couldn't be relied on


2. There was a bug in the coverageAgent.exit() code sequence

The way CoverageAgent.exit() works is that you call it from within your application and it:

1. Waits until the application has sent all remianing coverage data


2. Sends a message to the coverage viewer asking it to prepare for exit


3. The coverage viewer writes out any report files


4. Exits, and sends a message back to the main application telling it to exit

Somewhere in this back and forth coverage data was getting lost, and occasionally the build was failing because the report file was not getting written. I could have attempted to debug the code, but in my experience, debugging LocalConnections is a royal ballache. In the end I decided to bring the reporting side of the coverage viewer 'in-house' and avoid the need for a second application altogether. My next post will show this and give you a download to the swc so that you can do it yourself...

Just in case you are interested, I have attached the buildfile I used for running the coverageViewer application as part of the build. If you can wait till I post the next update to this, I'd suggest you do :)

Download build file archive