While it is important to create test cases and invoke them during development, it is also important that these test cases are of high quality---that they can reliably signal the presence of defects when introduced during development. One approach to assessing test quality is to collect coverage information, which indicates the thoroughness with which the system code is exercised by the test case code during run-time. For example, if there is system code that is never actually exercised during testing, then it is unlikely that the test cases can discover defects in that code. Monitoring coverage information over time can be very useful to determine if newly created source code has corresponding test cases, and if the level of testing quality as reflected in coverage is increasing or decreasing.
StackyHack implements support for two coverage tools: JBlanket and Emma. This section will cover the JBlanket tool.
To install JBlanket for use in StackyHack, download the .zip file containing the latest release from the JBlanket home page into a directory such as "java-lib", and unzip it. Then, define an environment variable called JBLANKET_HOME to point to the distribution directory. For example, "c:\java-lib\jblanket-4.4.0309". (Note that the file path should have no embedded spaces in it.)
Example 4.13, “jblanket.build.xml” shows the jblanket.build.xml file, which defines the Ant targets used to invoke the JBlanket tool and the JBlanket sensor.
Example 4.13. jblanket.build.xml
<project name="stackyhack.jblanket" default="jblanket">
<description>
Provides the JBlanket tool and the Hackystat JBlanket sensor.
</description>
<import file="build.xml"/>
<property environment="env"/>
<property name="jblanket.dir" location="${build.dir}/jblanket" />
<property name="junit.dir" location="${build.dir}/junit" />
<target name="jblanket" depends="jblanket.tool, jblanket.sensor" description="Runs JBlanket, followed by the JBlanket sensor."/>
<target name="jblanket.tool" depends="clean, compile"
description="Cleans, compiles, instruments byte codes, runs unit tests, generates JBlanket report.">
<!-- Check for the JBLANKET_HOME environment variable, failing the build if it can't be found. -->
<available file="${env.JBLANKET_HOME}/lib/ant/jblanket.jar" property="jblanket.available"/>
<fail unless="jblanket.available" message="Error: JBLANKET_HOME not set or jblanket.jar not found, indicating JBlanket is not installed."/>
<path id="jblanket.libraries" >
<pathelement location="${env.JBLANKET_HOME}/lib/ant/jblanket.jar" />
<pathelement location="${env.JBLANKET_HOME}/lib/ant/jdom.jar" />
<pathelement location="${env.JBLANKET_HOME}/lib/ant/bcel-5.1.jar" />
</path>
<taskdef name="jblanket" classname="csdl.jblanket.ant.JBlanketModifierTask" classpathref="jblanket.libraries"/>
<mkdir dir="${jblanket.dir}"/>
<jblanket testgrammar="Test*.class" excludeonelinemethods="on" verbose="${hackystat.verbose.mode}">
<sysproperty key="jblanket.dir" value="${jblanket.dir}" />
<fileset dir="${build.dir}/classes">
<include name="**/*.class" />
</fileset>
</jblanket>
<!-- Run unit tests with JBlanket modified code. -->
<mkdir dir="${junit.dir}"/>
<junit printsummary="withOutAndErr" fork="yes">
<classpath>
<pathelement path="${java.class.path};${build.dir}/classes" />
<path refid="jblanket.libraries"/>
</classpath>
<sysproperty key="jblanket.dir" value="${jblanket.dir}"/>
<formatter type="xml" />
<batchtest todir="${junit.dir}">
<fileset dir="${src.dir}">
<include name="**/Test*.java" />
</fileset>
</batchtest>
</junit>
<!-- Run the jblanket report. -->
<taskdef name="jblanketreport" classname="csdl.jblanket.ant.JBlanketReportTask" classpathref="jblanket.libraries"/>
<jblanketreport excludeonelinemethods="on" >
<sysproperty key="jblanket.dir" value="${jblanket.dir}" />
</jblanketreport>
<!-- Now delete the classes/ dir containing the instrumented .class files. -->
<delete dir="${build.dir}/classes" />
</target>
<target name="jblanket.sensor" description="Sends Coverage data to Hackystat using the JBlanket sensor.">
<!-- Define the jblanket sensor taskdef, failing the build if the sensor is not installed. -->
<available classname="org.hackystat.sensor.jblanket.JBlanketSensor" property="jblanket.sensor.available"/>
<fail unless="jblanket.sensor.available" message="Error: JBlanket sensor not installed."/>
<taskdef name="hacky-jblanket" classname="org.hackystat.sensor.jblanket.JBlanketSensor" />
<!-- Send Coverage data to Hackystat using the JBlanket sensor. -->
<hacky-jblanket verbose="${hackystat.verbose.mode}" methodsetsxmlfile="${jblanket.dir}/COVER-MethodSets.xml" />
</target>
</project>
The jblanket.build.xml defines the <jblanket.tool> target to obtain coverage information. The <jblanket.sensor> target uses the JBlanket sensor to read in the XML files generated by JBlanket and send the output to Hackystat. The <jblanket> target simply invokes both of these targets in sequence.
As the Ant build file illustrates, the <jblanket.tool> target produces coverage data using a three-step process. First, it performs a pass over all of the compiled .class files in StackyHack, inserting instrumentation bytecodes into these files to record information as the system is executed. Second, it invokes the JUnit tests on the instrumented bytecode, resulting in a set of files that provide information about what methods were invoked (and not invoked) during the test runs. Finally, the JBlanketReport task processes these data files to produce a summary report.
Example 4.14, “jblanket.tool invocation” shows the output from invoking the <jblanket.tool> target.
Example 4.14. jblanket.tool invocation
C:\svn-csdl\StackyHack>ant -f jblanket.build.xml jblanket.tool
Buildfile: jblanket.build.xml
clean:
[delete] Deleting directory C:\svn-csdl\StackyHack\build
compile:
[mkdir] Created dir: C:\svn-csdl\StackyHack\build\classes
[javac] Compiling 5 source files to C:\svn-csdl\StackyHack\build\classes
jblanket.tool:
[mkdir] Created dir: C:\svn-csdl\StackyHack\build\jblanket
[mkdir] Created dir: C:\svn-csdl\StackyHack\build\junit
[jblanket] JBlanket modify task completed (0 secs.)
[junit] Running edu.hawaii.stack.TestClearStack
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.281 sec
[junit] Running edu.hawaii.stack.TestStack
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.344 sec
[delete] Deleting directory C:\svn-csdl\StackyHack\build\classes
[jblanketreport] ********************************************************
[jblanketreport] Method-level Coverage:
[jblanketreport] All methods : {total=17}
[jblanketreport] Untestable methods : {total=0}
[jblanketreport] Excluded One-line methods : {total=5}
[jblanketreport] --------------------------------------------------------
[jblanketreport] Remaining methods : {total=12}
[jblanketreport] Tested methods : {total=12, percent=100%}
[jblanketreport] Untested methods : {total=0, percent=0%}
[jblanketreport] ********************************************************
[jblanketreport] JBlanket results in C:\svn-csdl\StackyHack\build\jblanket\index.html
[jblanketreport] JBlanket report task completed (0 secs.)
BUILD SUCCESSFUL
Total time: 4 seconds
As you can see, coverage data was collected by JBlanket for each of the methods in the StackyHack system. JBlanket supports a coverage method called "Extreme Coverage", in which coverage is assessed at the level of method invocations, and uninvoked methods that consist of a single line do not count against you. (The rationale is that one line getter and setter methods are so common in Java that it is more efficient to verify their correctness by visual inspection rather than by writing test code.)
After JBlanket has been invoked to generate data regarding the results of testing, the JBlanket sensor can be invoked to read in the resulting XML files produced by JBlanket and send the resulting data to Hackystat as Coverage sensor data.
To accomplish this, you must first install the JBlanket sensor, which can process the XML files output by JBlanket. General instructions on installing sensors are provided in Chapter 2, Client-side configuration: Tool sensor installation; instructions for specific sensors are provided in Chapter 26, Sensors. After installation, your hackyInstaller JBlanket configuration window should look something like the screen image in Figure 4.8, “ HackyInstaller configuration screen for JBlanket sensor ”.
Example 4.15, “jblanket.sensor invocation” illustrates the invocation of the JBlanket sensor.
Example 4.15. jblanket.sensor invocation
C:\svn-csdl\StackyHack>ant -f jblanket.build.xml jblanket.sensor Buildfile: jblanket.build.xml jblanket.sensor: [hacky-jblanket] Processing file: C:\svn-csdl\StackyHack\build\jblanket\COVER-MethodSets.xml [hacky-jblanket] Hackystat data on 5 JBlanket coverage entries sent to http://hackystat.ics.hawaii.edu/ (2 secs.) BUILD SUCCESSFUL Total time: 1 second
Once you've invoked the sensor, you can verify that this data was received at the server by using the List Sensor Data command. Figure 4.9, “ List Sensor Data with StackyHack JBlanket data ” illustrates what this page might look like after receiving the StackyHack data.
