You'd better download Hackystat and deploy it locally to analyze test subjects' sensor data. I will use a scenario-based approach to guide you through sensor data analysis with Zorro in next paragraph.
On March 31 of 2006 I developed bowling game scoring system in Test-Driven Development with Hackystat Eclipse sensor enabled and development activities were sent to public Hackystat server.
I compress all my sensor data from Mar 30, 2006 to Apr 1, 2006 into a zip file TddBowling.zip with "Zip Sensor Data" analysis on public Hackystat server and save it onto my desktop.
I open the zip file with WinZip and extract all the information into my local hackystat's data directory which is C:\hackystatdata. The new user's data directory name is anonymous_user because I don't want to disclose my user key. Then I rename the folder name from anonymous_user to 2277, which is the last four digits of my ID number in the validation study.
I log in to my local hackystat server at http://localhost:8080/hackystat with key 2277 and click preferences menu after I logged in. First I set C:\svn\hackystat\ as my workspace root with "Workspace Root Config" analysis because the bowling game project was saved in this directory. Project "TddBowling" can be defined with "Register Project" utility. Both start day and end day are Mar 31, 2006, this project has only one workspace "Game\".
Then I run Zorro analysis on the newly defined project. "TDD Stream Episode" analysis is located at "Sequence Analysis" block in "Extras" panel. I select project TddBowling, start day 3/31/2006 and end day 3/31/2006 to analyze. It gives me results similiar as shown in Figure 10.4, “ TDD Stream Analysis ”.
It is a table report with two columns. The left column contains TDD conformance evaluation results from Zorro and development activities are included in the right column. Compilation failure, unit test error and unit test success are highlighted in yellow, red and green colors respectively in this report.
You can follow this procedure to run Zorro analysis on every test subject's sensor data.
Eclipse Screen Recorder (ESR) records changes of Eclipse window into QuickTime movie file. Figure 10.5, “ ESR Video of Development in Eclipse ” is the screen dump of the video being played inside QuickTime. The timeline track is at the bottom of the video in red font.
ESR video analysis is the substitution for field observation. You can play back the recorded video and write down the development activities. Microsoft Excel is a good tool that you can use to bookkeep these data. You will get a manually created development stream after you write down all the observed activities and add the annotation (see Figure 10.6, “ ESR Video Analysis ”). By comparing the manually created development stream to the automatedly generated software development stream by Zorro you can validate Hackystat data collection. You will know whether there are missing or incorrect data, and come up with suggestion on how to improve Hackystat data collection. By watching the recorded video you will know whether the development is in TDD or not, and use observation to validate Zorro. The observation results may agree or disagree with Zorro. All these information can be written down in a single spreadsheet as seen in Figure 10.6, “ ESR Video Analysis ”.
Excel spreadsheet in Figure 10.6, “ ESR Video Analysis ” demonstrates how to conduct video analysis. From this chart we can see that the first episode and the second episode are recognized correctly by Zorro, while the third episode is in the wrong category because some editing actities were not captured by Hackystat Eclipse sensor. You can use this approach to validate Zorro episode by episode to each subject's TDD development video and summarize the results into one table.
Figure 10.7, “ Summar of Zorro Validation Analysis ” is the summary of a Zorro validation pilot study. In this study, we recruited 7 Java developers to work on stack problem in TDD to validate Zorro. Zorro classifies episodes of TDD into Test-Driven, Refactoring and Test-Last. An episode is wrongly classified if it is different from video observation and analysis.
Validation studies of Zorro will help us understand how precisely Zorro can recognize TDD, and how to improve it to support TDD conformance evaluation. In next section I will provide you with some support materials to help you conduct Zorro validation study.