Tool Mentor: Analyzing Test Results Using Rational Test RealTime

Overview

This tool mentor describes how to analyze the results of test execution from Rational Test RealTime. The value of testing early and often cannot be realized if the results of the testing activity cannot be quickly and easily interpreted and subsequently acted upon.

Using both a source code insertion technology as well as source code analysis, Rational Test RealTime is able to focus its component testing and runtime analysis functionality directly at the source code level. A variety of features have been added to ensure that fault isolation is quick and that it is easy and efficient to work on the associated source code:

source code can be opened and modified in the Test RealTime Text Editor.
version control programs (e.g. Rational ClearCase) can be manipulated to check-in/check-out versioned code.
change management programs (e.g. ClearQuest) can be directly accessed for defect submissions and enhancement requests

These features are used in conjunction with the analysis reports produced by the test tools. Each key function of Test RealTime-component testing, system testing, memory profiling, performance profiling, code coverage analysis, runtime tracing, static metric generation-has a dedicated report. These reports contain information extracted from a number of log files generated by source code parsers, source code instrumentors, test script generators, test script compilers, target deployment ports and report generators. These files can be opened automatically by Test RealTime at the conclusion of a run, or can be opened manually.

Each report is designed to be immediately interpretable to ensure problem diagnosis and resolution is quick and efficient.

Types of Reports

Report analysis for each Test RealTime feature will be discussed. Where relevant, report differences due to programming language used will be discussed:

Component Testing and System Testing
Memory Profiling
Application Performance Profiling
Code Coverage Analysis
Runtime Tracing
Static Metrics

Component Testing and System Testing

To open these reports using the Project Browser tab in the Project Window on the right-hand side of the Test RealTime user interface, right-click a test node and select View Report->Test. To open these reports using the Test RealTime menu:

Select the menu item File->Browse Reports.
Select the Add toolbar button.
Select Report from the Files of Type dropdown list
Browse to and then select the .xrd files corresponding to the reports you would like to view
Click the Open button

To open these reports from the command line, simply treat each .xrd file as a parameter to the command line studio.

The report is organized into two main sections, a summary section followed by a details section.

Summary - The summary section contains information such as the report file locations, time of test execution and a summary count of passed and failed tests.
Component Test Details - For component tests, this section provides information about the drivers/services/functions/methods/procedures under test. For each item, the report contains some or all of the following information, depending on the source language and the options selected:
- a test section which provides an overview and summarizes whether or not the test passed
- a variable value section, specifying the acquired value for each variable.
- if code coverage analysis was performed, a code coverage section
- additional information regarding contract (i.e. assertion) validation and verify queries for C++ and Java
System Test Details - For system tests, the following subsections are provided:
- an Initialization section, which details the steps involved in opening a connection to the component under test.
- one or more Scenario sections, which details the actual tests that were performed. Scenario blocks can be nested.
- a Termination section, which details the steps involved in terminating connection to the component under test.

The Report Explorer of the Test RealTime user interface can be used to maneuver through the test report (double-click on any node of interest); the Test Report menu item and the toolbar let you filter the report in order to only view failures. Multiple sections of the test report are hyperlinked to the test script; simply right-click those sections in which the mouse icon resembles a hand.

For detailed information about viewing test reports see the Rational Test RealTime User Guide, and refer to the chapter Automated Testing->Component Testing, in the sections discussing the test report.

For detailed technical information on working with report files, refer to the Rational Test RealTime Reference Guide, focusing on the sections discussing the various test report generators. In particular, see the page Appendices->File Types.

Memory Profiling

Memory profiling reports for all supported languages-C, C++ and Java-are stored in XML-based files within the Project folder or the TDP folder, depending on the language.

To open these reports using the Project Browser tab in the Project Window on the right-hand side of the Test RealTime user interface, right-click a test node and select View Report->Memory Profile.

To open these reports using the Test RealTime menu:

Select the menu item File->Browse Reports.
Select the Add toolbar button.
Select Memory Profiling from the Files of Type dropdown list.
Browse to and then select the files you are interested in viewing.
Repeat steps 2-4 as needed.
Click the Open button in the Report Browser window.

To open these reports from the command line, simply treat the one or more files you are interested in viewing as parameters to the command line studio.

The report is organized into two main sections, a summary section followed by a details section.

Summary - A Histogram and textual report section giving the total number of blocks and bytes allocated and freed, as well as the maximum number of blocks/bytes allocated at any given time.
C and C++ Details - Each error and warning is listed, with its associated call stack. The call stack is listed with the last called function/method mentioned first; all functions/methods in the call stack are hyperlinked to source code
Java Details - A listing is presented of every method that has allocated objects since the last snapshot. The basic table lists the number of objects and associated bytes allocated since the last snapshot for each method, as well as the total number of objects and bytes allocated by each method and its descendants (i.e. child methods of the parent).

For detailed information about viewing memory profiling reports see the Rational Test RealTime User Guide, and refer to the chapter Runtime Analysis->Memory Profiling.

Application Performance Profiling

Performance profiling reports for all supported languages-C, C++ and Java-are stored in XML-based files within the Project folder or the TDP folder, depending on the language.

To open these reports using the Project Browser tab in the Project Window on the right-hand side of the Test RealTime user interface, right-click a test node and select View Report->Memory Profile.

To open these reports using the Test RealTime menu:

Select the menu item File->Browse Reports.
Select the Add toolbar button.
Select Performance Profiling from the Files of Type dropdown list.
Browse to and then select the files you are interested in viewing.
Repeat steps 2-4 as needed.
Click the Open button in the Report Browser window.

To open these reports from the command line, simply treat the one or more files you are interested in viewing as parameters to the command line studio.

The report is organized as follows: at the top of each report is a pie chart detailing each function/method (up to six) that monopolize 5 or more percent of total execution time. Beneath the pie chart is a sortable table listing each function/method and various associated statistics. A mouse click on a column header will sort the overall list by the contents of that column; a mouse click on any function/method named opens the corresponding source code. No value is necessarily good or bad - large function execution times, or target function + descendant execution times, simply highlights potentially inefficient algorithms and thus candidates for optimization.

For detailed information about viewing performance profiling reports see the Rational Test RealTime User Guide, and refer to the chapter Runtime Analysis->Performance Profiling.

Code Coverage Analysis

Code coverage analysis reports for all supported languages-C, C++, Ada and Java-are stored in XML-based files within the Project folder or the TDP folder, depending on the language.

To open these reports using the Project Browser tab in the Project Window on the right-hand side of the Test RealTime user interface, right-click a test node and select View Report->Memory Profile.

To open these reports using the Test RealTime menu:

Select the menu item File->Browse Reports.
Select the Add toolbar button.
Select Code Coverage from the Files of Type dropdown list.
Browse to and then select the files you are interested in viewing.
Repeat steps 2-4 as needed.
Click the Open button in the Report Browser window.

To open these reports from the command line, simply treat the one or more files you are interested in viewing as parameters to the command line studio.

The report is organized into two main components, a Source component and a Rates component.

Source - The Source tab reflects two levels of code coverage information:
- When the Root folder is selected, the Source tab summarizes overall code coverage using a bar chart.
- When any other node in the Report Window is selected, the Source tab uses a color-coded display to reflect which code has and has not been covered. The Source tab displays information related to whichever node has been selected.
Rates - The Rates tab breaks down the level of coverage for each function/method/procedure into percentages.

For detailed information about viewing Code Coverage reports see the Rational Test RealTime User Guide, and refer to the chapter Runtime Analysis->Code Coverage.

Runtime Tracing

Runtime tracing reports for all supported languages-C, C++ and Java-are stored in XML-based files within the Project folder or the TDP folder, depending on the language.

To open these reports using the Project Browser tab in the Project Window on the right-hand side of the Test RealTime user interface, right-click a test node and select View Report->Memory Profile.

To open these reports using the Test RealTime menu:

Select the menu item File->Browse Reports.
Select the Add toolbar button.
Select Trace Files from the Files of Type dropdown list.
Browse to and then select the files you are interested in viewing.
Repeat steps 2-4 as needed.
Click the Open button in the Report Browser window.

To open these reports from the command line, simply treat the one or more files you are interested in viewing as parameters to the command line studio.

The runtime tracing report is a sequence diagram of all events that occurred during the execution of application or test code. This sequence diagram uses a notation taken from the Unified Modeling Language, thus it can be correctly referred to as a UML-based sequence diagram. Vertical lines are lifelines; each lifeline represents a Java object instance. The very first lifeline, represented by a stick figure, is considered the "world" - that is, the operating system. Horizontal lines connect one lifeline to another. Green lines are constructor calls, black lines are method calls, red lines are method returns, blue lines are destructor calls, orange lines are exceptions. Hovering the mouse over any method call to see the full text. Every call and call return is time stamped. Every vertical and horizontal line is hyperlinked to the monitored source code. The menu item Runtime Trace->Filters can be used to create filters that make the runtime tracing report more manageable.

For detailed information about viewing runtime tracing reports see the Rational Test RealTime User Guide, and refer to the chapter Runtime Analysis->Runtime Tracing .

Static Metrics

Memory profiling reports for all supported languages-C, C++ and Java-are stored in XML-based files within the Project folder or the TDP folder, depending on the language.

To open these reports using the Project Browser tab in the Project Window on the right-hand side of the Test RealTime user interface, right-click a test node and select View Report->Memory Profile.

To open these reports using the Test RealTime menu:

Select the menu item File->Browse Reports.
Select the Add toolbar button.
Select Metric from the Files of Type dropdown list.
Browse to and then select the files you are interested in viewing.
Repeat steps 2-4 as needed.
Click the Open button in the Report Browser window.

To open these reports from the command line, simply treat the one or more files you are interested in viewing as parameters to the command line studio.

The report is organized into two levels, the File view and the Object view:

File view - When the Root folder is selected, a bar graph of one of seven Halstead metrics is presented for every file that was selected. The particular metric that is displayed can be changed using the menu. When any other node is selected, static metrics specific to that node are presented.
Object View - When the Root folder is selected, a graph is presented. This graph compares certain static metrics for each function/method - this can be changed using the menu. When any other node is selected, static metrics specific to that node are presented.

For detailed information about viewing static metrics reports see the Rational Test RealTime User Guide, and refer to the chapter Runtime Analysis->Static Metrics.