Capture work status
Purpose:
|
To gain an objective, up-to-date understanding of the general status of the testing work against
plan.
|
There are different ways to approach this step, and much of the approach will depend on your project culture. Where
available, gather and collate progress reports prepared by individual team members or sub-teams. Project time sheets
are another possible source to consider. Where project scheduling systems such as Microsoft Project are actively used
and updated with actual progress this provides another useful information source. Where available and actively used,
you might also derive objective status or progress metrics from configuration and change management systems.
For this step and subsequent steps that deal with gathering information and assessing the test effort, try to obtain a
balanced view incorporating both objective and subjective measures. Remember that objective numbers only give part of
the picture and need to be supported and explained by the current project "climate". Conversely, don't rely purely on
hearsay and subjective speculation about the test effort: look for supporting objective evidence. We recommend you
supplement your objective data by discussion with either team leads or where possible individual team members to gather
subjective assessments and gauge how much confidence you can place in the objective data.
|
Gather test effort productivity and effectiveness metrics
Purpose:
|
To gather and examine objective data that enables the assessment of the testing performed by the test
team.
|
Investigate how much effort has been spent on the identification, definition, design, implementation and execution of
tests. Keep an eye out for signs of excessive effort being devoted to one aspect of the test effort to the detriment of
others. Look also for areas where effort may be unproductive or not showing sufficient benefit for the level of effort
being expended.
Look also at the effectiveness of the testing. Look for data that supports your initial observations of effectiveness.
Consider aspects such as defect discovery rate, defect severity counts, duplicate defect statistics, and defects
detected as test escapes.
|
Gather Change Request distribution, trend and aging metrics
Purpose:
|
To gather and examine objective data that enables the assessment of the issues and defects logged by the
test team.
|
Identify important trends evident in the Change Request data. In general it's less important for this task to spend
time analyzing data volumes and more important to identify what the relative data trends are indicating. Look for
positive signs such as a steady, continuous rate of defect discovery, or a light ongoing increase or decrease in
discovery rate over time. Be on the lookout for sharp peaks and troughs in discovery rate that indicate the test team
may be encountering process, environmental, political or other problems that are reducing their productivity.
Look at trends in defects closures. Look for significant increases of closures by development staff as "not
reproducible"; identify cases where this is a result of insufficient analysis of the failure having been performed by
the test team and quantify the extent of this problem. Look at trends in defects being closed by development staff as
"functioning as designed"; identify cases where this is a result of insufficient analysis of the specification having
been performed by the test team and quantify the extent of this problem. Be careful to confirm these indications are
not false and due instead to overworked developers triaging their workload. Some analysis should also be done of defect
verification trends as fixes to defects are released to the test team in subsequent builds: look out for trends that
indicate defects awaiting verification by the test team are aging or growing to an unmanageable number.
Look for other trends that indicate problems. Look at the the way in which defects and other change requests have been
recorded or managed by the test team: ambiguous and insufficient information on a change request is difficult and
frustrating for a developer to take action on. The team should take care to monitor that the quality of the information
recorded against defects remains-on average-relatively high. Take the opportunity to improve the clarity of the
associated Change Requests, eliminating ambiguity and emotive language and reasoning. Work together with the
individuals who created these work products to ensure the essence of the problem is clearly stated and encourage them
to find factual and accurate ways to approach discussing the Issues.
Also look out for imbalances in defect distribution on a number of different dimensions. Look for functional areas of
the application or the specification that have low defect counts raised against them: this may indicate an exposure
that insufficient testing has been undertaken in that functional area. Look also at distribution by test team member:
there may be indications that individual team members are overworked and that productivity is suffering.
|
Gather traceability, coverage and dependency metrics
Purpose:
|
To gather and examine objective data that enables the assessment asset traceability.
|
Analyze the state of the traceability relationships between the testing assets-Test Ideas, Test Cases, Test Scripts,
Test Suites and Change Requests-and the upstream and downstream assets they relate to. Look for signs that indicate the
test effort is focused on the correct areas and a useful set of motivations. Look also for negative indications that
suggest certain aspects of testing are missing or are no longer of importance: If the requirements or development teams
are working on areas not represented by the current test effort, this should raise concerns.
|
Evaluate metrics and formulate initial assessment
Purpose:
|
To evaluate and assess the metric data and formulate an initial assessment of the effectiveness of the test
effort against plan.
|
Collate all of the information you have gathered and evaluate it as a collective whole. Remember that each piece of the
data gathered only addresses one aspect of the total assessment, and you must formulate your assessment of the test
effort based on a balanced and considered view of all data.
Record you initial assessment in a format that will be suitable for the stakeholders to make comments and give feedback
on.
|
Record findings
Purpose:
|
To document summary findings for inclusion in project management reporting and to enable analysis of
subsequent status assessment against earlier assessments.
|
This task produces summary status information that is important to the project manager and other roles in the
management team. These roles will use the summary findings to make informed decisions about the project.
We recommend your record some aspects of the test effort assessment in a format that allows subsequent assessments to
be compared and contrasted with previous ones. This will enable you to analyze the relative trend in test effort
improvements over time.
|
Present assessment and gather feedback
Purpose:
|
To engage stakeholders and obtain their feedback on whether the actual testing effort is serving their
needs.
|
Present your assessment for stakeholders to comment and offer feedback on. The format or method for doing this will
differ from project to project: in some cases it will be a series of informal conversations, in another simply a
posting on a project intranet web-site, and in others a formal presentation-choose a format that suits your culture.
Even with the best planning and specification documents possible, there will usually be differences between the
original expectation and intent of those documents and the resulting end product. This is as true for testing and
evaluating software as it is for the software development itself. The value of this step is to take the opportunity to
elicit the stakeholders feedback and identify where the careful planning and documentation has not achieved what was
originally expected or intended.
|
Plan and implement improvement initiatives
Purpose:
|
To identify areas for improvement and formulate initial strategies for achieving those improvements.
|
Based on your analysis and the feedback you've received from various stakeholders, identify opportunities for
improvement. Look for ways to make the testing more effective, productive and efficient. This might involve:
reassigning staff, including pairing staff to work more effectively or employing specialized contractors; using
productivity tools to improve efficiency; finding alternative approaches and techniques that are more productive in
terms of finding defects.
In most cases it's better to make small, incremental improvements to the test effort and avoid the risk of derailing
the project with large, unsettling changes: In some cases a bigger change is warranted and useful. Use your best
judgment to formulate an appropriate approach to improvement and discuss your ideas with other management staff to get
their input before committing the team to embrace large changes.
|
Monitor and support improvement initiatives
Purpose:
|
To ensure that necessary improvement initiatives are achieved in a satisfactory and timely manner.
|
For the improvements to be effective, you will need to manage their success. Identify ways that you will be able to
monitor improvement initiatives-preferably in advance on their adoption-to assess their effectiveness. Either actively
monitor the progress being made in adopting the changes yourself, or appoint someone else on the team to do so.
Most changes meet resistance or problems that must be overcome for them to be ultimately successful. Allow time for and
be prepared to quickly address any issues that arise and prevent the initiative from succeeding. Be sensitive to
peoples natural reluctance to change and find ways to address their concerns appropriately.
|
Evaluate and verify your results
Purpose:
|
To verify that the task has been completed appropriately and that the resulting work products are
acceptable.
|
Now that you have completed the work, it is beneficial to verify that the work was of sufficient value, and that you
did not simply consume vast quantities of paper. You should evaluate whether your work is of appropriate quality, and
that it is complete enough to be useful to those team members who will make subsequent use of it as input to their
work. Where possible, use the checklists provided in RUP to verify that quality and completeness are "good enough".
Have the people performing the downstream tasks that rely on your work as input take part in reviewing your interim
work. Do this while you still have time available to take action to address their concerns. You should also evaluate
your work against the key input work products to make sure you have represented them accurately and sufficiently. It
may be useful to have the author of the input work product review your work on this basis.
Try to remember that that RUP is an iterative delivery process and that in many cases work products evolve over time.
As such, it is not usually necessary-and is often counterproductive-to fully-form a work product that will only be
partially used or will not be used at all in immediately subsequent work. This is because there is a high probability
that the situation surrounding the work product will change-and the assumptions made when the work product was created
proven incorrect-before the work product is used, resulting in wasted effort and costly rework. Also avoid the trap of
spending too many cycles on presentation to the detriment of content value. In project environments where presentation
has importance and economic value as a project deliverable, you might want to consider using an administrative resource
to perform presentation tasks.
|
|