Kuldeep - founder of KR Testing Solutions

About Me

2010-07-16T11:17:00.001-07:00

Professional History and Credentials

I have been involved in Software Testing as Software Test Engineer from past few years, and have tested a number of software applications on a variety of platforms. I have also been involved in some form of automated testing or another during this period.

At SGS Infotech, I was involved in testing Geographical Information System (GIS) applications written in ASP. Here I tested the “PROGis”, which was allow to view the acquired Land record of "Taj Expressway, Noida" using the different-2 maps (using Archview). I was also involved in testing the Enterprise Resource Planning (ERP) application written in Java. I tested the “JOfficeDesk”, which allowed to automate the complete SGS Infotech.

During my employment at Orion India I successfully implemented Testing Team. I was in charge of testing Complete Garment ERP AMTPlus. I tested the Complete ERP System there. You can get the summary on www.obsllc.com . I have played a good role of Team Lead there. Now lot of Garment players are using this system in US like SLFashion, Apollo Garments etc.

During my employment at WNS IT Services I successfully Tested Mortgage Banking application Risk Management DB (RMD)of First Magnus Finance Corporation, USA.

Current I am working with MNC as a Testing Architect.

KR TESTING SOLUTIONS

KR Testing Solutions is an independent quality assurance and testing Organisation. Founded in 2006. We provide comprehensive software testing services and solutions for all testing needs. Our goal is to provide the better quality product to our clients.

KR Testing Solutions delivers effective, customized solutions for clients in a wide array of industries and technologies, and at significant cost savings. We have the expertise to perform functional testing of applications, performance, load and stress testing of web sites, and automation of regression tests. In addition, we will check sites for browser compatibility and accessibility compliance.

I hope that you will give me a chance for providing a better solutions of your problems.

Go to KR Testing Solutions

Tutorial 1 - What is QTP?

2010-07-16T11:17:00.000-07:00

This tutorial introduces Quick Test Professional , features of QTP as Functional Automation Tool & the concept of ADD-ins

Summary

* Quick Test Professional , popularly know by its acronym QTP is the flagship functional automation testing tool from Mercury Interactive now acquired by HP
* It is an icon based tool, which automates the functional & regression testing of an application
* QTP is easier to use and implement for both technical & non technical testers in comparison to other functional testing tools available.
* Quick Test is the market leader in Functional Automation Tool with over 50% market share
* QTP's Scripting Language is VB Script which is easy to use , understand and program
* Quick Test Professional enables Business Process Testing (BPT)
Supports large pool of software development environments like SAP , Web , Oracle etc.
* The trainings have been recorded using QTP version 9.5 but you may use any higher or lower versions for your learning purposes
* For All Hands On in these trainings, we will be using “Flight Reservation” Application which comes bundled with QTP

Tutorial 2 - First Look Flight Reservation Application

2010-07-16T11:16:00.001-07:00

This tutorial introduces Flight Reservation Application which will be used for hands-on for the rest of the tutorials.

Notes:
* Flight Reservation Application comes pre-installed with QTP
* Using Flight Reservation , you can book a flight between two cities, even modify or delete and exisiting booking.
* You may also fax a booking to a customer with your custom signature.
* Flight Reservation has a Reports and Graphs section which helps analyze the ticket selling trends.
* A small number of bookings is pre-populated for any new agent , so that you do not have to create test data.
* The application has been designed to help learn all the features provided by QTP. At times , you may find this application buggy but your focus should not be technical accuracy of the application, but on its use as a tool, to learn various features of QTP.

Tutorial 3 - First Look QTP IDE

2010-07-16T11:16:00.000-07:00

This tutorial introduces the Quick Test Professional IDE

Video Transcript with Key Takeaways highlighted

* Without further ado , Lets look into QTP
* To launch QTP, In Start Menu, Choose Programs > Quick Test Professional Folder > Quick Test Professional
* The first time you start QTP, the Add-in Manager dialog box opens.It Displays list of all installed add-in along with license expiry date.
* It is recommended you select only the add-ins required for your particular testing session . Because at times , different add-in interfere with each other degrading object identification and QTP's performance. QTP will remember the add-ins you load so that the next time you open QTP the add-ins you selected in the previous session are selected by default.Also, If you do not want this dialog box to open the next time you start QTP clear the Show on startup check box.Click OK.
* The Start Page describes the new features in this release—including links to more information about these features.It also provides links to Process Guidance, a tool that offers best practices for working with QTP.You can open a document from the list of Recently Used Files,or you can click the buttons in the Welcome! area to open new or existing documents.If you do not want QTP to display the Start Page when you next open QTP ,select the "Don’t show the Start Page window on startup" check box.When you select this option, the Start Page is also automatically hidden for the current QTP session as soon as you open another QTP document.To display the Start Page again, select View > Start.
* Title Bar Displays the name of the active document. If changes have been made since it was last saved, an asterisk (*) is displayed next to the document name in the title bar.
* Menu Bar Displays menus of QTP commands.
Toolbars Contains buttons to assist you in managing your document
Document Tabs Enables you to navigate open documents
* Keyword View Displays test steps in a graphical representation
* Expert View Displays test steps as a VB Script line.
* Active Screen Provides a snapshot of your application as it appeared when you performed a certain step during the recording session.
* Data Table Assists you to parametrize your test.
Test Flow Displays the hierarchy of actions and action calls in the current test, and shows the order in which they are run.
* Below are Tabs For Other Panes.
* You can change the look and feel of the main QTP window, as required
* In the QTP window, select View > Window Theme, and then select the way the window should appear from the list of available themes.
* You can move the QTP window panes to suit your own personal preferences.Say we want to Move the Test Flow Pane .In the QTP window, drag the title bar of the Test Flow Pane.As you drag the pane, markers are displayed in the active pane and on each edge of the QTP window.Drag the Test Flow and hold the cursor over the various markers.A shaded area is displayed, indicating the new location of the pane.Lets move it to Right. Release the mouse button.The Test Flow Pane snaps into place and is displayed as a new pane in the shaded area.
That’s all to the QTP IDE

Tutorial 4 - HP Quick Test Professional

2010-07-16T11:15:00.001-07:00

This tutorial identifies the TEST STEPS that need to be automated for your First Test Script using QTP

Video Transcript with Key Takeaways highlighted

* Now, Lets Record Our First Script.For Our Application Under Test i.e. Flight Reservation - lets validate a simple test scenario out of possible many
The Scenario Would Be"Check that user successfully logs in to the application on inputting valid Agent Name & Password"
Test Steps required to validate this scenario would be
Step 1) Open Flight Reservation Application
Step 2) Enter Valid Agent Name
Step 3) Enter Valid Password
Step 4) Press Ok
Step 5) Close Application After Successful Login.
Lets automate these 5 steps in QTP

Tutorial 5 - Record and Run Settings QTP

2010-07-16T11:15:00.000-07:00

This tutorial introduces the Record & Run settings & It demonstrates , how to record a script in QTP

Video Transcript with Key Takeaways highlighted

* In QTP click the record button.The Record and Run settings Dialog Box Opens.This box opens automatically each time you begin recording a new test.
Record & Run settings shows a tabs corresponding to add-ins installed and loaded for your test.So, for example if you have SAP Add-in Installed and loaded you will see a SAP tab.
The Windows Application tab is always available and be used for environments, such as Visual Basic, ActiveX, and terminal emulators.
* For any Environment, the Record and Run settings can be classified into two generic groups
1) Record & Run on ANY window belonging to that particular environment
2) Record & Run on a SPECIFIC window belonging to that particular environment - which is the recommended Option
For the time being , lets stick to default settings .Once settings are done , QTP remembers and uses the same settings for additional record sessions on the same test, unless you manually open the Record and Run Settings dialog box to modify the settings.
Click okay
* QTP Starts Recoding Mode
Step # 1 is to Open Flight Reservation Application
Click Start Menu > Program Files > Quick Test Professional > Sample Applications > Flight Reservation
Step # 2 Enter a Valid Agent Name greater than 4 characters say Guru
Step # 3 Enter a Valid Password which is by default MERCURY
Step # 4 Click Okay
Step # 5 Check that the Flight Reservation opens successfully which it is . Click the close button.
* All the 5 steps are now recorded
In QTP , Stop recording
* Save the script as "LogIn"

Tutorial 6 - All about Object Identification

2010-07-16T11:14:00.002-07:00

This tutorial demonstrates how QTP identifies an on - screen GUI Object and the concept of QTP's Test Object Model

Video Transcript with Key Takeaways highlighted

* QTP also uses a "human" like technology for object identification
* During Record Time QTP tries to learn properties of a GUI object on which operation is performed.
* During Run-Time QTP compares the stored object properties with actual properties of object available on screen and uniquely identifies an object independent of its location on screen
* The stored object and together with its properties is called TEST Object
* During Run-Time, the actual object available on the application under test is called Run-Time Object
* This is Quick Tests "Test Object Model"
* Information about the Test Objects is stored in Object Repository
* Add-ins help in instructing Quick Test

Tutorial 7 - Expert View in QTP

2010-07-16T11:14:00.001-07:00

This tutorial interprets the Expert View and gives its Syntax

Video Transcript with Key Takeaways highlighted

* In the Expert View , each line represents a Test Step in VB Script
* To understand expert view better , lets analyze Step # 2 from our TEST recorded earlier
* An Object's Name is displayed in parentheses following the Object Type.Here the Object Name is Login and * Object Type is Dialog
* Objects in Object Hierarchy are separated by a "dot".Here Dialog and WinEdit are fall in the same Object Hierarchy
* Just to put things in perspective, Object Hierarchy is Object Oriented Concept where set of objects that are grouped together in a parent-child relationship.In our case Dialog Box is the Parent Object and WinEdit is the Child Object
* The Operation performed on the object is always displayed at the end of the statement followed by any values associated with the operation.Here the word "Guru" is inserted in the AgentName Edit Box using the Set Method
* Syntax for a statement in expert view is GUI object on which the operation is performed along with its complete hierarchy followed by the Operation on the Object and value associated with that Operation

Tutorial 8 - Compare Keyword view and Export view in QTP

2010-07-16T11:14:00.000-07:00

This tutorial interprets the Keyword View and compares the Expert & Keyword View

Video Transcript with Key Takeaways highlighted

* The Keyword View is comprised of a table-like view where
* Each step is a separate row in the table and
* Each column represents different parts of the steps.
* Item Column contains the item on which you want to perform the step. This column uses icons displays the hierarchy of the GUI object on which operation is performed
* Operation Column contains the operation to be performed on the item.
* Value Column contains the argument values for the selected operation,
* QTP automatically documents each step for easy understanding in the Documentation Column
* These 4 columns are default but you can also use assignment & comment columns in Keyword View
* That’s all to the Keyword View
* Lets Compare the Keyword and Expert View using Step no 2 from our script. You will observe that the same object hierarchy is displayed in both Expert & Keyword Views and they map to the same operation and argument value.
* Essentially , Keyword & Expert view contain the same data but arranged in different format.
* In fact you can perform all operations like create , modifying a step . using the Keyword View but to gain mastery over the tool we will restrict ourselves to the Expert View

Tutorial 9 - QTP

2010-07-16T11:13:00.001-07:00

This tutorial interprets the script that was recorded in earlier tutorials

Video Transcript with Key Takeaways highlighted

* Now , Lets go ahead and understand our recorded test.
* The first step is the SystemUtil.Run Command which is used by default by QTP to open a application.So during recording , using the Windows Start Menu , when we finally navigated to the "Flight Reservation" application QTP identified the location of its executable file and inserted the System.Util Command to Open it.
* Step No 2 is Setting Agent Name as Guru as shown in the Active Screen
* Step no 3 is Pressing the Tab key on keyboard to bring Focus from Agent Name Field To Password Field , which is exactly this step. Human users need to use tab or click operations to focus on a particular object on screen.. On the other hand , QTP can directly identify an object using object properties and does not required these "maneuvering" operations. We can delete this step , as QTP will still be able to set the password field without this operation
* Next Step is Setting the Password as Mercury. QTP automatically encrypts passwords entered while recording to avoid security breaches. This value can not be decrypted ie. There is no way to recover the original value using this cryptic data. You can explicitly encrypt a password using the Password Encoder Tool. For our learning purposes , we will use the password in its raw form. And the operation will also change to Set
* Next Step is clicking the okay button
* Next Step is closing the application after successful log on
* So the 5 steps mentioned in our test script are recorded in QTP

Tutorial 10 - What is Replay in QTP?

2010-07-16T11:13:00.000-07:00

This tutorial introduces the concept of REPLAY . It demonstrates the Run Settings available in Quick Test

Video Transcript with Key Takeaways highlighted

* Now , lets go ahead and REPLAY the script to ensure the test steps have recorded correctly.
* Click the Run Button.
* The Run Dialog Box Opens which enables you to specify the location in which you want to save the run session results.
* This option displays the default path and folder name in which results are stored. By default results are stored in Test Folder .A new sub-folder is created with the name RESn .The number n is incremented for each run.
* You may accept the default settings or specify a folder of your choice
* Temporary run results folder options saves the run results in a temporary folder. This option overwrites any results previously saved in this folder.
* Its recommended that while developing your test script choose the Temporary option and once the script is baseline you can use the new folder option
* In our case we will use the temporary option and click okay. QTP enters the run mode
and will start executing steps one by one.
* In QTP window you can see a yellow marker pointing at the step which is currently being executed.
* During Replay the script performed exactly the same 5 steps that were recorded which signifies that there were no errors in recording.
* Test Results are also shown

Tutorial 11 - Test Results in QTP

2010-07-16T11:12:00.003-07:00

This tutorial demonstrates the various features available in the Test Results generated by QTP

Video Transcript with Key Takeaways highlighted

* Lets understand the Test Results generated by QTP
* Right Hand Side shows Test Results Summary
* Left Hand Side is Test Results Tree - an icon-based view of the test steps that were performed while the test was running. Similar to the test tree in Keyword View
* If you select a step in the tree , the right panel gives it complete details
* This Apart
* You can capture Movie / Screenshots of entire Test Run using Tools > Options > Run Tab
* In our case this movie was recorded and , screen shots were captured
* You can print/export full/part of results in HTML , Word or PDF format.
* You can exports the results to Quality Center
* You can change the format of Results by using Results.xml and creating a XSL

Tutorial 12 - All about parameters in QTP

2010-07-16T11:12:00.002-07:00

This tutorial demonstrates parameterization

Video Transcript with Key Takeaways highlighted

* You may be wondering why take the Herculean effort to automate this simple scenario
* Check that user successfully logs in to the application on inputting valid Agent Name & Password
* The need becomes obvious if we extend the scope of the scenario to include a Combination of valid ALPHANUMERIC Agent Name & Password
* In this case , the Test Steps Will Remain the Same. But we will have more combinations of Data To TEST. We will restrict to just 3 of possible 8 combinations
* To achieve this you can either copy the six steps and give different data values which is in fact * something you would do manually. Or you can use Parameterization
* The easiest way to parameterize an argument , in our case Guru is in Keyword view , Click the Parameterization Icon
* Value Configuration Dialog Box Opens. Currently the value is set to a Constant. Click on Parameter Radio Button. QTP assigns a default name to these parameter. You can give a name of your choice. Click Okay
* In the Global Sheet , a column with Header "Agent Name" and value Guru is created. You can enter more values for this parameter
* Likewise you can also parameterize the argument Password and enter different test data sets.
* What this Data sheet means is QTP will iterate the same 6 steps that we have recorded 3 times. During * first iteration it will use the data in the first row. During second it will use data in the second row and so on
* Now lets run the script. This is the first iteration. This is the second iteration
* In the status bar QTP gives information of the row, it is currently using as test data and highlights the * corresponding row in the data sheet. The results will show summary of the 3 iterations.
* A brief recap of parameterization
* Parameterization allows us to pick different values at run time.
* It Reduces Time and Effort.
* Usage of Data Drivers allow us to use the same data for various input boxes. Data Drivers is feature provided by QTP which shows all the constants that could be parameterized in one single window

Tutorial 13 - What are the standard checkpoints in QTP?

2010-07-16T11:12:00.001-07:00

This tutorial introduces Checkpoints

Video Transcript with Key Takeaways highlighted

* You may have notice that results generated for our test script has no Pass/ Fail Status without which our automation is incomplete
* The expected TEST RESULT for our scenario should be - Flight Reservation Window
* which is this screen should appear after entering valid username and password. To accomplish this we will need to record an additional step # 6 which is Check Flight Reservation Window is Displayed..
* Automation of this verification step can be achieved using Checkpoints- A checkpoint is a verification point that compares the current value with the expected value for specified properties of an Object. If they current and expected value match it generates a PASS status otherwise FAIL status
* Lets go ahead and record step # 6
* To add a checkpoint , right click on the step # 5 after which checkpoint needs to be inserted .Choose Insert Standard Checkpoint. Checkpoint Properties Dialog Box Opens.
* QTP assigns a default name to checkpoint. You can input your preferred name
* The table shows all the recorded properties and their corresponding values for the object. The Selection mark indicates that these properties will be checked
* ABC icon indicated that the property values are a constant. If you parameterize any of the selected properties the icon changes correspondingly
* Lets stick to the default and insert the statement after the current step. Click okay
* A check statement with checkpoint name is inserted at line # 6.Lets replay the script
* The script gives a Run-Time Error
Well this is a common source of error for beginners and happens because at step # 5 , QTP closes the Flight * Reservation Screen and when the execution reaches step # 6 there is no Flight Reservation Screen Object available to very its properties. You need to ensure , that the object which are verifying is available while QTP executes the Checkpoint Step
* This error can be rectified by changing the sequence of Tests Steps. You need to switch
* Step 5 & step 6 . In the new scenario step 5 is verify the Flight Reservation Window Exists and step 6 is to Close the application
* In QTP , you just need to cut step # 6 and paste it at location of step # 5.Lets replay the script again. * The Script passes and the results tree gives the checkpoint values that were compared. That's all to standard checkpoints
* A recap of checkpoints-
* There are many types of checkpoints and the most generic is the standard checkpoint.
* Standard checkpoints compare the expected values of object properties captured during recording to the object's current values during a run session

Tutorial 14 - Types of checkpoints in QTP

2010-07-16T11:12:00.000-07:00

This tutorial demonstrates different types of Checkpoints in Quick Test Professional

points to remember

A Checkpoint is a confirmation or verification point in which the value of some property which is expected at a particular step is compared with the actual value which is displayed in the application. Based on the expected values Checkpoints are classified as follows

* Page Checkpoint : A Standard Checkpoint created for a web page can be called a Page Checkpoint. It is used to check total number of links & images on a web page. Page Checkpoints can be used to check Load Time i.e. time taken to load a web page.
Bitmap Checkpoint helps a user in checking the bitmap of an image or a full web page. It does a pixel by pixel comparison between actual and expected images.
* Image Checkpoint enable you to check properties like source file location of a web image. Unlike , Bitmap Checkpoint you can not check pixels(bitmaps) using image checkpoint.
* Text Checkpoint is Used to check expected text in a web-page or application. This text could be from a specific region of the application or a small portion of text displayed
* Accessibility Checkpoints verifies compliance with World Wide Web Consortium (W3C) instructions and guidelines for * Web-based technology and information systems. These Guidelines make it easy for disabled to access the web.
* Database Checkpoints create a query during record time and database values are stored as expected values. Same query is executed during run time and actual & expected values are compared.
In Table Checkpoint , you dynamically can check the contents of cells of a table (grid) appearing in your * environment. You can also check various table properties like row height , cell width and so on. Table Checkpoint is similar to Database Checkpoint
* Using XML Checkpoints you can verify XML Data ,XML Schema, XML Data

Tutorial 15 - How to insert output values in QTP?

2010-07-16T11:11:00.001-07:00

This tutorial demonstrates the concept of Output values in Quick Test Professional

points to remember

An output value step ,is a step in which a object property value is captured at a specific point in your test and stored at a desired location. The stored values can be used as input at a different points in test script. Multiple properties of an object can be selected and outputted.

Types of Output Values

* Standard output value
* Text /Text Area output value
* Data base output value
* Xml output value (from application/resources)

Tutorial 16 - Conditional Loops in QTP

2010-07-16T11:11:00.000-07:00

This tutorial demonstrates advanced coding in QTP using if and else loop

Video Transcript with Key Takeaways highlighted

* From your testing background you will certainly infer that a more accurate Test scenario would be * Validate the Login Functionality of Flight Reservation which should have two sub scenarios
* Check that user successfully logs in to the application on inputting a COMBINATION OF valid ALPHANUMERIC Agent Name & Password
* Check that user log on fails on inputting INVALID Agent Name & Password
* More so , a robut automation script should be able to accept and handle both valid and invalid login details
* We have the sub-scenarios already recorded So the challenge is to integrate them.
* You May observe for both the scripts - STEPS ,Launch Flight Reservation Application, Enter Agent Name ,Enter Password ,Click OK ,while steps Checkpoint ,,Close Flight Reservation Window, for positive scenario, and steps ,Output Error Information ,Close Error Info Window ,Close Login Dialog Box, for negative scenario are different
* There are many ways to integrate them and once of the ways ,is to use if and else loop and check whether error information screen exists after entering the agent name and password. if yes do the steps for negative scenario, if no do the steps for positive scenario
* In QTP Window, After step # 4,Add a step if else loop, The check condition is whether error information screen exists. Copy this step and paste it as check condition.
* Delete the Click Operation. And replace it with Exists method. This method is applicable to almost all objects and checks whether the particular objects exists on screen or not
* If result is true do the negative scenario steps. I will cut and paste the steps inside the loop
* Copy the steps from the positive scenario and paste it in the else loop. Lets run the test for one valid and one invalid login credentials. The test Runs successfully .Note is you saved the tests in the order mentioned in the tutorials
* PositiveLogon to NegativeLogon and Negative Logon to Combined you should have no problem running the test.
* Apart from if and else you can also use - if elseif., while wend.,do case ,for Loops. as per your requirements

Tutorial 17 - How to user Report.Report Event in QTP

2010-07-16T11:10:00.002-07:00

This tutorial demonstrates the use of function Reporter.Report Event and Results Formatting.The tutorial will ask you to develop a script. To maximise your learning , please do complete the scripting exercise.

points to remember about QTPs Test Results Formatting

* You can use Reporter.ReportEvent to report custom test steps in QTP's test results tree
Syntax -Reporter.ReportEvent EventStatus, ReportStepName, Details [, ImageFilePath]
* Event Status can have values
* 0 or micPass sends a pass status to test result window
* 1 or micFail sends a pass status to test result window
* 2 or micDone sends a message to test result window without affecting the Pass/Fail status
* 3 or micWarning sends a warning message to the result window
* When test cases are executed using automation tools it may be difficult for certain users to understand the test results You can use results.xml to create an xsl which will present the test results as per your preferences
* You can use VBScript Library functions to store the results in xls or a text file.

Tutorial 18 - Complete Guide on Actions in QTP

2010-07-16T11:10:00.001-07:00

This tutorial demonstrates Actions. It uses the vanilla Test Script created in previous tutorials with 5 steps to log in into Flight Reservation.This tutorial is the longest in all QTP tutorials and its recommended you take notes while viewing it.

key takeaways:

* Actions help divide your test into logical units or Business Processes
* Actions help create a script which is more modular and efficient.
* When a script is newly created it consists of only one action .A script can consist of one or more Actions .
* There two types of Actions 1)Reusable Actions & 2) Non-Reusable Actions
* Reusable Action can be used in other Tests. They can be used in the same Test Script multiple times.
* Non reusable Action can not be used in other Tests. They can be called in the same script only once
* You can split an existing action in two ways 1) Independent of each other which splits the selected action into two sibling actions 2) Nested Action which splits the selected action into a parent action whose last step calls the second, child action
Q* TP provides 2 type of datasheets 1) Global & 2) Local
* GLOBAL Datasheet : It is Unique for the entire test. Any Action can access and write data into Global Datasheet. Sheet is named “GLOBAL”
* LOCAL Datasheet : Equal to number of Actions in the sheet. An Action can read and write data into its own local Datasheet only. Sheet name = “ACTION NAME”
* There are two methods to import Actions into a Test 1) Call to Copy & 2) Call to Existing.
* Call to COPY of an Action : When you make a Copy of an Action , the action is copied in its entirety, including checkpoints, parameterization, and the corresponding action tab in the Data Table into the calling test . When you insert a copy of an existing action, you can make changes to the copied action, and your changes will not affect nor be affected by any other test. You can insert copies of both reusable and non-reusable actions
* Call to an EXISTING Action : Calls to actions are read-only in the calling test. They can only be modified in the test in which they were created. Enables you to use the same action in several tests and makes it easy to maintain tests. You can make calls to only “Reusable” actions.
* You can change the action iteration frequency by selecting Action Call Properties > Run Tab

Tutorial 19 - QTP Object Identification

2010-07-16T11:10:00.000-07:00

This tutorial demonstrates how Object Identification works in QTP.Before you begin , you must know an object property and its value is called Object Description used to identify the corresponding Object.For example , for a WebButton property "name" and its value "Login" together can be termed as Object Description for that Web Button.The term Unique Description implies finding typical Object Description for fast object identification.

Video Transcript with Key Takeaways highlighted

* QTP stores Object and its properties in Object Repository to identify them during run-time.
* An Object could have large number of properties associated with it. For example in Web Environment a Button could have the following properties associated.
* If QTP will try and remember all the available properties for an object, size of Object Repository will bloat and script execution time will increase drastically
* To avoid this, QTP by default , does not store all the properties of an object but a limited no of typical properties for an object which helps in its unique identification.
* These settings are store in Object Identification
* In QTP in Tools Menu , select Object Identification
* In Object Identification Dialog Box you can see a drop down of all the environments installed and loaded.
* You can select an environment and QTP auto populates list of all the supported objects belonging to that environment. On the right QTP lists the properties that will be stored for the object
* The list is divided into 2 columns 1) Mandatory & 2) Assistive
* 1) Mandatory properties will be stored by default for that object
* 2) In case during record time of script QTP can NOT create a unique description of the object it will store the assistive properties
* To understand this better consider the example of an Image. QTP will store value of alt ,html tag, image type, properties mandatory even if it can uniquely identity it using the alt property alone.
* In case it can not create unique description using mandatory property ,QTP will store assistive property. In this case QTP will store the class property .
* If this property helps create a unique description of the object, QTP will not store file name, height property. If class property is not sufficient to create unique description, QTP will store file name property . If file name property creates a unique description QTP will not store height property and so on.
* Suppose during Record Time, QTP has only stored class property
* At Run - time ,QTP will forget the distinction between mandatory and assistive properties and compare all the recorded properties.
* Even if any one of the properties does not match its stored value , Script fails

Tutorial 20 - All about Smart Identification in QTP

2010-07-16T11:09:00.004-07:00

This tutorial demonstrates SMART Identification process in QTP.For the first time, the tutorial makes use of a Web Environment.

points to remember

* If the usual object identification process fails , QTP triggers Smart Identification , which is more flexible mechanism for identifying objects provided it is enabled in Object Identification settings.
* Smart Identification uses two categories of properties 1) Base Filter Properties & 2) Optional Filter Properties
* Base Filter Properties. It contains the most fundamental properties of a particular test object class; those whose values cannot be changed without changing the essence of the original object
* Optional Filter Properties. Other properties that can help identify objects
* Quick Test "forgets" the learned test object description and creates a new object candidate list containing the objects that match all of the properties defined in the Base Filter Properties list.
* QTP the Uses Base Filter Property to reduce the Object Candidate list
* The idea is to narrow down only one object matching the some or all of the saved description properties
* If smart identification is invoked during a test run ,in the test results tree a warning message is generated indicating smart identification was invoked and a smart identification step is inserted.

Tutorial 21 - Object Property Modification

2010-07-16T11:09:00.003-07:00

This tutorial uses Object Property Modification to address the property change issue faced in Tutorial 20 (smart identification).It is recommeded you go through Tutorial 20 before taking this tutorial

Video Transcript with Key Takeaways highlighted

* You may have observe that smart identification slows down script execution which is not desirable
* To avoid smart identification , we can change the default object identification properties
* In QTP , Object Identification , lets remove "name" from the mandatory properties and replace it with "html id" to make our test independent of name changes
* You can re-record the same steps for the script
* Now if you change the name from "Submit" to "Login" and run the script the script executes without any smart identification
* Similarly , in your application under test if any of the mandatory or assistive properties change frequently for an object you can replace it with some other suitable property to enable faster script execution
* Property tweaking is an experience game field and you will pick it up as you age with the tool

Tutorial 22 - Ordinal Identifiers

2010-07-16T11:09:00.002-07:00

This tutorial introduces Ordinal Identifiers .It is recommended you complete Tutorials 19 , 20 & 21 before taking this tutorial.

points to note:

If mandatory and assistive properties are insufficient to identify an object during record session ,QTP uses Ordinal Identifier in addition to other captured properties to identify the objects during a record session
By default, an ordinal identifier type exists for each test object class.
In the Object Identification Window, you can modify the default Ordinal Identifier for an Object

There 3 types of Ordinal Identifiers
* 1) Index based
* 2) Location based
* 3) Creation Time

Index Based

* When using Index based ordinal identifier, while recording , QTP will assign a value to INDEX property of an object
* The value is based on the order in which the object appears within the source code.
* The first occurrence has value 0
* Index property values are object-specific.
* Therefore, if you use Index:=3 to describe a WebEdit test object, Quick Test searches for the fourth WebEdit object on the page.
* Likewise , you use Index:=1 to describe a WebButton test object, Quick Test searches for the third WebButton object on the page

Location Based
* When using location based ordinal identifier, while recording , QTP will assign a value to LOCATION property of an object to uniquely identify the object.
* The value is based on the order in which the object appears within the window, frame, or dialog box, in relation to other objects with identical properties.
* The first occurrence of the object is 0.
* Values are assigned in columns from top to bottom, and left to right.

Creation Time
* When using creation time ordinal identifier, while recording , QTP will assign a value to CreationTime property of an WebBrowser
* The Identifier is only available for the Web Environment
* This value indicates the order in which the browser was opened relative to other open browsers.
* The first browser that opens while recording receives the value CreationTime=0 and succeeding browsers are give values 1 , 2 , 3 … an so on

Tutorial 23 - Local Per-Action Repository in QTP

2010-07-16T11:09:00.001-07:00

This tutorial introduces the different types of Object Repositories and discusses Local Object Repository in detail

points to note:

* Local Object Repository is the default object repository
* It is specific to actions and can be used only for a particular action
* Local Object Repository is preferable when application is not dynamic with respect to time
* Local Object Repository cannot be reused
* You can perform many a operations in the local object repository such as -

* Highlight an object stored in repository on the application under test
* Check whether a particular object in your AUT is stored in the Object Repository
* Cut , Copy , Paste , Modify and Delete Objects
* In case you have accidentally modified the value of a property you can update its description from the application using update function.

Tutorial 24 - How to create shared repository in QTP

2010-07-16T11:09:00.000-07:00

This tutorial demonstartes Shared Object Repository in detail

Summary:

* Global or Shared Object Repository is preferable when application is dynamic and object description change frequently
* Between Shared and local object repository , shared object repository is more commonly used in automation projects
* But it has maintenance and administration overheads as compared to local object repository.

To create and use a shared object repository you need to perform three broad steps
* 1) Creating a Shared Object Repository
* 2) Associating a Shared Object Repository
* 3) Editing a Shared Object Repository

Lets look at them one at a time

1) Creating a Shared Object Repository
* All repositories are local by default . To create a Shared Object Repository, in the Object Repository Dialog Box , Click File > Export Local Objects
* Repository files have an extension .tsr .Give a suitable name say "guru99" and save
* The Shared Repository File is now created

2)Associating a Shared Object Repository
* Next step is to associate the repository to your test, which enables you to use it
* To associate repository with a test, Click Resources > Associate Repository
* You can select the Repository to associate with Actions available in your test.
* Now you can now use this shared repository to develop your test

3) Editing a Shared Object Repository
* You can use the Object Repository Manager to Edit a Share Repository.
* Select Resources > Object Repository Manager .Open the Object Repository we created "guru99"
* By Default , Repository is opened in Read-only mode. To enable editing click File > Enable Editing
* Once editing is enabled you can all the operations like cut , copy , pate , rename objects etc that you can also do in Object Repository
* Using Object Repository Manager is you can compare two Object Repositories .QTP will give you a static's of what's unique and common in both the repositories
* You can use the Object repository merge tool to merge two repositories into oneT

Tutorial 25 - Guide to develop a script in Export View of QTP

2010-07-16T11:08:00.003-07:00

This tutorial demonstartes how you can develop a script directly in EXPERT view in QTP

Video Transcript with Key Takeaways highlighted

* Suppose my objective is to code the following statement directly in keyword view
* Dialog(Login).WinEdit(Agent Name:).Set Guru99
* In QTP screen when I press Cntrl + spacebar ,a list containing of all possible properties , methods is shown
* Also the list shows the objects stored in the object repository
* Select Dialog
* As soon as I open the parentheses , object name login is auto populated , if there are more than one * object for the same object type a list is displayed
* On pressing the . key a list of all the methods for the Dialog object and its child objects are displayed. Select WinEdit
* On inputting the dot operator a list of methods for Winedit box is displayed select SE

Tutorial 26 - Type of Recording Modes

2010-07-16T11:08:00.002-07:00

This tutorial demonstrates the 3 types of Recording Modes in QTP

points to remember

QTP supports 3 types of recording modes

* 1) Context Sensitive
* 2)Analog
* 3) Low Level

Context Sensitive Recording mode
* Normal recording mode is also called Context Sensitive Mode
* It is the default mode of recording which takes full advantage of Quick Test Professional's test object model.
* It recognizes objects in application regardless of their location on the screen.
* It records the objects in your application and the operations performed on them

Analog Recording Mode

* In analog recording mode, Quick Test Professional records and tracks every movement of the mouse as you drag the mouse around a screen or window.
* QTP’s Analog recording is captured as Tracks and stored in the directory of your test
* It is useful for recording operations that cannot be recorded at the level of an object. Eg., A signature produced by dragging the mouse
* In Analog mode you can record 1) Record Relative to screen & 2) Relative to window
* When your analog operation are confined to just one window , use relative to window
* When your analog operation involve multiple screens like dragging and dropping an object from one window to other use the screen option

Low Level Mode
* This mode enables you to record on any object in your application, irrespective of QTP recognizes the specific object or the specific operation.
* This mode records at the object level and records all run-time objects as either Window or WinObject test objects..
* It is used when the exact coordinates of the object are important for your tests. A good example would be hashmaps where clicking different sections of a picture takes you to different links
* Used when recording tests in an environment (or on an object) not recognized by QTP
* Low level mode records the x,y coordinates of any clicks

Summary
* Analog recording and low-level recording require more disk space than normal recording mode.
* For both modes , screen positions during the record and run time needs to be identical otherwise script fails
* Hence, Use analog recording or low-level recording only when normal recording mode does not accurately record your operation.
* At times QTP automatically switches to low level mode while recording objects or environments not supported by QTP

Tutorial 27 - All about user defined functions in QTP

2010-07-16T11:08:00.001-07:00

This tutorial describes how you could use Functions in QTP

Video Transcript with Key Takeaways highlighted

* If you have segments of code that you need to use several times in your tests, you may want to create a user-defined function. By using user-defined functions, your tests are shorter, and easier to design, read, and maintain
* Your own function libraries, can contain VBScript functions, subroutines, modules etc.
* You need to follow 3 simple steps to use a function from a library in your test
* To create a new function library in QTP
* Select File > New > Function Library. It opens as a new tab in QTP
* Lets a create a very simple function which shows Message Box.So whenever this function is called a message box must be displayed
* You can have mutliple functions defined in the same file.Let's save the function
* A function has an extension .qfl
* Next Step Associate the library with your test
* Click File > Settings > Resources > Associate Function Library.Click Add. Select The Function Library File.Click Okay
* Last step to use the function
* In your test simply call the function. When you run the test as expected Message Box is displayed
* Using COM , DCOM objects you can create very advanced Functions
* Infact, many of the features provided by QTP can be coded using VBScript
* We have seen automation engineers who make it more of a VB project rather than automation project
* Our recommendation is to focus on 100% automation rather than flaunting your VB skills

Tutorial 28 - All about Transanctions in QTP

2010-07-16T11:08:00.000-07:00

This tutorial describes how you could use TRANSACTIONS in QTP

Video Transcript with Key Takeaways highlighted

* You can measure how long it takes to run a section of your test by defining transactions.
* You define transactions within your test by enclosing the appropriate sections of the test with start and end transaction statements.
* For example, you may want to note the time taken to book a flight
* In QTP , select the appropriate statement where you want to start your transaction .
* Select Insert Start Transaction.
* Start Transaction Dialog Box opens
* Give the transaction a suitable name, say "Booking Time"
* A start transaction statement is added in the test
* Select the statement where you want to end the transaction
* Click Insert > End Transaction
* End Transaction Dialog Box Opens with the list of all available transactions
* Click Okay. An end transaction statement is added
* Lets runt the test
* * In results , the end transaction statement gives the time taken to insert the order
* Transactions can be inserted anywhere in the script
* There is no limit to the number of transactions that can be added to a test.
* You can also insert a transaction within a transaction

Tutorial 29 - All about Recovery Scenario in QTP

2010-07-16T11:07:00.000-07:00

This tutorial demonstrates Recovery Scenarios in QTP

Video Transcript with Key Takeaways highlighted

Using Recovery Scenarios you can instruct QTP to recover from unexpected events and errors that occur in your testing environment during a run session.

Recovery scenario become crucial for large tests, which run unattended and are paused until recovery operation, is performed increasing the test execution time.

There are 6 steps involved in creating a recovery scenario

* 1) In QTP , Select Resources > Recovery Scenario Manager .Create new Scenario
* 2) Specify the Trigger Event. A Trigger Event is an event that interrupts your run session
* 3) Specigy the Recovery Operation which is the corrective action you take when trigger happens
* 4) Specify Post-recovery test run options which specify how to continue the run session after Quick Test Professional has identified the event and performed all of the specified recovery operations.
* 5) Check and verify Summary of the scenario you created.
* 6) Add Recovery Scenario to your test using File > Settings > Recovery.

The test results window show the details of the recovery scenario

Use can also use statements
On Error Resume Next :
On Error Go to 0 :

to handle errors in your script

Tutorial 30 - What is an Optional Step in QTP?

2010-07-16T11:06:00.003-07:00

This tutorial describes an OPTIONAL STEP and how it can be used.

Video Transcript with Key Takeaways highlighted

* An optional step is a step that is not necessarily required to successfully complete a run session.
* During a run session, if the object of an optional step does not exist in the application * QTP bypasses this step and continues to run the test
* To set a step as optional in keyword view right click on the step and select Optional Step
* Alternatively, you can directly write the keyword "OptionalStep" preceding a statement to make it optional

Tutorial 31 - Guide to Object Sopy | GetROProperty, GetTOProperty & SetTOPropert

2010-07-16T11:06:00.002-07:00

This tutorial demonstrates OBJECT SPY.Object Spy can help determine useful properties and methods associated with an object in your environment .The tutorials also describes GetROProperty, GetTOProperty & SetTOProperty

points to remember

GetRoProperty

* GetRoProperty – is an in-built method used to retrieve runtime value of an object property.
* There are 4 steps involved in using the GetRoProperty
* Step 1) Record the Object on which you want to use the GetRoProperty in Object Repository
* Step 2) For the recorded object identify the run-time property which could be used. You may use Object Spy.
* Step 3) Use GetRoProperty method to retrieve the identified Run-time property and store the value in a variable
* Step 4) Use this value for further deductions

SetToProperty & GetToProperty
* Consider a Web Button stored in the Object Repository
* When the test is run QTP creates a copy of this object called Test Object and compares it with the Run Time Object
* Using GetToProperty you can retrieve the value of a property of Test Object
* Using SetToProperty you can change the property value of a Test Object
* When the test is completed, this test object is discarded and so are any modifications you made in the object properties using the SetToProperty
* When the test is re-run, a new copy of the test object is created with original property values stored in the object repository
* You can consider using GetToProperty and SetToProperty when your test script has multiple lines of codes and your environment is sporadic
* For a note ,there is no SetRoProperty

Object Spy:
* Object spy is feature in QTP using which you can view both the test and run-time object properties and methods.
* It also gives the syntax for a selected method.
* Object Spy gives the complete hierarchy of the object you have selected

Tutorial 32 - Descriptive Programming

2010-07-16T11:06:00.001-07:00

This is the 1st part of a two part tutorial for Descriptive Programming. It introduces Descriptive Programming and its two type viz. static and dynamic.

Video Transcript with Key Takeaways Highlighted:

* To understand the concept of descriptive programming lets understand the significance of an objects name in QTP
* An object name is simply used to map an object in script with its description in object repository.
* Meaning if I change the object name here to ABC and correspondingly change the object name in script to ABC the script will still work fine.
* Consider this statement, which sets the Agent Name as Guru99. Let us do an experiment..Let's delete the Object Description of Agent Name Win Edit Box from the Object Repository.
* If I run the test again it will fail since it can not recognize the object. Lets examine the reason why the script is failing
* During Run Time, QTP identifies the operation that is performed on WinEdit box and the Object Description in Object Repository is stored as Agent Name
* It uses this name to track the object in object repository. For a parent you cannot have two child objects with the same name. Hence, QTP uniquely maps the object in the repository
* It then uses the stored description in the Object Repository and replaces the name with the description. It then uses this statement to identify the object in the application under test
* Since in our case we had deleted this object description all together the script fails
* But what if instead of QTP replacing the object description, you as a tester directly specify the object descriptions in your script. This is nothing but "Descriptive Programming"
* You can use Descriptive programming in two ways 1) Static 2) Dynamic
* Lets look at Static First. In Static Method , for object identification in the following format property:=values , you mention an objects property ,and its corresponding values separated by a colon and equal to sign. This format is called property value pair and is enclosed in inverted commas
* If your object uses multiple descriptions for identification, you can specify those using commas
* So in our case the description for Agent Name becomes "nativeclass:=Edit", "attached text:=Agent Name:"
* Next Step is to Replace Object Name with above description. In QTP , lets do the replacement and run the test. It runs fine . In the test results the Agent Name object is enclosed in parentheses indicating that Descriptive Programming was used in its identification. * This is the static method
* The second method of doing the same action is using Dynamic Descriptive programming
* In case your script uses the descriptive programming object candidate multiple times, * it will be very tiresome to specify all the property value pairs for each statement
* In such cases you can make use of Description Class provided by QTP
* Syntax for creating an description object is
Set MyDescription = Description.Create();
MyDescription("property").Value = "property-value";
This is the Dynamic Method

Tutorial 33 - Descriptive Programming Practical Application

2010-07-16T11:06:00.000-07:00

This is the 2nd part of a two part tutorial for Descriptive Programming. It describes practical application of Descriptive Programming .It makes use of ChildObjects() a method used to find all Child Objects for a Parent object.

Video Transcript with Key Takeaways highlighted

* The million dollar question is why use DP when the Object Identification process is handled by QTP
* Suppose you are assigned to test a job portal. You enter a search query into the portal and
your test expects you to select all available jobs .and click the apply job
But the no of jobs reflected will depend on the search query and jobs available at time of script execution but there is no way to predict in advance the no of jobs that would be reflected
* In such cases, you can use descriptive programming. Even though you do not know the number and names of the checkboxes you do know the class for the objects as "WebCheckBox"
* You can use the ChildObject method to return objects belonging to a particular parent
* A line of code like - Set allObjects = Browser("Jobs").Page("QTP").ChildObjects()
Will return all child objects for this page.
* But we want only WebCheckBox objects.To do so we can create a filter creation object and set its property as WebcheckBox and pass this filter as an argument for the ChildObjects method
* In this case only the checkboxes are returned.
* Next you can write a code like this which access the entire collection of checkboxes starting from zero and sets all checkboxes ON.
* Next you can click apply button to complete the test
* You can also use Descriptive Programming to run objects which are difficult to record like Auto-Hide Panels, Objects with changing hierarchies, Nested Inner Objects ,Sub-menus.
* You can also do advanced string manipulations using descriptive programming
* In conjunction with index property, descriptive programming could be very useful in identifying difficult objects.
* If you use programmatic description for an object in object hierarchy you will need to use description programming for succeeding child objects
* For example, for page object descriptive programming was used but for succeeding child object WinEdit Object Repository is used which is incorrect
* On the contrary here for both Page and WinEdit descriptive programming is used which is correct

Khols Shopping

2009-07-26T10:08:00.000-07:00

QTP 9.2

2008-03-04T20:39:00.000-08:00

QTP 9.2

Google Search

2008-03-04T19:44:00.000-08:00

More On QTP

2008-02-07T04:08:00.000-08:00

Automation Framework

2008-01-28T03:11:00.000-08:00

Automation Framework

Starting QTP

2008-01-28T03:00:00.000-08:00

Qtp Basics

ISTQB Syllabus

2007-02-15T21:00:00.000-08:00

Certified Tester Foundation Level Syllabus

1. Fundamentals of testing (K2) 155 minutes

Learning objectives for fundamentals of testing
The objectives identify what you will be able to do following the completion of each module.
1.1 Why is testing necessary? (K2)
 Describe, with examples, the way in which a defect in software can cause harm to a person, to
the environment or to a company. (K2)
 Distinguish between the root cause of a defect and its effects. (K2)
 Give reasons why testing is necessary by giving examples. (K2)
 Describe why testing is part of quality assurance and give examples of how testing contributes
to higher quality. (K2)
 Recall the terms mistake, defect, failure and corresponding terms error and bug. (K1)
1.2 What is testing? (K2)
 Recall the common objectives of testing. (K1)
 Describe the purpose of testing in software development, maintenance and operations as a
means to find defects, provide confidence and information, and prevent defects. (K2)
1.3 General testing principles (K2)
 Explain the fundamental principles in testing. (K2)
1.4 Fundamental test process (K1)
 Recall the fundamental test activities from planning to test closure activities and the main
tasks of each test activity. (K1)
1.5 The psychology of testing (K2)
 Recall that the success of testing is influenced by psychological factors (K1):
o clear objectives;
o a balance of self-testing and independent testing;
o recognition of courteous communication and feedback on defects.
 Contrast the mindset of a tester and of a developer. (K2)

1.1 Why is testing necessary (K2) 20 minutes
Terms
Bug, defect, error, failure, fault, mistake, quality, risk, software, testing.
1.1.1 Software systems context (K1)
Software systems are an increasing part of life, from business applications (e.g. banking) to consumer
products (e.g. cars). Most people have had an experience with software that did not work as expected.
Software that does not work correctly can lead to many problems, including loss of money, time or
business reputation, and could even cause injury or death.

1.1.2 Causes of software defects (K2)
A human being can make an error (mistake), which produces a defect (fault, bug) in the code, in
software or a system, or in a document. If a defect in code is executed, the system will fail to do what it
should do (or do something it shouldn’t), causing a failure. Defects in software, systems or documents
may result in failures, but not all defects do so.
Defects occur because human beings are fallible and because there is time pressure, complex code,
complexity of infrastructure, changed technologies, and/or many system interactions.
Failures can be caused by environmental conditions as well: radiation, magnetism, electronic fields,
and pollution can cause faults in firmware or influence the execution of software by changing hardware
conditions.
1.1.3 Role of testing in software development, maintenance and operations
(K2)
Rigorous testing of systems and documentation can help to reduce the risk of problems occurring in
an operational environment and contribute to the quality of the software system, if defects found are
corrected before the system is released for operational use.
Software testing may also be required to meet contractual or legal requirements, or industry-specific
standards.
1.1.4 Testing and quality (K2)
With the help of testing, it is possible to measure the quality of software in terms of defects found, for
both functional and non-functional software requirements and characteristics (e.g. reliability, usability,
efficiency and maintainability). For more information on non-functional testing see Chapter 2; for more
information on software characteristics see ‘Software Engineering – Software Product Quality’ (ISO
9126).
Testing can give confidence in the quality of the software if it finds few or no defects. A properly
designed test that passes reduces the overall level of risk in a system. When testing does find defects,
the quality of the software system increases when those defects are fixed.

Lessons should be learned from previous projects. By understanding the root causes of defects found
in other projects, processes can be improved, which in turn should prevent those defects reoccurring
and, as a consequence, improve the quality of future systems.
Testing should be integrated as one of the quality assurance activities (e.g. alongside development
standards, training and defect analysis).
1.1.5 How much testing is enough? (K2)
Deciding how much testing is enough should take account of the level of risk, including technical and
business product and project risks, and project constraints such as time and budget. (Risk is
discussed further in Chapter 5.)
Testing should provide sufficient information to stakeholders to make informed decisions about the
release of the software or system being tested, for the next development step or handover to
customers.

1.2 What is testing (K2) 30 minutes
Terms
Code, debugging, development (of software), requirement, review, test basis, test case, testing, test
objectives.
Background
A common perception of testing is that it only consists of running tests, i.e. executing the software.
This is part of testing, but not all of the testing activities.
Test activities exist before and after test execution, activities such as planning and control, choosing
test conditions, designing test cases and checking results, evaluating completion criteria, reporting on
the testing process and system under test, and finalizing or closure (e.g. after a test phase has been
completed). Testing also includes reviewing of documents (including source code) and static analysis.
Both dynamic testing and static testing can be used as a means for achieving similar objectives, and
will provide information in order to improve both the system to be tested, and the development and
testing processes.
There can be different test objectives:
 finding defects;
 gaining confidence about the level of quality and providing information;
 preventing defects.
The thought process of designing tests early in the life cycle (verifying the test basis via test design)
can help to prevent defects from being introduced into code. Reviews of documents (e.g.
requirements) also help to prevent defects appearing in the code.
Different viewpoints in testing take different objectives into account. For example, in development
testing (e.g. component, integration and system testing), the main objective may be to cause as many
failures as possible so that defects in the software are identified and can be fixed. In acceptance
testing, the main objective may be to confirm that the system works as expected, to gain confidence
that it has met the requirements. In some cases the main objective of testing may be to assess the
quality of the software (with no intention of fixing defects), to give information to stakeholders of the
risk of releasing the system at a given time. Maintenance testing often includes testing that no new
errors have been introduced during development of the changes. During operational testing, the main
objective may be to assess system characteristics such as reliability or availability.
Debugging and testing are different. Testing can show failures that are caused by defects. Debugging
is the development activity that identifies the cause of a defect, repairs the code and checks that the
defect has been fixed correctly. Subsequent confirmation testing by a tester ensures that the fix does
indeed resolve the failure. The responsibility for each activity is very different, i.e. testers test and
developers debug.
The process of testing and its activities is explained in Section 1.4.

1.3 General testing principles (K2) 35 minutes
Terms
Exhaustive testing.
Principles
A number of testing principles have been suggested over the past 40 years and offer general
guidelines common for all testing.
Principle 1 – Testing shows presence of defects
Testing can show that defects are present, but cannot prove that there are no defects. Testing
reduces the probability of undiscovered defects remaining in the software but, even if no defects are
found, it is not a proof of correctness.
Principle 2 – Exhaustive testing is impossible
Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial
cases. Instead of exhaustive testing, we use risk and priorities to focus testing efforts.
Principle 3 – Early testing
Testing activities should start as early as possible in the software or system development life cycle,
and should be focused on defined objectives.
Principle 4 – Defect clustering
A small number of modules contain most of the defects discovered during pre-release testing, or show
the most operational failures.
Principle 5 – Pesticide paradox
If the same tests are repeated over and over again, eventually the same set of test cases will no
longer find any new bugs. To overcome this “pesticide paradox”, the test cases need to be regularly
reviewed and revised, and new and different tests need to be written to exercise different parts of the
software or system to potentially find more defects.
Principle 6 – Testing is context dependent
Testing is done differently in different contexts. For example, safety-critical software is tested
differently from an e-commerce site.
Principle 7 – Absence-of-errors fallacy
Finding and fixing defects does not help if the system built is unusable and does not fulfill the users’
needs and expectations.

1.4 Fundamental test process (K1) 35 minutes
Terms
Confirmation testing, exit criteria, incident, regression testing, test basis, test condition, test coverage,
test data, test execution, test log, test plan, test strategy, test summary report, testware.
Background
The most visible part of testing is executing tests. But to be effective and efficient, test plans should
also include time to be spent on planning the tests, designing test cases, preparing for execution and
evaluating status.
The fundamental test process consists of the following main activities:
 planning and control;
 analysis and design;
 implementation and execution;
 evaluating exit criteria and reporting;
 test closure activities.
Although logically sequential, the activities in the process may overlap or take place concurrently.
1.4.1 Test planning and control (K1)
Test planning is the activity of verifying the mission of testing, defining the objectives of testing and the
specification of test activities in order to meet the objectives and mission.
Test control is the ongoing activity of comparing actual progress against the plan, and reporting the
status, including deviations from the plan. It involves taking actions necessary to meet the mission and
objectives of the project. In order to control testing, it should be monitored throughout the project. Test
planning takes into account the feedback from monitoring and control activities.
Test planning has the following major tasks:
 Determining the scope and risks, and identifying the objectives of testing.
 Determining the test approach (techniques, test items, coverage, identifying and interfacing
the teams involved in testing, testware).
 Determining the required test resources (e.g. people, test environment, PCs).
 Implementing the test policy and/or the test strategy.
 Scheduling test analysis and design tasks.
 Scheduling test implementation, execution and evaluation.
 Determining the exit criteria.
Test control has the following major tasks:
 measuring and analyzing results;
 monitoring and documenting progress, test coverage and exit criteria;
 initiation of corrective actions;
 making decisions.
1.4.2 Test analysis and design (K1)
Test analysis and design is the activity where general testing objectives are transformed into tangible
test conditions and test designs.
Test analysis and design has the following major tasks:

 Reviewing the test basis (such as requirements, architecture, design, interfaces).
 Identifying test conditions or test requirements and required test data based on analysis of test
items, the specification, behavior and structure.
 Designing the tests.
 Evaluating testability of the requirements and system.
 Designing the test environment set-up and identifying any required infrastructure and tools.
1.4.3 Test implementation and execution (K1)
Test implementation and execution is the activity where test conditions are transformed into test cases
and testware, and the environment is set up.
Test implementation and execution has the following major tasks:
 Developing and prioritizing test cases, creating test data, writing test procedures and,
optionally, preparing test harnesses and writing automated test scripts.
 Creating test suites from the test cases for efficient test execution.
 Verifying that the test environment has been set up correctly.
 Executing test cases either manually or by using test execution tools, according to the planned
sequence.
 Logging the outcome of test execution and recording the identities and versions of the
software under test, test tools and testware.
 Comparing actual results with expected results.
 Reporting discrepancies as incidents and analyzing them in order to establish their cause (e.g.
a defect in the code, in specified test data, in the test document, or a mistake in the way the
test was executed).
 Repeating test activities as result of action taken for each discrepancy. For example, reexecution
of a test that previously failed in order to confirm a fix (confirmation testing),
execution of a corrected test and/or execution of tests in order to ensure that defects have not
been introduced in unchanged areas of the software or that defect fixing did not uncover other
defects (regression testing).
1.4.4 Evaluating exit criteria and reporting (K1)
Evaluating exit criteria is the activity where test execution is assessed against the defined objectives.
This should be done for each test level.
Evaluating exit criteria has the following major tasks:
 Checking test logs against the exit criteria specified in test planning.
 Assessing if more tests are needed or if the exit criteria specified should be changed.
 Writing a test summary report for stakeholders.
1.4.5 Test closure activities (K1)
Test closure activities collect data from completed test activities to consolidate experience, testware,
facts and numbers. For example, when a software system is released, a test project is completed (or
cancelled), a milestone has been achieved, or a maintenance release has been completed.
Test closure activities include the following major tasks:
 Checking which planned deliverables have been delivered, the closure of incident reports or
raising of change records for any that remain open, and the documentation of the acceptance
of the system.
 Finalizing and archiving testware, the test environment and the test infrastructure for later
reuse.
 Handover of testware to the maintenance organization.
 Analyzing lessons learned for future releases and projects, and the improvement of test
maturity.

1.5 The psychology of testing (K2) 35 minutes
Terms
Independent testing.
Background
The mindset to be used while testing and reviewing is different to that used while analyzing or
developing. With the right mindset developers are able to test their own code, but separation of this
responsibility to a tester is typically done to help focus effort and provide additional benefits, such as
an independent view by trained and professional testing resources. Independent testing may be
carried out at any level of testing.
A certain degree of independence (avoiding the author bias) is often more effective at finding defects
and failures. Independence is not, however, a replacement for familiarity, and developers can
efficiently find many defects in their own code. Several levels of independence can be defined:
 Tests designed by the person(s) who wrote the software under test (low level of
independence).
 Tests designed by another person(s) (e.g. from the development team).
 Tests designed by a person(s) from a different organizational group (e.g. an independent test
team).
 Tests designed by a person(s) from a different organization or company (i.e. outsourcing or
certification by an external body).
People and projects are driven by objectives. People tend to align their plans with the objectives set by
management and other stakeholders, for example, to find defects or to confirm that software works.
Therefore, it is important to clearly state the objectives of testing.
Identifying failures during testing may be perceived as criticism against the product and against the
author. Testing is, therefore, often seen as a destructive activity, even though it is very constructive in
the management of product risks. Looking for failures in a system requires curiosity, professional
pessimism, a critical eye, attention to detail, good communication with development peers, and
experience on which to base error guessing.
If errors, defects or failures are communicated in a constructive way, bad feelings between the testers
and the analysts, designers and developers can be avoided. This applies to reviewing as well as in
testing.
The tester and test leader need good interpersonal skills to communicate factual information about
defects, progress and risks, in a constructive way. For the author of the software or document, defect
information can help them improve their skills. Defects found and fixed during testing will save time
and money later, and reduce risks.
Communication problems may occur, particularly if testers are seen only as messengers of unwanted
news about defects. However, there are several ways to improve communication and relationships
between testers and others:

 Start with collaboration rather than battles – remind everyone of the common goal of better
quality systems.
 Communicate findings on the product in a neutral, fact-focused way without criticizing the
person who created it, for example, write objective and factual incident reports and review
findings.
 Try to understand how the other person feels and why they react as they do.
 Confirm that the other person has understood what you have said and vice versa.
References
1.1.5 Black, 2001, Kaner, 2002
1.2 Beizer, 1990, Black, 2001, Myers, 1979
1.3 Beizer, 1990, Hetzel, 1998, Myers, 1979
1.4 Hetzel, 1998
1.4.5 Black, 2001, Craig, 2002
1.5 Black, 2001, Hetzel, 1998

2. Testing throughout the software life
cycle (K2)
135 minutes
Learning objectives for testing throughout the software life cycle
The objectives identify what you will be able to do following the completion of each module.
2.1 Software development models (K2)
 Understand the relationship between development, test activities and work products in the
development life cycle, and give examples based on project and product characteristics and
context (K2).
 Recognize the fact that software development models must be adapted to the context of
project and product characteristics. (K1)
 Recall reasons for different levels of testing, and characteristics of good testing in any life
cycle model. (K1)
2.2 Test levels (K2)
 Compare the different levels of testing: major objectives, typical objects of testing, typical
targets of testing (e.g. functional or structural) and related work products, people who test,
types of defects and failures to be identified. (K2)
2.3 Test types: the targets of testing (K2)
 Compare four software test types (functional, non-functional, structural and change-related) by
example. (K2)
 Recognize that functional and structural tests occur at any test level. (K1)
 Identify and describe non-functional test types based on non-functional requirements. (K2)
 Identify and describe test types based on the analysis of a software system’s structure or
architecture. (K2)
 Describe the purpose of confirmation testing and regression testing. (K2)
2.4 Maintenance testing (K2)
 Compare maintenance testing (testing an existing system) to testing a new application with
respect to test types, triggers for testing and amount of testing. (K2)
 Identify reasons for maintenance testing (modification, migration and retirement). (K1)
 Describe the role of regression testing and impact analysis in maintenance. (K2)

2.1 Software development models (K2)
Terms
Commercial off the shelf (COTS), incremental development model, test level, validation, verification,
V-model.
Background
Testing does not exist in isolation; test activities are related to software development activities.
Different development life cycle models need different approaches to testing.
2.1.1 V-model (K2)
Although variants of the V-model exist, a common type of V-model uses four test levels,
corresponding to four development levels.
The four levels used in this syllabus are:
 component (unit) testing;
 integration testing;
 system testing;
 acceptance testing.
In practice, a V-model may have more, fewer or different levels of development and testing, depending
on the project and the software product. For example, there may be component integration testing
after component testing, and system integration testing after system testing.
Software work products (such as business scenarios or use cases, requirement specifications, design
documents and code) produced during development are often the basis of testing in one or more test
levels. References for generic work products include Capability Maturity Model Integration (CMMI) or
‘Software life cycle processes’ (IEEE/IEC 12207). Verification and validation (and early test design)
can be carried out during the development of the software work products.
2.1.2 Iterative development models (K2)
Iterative development is the process of establishing requirements, designing, building and testing a
system, done as a series of smaller developments. Examples are: prototyping, rapid application
development (RAD), Rational Unified Process (RUP) and agile development models. The increment
produced by an iteration may be tested at several levels as part of its development. An increment,
added to others developed previously, forms a growing partial system, which should also be tested.
Regression testing is increasingly important on all iterations after the first one. Verification and
validation can be carried out on each increment.
2.1.3 Testing within a life cycle model (K2)
In any life cycle model, there are several characteristics of good testing:
 For every development activity there is a corresponding testing activity.
 Each test level has test objectives specific to that level.
 The analysis and design of tests for a given test level should begin during the corresponding
development activity.
 Testers should be involved in reviewing documents as soon as drafts are available in the
development life cycle.
Test levels can be combined or reorganized depending on the nature of the project or the system
architecture. For example, for the integration of a commercial off the shelf (COTS) software product
into a system, the purchaser may perform integration testing at the system level (e.g. integration to the

infrastructure and other systems, or system deployment) and acceptance testing (functional and/or
non-functional, and user and/or operational testing).

2.2 Test levels (K2) 60 minutes
Terms
Alpha testing, beta testing, component testing (also known as unit, module or program testing),
contract acceptance testing, drivers, field testing, functional requirements, integration, integration
testing, non-functional requirements, operational (acceptance) testing, regulation acceptance testing,
robustness testing, stubs, system testing, test-driven development, test environment, user acceptance
testing.
Background
For each of the test levels, the following can be identified: their generic objectives, the work product(s)
being referenced for deriving test cases (i.e. the test basis), the test object (i.e. what is being tested),
typical defects and failures to be found, test harness requirements and tool support, and specific
approaches and responsibilities.
2.2.1 Component testing (K2)
Component testing searches for defects in, and verifies the functioning of, software (e.g. modules,
programs, objects, classes, etc.) that are separately testable. It may be done in isolation from the rest
of the system, depending on the context of the development life cycle and the system. Stubs, drivers
and simulators may be used.
Component testing may include testing of functionality and specific non-functional characteristics,
such as resource-behavior (e.g. memory leaks) or robustness testing, as well as structural testing (e.g.
branch coverage). Test cases are derived from work products such as a specification of the
component, the software design or the data model.
Typically, component testing occurs with access to the code being tested and with the support of the
development environment, such as a unit test framework or debugging tool, and, in practice, usually
involves the programmer who wrote the code. Defects are typically fixed as soon as they are found,
without formally recording incidents.
One approach in component testing is to prepare and automate test cases before coding. This is
called a test-first approach or test-driven development. This approach is highly iterative and is based
on cycles of developing test cases, then building and integrating small pieces of code, and executing
the component tests until they pass.
2.2.2 Integration testing (K2)
Integration testing tests interfaces between components, interactions to different parts of a system,
such as the operating system, file system, hardware or interfaces between systems.
There may be more than one level of integration testing and it may be carried out on test objects of
varying size. For example:
 Component integration testing tests the interactions between software components and is
done after component testing;
 System integration testing tests the interactions between different systems and may be done
after system testing. In this case, the developing organization may control only one side of the
interface, so changes may be destabilizing. Business processes implemented as workflows
may involve a series of systems. Cross-platform issues may be significant.
The greater the scope of integration, the more difficult it becomes to isolate failures to a specific
component or system, which may lead to increased risk.
Systematic integration strategies may be based on the system architecture (such as top-down and
bottom-up), functional tasks, transaction processing sequences, or some other aspect of the system or
© International Software Testing Qualifications Board
component. In order to reduce the risk of late defect discovery, integration should normally be
incremental rather than “big bang”.
Testing of specific non-functional characteristics (e.g. performance) may be included in integration
testing.
At each stage of integration, testers concentrate solely on the integration itself. For example, if they
are integrating module A with module B they are interested in testing the communication between the
modules, not the functionality of either module. Both functional and structural approaches may be
used.
Ideally, testers should understand the architecture and influence integration planning. If integration
tests are planned before components or systems are built, they can be built in the order required for
most efficient testing.
2.2.3 System testing (K2)
System testing is concerned with the behavior of a whole system/product as defined by the scope of a
development project or programme.
In system testing, the test environment should correspond to the final target or production environment
as much as possible in order to minimize the risk of environment-specific failures not being found in
testing.
System testing may include tests based on risks and/or on requirements specifications, business
processes, use cases, or other high level descriptions of system behavior, interactions with the
operating system, and system resources.
System testing should investigate both functional and non-functional requirements of the system.
Requirements may exist as text and/or models. Testers also need to deal with incomplete or
undocumented requirements. System testing of functional requirements starts by using the most
appropriate specification-based (black-box) techniques for the aspect of the system to be tested. For
example, a decision table may be created for combinations of effects described in business rules.
Structure-based techniques (white-box) may then be used to assess the thoroughness of the testing
with respect to a structural element, such as menu structure or web page navigation. (See Chapter 4.)
An independent test team often carries out system testing.
2.2.4 Acceptance testing (K2)
Acceptance testing is often the responsibility of the customers or users of a system; other
stakeholders may be involved as well.
The goal in acceptance testing is to establish confidence in the system, parts of the system or specific
non-functional characteristics of the system. Finding defects is not the main focus in acceptance
testing. Acceptance testing may assess the system’s readiness for deployment and use, although it is
not necessarily the final level of testing. For example, a large-scale system integration test may come
after the acceptance test for a system.
Acceptance testing may occur as more than just a single test level, for example:
 A COTS software product may be acceptance tested when it is installed or integrated.
 Acceptance testing of the usability of a component may be done during component testing.
 Acceptance testing of a new functional enhancement may come before system testing.
Typical forms of acceptance testing include the following:
User acceptance testing
Typically verifies the fitness for use of the system by business users.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 23 of 73 July 1st 2005
© International Software Testing Qualifications Board
Operational (acceptance) testing
The acceptance of the system by the system administrators, including:
 testing of backup/restore;
 disaster recovery;
 user management;
 maintenance tasks;
 periodic checks of security vulnerabilities.
Contract and regulation acceptance testing
Contract acceptance testing is performed against a contract’s acceptance criteria for producing
custom-developed software. Acceptance criteria should be defined when the contract is agreed.
Regulation acceptance testing is performed against any regulations which must be adhered to, such
as governmental, legal or safety regulations.
Alpha and beta (or field) testing
Developers of market, or COTS, software often want to get feedback from potential or existing
customers in their market before the software product is put up for sale commercially. Alpha testing is
performed at the developing organization’s site. Beta testing, or field testing, is performed by people at
their own locations. Both are performed by potential customers, not the developers of the product.
Organizations may use other terms as well, such as factory acceptance testing and site acceptance
testing for systems that are tested before and after being moved to a customer’s site.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 24 of 73 July 1st 2005
© International Software Testing Qualifications Board
2.3 Test types: the targets of testing (K2) 40 minutes
Terms
Automation, black-box testing, code coverage, confirmation testing, functional testing, interoperability
testing, load testing, maintainability testing, performance testing, portability testing, regression testing,
reliability testing, security testing, specification-based testing, stress testing, structural testing, test
suite, usability testing, white-box testing.
Background
A group of test activities can be aimed at verifying the software system (or a part of a system) based
on a specific reason or target for testing.
A test type is focused on a particular test objective, which could be the testing of a function to be
performed by the software; a non-functional quality characteristic, such as reliability or usability, the
structure or architecture of the software or system; or related to changes, i.e. confirming that defects
have been fixed (confirmation testing) and looking for unintended changes (regression testing).
A model of the software may be developed and/or used in structural and functional testing. For
example, in functional testing a process flow model, a state transition model or a plain language
specification; and for structural testing a control flow model or menu structure model.
2.3.1 Testing of function (functional testing) (K2)
The functions that a system, subsystem or component are to perform may be described in work
products such as a requirements specification, use cases, or a functional specification, or they may be
undocumented. The functions are “what” the system does.
Functional tests are based on these functions and features (described in documents or understood by
the testers), and may be performed at all test levels (e.g. tests for components may be based on a
component specification).
Specification-based techniques may be used to derive test conditions and test cases from the
functionality of the software or system. (See Chapter 4.) Functional testing considers the external
behaviour of the software (black-box testing).
A type of functional testing, security testing, investigates the functions (e.g. a firewall) relating to
detection of threats, such as viruses, from malicious outsiders.
2.3.2 Testing of software product characteristics (non-functional testing) (K2)
Non-functional testing includes, but is not limited to, performance testing, load testing, stress testing,
usability testing, interoperability testing, maintainability testing, reliability testing and portability testing.
It is the testing of “how” the system works.
Non-functional testing may be performed at all test levels. The term non-functional testing describes
the tests required to measure characteristics of systems and software that can be quantified on a
varying scale, such as response times for performance testing. These tests can be referenced to a
quality model such as the one defined in ‘Software Engineering – Software Product Quality’ (ISO
9126).
2.3.3 Testing of software structure/architecture (structural testing) (K2)
Structural (white-box) testing may be performed at all test levels. Structural techniques are best used
after specification-based techniques, in order to help measure the thoroughness of testing through
assessment of coverage of a type of structure.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 25 of 73 July 1st 2005
© International Software Testing Qualifications Board
Coverage is the extent that a structure has been exercised by a test suite, expressed as a percentage
of the items being covered. If coverage is not 100%, then more tests may be designed to test those
items that were missed and, therefore, increase coverage. Coverage techniques are covered in
Chapter 4.
At all test levels, but especially in component testing and component integration testing, tools can be
used to measure the code coverage of elements, such as statements or decisions. Structural testing
may be based on the architecture of the system, such as a calling hierarchy.
Structural testing approaches can also be applied at system, system integration or acceptance testing
levels (e.g. to business models or menu structures).
2.3.4 Testing related to changes (confirmation and regression testing) (K2)
When a defect is detected and fixed then the software should be retested to confirm that the original
defect has been successfully removed. This is called confirmation testing. Debugging (defect fixing) is
a development activity, not a testing activity.
Regression testing is the repeated testing of an already tested program, after modification, to discover
any defects introduced or uncovered as a result of the change(s). These defects may be either in the
software being tested, or in another related or unrelated software component. It is performed when the
software, or its environment, is changed. The extent of regression testing is based on the risk of not
finding defects in software that was working previously.
Tests should be repeatable if they are to be used for confirmation testing and to assist regression
testing.
Regression testing may be performed at all test levels, and applies to functional, non-functional and
structural testing. Regression test suites are run many times and generally evolve slowly, so
regression testing is a strong candidate for automation.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 26 of 73 July 1st 2005
© International Software Testing Qualifications Board
2.4 Maintenance testing (K2) 15 minutes
Terms
Impact analysis, maintenance testing, migration, modifications, retirement.
Background
Once deployed, a software system is often in service for years or decades. During this time the system
and its environment is often corrected, changed or extended. Maintenance testing is done on an
existing operational system, and is triggered by modifications, migration, or retirement of the software
or system.
Modifications include planned enhancement changes (e.g. release-based), corrective and emergency
changes, and changes of environment, such as planned operating system or database upgrades, or
patches to newly exposed or discovered vulnerabilities of the operating system.
Maintenance testing for migration (e.g. from one platform to another) should include operational tests
of the new environment, as well as of the changed software.
Maintenance testing for the retirement of a system may include the testing of data migration or
archiving if long data-retention periods are required.
In addition to testing what has been changed, maintenance testing includes extensive regression
testing to parts of the system that have not been changed. The scope of maintenance testing is
related to the risk of the change, the size of the existing system and to the size of the change.
Depending on the changes, maintenance testing may be done at any or all test levels and for any or
all test types.
Determining how the existing system may be affected by changes is called impact analysis, and is
used to help decide how much regression testing to do.
Maintenance testing can be difficult if specifications are out of date or missing.
References
2.1.3 CMMI, Craig, 2002, Hetzel, 1998, IEEE 12207
2.2 Hetzel, 1998
2.2.4 Copeland, 2004, Myers, 1979
2.3.1 Beizer, 1990, Black, 2001, Copeland, 2004
2.3.2 Black, 2001, ISO 9126
2.3.3 Beizer, 1990, Copeland, 2004, Hetzel, 1998
2.3.4 Hetzel, 1998, IEEE 829
2.4 Black, 2001, Craig, 2002, Hetzel, 1998, IEEE 829
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 27 of 73 July 1st 2005
© International Software Testing Qualifications Board
3. Static techniques (K2) 60 minutes
Learning objectives for static techniques
The objectives identify what you will be able to do following the completion of each module.
3.1 Reviews and the test process (K2)
 Recognize software work products that can be examined by the different static techniques.
(K1)
 Describe the importance and value of considering static techniques for the assessment of
software work products. (K2)
 Explain the difference between static and dynamic techniques. (K2)
3.2 Review process (K2)
 Recall the phases, roles and responsibilities of a typical formal review. (K1)
 Explain the differences between different types of review: informal review, technical review,
walkthrough and inspection. (K2)
 Explain the factors for successful performance of reviews. (K2)
3.3 Static analysis by tools (K2)
 Describe the objective of static analysis and compare it to dynamic testing. (K2)
 Recall typical defects and errors identified by static analysis and compare them to reviews and
dynamic testing. (K1)
 List typical benefits of static analysis. (K1)
 List typical code and design defects that may be identified by static analysis tools. (K1)
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 28 of 73 July 1st 2005
© International Software Testing Qualifications Board
3.1 Reviews and the test process (K2) 15 minutes
Terms
Dynamic testing, reviews, static analysis.
Background
Static testing techniques do not execute the software that is being tested; they are manual (reviews) or
automated (static analysis).
Reviews are a way of testing software work products (including code) and can be performed well
before dynamic test execution. Defects detected during reviews early in the life cycle are often much
cheaper to remove than those detected while running tests (e.g. defects found in requirements).
A review could be done entirely as a manual activity, but there is also tool support. The main manual
activity is to examine a work product and make comments about it. Any software work product can be
reviewed, including requirement specifications, design specifications, code, test plans, test
specifications, test cases, test scripts, user guides or web pages.
Benefits of reviews include early defect detection and correction, development productivity
improvements, reduced development timescales, reduced testing cost and time, lifetime cost
reductions, fewer defects and improved communication. Reviews can find omissions, for example, in
requirements, which are unlikely to be found in dynamic testing.
Reviews, static analysis and dynamic testing have the same objective – identifying defects. They are
complementary: the different techniques can find different types of defect effectively and efficiently. In
contrast to dynamic testing, reviews find defects rather than failures.
Typical defects that are easier to find in reviews than in dynamic testing are: deviations from
standards, requirement defects, design defects, insufficient maintainability and incorrect interface
specifications.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 29 of 73 July 1st 2005
© International Software Testing Qualifications Board
3.2 Review process (K2) 25 minutes
Terms
Entry criteria, exit criteria, formal review, informal review, inspection, kick-off, metrics, moderator/
inspection leader, peer review, reviewer, review meeting, review process, scribe, technical review,
walkthrough.
Background
Reviews vary from very informal to very formal (i.e. well structured and regulated). The formality of a
review process is related to factors such as the maturity of the development process, any legal or
regulatory requirements or the need for an audit trail.
The way a review is carried out depends on the agreed objective of the review (e.g. find defects, gain
understanding, or discussion and decision by consensus).
3.2.1 Phases of a formal review (K1)
A typical formal review has the following main phases:
 Planning: selecting the personnel, allocating roles; defining the entry and exit criteria for more
formal review types (e.g. inspection); and selecting which parts of documents to look at.
 Kick-off: distributing documents; explaining the objectives, process and documents to the
participants; and checking entry criteria (for more formal review types).
 Individual preparation: work done by each of the participants on their own before the review
meeting, noting potential defects, questions and comments.
 Review meeting: discussion or logging, with documented results or minutes (for more formal
review types). The meeting participants may simply note defects, make recommendations for
handling the defects, or make decisions about the defects.
 Rework: fixing defects found, typically done by the author.
 Follow-up: checking that defects have been addressed, gathering metrics and checking on
exit criteria (for more formal review types).
3.2.2 Roles and responsibilities (K1)
A typical formal review will include the roles below:
 Manager: decides on the execution of reviews, allocates time in project schedules and
determines if the review objectives have been met.
 Moderator: the person who leads the review of the document or set of documents, including
planning the review, running the meeting, and follow-up after the meeting. If necessary, the
moderator may mediate between the various points of view and is often the person upon
whom the success of the review rests.
 Author: the writer or person with chief responsibility for the document(s) to be reviewed.
 Reviewers: individuals with a specific technical or business background (also called checkers
or inspectors) who, after the necessary preparation, identify and describe findings (e.g.
defects) in the product under review. Reviewers should be chosen to represent different
perspectives and roles in the review process and they take part in any review meetings.
 Scribe (or recorder): documents all the issues, problems and open points that were identified
during the meeting.
Looking at documents from different perspectives, and using checklists, can make reviews more
effective and efficient, for example, a checklist based on perspectives such as user, maintainer, tester
or operations, or a checklist of typical requirements problems.
3.2.3 Types of review (K2)
A single document may be the subject of more than one review. If more than one type of review is
used, the order may vary. For example, an informal review may be carried out before a technical
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 30 of 73 July 1st 2005
© International Software Testing Qualifications Board
review, or an inspection may be carried out on a requirement specification before a walkthrough with
customers. The main characteristics, options and purposes of common review types are:
Informal review
Key characteristics:
 no formal process;
 there may be pair programming or a technical lead reviewing designs and code;
 optionally may be documented;
 may vary in usefulness depending on the reviewer;
 main purpose: inexpensive way to get some benefit.
Walkthrough
Key characteristics:
 meeting led by author;
 scenarios, dry runs, peer group;
 open-ended sessions;
 optionally a pre-meeting preparation of reviewers, review report, list of findings and scribe
(who is not the author)
 may vary in practice from quite informal to very formal;
 main purposes: learning, gaining understanding, defect finding.
Technical review
Key characteristics:
 documented, defined defect-detection process that includes peers and technical experts;
 may be performed as a peer review without management participation;
 ideally led by trained moderator (not the author);
 pre-meeting preparation;
 optionally the use of checklists, review report, list of findings and management participation;
 may vary in practice from quite informal to very formal;
 main purposes: discuss, make decisions, evaluate alternatives, find defects, solve technical
problems and check conformance to specifications and standards.
Inspection
Key characteristics:
 led by trained moderator (not the author);
 usually peer examination;
 defined roles;
 includes metrics;
 formal process based on rules and checklists with entry and exit criteria;
 pre-meeting preparation;
 inspection report, list of findings;
 formal follow-up process;
 optionally, process improvement and reader;
 main purpose: find defects.
3.2.4 Success factors for reviews (K2)
Success factors for reviews include:
 Each review has a clear predefined objective.
 The right people for the review objectives are involved.
 Defects found are welcomed, and expressed objectively.
 People issues and psychological aspects are dealt with (e.g. making it a positive experience
for the author).
 Review techniques are applied that are suitable to the type and level of software work
products and reviewers.
 Checklists or roles are used if appropriate to increase effectiveness of defect identification.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 31 of 73 July 1st 2005
© International Software Testing Qualifications Board
 Training is given in review techniques, especially the more formal techniques, such as
inspection.
 Management supports a good review process (e.g. by incorporating adequate time for review
activities in project schedules).
 There is an emphasis on learning and process improvement.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 32 of 73 July 1st 2005
© International Software Testing Qualifications Board
3.3 Static analysis by tools (K2) 20 minutes
Terms
Compiler, complexity, control flow, data flow, static analysis.
Background
The objective of static analysis is to find defects in software source code and software models. Static
analysis is performed without actually executing the software being examined by the tool; dynamic
testing does execute the software code. Static analysis can locate defects that are hard to find in
testing. As with reviews, static analysis finds defects rather than failures. Static analysis tools analyze
program code (e.g. control flow and data flow), as well as generated output such as HTML and XML.
The value of static analysis is:
 Early detection of defects prior to test execution.
 Early warning about suspicious aspects of the code or design, by the calculation of metrics,
such as a high complexity measure.
 Identification of defects not easily found by dynamic testing.
 Detecting dependencies and inconsistencies in software models, such as links.
 Improved maintainability of code and design.
 Prevention of defects, if lessons are learned in development.
Typical defects discovered by static analysis tools include:
 referencing a variable with an undefined value;
 inconsistent interface between modules and components;
 variables that are never used;
 unreachable (dead) code;
 programming standards violations;
 security vulnerabilities;
 syntax violations of code and software models.
Static analysis tools are typically used by developers (checking against predefined rules or
programming standards) before and during component and integration testing, and by designers
during software modeling. Static analysis tools may produce a large number of warning messages,
which need to be well managed to allow the most effective use of the tool.
Compilers may offer some support for static analysis, including the calculation of metrics.
References
3.2 IEEE 1028
3.2.2 Gilb, 1993, van Veenendaal, 2004
3.2.4 Gilb, 1993, IEEE 1028
3.3 Van Veenendaal, 2004
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 33 of 73 July 1st 2005
© International Software Testing Qualifications Board
4. Test design techniques (K3) 255 minutes
Learning objectives for test design techniques
The objectives identify what you will be able to do following the completion of each module.
4.1 Identifying test conditions and designing test cases (K3)
 Differentiate between a test design specification, test case specification and test procedure
specification. (K1)
 Compare the terms test condition, test case and test procedure. (K2)
 Write test cases: (K3)
o showing a clear traceability to the requirements;
o containing an expected result.
 Translate test cases into a well-structured test procedure specification at a level of detail
relevant to the knowledge of the testers. (K3)
 Write a test execution schedule for a given set of test cases, considering prioritization, and
technical and logical dependencies. (K3)
4.2 Categories of test design techniques (K2)
 Recall reasons that both specification-based (black-box) and structure-based (white-box)
approaches to test case design are useful, and list the common techniques for each. (K1)
 Explain the characteristics and differences between specification-based testing, structurebased
testing and experience-based testing. (K2)
4.3 Specification-based or black-box techniques (K3)
 Write test cases from given software models using the following test design techniques: (K3)
o equivalence partitioning;
o boundary value analysis;
o decision tables;
o state transition diagrams.
 Understand the main purpose of each of the four techniques, what level and type of testing
could use the technique, and how coverage may be measured. (K2)
 Understand the concept of use case testing and its benefits. (K2)
4.4 Structure-based or white-box techniques (K3)
 Describe the concept and importance of code coverage. (K2)
 Explain the concepts of statement and decision coverage, and understand that these concepts
can also be used at other test levels than component testing (e.g. on business procedures at
system level). (K2)
 Write test cases from given control flows using the following test design techniques: (K3)
o statement testing;
o decision testing.
 Assess statement and decision coverage for completeness. (K3)
4.5 Experience-based techniques (K2)
 Recall reasons for writing test cases based on intuition, experience and knowledge about
common defects. (K1)
 Compare experience-based techniques with specification-based testing techniques. (K2)
4.6 Choosing test techniques (K2)
 List the factors that influence the selection of the appropriate test design technique for a
particular kind of problem, such as the type of system, risk, customer requirements, models for
use case modeling, requirements models or tester knowledge. (K2)
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 34 of 73 July 1st 2005
© International Software Testing Qualifications Board
4.1 Identifying test conditions and designing test
cases (K3)
15 minutes
Terms
Test cases, test case specification, test condition, test data, test procedure specification, test script,
traceability.
Background
The process of identifying test conditions and designing tests consists of a number of steps:
 Designing tests by identifying test conditions.
 Specifying test cases.
 Specifying test procedures.
The process can be done in different ways, from very informal with little or no documentation, to very
formal (as it is described in this section). The level of formality depends on the context of the testing,
including the organization, the maturity of testing and development processes, time constraints, and
the people involved.
During test design, the test basis documentation is analyzed in order to determine what to test, i.e. to
identify the test conditions. A test condition is defined as an item or event that could be verified by one
or more test cases (e.g. a function, transaction, quality characteristic or structural element).
Establishing traceability from test conditions back to the specifications and requirements enables both
impact analysis, when requirements change, and requirements coverage to be determined for a set of
tests. During test design the detailed test approach is implemented based on, among other
considerations, the risks identified (see Chapter 5 for more on risk analysis).
During test case specification the test cases and test data are developed and described in detail by
using test design techniques. A test case consists of a set of input values, execution preconditions,
expected results and execution post-conditions, developed to cover certain test condition(s). The
‘Standard for Software Test Documentation’ (IEEE 829) describes the content of test design
specifications and test case specifications.
Expected results should be produced as part of the specification of a test case and include outputs,
changes to data and states, and any other consequences of the test. If expected results have not
been defined then a plausible, but erroneous, result may be interpreted as the correct one. Expected
results should ideally be defined prior to test execution.
The test cases are put in an executable order; this is the test procedure specification. The test
procedure (or manual test script) specifies the sequence of action for the execution of a test. If tests
are run using a test execution tool, the sequence of actions is specified in a test script (which is an
automated test procedure).
The various test procedures and automated test scripts are subsequently formed into a test execution
schedule that defines the order in which the various test procedures, and possibly automated test
scripts, are executed, when they are to be carried out and by whom. The test execution schedule will
take into account such factors as regression tests, prioritization, and technical and logical
dependencies.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 35 of 73 July 1st 2005
© International Software Testing Qualifications Board
4.2 Categories of test design techniques (K2) 15 minutes
Terms
Black-box techniques, experience-based techniques, specification-based techniques, structure-based
techniques, white-box techniques.
Background
The purpose of a test design technique is to identify test conditions and test cases.
It is a classic distinction to denote test techniques as black box or white box. Black-box techniques
(also called specification-based techniques) are a way to derive and select test conditions or test
cases based on an analysis of the test basis documentation, whether functional or non-functional, for
a component or system without reference to its internal structure. White-box techniques (also called
structural or structure-based techniques) are based on an analysis of the internal structure of the
component or system.
Some techniques fall clearly into a single category; others have elements of more than one category.
This syllabus refers to specification-based or experience-based approaches as black-box techniques
and structure-based as white-box techniques.
Common features of specification-based techniques:
 Models, either formal or informal, are used for the specification of the problem to be solved,
the software or its components.
 From these models test cases can be derived systematically.
Common features of structure-based techniques:
 Information about how the software is constructed is used to derive the test cases, for
example, code and design.
 The extent of coverage of the software can be measured for existing test cases, and further
test cases can be derived systematically to increase coverage.
Common features of experience-based techniques:
 The knowledge and experience of people are used to derive the test cases.
o knowledge of testers, developers, users and other stakeholders about the software, its
usage and its environment;
o knowledge about likely defects and their distribution.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 36 of 73 July 1st 2005
© International Software Testing Qualifications Board
4.3 Specification-based or black-box techniques
(K3)
120 minutes
Terms
Boundary value analysis, decision table testing, equivalence partitioning, state transition testing, use
case testing.
4.3.1 Equivalence partitioning (K3)
Inputs to the software or system are divided into groups that are expected to exhibit similar behavior,
so they are likely to be processed in the same way. Equivalence partitions (or classes) can be found
for both valid data and invalid data, i.e. values that should be rejected. Partitions can also be identified
for outputs, internal values, time-related values (e.g. before or after an event) and for interface
parameters (e.g. during integration testing). Tests can be designed to cover partitions. Equivalence
partitioning (EP) is applicable at all levels of testing.
Equivalence partitioning as a technique can be used to achieve input and output coverage. It can be
applied to human input, input via interfaces to a system, or interface parameters in integration testing.
4.3.2 Boundary value analysis (K3)
Behavior at the edge of each equivalence partition is more likely to be incorrect, so boundaries are an
area where testing is likely to yield defects. The maximum and minimum values of a partition are its
boundary values. A boundary value for a valid partition is a valid boundary value; the boundary of an
invalid partition is an invalid boundary value. Tests can be designed to cover both valid and invalid
boundary values. When designing test cases, a value on each boundary is chosen.
Boundary value analysis can be applied at all test levels. It is relatively easy to apply and its defectfinding
capability is high; detailed specifications are helpful.
This technique is often considered an extension of equivalence partitioning and can be used on input
by humans as well as, for example, on timing or table boundaries. Boundary values may also be used
for test data selection.
4.3.3 Decision table testing (K3)
Decision tables are a good way to capture system requirements that contain logical conditions, and to
document internal system design. They may be used to record complex business rules that a system
is to implement. The specification is analyzed, and conditions and actions of the system are identified.
The input conditions and actions are most often stated in such a way that they can either be true or
false (Boolean). The decision table contains the triggering conditions, often combinations of true and
false for all input conditions, and the resulting actions for each combination of conditions. Each column
of the table corresponds to a business rule that defines a unique combination of conditions that result
in the execution of the actions associated with that rule. The coverage standard commonly used with
decision table testing is to have at least one test per column, which typically involves covering all
combinations of triggering conditions.
The strength of decision table testing is that it creates combinations of conditions that might not
otherwise have been exercised during testing. It may be applied to all situations when the action of the
software depends on several logical decisions.
4.3.4 State transition testing (K3)
A system may exhibit a different response depending on current conditions or previous history (its
state). In this case, that aspect of the system can be shown as a state transition diagram. It allows the
tester to view the software in terms of its states, transitions between states, the inputs or events that
trigger state changes (transitions) and the actions which may result from those transitions. The states
of the system or object under test are separate, identifiable and finite in number. A state table shows
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 37 of 73 July 1st 2005
© International Software Testing Qualifications Board
the relationship between the states and inputs, and can highlight possible transitions that are invalid.
Tests can be designed to cover a typical sequence of states, to cover every state, to exercise every
transition, to exercise specific sequences of transitions or to test invalid transitions.
State transition testing is much used within the embedded software industry and technical automation
in general. However, the technique is also suitable for modeling a business object having specific
states or testing screen-dialogue flows (e.g. for internet applications or business scenarios).
4.3.5 Use case testing (K2)
Tests can be specified from use cases or business scenarios. A use case describes interactions
between actors, including users and the system, which produce a result of value to a system user.
Each use case has preconditions, which need to be met for a use case to work successfully. Each use
case terminates with post-conditions, which are the observable results and final state of the system
after the use case has been completed. A use case usually has a mainstream (i.e. most likely)
scenario, and sometimes alternative branches.
Use cases describe the “process flows” through a system based on its actual likely use, so the test
cases derived from use cases are most useful in uncovering defects in the process flows during realworld
use of the system. Use cases, often referred to as scenarios, are very useful for designing
acceptance tests with customer/user participation. They also help uncover integration defects caused
by the interaction and interference of different components, which individual component testing would
not see.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 38 of 73 July 1st 2005
© International Software Testing Qualifications Board
4.4 Structure-based or white-box techniques (K3) 60 minutes
Terms
Code coverage, decision coverage, statement coverage, structural testing, structure-based testing,
white-box testing.
Background
Structure-based testing/white-box testing is based on an identified structure of the software or system,
as seen in the following examples:
 Component level: the structure is that of the code itself, i.e. statements, decisions or branches.
 Integration level: the structure may be a call tree (a diagram in which modules call other
modules).
 System level: the structure may be a menu structure, business process or web page structure.
In this section, two code-related structural techniques for code coverage, based on statements and
decisions, are discussed. For decision testing, a control flow diagram may be used to visualize the
alternatives for each decision.
4.4.1 Statement testing and coverage (K3)
In component testing, statement coverage is the assessment of the percentage of executable
statements that have been exercised by a test case suite. Statement testing derives test cases to
execute specific statements, normally to increase statement coverage.
4.4.2 Decision testing and coverage (K3)
Decision coverage, related to branch testing, is the assessment of the percentage of decision
outcomes (e.g. the True and False options of an IF statement) that have been exercised by a test
case suite. Decision testing derives test cases to execute specific decision outcomes, normally to
increase decision coverage.
Decision testing is a form of control flow testing as it generates a specific flow of control through the
decision points. Decision coverage is stronger than statement coverage: 100% decision coverage
guarantees 100% statement coverage, but not vice versa.
4.4.3 Other structure-based techniques (K1)
There are stronger levels of structural coverage beyond decision coverage, for example, condition
coverage and multiple condition coverage.
The concept of coverage can also be applied at other test levels (e.g. at integration level) where the
percentage of modules, components or classes that have been exercised by a test case suite could be
expressed as module, component or class coverage.
Tool support is useful for the structural testing of code.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 39 of 73 July 1st 2005
© International Software Testing Qualifications Board
4.5 Experience-based techniques (K2) 30 minutes
Terms
Error guessing, exploratory testing.
Background
Perhaps the most widely practiced technique is error guessing. Tests are derived from the tester’s skill
and intuition and their experience with similar applications and technologies. When used to augment
systematic techniques, intuitive testing can be useful to identify special tests not easily captured by
formal techniques, especially when applied after more formal approaches. However, this technique
may yield widely varying degrees of effectiveness, depending on the testers’ experience. A structured
approach to the error guessing technique is to enumerate a list of possible errors and to design tests
that attack these errors. These defect and failure lists can be built based on experience, available
defect and failure data, and from common knowledge about why software fails.
Exploratory testing is concurrent test design, test execution, test logging and learning, based on a test
charter containing test objectives, and carried out within time-boxes. It is an approach that is most
useful where there are few or inadequate specifications and severe time pressure, or in order to
augment or complement other, more formal testing. It can serve as a check on the test process, to
help ensure that the most serious defects are found.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 40 of 73 July 1st 2005
© International Software Testing Qualifications Board
4.6 Choosing test techniques (K2) 15 minutes
Terms
No specific terms.
Background
The choice of which test techniques to use depends on a number of factors, including the type of
system, regulatory standards, customer or contractual requirements, level of risk, type of risk, test
objective, documentation available, knowledge of the testers, time and budget, development life cycle,
use case models and previous experience of types of defects found.
Some techniques are more applicable to certain situations and test levels; others are applicable to all
test levels.
References
4.1 Craig, 2002, Hetzel, 1998, IEEE 829
4.2 Beizer, 1990, Copeland, 2004
4.3.1 Copeland, 2004, Myers, 1979
4.3.2 Copeland, 2004, Myers, 1979
4.3.3 Beizer, 1990, Copeland, 2004
4.3.4 Beizer, 1990, Copeland, 2004
4.3.5 Copeland, 2004
4.4.3 Beizer, 1990, Copeland, 2004
4.5 Kaner, 2002
4.6 Beizer, 1990, Copeland, 2004
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 41 of 73 July 1st 2005
© International Software Testing Qualifications Board
5. Test management (K3) 180 minutes
Learning objectives for test management
The objectives identify what you will be able to do following the completion of each module.
5.1 Test organization (K2)
 Recognize the importance of independent testing. (K1)
 List the benefits and drawbacks of independent testing within an organization. (K2)
 Recognize the different team members to be considered for the creation of a test team. (K1)
 Recall the tasks of typical test leader and tester. (K1)
5.2 Test planning and estimation (K2)
 Recognize the different levels and objectives of test planning. (K1)
 Summarize the purpose and content of the test plan, test design specification and test
procedure documents according to the ‘Standard for Software Test Documentation’ (IEEE
829). (K2)
 Recall typical factors that influence the effort related to testing. (K1)
 Differentiate between two conceptually different estimation approaches: the metrics-based
approach and the expert-based approach. (K2)
 Differentiate between the subject of test planning for a project, for individual test levels (e.g.
system test) or specific test targets (e.g. usability test), and for test execution. (K2)
 List test preparation and execution tasks that need planning. (K1)
 Recognize/justify adequate exit criteria for specific test levels and groups of test cases (e.g.
for integration testing, acceptance testing or test cases for usability testing). (K2)
5.3 Test progress monitoring and control (K2)
 Recall common metrics used for monitoring test preparation and execution. (K1)
 Understand and interpret test metrics for test reporting and test control (e.g. defects found and
fixed, and tests passed and failed). (K2)
 Summarize the purpose and content of the test summary report document according to the
‘Standard for Software Test Documentation’ (IEEE 829). (K2)
5.4 Configuration management (K2)
 Summarize how configuration management supports testing. (K2)
5.5 Risk and testing (K2)
 Describe a risk as a possible problem that would threaten the achievement of one or more
stakeholders’ project objectives. (K2)
 Remember that risks are determined by likelihood (of happening) and impact (harm resulting if
it does happen). (K1)
 Distinguish between the project and product risks. (K2)
 Recognize typical product and project risks. (K1)
 Describe, using examples, how risk analysis and risk management may be used for test
planning. (K2)
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 42 of 73 July 1st 2005
© International Software Testing Qualifications Board
5.6 Incident Management (K3)
 Recognize the content of the ‘Standard for Software Test Documentation’ (IEEE 829) incident
report. (K1)
 Write an incident report covering the observation of a failure during testing. (K3)
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 43 of 73 July 1st 2005
© International Software Testing Qualifications Board
5.1 Test organization (K2) 30 minutes
Terms
Tester, test leader, test manager.
5.1.1 Test organization and independence (K2)
The effectiveness of finding defects by testing and reviews can be improved by using independent
testers. Options for independence are:
 Independent testers within the development teams.
 Independent test team or group within the organization, reporting to project management or
executive management.
 Independent testers from the business organization, user community and IT.
 Independent test specialists for specific test targets such as usability testers, security testers
or certification testers (who certify a software product against standards and regulations).
 Independent testers outsourced or external to the organization.
For large, complex or safety critical projects, it is usually best to have multiple levels of testing, with
some or all of the levels done by independent testers. Development staff may participate in testing,
especially at the lower levels, but their lack of objectivity often limits their effectiveness. The
independent testers may have the authority to require and define test processes and rules, but testers
should take on such process-related roles only in the presence of a clear management mandate to do
so.
The benefits of independence include:
 Independent testers see other and different defects, and are unbiased.
 An independent tester can verify assumptions people made during specification and
implementation of the system.
Drawbacks include:
 Isolation from the development team (if treated as totally independent).
 Independent testers may be the bottleneck as the last checkpoint.
 Developers lose a sense of responsibility for quality.
Testing tasks may be done by people in a specific testing role, or may be done by someone in another
role, such as a project manager, quality manager, developer, business and domain expert,
infrastructure or IT operations.
5.1.2 Tasks of the test leader and tester (K1)
In this syllabus two test positions are covered, test leader and tester. The activities and tasks
performed by people in these two roles depend on the project and product context, the people in the
roles, and the organization.
Sometimes the test leader is called a test manager or test coordinator. The role of the test leader may
be performed by a project manager, a development manager, a quality assurance manager or the
manager of a test group. In larger projects two positions may exist: test leader and test manager.
Typically the test leader plans, monitors and controls the testing activities and tasks as defined in
Section 1.4.
Typical test leader tasks may include:
 Coordinate the test strategy and plan with project managers and others.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 44 of 73 July 1st 2005
© International Software Testing Qualifications Board
 Write or review a test strategy for the project, and test policy for the organization.
 Contribute the testing perspective to other project activities, such as integration planning.
 Plan the tests – considering the context and understanding the risks – including selecting test
approaches, estimating the time, effort and cost of testing, acquiring resources, defining test
levels, cycles, approach, and objectives, and planning incident management;
 Initiate the specification, preparation, implementation and execution of tests, and monitor and
control the execution.
 Adapt planning based on test results and progress (sometimes documented in status reports)
and take any action necessary to compensate for problems.
 Set up adequate configuration management of testware for traceability.
 Introduce suitable metrics for measuring test progress and evaluating the quality of the testing
and the product.
 Decide what should be automated, to what degree, and how.
 Select tools to support testing and organize any training in tool use for testers.
 Decide about the implementation of the test environment.
 Schedule tests.
 Write test summary reports based on the information gathered during testing.
Typical tester tasks may include:
 Review and contribute to test plans.
 Analyze, review and assess user requirements, specifications and models for testability.
 Create test specifications.
 Set up the test environment (often coordinating with system administration and network
management).
 Prepare and acquire test data.
 Implement tests on all test levels, execute and log the tests, evaluate the results and
document the deviations from expected results.
 Use test administration or management tools and test monitoring tools as required.
 Automate tests (may be supported by a developer or a test automation expert).
 Measure performance of components and systems (if applicable).
 Review tests developed by others.
People who work on test analysis, test design, specific test types or test automation may be
specialists in these roles. Depending on the test level and the risks related to the product and the
project, different people may take over the role of tester, keeping some degree of independence.
Typically testers at the component and integration level would be developers, testers at the
acceptance test level would be business experts and users, and testers for operational acceptance
testing would be operators.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 45 of 73 July 1st 2005
© International Software Testing Qualifications Board
5.2 Test planning and estimation (K2) 50 minutes
Terms
Entry criteria, exit criteria, exploratory testing, test approach, test level, test plan, test procedure, test
strategy.
5.2.1 Test planning (K2)
This section covers the purpose of test planning within development and implementation projects, and
for maintenance activities. Planning may be documented in a project or master test plan, and in
separate test plans for test levels, such as system testing and acceptance testing. Outlines of test
planning documents are covered by the ‘Standard for Software Test Documentation’ (IEEE 829).
Planning is influenced by the test policy of the organization, the scope of testing, objectives, risks,
constraints, criticality, testability and the availability of resources. The more the project and test
planning progresses the more information is available and the more detail that can be included in the
plan.
Test planning is a continuous activity and is performed in all life cycle processes and activities.
Feedback from test activities is used to recognize changing risks so that planning can be adjusted.
5.2.2 Test planning activities (K2)
Test planning activities may include:
 Defining the overall approach of testing (the test strategy), including the definition of the test
levels and entry and exit criteria.
 Integrating and coordinating the testing activities into the software life cycle activities:
acquisition, supply, development, operation and maintenance.
 Making decisions about what to test, what roles will perform the test activities, when and how
the test activities should be done, how the test results will be evaluated, and when to stop
testing (exit criteria).
 Assigning resources for the different tasks defined.
 Defining the amount, level of detail, structure and templates for the test documentation.
 Selecting metrics for monitoring and controlling test preparation and execution, defect
resolution and risk issues.
 Setting the level of detail for test procedures in order to provide enough information to support
reproducible test preparation and execution.
5.2.3 Exit criteria (K2)
The purpose of exit criteria is to define when to stop testing, such as at the end of a test level or when
a set of tests has a specific goal.
Typically exit criteria may consist of:
 Thoroughness measures, such as coverage of code, functionality or risk.
 Estimates of defect density or reliability measures.
 Cost.
 Residual risks, such as defects not fixed or lack of test coverage in certain areas.
 Schedules such as those based on time to market.
5.2.4 Test estimation (K2)
Two approaches for the estimation of test effort are covered in this syllabus:
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 46 of 73 July 1st 2005
© International Software Testing Qualifications Board
 Estimating the testing effort based on metrics of former or similar projects or based on typical
values.
 Estimating the tasks by the owner of these tasks or by experts.
Once the test effort is estimated, resources can be identified and a schedule can be drawn up.
The testing effort may depend on a number of factors, including:
 Characteristics of the product: the quality of the specification and other information used for
test models (i.e. the test basis), the size of the product, the complexity of the problem domain,
the requirements for reliability and security, and the requirements for documentation.
 Characteristics of the development process: the stability of the organization, tools used, test
process, skills of the people involved, and time pressure.
 The outcome of testing: the number of defects and the amount of rework required.
5.2.5 Test approaches (test strategies) (K2)
One way to classify test approaches or strategies is based on the point in time at which the bulk of the
test design work is begun:
 Preventative approaches, where tests are designed as early as possible.
 Reactive approaches, where test design comes after the software or system has been
produced.
Typical approaches or strategies include:
 Analytical approaches, such as risk-based testing where testing is directed to areas of
greatest risk.
 Model-based approaches, such as stochastic testing using statistical information about failure
rates (such as reliability growth models) or usage (such as operational profiles).
 Methodical approaches, such as failure based (including error guessing and fault-attacks),
check-list based, and quality characteristic based.
 Process- or standard-compliant approaches, such as those specified by industry-specific
standards or the various agile methodologies.
 Dynamic and heuristic approaches, such as exploratory testing where testing is more reactive
to events than pre-planned, and where execution and evaluation are concurrent tasks.
 Consultative approaches, such as those where test coverage is driven primarily by the advice
and guidance of technology and/or business domain experts outside the test team.
 Regression-averse approaches, such as those that include reuse of existing test material,
extensive automation of functional regression tests, and standard test suites.
Different approaches may be combined, for example, a risk-based dynamic approach.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 47 of 73 July 1st 2005
© International Software Testing Qualifications Board
The selection of a test approach should consider the context, including:
 Risk of failure of the project, hazards to the product and risks of product failure to humans, the
environment and the company.
 Skills and experience of the people in the proposed techniques, tools and methods.
 The objective of the testing endeavour and the mission of the testing team.
 Regulatory aspects, such as external and internal regulations for the development process.
 The nature of the product and the business.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 48 of 73 July 1st 2005
© International Software Testing Qualifications Board
5.3 Test progress monitoring and control (K2) 20 minutes
Terms
Defect density, failure rate, test control, test coverage, test monitoring, test report.
5.3.1 Test progress monitoring (K1)
The purpose of test monitoring is to give feedback and visibility about test activities. Information to be
monitored may be collected manually or automatically and may be used to measure exit criteria, such
as coverage. Metrics may also be used to assess progress against the planned schedule and budget.
Common test metrics include:
 Percentage of work done in test case preparation (or percentage of planned test cases
prepared).
 Percentage of work done in test environment preparation.
 Test case execution (e.g. number of test cases run/not run, and test cases passed/failed).
 Defect information (e.g. defect density, defects found and fixed, failure rate, and retest
results).
 Test coverage of requirements, risks or code.
 Subjective confidence of testers in the product.
 Dates of test milestones.
 Testing costs, including the cost compared to the benefit of finding the next defect or to run
the next test.
5.3.2 Test Reporting (K2)
Test reporting is concerned with summarizing information about the testing endeavour, including:
 What happened during a period of testing, such as dates when exit criteria were met.
 Analyzed information and metrics to support recommendations and decisions about future
actions, such as an assessment of defects remaining, the economic benefit of continued
testing, outstanding risks, and the level of confidence in tested software.
The outline of a test summary report is given in ‘Standard for Software Test Documentation’ (IEEE
829).
Metrics should be collected during and at the end of a test level in order to assess:
 The adequacy of the test objectives for that test level.
 The adequacy of the test approaches taken.
 The effectiveness of the testing with respect to its objectives.
5.3.3 Test control (K2)
Test control describes any guiding or corrective actions taken as a result of information and metrics
gathered and reported. Actions may cover any test activity and may affect any other software life cycle
activity or task.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 49 of 73 July 1st 2005
© International Software Testing Qualifications Board
Examples of test control actions are:
 Re-prioritize tests when an identified risk occurs (e.g. software delivered late).
 Change the test schedule due to availability of a test environment.
 Set an entry criterion requiring fixes to have been retested by a developer before accepting
them into a build.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 50 of 73 July 1st 2005
© International Software Testing Qualifications Board
5.4 Configuration management (K2) 10 minutes
Terms
Configuration management, version control.
Background
The purpose of configuration management is to establish and maintain the integrity of the products
(components, data and documentation) of the software or system through the project and product life
cycle.
For testing, configuration management may involve ensuring that:
 All items of testware are identified, version controlled, tracked for changes, related to each
other and related to development items (test objects) so that traceability can be maintained
throughout the test process.
 All identified documents and software items are referenced unambiguously in test
documentation.
For the tester, configuration management helps to uniquely identify (and to reproduce) the tested item,
test documents, the tests and the test harness.
During test planning, the configuration management procedures and infrastructure (tools) should be
chosen, documented and implemented.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 51 of 73 July 1st 2005
© International Software Testing Qualifications Board
5.5 Risk and testing (K2) 30 minutes
Terms
Product risk, project risk, risk, risk-based testing.
Background
Risk can be defined as the chance of an event, hazard, threat or situation occurring and its
undesirable consequences, a potential problem. The level of risk will be determined by the likelihood
of an adverse event happening and the impact (the harm resulting from that event).
5.5.1 Project risks (K1, K2)
Project risks are the risks that surround the project’s capability to deliver its objectives, such as:
 Supplier issues:
o failure of a third party;
o contractual issues.
 Organizational factors:
o skill and staff shortages;
o personal and training issues;
o political issues, such as
 problems with testers communicating their needs and test results;
 failure to follow up on information found in testing and reviews (e.g. not
improving development and testing practices).
o improper attitude toward or expectations of testing (e.g. not appreciating the value of
finding defects during testing).
 Technical issues:
o problems in defining the right requirements;
o the extent that requirements can be met given existing constraints;
o the quality of the design, code and tests.
When analyzing, managing and mitigating these risks, the test manager is following well established
project management principles. The ‘Standard for Software Test Documentation’ (IEEE 829) outline
for test plans requires risks and contingencies to be stated.
5.5.2 Product Risks (K2)
Potential failure areas (adverse future events or hazards) in the software or system are known as
product risks, as they are a risk to the quality of the product, such as:
 Error-prone software delivered.
 The potential that the software/hardware could cause harm to an individual or company.
 Poor software characteristics (e.g. functionality, security, reliability, usability and performance).
 Software that does not perform its intended functions.
Risks are used to decide where to start testing and where to test more; testing is used to reduce the
risk of an adverse effect occurring, or to reduce the impact of an adverse effect.
Product risks are a special type of risk to the success of a project. Testing as a risk-control activity
provides feedback about the residual risk by measuring the effectiveness of critical defect removal and
of contingency plans.
A risk-based approach to testing provides proactive opportunities to reduce the levels of product risk,
starting in the initial stages of a project. It involves the identification of product risks and their use in
guiding the test planning, specification, preparation and execution of tests. In a risk-based approach
the risks identified may be used to:
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 52 of 73 July 1st 2005
© International Software Testing Qualifications Board
 Determine the test techniques to be employed.
 Determine the extent of testing to be carried out.
 Prioritize testing in an attempt to find the critical defects as early as possible.
 Determine whether any non-testing activities could be employed to reduce risk (e.g. providing
training to inexperienced designers).
Risk-based testing draws on the collective knowledge and insight of the project stakeholders to
determine the risks and the levels of testing required to address those risks.
To ensure that the chance of a product failure is minimized, risk management activities provide a
disciplined approach to:
 Assess (and reassess on a regular basis) what can go wrong (risks).
 Determine what risks are important to deal with.
 Implement actions to deal with those risks.
In addition, testing may support the identification of new risks, may help to determine what risks
should be reduced, and may lower uncertainty about risks.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 53 of 73 July 1st 2005
© International Software Testing Qualifications Board
5.6 Incident management (K3) 40 minutes
Terms
Incident logging.
Background
Since one of the objectives of testing is to find defects, the discrepancies between actual and
expected outcomes need to be logged as incidents. Incidents should be tracked from discovery and
classification to correction and confirmation of the solution. In order to manage all incidents to
completion, an organization should establish a process and rules for classification.
Incidents may be raised during development, review, testing or use of a software product. They may
be raised for issues in code or the working system, or in any type of documentation including
development documents, test documents or user information such as “Help” or installation guides.
Incident reports have the following objectives:
 Provide developers and other parties with feedback about the problem to enable identification,
isolation and correction as necessary.
 Provide test leaders a means of tracking the quality of the system under test and the progress
of the testing.
 Provide ideas for test process improvement.
A tester or reviewer typically logs the following information, if known, regarding an incident:
 Date of issue, issuing organization, author, approvals and status.
 Scope, severity and priority of the incident.
 References, including the identity of the test case specification that revealed the problem.
Details of the incident report may include:
 Expected and actual results.
 Date the incident was discovered.
 Identification or configuration item of the software or system.
 Software or system life cycle process in which the incident was observed.
 Description of the anomaly to enable resolution.
 Degree of impact on stakeholder(s) interests.
 Severity of the impact on the system.
 Urgency/priority to fix.
 Status of the incident (e.g. open, deferred, duplicate, waiting to be fixed, fixed awaiting
confirmation test or closed).
 Conclusions and recommendations.
 Global issues, such as other areas that may be affected by a change resulting from the
incident.
 Change history, such as the sequence of actions taken by project team members with respect
to the incident to isolate, repair and confirm it as fixed.
The structure of an incident report is covered in the ‘Standard for Software Test Documentation’ (IEEE
829) and is called an anomaly report.
References
5.1.1 Black, 2001, Hetzel, 1998
5.1.2 Black, 2001, Hetzel, 1998
5.2.5 Black, 2001, Craig, 2002, IEEE 829, Kaner 2002
5.3.3 Black, 2001, Craig, 2002, Hetzel, 1998, IEEE 829
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 54 of 73 July 1st 2005
© International Software Testing Qualifications Board
5.4 Craig, 2002
5.5.2 Black, 2001 , IEEE 829
5.6 Black, 2001, IEEE 829
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 55 of 73 July 1st 2005
© International Software Testing Qualifications Board
6. Tool support for testing (K2) 80 minutes
Learning objectives for tool support for testing
The objectives identify what you will be able to do following the completion of each module.
6.1 Types of test tool (K2)
 Classify different types of test tools according to the test process activities. (K2)
 Recognize tools that may help developers in their testing. (K1)
6.2 Effective use of tools: potential benefits and risks (K2)
 Summarize the potential benefits and risks of test automation and tool support for testing. (K2)
 Recognize that test execution tools can have different scripting techniques, including data
driven and keyword driven. (K1)
6.3 Introducing a tool into an organization (K1)
 State the main principles of introducing a tool into an organization. (K1)
 State the goals of a proof-of-concept/piloting phase for tool evaluation. (K1)
 Recognize that factors other than simply acquiring a tool are required for good tool support.
(K1)
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 56 of 73 July 1st 2005
© International Software Testing Qualifications Board
6.1 Types of test tool (K2) 45 minutes
Terms
Configuration management tool, coverage measurement tool, debugging tool, driver, dynamic analysis
tool, incident management tool, load testing tool, modeling tool, monitoring tool, performance testing
tool, probe effect, requirements management tool, review process support tool, security tool, static
analysis tool, stress testing tool, stub, test comparator, test data preparation tool, test design tool, test
harness, test execution tool, test management tool, unit test framework tool.
6.1.1 Test tool classification (K2)
There are a number of tools that support different aspects of testing. Tools are classified in this
syllabus according to the testing activities that they support.
Some tools clearly support one activity; others may support more than one activity, but are classified
under the activity with which they are most closely associated. Some commercial tools offer support
for only one type of activity; other commercial tool vendors offer suites or families of tools that provide
support for many or all of these activities.
Testing tools can improve the efficiency of testing activities by automating repetitive tasks. Testing
tools can also improve the reliability of testing by, for example, automating large data comparisons or
simulating behavior.
Some types of test tool can be intrusive in that the tool itself can affect the actual outcome of the test.
For example, the actual timing may be different depending on how you measure it with different
performance tools, or you may get a different measure of code coverage depending on which
coverage tool you use. The consequence of intrusive tools is called the probe effect.
Some tools offer support more appropriate for developers (e.g. during component and component
integration testing). Such tools are marked with “(D)” in the classifications below.
6.1.2 Tool support for management of testing and tests (K1)
Management tools apply to all test activities over the entire software life cycle.
Test management tools
Characteristics of test management tools include:
 Support for the management of tests and the testing activities carried out.
 Interfaces to test execution tools, defect tracking tools and requirement management tools.
 Independent version control or interface with an external configuration management tool.
 Support for traceability of tests, test results and incidents to source documents, such as
requirement specifications.
 Logging of test results and generation of progress reports.
 Quantitative analysis (metrics) related to the tests (e.g. tests run and tests passed) and the
test object (e.g. incidents raised), in order to give information about the test object, and to
control and improve the test process.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 57 of 73 July 1st 2005
© International Software Testing Qualifications Board
Requirements management tools
Requirements management tools store requirement statements, check for consistency and undefined
(missing) requirements, allow requirements to be prioritized and enable individual tests to be traceable
to requirements, functions and/or features. Traceability may be reported in test management progress
reports. The coverage of requirements, functions and/or features by a set of tests may also be
reported.
Incident management tools
Incident management tools store and manage incident reports, i.e. defects, failures or perceived
problems and anomalies, and support management of incident reports in ways that include:
 Facilitating their prioritization.
 Assignment of actions to people (e.g. fix or confirmation test).
 Attribution of status (e.g. rejected, ready to be tested or deferred to next release).
These tools enable the progress of incidents to be monitored over time, often provide support for
statistical analysis and provide reports about incidents. They are also known as defect tracking tools.
Configuration management tools
Configuration management (CM) tools are not strictly testing tools, but are typically necessary to keep
track of different versions and builds of the software and tests.
Configuration Management tools:
 Store information about versions and builds of software and testware.
 Enable traceability between testware and software work products and product variants.
 Are particularly useful when developing on more than one configuration of the
hardware/software environment (e.g. for different operating system versions, different libraries
or compilers, different browsers or different computers).
6.1.3 Tool support for static testing (K1)
Review process support tools
Review process support tools may store information about review processes, store and communicate
review comments, report on defects and effort, manage references to review rules and/or checklists
and keep track of traceability between documents and source code. They may also provide aid for
online reviews, which is useful if the team is geographically dispersed.
Static analysis tools (D)
Static analysis tools support developers, testers and quality assurance personnel in finding defects
before dynamic testing. Their major purposes include:
 The enforcement of coding standards.
 The analysis of structures and dependencies (e.g. linked web pages).
 Aiding in understanding the code.
Static analysis tools can calculate metrics from the code (e.g. complexity), which can give valuable
information, for example, for planning or risk analysis.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 58 of 73 July 1st 2005
© International Software Testing Qualifications Board
Modeling tools (D)
Modeling tools are able to validate models of the software. For example, a database model checker
may find defects and inconsistencies in the data model; other modeling tools may find defects in a
state model or an object model. These tools can often aid in generating some test cases based on the
model (see also Test design tools below).
The major benefit of static analysis tools and modeling tools is the cost effectiveness of finding more
defects at an earlier time in the development process. As a result, the development process may
accelerate and improve by having less rework.
6.1.4 Tool support for test specification (K1)
Test design tools
Test design tools generate test inputs or the actual tests from requirements, from a graphical user
interface, from design models (state, data or object) or from code. This type of tool may generate
expected outcomes as well (i.e. may use a test oracle). The generated tests from a state or object
model are useful for verifying the implementation of the model in the software, but are seldom
sufficient for verifying all aspects of the software or system. They can save valuable time and provide
increased thoroughness of testing because of the completeness of the tests that the tool can
generate.
Other tools in this category can aid in supporting the generation of tests by providing structured
templates, sometimes called a test frame, that generate tests or test stubs, and thus speed up the test
design process.
Test data preparation tools (D)
Test data preparation tools manipulate databases, files or data transmissions to set up test data to be
used during the execution of tests. A benefit of these tools is to ensure that live data transferred to a
test environment is made anonymous, for data protection.
6.1.5 Tool support for test execution and logging (K1)
Test execution tools
Test execution tools enable tests to be executed automatically, or semi-automatically, using stored
inputs and expected outcomes, through the use of a scripting language. The scripting language makes
it possible to manipulate the tests with limited effort, for example, to repeat the test with different data
or to test a different part of the system with similar steps. Generally these tools include dynamic
comparison features and provide a test log for each test run.
Test execution tools can also be used to record tests, when they may be referred to as capture
playback tools. Capturing test inputs during exploratory testing or unscripted testing can be useful in
order to reproduce and/or document a test, for example, if a failure occurs.
Test harness/unit test framework tools (D)
A test harness may facilitate the testing of components or part of a system by simulating the
environment in which that test object will run. This may be done either because other components of
that environment are not yet available and are replaced by stubs and/or drivers, or simply to provide a
predictable and controllable environment in which any faults can be localized to the object under test.
A framework may be created where part of the code, object, method or function, unit or component
can be executed, by calling the object to be tested and/or giving feedback to that object. It can do this
by providing artificial means of supplying input to the test object, and/or by supplying stubs to take
output from the object, in place of the real output targets.
Test harness tools can also be used to provide an execution framework in middleware, where
languages, operating systems or hardware must be tested together.
They may be called unit test framework tools when they have a particular focus on the component test
level. This type of tool aids in executing the component tests in parallel with building the code.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 59 of 73 July 1st 2005
© International Software Testing Qualifications Board
Test comparators
Test comparators determine differences between files, databases or test results. Test execution tools
typically include dynamic comparators, but post-execution comparison may be done by a separate
comparison tool. A test comparator may use a test oracle, especially if it is automated.
Coverage measurement tools (D)
Coverage measurement tools can be either intrusive or non-intrusive depending on the measurement
techniques used, what is measured and the coding language. Code coverage tools measure the
percentage of specific types of code structure that have been exercised (e.g. statements, branches or
decisions, and module or function calls). These tools show how thoroughly the measured type of
structure has been exercised by a set of tests.
Security tools
Security tools check for computer viruses and denial of service attacks. A firewall, for example, is not
strictly a testing tool, but may be used in security testing. Other security tools stress the system by
searching specific vulnerabilities of the system.
6.1.6 Tool support for performance and monitoring (K1)
Dynamic analysis tools (D)
Dynamic analysis tools find defects that are evident only when software is executing, such as time
dependencies or memory leaks. They are typically used in component and component integration
testing, and when testing middleware.
Performance testing/load testing/stress testing tools
Performance testing tools monitor and report on how a system behaves under a variety of simulated
usage conditions. They simulate a load on an application, a database, or a system environment, such
as a network or server. The tools are often named after the aspect of performance that it measures,
such as load or stress, so are also known as load testing tools or stress testing tools. They are often
based on automated repetitive execution of tests, controlled by parameters.
Monitoring tools
Monitoring tools are not strictly testing tools but provide information that can be used for testing
purposes and which is not available by other means.
Monitoring tools continuously analyze, verify and report on usage of specific system resources, and
give warnings of possible service problems. They store information about the version and build of the
software and testware, and enable traceability.
6.1.7 Tool support for specific application areas (K1)
Individual examples of the types of tool classified above can be specialized for use in a particular type
of application. For example, there are performance testing tools specifically for web-based
applications, static analysis tools for specific development platforms, and dynamic analysis tools
specifically for testing security aspects.
Commercial tool suites may target specific application areas (e.g. embedded systems).
6.1.8 Tool support using other tools (K1)
The test tools listed here are not the only types of tools used by testers – they may also use
spreadsheets, SQL, resource or debugging tools (D), for example.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 60 of 73 July 1st 2005
© International Software Testing Qualifications Board
6.2 Effective use of tools: potential benefits and
risks (K2)
20 minutes
Terms
Data-driven (testing), keyword-driven (testing), scripting language.
6.2.1 Potential benefits and risks of tool support for testing (for all
tools) (K2)
Simply purchasing or leasing a tool does not guarantee success with that tool. Each type of tool may
require additional effort to achieve real and lasting benefits. There are potential benefit opportunities
with the use of tools in testing, but there are also risks.
Potential benefits of using tools include:
 Repetitive work is reduced (e.g. running regression tests, re-entering the same test data, and
checking against coding standards).
 Greater consistency and repeatability (e.g. tests executed by a tool, and tests derived from
requirements).
 Objective assessment (e.g. static measures, coverage and system behavior).
 Ease of access to information about tests or testing (e.g. statistics and graphs about test
progress, incident rates and performance).
Risks of using tools include:
 Unrealistic expectations for the tool (including functionality and ease of use).
 Underestimating the time, cost and effort for the initial introduction of a tool (including training
and external expertise).
 Underestimating the time and effort needed to achieve significant and continuing benefits from
the tool (including the need for changes in the testing process and continuous improvement of
the way the tool is used).
 Underestimating the effort required to maintain the test assets generated by the tool.
 Over-reliance on the tool (replacement for test design or where manual testing would be
better).
6.2.2 Special considerations for some types of tool (K1)
Test execution tools
Test execution tools replay scripts designed to implement tests that are stored electronically. This type
of tool often requires significant effort in order to achieve significant benefits.
Capturing tests by recording the actions of a manual tester seems attractive, but this approach does
not scale to large numbers of automated tests. A captured script is a linear representation with specific
data and actions as part of each script. This type of script may be unstable when unexpected events
occur.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 61 of 73 July 1st 2005
© International Software Testing Qualifications Board
A data-driven approach separates out the test inputs (the data), usually into a spreadsheet, and uses
a more generic script that can read the test data and perform the same test with different data. Testers
who are not familiar with the scripting language can enter test data for these predefined scripts.
In a keyword-driven approach, the spreadsheet contains keywords describing the actions to be taken
(also called action words), and test data. Testers (even if they are not familiar with the scripting
language) can then define tests using the keywords, which can be tailored to the application being
tested.
Technical expertise in the scripting language is needed for all approaches (either by testers or by
specialists in test automation).
Whichever scripting technique is used, the expected results for each test need to be stored for later
comparison.
Performance testing tools
Performance testing tools need someone with expertise in performance testing to help design the
tests and interpret the results.
Static analysis tools
Static analysis tools applied to source code can enforce coding standards, but if applied to existing
code may generate a lot of messages. Warning messages do not stop the code being translated into
an executable program, but should ideally be addressed so that maintenance of the code is easier in
the future. A gradual implementation with initial filters to exclude some messages would be an
effective approach.
Test management tools
Test management tools need to interface with other tools or spreadsheets in order to produce
information in the best format for the current needs of the organization. The reports need to be
designed and monitored so that they provide benefit.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 62 of 73 July 1st 2005
© International Software Testing Qualifications Board
6.3 Introducing a tool into an organization (K1) 15 minutes
Terms
No specific terms.
Background
The main principles of introducing a tool into an organization include:
 Assessment of organizational maturity, strengths and weaknesses and identification of
opportunities for an improved test process supported by tools.
 Evaluation against clear requirements and objective criteria.
 A proof-of-concept to test the required functionality and determine whether the product meets
its objectives.
 Evaluation of the vendor (including training, support and commercial aspects).
 Identification of internal requirements for coaching and mentoring in the use of the tool.
The proof-of-concept could be done in a small-scale pilot project, making it possible to minimize
impacts if major hurdles are found and the pilot is not successful.
A pilot project has the following objectives:
 Learn more detail about the tool.
 See how the tool would fit with existing processes and practices, and how they would need to
change.
 Decide on standard ways of using, managing, storing and maintaining the tool and the test
assets (e.g. deciding on naming conventions for files and tests, creating libraries and defining
the modularity of test suites).
 Assess whether the benefits will be achieved at reasonable cost.
Success factors for the deployment of the tool within an organization include:
 Rolling out the tool to the rest of the organization incrementally.
 Adapting and improving processes to fit with the use of the tool.
 Providing training and coaching/mentoring for new users.
 Defining usage guidelines.
 Implementing a way to learn lessons from tool use.
 Monitoring tool use and benefits.
References
6.2.2 Buwalda, 2001, Fewster, 1999
6.3 Fewster, 1999
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 63 of 73 July 1st 2005
© International Software Testing Qualifications Board
7. References
Standards
ISTQB Glossary of terms used in Software Testing Version 1.0
[CMMI] Chrissis, M.B., Konrad, M. and Shrum, S. (2004) CMMI, Guidelines for Process Integration
and Product Improvement, Addison Wesley: Reading, MA
See section 2.1
[IEEE 829] IEEE Std 829™ (1998/2005) IEEE Standard for Software Test Documentation (currently
under revision)
See sections 2.3, 2.4, 4.1, 5.2, 5.3, 5.5, 5.6
[IEEE 1028] IEEE Std 1028™ (1997) IEEE Standard for Software Reviews
See section 3.2
[IEEE 12207] IEEE 12207/ISO/IEC 12207-1996, Software life cycle processes
See section 2.1
[ISO 9126] ISO/IEC 9126-1:2001, Software Engineering – Software Product Quality
See section 2.3
Books
[Beizer, 1990] Beizer, B. (1990) Software Testing Techniques (2nd edition), Van Nostrand Reinhold:
Boston
See sections 1.2, 1.3, 2.3, 4.2, 4.3, 4.4, 4.6
[Black, 2001] Black, R. (2001) Managing the Testing Process (2nd edition), John Wiley & Sons: New
York
See sections 1.1, 1.2, 1.4, 1.5, 2.3, 2.4, 5.1, 5.2, 5.3, 5.5, 5.6
[Buwalda, 2001] Buwalda, H. et al. (2001) Integrated Test Design and Automation, Addison Wesley:
Reading, MA
See section 6.2
[Copeland, 2004] Copeland, L. (2004) A Practitioner’s Guide to Software Test Design, Artech House:
Norwood, MA
See sections 2.2, 2.3, 4.2, 4.3, 4.4, 4.6
[Craig, 2002] Craig, Rick D. and Jaskiel, Stefan P. (2002) Systematic Software Testing, Artech House:
Norwood, MA
See sections 1.4.5, 2.1.3, 2.4, 4.1, 5.2.5, 5.3, 5.4
[Fewster, 1999] Fewster, M. and Graham, D. (1999) Software Test Automation, Addison Wesley:
Reading, MA
See sections 6.2, 6.3
[Gilb, 1993]: Gilb, Tom and Graham, Dorothy (1993) Software Inspection, Addison Wesley: Reading,
MA
See sections 3.2.2, 3.2.4
[Hetzel, 1988] Hetzel, W. (1988) Complete Guide to Software Testing, QED: Wellesley, MA
See sections 1.3, 1.4, 1.5, 2.1, 2.2, 2.3, 2.4, 4.1, 5.1, 5.3
[Kaner, 2002] Kaner, C., Bach, J. and Pettticord, B. (2002) Lessons Learned in Software Testing,
John Wiley & Sons:
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 64 of 73 July 1st 2005
© International Software Testing Qualifications Board
See sections 1.1, 4.5, 5.2
[Myers 1979] Myers, Glenford J. (1979) The Art of Software Testing, John Wiley & Sons:
See sections 1.2, 1.3, 2.2, 4.3
[van Veenendaal, 2004] van Veenendaal, E. (ed.) (2004) The Testing Practitioner (Chapters 6, 8, 10),
UTN Publishers: The Netherlands
See sections 3.2, 3.3
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 65 of 73 July 1st 2005
© International Software Testing Qualifications Board
Appendix A – Syllabus background
History of this document
This document was prepared during 2004–2005 by a working party comprising members appointed by
the International Software Testing Qualifications Board (ISTQB). It was initially reviewed by a selected
review panel, and then by representatives drawn from the international software testing community.
The rules used in the production of this document are shown in Appendix C.
This document is the syllabus for the International Foundation Certificate in Software Testing, the first
level international qualification approved by the ISTQB (www.istqb.org). At the time of writing (2005),
ISTQB membership includes Austria, Denmark, Finland, France, Germany, India, Israel, Japan,
Korea, Norway, Poland, Portugal, Spain, Sweden, Switzerland, The Netherlands, UK and USA.
Objectives of the Foundation Certificate qualification
 To gain recognition for testing as an essential and professional software engineering
specialization.
 To provide a standard framework for the development of testers' careers.
 To enable professionally qualified testers to be recognized by employers, customers and
peers, and to raise the profile of testers.
 To promote consistent and good testing practices within all software engineering disciplines.
 To identify testing topics that are relevant and of value to industry.
 To enable software suppliers to hire certified testers and thereby gain commercial advantage
over their competitors by advertising their tester recruitment policy.
 To provide an opportunity for testers and those with an interest in testing to acquire an
internationally recognized qualification in the subject.
Objectives of the international qualification (adapted from ISTQB meeting
at Sollentuna, November 2001)
 To be able to compare testing skills across different countries.
 To enable testers to move across country borders more easily.
 To enable multi-national/international projects to have a common understanding of testing
issues.
 To increase the number of qualified testers worldwide.
 To have more impact/value as an internationally based initiative than from any country-specific
approach.
 To develop a common international body of understanding and knowledge about testing
through the syllabus and terminology, and to increase the level of knowledge about testing for
all participants.
 To promote testing as a profession in more countries.
 To enable testers to gain a recognized qualification in their native language.
 To enable sharing of knowledge and resources across countries.
 To provide international recognition of testers and this qualification due to participation from
many countries.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 66 of 73 July 1st 2005
© International Software Testing Qualifications Board
Entry requirements for this qualification
The entry criterion for taking the ISTQB Foundation Certificate in Software Testing examination is that
candidates have an interest in software testing. However, it is strongly recommended that candidates
also:
 Have at least a minimal background in either software development or software testing, such
as six months experience as a system or user acceptance tester or as a software developer.
 Take a course that has been accredited to ISTQB standards (by one of the ISTQB-recognized
national boards).
Background and history of the Foundation Certificate in Software Testing
The independent certification of software testers began in the UK with the British Computer Society's
Information Systems Examination Board (ISEB), when a Software Testing Board was set up in 1998
(www.bcs.org.uk/iseb). In 2002, ASQF in Germany began to support a German tester qualification
scheme (www.asqf.de). This syllabus is based on the ISEB and ASQF syllabi; it includes reorganized,
updated and some new content, and the emphasis is directed at topics that will provide the most
practical help to testers.
An existing Foundation Certificate in Software Testing (e.g. from ISEB, ASQF or an ISTQB-recognized
national board) awarded before this International Certificate was released, will be deemed to be
equivalent to the International Certificate. The Foundation Certificate does not expire and does not
need to be renewed. The date it was awarded is shown on the Certificate.
Within each participating country, local aspects are controlled by a national ISTQB-recognized
Software Testing Board. Duties of national boards are specified by the ISTQB, but are implemented
within each country. The duties of the country boards are expected to include accreditation of training
providers and the setting of exams.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 67 of 73 July 1st 2005
© International Software Testing Qualifications Board
Appendix B – Learning objectives/level of
knowledge
The following learning objectives are defined as applying to this syllabus. Each topic in the syllabus
will be examined according to the learning objective for it.
Level 1: Remember (K1)
The candidate will recognize, remember and recall a term or concept.
Example
Can recognize the definition of “failure” as:
 “non-delivery of service to an end user or any other stakeholder” or
 “actual deviation of the component or system from its expected delivery, service or result”.
Level 2: Understand (K2)
The candidate can select the reasons or explanations for statements related to the topic, and can
summarize, compare, classify and give examples for the testing concept.
Examples
Can explain the reason why tests should be designed as early as possible:
 To find defects when they are cheaper to remove.
 To find the most important defects first.
Can explain the similarities and differences between integration and system testing:
 Similarities: testing more than one component, and can test non-functional aspects.
 Differences: integration testing concentrates on interfaces and interactions, and system testing
concentrates on whole-system aspects, such as end to end processing.
Level 3: Apply (K3)
The candidate can select the correct application of a concept or technique and apply it to a given
context.
Example
 Can identify boundary values for valid and invalid partitions.
 Can select test cases from a given state transition diagram in order to cover all transitions.
Reference
(For the cognitive levels of learning objectives)
Anderson, L. W. and Krathwohl, D. R. (eds) (2001) A Taxonomy for Learning, Teaching, and
Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, Allyn & Bacon:
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 68 of 73 July 1st 2005
© International Software Testing Qualifications Board
Appendix C – Rules applied to the ISTQB
Foundation syllabus
The rules listed here were used in the development and review of this syllabus. (A “TAG” is shown
after each rule as a shorthand abbreviation of the rule.)
General rules
SG1. The syllabus should be understandable and absorbable by people with 0 to 6 months (or more)
experience in testing. (6-MONTH)
SG2. The syllabus should be practical rather than theoretical. (PRACTICAL)
SG3. The syllabus should be clear and unambiguous to its intended readers. (CLEAR)
SG4. The syllabus should be understandable to people from different countries, and easily
translatable into different languages. (TRANSLATABLE)
SG5. The syllabus should use American English. (AMERICAN-ENGLISH)
Current content
SC1. The syllabus should include recent testing concepts and should reflect current best practice in
software testing where this is generally agreed. The syllabus is subject to review every three to five
years. (RECENT)
SC2. The syllabus should minimize time-related issues, such as current market conditions, to enable it
to have a shelf life of three to five years. (SHELF-LIFE).
Learning Objectives
LO1. Learning objectives should distinguish between items to be recognized/remembered (cognitive
level K1), items the candidate should understand conceptually (K2) and those which the candidate
should be able to practice/use (K3). (KNOWLEDGE-LEVEL)
LO2. The description of the content should be consistent with the learning objectives. (LOCONSISTENT)
LO3. To illustrate the learning objectives, sample exam questions for each major section should be
issued along with the syllabus. (LO-EXAM)
Overall structure
ST1. The structure of the syllabus should be clear and allow cross-referencing to and from other parts,
from exam questions and from other relevant documents. (CROSS-REF)
ST2. Overlap between sections of the syllabus should be minimized. (OVERLAP)
ST3. Each section of the syllabus should have the same structure. (STRUCTURE-CONSISTENT)
ST4. The syllabus should contain version, date of issue and page number on every page. (VERSION)
ST5. The syllabus should include a guideline for the amount of time to be spent in each section (to
reflect the relative importance of each topic). (TIME-SPENT)
References
SR1. Sources and references will be given for concepts in the syllabus to help training providers find
out more information about the topic. (REFS)
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 69 of 73 July 1st 2005
© International Software Testing Qualifications Board
SR2. Where there are not readily identified and clear sources, more detail should be provided in the
syllabus. For example, definitions are in the Glossary, so only the terms are listed in the syllabus.
(NON-REF DETAIL)
Sources of information
Terms used in the syllabus are defined in ISTQB’s Glossary of terms used in Software Testing. A
version of the Glossary is available from ISTQB.
A list of recommended books on software testing is also issued in parallel with this syllabus. The main
book list is part of the References section.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board
Version 2005 Page 70 of 73 July 1st 2005
© International Software Testing Qualifications Board
Appendix D – Notice to training providers
Each major subject heading in the syllabus is assigned an allocated time in minutes. The purpose of
this is both to give guidance on the relative proportion of time to be allocated to each section of an
accredited course, and to give an approximate minimum time for the teaching of each section.
Training providers may spend more time than is indicated and candidates may spend more time again
in reading and research. A course curriculum does not have to follow the same order as the syllabus.
The syllabus contains references to established standards, which must be used in the preparation of
training material. Each standard used must be the version quoted in the current version of this
syllabus. Other publications, templates or standards not referenced in this syllabus may also be used
and referenced, but will not be examined.
The specific areas of the syllabus requiring practical exercises are as follows:
4.3 Specification-based or black-box techniques
Practical work (short exercises) should be included covering the four techniques: equivalence
partitioning, boundary value analysis, decision table testing and state transition testing. The lectures
and exercises relating to these techniques should be based on the references provided for each
technique.
4.4 Structure-based or white-box techniques
Practical work (short exercises) should be included to assess whether or not a set of tests achieve
100% statement and 100% decision coverage, as well as to design test cases for given control flows.
5.6 Incident management
Practical work (short exercise) should be included to cover the writing and/or assessment of an
incident report.
Certified Tester
Foundation Level Syllabus
International
Software Testing
Qualifications Board

Automated Testing Lifecycle Methodology

2007-02-15T20:57:00.000-08:00

Introducing the ATLM
Software project managers and software developers building today's applications face the challenge of doing so within an ever-shrinking schedule and with minimal resources. As part of their attempt to do more with less, organizations want to test software adequately, but as quickly and thoroughly as possible. To accomplish this goal, organizations are turning to automated testing.
Faced with this reality and realizing that many tests cannot be executed manually, such as simulating 1,000 virtual users for volume testing, software professionals are introducing automated testing to their projects. But these software professionals may not know what's involved in introducing an automated test tool to a software project, and they may be unfamiliar with the breadth of application that automated test tools have today. The Automated Testing Lifecycle Methodology (ATLM), depicted in Figure 1, provides guidance in these areas.
Figure 1 The Automated Test Lifecycle Methodology (ATLM).
By using the systematic approach outlined within the ATLM, organizations can organize and execute test activities in such a way as to maximize test coverage within the limits of testing resources. This structured test methodology involves a multi-stage process, supporting the detailed and interrelated activities that are required to introduce and utilize an automated test tool:
· Develop test design.
· Develop and execute test cases.
· Develop and manage test data and the test environment.
· Document, track, and obtain closure on issue/trouble reports.
Clearly, the emphasis on automated testing represents a paradigm change for the software industry. This change doesn't simply involve the application of tools and the performance of test automation. Rather, it blankets the entire test lifecycle and the system development lifecycle. The ATLM implementation takes place in parallel with the system development lifecycle. For software professionals to make a successful leap to automated testing, they must embrace structured approaches to testing. The ATLM is revolutionary in the fact that it promulgates a new structured, building-block approach to the entire test lifecycle, which enables software test professionals to approach software testing in a methodical and repeatable fashion.
The growth of automated test capability has stemmed in large part from the growing popularity of the iterative and incremental development lifecycle, a software development methodology that focuses on minimizing the development schedule while providing frequent, incremental software builds. The objective of this incremental and iterative development is to engage the user and the test team early throughout the design and development of each build in order to refine the software, thereby ensuring that it more closely reflects the needs and preferences of the user and thus addressing the riskiest aspects of development in early builds.
In this environment of continual changes and additions to the software through each software build, software testing itself takes on an iterative nature. Each new build is accompanied by a considerable number of new tests as well as rework to existing test scripts, just as there is rework on previously released software modules. Given the continual changes and additions to software applications, especially Web applications, automated software testing becomes an important control mechanism to ensure accuracy and stability of the software through each build.
The ATLM, invoked to support testing efforts involving automated test tools, incorporates a multi-stage process. The methodology supports the detailed and interrelated activities that are required to decide whether to acquire an automated testing tool. The methodology includes the process of how to introduce and utilize an automated test tool, covers test development and test design, and addresses test execution and management. The methodology also supports the development and management of test data and the test environment, and addresses test documentation to include problem reports.
The ATLM methodology represents a structured approach, which depicts a process with which to approach and execute testing. This structured approach is necessary to help steer the test team away from these common test program mistakes:
· Implementing the use of an automated test tool without a testing process in place, resulting in an ad hoc, non-repeatable, non-measurable test program.
· Implementing a test design without following any design standards, resulting in the creation of test scripts that are not repeatable and therefore not reusable for incremental software builds.
· Attempting to automate 100% of test requirements, when tools or in-house–developed automated test harnesses do not support automation of all tests required.
· Using the wrong tool or developing a too-elaborate in-house test harness.
· Initiating test tool implementation too late in the application-development lifecycle—not allowing sufficient time for tool setup and test tool introduction process (learning curve).
· Initiating test engineer involvement too late in the application-development lifecycle, resulting in poor understanding of the application and system design, which results in incomplete testing.
The Automated Test Lifecycle Methodology (ATLM) comprises six primary processes or components:
1. Decision to Automate Testing
2. Test Tool Acquisition
3. Automated Testing Introduction Process
4. Test Planning, Design, and Development
5. Execution and Management of Tests
6. Test Program Review and Assessment
The following sections describe each primary process, as well as the subordinate processes contained within each primary process.
Phase 1: Decision to Automate Testing
The decision to automate testing represents the first phase of the ATLM. This phase covers the entire process that goes into the automated testing decision. During this phase, it's important for the test team to manage automated testing expectations and to outline the potential benefits of automated testing when implemented correctly. A test tool proposal needs to be outlined, which will be helpful in acquiring management support.
Overcoming False Expectations for Automated Testing
While it has been proven that automated testing is valuable and can produce a successful return on investment, there isn't always an immediate payback on investment. It's important to address some of the misconceptions that persist in the software industry and to manage the automated testing utopia. Following is a list of just a few of the misconceptions that need to be addressed. People often see test automation as a silver bullet; when they find that test automation requires a significant short-term investment of time and energy to achieve a long-term return on investment (ROI) of faster and cheaper regression testing (for example), the testing tool often becomes "shelfware." This is why it's important to manage expectations in order to introduce automated testing correctly into a project.
Automatic Test Plan Generation
Currently, there is no commercially available tool that can automatically create a comprehensive test plan while also supporting test design and execution.
Throughout a software test career, the test engineer can expect to witness test tool demonstrations and review an abundant amount of test tool literature. Often the test engineer will be asked to stand before one or more senior managers to give a test tool functionality overview. As always, the presenter must bear in mind the audience. In this case, the audience may represent individuals with just enough technical knowledge to make them enthusiastic about automated testing, while unaware of the complexity involved with an automated test effort. Specifically, the managers may have obtained secondhand information about automated test tools, and may have reached the wrong interpretation of the actual capability of automated test tools.
What the audience at the management presentation may be waiting to hear is that the tool you're proposing automatically develops the test plan, designs and creates the test procedures, executes all the test procedures, and analyzes the results automatically. Meanwhile, you start out the presentation by informing the group that automated test tools should be viewed as enhancements to manual testing, and that automated test tools will not automatically develop the test plan, design and create the test procedures, or execute the test procedures.
Shortly into the presentation and after several management questions, it becomes very apparent just how much of a divide exists between the reality of the test tool capabilities and the perceptions of the individuals in the audience. The term automated test tool seems to bring with it a great deal of wishful thinking that's not closely aligned with reality. An automated test tool will not replace the human factor necessary for testing a product. The proficiencies of test engineers and other quality assurance experts will still be needed to keep the test machinery running. A test tool can be viewed as an additional part of the machinery that supports the release of a good product.
One Test Tool Fits All
Currently, no single test tool exists that can be used to support all operating system environments.
Generally, a single test tool will not fulfill all the testing requirements for an organization. Consider the experience of one test engineer encountering such a situation. The test engineer was asked by a manager to find a test tool that could be used to automate the testing of all the department's applications. The department was using various technologies including mainframe computers and Sun workstations; operating systems such as Windows 3.1, Windows 95, Windows NT, and Windows 2000; programming languages such as Visual C++ and Visual Basic; other client/server technologies; and Web technologies such as DHTML, XML, ASP, and so on.
After conducting a tool evaluation, the test engineer determined that the tool of choice was not compatible with the Visual C++ third-party add-ons (in this case, Stingray grids). Another tool had to be brought in that was compatible with this specific application.
Immediate Reduction in Schedule
An automated test tool will not immediately minimize the testing schedule.
Another automated test misconception is the expectation that the use of an automated testing tool on a new project will immediately minimize the test schedule. The testing schedule will not experience the anticipated decrease at first, and an allowance for schedule increase is required when initially introducing an automated test tool. This is due to the fact that when rolling out an automated test tool, the current testing process has to be augmented or an entirely new testing process has to be developed and implemented. The entire test team—and possibly the development team—needs to become familiar with this new automated testing process (such as ATLM) and needs to follow it. Once an automatic testing process has been established and effectively implemented, the project can expect to experience gains in productivity and turnaround time that have a positive effect on schedule and cost.
Benefits of Automated Testing
The previous discussion points out and clarifies some of the false automated testing expectations that exist. The test engineer will also need to be able to elaborate on the true benefits of automated testing, when automated testing is implemented correctly and a process is followed. The test engineer must evaluate whether potential benefits fit required improvement criteria and whether the pursuit of automated testing on the project is still a logical fit, given the organizational needs. There are three significant automated test benefits (in combination with manual testing):
· Producing a reliable system.
· Improving the quality of the test effort.
· Reducing test effort and minimizing schedule.
Many return on investment case studies have been done with regard to the implementation of automated testing. One example is a research effort conducted by imbus GmbH. They conducted a test automation value study in order to collect test automation measurements with the purpose of studying the benefits of test automation versus the implementation of manual test methods. Their research determined that the breakeven point of automated testing is on average at 2.03 test runs. (T. Linz and M. Daigl, "GUI Testing Made Painless: Implementation and Results of the ESSI Project Number 24306," 1998.)
Acquiring Management Support
Whenever an organization tries to adopt a new technology, they encounter a significant effort when determining how to apply it to their needs. Even with completed training, organizations wrestle with time-consuming false starts before they become capable with the new technology. For the test team interested in implementing automated test tools, the challenge is how to best present the case for a new test automation technology and its implementation to the management team.
Test engineers need to influence management's expectations for the use of automated testing on projects. Test engineers can help to manage expectations of others in the organization by forwarding helpful information to the management staff. Bringing up test tool issues during strategy and planning meetings can also help develop better understanding of test tool capabilities for everyone involved on a project or within the organization. A test engineer can develop training material on the subject of automated testing and can advocate to management that a seminar be scheduled to conduct the training.
The first step in moving toward a decision to automate testing on a project requires that the test team adjust management understanding of the appropriate application of automated testing for the specific need at hand. For example, the test team needs to check early on whether management is cost-averse and would be unwilling to accept the estimated cost of automated test tools for a particular effort. If so, test personnel need to convince management about the potential return on investment by conducting cost/benefit analysis.
If management is willing to invest in an automated test tool, but is unable or unwilling to staff a test team with individuals having the proper software skill level or to provide for adequate test tool training, the test team needs to point out the risks involved and/or may need to reconsider a recommendation to automate test.
Management also needs to be made aware of the additional cost involved when introducing a new tool—not only for the tool purchase, but for initial schedule/cost increase, additional training costs, and for enhancing an existing testing process or implementing a new testing process.
Test automation represents highly flexible technology, which provides several ways to accomplish an objective. Use of this technology requires new ways of thinking, which only amplifies the problem of test tool implementation. Many organizations can readily come up with examples of their own experience of technology that failed to deliver on its potential because of the difficulty of overcoming the "Now what?" syndrome. The issues that organizations face when adopting automated test systems include those outlined below:
· Finding/hiring test tool experts.
· Using the correct tool for the task at hand.
· Developing and implementing an automated testing process, which includes developing automated test design and development standards.
· Analyzing various applications to determine those that are best suited for automation.
· Analyzing the test requirements to determine the ones suitable for automation.
· Training the test team on the automated testing process, automated test design, development, and execution.
· Initial increase in schedule and cost.
The Automated Testing Lifecycle Methodology (ATLM)
Article Information
Contents
1. Introducing the ATLM
2. Phase 1: Decision to Automate Testing
3. Phase 2: Test Tool Acquisition
4. Phase 3: Automated Testing Introduction Process
5. Phase 4: Test Planning, Design, and Development
6. Phase 5: Execution and Management of Tests
7. Phase 6: Test Program Review and Assessment
8. Summary
Phase 2: Test Tool Acquisition
Test tool acquisition represents the second phase of the ATLM. This phase guides the test engineer through the entire test tool evaluation and selection process, starting with confirmation of management support. Since a tool should support most of the organizations' testing requirements, whenever feasible the test engineer will need to review the system's engineering environment and other organizational needs and come up with a list of tool evaluation criteria. A review of the different types of tools available to support aspects of the entire testing lifecycle is provided in Automated Software Testing: Introduction, Management, and Performance (as part of the ATLM), enabling the reader to make an informed decision with regard to the types of tests to be performed on a particular project. The test engineer then needs to define an evaluation domain to pilot the test tool. Finally, after all those steps have been completed, the test engineer can make vendor contact to bring in the selected tool(s). Test personnel then evaluate the tool, based on sample criteria provided.
Phase 3: Automated Testing Introduction Process
The process of introducing automated testing to a new project team constitutes the third phase of the ATLM. This phase outlines the steps necessary to successfully introduce automated testing to a new project, which are summarized in the following sections.
Test Process Analysis
Test process analysis ensures that an overall test process and strategy are in place and are modified, if necessary, to allow automated testing to be introduced in a successful fashion. The test engineers define and collect test process metrics in order to allow for process improvement. Here test goals/objectives and strategies need to be defined and test process needs to be documented and communicated to the test team. In this phase, the kinds of testing applicable for the technical environment will be defined, and tests are defined that can be supported by automated tools.
During the test process analysis, techniques are defined. Best practices are laid out, such as conducting performance testing during the unit-testing phase.
Plans for user involvement are assessed, and test team personnel skills are analyzed against test requirements and planned test activities. Early test team participation is emphasized, supporting refinement of requirement specifications into terms that can be adequately tested while also supporting test team understanding of application requirements and design.
Test Tool Consideration
The test tool consideration process includes steps that investigate whether incorporation of automated test tools that have been brought into the company without a specific project in mind now would be beneficial to a specific project, given the project testing requirements, available test environment, personnel resources, user environment, platform, and product features of the application under test. Schedule is reviewed to ensure sufficient time for test tool setup and development of requirements hierarchy; potential test tools and utilities are mapped to test requirements, test tool compatibility with the application and environment is verified, and workaround solutions are investigated for incompatibility problems that surfaced during compatibility tests.
Phase 4: Test Planning, Design, and Development
Test planning, design, and development is the fourth phase of the ATLM. These subjects are summarized in the following sections.
Test Planning
The test planning stage represents the need to review long–lead-time test planning activities. During this phase, the test team identifies test procedure creation standards and guidelines; hardware, software, and network required to support test environment; test data requirements; a preliminary test schedule; performance measure requirements; a procedure to control test configuration and environment; as well as defect-tracking procedure(s) and associated tracking tool(s).
The test plan contains the results of each preliminary phase of the structured test methodology (ATLM). The test plan will define roles and responsibilities, project test schedule, test planning and design activities, test environment preparation, test risks and contingencies, and acceptable level of thoroughness (test acceptance criteria). Test plan appendices may include test procedures, naming conventions, test procedure format standards, and a test procedure traceability matrix.
The test environment setup is part of test planning. It represents the need to plan, track, and manage test environment setup activities, where material procurements may have long lead times. The test team needs to schedule and track environment setup activities; install test environment hardware, software, and network resources; integrate and install test environment resources; obtain/refine test databases; and develop environment setup scripts and test bed scripts.
Test Design
The test design component addresses the need to define the number of tests to be performed, the ways that testing will be approached (paths, functions), and the test conditions that need to be exercised. Test design standards need to be defined and followed.
An effective test program, incorporating the automation of software testing, involves a mini-development lifecycle of its own, complete with strategy and goal planning, test requirement definition, analysis, design, and coding. Similar to software application development, test requirements must be specified before test design is constructed. Test requirements need to be clearly defined and documented, so that all project personnel will understand the basis of the test effort. Test requirements are defined within requirement statements as an outcome of test requirement analysis.
After test requirements have been derived using the described techniques, test procedure design can begin. Test procedure design consists of the definition of logical groups of test procedures and a naming convention for the suite of test procedures. With a test procedure definition in place, each test procedure is then identified as either an automated or a manual test. During the test planning phase, the test team gets an understanding of the number of test techniques being employed and an estimate for the number of test procedures that will be required. The test team also will have an estimate of the number of test procedures that will need to be performed manually, as well as with an automated test tool.
Much like a software development effort, the test program must be mapped out and consciously designed to ensure that test activities performed represent the most efficient and effective tests for the system under test. Test program resources are limited, yet ways of testing the system are endless. A test design is developed to portray the test effort, in order to give project and test personnel a mental framework on the boundary and scope of the test program.
Following test analysis, the test team develops the test program design models. The first of these design models, the test program model, consists of a graphical illustration that depicts the scope of the test program. This model typically depicts the test techniques required to support the dynamic test effort and also outline static test strategies.
Having defined a test program model, the test team constructs a test architecture, which depicts the structure of the test program and defines the way that test procedures will be organized in support of the test effort.
The next step in the test procedure design process (see Table 1) is to identify those test procedures that stand out as being more sophisticated, and as a result are required to be defined further as part of detailed test design. These test procedures are flagged and a detailed design document is prepared in support of the more sophisticated test procedures. Following detailed test design, test data requirements are mapped against the defined test procedures. To create a repeatable, reusable process for producing test procedures, the test team needs to create a document that outlines test procedure design standards. Only when these standards are followed can the automated test program achieve real efficiency and success, by being repeatable and maintainable.
Table 1 Test Procedure Design Process
Step
Description
1
Test Architecture Review. The test team reviews the test architecture in order to identify the test techniques that apply.
2
Test Procedure Definition (Development Level). A test procedure definition is constructed at the development test level, identifying the test procedure series that applies for the various design components and test techniques.
3
Test Procedure Definition (System Level). A test procedure definition is constructed at the system test level, identifying the test procedure series that applies for the various test techniques.
4
Test Procedure Design Standards. Design standards are adopted and a unique naming convention is adopted that distinguishes the test procedures on the project from test procedures developed in the past or on other projects.
5
Manual Versus Automated Tests. Test procedures will be depicted as being either performed manually or as part of an automated test.
6
Test Procedures Flagged for Detailed Design. Test procedures that stand out as more sophisticated are flagged. These test procedures are further defined as part of detailed test design.
7
Detailed Design. Those test procedures flagged as part of step 7 are designed in further detail within a detailed test design file or document. Test procedure detailed design may consist of pseudo-code of algorithms, preliminary test step definition, or pseudo-code of test automation programs.
8
Test Data Mapping. Test procedure matrix is modified to reflect test data requirements for each test procedure.

The exercise of developing the test procedure definition not only aids in test development, but helps to quantify or bound the test effort. The development of the test procedure definition involves the identification of the suite of test procedures that need to be developed and executed in support of the test effort. The design exercise involves the organization of test procedures into logical groups and the definition of a naming convention for the suite of test procedures.
At the system level, it may be worthwhile to develop a detailed test design for sophisticated tests. These tests might involve test procedures that perform complex algorithms, consist of both manual and automated steps, and test programming scripts that are modified for use in multiple test procedures. The first step in the detailed design process is to review the test procedure definition at the system test level. This review is conducted for the purpose of identifying those test procedures that stand out as being more sophisticated and that, as a result, are required to be defined further as part of detailed test design.
Detailed test design may take the form of test program pseudo-code, when test programming is required. The detailed design may be represented simply as a sequence of steps that need to be performed in support of a test. When programming variables and multiple data values are involved, the detailed design may reflect the programming construct of a loop supporting an iterative series of tests involving different values, together with a list or table identifying the kinds of data or ranges of data required for the test.
Following the performance of detailed test design, test data requirements need to be mapped against the defined test procedures. Once test data requirements are outlined, the test team needs to plan the means for obtaining, generating, or developing the test data.
The structure of the test program (test architecture) is commonly portrayed in two ways. One test procedure organization method involves the logical grouping of test procedures with the system application design components, and is referred to as a design-based test architecture. Another method represents a test technique perspective and associates test procedures with the various kinds of test techniques represented within the test program model, and is referred to as a technique-based test architecture.
An understanding of test techniques is necessary when developing test design and the test program design models. Personnel performing testing need to be familiar with the test techniques associated with the white box and black box test-approach methods. White box test techniques are aimed at exercising software program internals; black box techniques generally compare the application under test behavior against requirements that address testing via established public interfaces such as the user interface or the published application programming interface (API).
Test Development
For automated tests to be reusable, repeatable, and maintainable, test development standards need to be defined and followed.
After performing test analysis and design, the test team is now ready to perform test development.
Keep in mind that the test design and development activities follow an iterative and incremental approach, in order to address the highest risk functionality up front. Table 2 correlates the development process phases to the test process phases. The testing processes and steps outlined in the table are strategically aligned with the development process, and the execution of these steps results in the refinement of test procedures at the same time as developers are creating the software modules. Automated and/or manual test procedures are developed during the integration test phase with the intention of reusing them during the system test phase.
Table 2 Development/Test Relationship
Phase
Development Process
Test Process
Module (Unit) Development
Design module from requirements.
Perform test planning and test environment setup.

Code module.
Create test design and develop test data.

Debug module.
Write test scripts or record test scenario using module.

Unit test module.
Debug automated test script by running against module. Use tools that support unit testing.

Correct defects.
Rerun automated test script to regression test as defects are corrected.

Conduct performance testing.
Verify that system is scalable and will meet performance requirements.
Integration
Build system by connecting modules.
Integration-test connected modules.
Review trouble reports.
Combine unit test scripts and add new scripts that demonstrate module interconnectivity. Use test tool to support automated integration testing.

Correct defects and update defect status.
Rerun automated test script as part of regression test, as defects are corrected.

Continued performance testing activities.
Verify that system is scalable and will meet performance requirements.
System Test
Review trouble reports.
Integrate automated test scripts into system-level test procedures where possible, and develop additional system-level test procedures. Execute system test and record test results.

Correct defects and update defect status.
Rerun automated test script as part of regression test as defects are corrected.
Acceptance Test
Review incident reports.
Perform subset of system test as part of demonstration of user acceptance test.

Correct defects.
Rerun automated test script as part of regression test as defects are corrected.

Many preparation activities need to take place before test development can begin. A test development architecture is developed (described in the next section), which provides the test team with a clear picture of the test development preparation activities or building blocks necessary for the efficient creation of test procedures. The test team will need to tailor the sample test development architecture to reflect the priorities of their particular project. Part of these setup and preparation activities involves the need to track and manage test environment set up activities, where material procurements may have long lead times. Prior to the commencement of test development, the test team also needs to perform analysis to identify the potential for reuse of existing test procedures and scripts within the automation infrastructure (reuse library).
The test team needs to develop test procedures according to a test procedure development/execution schedule. This schedule needs to allocate personnel resources and reflect development due dates, among other factors. The test team needs to monitor development progress and produce progress status reports. Prior to the creation of a complete suite of test procedures, the test team performs a modularity relationship analysis. The results of this analysis help to incorporate data dependencies, plan for workflow dependencies between tests, and identify common scripts that can be applied repeatedly to the test effort. As test procedures are being developed, the test team needs to ensure that configuration control is performed for the entire test bed to include test design, test scripts, and test data, as well as for each individual test procedure. The test bed needs to be baselined using a configuration management tool.
Test development involves the development of test procedures that are maintainable, reusable, simple, and robust, which in itself can be as challenging as the development of the application under test. Test procedure development standards need to be in place supporting structured and consistent development of automated tests. Test development standards can be based on the scripting language standards of a particular test tool. For example, Rational's Robot uses SQABasic, a Visual Basic–like scripting language, and therefore the script development standards could be based on the Visual Basic development standards, outlined in a number of books on the subject.
Usually internal development standards exist that can be followed if the organization is developing in a language similar to the tool's scripting language. The adoption or slight modification of existing development standards is generally a better approach than creating a standard from scratch. If no development standards exist within the organization for the particular tool scripting language, it's important for the test team to develop script development guidelines. Such guidelines can include directions on context independence, which addresses the particular place where a test procedure should start and where it should end. Additionally, modularity and reusability guidelines need to be addressed.
By developing test procedures based on development guidelines, the test team creates the initial building blocks for an automation infrastructure. The automation infrastructure will eventually contain a library of common, reusable scripts. Throughout the test effort and in future releases, the test engineer can make use of the automation infrastructure to support reuse of archived test procedures, minimize duplication, and thus enhance the entire automation effort.
Test Development Architecture
Test team members responsible for test development need to be prepared with the proper materials. Test team personnel need to follow a test development architecture that includes, for example, a listing of the test procedures assigned to them and a listing of the outcome of automated versus manual test analysis. Also, test team personnel need to decide when to automate. At times a test team might want to avoid automating using a GUI testing tool before the interface—whether API, character UI, or GUI—is stabilized, to avoid having to reengineer the automated tests in response to non–bug-related changes. At other times, the test team might find workaround solutions when automating an unstable GUI, such as focusing automation on the known stable parts only.
The test engineer needs to adhere to the test procedure development and execution schedule, test design information, automated test tool user manuals, and test procedure development guidelines. Armed with the proper instructions, documentation, and guidelines, test engineers will have the foundation that allows them to develop a more cohesive and structured set of test procedures. Repeating a process and repeatedly demonstrating a strong test program depends on the availability of documented processes and standard guidelines such as the test development architecture.
Figure 2 shows an example of a graphical illustration containing the major activities to be performed as part of the test development architecture. Test development starts with test environment setup and preparation activities, discussed earlier. Once they're concluded, the test team needs to make sure that all pertinent information necessary to support development has been documented or gathered. The test team will need to tailor the sample test development architecture in Figure 2 to reflect the priorities of their particular project. Note that Figure 2 should be read from bottom to top.
Figure 2 Building blocks of the test development architecture.
Technical Environment
Test procedure development needs to be preceded by several setup activities. The test development activity needs to be supported by a technical environment, which facilitates the development of test procedures. As a result, the test environment needs to be set up and ready to go. The test environment includes the technical environment, which may include facility resources as well as the hardware and software necessary to support test development and execution. The test team needs to ensure that there are enough workstations to support the entire team. The various elements of the test environment need to be outlined within the test plan, as discussed earlier.
Environment setup activities can also include the use of an environment setup script to load test data or restore a drive image, and to calibrate the test tool to the environment. When test tool compatibility problems arise with the application under test, workaround solutions have to be identified. When developing test procedures, it's important that the schedule for developing test procedures is consistent with the test execution schedule. It's also important that the test team follow test procedure development guidelines.
The test team must ensure that the proper test room or laboratory facilities are reserved and set up. Once the physical environment is established, the test team ensures that all necessary equipment is installed and operational. The test plan defined the required technical environment and addressed test environment planning. Within the test environment section of the test plan, the test team has already identified operational support required to install and check out the operational readiness of the technical environment. The test team needs to ensure that operational support activities have been properly scheduled and must monitor progress of these tasks.
Specific tasks and potential issues outlined in the test plan should now have been addressed and resolved. Such issues could include network installation, network server configuration and allocated disk space, network access privileges, required desktop computer processing speed and memory, number and types of desktop computers (clients), video resolution requirements, and any additional software required to support the application, such as browser software. Automated test tools that apply should have been scheduled for installation and checkout. These tools now should be configured to support the test team and be operational within the test environment.
The test environment setup activity includes the need to track and manage test environment setup activities, where material procurements may have long lead times. These activities include the need to schedule and track environment setup activities; install test environment hardware, software, and network resources; integrate and test-install test environment resources; obtain/refine test databases; and develop environment setup scripts and test bed scripts.
The hardware supporting the test environment must be sufficient to ensure complete functionality of the production application. Test environment hardware needs to be sufficient to support performance analysis. In cases where the test environment utilizes hardware resources that are also supporting other development or management activities, special arrangements may be necessary during actual performance testing. During system test, the software configuration loaded within the test environment must be a complete, fully integrated release with no patches and no disabled sections. The hardware configuration supporting the test environment needs to be designed to support processing, storage, and retrieval activities, which may be performed across a local or wide area network, reflecting the target environment.
The test environment design also needs to consider stress testing requirements. Stress and load tests may require the use of multiple workstations that will run multiple test procedures simultaneously; some automated test tools include a virtual user simulation functionality that eliminates or minimizes the need for multiple workstations.
Test data will need to be obtained with enough lead time to support refinement and manipulation to support test requirements. Data preparation activities include the identification of conversion data requirements, the preprocessing of raw data files, loading of temporary tables—possibly in a relational database management system format, and the performance of consistency checks. Identifying conversion data requirements involves performing in-depth analysis on data elements, which includes defining data-mapping criteria, clarifying data-element definitions, confirming primary keys, and defining data-acceptable parameters.
During test planning, the test team defined and scheduled the test environment activities. Now the team needs to track the test environment setup activities. Resources need to be identified to install hardware, software, and network resources into the test environment and integrate and test installed test environment resources. The test environment materials and the application under test need to be baselined within a configuration management tool. Additionally, test environment materials may include test data and test processes.
The test team needs to obtain and modify test databases necessary to exercise software applications, and develop environment setup scripts and test bed scripts. The test team should perform product reviews and validation of all test source materials. The location of the test environment for each project or task should be defined within the test plan for each project. Early identification of the test site is critical to cost-effective test environment planning and development.
Phase 5: Execution and Management of Tests
At this stage, the test team has addressed test design and test development. Test procedures are now ready to be executed in support of exercising the application under test. Also, test environment setup planning and implementation was addressed consistent with the test requirements and guidelines provided within the test plan.
With the test plan in hand and the test environment now operational, it's time to execute the tests defined for the test program. When executing test procedures, the test team must comply with a test procedure execution schedule, as discussed earlier. The test procedure execution schedule implements the strategy defined within the test plan. Plans for unit, integration, system, and user acceptance testing are executed. Together, these testing phases make up the steps that are required to test the system as a whole. The various steps involved during execution and management of tests are outlined below.
· When executing test procedures, the test team needs to comply with a test procedure execution schedule. Following test execution, test outcome evaluations are performed and test result documentation is prepared.
· Plans for unit, integration, system, and user acceptance testing are executed, which together make up the steps that are required to test the system as a whole. During the unit test phase, code profiling can be performed. Traditionally, profiling is a tuning process that determines whether an algorithm is inefficient or a function is being called too frequently. Profiling can discover instances where there is improper scaling of algorithms, instantiations, and resource utilization.
· Integration testing is performed, which focuses on the application internals. During integration testing, units are incrementally integrated and tested together based on control flow. Since units may consist of other units, some integration testing, also called module testing, may take place during unit testing.
· During system test, the test engineer is testing the integration of parts that comprise the entire system. A separate test team usually performs system-level tests. The test team implements the test procedure execution schedule and the system test plan.
· The test team also performs analysis to identify particular components or functionality that are experiencing a greater relative number of problem reports. As a result of this analysis, additional test procedures and test effort may need to be assigned to the components. Test results analysis can also confirm whether executed test procedures are proving to be worthwhile in terms of identifying errors.
· Each test team needs to perform problem-reporting operations in compliance with a defined process. The documentation and tracking of software problem reports is greatly facilitated by an automated defect-tracking tool.
· The test team manager is responsible for ensuring that tests are executed according to schedule, and test personnel are allocated and redirected when necessary to handle problems that arise during the test effort. To perform this oversight function effectively, the test manager needs to perform test program status tracking and management reporting.
· Test metrics provide the test manager with key indicators of the test coverage, progress, and the quality of the test effort. During white box testing, the test engineer measures the depth of testing, by collecting data relative to path coverage and test coverage. During black box testing, metrics collection focuses on the breadth of testing, to include the amount of demonstrated functionality and the amount of testing that has been performed.
Phase 6: Test Program Review and Assessment
Test program review and assessment activities need to be conducted throughout the testing lifecycle, to allow for continuous improvement activities. Throughout the testing lifecycle and following test execution activities, metrics need to be evaluated and final review and assessment activities need to be conducted to allow for process improvement. The various steps necessary for test program review and assessment are outlined below.
· Following test execution, the test team needs to review the performance of the test program to determine where changes can be implemented to improve the test program performance on the next project. This test program review represents the final phase of the Automated Test Lifecycle Methodology (ATLM).
· Throughout the test program, the test team collected various test metrics. The focus of the test program review includes an assessment of whether the application satisfies acceptance criteria and is ready to go into production. The review also includes an evaluation of earned value progress measurements and other metrics collected.
· As part of its culture, the test team needs to adopt an ongoing iterative process of lessons learned activities. Such a program encourages test engineers to take the responsibility to raise corrective action proposals immediately, when such actions potentially have significant impact on test program performance. Throughout the entire test lifecycle, it's good practice to document and begin to evaluate lessons learned at each milestone. The metrics that are collected throughout the test lifecycle and especially during the test execution phase help pinpoint problems that need to be addressed.
· Lessons learned, metrics evaluations, and corresponding improvement activity or corrective action need to be documented throughout the entire test process in a central repository that's easily accessible.
· After collecting lessons learned and other metrics, and defining corrective actions, test engineers also need to assess the effectiveness of the test program to include an evaluation of the test program return on investment. Test engineers capture measures of the benefits of automation realized throughout the test lifecycle in order to support this assessment.
· Test teams can perform their own surveys to inquire about the potential value of process and tool changes. A survey form can be used to solicit feedback on the potential use of requirement-management tools, design tools, and development tools. Surveys are helpful to identify potential misconceptions and gather positive feedback.
Summary
ATLM is a structured methodology geared toward ensuring successful implementation of automated testing. The ATLM approach mirrors the benefits of modern rapid application development efforts, where such efforts engage the user early in the development cycle. The end user of the software product is actively involved throughout analysis, design, development, and test of each software build, which is augmented in an incremental fashion.
Rational Corporation lists Automated Software Testing: Introduction, Management, and Performance and the ATLM as recommended reading as part of the Rational Unified Process. Many universities and professional software test training organizations have adopted the book for their classroom. Many companies (such as Imbus GMBH, Moehrendorf, Germany) have adopted the book and ATLM as their company standard for automated software testing. Others believe that industry automated test tool vendors will soon be incorporating the book's structured methodology within their tools. Instead of performing the entire test lifecycle haphazardly, software test managers will use an ATLM-compliant test tool that automatically supports (and possibly enforces) the book's sound building-block approach to the test effort.