Functional testing guidelines

#1 Introduction

This document provides detailed information about how to do functional testing of a web based software. Intended audience of this document consists of testing team, developers, project manager and stack-holders.

Please read document quality assurance guidelines before reading this document.

#2 What is Functional Testing?

Functional testing is a testing strategy, which needs little or no need of internal design or code etc. The types of testing under this strategy are totally based/focused on the testing for requirements and functionality of the work product/software application. Functionality testing (FT) is sometimes also called as Black Box testing, Opaque Testing, Behavioral Testing, Closed Box Testing etc.

The base of the FT strategy lies in the selection of appropriate data as per functionality and testing it against the functional specifications in order to check for normal and abnormal behavior of the software.

In order to implement FT Strategy, the tester is needed to be thorough with the requirement specifications of the system and as a user, should know, how the system should behave in response to the particular action.

FT works on TDD methodology, so It should be created once the testers exercised the code for completed stories (functionalities) to verify that functional requirements had been met. It should be synchronized with functionality of the project.

#2.1 Advantages

1. Tester needs little or no knowledge of implementation, including specific programming languages.
2. Tester and programmer are independent of each other.
3. Tests are done from a user's point of view.
4. Will help to expose any ambiguities or inconsistencies in the specifications.
5. Test cases can be designed as soon as the specifications are complete.

#2.2 Disadvantages

1. Only a small number of possible inputs can actually be tested, to test every possible input stream would take nearly forever.
2. Without clear and concise functional requirement specifications, test cases are hard to design.
3. There may be unnecessary repetition of test inputs if the tester is not informed of test cases the programmer has already tried.
4. May leave many program paths untested.
5. Cannot be directed toward specific segments of code which may be very complex (and therefore more error prone).

#3 Implementation

For web based software, test cases are first designed into HTML format for reporting, documentation, verification then their corresponding test cases are recorded and executed using selenium software.

Selenium is an open source tool to do automated functional tests for web-based applications. It is based on java scripting to a large extent. It’s simplicity and robustness makes it an excellent candidate for introducing automated functional testing in our project.

While Selenium-IDE operates as a Firefox add-on and provides an easy-to-use interface for developing and running individual test cases or entire test suites. It has a recording feature, which will keep account of user actions as they are performed and store them as a reusable script to play back. It also has a context menu (right-click) integrated with the Firefox browser, which allows the user to pick from a list of assertions and verifications for the selected location. Selenium-IDE also offers full editing of test cases for more precision and control.

Hence there will be 2 types of artifacts for functional test cases:

1. Designed test cases in HTML format.
2. Corresponding Selenium test case.

#4 Guidelines and principles

QA team should follow below guidelines and principles for designing, executing and documenting functional test cases.

#4.1 Designing test cases

1. Test cases are directly mapped Functional Requirement Specification. Hence each FRS should have corresponding test case.
2. Test cases should be written in enough detail that they could be given to a new team member who would be able to quickly start to carry out the tests and find defects.
3. Test cases should be first designed in HTML format as shown in following table. That is; there have to be sections; Unique ID, Purpose, URL, Prerequisites (optional), Test data, Steps, Notes/Question (optional). After that each test case will be recorded and executed using Selenium IDE.
4. The crucial issue in constructing functional test case is selection of range of input data. If range is too narrow, then the tests will be trivial and the objects might pass the tests.

#4.1.1 Test case format

UniqueTestCaseID: Test Case Title
Purpose:	Short sentence or two about the aspect of the system is being tested. If this gets too long, break the test case up or put more information into the feature descriptions.
URL:	URL where test is to be performed.
Prerequisites:	Assumptions that must be met before the test case can be run. e.g., logged in, guest login allowed, user testuser exists.
Test Data:	List of variables and their possible values used in the test case. You can list specific values or describe value ranges. The test case should be performed once for each combination of values. These values are written in set notation, one per line, like; loginID = {Valid loginID, invalid loginID, valid email, invalid email, empty} password = {valid, invalid, empty}
Steps:	Steps to carry out the test. See step formatting rules below. visit LoginPage enter userID enter password click login see the terms of use page click agree radio button at page bottom click submit button see PersonalPage verify that welcome message is correct username
Notes and Questions:	If any

Each step can be written very tersely using the following keywords:

login [as ROLE-OR-USER]

Log into the system with a given user or a user of the given type. Usually only stated explicitly when the test case depends on the permissions of a particular role or involves a workflow between different users.

visit LOCATION

Visit a page or screen. For web applications, LOCATION may be a hyperlink. The location should be a well-known starting point (e.g., the Login screen), drilling down to specific pages should be part of the test.

enter FIELD-NAME [as VALUE] [in SCREEN-LOCATION]

Fill in a named form field. VALUE can be a literal value or the name of a variable defined in the Test Data section. The FIELD-NAME itself can be a variable name when the UI field for that value is clear from context, e.g., enter password.

enter FIELDS

Fill in all fields in a form when their values are clear from context or when their specific values are not important in this test case.

click "LINK-LABEL" [in SCREEN-LOCATION]

Follow a labeled link or press a button. The screen location can be a predefined panel name or English phrase. Predefined panel names are based on GUI class names, master template names, or titles of boxes on the page.

click BUTTON-NAME [in SCREEN-LOCATION]

Press a named button. This step should always be followed by a see step to check the results.

see SCREEN-OR-PAGE

The tester should see the named GUI screen or web page. The general correctness of the page should be testable based on the feature description.

verify CONDITION

The tester should see that the condition has been satisfied. This type of step usually follows a see step at the end of the test case.

verify CONTENT [is VALUE]

The tester should see the named content on the current page, the correct values should be clear from the test data, or given explicitly. This type of step usually follows a see step at the end of the test case.

perform TEST-CASE-NAME

This is like a subroutine call. The tester should perform all the steps of the named test case and then continue on to the next step of this test case.

Every test case must include a verify step at the end so that the expected output is very clear. A test case can have multiple verify steps in the middle or at the end. Having multiple verify steps can be useful if you want a smaller number of long tests rather than a large number of short tests.

#4.2 Recording and Executing test cases

Once test cases are designed, they are be recorded and executed using Selenium. Selenium IDE is the tool to develop Selenium test cases which is an easy-to-use Firefox plug-in and is generally the most efficient way to develop test cases.

While recording and executing Selenium test cases, following checklist is to be kept in mind.

• Links

1. Check that the link takes you to the page it said it would.
2. Ensure to have no orphan pages normally (a page that has no links to it).
3. Are all referenced web sites or email addresses hyperlinked if not generated by image?
4. Check all of your links to other websites.
5. If we have removed some of the pages from our own site, set up a custom 404 page that redirects your visitors to your home page (or a search page) when the user try to access a page that no longer exists.
6. Check all mailto links and whether it reaches properly.

• Forms

1. Acceptance of invalid input.
2. Optional versus mandatory fields.
3. Input longer than field allows.
4. Default values on page load/reload.
5. Is Command Button can be used for hyperLinks and Continue Links?
6. Is all the data inside combo/list box are arranged in required order?
7. Are all of the parts of a table or form present? Correctly laid out? Can you confirm that selected texts are in the right place?
8. Does a scroll-bar appear if required? 

• Data verification and validation

1. Is the Privacy Policy clearly defined and available for user access?
2. At no point of time the system should behave awkwardly when an invalid data is fed.
3. Check to see what happens if a user deletes cookies while in site.
4. Check to see what happens if a user deletes cookies after visiting a site.

• Data integration

1. Check the maximum field lengths to ensure that there are no truncated characters?
2. If numeric fields accept negative values can these be stored correctly on the database and does it make sense for the field to accept negative numbers?
3. If a particular set of data is saved to the database check that each value gets saved fully to the database. (i.e.) Beware of truncation (of strings) and rounding of numeric values.

• Date field checks

1. Assure that leap years are validated correctly & do not cause errors/miscalculations.
2. Assure that Feb. 28, 29, 30 are validated correctly & do not cause errors/ miscalculations.

• Numeric fields

1. Assure that lowest and highest values are handled correctly.
2. Assure that numeric fields with a blank in position 1 are processed or reported as an error.
3. Assure that fields with a blank in the last position are processed or reported as an error an error.
4. Assure that both + and - values are correctly processed.
5. Assure that division by zero does not occur.
6. Include value zero in all calculations.
7. Assure that upper and lower values in ranges are handled correctly.
8. Alphanumeric field checks
9. Use blank and non-blank data.
10. Include lowest and highest values.
11. Include invalid characters & symbols.
12. Include valid characters.
13. Include data items with first position blank.
14. Include data items with last position blank.

#4.3 Organizing test cases

While test cases are successfully designed, recorded and executed they should be stored on file-system to re-run them for regression testing, documentation, demonstration etc. There have been designed certain standards and conventions for that, as shown below;

#4.3.1 HTML test cases

1. HTML test case should be named as ModuleAction.html. For example if module is User and action is Login then test case name would be UserLogin.html. Some more examples are UserAdd.html, ImageAttach.html etc.
2. These test cases are to be organized into single test suite to view them at same time. For that name TestSuite.html is to be used. A test suite document is an organized table of contents for our test cases: it simply lists the names of all test cases that we intend to write. The suite can be organized in several ways. For example, we can list all the system components, and then list test cases under each. Or, we could list major product features, and then list test cases for each of those.
3. Test suite is organized in a grid where the rows are types of business objects or module names and the columns are types of operations or actions. Each cell in the grid lists test cases that test one type of operation on one type of object. Each individual operations/action should also contain hyperlink to corresponding test case document. Format of TestSuite.html should be as follows:

Test cases by modules and actions or business objects and operations
Modules/Actions BOs/Operations	add	list/browse	edit	delete	search
User	UserAdd	UserList	UserEdit	UserDelete	UserSearch
Ad	AdAdd	AdList	AdEdit	AdDelete	AdSearch
Club	ClubAdd	ClubList	ClubEdit	ClubDelete	ClubSearch
Element	ElementAdd	ElementList	ElementEdit	ElementDelete	ElementSearch

Once all these objects are created they should be stored at location; tests/functionaltests/html/ location for concerned project. Other artifacts like documents, instructions etc. also should be put at same location.

Since these objects are for internal purpose; they may not be publicly visible. However separate/sub domain can be created to view them from web if needed.

#4.3.2 Selenium test cases

1. Selenium test case should also be named as ModuleAction.html. For example if module is User and action is Login then test case name would be UserLogin.html. Some more examples are UserAdd.html, ImageAttach.html etc.
2. These test cases are to be organized into single test suite HTML file to re-run them at same time. For that name TestSuite.html is to be used.
3. Once all these objects are created they should be stored at location; tests/functionaltests/selenium/ location for concerned project. Other artifacts like documents, instructions etc. also should be put at same location.
4. Since these objects are for internal purpose; they may not be publicly visible. However separate/sub domain can be created to re-run them from web on other servers if needed.

#4.4 Scope of testing

1. Minimum scope of overall testing procedure is to test all features in all popular browsers and platforms.
2. Stress testing should be performed (try to overload the program with inputs to see where it reaches its maximum capacity), especially with real time systems.
3. Crash testing should be performed to see what it takes to bring the system down.
4. Other scope can be defined according to FRS when test plan is defined.

#4.5 What test should be written

While doing functional testing, it is important to know that what kind of tests should be done. There has been provided some insight about type of some tests.

1. All positive tests: This set of tests ensures that everything works as expected.
2. All failure tests: Use these tests on a one-by-one basis to ensure that every failure or exception case works.
3. Positive sequence tests: This set of tests ensures that calls in the correct order work as expected.
4. Negative sequence tests:This set of tests ensures that when calls are made out of order, they fail.
5. Load tests: When appropriate, you can perform a small set of tests to determine that the performance of those tests is within an expected range. For example; while uploading image, how large file software is able to handle.

First 4 types of tests are must for every test case. Rest can be implemented according to FRS.

#4.6 Functional testing in the context of team

Functional testing team is also part of development team, hence they should be included in every meeting, discussion, planning etc. to make them aware about functionalities are to be developed.

Testing strategy described here is to be performed by dedicated testers only. However standard practice is that developers do functional testing immediately after they develop required functionality. Then testing related to that functionality can be given to dedicated testers to test it from user point of view.

FT team may not always have daily tasks of testing, hence in spare time they can also assist developers in developing new functionality if they wish.

#5 Links

http://www.w3.org/TR/2004/WD-qaframe-test-20040225/
http://seleniumhq.org/
http://seleniumhq.org/projects/ide/