Just another blog

Quality Assurance guidelines

2010-02-01T13:01:00.000+05:30

#1 Introduction

This document provides detailed information about Quality Assurance plan for web based software. Intended audience of this document consists of developers, testing team, project manager, stack-holders and clients.

#2 Quality objectives

To achieve high quality of code and functionality to it's customers, team can design a detailed QA plan to be followed throughout development life cycle and post-development life span. To achieve this aim, following objectives should be decided by development team.

No untested code should be shipped to clients.
Test criteria must cover at least 75% of code for testing procedure.
Whatever has been tested is documented for various purposes.
Each functionality of the software must be tested according to requirement specifications and must run in most popular browsers and platforms in similar way.

#3 Management

#3.1 Team

Quality Assurance team primarily involves Test manager, Developers and Dedicated & Ad-hoc testers. It directly reports to Project manager and/or Development manager.

#3.2 Tasks and Responsibilities

QA team performs following tasks.

Doing unit testing;
Doing functionality testing;
Doing usability testing;
Managing testing documentation;
Managing testing guidelines;
Reviewing test cases;
Doing continuous testing for core code;
Providing training to new members;

Breakdown of above tasks by team members is as follows;

Developers do unit testing of code and make sure that it doesn't break even after making changes in it for new/updated features;
Dedicated and Ad-hoc testers then test those newly added/updated features in various browsers and tools to make sure that features behave according to requirement specifications;
Test manager checks created unit & functional tests and re-run them as and when needed. He also writes complex test cases to test code and features from various aspects like stress, load bearing, sanity etc.
Test manager is also responsible for preparing testing plans and guidelines for existing and new testers. Test manager also manages documentation of test cases and suits in repository for proof of quality control and easier access.
Development and Test manager together work for running unit tests continuously on test server for consistent quality check.
Training to new tester is usually provided by Test manager as and when needed during development life cycle.

Details about each task has been provided in following sections.

#4 Testing overview

Software testing is the process used to help identify the correctness, completeness, security, and quality of developed computer software. Testing is a process of technical investigation, performed on behalf of stakeholders, that is intended to reveal quality-related information about the product with respect to the context in which it is intended to operate.

#4.1 Unit testing

When it comes to source code, unit testing of code is performed in order to be sure that there is no problem at code level. In unit testing, unit is known as a part of code which can be a function, a method or a whole class. In short, unit testing tests code of software into various pieces. Hence successful unit testing shows that software is now ready for functional testing.

PHPUnit is a member of the xUnit family of testing frameworks and provides both a framework that makes the writing of tests easy as well as the functionality to easily run the tests and analyse their results.

#4.2 Functionality testing

Purpose of functionality testing is to test whole system (or software sometimes) using automatic and manual tools. Whole system may include PHP code, JS code, HTML etc. In this type of testing, various components of software are tested together. Software like selenium is used to perform functionality testing.

Selenium IDE is an integrated development environment for Selenium tests. It is implemented as a Firefox extension, and allows you to record, edit, and debug tests.

#4.3 User (Usability) testing

Usability testing is for measuring how well people can use some human-made object (such as a web page, a computer interface, a document, or a device) for its intended purpose, i.e. usability testing measures the usability of the object. Usability testing focuses on a particular object or a small set of objects, whereas general human-computer interaction studies attempt to formulate universal principles. Software like ab and mysqlprofile can be used to perform usability testing.

#5 Deliverables

QA deliverables include testing guidelines, manual and automated test cases, statistical and coverage reports and overall documentation related to QA. These are well organized and kept in version control for concerned project's application.

#6 Testing Guidelines

Testing guidelines and goals help achieve highest standards in quality control. They have been prepared to assist developers and testers to write effective test cases and to document them for various purposes.

#6.1 Unit testing

To read unit testing related guidelines and principles, read section #4 in this document.

#6.2 Functionality testing

To read functional testing related guidelines and principles, read section #4 in this document.

#7 Review process

Review of test cases is checking test cases by test colleagues and/or test manager in order to make sure that those are correct or any misunderstandings in the requirements or anything missed out or not. Since review is a continuous process, test manager regularly checks previously designed test cases and re-run them whenever needed.

Review process comprises following things:
Completeness- Do test cases contain enough conditions of code and features to be tested?
Accuracy- Are test cases accurately defined and logical? Are there missing elements?
Testable- Are test cases testable?

#8 Continuous testing

Continuous testing uses excess cycles on a workstation to continuously run regression tests in the background, providing rapid feedback about test failures as source code is edited. It reduces the time and energy required to keep code well-tested, and prevents regression errors from persisting uncaught for long periods of time.

For more information about continuous testing, please refer this document.

#9 Training

Training is important part of any department during software development life cycle. Test manager regularly provides training to QA team to improve testing procedure and to make it as effective as possible.

#10 References

http://phing.info/trac/

PHP project build system

2009-07-17T13:01:00.000+05:30

#1 Pre-requisites

Before reading this document, it is required to read unit testing guidelines , which is part of this package.

#2 PHPUnit, CVS and Phing

For a typical web based software, our main requirements are to automate several routine tasks like unit testing, updating latest data from version control, loading packages on different servers, cleaning up log files and performing other daily routine task to save manual work.

Hence here we will have brief look at how Phing can be used to automate some of the above mentioned tasks. But before that let's see what Phing is and how it works.

#2.1 What is Phing

Phing (PHing Is Not GNU make) is a project build system based on Apache Ant. In the context of PHP, where it is not required to build and compile sources, the intention of Phing is to ease the packaging, deployment, and testing of applications. For these tasks, Phing provides numerous out-of-the-box operation modules ("tasks") and an easy-to-use, object-oriented model for adding our own custom tasks.
Phing can be installed using the PEAR Installer, as shown in the following command line:

$ pear channel-discover pear.phing.info

$ pear install phing/phing

For other modes of installation and more details read Phing documentation.
Note: Please do not forget to install other dependencies also that Phing might ask during it's installation.

#2.2 How it works

Phing uses XML buildfiles that contain a description of the things to do. The buildfile is structured into targets (groups of task/s) that contain the actual commands to perform (e.g. commands to copy a file, delete a directory, perform a DB query, etc.). So, to use Phing, we would first write our buildfile and then run phing specifying the target in buildfile that we want to execute.

$ phing -f [mybuildfile.xml] [mytarget]

By default Phing will look for a buildfile named build.xml (so you don't have to specify the buildfile name unless it is not build.xml) and if no target is specified Phing will try to execute the default target, as specified in the tag.

A valid Phing buildfile has the following basic structure:

The document prolog,
Exactly one root element called ,
Several Phing type elements (i.e. , , etc.),
One or more elements containing built-in or user defined Phing task elements (i.e. , , etc).

It is beyond the scope of this document, to explain above structure. Hence please read phing documentation.

#2.3 Phing and automated testing

Phing has several built in tasks to perform various operations. Some of them, for our purpose, are exec, delete, mkdir, coverage-setup, phpunit, phpunitreport and coverage-report.

exec task can be used to execute any command of OS environment and applications;
delete can be used to remove files/folders;
mkdir to make new directories;
phpunit to run unit tests created by developers (as described in this document);
phpunitreport to generate report about unit tests;
coverage-setup and coverage-report tasks are for providing detailed report of code-coverage analysis.

#2.5 Agile documentation

Continuous testing using Phing is performed everyday on test server and 2 types of documentation is generated at following locations for every project;

tests/unittests/reports/index.html and
tests/unittests/reports/coverage/index.html (for reporting purpose)

Since this document is generated anytime on the fly and gets updated after every run, it doesn't require to add them into version control.

#2.6 Guidelines about using Phing

Phing has vast application areas hence in beginning it is used for limited tasks only like updating local repository from CVS, running unit tests, code analysis, packaging and unpacking files etc. Later it will be expanded to handle more complex tasks.
Build management system such as Phing, should not be used by individual developers as it affects whole system. Hence it's care is taken by dedicated test/project/development manager.
Phing can be extended to create custom tasks also. But that requires extensive study of it's existing functionality.

#3 Links

http://phing.info/docs/guide/current/
http://www.phpunit.de/manual/3.3/en/build-automation.html#build-automation.phing

Unit testing guidelines

2009-05-26T13:02:00.000+05:30

#1 Introduction

This document provides detailed information about how to unit test code of PHP based software. Intended audience of this document consists of developers, testing team, project manager and stack-holders.

#2 What is unit testing

Unit testing is a software verification and validation method in which a programmer tests that individual units of source code are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit may be an individual program, function, procedure, etc., while in object-oriented programming, the smallest unit is a class, which may belong to a base/super class, abstract class or derived/child class.

The goal of unit testing is to isolate each part of the program and to show that the individual parts are correct. Unit testing provides a strict, written contract that the piece of code must satisfy. As a result, it affords several benefits. In this article we will follow PHPUnit as a unit testing software.

#2.1 Advantages of unit testing

#2.1.1 Facilitates change

Unit testing allows the programmer to refactor code at a later date, and make sure the module still works correctly (i.e. regression testing). This provides the benefit of encouraging programmers to make changes to the code since it is easy for the programmer to check if the piece is still working properly. Good unit test design produces test cases that cover all paths through the unit with attention paid to loop conditions. In continuous unit testing environments, through the inherent practice of sustained maintenance, unit tests will continue to accurately reflect the intended use of the executable and code in the face of any change. Depending upon established development practices and unit test coverage, up-to-the-second accuracy can be maintained.

#2.1.2 Simplifies integration

Unit testing helps eliminate uncertainty in the units themselves and can be used in a bottom-up testing style approach. By testing the parts of a program first and then testing the sum of its parts, integration testing becomes much easier.

A heavily debated matter exists in assessing the need to perform manual integration testing. While an elaborate hierarchy of unit tests may seem to have achieved integration testing, this presents a false sense of confidence since integration testing evaluates many other objectives that can only be proved through the human factor. Some argue that given a sufficient variety of test automation systems, integration testing by a human test group is unnecessary. Realistically, the actual need will ultimately depend upon the characteristics of the product being developed and its intended uses.

#2.1.3 Documentation

Unit testing provides a sort of "living document". Clients and other developers looking to learn how to use the module can look at the unit tests to determine how to use the module to fit their needs and gain a basic understanding of the API.

Unit test cases embody characteristics that are critical to the success of the unit. These characteristics can indicate appropriate/inappropriate use of a unit as well as negative behaviors that are to be trapped by the unit. A unit test case, in and of itself, documents these critical characteristics, although many software development environments do not rely solely upon code to document the product in development.

Ordinary documentation, on the other hand, is more susceptible to drifting from the implementation of the program and will thus become outdated (e.g. design changes, feature creep, relaxed practices to keep documents up to date).

#2.1.4 Separation of interface from implementation

Because some classes may have references to other classes, testing a class can frequently spill over into testing another class. A common example of this is classes that depend on a database: in order to test the class, the tester often writes code that interacts with the database. This is a mistake, because a unit test should never go outside of its own class boundary. As a result, the software developer abstracts an interface around the database connection, and then implements that interface with their own mock object. This results in loosely coupled code, minimizing dependencies in the system.

#2.2 Limitations of unit testing

Unit testing will not catch every error in the program. By definition, it only tests the functionality of the units themselves. Therefore, it will not catch integration errors, performance problems or any other system-wide issues. In addition, it may not be easy to anticipate all special cases of input the program unit under study may receive in reality. Unit testing is only effective if it is used in conjunction with other software testing activities.

It is unrealistic to test all possible input combinations for any non-trivial piece of software. A unit test can only show the presence of errors; it cannot show the absence of errors. Though these two limitations apply to any form of software test.

#3 PHPUnit

PHPUnit is an excellent unit testing software for PHP programming language based softwares, which is derived from JUnit software (in Java technology). To learn PHPUnit, well understanding of OOP is must.

Testing with PHPUnit is not a totally different activity from what you should already be doing. It is just a different way of doing it. The difference is between testing, that is, checking that your program behaves as expected, and performing a battery of tests, runnable code-fragments that automatically test the correctness of parts (units) of the software. These runnable code-fragments are called unit tests.

It is to be clearly kept in mind that unit testing should be performed as soon as coding is finished, and not after days and month. When software's code gets changed, corresponding tests should also get changed and tested.

#3.1 How PHPUnit works

PHPUnit API has several built in methods to perform various kind of testing on your given code. Developer first write certain code according to requirement of software. Then he will need to create certain test cases (or suits) using PHPUnit's built in classes and finally to test those test cases (or suits) to be sure that designed code works properly.

Test cases (or suits) are nothing but PHP scripts that test code of software using PHPUnit's built in tests. Hence there are 3 components in unit testing:
Software's code
PHPUnit's built in tests and
Test cases (or suits).

#3.2 Installing PHPUnit

PHPUnit should be installed using the PEAR Installer. This installer is the backbone of PEAR, which provides a distribution system for PHP packages, and is shipped with every release of PHP since version 4.3.0.

The PEAR channel (pear.phpunit.de) that is used to distribute PHPUnit needs to be registered with the local PEAR environment:

$ pear channel-discover pear.phpunit.de

This has to be done only once. Now the PEAR Installer can be used to install packages from the PHPUnit channel:

# pear install phpunit/PHPUnit

After the installation you can find the PHPUnit source files inside your local PEAR directory; the path is usually /usr/lib/php/PHPUnit.

Although using the PEAR Installer is the only supported way to install PHPUnit, you can install PHPUnit manually. For manual installation, do the following:

Download a release archive from http://pear.phpunit.de/get/ and extract it to a directory that is listed in the include_path of your php.ini configuration file.

Prepare the phpunit script:

Rename the phpunit.php script to phpunit.
Replace the @php_bin@ string in it with the path to your PHP command-line interpreter (usually /usr/bin/php).
Copy it to a directory that is in your path and make it executable (chmod +x phpunit).

Prepare the PHPUnit/Util/Fileloader.php script:

Replace the @php_bin@ string in it with the path to your PHP command-line interpreter (usually /usr/bin/php).

After successful installation of PHPUnit, you can use command phpunit /PATH/TO/TESTSCRIPT to run your tests. For all the available options of this command type phpunit -–help.

#3.3 How to learn and use PHPUnit

Road map of learning and using PHPUnit has been shown below.

Get good understanding of OOP.
Read whole documentation of PHPUnit software to understand what it is and how it works.
Install PHPUnit to build up testing environment.
Study code of all scripts of PHPUnit software even if at first time, you don't understand much.
Having look at the testing examples provided by PHPUnit, understand how unit testing works.
Now try to build your own test cases of software on which you are working.
After having enough experience of building test cases, try to build test suit that can perform all test at once (which is the goal of the unit testing).
Try to build more complex and fully automated test suits using PHPUnit's extensions and other utilities or by building your own custom extensions according to requirements of your software.

To create test cases in your live projects, you may require to reorganize your classes into packages and sub packages. For test cases, you may create new folder test/unittest in your project's main directory and then create test cases (scripts) according to packages and sub packages defined above.

#4 Guidelines and principles

Here are guidelines and principles of implementing unit testing to be followed by developer tram.

#4.1 How to write test cases

Best way to write test cases is to study an example given here and some more from PHPUnit documentation.
1st part of test script contains code to include necessary files/fixtures, while 2nd part contain test case class.
In beginning develop simple tests for single method or collection of methods. After having certain experience, build more complex test cases. Finally go to suite level which provides fully automated testing environment.
Create readable tests - write comments in Asserts and in UT. Write descriptive method names (even very long). Use local variables in Asserts. Use constants. UT should be readable like a book.
Test cases must cover at least 75% of code. However this will be shown in reports generated by Phing when such tests are run from there. The danger of not implementing a unit test on every method is that the coverage may be incomplete. Just because we don't test every method explicitly doesn't mean that methods can get away with not being tested.
You may require configuration parameters for database, paths and locations of other dependent software to run unit tests. For that there has been created a common file tests/unittests/config.inc.php which would be included in your test cases. Since it is common file or all tests, it is normally managed test/build manager. Hence if it is required to update that file, test manager must be consulted.
Moreover this file should be included in your test cases as include_once or require_once constructs only otherwise there will be warnings/errors of duplication of code while running all tests together as suite from build system.
If you find difficulties in creating/running tests cases for any of your class method etc., it means it requires regression that is to re-organize structure of class/class package to be able to test them in proper way.
Since private methods of class can not be tested directly, it's corresponding public method should be tested. However every private method always has a parent public method which uses that private method. For more information visit Links sections.
The concepts that you may need to be consider while building test cases (or suits) are Test-first programming, Code-coverage analysis, Refactoring, Incomplete tests, Agile documentation, Debugging Tests etc.

#4.2 How to organize and run them

Developers should maintain hierarchy of classes as well as their equivalent test cases in same way. So for example if class file is grouped as Validate/Numeric.php then it's corresponding test case should also be grouped as Validate/NumericTest.php inside folder tests/unittests. Moreover name of test case class should also follow name of class that is to be tested. Same is true for phpDocumentator tags in test script. That is; tests scripts must be maintained as any other code- like Keep It Simple, Stupid (KISS)
Fixtures (such as images, data files etc.) of test cases should be kept in same folder where test case/s reside/s. However for larger contents, they can be put in separate folder.
All test cases are to be run from project's base directory only as shown in earlier example. This is necessary because it is required to generate agile documentation of unit tests cases and their classes.

#4.3 Testing instructions

Test cases should not be designed in such a way that they require specific order of execution; that is they must run independent of other implementation.
Test cases like to test Mail system, Database, File-system etc. requires additional setup apart from test script. Instructions about such setup and dependencies should clearly be mentioned in text file (like README.txt) inside corresponding test case's package. Such instructions can also contain information about how to run tests cases, other prerequisites, dependencies etc. wherever applicable.
While overall instructions are to be kept at tests/unittests/README.txt file only.

#4.4 Scope of unit testing

The crucial issue in constructing a unit test is scope. If the scope is too narrow, then the tests will be trivial and the objects might pass the tests, but there will be no design of their interactions.
Likewise, if the scope is too broad, then there is a high chance that not every component of the new code will get tested. The programmer is then reduced to testing-by-poking-around, which is not an effective test strategy.
It is recommended to write test cases for logical part only i.e static code/data need not to get tested. For complex test cases such as for Database, Mailing etc., it may require certain environmental settings. For that, concerned higher positioned person can be contacted for availability of it.

Bottom line of unit testing is that: it all depends upon the test case that you create. The better the cases, that covers all possible areas to be tested, the more worth the test is. If your test is poor, you wont yield advantages of unit testing.

#4.5 What tests should be written

While doing unit testing, it is important to know that what kind of tests should be done. There has been provided some insight about type of some tests.

All positive test: This set of tests ensures that everything works as expected.
All failure test: Use these tests on a one-by-one basis to ensure that every failure or exception case works.
Positive sequence tests: This set of tests ensures that calls in the correct order work as expected.
Negative sequence tests: This set of tests ensures that when calls are made out of order, they fail.
Load tests: When appropriate, you can perform a small set of tests to determine that the performance of those tests is within an expected range. For example; 500 mails should be sent within 10-15 seconds.
Resource tests: These tests ensure that the application program interface (API) properly allocates and frees resources-- for example, opening, writing, and closing a file-based API several times in a row to ensure that no files remain open.
Callback tests: For APIs that have callback methods, these tests ensure that the code runs properly if callbacks are not defined. In addition, these tests ensure that the code runs properly when callbacks are defined but behave inappropriately or generate exceptions.

First 4 types of tests are must for every test-case. Rest can be implemented according to requirements in code.

#4.6 Unit testing in the context of team

Developers should do unit testing of their developed libraries and dependent libraries only, i.e they don't require to do unit testing of whole library since other code might have other requirement/dependency about which they are not aware of. That means test-suite should be run by Test/Project/Development manager only.
Above positioned persons can use build tools like phing to automate unit testing process, whenever there are changes in code, without manually running every time. Procedure of using Phing in unit testing and other tasks has been mentioned in this document.
No untested code should be committed to version control, that is each package should have corresponding test cases.

#4.7 Re-usability of test cases/data

Although it may seem like a good idea to throw random data at an interface, try to avoid it because the data is hard to debug. If data is generated randomly on each invocation, you may get an error on one pass that you don't get on another. If your test requires random data, generate the data in a file, then use that file on every run. In this way, you can have noisy data, but still be able to debug errors.
In unit tests since each test case is tightly attached to particular piece of code, it is difficult to write re-usable test cases. However whenever possible, test cases/data should be made re-usable and hence should be kept in test library in order to re-use them for other projects.
To create such library, Version control system can be used to store them in organized way under namespace like unit_test_library. And underneath that, there can be created several pages to describe nature of test case/data and how to use it in other projects. Similar practice can be used for functional test cases.

#5 Links

http://www.phpunit.de/manual/3.3/en/index.html
http://www.acm.org/ubiquity/views/t_burns_1.html
http://www.simplifytheweb.com/files/presentation.ppt
http://readyset.tigris.org/nonav/templates/test-suite.html
http://stackoverflow.com/questions/34571/whats-the-best-way-of-unit-testing-private-methods
http://stackoverflow.com/questions/643898/mocking-methods-used-in-static-methods-phpunit
http://stackoverflow.com/questions/243274/best-practice-unit-testing-abstract-classes
http://www.artima.com/weblogs/viewpost.jsp?thread=203994

Functional testing guidelines

2009-02-05T13:00:00.000+05:30

#1 Introduction

This document provides detailed information about how to do functional testing of a web based software. Intended audience of this document consists of testing team, developers, project manager and stack-holders.

Please read document quality assurance guidelines before reading this document.

#2 What is Functional Testing?

Functional testing is a testing strategy, which needs little or no need of internal design or code etc. The types of testing under this strategy are totally based/focused on the testing for requirements and functionality of the work product/software application. Functionality testing (FT) is sometimes also called as Black Box testing, Opaque Testing, Behavioral Testing, Closed Box Testing etc.

The base of the FT strategy lies in the selection of appropriate data as per functionality and testing it against the functional specifications in order to check for normal and abnormal behavior of the software.

In order to implement FT Strategy, the tester is needed to be thorough with the requirement specifications of the system and as a user, should know, how the system should behave in response to the particular action.

FT works on TDD methodology, so It should be created once the testers exercised the code for completed stories (functionalities) to verify that functional requirements had been met. It should be synchronized with functionality of the project.

#2.1 Advantages

Tester needs little or no knowledge of implementation, including specific programming languages.
Tester and programmer are independent of each other.
Tests are done from a user's point of view.
Will help to expose any ambiguities or inconsistencies in the specifications.
Test cases can be designed as soon as the specifications are complete.

#2.2 Disadvantages

Only a small number of possible inputs can actually be tested, to test every possible input stream would take nearly forever.
Without clear and concise functional requirement specifications, test cases are hard to design.
There may be unnecessary repetition of test inputs if the tester is not informed of test cases the programmer has already tried.
May leave many program paths untested.
Cannot be directed toward specific segments of code which may be very complex (and therefore more error prone).

#3 Implementation

For web based software, test cases are first designed into HTML format for reporting, documentation, verification then their corresponding test cases are recorded and executed using selenium software.

Selenium is an open source tool to do automated functional tests for web-based applications. It is based on java scripting to a large extent. It’s simplicity and robustness makes it an excellent candidate for introducing automated functional testing in our project.

While Selenium-IDE operates as a Firefox add-on and provides an easy-to-use interface for developing and running individual test cases or entire test suites. It has a recording feature, which will keep account of user actions as they are performed and store them as a reusable script to play back. It also has a context menu (right-click) integrated with the Firefox browser, which allows the user to pick from a list of assertions and verifications for the selected location. Selenium-IDE also offers full editing of test cases for more precision and control.

Hence there will be 2 types of artifacts for functional test cases:

Designed test cases in HTML format.
Corresponding Selenium test case.

#4 Guidelines and principles

QA team should follow below guidelines and principles for designing, executing and documenting functional test cases.

#4.1 Designing test cases

Test cases are directly mapped Functional Requirement Specification. Hence each FRS should have corresponding test case.
Test cases should be written in enough detail that they could be given to a new team member who would be able to quickly start to carry out the tests and find defects.
Test cases should be first designed in HTML format as shown in following table. That is; there have to be sections; Unique ID, Purpose, URL, Prerequisites (optional), Test data, Steps, Notes/Question (optional). After that each test case will be recorded and executed using Selenium IDE.
The crucial issue in constructing functional test case is selection of range of input data. If range is too narrow, then the tests will be trivial and the objects might pass the tests.

#4.1.1 Test case format

UniqueTestCaseID: Test Case Title
Purpose:	Short sentence or two about the aspect of the system is being tested. If this gets too long, break the test case up or put more information into the feature descriptions.
URL:	URL where test is to be performed.
Prerequisites:	Assumptions that must be met before the test case can be run. e.g., logged in, guest login allowed, user testuser exists.
Test Data:	List of variables and their possible values used in the test case. You can list specific values or describe value ranges. The test case should be performed once for each combination of values. These values are written in set notation, one per line, like; loginID = {Valid loginID, invalid loginID, valid email, invalid email, empty} password = {valid, invalid, empty}
Steps:	Steps to carry out the test. See step formatting rules below. visit LoginPage enter userID enter password click login see the terms of use page click agree radio button at page bottom click submit button see PersonalPage verify that welcome message is correct username
Notes and Questions:	If any

Each step can be written very tersely using the following keywords:

login [as ROLE-OR-USER]

Log into the system with a given user or a user of the given type. Usually only stated explicitly when the test case depends on the permissions of a particular role or involves a workflow between different users.

visit LOCATION

Visit a page or screen. For web applications, LOCATION may be a hyperlink. The location should be a well-known starting point (e.g., the Login screen), drilling down to specific pages should be part of the test.

enter FIELD-NAME [as VALUE] [in SCREEN-LOCATION]

Fill in a named form field. VALUE can be a literal value or the name of a variable defined in the Test Data section. The FIELD-NAME itself can be a variable name when the UI field for that value is clear from context, e.g., enter password.

enter FIELDS

Fill in all fields in a form when their values are clear from context or when their specific values are not important in this test case.

click "LINK-LABEL" [in SCREEN-LOCATION]

Follow a labeled link or press a button. The screen location can be a predefined panel name or English phrase. Predefined panel names are based on GUI class names, master template names, or titles of boxes on the page.

click BUTTON-NAME [in SCREEN-LOCATION]

Press a named button. This step should always be followed by a see step to check the results.

see SCREEN-OR-PAGE

The tester should see the named GUI screen or web page. The general correctness of the page should be testable based on the feature description.

verify CONDITION

The tester should see that the condition has been satisfied. This type of step usually follows a see step at the end of the test case.

verify CONTENT [is VALUE]

The tester should see the named content on the current page, the correct values should be clear from the test data, or given explicitly. This type of step usually follows a see step at the end of the test case.

perform TEST-CASE-NAME

This is like a subroutine call. The tester should perform all the steps of the named test case and then continue on to the next step of this test case.

Every test case must include a verify step at the end so that the expected output is very clear. A test case can have multiple verify steps in the middle or at the end. Having multiple verify steps can be useful if you want a smaller number of long tests rather than a large number of short tests.

#4.2 Recording and Executing test cases

Once test cases are designed, they are be recorded and executed using Selenium. Selenium IDE is the tool to develop Selenium test cases which is an easy-to-use Firefox plug-in and is generally the most efficient way to develop test cases.

While recording and executing Selenium test cases, following checklist is to be kept in mind.

Links

Check that the link takes you to the page it said it would.
Ensure to have no orphan pages normally (a page that has no links to it).
Are all referenced web sites or email addresses hyperlinked if not generated by image?
Check all of your links to other websites.
If we have removed some of the pages from our own site, set up a custom 404 page that redirects your visitors to your home page (or a search page) when the user try to access a page that no longer exists.
Check all mailto links and whether it reaches properly.

Forms

Acceptance of invalid input.
Optional versus mandatory fields.
Input longer than field allows.
Default values on page load/reload.
Is Command Button can be used for hyperLinks and Continue Links?
Is all the data inside combo/list box are arranged in required order?
Are all of the parts of a table or form present? Correctly laid out? Can you confirm that selected texts are in the right place?
Does a scroll-bar appear if required?

Data verification and validation

Is the Privacy Policy clearly defined and available for user access?
At no point of time the system should behave awkwardly when an invalid data is fed.
Check to see what happens if a user deletes cookies while in site.
Check to see what happens if a user deletes cookies after visiting a site.

Data integration

Check the maximum field lengths to ensure that there are no truncated characters?
If numeric fields accept negative values can these be stored correctly on the database and does it make sense for the field to accept negative numbers?
If a particular set of data is saved to the database check that each value gets saved fully to the database. (i.e.) Beware of truncation (of strings) and rounding of numeric values.

Date field checks

Assure that leap years are validated correctly & do not cause errors/miscalculations.
Assure that Feb. 28, 29, 30 are validated correctly & do not cause errors/ miscalculations.

Numeric fields

Assure that lowest and highest values are handled correctly.
Assure that numeric fields with a blank in position 1 are processed or reported as an error.
Assure that fields with a blank in the last position are processed or reported as an error an error.
Assure that both + and - values are correctly processed.
Assure that division by zero does not occur.
Include value zero in all calculations.
Assure that upper and lower values in ranges are handled correctly.
Alphanumeric field checks
Use blank and non-blank data.
Include lowest and highest values.
Include invalid characters & symbols.
Include valid characters.
Include data items with first position blank.
Include data items with last position blank.

#4.3 Organizing test cases

While test cases are successfully designed, recorded and executed they should be stored on file-system to re-run them for regression testing, documentation, demonstration etc. There have been designed certain standards and conventions for that, as shown below;

#4.3.1 HTML test cases

HTML test case should be named as ModuleAction.html. For example if module is User and action is Login then test case name would be UserLogin.html. Some more examples are UserAdd.html, ImageAttach.html etc.
These test cases are to be organized into single test suite to view them at same time. For that name TestSuite.html is to be used. A test suite document is an organized table of contents for our test cases: it simply lists the names of all test cases that we intend to write. The suite can be organized in several ways. For example, we can list all the system components, and then list test cases under each. Or, we could list major product features, and then list test cases for each of those.
Test suite is organized in a grid where the rows are types of business objects or module names and the columns are types of operations or actions. Each cell in the grid lists test cases that test one type of operation on one type of object. Each individual operations/action should also contain hyperlink to corresponding test case document. Format of TestSuite.html should be as follows:

Test cases by modules and actions or business objects and operations
Modules/Actions BOs/Operations	add	list/browse	edit	delete	search
User	UserAdd	UserList	UserEdit	UserDelete	UserSearch
Ad	AdAdd	AdList	AdEdit	AdDelete	AdSearch
Club	ClubAdd	ClubList	ClubEdit	ClubDelete	ClubSearch
Element	ElementAdd	ElementList	ElementEdit	ElementDelete	ElementSearch

Once all these objects are created they should be stored at location; tests/functionaltests/html/ location for concerned project. Other artifacts like documents, instructions etc. also should be put at same location.

Since these objects are for internal purpose; they may not be publicly visible. However separate/sub domain can be created to view them from web if needed.

#4.3.2 Selenium test cases

Selenium test case should also be named as ModuleAction.html. For example if module is User and action is Login then test case name would be UserLogin.html. Some more examples are UserAdd.html, ImageAttach.html etc.
These test cases are to be organized into single test suite HTML file to re-run them at same time. For that name TestSuite.html is to be used.
Once all these objects are created they should be stored at location; tests/functionaltests/selenium/ location for concerned project. Other artifacts like documents, instructions etc. also should be put at same location.
Since these objects are for internal purpose; they may not be publicly visible. However separate/sub domain can be created to re-run them from web on other servers if needed.

#4.4 Scope of testing

Minimum scope of overall testing procedure is to test all features in all popular browsers and platforms.
Stress testing should be performed (try to overload the program with inputs to see where it reaches its maximum capacity), especially with real time systems.
Crash testing should be performed to see what it takes to bring the system down.
Other scope can be defined according to FRS when test plan is defined.

#4.5 What test should be written

While doing functional testing, it is important to know that what kind of tests should be done. There has been provided some insight about type of some tests.

All positive tests: This set of tests ensures that everything works as expected.
All failure tests: Use these tests on a one-by-one basis to ensure that every failure or exception case works.
Positive sequence tests: This set of tests ensures that calls in the correct order work as expected.
Negative sequence tests:This set of tests ensures that when calls are made out of order, they fail.
Load tests: When appropriate, you can perform a small set of tests to determine that the performance of those tests is within an expected range. For example; while uploading image, how large file software is able to handle.

First 4 types of tests are must for every test case. Rest can be implemented according to FRS.

#4.6 Functional testing in the context of team

Functional testing team is also part of development team, hence they should be included in every meeting, discussion, planning etc. to make them aware about functionalities are to be developed.

Testing strategy described here is to be performed by dedicated testers only. However standard practice is that developers do functional testing immediately after they develop required functionality. Then testing related to that functionality can be given to dedicated testers to test it from user point of view.

FT team may not always have daily tasks of testing, hence in spare time they can also assist developers in developing new functionality if they wish.

#5 Links

http://www.w3.org/TR/2004/WD-qaframe-test-20040225/
http://seleniumhq.org/
http://seleniumhq.org/projects/ide/

SEO guidelines

2008-09-09T16:07:00.000+05:30

#1 About this document

This document aims to defines generic search engine optimization requirements for various projects.

At this moment this document contains general guidelines of SEO. In future, at the time of taking training session, this document will be expanded further in order to be used as perfect resource for almost all SEO requirements.

#2 General requirements

#2.1 Server location

The server should be located in same country from where it will be mostly accessed. Moreover If the service will have it’s own domain then it should reside on a dedicated server. Wildcard DNS should not be allowed as well as all sub domains, if any, should be activated separately.

#2.2 Robots.txt

The robots.txt file is to be placed in the root (value of DocumentRoot directive in case web server is Apache) directory of the software. It should allow the search engines to crawl all directories where information related to various entities will be shown.

Personal pages such as listing owner's entities, posting/editing entities that require login should be blocked. Most search engines now a days are able to find out this behavior hence you may omit such entries into robots.txt file.

#2.3 Encoding

If there are will be used special characters in language of website they will need to get encoded in URLs (maybe using PHP function like urlencode()) and Filenames using UTF-8 encoding. For full documentation of encoding of such characters, please visit http://www1.tip.nl/~t876506/utf8tbl.html). As practically all browsers supports the Unicode UTF-8 standard, it should not be important to encode the characters in the actual content. The suitable HTML entities can be taken from this address: http://leftlogic.com/lounge/articles/entity-lookup/ anyhow.

There should be a 301 redirect from any page with special characters in the URL where someone writes the URL using the special characters and not the encoded ones if that user has a browser that over writes the UTF-8 character set with some other character set. See how Wikipedia functions for an example. This prevents links with the wrong character set to be used on external pages.

#2.4 Header responses

#2.4.1 Page not found (404 error)

Entities that are removed from database/software should not be shown. When someone accesses the removed listings page the server should respond with a 404 header response (and not a 200 response) and show an error message (or optionally a separate page) saying that the entity is already deleted/expired/sold etc. Furthermore the relevant listing page should be shown.

#2.4.2 Redirects (301 error)

As a general rule of thumb all redirects should be done using the 301 permanently moved response. All sub domains should be redirected this way (example.com -> 301 -> www.example.com) and also all other domains that contain the same information, as shown below;

www.example.net -> 301 -> www.example.com

www.example.in -> 301 -> www.example.com 

Assure also that only the specified URLs work and make a 301 redirect rule for all non-specified URL’s when called missing.

#3 General page requirements

#3.1 Using standards

The site should comply with the World Wide Web consortium’s (http://www.w3.org/) recommendations for creating web pages (XHTML 1.0 Transitional should be enough) and also comply with the Americans with disabilities act (http://www.ada.gov/) if required.

#3.2 Page design

The pages should be designed with CSS positioning and the content part of the page should appear in the source code as early as possible preferably before other body content such as navigational blocks.

The navigation should be implemented with anchor tags and text and the links should not redirect.

Breadcrumb navigation would increase SEO with internal back links and usability in a sense that the visitor would see their location on site. Example of the breadcrumb navigation: Home => List furniture items => View table => ...

Scripts and other elements (CSS) should be put in external files. The source code should be kept clean with little or no unused code. The preferred maximum file size for HTML code is 100 KB.

#3.3 Elements of a page

The following elements should always be included (and be editable somehow) on a page which is to be indexed by search engine:

Page title ([title]-element in the header)
Meta description, robots and keywords (in the header)
Page heading (one [h1] per page)

#3.3.1 Page title ([title]-element in the header)

A page title should be as specific and concise as possible with respect to the document. This will insure its uniqueness and click-through in Search Engine Result Pages. A structure similar to "Page name | Section name | Site title - Tagline" is encouraged for clarity, uniqueness and better usability for the visitor. Focus on delivering a title that spans from specific (closer to the beginning) to general keywords. The length of the title needs no more then 80 characters.

#3.3.2 Meta description, robots and keywords (in the header)

HTML meta description around 150 characters should be sufficient. Although it doesn't hurt to be a little more, this data should contain the most concise information about the document. The uniqueness of this information also plays a fair role as far as Search Engine Result Pages are concerned.

Meta keywords on the other hand are not quite necessary since it is the responsibility of the search engine indexers to determine the nature and the relevancy of the document. For the purposes of accuracy, they can't rely on what the document claims it to be. There comes a transition on the Web which provides this sort of meta information about the document. Today, the results gained from meta keywords are negligible. See below some examples of well written meta tags;

<meta name="description" content="Suppliers of quality office furniture and accessories at discount prices.">
<meta name="keywords" content="furniture, office, store, shop, retail, discount">

#3.3.3 Page heading (one <h1> per page)

A proper structured document will consist of headings, paragraphs, lists, tables, and forms, and use an external stylesheet to style them. Many search engines place more emphasis on text within heading tags (and not just on keywords provided in meta elements), so make sure they use keywords. Use one <h1> tag per page with the most important keywords. You can also use other head tags ( <h2>, <h3> etc.) to provide variations and support the main heading.

Some example of tags are;

<h1>Tables</h1>
<h2>Round tables</h2>
<p>... information about round tables ...</p>
<h2>Square Desks</h2>
<p>... information about square desks, etc.</p>

#3.3.4 Body text

Make sure the text of your web pages contain keywords and common phrases which people might search for. Be careful with the frequency of your keywords - you want to have them occur at least a few times if possible, but don't repeat yourself so much that the copy becomes unnatural. The idea is to discretely spread keywords around without making it obvious.

A well written document will naturally use keywords that are appropriate and in proportion. Search engine algorithms essentially compare similar documents to get a better understanding of the nature of the document. If a document is not well written and gives off-balanced scores then it will raise flags and possibly mark it as not relevant as it indicates a document that is written for the machine and not for the human reader. Keep in mind that indexing is in place to assist human searches. An example of good body text could be like;

[p]Buy office furniture at affordable prices from any of our retail stores.[/p]

#3.3.5 Images and Pictures

When pictures, that are not part of the page template, are used they should always include an ALT description. This description should either be automated or editable (This is partly already a requirement of the Americans with disabilities act).

#3.4 Automation

The title element and the meta description and keywords need to be automatically generated according to different templates. These templates will include page- and directory specific elements as well as generic elements. An example of a template for the title element for a page called Search results page could be:

[Results] - [category] - Search results – My furniture example.com

Different elements that could be included are
Results = Search results pages (New, Old, All)
Category name = Such as Wood tables, Wood chairs, Metal chairs
Area = Can represents location of entity.
Page number in a Search results, if applicable
The category name or area might not be in basic form – different grammatical forms might be needed.

In the title, meta information and headings the keywords or key phrases are added as is or in another grammatical form but when automating (URL rewriting) the URL, it may need some encoding if other language has been used:

Non-ASCII keywords (and phrases) included in URLs need to be encoded in hex values (maybe using PHP function like urlencode()) like:
www.example.com/product/table/એપલ => www.example.com/product/table/%E0%AA%8F%E0%AA%AA%E0%AA%B2

#4 Index page

Index page of your website is the most likely to get the highest number of inbound links since it is entry point of your website. Hence linking other pages of website from this page becomes very important. By theory this page should host almost links of all pages that starts from here.

However number of links in such page should be around 100, in many projects it may not be possible to display all links. In such cases most important links should be made visible from here. And remaining pages could be linked from there because our purpose is to chain all important pages to be get indexed.

To make this working it becomes important to identify those important links. For example if you are selling something then this home page can have link of those pages that display list of items per category of products. Similarly if they are bound to certain geographical location and if you website displays list of selling items per province/city/area then links to those pages could be placed on this page.

#5 Search pages

Search pages whether simple or extended, may not be indexed as they are not containing, be default, any information to be searched for.

However for usability point of view, their URLs, page design and on-page information should be properly designed and implemented.

#6 Listing pages

Listing pages are the 2nd most important pages for any website as they display information about entities for which website is created. Listings entities can include various types of stuffs ranging from selling items, ads, jobs etc.

Such listing may contain pagination and sorting links depending upon results and interest of users. It is recommended to keep pagination links in text mode so that search engine can crawl through all available pages and can index those pages. However sorting links may be implemented using JS (Ajax) etc. so that additional query to server can be minimized. From search engine point of view, it doesn't matter in what order information displays.

If possible, URL scheme of such pages can be made self-informative. For example for furniture selling website URLs can be designed like below;

www.myfurniture.com/tables/round
www.myfurniture.com/tables/square
www.myfurniture.com/tables/plastic
www.myfurniture.com/chairs/rocking
www.myfurniture.com/chairs/revolving

Text appearing on such pages should be as informative as possible and number of entity per list should be kept around 10 to 30 entities. Listing pages may also contain links to other important pages which are to be indexed.

#7 View pages

View entity pages shows detailed information about entities listed in listing pages. Title, Meta description, Meta keywords, H1 tag should contain information about entity that is expected to be viewed.

#8 Other pages

Other pages may include pages like Login pages, Posting/Editing entity pages etc.

#8.1 Login pages

Such pages should not get indexed as they don't contain any public searchable information.

#8.2 Posting/Editing entity pages

Any page that contains forms to be submitted are not normally indexed as they don't display any searchable information to general public.

General rule of thumb is that those pages which changes stat of the server (like data is inserted/updated, file is created/delete etc.) or those pages which are personal to users are not indexed as they are tightly integrated with data of the website.

#9 Resources

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=35769
http://help.yahoo.com/l/us/yahoo/search/basics/basics-18.html

Lighttpd vs. Apache

2008-03-13T17:15:00.000+05:30

#1 Lighttpd overview

Lighttpd is an open source web server (similar like Apache) to server web pages. It has been developed by a MySQL developer named Jan Kneschke who developed this web server as a part of the C10K problem. Hence immediate reason of birth of Lighttpd is to overcome weakness, like reducing high memory footprint, of Apache web server.

The prefork model that Apache uses consumes a lot of memory (> 20 MB normally) per process. Which means if we multiply number of process to run simultaneously then RAM of server gets exhausted quickly. Lighttpd here beats Apache by using very low memory footprint (just 6MB) which means faster output from web server. The response appears even more faster when static contents are to be delivered. In Netcraft's latest web server survey, we can see Lighttpd among top 5 web servers currently used on Internet.

#2 How to set it up

Normal and most preferred installation instruction can be found from this installation page. For Yum users, a single command yum install zlib pcre lighttpd lighttpd-fastcgi will do almost all things.

If you want to start and stop Lighttpd manually, you're done. To install Lighttpd as a service like Apache, edit and install the init script (only if you have installed Lighttpd from source):

# sed -e 's/FOO/lighttpd/g' doc/rc.lighttpd > lighttpd.init
# chmod a+rx lighttpd.init
# cp lighttpd.init /etc/init.d/lighttpd
# cp -p doc/sysconfig.lighttpd /etc/sysconfig/lighttpd
# install -Dp ./doc/lighttpd.conf /etc/lighttpd/lighttpd.conf
# chkconfig lighttpd on

If you have installed Lighttpd using Yum then just follow last step. You may also use various other commands to start and stop Lighttpd web service like /etc/init.d/lighttpd start|stop|restart|condrestart|reload|status or service lighttpd start|stop|restart|condrestart|reload|status.

To just test lighttpd.conf, run command lighttpd -t -f /PATH/TO/CONF/lighttpd.conf

#3 Differences between Apache and Lighttpd

#3.1 General

The main difference between Apache and Lighttpd is the serving model, Lighttpd is event-driven and Apache is threaded or pre-forked.

Apache provides different multiprocessing models (MPMs) for different runtime environments. The prefork model that Apache uses creates number of processes at startup of service and manages them in a pool. However each process requires lot of memory to handle requests which means the more the processes the more memory will require. That is simultaneous apache processes quickly eat available RAM.

On the other hand Lighttpd uses single process, single thread and non-blocking I/O. For that it uses fastest even handler in the target system like: poll, epoll, kqueue or /dev/poll. This difference makes Lighttpd faster than Apache in serving static files.

However the biggest difference between both is how they support scripting languages (specially like PHP). Apache has upper hand here because it supports easy to use Shared module version, CGI and FastCGI all together while Lighttpd supports only FastCGI at this moment.

#3.2 Configuration level

There is visible difference between styles of configuration files of Lighttpd (lighttpd.conf) and Apache (httpd.conf). Syntax of lighttpd.conf will look more like syntax of php.ini while httpd.conf has XML type syntax. Here is an example of some basic configuration:

#3.2.1 Basic Configuration

Apache:

DocumentRoot /var/www/html

CustomLog /var/www/logs/access

ErrorLog /var/www/logs/error

User apache

Group apache

Lighttpd:

server.document-root="/var/www/html"

accesslog.filename="/var/www/logs/access"

server.errorlog="/var/www/logs/error"

server.username="apache"

server.groupname="apache"

server.modules=("mod_cml")

#3.2.2 Virtual Hosts

Below is an example of difference between VirtualHosts of Apache and Lighttpd. Example is shown for myproject project.

Apache:

NameVirtualHost *

<VirtualHost *:80>
 ServerName 'www.myproject.com'
 DocumentRoot '/web/myproject/web'
 ErrorLog '/web/logs/myproject_error'
</VirtualHost>

Include conf.d/virtualhosts/*.conf

Lighttpd:

$HTTP[“host”] == “www.myproject.com” {

 server.document-root=”/web/myproject/web”

 server.errorlog="/web/myproject_error"

}

#3.2.3 Authentication and Authorization

Lighttpd, at this moment, does not support .htaccess files, so all settings must be specified in the lighttpd.conf file, or the configuration files that it includes. However it understands Apache user files for basic and digest authentication, but group file support is not yet implemented but will be implemented soon. Here is an example of authentication and authorization:

Apache:

<Directory ~>
  AuthName "Authentication required to access this area."
  AuthType Basic
  AuthUserFile /web/myproject/docs/valid.users
  Order deny,allow
  Require valid-user
</Directory>
 

Lighttpd:

auth.backend="htpasswd"

auth.backend.htpasswd.userfile="/web/myproject/docs/valid.users"

auth.require=

("~" =>

(

    "method" =>"basic",

    "realm"  =>"Authentication required to access this area.",

    "require"=>"valid-user"

)

)

Summarily, configuration file of Lighttpd server behaves like an active script in which you can declare variables, write logic, do computation based upon criteria etc. similar like programming script. This feature makes configuration file alive and agile.

#4 How to run PHP under Lighttpd

#4.1 Configuring PHP under Lighttpd

Apache processes PHP internally i.e using it as Shared module mod_php while Lighttpd runs PHP under FastCGI. Although Apache also supports FastCGI, using PHP under FastCGI with Apache is neglected and is not used. However with Lighttpd, only option is to run under FastCGI, PHP must be compiled with FastCGI option (thought it is not used with Apache). For more information, please read http://trac.lighttpd.net/trac/wiki/TutorialLighttpdAndPHP. Below is the example of difference between running PHP under Apache and lighttpd.

Apache:

LoadModule php5_module modules/libphp5.so
AddType application/x-httpd-php .php

Lighttpd:

server.modules=( ..., "mod_fastcgi", ... )
fastcgi.server=( ".php" =>
                 (
                   (
                     "socket" => "/tmp/php-fastcgi.socket",
                     "bin-path" => "/usr/bin/php-cgi",
                     "broken-scriptfilename" => "enable",
                     "bin-environment" =>
                     (
                       "PHP_FCGI_CHILDREN" => "2",
                       "PHP_FCGI_MAX_REQUESTS" => "5000"
                     ),
                     "min-procs" => 1,
                     "max-procs" => 2,
                     "idle-timeout" => 60
                   )
                 )
               )

You may require to set path of php-cgi according to your setup. Please note that directive server.modules actually exists along with other modules on top of configuration file hence above line indicates that mod_fastcgi should be enabled in lighttpd.cnf.

Then it will require to set 1 directive in following way in php.ini configuration file if it exists, if doesn't then nothing to do.

cgi.fix_pathinfo=1

Last 4 directives of above mentioned configuration are for running PHP scripts in better ways.

#4.2 Application wise changes

As FastCGI is a separate process, we can't handle directives of PHP into configuration file of web server (i.e lighttpd.conf). This is one of the biggest drawback of FastCGI that is why PHP is not used under FastCGI. Moreover under FastCGI mode, your PHP script would get limited support from web server which may force you to change or rewrite your scripts. Hence it is not recommended to use PHP under Lighttpd (at this moment because Lighttpd currently supports only FastCGI mode) because of it's lack of features that PHP will require like enabling configuration options of php.ini in configuration file for all hosts or per host base.

Moreover many benchmarks shows that PHP runs slower under FastCGI than under shared version on Apache.

However if it is required to run PHP on Lighttpd, then 2 major changes will require. They are:

To move all PHP related setting either in php.ini directly or in configuration or global file of the application.
Removing all Apache web server related variables and settings from application.

Once these changes are done, it will require to test application heavily to find whether functionalities of application get broken somewhere or not.

#5 mod_uploadprogress and Prototype

This feature will be available in Lighttpd version 1.5.0 which is not released yet, hence can not test or write more about it. More information about this module can be found at here. However when it will get released, it would surely be one of the finest module of Lighttpd server because it can be easily integrated with front-end applications using JSON.

#6 Lighttpd and output compression

Lighttpd provides output compression for static data through mod_compress module. Which means before sending static contents to client, mod_compress compresses it and saves at specified path. This compressed and cached copy will be served directly from cached location when similar request is made from same or different client. Thus saving valuable bandwidth and increasing response time.

Lighttpd supports 3 types of compressions viz. deflate, gzip and bzip2. The limitation of compressing and caching is that Lighttpd can not compress files with size more than 128 MByte and less than 128 Bytes.

To enable compression we need to set 3 directives in lighttpd.conf file. They are:

compress.cache-dir="/var/www/cache/myproject/"
compress.filetype=("text/plain", "text/html")
compress.max-filesize=1 MB

However since there is upper limit of file size of 128 Mbytes, the last directive is not necessary to declare. While compressing various types of static data, it should be kept in mind that if no file type or wrong file type is mentioned then no file will get compressed.

You may require to manually create cache folder and assign necessary write permissions to it. These cached contents do not automatically get cleared hence it is left to developer to clean it at periodic level when required. Following type of command can be used to remove contents that are older than a week.

$ find /var/www/cache/myproject/ -type f -mtime +7 | xargs -r rm

To compress dynamic contents, we need to reply on PHP itself as PHP natively supports good compression of dynamic contents. For that following 2 directives are to be set in php.ini or in equivalent configuration file.

zlib.output_compression=1
zlib.output_handler=On

Please note that to use zlib.output_compression, value of output_handler should be zlib.output_handler instead of standard output_handler. To do so, output_handler directive is to be set in following way:

output_handler=zlib.output_handler

or

zlib.output_handler=On

#7 Lighttpd and caching

#7.1 Caching overview

Caching is also another method to gain better performance in serving contents and increasing response time of your PHP scripts. There are several types of caching softwares available for PHP. Some important from them are Zend Platform, APC (APC GUI), XCache, eAccelerator, ionCube Encoder and PHP Accelerator. Certain web servers like Lighttpd provides built in modules for caching static contents at web server level. They are mod_expire, mod_mem_cache and mod_cml. Hence using combination of caching static and dynamic contents effectively, we can gain lot of speed in serving contents. However all of these mechanisms are not similar.

Aforementioned independent softwares are for Opcode/Bytecode caching i.e caching your PHP script into compiled state so that when new request arrives for same script, cache software will server compiled version of code directly from cache rather than reading file again from the disk and then compiling. From these 6, eAccelerator, XCache and APC are widely used caching softwares. This benchmarks also show that how XCache and APC are better than others. We will learn more about XCache in a short while.

As said earlier that only good combination of static and dynamic contents can give considerable boost in performance, we should try to cache as much contents as possible. To cache static contents, integrated modules of web server are the best candidates. In case of Lighttpd they are mod_expire and mod_mem_cache (however this is not provided as default).

#7.2 mod_expire

Mod_expire controls the Expire header in the Response Header of HTTP/1.0 messages. It is useful to set it for static files which should be cached like images, style-sheets etc. To use this module, first it needs to get enabled in server.modules directive array. Then module specific directives are to be set in server's configuration file as shown below.

<access|modification> <number> <years|months|days|hours|minutes|seconds>

Some examples could be like:

Cache contents of folder images for 2 hours.

expire.url = ( "/images/" => "access 2 hours" )

Cache contents of all sub-folders of images folder for 2 hours.

$HTTP["url"] =~ "^/images/" {
expire.url = ( "" => "access 2 hours" )
}

Values can be hours, months, days etc. depending upon requirement.

#7.3 mod_mem_cache

Mod_mem_cache is a plugin which stores content of files in memory for faster serving. That is it stores specified file types into memory to serve directly from there without going to read it from disk from specified location thus saving disk read access time. This module is a 3rd party module, hence is not included in the official distribution of Lighttpd.

This module doesn't seem that much promising to use effectively for caching as memory should be used for processing data rather than storing data. Moreover memory should not be occupied for serving files that can reside and easily managed on disks. For example when we have thousands of images to be served then it is not advisable to store them into memory just to serve it faster. More information about this plugin can be found at here.

#7.4 mod_cml (Cache Meta Language)

Mod_cml is an another caching module similar like mod_expire which is provided by Lighttpd to cache static contents of dynamic pages. The difference between mod_expire and mod_cml is that mod_cml can cache fragmented static contents which are part of dynamic contents. For example a dynamic page called index.php might have static contents like menu.html, banner.html inside it which are not integral part of index.php. In such case using mod_cml, these 2 static contents can be cached and can be delivered directly from there.

But such type of caching can not be handled directly by Lighttpd web server and mod_cml hence we need to write some code in PHP or in special CML scripts for mod_cml which is written in lua programming language.

To use mod_cml, it requires to install lua programming language and libmemcache-1.3.x. Additionally Lighttpd must be compiled with 2 options --with-lua and –with-memcache.

#7.5 XCache

XCache is a newly emerging candidate in the market of caching PHP scripts. This is an independent software and not a module of Lighttpd. However it has been written by developers of Lighttpd.

XCache is an open-source opcode cacher, which means that it accelerates the performance of PHP on servers. It optimizes performance by removing the compilation time of PHP scripts by caching the compiled state of PHP scripts into the shm (RAM) and uses the compiled version straight from the RAM. This will increase the rate of page generation time by up to 5 times as it also optimizes many other aspects of php scripts and reduce server load. Some of the good features of XCache are:

Optimized opcode cache.
Using a generator to produce C code, reduces human mistake greatly.
Running stable on PHP_4_3/PHP_4_4
Supported and tested on all latest php cvs branches, such as PHP_4_3 PHP_4_4 PHP_5_0 PHP_5_1 PHP_5_2 HEAD (6.x)
Alpha supported for in-alpha-php6, with Unicode enabled.
Read-only Cacher Protection that prevents the cache from being corrupted by php-core/extension or any code other than XCache itself.
Atomic get/set/inc/dec API operation on var cache for php programmers.
Optimizer
Encoder/Decoder(Loader)
Administrator Script

view statistics
to see if it's AutoDisableOnCorrupted?
view cached php/variable list
clear cache

The last feature allows administrator to view statistic and cached PHP variables and manage caching behavior of XCache.

#7.5.1 Installing XCache

The standard way to install XCache is from source. Get your desired version of XCache from here. Then follow below steps to install it.

# tar -zxf xcache-*.tar.gz
# cd xcache
# phpize
# ./configure --enable-xcache
# make
# su
# make install
# cat xcache.ini >> /etc/php.ini

To make sure XCache is properly installed, run below command.

$ php-fcgi -v

It will show string like with XCache vX.X, Copyright (c) XXXX-XXXX, by XXX. Same can be checked from output of phpinfo() function also. Once XCache is installed, it will require to edit xcache.ini which contain various caching related directives to be used. However it is not mandatory to edit or change. A complete explanation of all the directives can be found from http://trac.lighttpd.net/xcache/wiki/PhpIni.

#7.5.2 Configuring Administrator panel

XCache Administrator panel is an important web interface that you can monitor and operate your opcode cache, seeing how well(or bad) it goes. Since this page is protected by http-auth, it will require to provide certain values in xcache.ini. For that set below 2 directives.

xcache.admin.user='USER'
xcache.admin.pass='MD5(PASSWORD)'

where USER is name of user you wish to use and MD5(PASSWORD) is MD5 encrypted string of password that you wish to use for given USER.

To set up web interface, copy xcache/admin/ (the whole directory) to your web document-root or sub-directory of it then request it from your browser, a http-auth prompt will popup where you will require to provide above USER and PASSWORD (as a normal string, not MD5 encrypted string). However sometimes installing XCache from rpm based utilities it may require to alias in web server instead of copying the script. To do so, add below directive in your server configuration file.

Apache:

Alias /xcache-admin/ /usr/share/xcache/admin/

Lighttpd:

alias.url += ("/xcache-admin/" => "/usr/share/xcache/admin/")

Gaining performance boost by using caching mechanism is tricky. Unless used carefully, it cannot give required boost. As we know that we can't cache everything (specially dynamic contents), we should try to cache whatever is left. This can be achieved by various types of caching as discussed above. Static contents are well cached by clients, if not then can be cached by web servers. PHP scripts can be cached using Opcode caching softwares like APC, XCache etc. While static part of dynamic data can be cached by modules like mod_cml (for lighttpd web server only).

#8 Summary

Summarily, Lighttpd web server is surely worth to have look at it and to be used for serving static data. For dynamic contents like PHP scripts, it is not optimized (because of support of only FastCGI) hence we have to wait until Shared module version PHP get started to support by it. At this moment it is widely used to server static contents only. So it will take time for it to really start competing with Apache.

However certain modules like mod_secdownload, mod_compress, mod_geoip, mod_trigger_b4_dl, mod_uploadprogress, mod_useronline etc. are peculiar modules of Lighttpd which can make it stand firmly with currently popular web servers.

#9 Links

http://www.onlamp.com/pub/a/onlamp/2007/04/05/the-lighttpd-web-server.html
http://survey.netcraft.com/Reports/0703/
http://schlitt.info/applications/blog/index.php?/archives/504-Apache-vs.-Lighttpd-echo-performance.html
http://trac.lighttpd.net/trac/wiki/TutorialInstallation
http://trac.lighttpd.net/trac/wiki/TutorialConfiguration
http://trac.lighttpd.net/trac/wiki/Docs:ConfigurationOptions
http://trac.lighttpd.net/trac/wiki/TutorialLighttpdAndPHP
http://trac.lighttpd.net/trac/wiki/Docs:ModUploadProgress
http://trac.lighttpd.net/trac/wiki/Docs:ModCompress
http://blog.lighttpd.net/articles/2006/08/01/mod_uploadprogress-is-back
http://trac.lighttpd.net/trac/wiki/Docs:ModCML
http://www-128.ibm.com/developerworks/library/os-php-fastapps1/index.html
http://trac.lighttpd.net/xcache/wiki/Faq

Varnish accelerator

2008-01-02T17:14:00.000+05:30

#1 Introduction

Varnish is a high performance HTTP accelerator (more precisely a Reverse proxy server) designed for content-heavy dynamic web sites. In contrast to other HTTP accelerators, many of which began life as client-side proxies or origin servers, Varnish was designed from the ground up as an HTTP accelerator. The Varnish web site claims that Varnish is ten to twenty times faster than the popular Squid cache on the same hardware.

Varnish is installed within the neighbourhood of one or more webservers. All connections coming from the Internet addressed to one of the webservers are routed through the proxy server, which may either deal with the request itself or pass the request wholly or partially to the main webserver.

There are various reasons to install reverse proxies. They are:

Security: the proxy server is an additional layer of defence and therefore protects the webservers further up the chain.
Encryption / SSL acceleration: when secure websites are created, the SSL encryption is sometimes not done by the webserver itself, but by a reverse proxy that is equipped with SSL acceleration hardware.
Load distribution: the reverse proxy can distribute the load to several servers, each server serving its own application area. In the case of reverse proxying in the neighbourhood of webservers, the reverse proxy may have to rewrite the URLs in each webpage (translation from externally known URLs to the internal locations).
Caching static content: A reverse proxy can offload the webservers by caching static content, such as images. Proxy caches of this sort can often satisfy a considerable amount of website requests, greatly reducing the load on the central web server.
Compression: the proxy server can optimize and compress the content to speed up the load time.
Spoon feeding: if a program is producing the webpage on the webservers, the webservers can produce it, serve it to the reverse-proxy, which can spoon-feed it however slowly the clients need and then close the program rather than having to keep it open while the clients insist on being spoon fed.

#2 Architecture

Varnish is heavily threaded, with each client connection being handled by a separate worker thread. When the configured limit on the number of active worker threads is reached, incoming connections are placed in an overflow queue; only when this queue reaches its configured limit will incoming connections be rejected.

The principal configuration mechanism is VCL (Varnish Configuration Language), a DSL used to write hooks which are called at critical points in the handling of each request. Most policy decisions are left to VCL code, making Varnish far more configurable and adaptable than most other HTTP accelerators. When a VCL script is loaded, it is translated to C, compiled to a shared object by the system compiler, and linked directly into the accelerator.

A number of run-time parameters control things such as the maximum and minimum number of worker threads, various timeouts etc. A command-line management interface allows these parameters to be modified, and new VCL scripts to be compiled, loaded and activated, without restarting the accelerator.

In order to reduce the number of system calls in the fast path to a minimum, log data is stored in shared memory, and the task of filtering, formatting and writing log data to disk is delegated to a separate application.

#3 Installation

Here we will go through quick installation process. Please get latest version of Varnish from here or check it out from repository.

#3.1 Prerequisites

The following tools are required to build Varnish:

A recent version of GCC.
A POSIX compatible make.
Recent versions of GNU autotools like automake, autoconf, libtool.

Latest versions of OSes are most likely to contain above mentioned items.

#3.2 Configuring and Building

$ ./autogen.sh

You may see some error messages. Check if configure and Makefile.in were generated. If they weren't, you probably need newer versions of the GNU autotools. If they were; run autogen.sh again: any error messages it still shows the second time around are most likely caused by bugs in autoconf macros installed by other software you have on your machine, and can safely be ignored.

Next, run configure. In most cases, the defaults are correct and you do not need to specify any command-line options, except perhaps --prefix. If you plan on hacking the Varnish sources, however, you will most likely want to turn on stricter error checks and dependency tracking:

$ ./configure

OR

$ ./configure --enable-debugging-symbols --enable-developer-warnings –enable-dependency-tracking

If configure completes without any errors, simply run below two commands to compile and install Varnish.

$ make
$ make install

For more information please visit this link .

#3.3 Enabling Varnish caching

Varnish API comes with Management console (telnet HOST/IP PORT), Caching process as a child process of management process (varnishd), and some utilities for logging (varnishlog and varnishncsa), statistics of caching (varnishstat), histogram (varnishhist) and log entry ranking (varnishtop).

Following commands can be used to enable varnish caching on your servers.

$ varnishd -a www.example.com:80 -b www.example.com:8080
$ varnishd -a www.example.com:80 -f /usr/local/etc/varnish/myconf.vcl
$ varnishd -a www.example.com:80 -b www.example.com:8080 -T www.example.com:6082
$ varnishd -a www.example.com:80 -f /usr/local/etc/varnish/myconf.vcl -T www.example.com:6082

1st command denotes that website www.example.com is originally running on port 8080 on Apache web server but it's running through Varnish under port 80 which is default port for http. This is must for production server but for development and/or test server, ports could be exactly in reverse because during development and testing you may want to run your websites without caching.

Sometimes we might want to use different caching policies (like caching documents having cookies) which is written in special configuration syntax called VCL; in that case 2nd command is useful to tell Varnish to use modified configuration language file than the default one. When -f switch is used, -b switch cannot be used together because values of -b switch is now mentioned in configuration file.

Once caching is started it can be controlled by management console from which caching can be started, stopped and various configuration values can be set and unset. For that 2 steps are needed.

enabling Varnish as shown in command 3 or 4 and
using Telnet utility to open management console on given port for given host (like telnet www.example.com 6082).

Please note that to start and stop caching do not just kill process, instead use management console to control caching for particular host.

Varnish stores log into memory hence to dump it in regular file on disk, use varnishlog or varnishncsa utilities. For more information and how to use these and other utilities, please check their man pages.

#4 VCL

#4.1 Description

VCL is an acronym for Varnish Configuration Language. In a VCL file, you configure how Varnish should behave. It is like Apache web server's httpd.conf and PHP's php.ini configuration files.

#4.2 Syntax

The VCL syntax is very simple, and deliberately similar to C and Perl. Blocks are delimited by curly braces, statements end with semicolons, and comments may be written as in C, C++ or Perl according to your own preferences.

In addition to the C-like assignment (=), comparison (==) and boolean (!, && and ||) operators, VCL sup-ports regular expression and ACL matching using the ~ operator.

Unlike C and Perl, the backslash (\) character has no special meaning in strings in VCL, so it can be freely used in regular expressions without doubling.

Assignments are introduced with the set keyword. There are no user-defined variables; values can only be assigned to variables attached to backend, request or document objects. Most of these are typed, and the values assigned to them must have a compatible unit suffix.

VCL has if tests, but no loops.

The contents of another VCL file may be inserted at any point in the code by using the include keyword followed by the name of the other file as a quoted string.

#4.3 How to

#4.3.1 refresh (purge) document when it gets changed on server?

Refreshing is often called purging a document. There are 2 different ways in Varnish to refresh (purge) any document/s:

From management console you can type below commands to control purging of desired documents. Regular expressions are allowed in syntax so many documents can be purged by giving few commands.

url.purge ^/$

url.purge .*html$

In VCL we can write logic to purge any document when request is method is PURGE. Which means any document that needs to get purged, will require to call same document by PURGE method to remove itself from cache. This is the most convenient and practical way to keep fresh copies of documents in cache. It is also automatic way so server administrator need not to manually purge large amount of documents.

Define all possible hosts only from which purging request will be accepted. This is good precaution so that not everyone can purge what is in cache.

acl purge
{
"myhost"; "123.456.789.1";
}

When request is received.

sub vcl_recv
{
if (req.request == "PURGE")
{
    if (!client.ip ~ purge)
    {
      error 405 "Not allowed.";
    }
    lookup;
}
}

When cache is hit (i.e document is to be served from cache).

sub vcl_hit
{
if(req.request == "PURGE")
{
    set obj.ttl = 0s;
    error 200 "Purged.";
}
}

When cache is missed (i.e document is to be served directly from backend server).

sub vcl_miss
{
if(req.request == "PURGE")
{
    error 404 "Not in cache.";
}
}

#4.3.2 cache documents even when cookies are present?

When request is received.

sub vcl_recv
{
if (req.request == "GET" && req.http.cookie)
{
    lookup;
}
}

Fetch document from backend server.

sub vcl_fetch
{
if (resp.http.Set-Cookie)
{
    insert;
}
}

#4.3.3 support multiple sites running on separate backends in the same Varnish instance?

Define all backend WWW servers which are to be used for caching.

backend www
{
set backend.host = "www.example.com";
set backend.port = "8080";
}

Define all backend Image servers which are to be used for caching.

backend images
{
set backend.host = "images.example.com";
set backend.port = "8080";
}

When request is received.

sub vcl_recv
{

if (req.http.host ~ "^(www.)?example.com.com$")
{
    set req.backend = www;
}
elsif (req.http.host ~ "^images.example.com")
{
    set req.backend = images;
}
else
{
    error 404 "Unknown virtual host";
}
}

#4.3.4 force a minimum TTL for all documents?

Fetch document from backend server.

sub vcl_fetch
{

  if (obj.ttl < 120s)

{
set obj.ttl = 120s;
}
}

#5 Performance

While Varnish is designed to reduce contention between threads to a minimum, its performance will only be as good as that of the system's pthreads implementation. Additionally, a poor malloc implementation may add unnecessary contention and thereby limit performance. On FreeBSD (using libthr) and Linux (using native threads), it is believed that performance is limited only by hardware.

When the requested document is in cache, response time is typically measured in microseconds. This is significantly better than most HTTP servers, so even sites consisting mostly of static content will mostly benefit from Varnish.

#6 Limitations

Current versions of Varnish do not understand the HTTP Vary: header, which can lead to problems with sites which support content negotiation.
the HTTP Host: header is always included in the object hash, so sites which can be accessed under multiple different names will have multiple copies of the same content cached.
Default policy of Varnish doesn't allow caching documents having cookies/sessions, which means websites heavily dependent upon cookies and session can not use Varnish out of the box for dynamic documents. To solve this problem VCL is to be tweaked as shown in section 4.3.2.
Varnish’s internal caching mechanism doesn’t obey even the minimum requisite client-side HTTP caching pragmas. It fails to obey other established caching headers, and support for them cannot even be implemented by end users through configuration, because there’s no mechanism to control cache behavior based on Web server HTTP headers — only on client headers. Which means preventing caching of files without an ETag response header is very hard to implement.
Varnish refuses to start if your /tmp is mounted noexec. Because Varnish attempts to compile a “shared lib” and load it from /tmp. Such problems are very hard to detect because the startup script doesn’t give any indication, and the log files don’t either.
There is lack of proper documentation for Varnish and VCL. There is some documentation in man pages but it is accessible only when you have Varnish installed on your PC.

Most of these limitations have been or are being addressed in the development version.

#7 Conclusion

Web accelerators (here caching software) are not install and forget type of software. They require constant monitoring and inspection on them for their behaviour and effectiveness. Software like Varnish have their limitations as shown in section 6 which must be kept in mind before using them. Then there are other things to be taken care of in your project to use caching most effectively.

Caching of documents is implemented on GET and HEAD methods only. Hence your project must have maximum documents using above 2 methods.
URL structure should be caching friendly.
For dynamic document session IDs should not get appended into URL because they are dynamic and different every time they are generated hence same document having such different session IDs makes caching of documents less effective because same document will have different versions in cache as session IDs are different.

#8 Links

http://varnish.linpro.no/
http://phk.freebsd.dk/pubs/varnish.pdf
http://rudd-o.com/archives/2007/07/02/why-the-varnish-cache-sucks-with-bonus-varnish-dev-whining-about-me/
http://projects.linpro.no/pipermail/varnish-misc/2007-July/000577.html
http://www.version2.dk/artikel/3084
http://varnish.projects.linpro.no/wiki/StatsExplained
http://varnish.projects.linpro.no/wiki/FAQ

Designing web service

2007-08-09T17:13:00.000+05:30

#1 New way of Programming experience

Designing API requires different way of thinking in programming because API is mainly related to handling raw data and header management. Hence apart from traditional web development, it requires to have good understanding and experience of HTTP Headers, HTTP request methods, REST architectural style, OOP and Responsiveness.

Another difference between API and normal web application is that once API is being started to use it grows vertically hence it's URL scheme, response headers and types can not be changed that easily so proper care should be taken before designing it. Best practice is to start with providing simple interface of only GETting data. And from that experience more functionality for adding, updating and deleting resources can be added.

#2 Handling input data

While designing APIs, developers should not rely on browsers because except POST and GET methods; data coming by other methods will directly come to server skipping the browser. Hence population of incoming data (which is done by PHP by storing them in $_GET and $_POST super globals) would not be possible particularly for PUT and DELETE methods. Which means all data should be handled by standards input which can be accessed from php://input. After that they can be parsed to extract information in chunks to finally execute requests. Please study example API for more details about how to handle and implement raw data. This link explains all possible HTTP methods and how they work.

#3 Structure of codebase

Normally API would require 1 configuration file to store server specific data, 1 global file to store other global settings, 1 front controller and then bunch of classes according to required modules. Since API doesn't have GUI, multiple controllers are not required. And anyway modern practices recommend having only 1 single controller to receive and dispatch data.

URL of API could be like http://api.mysite.fi or http://www.mysite.com/api/. As far as directory structure is concerned there can be created folders to store API related includable files, classes and controller/s. Since API is part of main application, file and folder structure should conform to existing standards of main application.

#4 Authentication and session management

Access control and session management are important aspects of API design. When it is required to change state of resource on server then it becomes important to identify client who has made such request. This can be done by asking requester to provide details of account on behalf of which request is made.

For example, to build generic authentication mechanism a URL like http://www.mysite.fi/api/api_login/ can be provided where clients can submit his/her ID and Password to log on. The API authentication module will verify ID and Password and if found ok then can issue an api_token which can then be used to execute subsequent requests. Depending upon sensitiveness of data various authentication mechanisms can be implemented which includes session based authentication and basic/digest authentications.

For more information about authentication following resources can be studied:

http://code.google.com/apis/gdata/basics.html
http://code.google.com/apis/gdata/auth.html
http://code.google.com/apis/accounts/AuthForInstalledApps.html

#5 Access control

When API get considerable traffic from various clients it becomes important to restrict clients to not to make so many requests at particular time. Depending upon available resources and amount traffic to various resources, various timing related mechanism can be applied so that clients can not submit particular request more than once or twice or thrice for each minute or hour or so and so. This is important to keep hackers/crackers away who can damage API service at various levels.

For advanced usage, access control can also be applied to allow usage of other users' data. Hence if API client want to use other users' resources, he/she first get confirmation from respective user/s to use his/her data for various purposes.

#6 Caching of common resources

Once API starts receiving high amount of traffic to particular resource/s, it may become necessary to cache certain resources into memory or file to serve them faster and to save bandwidth. There can be some resources like top rated image of users, which may not be changing frequently, could be cached.

#7 Designing URLs

As we have adopted REST architectural style for implementing API, we must adhere to it's standards. For that it would require to properly understand REST principles. In REST architectural style, URL of any resource is tightly coupled to request method of that resource. Which means URL can be same but depending upon request method, response from API could change. For example resource http://www.mysite.fi/api/users/ requested by POST method would mean to add new user to existing users; with GET method will provide list of users. Similarly resource http://www.mysite.fi/api/user/1/ requested by PUT method would update information of particular user; with GET method will provide details of user and; with DELETE method will remove that user from database.

Please note that if resource http://www.mysite.fi/api/user/1/ is requested by POST method then there should be appropriate error response telling API client that POST method is not allowed for such resource. Similarly no resources can be deleted by GET, POST or PUT methods. For that only DELETE method must be used. Such problem exists with Del.ico.us API where everything is done via GET including creating, changing and removing resources which is unRESTful way of implementation.

#8 Programming approach

A typical API can contain following objects/modules. Each object/module could be a single class to handle request/s at various stages.

#8.1 Request handler

This is the 1st module which will receive request. It will then determine request method, will log client's information and will store raw data received in request. Then it will give control to authentication or session manager.

#8.2 Authenticator/Session manager

This module, if required, will check whether authentication is required for particular request or not. If yes then will inform client via error handler module to send credentials. This module will generate authentication tokens to client for subsequent requests.

#8.3 Parser

Once client is authenticated next step would be to parse raw data to extract information from it to for execution. This extracted data will then be handed over to data validator.

#8.4 Data Validator

This module will validate (will check accept type, range and type of data etc.) extracted data for particular requests. If everything is ok then Executer module will be called to execute request for which request is made.

Here Parser and Data validator modules can be combined together also depending upon convenience of developers.

#8.5 Executer

This module will be collection of various classes which will actually execute request and generate output in Arrays. Such modules could be same as primary objects of requests like image, user, tag etc.

These classes should preferably generate data in associate arrays so that it can be used in any format. Once output is generated it will be handed over to renderer module.

#8.6 Renderer (XML/XHTML/JSON etc.)

This module will redder generated data and will send back to client. Default format of data could be XML but depending upon query string different format can be provided. There can be separate templates for each format so that in final output dynamic part will be replaced with data and static part will remain as it is.

#8.7 Error handler

This module can be used with any module at any level to inform client about occurrence of any error while executing any request. Format of errors could be like below:

<?xml version="1.0" encoding="utf-8" ?>
<res status="0">
<error code="123">Invalid data. Please read documentation for more details.</error>
</res>

Note that whenever error occurs, "status" attribute will be set to "0" rather than 1. Apart from custom error messages and codes, standard HTTP headers can also be used to minimize parsing of output at client side to determine whether request was executed successfully or not.

All these modules classes could be designed by using latest concepts of PHP5. One such concept is to have Getter and Setter methods to set and get data whenever required. It is not easy to test and debug API by conventional browsers because developers can't send raw headers and data by browsers. Hence either API client will require to be designed first or to use other options. One such option is to use LiveHTTPHeaders extension in FF browser. It is an excellent way to send and receive raw data over HTTP to API. By this way developers can easily see what response is received when particular request is made. For more details about this extension, visit link http://livehttpheaders.mozdev.org/

Excellent resource about how to design RESTful API can be found here http://www.peej.co.uk/articles/restfully-delicious.html.

#9 Sample of output of data

Below is given samples of XML responses. Root level node would be having attribute 1 or 0 to indicate API client whether request was executed successfully or not. Rest of nodes can be designed as per requirements. It would be noted that most common way of response is XML. Later when API gets more demand, more response format can be provided like Serialized PHP, JSON etc.

=> Basic XML structure of response:

<?xml version="1.0" encoding="utf-8" ?>
<res status="1"></res>

=> XML structure of response with data:

<?xml version="1.0" encoding="utf-8" ?>
<res status="1">
  <image>
    <resource>/image/recent</resource>
    <url>http://www.mysite.fi/images/123456.jpg</url>
    <width>120</width>
    <height>90</height>
    <metadata>
      <camera>Canon EOS 60</camera>
      <date-taken>10-09-2007</date-taken>
    </metadata>
  </image>
</res>

REST architectural style emphasizes on following 2 rules while responding requests, they are:

#9.1 Resources should be interconnected

which means each response should contain information about previous and next resource. For example while displaying list of users if limit of response is 30 records per page then each response should contain link of next and previous list of users if applicable. This tells that by inspecting any response, API client could find what was previous response and what will be the next one. This can be accomplished either by setting attribute/s like next-url and previous-url or by providing attributes like tot-page, cur-page, next-page etc. in response XM so that those can be used as query string of request to access next or previous resources.

Similarly whenever new resource is created, response sent back to client should contain URL of that newly created resource. This is how inter-connectivity works.

#9.2 Revelation of information should be step by step

which means all information should not be responded with just one call. For example request like http://www.mysite.fi/api/images/ should not contain details of each image, instead it can contain link to obtain information of each image (http://www.mysite.fi/api/image/12345_6789.jpg) and after calling that resource, actual information of image would be responded.

All response should be using UTF-8 characters and encoded in same way before sending to client.

#10 Documentation

Documentation is must while designing any API. Without documentation none would use your API. Hence it should be prepared in parallel with developing of code. It also should be easy to understand and use.

#11 Example of API

It is important to study existing API while building API first time. Below link contains source code of such API. It is REST implementation of managing users and companies. However it is very old and developed in PHP4 but still worth to have look at it specially it's code to understand how to handle, execute and respond various requests.

http://nchc.dl.sourceforge.net/sourceforge/phprestsql/phprestsql.tar.gz
http://phprestsql.sourceforge.net/tutorial.html

CVS best practices

2007-06-11T17:15:00.000+05:30

Best practices often requires thorough knowledge of technology that is used in software development. To fully use CVS, we now need to know about tag, trunk, branch, merging etc. to minimize or eliminate certain problems arising of out insufficient use of CVS. In this document it has been shown how we can use CVS in most effective way to minimize such problems. Here are some policies that have been designed to follow whenever it is possible.

#1 Sandbox

The developer sandbox is where each developer keeps his or her working copy of the code base. In CVS this is referred to as the working directory. This is where they build, test and debug the modules that they are working on. A sandbox can also be the area where the staging build or the production build is done. Changes made in the work area are checked into the CVS repository. In addition, changes made in the repository by others have to be updated in the sandbox on a regular basis.

The best practices related to developers sandbox are:

#1.1 Keep System clocks in Sync

CVS tracks change to source files by using the timestamps on the file. If each client system date and time is not in sync, there is a definite possibility of CVS getting confused. Thus system clocks must be kept in sync by use of a central time server or similar mechanism.

CVS is designed from ground up to handle multiple timezones. As long as the host operating system has been setup and configured correctly, CVS will be able to track changes correctly.

#1.2 Stay in sync with the repository

To gain the benefits of working within a sandbox as mentioned above, the developer must keep his or her sandbox in sync with the main repository. A regular cvs update with the appropriate tag or branch name will ensure that the sandboxes are kept up to date.

#1.3 Do not share the sandbox

Sandboxes have to be unique for each developer or purpose. They should not be used for multiple things at the same time. A sandbox can be a working area for a developer or the build area for the final release. If such sandboxes are shared, then the owner of the sandbox will not be aware of the changes made to the files resulting in confusion.

In CVS, the sandbox is created automatically when a working copy is checked out for a CVS project using the cvs checkout [options] MODULES command. In very large projects, it does not make sense for the developers to check−out the entire source into the local sandbox. In such cases, they can take only certain modules in which they are working.

#1.4 Do not work outside the sandbox

The sandbox can be thought of as a controlled area within which CVS can track for changes made to the various source files. Files belonging to other developers will be automatically updated by CVS in the developer's sandbox. Thus the developer who lives within the sandbox will stand to gain a lot of benefits of concurrent development.

#1.5 Cleanup after completion

Make sure that the sandbox is cleaned up after completion of work on the files. Clean up can be done in CVS by using the cvs release [-d] [DIRECTORIES] command. This ensures that no old version of the files exists in the development sandbox.

#1.6 Check−in often

To help other developers keep their code in sync with your code, you must check−in (commit) your code often into the CVS repository. The best practice would be to check−in soon as a piece of code is completed, reviewed and tested, check−in the changes with command cvs commit [options] [-m LOG_MESSAGE | -F FILE] [-r revision] [FILES] to ensure that your changes are committed to the CVS repository.

CVS promotes concurrent development. Concurrent development is possible only if all the other developers are aware of the ongoing changes on a regular basis. This awareness can be termed as "situation awareness". One of the "bad" practices that commonly occur is the sharing of files between developers by email. This works against most of the best practices mentioned above. To share updates between two developers, CVS must be used as the communication medium. This will ensure that CVS is aware of the changes and can track them. Thus, audit trail can be established if necessary.

When you commit a change to the repository, make sure your change reflects a single purpose: the fixing of a specific bug, the addition of a new feature, or some particular task. Your commit will create a new revision number which can forever be used as a name for the change. You can mention this revision number in bug databases, or use it as an argument to CVS merge should you want to undo the change or port it to another branch.

#1.7 Add/Commit data in proper way

CVS is not good in handling directories. Hence once any directory is added, can't be removed from repository in normal way. Hence be careful when dealing with directories.

Moreover CVS tend to exclude empty directories while checking out any module. Which means any directory that is supposed to be empty at check-out time wont be included in checked-out copy of module. This problem often occurs when any directory is used to store temporary files which are not required to keep in CVS. Hence if directory is not present in checked-out module, your local sandbox might not work as expected. To solve this problem an empty file called .keepme can be added to empty directory.

#1.8 Use the issue-tracker wisely

Try to create as many two-way links between CVS changesets and your issue-tracking (gForge, Bugzilla, Mantis etc.) database as possible:

If possible, refer to a specific issue ID in every commit log message. When appending information to an issue (to describe progress, or to close the issue) name the revision number(s) responsible for the change.

#2 Branching and Merging

Branching in CVS splits a project's development into separate, parallel histories. Changes made on one branch do not affect the other branches. Branching can be used extensively to maintain multiple versions of a product for providing support and new features.

Merging converges the branches back to the main trunk. In a merge, CVS calculates the changes made on the branch between the point where it diverged from the trunk and the branch's tip (its most recent state), then applies those differences to the project at the tip of the trunk.

#2.1 Know when to create branches

This is a hotly debated question, and it really depends on the culture of your software project. Rather than prescribe a universal policy, we'll describe three common ones here.

#2.1.1 The Never-Branch system

(Often used by nascent projects that don't yet have runnable code.) Users commit their day-to-day work on /trunk. Occasionally /trunk "breaks" (doesn't compile, or fails functional tests) when a user begins to commit a series of complicated changes.

Pros: Very easy policy to follow. New developers have low barrier to entry. Nobody needs to learn how to branch or merge.

Cons: Chaotic development, code could be unstable at any time.

Note: this sort of development is a bit less risky in Subversion than in CVS. Because Subversion commits are atomic, it's not possible for a checkout or update to receive a "partial" commit while somebody else is in the process of committing.

#2.1.2 The Always-Branch system

(Often used by projects that favor heavy management and supervision.) Each user creates/works on a private branch for every coding task. When coding is complete, someone (original coder, peer, or manager) reviews all private branch changes and merges them to /trunk.

Pros: /trunk is guaranteed to be extremely stable at all times.

Cons: Coders are artificially isolated from each other, possibly creating more merge conflicts than necessary. Requires users to do lots of extra merging.

#2.1.3 The Branch-When-Needed system

Users commit their day-to-day work on /trunk.

Rule #1: /trunk must compile and pass regression tests at all times. Committers who violate this rule are publicly humiliated.

Rule #2: a single commit (changeset) must not be so large so as to discourage peer-review.

Rule #3: if rules #1 and #2 come into conflict (i.e. it's impossible to make a series of small commits without disrupting the trunk), then the user should create a branch and commit a series of smaller changesets there. This allows peer-review without disrupting the stability of /trunk.

Pros: /trunk is guaranteed to be stable at all times. The hassle of branching/merging is somewhat rare.

Cons: Adds a bit of burden to users' daily work: they must compile and test before every commit.

#2.3 Assign ownership to Trunk and Branches

The main trunk of the source tree and the various branches should have a owner assigned who will be responsible for.

#2.3.1 Keeping the list of configurable items for the branch or trunk

The owner will be the maintainer of the contents list for the branch or trunk. This list should contain the item name and a brief description about the item. This list is essential since new artifacts are always added to or removed from the repository on an ongoing basis. This list will be able to track the new additions/deletions to the repository for the respective branch.

#2.3.2 Establishing a working policy for the branch or trunk

The owner will establish policies for check−in and check−out. The policy will define when the code can be checked in (after coding or after review etc.,). Who is responsible to merge changes on the same file and resolve conflicts (the author or the person who recently changed the file).

#2.3.3 Identifying and document policy deviations

Policies once established tend to have exceptions. The owner will be responsible for identifying the workaround and tracking/documenting the same for future use.

#2.3.4 Merging with the trunk

The branch owner will be responsible for ensuring that the changes in the branch can be successfully merged with the main trunk at a reasonable point in time.

#2.3 Tag each release

As part of the release process, the entire code base must be tagged (by cvs tag [options] SYMBOLIC_TAG [FILES] command) with an identifier that can help in uniquely identifying the release. A tag gives a label to the collection of revisions represented by one developer's working copy (usually, that working copy is completely up to date so the tag name is attached to the "latest and greatest" revisions in the repository).

The identifier for the tag should provide enough information to identify the release at any point in time in the future. One suggested tag identifier is of the form.

release_{major version #}_{minor version #}

Checkout the entire codebase using the tag, and then proceed to go through a build / deploy / test process before making the actual release. This will absolutely ensure that what "leaves the door " is a verified and tested codebase.

#2.4 Create a branch after each release

After each software release, once the CVS repository is tagged, a branch has to be immediately created. This branch will serve as the bug fix baseline for that release. This branch is created only if the release is not a bug fix or patch release in the first place. Patches that have to be made for this release at any point in time in the future will be developed on this branch. The main trunk will be used for ongoing product development.

With this arrangement, the changes in the code for the ongoing development will be on the main trunk and the branch will provide a separate partition for hot fixes and bug fix releases.The identifier for the branch name can be of the form.

A branch can be created using cvs tag -b BRANCH_NAME command.

#2.5 Make bug fixes to branches only

This practice extends from the previous practice of creating a separate branch after a major release. The branch will serve as the code base for all bug fixes and patch release that have to be made. Thus, there is a separate repository "sandbox" where the hot fixes and patches can be developed apart from the mainstream development.

This practice also ensures that bug fixes done to previous releases do not mysteriously affect the mainstream version. In addition, new features added to the mainstream version do not creep into the patch release accidentally.

#2.6 Make patch releases from branches only

Since all the bug fixes for a given release are done on its corresponding branch, the patch releases are made from the branch. This ensures that there is no confusion on the feature set that is released as part of the patch release.. After the patch release is made, the branch has to be tagged using the release tagging practice (see Tag each release).

#2.7 Merge branch with the trunk after release

After each release from a branch, the changes made to the branch should be merged (by cvs update -j BRANCH command) with the trunk. This ensures that all the bug fixes made to the patch release are properly incorporated into future releases of the application.

This merge could potentially be time consuming depending on the amount of changes made to the trunk and the branch being merged. In fact, it will probably result in a lot of conflicts in CVS resulting in manual merges. After the merge, the trunk code base must be tested to verify that the application is in proper working order. This must be kept in mind while preparing the project schedule.

In the case of changes occurring on branches for a long period, these changes can be merged to the main branch on a regular basis even before the release is made. The frequency of merge is done based on certain logical points in the branch's evolution. To ensure that duplicate merging does not occur, the following practice can be adopted.

In addition to the branch tag, a tag called {branch_name}_MERGED should be created. This is initially at the same level as the last release tag for the branch. This tag is then "moved" after each intermediate merge by using the −F option. This eliminates duplicate merging issues during intermediate merges.

#3 Summary

it is not said that all of the above mentioned policies must be followed as they are there. There are always exceptions depending upon type of project, amount of work that is to be done. Hence policies need to be followed under most suitable way.

Localization

2007-02-27T12:40:00.000+05:30

#1 L10N overview

Internationalization and Localization are means of adapting products such as publications, hardware and software for non-native environments, especially other nations and cultures.

When you implement i18n, l10n automatically knocks your door to get implemented. Reason is that if you allow users from different countries and cultures to use your software, they will expect that apart from language transformation, real data should also get transformed into localized ways. This expectation is reasonable because it might be possible that different countries are using different standards for displaying dates, currencies, units etc. For example US people won't understand Kilometers because they use Mile as unit, however for Indian people Kilometer is quite familiar unit. The more the cultures, the more the varieties one can see in communication, displaying information etc.

#1.1 Locale and formats

Before implementing l10n we first need to have proper understanding of terms locale and format. Locale represents a whole culture that can contain information about how to display dates, how to show currencies, which measurements units are to be used for conversions etc. For example for Indian locale, it is like below:

Date display:DD-MM-YYYY
Currency: 1,11,111
Unit to measure distance: Km

While for US, it can be like below:

Date display: MM-DD-YYYY
Currency: 111,111
Unit to measure distance: Mile

Hence locale should be seen in broader way as it represents set of various localized items. But sometimes apart from locale, users prefer more customization in locals, hence there comes the term format which means allowing 1 more level of customization. For example a US user might like to format date from 02-01-2007 to February 1st 2007. In short locale includes language, default format, glyph and other instruction set for particular locale while format is nothing but the different representation of same values. Hence overall a locale can have more than 1 formats.

#2 How to implement it?

While implementing l10n in softwares, software administrator or team members first need to determine that how many locales and formats should be used. Chosen locale and formats can be stored in file system or in database. Once it is decided, it can be implemented at 2 levels.

#2.1 Backend

Each software normally has administrative area from where whole software is managed. This area should be used to select number of formats for respective locales for particular software.

From selected formats, there should be chosen 1 default format for each l10n entity which will be used at client area. This default format is applicable to whole client area of software until it is overridden at user-level.

For certain entities like number format and currency format only 1 format should be set and user-level option may not be allowed. It should also be kept in mind that formats of one locale should not be used in other locale.

For softwares likes FS and Flog where client is registered from administrative area, user-level locales and formats could be selected directly.

#2.2 Frontend

At client area, if user is not given option to set his/her own locales & formats or if it is provided but user is not logged in then locale and formats set as default at administrative area should be used.

#2.2.1 Setting locale

While implementing l10n in PHP based softwares, developers need to set locales first. This locale can be decided upon selection of language. For example if English language is selected by user then locate should be set as en_US, for Finnish language locale should be set as fi_FI. To set locale in PHP, you can use function setlocale(). You can set locale for various categories like to display monetary items, or dates or messages etc. Please refer PHP manual for more details about how to use this function.

There are various PHP functions which behaves depending upon locales. Some of them are strcoll(), strftime(), date() etc.

#2.2.2 Displaying data in various formats based upon locales

Once locale is set for particular language, locale related functions behaves in different ways. For example below code will display day in different language for different locales. You can see that code remains same but information displays in different way.

// Displays “Wednesday” for English language.
setlocale(LC_TIME,'C');
echo strftime('%A');

// Displays “keskiviikko” for Finnish language.
setlocale(LC_TIME,'fi_FI');
echo strftime('%A');

// Displays “mercredi” for French language.
setlocale(LC_TIME,'fr_FR');
echo strftime('%A');

// Displays “Mittwoch” for German language.
setlocale(LC_TIME,'de_DE');
echo strftime('%A');

// Displays “बधवार” for Hindi language.
setlocale(LC_TIME,'hi_IN');
echo strftime('%A');

Similarly locales can be set for entities like currency, number format etc. To set locale for all entities, constant LC_ALL should be used.

At code level there might be problems during implementing different formats because for different locales default formats can be different. Hence above code doesn't actually serve our purpose. See example below.

// Displays 'Friday December 22 1978' in English.
setlocale(LC_ALL, 'en_US');
echo strftime('%A %B %d %Y', mktime(0, 0, 0, 12, 22, 1978))."\n";

// Displays 'perjantai 22 joulukuu 1978' in Finnish.
setlocale(LC_ALL, 'fi_FI');
echo strftime('%A %d %B %Y', mktime(0, 0, 0, 12, 22, 1978))."\n";

// Displays 'vendredi 22 d�embre 1978' in French.
setlocale(LC_ALL, 'fr_FR');
echo strftime('%A %d %B %Y', mktime(0, 0, 0, 12, 22, 1978))."\n";

// Displays 'Freitag 22 Dezember 1978' in German.
setlocale(LC_ALL, 'de_DE');
echo strftime('%A %d %B %Y', mktime(0, 0, 0, 12, 22, 1978))."\n";

// Displays '22 दिसमबर शकरवार 1978' in Hindi.
setlocale(LC_ALL, 'hi_IN');
echo strftime('%d %B %A %Y', mktime(0, 0, 0, 12, 22, 1978))."\n";

In this example there are different formats for different locales, hence to make implementation easy at code level, we should store conversion specifier into database/file and using it directly into function. For example for Finnish languages locale, conversion specifier %A %d %B %Y would stored as string and should be used directly into function like above. Similarly this type of conversion specifiers can be used for all formats of all entities.

#3 Limitations of l10n

Native support of l10n in script or database is limited to display information into different format and glyphs only. It doesn't actually convert values according to localization. For example if price of any item is stored in $ currency, then that price, when displays to users who has selected Finnish language (or locale), wont get displayed automatically into his/her own chosen currency (i.e. €). This is because conversion rates between 2 units gets constantly changed.

For such issues, l10n should be implemented in customized way in your software where unit conversion functions can be built and used according to chosen format. However information should be stored in database in only one format and should be formatted only while displaying it to users.

However there is one one exception in displaying date and time, which can be displayed with different values if time zone related functions are used. Normally software logs date and time into it's own locale, but it could be possible that the user who is using that software located in different country where date/time is different than server time. Hence in such cases software should provide option to select timezone so that date/time can be displayed with localized values. Such option is essential for softwares that provides email services.

#4 Links

http://en.wikipedia.org/wiki/l10n
http://www.useit.com/alertbox/9608.html

Internationalization

2007-02-23T17:16:00.000+05:30

#1 I18N overview

Internationalization and localization are means of adapting products such as publications, hardware and software for non-native environments, especially other nations and cultures.

I18n includes many non-english and non-european languages like Hindi, Gujarati etc. that require multi bytes to store characters. To support such languages, software should use utf-8 encoding scheme to input, process, store, search and output data in same language.

#2 How to implement it?

Just few years ago i18n was headache for developers to implement because of limited support from database, scripting language, browsers, OS and other middle layers. But now a days with transparent support of utf-8 at each layer it has been easy to implement i18n.

In this document I have mentioned steps about how to implement i18n for LAMP based softwares with MySQL 4.1 and higher and PHP 5.0 and higher. For lower versions than these certain steps may not work.

#2.1 Server side

This sections includes changes are to be made at server side.

#2.1.1 OS level

At OS level, the only requirement is that OS should support utf-8 encoding which modern OSes like FC1...6, CentOS etc. support very well.

#2.1.2 Database level

At database level, you should use utf-8_* encoding as default for database communication and collation. For that you need to add following entries into my.cnf (MySQL software's configuration file) file at section [mysqld].

# To support Asiatic languages use utf-8. 

init-connect='SET NAMES utf8' 

default-character-set=utf8 

Sometimes customized configuration my.cnf has more sections like [client] etc. In that case add below entry in [client] section also.

# To support Asiatic languages use utf-8. 

default-character-set=utf8 

After making above entries, restart MySQL service. Whenever you create new database, use connection collation as utf8_* and use utf-8 as character set of the file. However if you have set above 2 values then these changes are not required, but still it is advised to check them as sometimes you are dumping databases from different versions of MySQL.

To test what is set; run below SQL queries.

SHOW VARIABLES LIKE 'character_set%';

SHOW VARIABLES LIKE 'collation%';

It will output all values containing utf-8 word in it. Sometimes it is not possible to add above type of entries in my.cnf specially on shared hosting server. In such case, execute below SQL query before execution of any query (in your PHP script).

SET NAMES 'utf8';

It does same thing for which we added entries in my.cnf except that this is runtime and applicable in local scope only. However database and tables must be created in utf-8 format and using same as collation.

#2.1.3 PHP script level

This is the main area where important changes are to be made. PHP natively doesn't support handling of i18n hence we have to use certain extensions to fulfill our requirements. These extensions are iconv and mbstring. But from these 2, mbstring is popular and works very well. As mbstring extension is not part of standard PHP installation, we need to enable it manually.

If you have configured your web server using utilities like YUM then it is very easy. Just run below command as root user and restart httpd service.

[root@mypc ~]# yum install php-mbstring 

For manual installation, you need to set following option to enable all the supported languages.

–enable-mbstring=all 

Once this extension is enabled in PHP, we need to set certain directives to make it working. These directives can be set in php.ini for global usage, httpd.conf for host wise usage and in PHP script itself for page or project wise usage. But I recommend to use it in PHP script itself so that it's usage remains limited to specific application or project. See below section for implementation.

Apart from these changes, all your PHP and other required scripts should be saved in utf-8 character set encoding because we store static i18n data in flat files (like fl_fi.inc.php). Hence editors needs to be configured in that way. Most popular editors provide setting of character set encoding; hence select utf-8 as standard encoding type.

#2.1.4 Application level

At application level, you need set certain directives using function ini_get() to enable mbstring extension. These directives are:

// Settings for i18n support. 

ini_set('mbstring.internal_encoding','utf-8'); 

ini_set('mbstring.func_overload',7); 

ini_set('mbstring.encoding_translation',1); 

If your application is using output buffering (i.e. output_buffering=On) then output handler should be set as mb_output_handler like below:

ini_set('output_handler','mb_output_handler'); 

Once these directives are set, all string related, regular expression related and mail system related functions will work transparently for all kind of languages. For more information about above directives, refer PHP manual.

While sending emails, utf-8 should be set as character set encoding in headers. Similarly while outputting HTML/XHTML to browser you need to explicitly set character set encoding into meta tags like below:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

Moreover sometimes if your application outputs contents to browser directly from running script even then you need to pass certain header like below:

In short wherever and whenever character set encoding is required, pass utf-8 as encoding scheme.

If you are using Ajax mechanism in your website then you also require to encode your Ajax query string before submitting to server for execution. This is specially required while running application in MSIE browsers. Firefox and other Gecko based browsers handles Ajax queries that contains i18n data correctly. For more info read thread http://news.php.net/php.i18n/1059.

#2.2 Client side

At client side minimal support is required.

#2.2.1 OS level

Client's OS should be able to understand utf-8 encoding, which is quite possible in latest versions of OSes. Apart from that, OS should have support to type characters in native language which is called as Keyboard layout and Regional settings.

#2.2.2 Browser level

The only requirement at browser level is that browser should be able to interpret utf-8 character set encoding which is supported by almost all modern browsers. Contents sent from server (refer section 2.1.4) tell browser that which encoding scheme is to be used.

#3 How to implement in existing projects?

To implement i18n in existing projects requires above implementation plus conversion of existing data.

#3.1 Converting database

#3.1.1 Using commands

To convert existing database encoded into utf-8, take dump of data by running below command.

mysqldump -hHOST -uUSER -p --opt --default-character-set=CHARSET --skip-set-charset DB_NAME|sed -e 's/SOURCE_CHARSET/DEST_CHARSET/g' > DB_NAME.sql 

It will create SQL file containing all data of selected database which you can use to re-create new database. Now convert existing file into utf-8 format by running below command.

iconv -f SOURCE_CHARSET -t UTF-8 DB_NAME.sql > UTF_DB_NAME.sql 

To dump above data into newly created database, run below command:

mysql -hHOST -uUSER -p --default-character-set=DEST_CHARSET DB_NAME < 

UTF_DB_NAME.sql 

#3.1.2 Converting database directly

Sometimes it is not possible do this using command line interface (due to shared hosting environment), then it will require to execute certain SQL queries to change character set of existing database without copying or moving it anywhere. To do so; run below SQL queries wherever it is applicable. Please note the order of execution of queries.

# First change all i18n fields of all tables into BLOB.

ALTER TABLE TABLE MODIFY FIELD BLOB; 

# Now change character set of database as UTF-8. 

ALTER DATABASE DATABASE charset=utf8; 

# Then change character set of each table.

ALTER TABLE TABLE charset=utf8; 

# Now change all i18n fields of all tables into UTF-8 character set.

ALTER TABLE TABLE MODIFY FIELD ORIG_FIELD_TYPE CHARACTER SET utf8; 

Ideally above queries should be run by making shell or PHP script so that it can be used later or for other projects.

#3.2 Converting file system

You will also require to change encoding of existing PHP scripts and other files if they contain data that requires utf-8 encoding. Normally you would require to change format of files containing non-english text only. But it is recommended to use same encoding for all type of files throughout your software. To change encoding of file iconv utility can be used. One example is provided below.

iconv -f SOURCE_CHARSET -t UTF-8 FILENAME > FILENAME 

You can also use mb_string_* functions of PHP to change character set encoding of your files. However this is PHP based function so you need to design a script to convert all existing files.

#3.3 Translating your project

When you design multilingual projects, it becomes mandatory to dynamically handle static text of your application. Your application can have several options for that depending upon tools used to build it. Symfony based projects can have XLIFF as standard mechanism to translate static data. But overall if we want such mechanism in every PHP application then there are 2 ways to do so.

Declaring PHP variables where static text gets displayed, and then putting those variables in language files, according to each language used in application, and then including it in scripts where those translations are required. This is standard way but there are some disadvantages of it. Since file is PHP based, third party translators will find difficulties in adding/updating translations as they might be from non-technical area. Hence there exists professional mechanism to overcome this problem.
For large applications, where translators are from various cultures and backgrounds, preferred approach is it to use PHP extension called Gettext (in PHP it is bundled as php-gettext).

To check whether it is installed or not, run command php -m from command line. If output contains text like gettext then it is installed. Otherwise using various options it can be installed. The most popular way is using Yum utility. Just type below command as root user and that's it.

[root@mypc ~]# yum install php-gettext 

php-gettext works in this way. 1 language file is prepared using specials editors like KBabel, poEdit where all translations are kept in pairs of 2 special variable msgid and msgstr. msgid variable denotes variable name to be used in PHP script while msgstr denotes language translation associated with that msgid variable. Using pair of msgid and msgstr, we can build as much translations as required for any language. There will be separate files for each language. This version of file has .po extension but it is only for editing translations. Actual file which is used by PHP to apply these translations is not this. For that .po file needs to get compiled into .mo binary file. For that following command can be used.

[user@mypc ~]# msgfmt -cv -o FILE.mo FILE.po 

Normally both files are kept at same location for easy maintenance. Since we have generated files for language translations, I will describe how to use them in your application. In your global includable script following code needs to get added to bind your created language files and your application.

// Set environment variable. 

putenv('LC_ALL=en_US'); 

// Set selected locale (language). 

setlocale(LC_ALL, en_US); 

// Specify location of translation tables. 

bindtextdomain('frontend', '/web/projects/myproject/translations/'); 

bindtextdomain('backend', '/web/projects/myproject/translations/'); 

// Choose domain for application. 

textdomain('frontend'); 

// Bind specific character set to be used with selected domain. 

bind_textdomain_codeset('frontend','UTF-8'); 

bind_textdomain_codeset('backend','UTF-8'); 

In above code snippet, items marked in bold are to be replaced by your application specific needs. Here en_US is a locale which can be changed as required.

Since gettext accesses translation files in special way, they needs to get stored according to rules defined by gettext. With reference to above code-snippet, if root folder of language translation files is like /web/projects/myproject/translations/ and application areas are admin and client then language related files should be stored in following way.

/web/projects/myproject/translations/en_US/LC_MESSAGES/backend.mo, [admin.po]
/web/projects/myproject/translations/en_US/LC_MESSAGES/frontend.mo, [client.po]

/web/projects/myproject/translations/en_GB/LC_MESSAGES/backend.mo, [admin.po]
/web/projects/myproject/translations/en_GB/LC_MESSAGES/frontend.mo, [client.po]

/web/projects/myproject/translations/fi_FI/LC_MESSAGES/backend.mo, [admin.po]
/web/projects/myproject/translations/fi_FI/LC_MESSAGES/frontend.mo, [client.po]

Hence by switching values in function textdomain(), desired translation files can be included in each application. Now how to use these translation in PHP script?. For that each msgid variable should be written like _('myVar') in PHP script to include corresponding translation from selected file.

That's it, whenever translation files are modified, they need to get recompiled using command msgfmt as described earlier to avail latest translations. Since compiled binary .mo files are cached by PHP, modifications might not get reflected immediately. In such case web service should be gracefully restarted.

#4 Summary

Advantages of implementing i18n using utf-8 character set encoding is that users can now input data in their localized language and those contents would saved and displayed back in browser in same language. Not only these, but database can make searching records in native language also. For example if you are required to retrieve records of all users whose last name is “ઝાલા”, then writing SQL like below will work successfully.

SELECT * FROM user WHERE last_name='ઝાલા'; 

At PHP script level you can compare, sort, split Unicode strings in same way like you are doing for normal strings. Next version of PHP (i.e PHP 6) is going to support Unicode by default hence there will not require extensions or setting to enable Unicode string.

#5 Links

http://www.nyphp.org/content/presentations/smallworld/April2006-nyphp-Presentation.ppt
http://en.wikipedia.org/wiki/I18n
http://www.useit.com/alertbox/9608.html
http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html

REST

2006-11-29T17:14:00.000+05:30

#1 REST overview

Representational State Transfer or REST is a style of software architecture for distributed hypermedia systems such as the World Wide Web. The term was originated by Roy Fielding, one of the principal authors of the HTTP protocol specification.

Roy Fielding describes REST as:

Representational State Transfer is intended to evoke an image of how a well-designed Web application behaves: a network of web pages (a virtual state-machine), where the user progresses through an application by selecting links (state transitions), resulting in the next page (representing the next state of the application) being transferred to the user and rendered for their use.

The term is also often used in a loose sense to describe any simple interface that transmits domain-specific data over HTTP without an additional messaging layer such as SOAP or session tracking via HTTP cookies. Systems that follow Fielding's REST principles are often referred to as RESTful.

The Web is the key example of existing RESTful design. Much of it conforms or can be made to conform to REST principles. The Web consists of various protocols like HTTP, FTP, HTTPS, SMTP etc, content types including the HTML, XML and other Internet technologies such as the Domain Name Service (DNS).

HTML can include Javascript and Applets that has implicit support for hyper-links. HTTP has a uniform interface for accessing resources which consists of URIs, Request methods (GET, POST, PUT, DELETE), Status codes (401, 404, 500), Headers, and Content distinguished by MIME type (application/plain-text, image/gif, image/png etc.).

HTTP separates the notions of a web server and a web browser. This allows the implementation of each to vary from the other based on the client/server principle. When used RESTfully, HTTP is stateless. Each message contains all the information necessary to understand the request when combined with state at the resource. As a result, neither the client nor the server needs to remember any communication-state between messages. Any state retained by the server must be modeled as a resource.

Before further reading this document, I advise to read this article http://www.xfront.com/REST.html in order to understand REST in proper way.

Let's understand REST in the by taking example of "Camera purchasing store" project. In RESTful application, only single base URL is given to client and after that each representation of resources gives more URLs to be visited and so on. To get information of cameras available on camera purchasing website, a base URL can be given like http://mycamera.com/camera. This can be called as a main resource of available cameras. Calling this resource will result into primary information of available cameras on the website. To obtain more information about each camera link to more resources are provided. They are: http://mycamera.com/camera/view/camera_id/1 and http://mycamera.com.com/camera/view/camera_id/2. Both these resources provides details of respective cameras. These details may also contain links to more resources like 1st resource contains link to another resource http://en.wikipedia.org/wiki/Nikon_D70.

Each of the above request URI is stateless and self-explanatory which means server at any point doesn't require to ask client to provide more information about the request. Moreover each resultant representation provides required information of requested resource as well hyper-link to additional resource attached to main resource.

#2 REST characteristics and principles

#2.1 REST characteristics

Characteristics of the RESTful system would be:

Client-Server: a pull-based interaction style: consuming components pull representations.
Stateless: each request from client to server must contain all the information necessary to understand the request, and cannot take advantage of any stored context on the server.
Cache: to improve network efficiency responses must be capable of being labelled as cacheable or non-cacheable.
Uniform interface: all resources are accessed with a generic interface (e.g., GET, POST, PUT, DELETE).
Named resources - the system is comprised of resources which are named using a URL.
Interconnected resource representations - the representations of the resources are interconnected using URLs, thereby enabling a client to progress from one state to another.
Layered components - intermediaries, such as proxy servers, cache servers, gateways, etc, can be inserted between clients and resources to support performance, security, etc.

#2.2 REST principles

Key principles of RESTful system are:

The key to creating Web Services in a REST network (i.e., the Web) is to identify all of the conceptual entities that you wish to expose as services.
Create a URL to each resource. The resources should be nouns, not verbs. For example, do not use this:

http://mycamera.com/camera/viewCamera.php?camera_id=1

Note the verb, viewCamera. Instead, use a noun:

http://mycamera.com/camera/view/camera_id/1

Categorize your resources according to whether clients can just receive a representation of the resource, or whether clients can modify (add to) the resource. For the former, make those resources accessible using an HTTP GET. For the later, make those resources accessible using HTTP POST, PUT, and/or DELETE.
All resources accessible via HTTP GET should be side-effect free. That is, the resource should just return a representation of the resource. Invoking the resource should not result in modifying the resource.
No man/woman is an island. Likewise, no representation should be an island. In other words, put hyper-links within resource representations to enable clients to drill down for more information, and/or to obtain related information.
Design to reveal data gradually. Don't reveal everything in a single response document. Provide hyper-links to obtain more details.
Specify the format of response data using a schema (DTD, W3C Schema, RelaxNG, or Schematron). For those services that require a POST or PUT to it, also provide a schema to specify the format of the response.
Describe how your services are to be invoked using either a WSDL document, or simply an HTML document.

#2.3 How to achieve them?

To achieve above principles while developing RESTful application, developers will require to start thinking web applications in context of resource and it's representation.

It is not difficult to achieve some of the REST principles, but it is difficult to achieve all of them. For example we hardly use PUT and DELETE methods of HTTP. Below are guidelines about how to achieve them.

Developers should be able to identify all of the conceptual entities that is to be provided as service.
These entities or resources must have a URI that can represent them.
The format of URIs should be easy to understand by users. For example prefer noun over verbs i.e use http://mycamera.com/camera/view/camera_id/1 instead of http://mycamera.com/camera/viewCamera.php?camera_id=1.
Categorize all available resources that are to be exposed so that users can easily find them and visit them one by one.
Coding should be done in such a way that all resources to be accessed by GET method should remain side-effect free i.e do not modify, delete, alter resources using parameters in GET method like http://mycamera.com/viewUser.php?id_user=1&delete=1.
In RESTful application, there is not any end-point i.e. Representation of 1 resource leads to access of another resource as shown in camera purchasing example (where list of camera get displayed first, then hyper-link to details of each camera can be accessed from there). However resources need not to be delivered from only 1 domain i.e they can be delivered from different domains also. Even in case of error/s occurring on server while accessing certain resources, server should return status code with proper information that what to do next or what steps to be followed by users so that error/s would not occur again.
While developing application, data should not get revealed altogether. Which means first generic list of resources should get displayed. That list should contain link to details of each individual resources and so on. The purpose is that daily users might not be interested in seeing details of every resource each time.
For example in camera purchasing website, URL http://mycamera.com/camera just displays list of cameras. From there, link to details of each individual camera (http://mycamera.com/camera/view/camera_id/1) or purchase of camera (http://mycamera.com/camera/purchase/camera_id/1) can be requested.
Sometimes different format of data is requested by client; in such case client application should pass desired format related instructions in request directly and server should be able to understand and deliver it according to specifications. For example if details of camera is needed in XML format then query string could be like http://mycamera.com/camera/view/camera_id/1/xml.

#3 REST and non-REST

Simplest definition between REST and non-REST system is that a system that is not following REST principles is called a non-REST system. That is there is not any specific guidelines or rules to differentiate REST vs. non-REST.

Most of web application are non-REST because it is not possible to create 100% RESTful application due to it's constraints. Often web application needs to use HTTP cookies and Session to keep track of client. Using such components violate REST principle/s (not being stateless). To make application 100% RESTful such information can not be passed in URI due to security reasons.

#4 REST and SOAP

REST can not be directly compared with SOAP as REST is an architectural style while SOAP is a message layer. However RESTful service can be compared with SOAP.

The key difference between REST and SOAP is message style. SOAP is strictly bound to XML while REST doesn't have such constraints hence can use HTML, XML or any other MIME types. Other differences between REST and SOAP are:

REST can use proxy servers in serving contents as resource can easily be identified from URL. Proxies are useful in defining policies of serving resources. Which means if certain resources are not be served, it can be directly denied from there without sending request to actual web server.

On the other hand when resource is requested through SOAP, proxy cannot determine whether resource is to be allowed or not because information of resource is hidden in message envelope. Furthermore the URL is not to the target resource, but rather to a SOAP server which means proxy server will require to look inside each SOAP message to determine what is being asked. To do so proxy server would require to understand semantics of message. Which is difficult task as each SOAP message can have different semantics. However there are other solutions like using RDF/DAML format to allow proxy server to dynamically discover the resource or method to be discovered.

Similar like proxy servers, there are other web intermediaries like gateways, caches etc. which can not be taught to understand semantics of every SOAP message.
On the contrary REST request is self-explanatory hence these web intermediaries can work effectively in serving resources.

Each client application can be considered as a state machine which means each resource representation that it receives causes it to transition to the next state.
In REST based application each representation contains hyper-links; following them puts client into next state. That is there is not endpoint for any state in REST based application.

While in SOAP based implementation each SOAP message is just data i.e no hyper-links which means which resources is to be accessed next must be made out of the band. Following 2 figures gives more details about this difference.

State Transitions in a REST-based Network

State Transitions in a SOAP-based Network

In REST based implementation, caching can play important role in saving bandwidth and increasing response time because each resources has it's own location hence cache servers can easily cache them (if defined in headers) and when new request arrives resource can be served directly from cache.

When resources is requested in SOAP system, message is always sent using POST method even if purpose is to get data. Moreover with SOAP server URI is always same and not the target resource hence a cache server would not know from the URI what resource is being requested. That is with SOAP message, the cache server cannot determine if data is being requested, nor what resource is being requested. Thus no caching possible with SOAP.

In the context of semantic web, according to vision of Tim Berners-Lee, every resource can be given a URI and in fact any resource of significance should be given a URI. The REST architectural style is consistent with this vision due to having unique location for each resource.

While with SOAP, since all requests are submitted at only one SOAP server, it breaks above vision of semantic web.

A key feature of REST (and the Web) is that every resource have a generic interface. Namely, access to every resource is accomplished using HTTP GET, POST, PUT, and DELETE.

With SOAP there is no defined set of methods. Each SOAP application is free to define its own set of methods.

The key to interoperability is standardization. The reason why independent resources on the Web are able to interoperate today is because the Web has standardized on:

Addressing and naming resources -> URI
Generic resource interface -> HTTP GET, POST, PUT, DELETE
Resource representations -> HTML, XML, GIF, JPEG, etc
Media types -> MIME types (text/html, text/plain, etc)

While SOAP much more depends upon customization. Because each SOAP message provides its own unique method of naming, a resource and each SOAP application defines its own interface.

#5 REST and Symfony

Since REST is an architectural style and not a standard or a feature, it doesn't depend upon any programming language or framework. However a well designed framework can help a lot in developing RESTful applications.

Symfony framework assists in generating Smart URLs and usage of “/” in URI to represent parent-child relationship through it's routing mechanism (together with Apache web server's mod_rewrite module). Which are important characteristics of a RESTful application.

#6 Examples of REST implementation

Our above example of "Camera purchasing store" is a RESTful web application. Hence we can see some examples of REST principles using HTTP GET and POST methods.

#6.1 Using GET method

For example if users want to retrieve information of available cameras (i.e resources in REST way) from the website, then URL http://mycamera.com/camera provides them list of them and links to see further details of each camera. You can see that it has been implemented using GET method.

This representation contains URL of each camera resource like http://mycamera.com/camera/view/camera_id/1 and http://mycamera.com/camera/view/camera_id/2. Which means data is linked to more data. By visiting above resources user might get links to more camera related resources.

Main resource also provides links to get information about purchase of each camera like http://mycamera.com/camera/purchase/camera_id/1 and http://mycamera.com/camera/purchase/camera_id/2 that is another resources are there. When any of these resources, server returns purchase made for each camera as well as link to other resources like viewing details of each person (http://mycamera.com/user/view/user_id/19) who has purchased particular camera.

#6.2 Using POST method

HTTP POST is an another popular HTTP method to manage resources for any application. As GET is used to retrieve resources from server, POST is used to add/alter/remove resources. Normally POST is used with web forms where user input is required. Hence request submitted through POST method should be handled properly at server side.

RESTful action of POST method for our website is "to purchase cameras”. Consider a base URL http://mycamera.com/camera that provides list of cameras. From this list; link to purchase particular camera can be requested. This link http://mycamera.com/camera/purchase/camera_id/1 is a resource to purchase 1st camera. It will display a form to select desired from and to dates to purchase selected camera. Upon selection and submission of data, camera will get purchased (after making payments etc.). Upon successful submission another resource http://mycamera.com/user is returned which displays confirmation of order made by this user. From this resource order history can be view and deleted if required. Cancellation of order/s (i.e resource) will require another HTTP POST operation.

#6.3 PUT and DELETE methods

DELETE and PUT methods; unlike GET and POST; are not supported by browsers. Hence we have to use POST method to update (PUT) and remove (DELETE) resources in desired way. To do so we may provide PUT and DELETE related information via hidden fields of forms.

Resource http://mycamera.com/user is an example of deleting order which is equivalent to DELETE method while resource http://mycamera.com/user/edit is an example of updating user information which is equivalent to PUT method.

#7 How to create REST based web services

Web services have been designed to share data across many platforms and hardware configurations. For example a web services created on PHP platform can be used by .NET based platform as well as messaging format between client and server is common i.e XML or other HTTP based ones.

Currently there are 3 web services which are popular on the web. They are XML-RPC, SOAP and REST. First 2 services are strictly using XML messaging format while 3rd can use standard HTML, XML, GIF or JPEG etc. In this document important guidelines have been provided to create REST based web-services.

Implementing REST web service correctly requires a resource-oriented view of the world instead of the object-oriented views many developers are familiar with.

In REST based web service resource is the key concept. Hence whole REST web service is designed around managing resources. Since resource requires some mechanism to get identified by web, they are represented by URI.

The creator of a URI decides the encoding of the URI, and users should not derive metadata from the URI itself. URI opacity only applies to the path of a URI. The query string and fragment have special meaning that can be understood by users. There must be a shared vocabulary between a service and its consumers.

A service provider should ignore any query parameters it does not understand during processing. If it needs to consume other services, it should pass all ignored parameters along. This practice allows new functionality to be added without breaking existing services. Here XML schema can be used to validate query parameters.

The resource should be delivered by either of these 4 ways.

Server-driven negotiation.
Client-driven negotiation.
Proxy-driven negotiation.
URI-specified representation.

A service represents a specialized business function. Hence it should be safe and should not incur any obligations from its invoking client, even if this service may cause a change of state on the server side.

A safe service should be invoked by the GET method of HTTP. Parameters needed to invoke the service can be embedded in the query string of a URI. The main purpose of a safe service is to obtain a representation of a resource.

If there is more than one representation available for a resource, the service should negotiate with the client as discussed above. When returning a representation, a service provider should set the HTTP headers that relate to caching policies for better performance.

Obligated services should be implemented using POST. A request to an obligated service should be described by some kind of XML instance, which should be constrained by a schema. The schema should be written in W3C XML Schema or Relax NG. An obligated service should be made idempotent so that if a client is unsure about the state of its request, it can send it again.

Web service should be asynchronous i.e it needs to :

Return a receipt immediately upon receiving a request.
Validate the request.
If the request if valid, the service must act on the request as soon as possible. It must report an error if the service cannot process the request after a period of time defined in the service contract.

The receipt element should include a received attribute, the value of which is the time the server received the request in WXS dateTime type format.

A transaction represents an atomic unit of work done by a server. The goal of a transaction is to complete the work successfully or return to the original state if an error occurs. For example, a transaction in a purchase order service should either place the order successfully or not place the order at all, in which case the client incurs no obligation.

The status resource should not vary much from it's transaction URI. For example:

Transaction URI: http://mycamera.com/xyz1234

Transaction Status URI: http://mycamera.com/xyz1234?option=status

Life cycle of transaction should be as shown in below image.

While receiving and sending XML, one should follow the principle of strict out and loose in. When sending XML, one must ensure it is validated against the relevant schema. When receiving an XML document, one should only validate the XML against the smallest set of schema that is really needed. Any software agent must not change XML it does not understand.

Typical architectural style of a REST web service could have a pipe-and-filter style. This classical and robust architectural style was used by the famous physicist, Richard Feynman, to build the first atomic bomb in his computing team. It is made of front-end filters, a connector and back-end filters as shown in below image

In this style a request is processed by a chain of filters and each filter is responsible for a well-defined unit of work. Those filters are further classified as two distinct groups: front-end and back-end. Front-end filters are responsible to handle common Web service tasks and they must be light weight. Before or at the end of front-end filters, a response is returned to the invoking client.

All front-end filters must be lightweight and must not cause serious resource drain on the host. A common filter is a bouncer filter, which checks the eligibility of the request using some simple techniques:

IP filtering. Only requests from eligible IPs are allowed.
URL mapping. Only certain URL patterns are allowed.
Time-based filtering. A client can only send a certain number of requests per second.
Cookie-based filtering. A client must have a cookie to be able to access this service.
Duplication-detection filter. This filter checks the content of a request and determines whether it has received it before. A simple technique is based on the hash value of the received message. However, a more sophisticated technique involves normalizing the contents using an application-specific algorithm.

A connector, whose purpose is to decouple the time dependency between front-end filters and back-end filters, connects front-end filters and back-end filters. If back-end processing is lightweight, the connector serves mainly as a delegator, which delegates requests to its corresponding back-end processors. If back-end processing is heavy, the connector is normally implemented as a queue.

Back-end filters are usually more application specific or heavy. They should not respond directly to requests but create or update resources.

#8.1 Determining data interchange format

Most REST web services use XML as data interchange format. However it is not necessary to use only XML as different web services might have different requirements. There can be used plain text as well as HTML.

Why XML is used the most is because it is well structured and information can easily be extracted from returned data by using ready made XML parsers. For Plain text and HTML it is difficult to extract certain piece of information unless used at whole.

#8.1.1 Data interchange format and JSON

Apart from XML, JSON is an another popular data interchange format for REST web services. It is Javascript object notation which is used at client side to interpret response in Javascript. Response provided by server can directly be used in JS if web service outputs data in JSON format. JSON:

conforms REST principles.
handles simple and complex data types,
can be used with Ajax and
can be used at server side.

Since no other middle layer processing is required, to process JSON data before using it, it is widely used in client applications. Below is given an example of REST web service using JSON as output format.

Server side implementation

// Class to convert Rupees into Euro, USD and Pound.

class Converter

{

  public function convert($curr='E', $value=0)

{

    if($curr == 'E')

      return ($value / 55);

    if($curr == 'D')

      return ($value / 44);

    if($curr == 'P')

      return ($value / 75);

...

}

}

// Create instance.

$converter=new Converter;

// Convert value. When call is made like

// "converter.php?curr=E&value=3000"

$value=$converter->convert($_GET['curr'], $_GET['value']);

// Store result into array.

$result=array('value'=> $value);

// Output JSON encoded string to be used by JS.

echo 'var result='.json_encode($result);

Client side implementation

[html]
[head]
[script type="text/javascript" src="converter.php?curr=E&value=3000"]
[/script]
[/head]

[body]
[script type="text/javascript"]
var json=eval(result);
alert(json.value) // Alerts "54.54".
[/script]
[/body]
[/html]

#8.2 Building API

REST API should respect REST principles. Hence while building API, following technical points should be kept in mind.

URL of REST web service API should be easy to read and understand and should be self-descriptive. For example for our example project project, URL could be like http://rest.mycamera.com or http://webservice.mycamera.com.
API can be designed using existing structure of the project so that project's common libraries can also be used.
REST doesn't constrain how request should be handled at back-end. Hence developers have full control and convenience over design and implementation of API. Therefore only single controller can be used to understand incoming requests. This controller then can call other scripts or code to generate proper response in order to satisfy request.
Conventions used in scripts and resources should be self-descriptive and easy to navigate in between. For example resource that provides information of particular user could be as simple as like http://rest.mycamera.com/user/XXXX (where XXXX is ID of user). Similarly information of 1st album of this user can be represented by resource like http://rest.mycamera.com/user/XXXX/album/YYYY (where YYYY is ID of 1st album). However for same resource, there can be updation or deletion operations also. This can be determined by type of HTTP request. For example, for URL http://rest.mycamera.com/user/XXXX, if GET method is used then it means information of user has been requested. But if PUT (sometimes POST) method is used then information is to be updated and if DELETE method is used then resource is to be deleted (i.e user is to be deleted).

As we know that handling above type of resources is not easy at server side due to web server constraints, hence frameworks like Symfony (together with URL rewriting mechanism called mod_rewrite for Apache web server) can be used to easily extract operation and values from resources.

API should be well documented.
If data transferring between client and server are sensitive then secure HTTP i.e https should be used.

#8.3 Authenticating users

It is often required to ask users to provide authentication before allowing them to access web service or certain part of web service. There are various ways for to accomplish this task. Since REST architectural style is stateless, we can't ask users to provide valid credentials every time they submit requests. We can have 3 types of solutions.

Standard solution of this problem is to use HTTP authentication. Once user is authenticated, he/she will just need to provide valid token for validation instead of providing credential every time. This token can then be used to response subsequent requests.
Another approach is to use Session mechanism that PHP provides. Once API validates users' credentials, it should create session for associate user. PHP then issues a session ID (referred as PHPSESSID) that can be used to associate particular user with existing session on server. This session ID is stored in form of cookie at user side (if user has enabled cookies) and will be read directly from there by PHP's session mechanism. If cookies are disabled (or not accepted) then this session ID can be appended to query string each time new request is made to web service. PHP stores sessions in form of files at specified folders (usually /tmp). This session ID stores user's information such as browser's user agent, IP address of user's machine etc. which means even if someone else tries to use this session ID to act as authenticated user, he/she can't do so. Moreover, by default, upon inactivity, these session IDs are destroyed after 20 minutes which means in future also none can use it.
3rd solutions is to use customized interface to authenticate users. For that database storage system can be used where user's information can be stored. When user submits request 1st time, he/she will require to provide ID and Password to log on web service. At server side API will do authentication and will issue a unique hash or token back to user to be used for subsequent requests. This hash will be appended to request URL every time new request will be made. At server side this hash or token will be stored in database along with other information of user for future validations. By this way we can increase/decrease inactivity time out, can store more information about user and can retain certain information of client for logging purpose etc.

#9 Summary

There are many components of web that exist since long time but were not utilized in best way. REST and Ajax are examples of them. Few years ago web was simple hence people were using REST principles unknowingly. At that time it was not required to have look at them in ordered way. But when web became complex flooding with so many complex techniques, it is again required to make things simple.

REST has been recalled to make things simple. Currently REST is gaining rapid growth because of it's simplicity of implementation and usage over XML-RPC and SOAP. Hence it is the key of next generation web services. Overall, the thing to keep in mind is that REST is about exposing resources through URIs, not services through messaging interfaces.

#10 Links

http://www.xfront.com/REST.ppt
http://en.wikipedia.org/wiki/Representational_State_Transfer
http://rest.blueoxen.net/cgi-bin/wiki.pl?FrontPage
http://rest.blueoxen.net/cgi-bin/wiki.pl?RestFaq
http://www.xfront.com/REST-Web-Services.html
http://www.w3.org/DesignIssues/Axioms.html
http://developer.yahoo.com/maps/rest/V1/geocode.html
http://www.onlamp.com/pub/a/php/2003/07/03/php_amazon_soap.html http://www.onlamp.com/pub/a/php/2003/10/30/amazon_rest.html http://webservices.xml.com/pub/a/ws/2004/03/24/phpws.html
http://www.xml.com/pub/a/2004/08/11/rest.html
http://ajaxpatterns.org/archive/RESTful_Service.php

Various types of caching

2006-09-06T22:11:00.000+05:30

#1 Caching overview

Caching is one of the important techniques to gain better performance in serving contents and increasing response time of your web sites. There are several types of caching mechanism that can be used to cache HTML, Media, Raw scripts, Compiled scripts etc. Let's understand what can be cached at what level from below diagram.

A request traverses through 3 components viz. Client, Internet and Server environment. At each component there can be implemented various caching mechanism according to requirement.

At client level browsers are good candidates of caching static media like CSS, Images, JS, Videos etc. Most browsers take such data from cache instead of requesting to serve as fresh copy. They handle caching of such media automatically hence no special instructions or headers are required to tell browser to how to cache such media. However browsers tend to cache almost everything that it renders hence special HTTP headers are passed to inform them to not to cache such items (usually PHP handles this automatically). Browser cache can be also managed by web server by sending certain HTTP headers if browsers are not behaving in normal ways. For that, web server module like mod_expire (for Apache) is used send cache control related HTTP headers. More details about mod_expire can be found later section.
Internet is intermediate between client and server which includes Gateways, ISPs, Proxies and several other components. At this level, ISP and Proxy cache can be implemented to cache pages, but since it is not in control of client or server it is not good option of caching.
Server environment is made of components like Apache web server, PHP scripting language, Middlewares like frameworks and database like MySQL. For each component, there exists separate caching mechanisms to boost performance of web site. Let's discuss about each in detail.

#2 Levels of caching in server environment

#2.1 Web server (Apache)

Certain web servers like Apache, Lighttpd etc. provide built in modules for caching contents at web server level. They are mod_cache, mod_disk_cache, mod_expire, mod_file_cache, mod_mem_cache.

#2.1.1 mod_cache

This module of Apache web server required to implement caching of HTTP contents. However it is always used along with other caching modules like mod_disk_cache or mod_mem_cache depending upon requirements. Detailed information about this module can be found from here.

#2.2.2 mod_disk_cache

This module is used to cache files using disk based storage manager i.e cached files are stored on disk on URI based keys and when same URI is requested, cache content is served directly from cache. This mechanism is similar like how browser caches entire page on local disk. The only difference here is that content is stored on server instead of client. Detailed information about this module can be found from here.

#2.2.3 mod_file_cache

This module provides two techniques for caching frequently requested static files. Through configuration directives, mod_file_cache can be directed to either open then mmap() a file, or to pre-open a file and save the file's open file handle. Both techniques reduce server load when processing requests for these files by doing part of the work (specifically, the file I/O) for serving the file when the server is started rather than during each request.

Not all platforms support both techniques, hence it is to be found out that which technique will work for specific environment.

This mmap() ing is done once at server start or restart, only. So whenever one of the mapped files changes on the filesystem, web server is to be restarted. However if the files are modified in place without restarting the server, only cached contents will be served. Hence files should get updated by unlinking the old copy and putting a new copy in place. Most tools such as rdist and mv do this. The reason why this modules do't take care of changes to the files is that this check would need an extra stat() every time which is a waste and against the intent of I/O reduction.

For certain systems, configuration and global files are ideal candidates for this module because they do not get changed frequently. This mechanism should not be used to cache media files as they can be effectively cached at client side. Detailed information about this module can be found from here.

#2.2.4 mod_mem_cache

As opposite to mod_disk_cache, mod_mem_cache implements memory based caching of contents which provides faster access to cached contents than disk. This module can be implemented in 2 ways;

by caching open file descriptors or
caching objects in heap storage. This module is most useful when it is used to cache locally generated content or to cache backend server content for mod_proxy configured as reverse proxy. Content is stored in and retrieved from the cache using URI based keys.

This module can be used to store session files in memory to share session across different services of particular system. Detailed information about this module can be found from here.

#2.2.5 mod_expire

This module controls the setting of the Expires HTTP header and the max-age directive of the Cache-Control HTTP header in server responses. The expiration date can set to be relative to either the time the source file was last modified, or to the time of the client access.

These HTTP headers are an instruction to the client about the document's validity and persistence. If cached, the document may be fetched from the cache rather than from the source until this time has passed. After that, the cache copy is considered "expired" and invalid, and a new copy must be obtained from the source.

This module is not useful when caching is turned off from browser because nothing is cached at server side. Since most browsers automatically handle caching of static media, this module may not be as useful as it seems. Detailed information about this module can be found from here.

#2.2.6 Reverse proxy (Varnish)

See here.

#2.2 Scripting language (PHP)

#2.2.1 Opcode/Bytecode cache

Opcode/Bytecode caching is caching your PHP script into compiled state so that when new request arrives for same script, cache software will server compiled version of code directly from cache rather than reading file from the disk and then compiling. Some examples of opcode caching softwares are Zend Platform, APC (APC GUI), XCache, eAccelerator, ionCube Encoder and PHP Accelerator. From these 6, eAccelerator, XCache and APC are widely used caching softwares. This benchmarks also show that how XCache and APC are better than others.

#2.3 Middleware (Symfony)

Symfony framework provides 3 types of caching of data. They are HTML cache of output, cache of configuration scripts and cache of translated templates. Last 2 mechanisms are automatic and handled by Symfony directly.

#2.3.1 HTML cache

HTML cache is nothing but output of script that is sent to browser for display. A web page is made of many sections. These sections are mostly designed in separate templates and files. Hence depending upon type of output, different type of HTML caching can be enabled in Symfony environment. Caching can be enabled by changing off => on in settings.yml file as shown below.

prod:

dev:

.settings:

cache : on

There are 5 types of HTML cache that can be implemented in Symfony. They are Action, Partial, Component, Page and Fragment of template. Each type is useful depending upon type of data. Enabling each type of cache requires specific settings in configuration files at various levels. For more details, please refer to Symfony API manual.

In modern web applications, most of data is dynamic hence great care should be taken while enabling caching in particular action or page.

However following care should be taken before implementing HTML cache in Symfony.

Setting less time of caching data may not effectively boost performance, similarly setting long time than required may result in annoyance of users since they could be viewing old data.
Structure of page should be divided into sections like which section can be cached and which can not be so that cached part can easily be identified and handled.
Enabling or Disabling cache should not be entirely depending upon just any programmer, instead it should be well discussed before making any change in it.
Static templates should not be cached as anyway they do not require extra processing time.
Cache should get cleared regularly either by automatic or manual way.

#2.3.3 Template Translations cache

Multilingual projects requires separate language files for each language. These language translation files are also cached by Symfony. Location of these cached files is /..../PROJECT/cache/i18n/. This cache is automatically handled by Symfony hence at code level nothing is to be done.

#2.4 Database (MySQL)

At database level (specially MySQL), query cache can be implemented which stores the text of a SELECT statement together with the corresponding result that was sent to the client. If an identical statement is received later, the server retrieves the results from the query cache rather than parsing and executing the statement again.

The query cache is extremely useful in an environment where you have tables that do not change very often and for which the server receives many identical queries. This is a typical situation for many web servers that generate many dynamic pages based on database content.
There are some points to be considered before implementing query cache.
The query cache does not return stale data. When tables are modified, any relevant entries in the query cache are flushed.
The query cache does not work in an environment where you have multiple mysqld servers updating the same MyISAM tables.
The query cache is not used for server-side prepared statements. If you're using server-side prepared statements consider that these statement won't be satisfied by the cache.

#3 Summary

Gaining performance boost by using caching mechanism is tricky. Unless used ca
refully, it cannot give required boost. As we know that we can't cache everything (specially dynamic contents), we should try to cache whatever is left. This can be achieved by various types of caching as discussed above. Static contents are well cached by clients, if not then can be cached by web servers. PHP scripts can be cached using Opcode caching softwares like APC, XCache etc. While static part of dynamic data can be cached by middleware like Symfony framework.

However only drawback of caching is that exact calculation of pages served, displayed, data transferred etc. becomes almost impossible which may effect rankings of website and thus effects popularity, revenue generations etc.