Video uploading guide

#1 Introduction

This document provides information about setting up video uploading and streaming for PHP based websites. This guide has been prepared by studying various resources from Internet hence this is tried and tested and almost de-facto standard in video uploading, processing and streaming.

#2 Video uploading stages

There are 3 stages in video uploading viz. Uploading video, Processing it for streaming and Streaming.

#2.1 Uploading video

Generally video files are large in terms of file size hence separate page/interface is designed to handle long uploading process. By this interface, user can send their video to your server for streaming attached to ad.

#2.2 Processing video

Processing video involves activities such as creating video thumbnails (for promotion, preview etc.), converting video formats suitable for various browsers, extracting meta-data from video for various purposes.

#2.3 Streaming video

In 3rd stage, converted videos are streamed through flash player or by browser's built in media players supporting those video types.

#3 Implementation guidelines

This guidelines mainly emphasizes on set up of web server because it is the most important part and 99% remain same for most of video uploading and streaming purpose; processing and streaming is less critical since it varies from project to project.

#3.1 Uploading video

To upload various types of videos, we first need to set up webserver so that it can accept video files. It is also better to have separate machine for video uploading, processing and streaming so that website which used those videos will not share load given by video related operations as such operations heavily consumes memory and CPU.

In this article I have decided to use Lighttpd 1.5 as video uploading and streaming server mainly for 2 reasons:

1. it is specially designed to serve static contents,
2. it has such modules/plugins which provides information about uploading progress directly to caller script which is very convenient to developers to design interface with minium coding.

There are 2 alternating solutions also viz. Apache + apache-upload-progress-module and Nginx + nginx-upload-progress-module & nginx-upload-module. However there is not much feedback available about these 2 solutions, hence I decided not to use them and sticked to lighttpd since it is popular and trusted.

#3.1.1 Installing and configuring Lighttpd

For rpm based distributions, use following command to install lighttpd server related packages:

yum install pcre-devel glib2-devel zlib-devel openssl-devel spwan-fcgi php php-cli

Lighttpd 1.5 is not yet available in any yum repository hence we have to compile and configure it manually as shown below:

• Download lighttpd.

cd /tmp/
wget http://download.lighttpd.net/lighttpd/snapshots-1.5/lighttpd-1.5.0-r2698.tar.gz
tar -zxvf lighttpd-1.5.0-r2698.tar.gz
cd lighttpd-1.5.0

• Configure and install

./configure --program-prefix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --with-pcre

make
make install

• Add necessary user/group, directories and files

adduser -m -d /var/www -s /sbin/nologin lighttpd
mkdir /etc/lighttpd/
mkdir -p /www/logs/
mkdir -p /web/pages/
chown lighttpd:lighttpd /var/log/lighttpd
cp doc/lighttpd.conf /etc/lighttpd/

• Make changes as per your setup by editing “/etc/lighttpd/lighttpd.conf” file

server.modules = ("mod_rewrite",                  "mod_access",
                  "mod_status",
                  "mod_uploadprogress",
                  "mod_proxy_core",
                  "mod_proxy_backend_fastcgi"
                  "mod_accesslog"
                )

server.max-request-size = 150000  // to support approx 120/150 MB of file.
upload-progress.progress-url = "/progress"
upload-progress.remove-timeout = 10

#### mod-proxy-core module
## read mod-proxy-core.txt for more info
## for PHP don't forget to set cgi.fix_pathinfo = 1 in the php.ini
$PHYSICAL["existing-path"] =~ "\.php$" {
  proxy-core.balancer = "round-robin"
  proxy-core.allow-x-sendfile = "enable"
  proxy-core.protocol = "fastcgi"
  proxy-core.backends = ( "unix:/tmp/php-fastcgi.sock" )
  proxy-core.max-pool-size = 16
}

# setup of host specific to video upload.
$HTTP["host"] =~ "video.myproject.com" {
  server.document-root = "/web/video.myproject.com"
  server.errorlog = "/web/logs/video.myproject.com_error.log"
  #accesslog.filename = "/web/logs/video.myproject.com_access.log"
  server.error-handler-404 = "http://www.myproject.com"

  $HTTP["url"]  =~ "^/upload" {
    proxy-core.balancer = "round-robin"
    proxy-core.protocol = "fastcgi"
    proxy-core.allow-x-sendfile = "enable"
    proxy-core.backends = (
      "unix:/tmp/upload_socket_1.sock",
      "unix:/tmp/upload_socket_2.sock",
      #"unix:/tmp/upload_socket_N.sock",
    )
    proxy-core.max-pool-size = 2  # as per backend.
  }
}

In above setup, what we are doing is that when video is uploaded to URI upload, we are proxying request to more than 1 socket using fastcgi protocol so that we can handle 2 to N uploads at a time on dedicated unix sockets. We do not need to worry about which socket is to be used and which is not, since webserver handles it on own. You can create more than 2 sockets also to handle more concurrent video uploads.

Here PHP script upload will contain code to move/copy video file at desired location making it available for further processing. This script will be normal PHP CGI script containing valid PHP code. Please note that for copying/renaming etc. you need file name so it is better to pass it from website as hidden variable of the form so that this script can rename video by that name.

Please note that for URI other than upload, dedicated php-fastcgi.sock will be used. Also do not forget to rotate 404 error log :)

• Verify installation by running following command

lighttpd -t -f /etc/lighttpd/lighttpd.conf

• Create init.d file “/etc/init.d/lighttpd” as shown below

#!/bin/sh
#
# lighttpd     Startup script for the lighttpd server
#
# chkconfig: - 85 15
# description: Lightning fast webserver with light system requirements
#
# processname: lighttpd
# config: /etc/lighttpd/lighttpd.conf
# config: /etc/sysconfig/lighttpd
# pidfile: /var/run/lighttpd.pid
#
# Note: pidfile is assumed to be created
# by lighttpd (config: server.pid-file).

# Source function library
. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/lighttpd ]; then
  . /etc/sysconfig/lighttpd
fi

if [ -z "$LIGHTTPD_CONF_PATH" ]; then
  LIGHTTPD_CONF_PATH="/etc/lighttpd/lighttpd.conf"
fi

prog="lighttpd"
lighttpd="/usr/sbin/lighttpd"
RETVAL=0

start() {
  echo -n $"Starting $prog: "
  daemon $lighttpd -f $LIGHTTPD_CONF_PATH
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog
  /usr/bin/spawn-fcgi -s /tmp/php-fastcgi.sock -f /usr/bin/php-cgi -u lighttpd -g lighttpd -P /var/run/spawn-fcgi.pid
  /usr/bin/spawn-fcgi -s /tmp/upload_socket_1.sock -f /usr/bin/php-cgi -u lighttpd -g lighttpd -P /var/run/upload_socket_1.pid
  /usr/bin/spawn-fcgi -s /tmp/upload_socket_2.sock -f /usr/bin/php-cgi -u lighttpd -g lighttpd -P /var/run/upload_socket_2.pid
  # /usr/bin/spawn-fcgi -s /tmp/upload_socket_N.sock -f /usr/bin/php-cgi -u lighttpd -g lighttpd -P /var/run/upload_socket_N.pid
  return $RETVAL
}

stop() {
  echo -n $"Stopping $prog: "
  killproc $lighttpd
  killproc php-cgi
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog /tmp/php-fastcgi.sock
/var/run/spawn-fcgi.pid /tmp/upload_socket_1.sock /var/run/upload_socket_1.pid
/tmp/upload_socket_2.sock /var/run/upload_socket_2.pid
  return $RETVAL
}

reload() {
  echo -n $"Reloading $prog: "
  killproc $lighttpd -HUP
  RETVAL=$?
  echo
  return $RETVAL
}

case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart)
    stop
    start
    ;;
  condrestart)
    if [ -f /var/lock/subsys/$prog ]; then
      stop
      start
    fi
    ;;
  reload)
    reload
    ;;
  status)
    status $lighttpd
    RETVAL=$?
    ;;
  *)
  echo $"Usage: $0 {start|stop|restart|condrestart|reload|status}"
  RETVAL=1
esac

exit $RETVAL

In above init.d file, I have merged creation of spawn-fcgi process along with lighttpd process because wihout “spawn-fcgi” process your PHP script can not receive data from webserver.

• Start lighttpd service

chmod +x /etc/init.d/lighttpd
/etc/init.d/lighttpd start

#3.1.2 Creating interface to upload videos

To create interface on your website, follow this best example. It explains how to create HTML form, Jquery based JS code and some basic stylsheets. Please do not forget to validate video file name by extension. If you face cross site domain issue, follow this native example using iframe.

#3.2 Processing video

Once video is copies/moved at desired location, it needs to be processed by a script for various purposes. These operations are like converting video; extract metadata; creating thumbnails; etc. for streaming purpose.

This should be done by separate process script. Let's call it as “process.php” script. But we also need various other software for processing. Install them on video server, using following command:

yum install ffmpeg, flvtool2, compat-readline5, php-gd php-devel libaio-devel

This process.php script will be set into crontab and should run every 1 minute so that newly uploaded videos can be processed as fast as possible.

#3.2.1 Converting video

User might have uploaded videos from any source, so there is no guarantee that it can be played in any browser since all browsers do not support all codecs. Hence we must convert uploaded video into desired format. We decided to use flash format.

Run following command from your php script to convert video into FLV format:

ffmpeg -i INPUT_VIDEO -ar 22050 -ab 32 -ac 1 -f flv -b 700k -r 15 -s ASPECT_RATIO - 2>/dev/null | flvtool2 -U stdin OUTPUT_VIDEO.flv > /dev/null

In above command, we are combining use of flvtool2 to embed keyframe markers for streaming. In your script, you will require to adjust ASPECT_RATIO.

#3.2.2 Creating thumbnails

To extract thumbnail from video file, following command can be used:

ffmpeg -itsoffset -4 -i VIDEO_FILE -vcodec CODEC -vframes 1 -an -f rawvideo -s 320x240 OUTPUT.jpg

This command generates a 320×240 sized JPG thumbnail at the 4th second in the video. You can use this example to randomly create thumbnails according to length of the video.

#3.2.3 Extract metadata

To extract metadata, following command can be used:

ffmpeg -i INPUT_VIDEO

It will print lot of metadata about video in text format which can be stored in database or used while streaming video.

#3.3 Streaming video

Streaming video requires support from webserver, JS, Flash player and some HTML work.

#3.3.1 Preparing server

Lighttpd server has built in streaming support to stream video files. To support streaming using keyframes, enable required module in server “/etc/lighttpd/lighttpd.conf” configuration file:

server.modules += ( "mod_flv_streaming" )
flv-streaming.extensions = ( ".flv" )

Restart web service to reflect above changes. Now server is ready to stream video files in flv format with support of keyframes.

#3.3.2 Streaming through HTML5

There are 2 ways to stream video. They are either using HTML5's native “video” tag or using Flash player as container.

Streaming video through HTML5 is as easy as showing image in browser, but unfortunately not all browsers support HTML5 because their support started to arrive in latest browser only in beginning of year 2011. Moreover even if browser supports HTML5, not all browsers supports all codec (another round of browser war) hence if user uploads video with H.264 codec, it will not be played in firefox and chrome browsers. Similarly if video is encoded using Theora codec then it will not be played in IE. More information about this situation can be found from here.

However if still it is decided to use HTML5 then following HTML tag can be used:

<video src="movie.mpeg" controls="controls">
Fallback flash player based video streaming code.
</video>

That's it, by this way any video file can be played without any JS/HTML code if browser supports video file's codec natively.

#3.3.3 Streaming through Flash player

Unfortunately standard solution is to use Flash player based video streaming method which streams video inside flash container. That is why earlier we had to convert video into “flv” format :). Because flash player natively supports almost all codec.

To stream video using flash container, follow this excellent tutorial.

#4 Improvements

1. In this article, I have not discussed about realtime video format validation to prevent users from uploading junks.
2. Since video server mostly serves video files and occasionally JS and HTML, you should deny access for other files than these.
3. When you will require more features for video processing, streaming you will need  to use wrapper classes like phpvideotoolkit and native ffmpeg-php extension.

#5 Resources

http://flowplayer.org/plugins/streaming/pseudostreaming.html
http://en.wikipedia.org/wiki/Flash_Video#Format_details
http://praegnanz.de/html5video/
http://uakino.net/media/document/1009.pdf
http://diveintohtml5.info/video.html

Lighttpd vs. Apache

#1 Lighttpd overview

Lighttpd is an open source web server (similar like Apache) to server web pages. It has been developed by a MySQL developer named Jan Kneschke who developed this web server as a part of the C10K problem. Hence immediate reason of birth of Lighttpd is to overcome weakness, like reducing high memory footprint, of Apache web server.

The prefork model that Apache uses consumes a lot of memory (> 20 MB normally) per process. Which means if we multiply number of process to run simultaneously then RAM of server gets exhausted quickly. Lighttpd here beats Apache by using very low memory footprint (just 6MB) which means faster output from web server. The response appears even more faster when static contents are to be delivered. In Netcraft's latest web server survey, we can see Lighttpd among top 5 web servers currently used on Internet.

#2 How to set it up

Normal and most preferred installation instruction can be found from this installation page. For Yum users, a single command yum install zlib pcre lighttpd lighttpd-fastcgi will do almost all things.

If you want to start and stop Lighttpd manually, you're done. To install Lighttpd as a service like Apache, edit and install the init script (only if you have installed Lighttpd from source):

# sed -e 's/FOO/lighttpd/g' doc/rc.lighttpd > lighttpd.init
# chmod a+rx lighttpd.init
# cp lighttpd.init /etc/init.d/lighttpd
# cp -p doc/sysconfig.lighttpd /etc/sysconfig/lighttpd
# install -Dp ./doc/lighttpd.conf /etc/lighttpd/lighttpd.conf
# chkconfig lighttpd on

If you have installed Lighttpd using Yum then just follow last step. You may also use various other commands to start and stop Lighttpd web service like /etc/init.d/lighttpd start|stop|restart|condrestart|reload|status or service lighttpd start|stop|restart|condrestart|reload|status.

To just test lighttpd.conf, run command lighttpd -t -f /PATH/TO/CONF/lighttpd.conf

#3 Differences between Apache and Lighttpd

#3.1 General

The main difference between Apache and Lighttpd is the serving model, Lighttpd is event-driven and Apache is threaded or pre-forked.

Apache provides different multiprocessing models (MPMs) for different runtime environments. The prefork model that Apache uses creates number of processes at startup of service and manages them in a pool. However each process requires lot of memory to handle requests which means the more the processes the more memory will require. That is simultaneous apache processes quickly eat available RAM.

On the other hand Lighttpd uses single process, single thread and non-blocking I/O. For that it  uses fastest even handler in the target system like: poll, epoll, kqueue or /dev/poll. This difference makes Lighttpd faster than Apache in serving static files.

However the biggest difference between both is how they support scripting languages (specially like PHP). Apache has upper hand here because it supports easy to use Shared module version, CGI and FastCGI all together while Lighttpd supports only FastCGI at this  moment.

#3.2 Configuration level

There is visible difference between styles of configuration files of Lighttpd (lighttpd.conf) and Apache (httpd.conf). Syntax of lighttpd.conf will look more like syntax of php.ini while httpd.conf has XML type syntax. Here is an example of some basic configuration:

#3.2.1 Basic Configuration

Apache:

DocumentRoot /var/www/html

CustomLog /var/www/logs/access

ErrorLog /var/www/logs/error

User apache

Group apache

Lighttpd:

server.document-root="/var/www/html"

accesslog.filename="/var/www/logs/access"

server.errorlog="/var/www/logs/error"

server.username="apache"

server.groupname="apache"

server.modules=("mod_cml")

#3.2.2 Virtual Hosts

Below is an example of difference between VirtualHosts of Apache and Lighttpd. Example is shown for myproject project.

Apache:

NameVirtualHost *

<VirtualHost *:80>
 ServerName 'www.myproject.com'
 DocumentRoot '/web/myproject/web'
 ErrorLog '/web/logs/myproject_error'
</VirtualHost>

Include conf.d/virtualhosts/*.conf

Lighttpd:

$HTTP[“host”] == “www.myproject.com” {

 server.document-root=”/web/myproject/web”

 server.errorlog="/web/myproject_error"

}

#3.2.3 Authentication and Authorization

Lighttpd, at this moment, does not support .htaccess files, so all settings must be specified in the lighttpd.conf file, or the configuration files that it includes. However it understands Apache user files for basic and digest authentication, but group file support is not yet implemented but will be implemented soon. Here is an example of authentication and authorization:

Apache:

<Directory ~>
  AuthName "Authentication required to access this area."
  AuthType Basic
  AuthUserFile /web/myproject/docs/valid.users
  Order deny,allow
  Require valid-user
</Directory>
 

Lighttpd:

auth.backend="htpasswd"

auth.backend.htpasswd.userfile="/web/myproject/docs/valid.users"

auth.require=

("~" =>

  (

    "method" =>"basic",

    "realm"  =>"Authentication required to access this area.",

    "require"=>"valid-user"

  )

)

Summarily, configuration file of Lighttpd server behaves like an active script in which you can declare variables, write logic, do computation based upon criteria etc. similar like programming script. This feature makes configuration file alive and agile.

#4 How to run PHP under Lighttpd

#4.1 Configuring PHP under Lighttpd

Apache processes PHP internally i.e using it as Shared module mod_php while Lighttpd runs PHP under FastCGI. Although Apache also supports FastCGI, using PHP under FastCGI with Apache is neglected and is not used. However with Lighttpd, only option is to run under FastCGI, PHP must be compiled with FastCGI option (thought it is not used with Apache). For more information, please read http://trac.lighttpd.net/trac/wiki/TutorialLighttpdAndPHP. Below is the example of difference between running PHP under Apache and lighttpd.

Apache:

LoadModule php5_module modules/libphp5.so
AddType application/x-httpd-php .php

Lighttpd:

server.modules=( ..., "mod_fastcgi", ... )
fastcgi.server=( ".php" =>
                 (
                   (
                     "socket" => "/tmp/php-fastcgi.socket",
                     "bin-path" => "/usr/bin/php-cgi",
                     "broken-scriptfilename" => "enable",
                     "bin-environment" =>
                     (
                       "PHP_FCGI_CHILDREN" => "2",
                       "PHP_FCGI_MAX_REQUESTS" => "5000"
                     ),
                     "min-procs" => 1,
                     "max-procs" => 2,
                     "idle-timeout" => 60
                   )
                 )
               )

You may require to set path of php-cgi according to your setup. Please note that directive server.modules actually exists along with other modules on top of configuration file hence above line indicates that mod_fastcgi should be enabled in lighttpd.cnf.

Then it will require to set 1 directive in following way in php.ini configuration file if it exists, if  doesn't then nothing to do.

cgi.fix_pathinfo=1

Last 4 directives of above mentioned configuration are for running PHP scripts in better ways.

#4.2 Application wise changes

As FastCGI is a separate process, we can't handle directives of PHP into configuration file of web server (i.e lighttpd.conf). This is one of the biggest drawback of FastCGI that is why PHP is not used under FastCGI. Moreover under FastCGI mode, your PHP script would get limited support from web server which may force you to change or rewrite your scripts. Hence it is not recommended to use PHP under Lighttpd (at this moment because Lighttpd currently supports only FastCGI mode) because of it's lack of features that PHP will require like enabling configuration options of php.ini in configuration file for all hosts or per host base.

Moreover many benchmarks shows that PHP runs slower under FastCGI than under shared version on Apache.

However if it is required to run PHP on Lighttpd, then 2 major changes will require. They are:

1. To move all PHP related setting either in php.ini directly or in configuration or global file of the application.
2. Removing all Apache web server related variables and settings from application.

Once these changes are done, it will require to test application heavily to find whether functionalities of application get broken somewhere or not.

#5 mod_uploadprogress and Prototype

This feature will be available in Lighttpd version 1.5.0 which is not released yet, hence can not test or write more about it. More information about this module can be found at here. However when it will get released, it would surely be one of the finest module of Lighttpd server because it can be easily integrated with front-end applications using JSON.

#6 Lighttpd and output compression

Lighttpd provides output compression for static data through mod_compress module. Which means before sending static contents to client, mod_compress compresses it and saves at specified path. This compressed and cached copy will be served directly from cached location when similar request is made from same or different client. Thus saving valuable bandwidth and increasing response time.

Lighttpd supports 3 types of compressions viz. deflate, gzip and bzip2. The limitation of compressing and caching is that Lighttpd can not compress files with size more than 128 MByte and less than 128 Bytes.

To enable compression we need to set 3 directives in lighttpd.conf file. They are:

compress.cache-dir="/var/www/cache/myproject/"
compress.filetype=("text/plain", "text/html")
compress.max-filesize=1 MB

However since there is upper limit of file size of 128 Mbytes, the last directive is not necessary to declare. While compressing various types of static data, it should be kept in mind that if no file type or wrong file type is mentioned then no file will get compressed.

You may require to manually create cache folder and assign necessary write permissions to it. These cached contents do not automatically get cleared hence it is left to developer to clean it at periodic level when required. Following type of command can be used to remove contents that are older than a week.

$ find /var/www/cache/myproject/ -type f -mtime +7 | xargs -r rm

To compress dynamic contents, we need to reply on PHP itself as PHP natively supports good compression of dynamic contents. For that following 2 directives are to be set in php.ini or in equivalent configuration file.

zlib.output_compression=1
zlib.output_handler=On

Please note that to use zlib.output_compression, value of output_handler should be zlib.output_handler instead of standard output_handler. To do so, output_handler directive is to be set in following way:

output_handler=zlib.output_handler

or

zlib.output_handler=On

#7 Lighttpd and caching

#7.1 Caching overview

Caching is also another method to gain better performance in serving contents and increasing response time of your PHP scripts. There are several types of caching softwares available for PHP. Some important from them are Zend Platform, APC (APC GUI), XCache, eAccelerator, ionCube Encoder and PHP Accelerator. Certain web servers like Lighttpd provides built in modules for caching static contents at web server level. They are mod_expire, mod_mem_cache and mod_cml. Hence using combination of caching static and dynamic contents effectively, we can gain lot of speed in serving contents. However all of these mechanisms are not similar.

Aforementioned independent softwares are for Opcode/Bytecode caching i.e caching your PHP script into compiled state so that when new request arrives for same script, cache software will server compiled version of code directly from cache rather than reading file again from the disk and then compiling. From these 6, eAccelerator, XCache and APC are widely used caching softwares. This benchmarks also show that how XCache and APC are better than others. We will learn more about XCache in a short while.

As said earlier that only good combination of static and dynamic contents can give considerable boost in performance, we should try to cache as much contents as possible. To cache static contents, integrated modules of web server are the best candidates. In case of Lighttpd they are mod_expire and mod_mem_cache (however this is not provided as default).

#7.2 mod_expire

Mod_expire controls the Expire header in the Response Header of HTTP/1.0 messages. It is useful to set it for static files which should be cached like images, style-sheets etc. To use this module, first it needs to get enabled in server.modules directive array. Then module specific directives are to be set in server's configuration file as shown below.

<access|modification> <number> <years|months|days|hours|minutes|seconds>

Some examples could be like:

• Cache contents of folder images for 2 hours.

expire.url = ( "/images/" => "access 2 hours" )

• Cache contents of all sub-folders of images folder for 2 hours.

$HTTP["url"] =~ "^/images/" {
     expire.url = ( "" => "access 2 hours" )
}

Values can be hours, months, days etc. depending upon requirement.

#7.3 mod_mem_cache

Mod_mem_cache is a plugin which stores content of files in memory for faster serving. That is it stores specified file types into memory to serve directly from there without going to read it from disk from specified location thus saving disk read access time. This module is a 3rd party module, hence is not included in the official distribution of Lighttpd.

This module doesn't seem that much promising to use effectively for caching as memory should be used for processing data rather than storing data. Moreover memory should not be occupied for serving files that can reside and easily managed on disks. For example when we have thousands of images to be served then it is not advisable to store them into memory just to serve it faster. More information about this plugin can be found at here.

#7.4 mod_cml (Cache Meta Language)

Mod_cml is an another caching module similar like mod_expire which is provided by Lighttpd to cache static contents of dynamic pages. The difference between mod_expire and mod_cml is that mod_cml can cache fragmented static contents which are part of dynamic contents. For example a dynamic page called index.php might have static contents like menu.html, banner.html inside it which are not integral part of index.php. In such case using mod_cml, these 2 static contents can be cached and can be delivered directly from there.

But such type of caching can not be handled directly by Lighttpd web server and mod_cml hence we need to write some code in PHP or in special CML scripts for mod_cml which is written in lua programming language.

To use mod_cml, it requires to install lua programming language and libmemcache-1.3.x. Additionally Lighttpd must be compiled with 2 options --with-lua and –with-memcache.

#7.5 XCache

XCache is a newly emerging candidate in the market of caching PHP scripts. This is an independent software and not a module of Lighttpd. However it has been written by developers of Lighttpd.

XCache is an open-source opcode cacher, which means that it accelerates the performance of PHP on servers. It optimizes performance by removing the compilation time of PHP scripts by caching the compiled state of PHP scripts into the shm (RAM) and uses the compiled version straight from the RAM. This will increase the rate of page generation time by up to 5 times as it also optimizes many other aspects of php scripts and reduce server load. Some of the good features of XCache are:

1. Optimized opcode cache.
2. Using a generator to produce C code, reduces human mistake greatly.
3. Running stable on PHP_4_3/PHP_4_4
4. Supported and tested on all latest php cvs branches, such as PHP_4_3 PHP_4_4 PHP_5_0 PHP_5_1 PHP_5_2 HEAD (6.x)
5. Alpha supported for in-alpha-php6, with Unicode enabled.
6. Read-only Cacher Protection that prevents the cache from being corrupted by php-core/extension or any code other than XCache itself.
7. Atomic get/set/inc/dec API operation on var cache for php programmers.
8. Optimizer
9. Encoder/Decoder(Loader)
10. Administrator Script

1. view statistics
2. to see if it's AutoDisableOnCorrupted?
3. view cached php/variable list
4. clear cache

The last feature allows administrator to view statistic and cached PHP variables and manage caching behavior of XCache.

#7.5.1 Installing XCache

The standard way to install XCache is from source. Get your desired version of XCache from here. Then follow below steps to install it.

# tar -zxf xcache-*.tar.gz
# cd xcache
# phpize
# ./configure --enable-xcache
# make
# su
# make install
# cat xcache.ini >> /etc/php.ini

To make sure XCache is properly installed, run below command.

$ php-fcgi -v

It will show string like with XCache vX.X, Copyright (c) XXXX-XXXX, by XXX. Same can be checked from output of phpinfo() function also. Once XCache is installed, it will require to edit xcache.ini which contain various caching related directives to be used. However it is not mandatory to edit or change. A complete explanation of all the directives can be found from http://trac.lighttpd.net/xcache/wiki/PhpIni.

#7.5.2 Configuring Administrator panel

XCache Administrator panel is an important web interface that you can monitor and operate your opcode cache, seeing how well(or bad) it goes. Since this page is protected by http-auth, it will require to provide certain values in xcache.ini. For that set below 2 directives.

xcache.admin.user='USER'
xcache.admin.pass='MD5(PASSWORD)'

where USER is name of user you wish to use and MD5(PASSWORD) is MD5 encrypted string of password that you wish to use for given USER.

To set up web interface, copy xcache/admin/ (the whole directory) to your web document-root or sub-directory of it then request it from your browser, a http-auth prompt will popup where you will require to provide above USER and PASSWORD (as a normal string, not MD5 encrypted string). However sometimes installing XCache from rpm based utilities it may require to alias in web server instead of copying the script. To do so, add below directive in your server configuration file.

Apache:

Alias /xcache-admin/ /usr/share/xcache/admin/

Lighttpd:

alias.url += ("/xcache-admin/" => "/usr/share/xcache/admin/")

Gaining performance boost by using caching mechanism is tricky. Unless used carefully, it cannot give required boost. As we know that we can't cache everything (specially dynamic contents), we should try to cache whatever is left. This can be achieved by various types of caching as discussed above. Static contents are well cached by clients, if not then can be cached by web servers. PHP scripts can be cached using Opcode caching softwares like APC, XCache etc. While static part of dynamic data can be cached by modules like mod_cml (for lighttpd web server only).

#8 Summary

Summarily, Lighttpd web server is surely worth to have look at it and to be used for serving static data. For dynamic contents like PHP scripts, it is not optimized (because of support of only FastCGI) hence we have to wait until Shared module version PHP get started to support by it. At this moment it is widely used to server static contents only. So it will take time for it to really start competing with Apache.

However certain modules like mod_secdownload, mod_compress, mod_geoip, mod_trigger_b4_dl, mod_uploadprogress, mod_useronline etc. are peculiar modules of Lighttpd which can make it stand firmly with currently popular web servers.

#9 Links

http://www.onlamp.com/pub/a/onlamp/2007/04/05/the-lighttpd-web-server.html
http://survey.netcraft.com/Reports/0703/
http://schlitt.info/applications/blog/index.php?/archives/504-Apache-vs.-Lighttpd-echo-performance.html
http://trac.lighttpd.net/trac/wiki/TutorialInstallation
http://trac.lighttpd.net/trac/wiki/TutorialConfiguration
http://trac.lighttpd.net/trac/wiki/Docs:ConfigurationOptions
http://trac.lighttpd.net/trac/wiki/TutorialLighttpdAndPHP
http://trac.lighttpd.net/trac/wiki/Docs:ModUploadProgress
http://trac.lighttpd.net/trac/wiki/Docs:ModCompress
http://blog.lighttpd.net/articles/2006/08/01/mod_uploadprogress-is-back
http://trac.lighttpd.net/trac/wiki/Docs:ModCML
http://www-128.ibm.com/developerworks/library/os-php-fastapps1/index.html
http://trac.lighttpd.net/xcache/wiki/Faq