zope_optimisation.txt

File contents

Optimising Zope
---------------

:Author: Richard Jones, with comments from Anthony Baxter, Bakhtiar A Hamid
          and Chris Withers.

We (http://www.ekit.com/) have a popular website, and it's hit fairly hard
(around a 6 hits [*]_ a second). I've therefore been working on ways to get Zope
to go as fast as possible. Here's some hints for others who are running (or
are expecting to run) high-volume Zope sites:

1. CallProfiler up the wazoo. Perform lots of CallProfiler analysis and
   fixing of hotspots. We managed halve our average page load time just
   doing this.

2. Use RAM caches in Zope - find the HTML chunks that don't get changed for
   every visitor (but rather large groups or all visitors). Use the
   CallProfiler to isolate calls that change very little between users. RAM
   cache those. Again, we have achieved halving, or more, through caching
   just a couple of common components.
 
3. Make static content (images, stylesheets and some pages) cacheable. Put
   stylesheets and images in an area of your Zope that isn't authenticated
   by cookies. Most browsers won't cache images or a stylesheet if a cookie
   was sent along with the request. Similarly, don't request stuff by HTTPS
   when you don't need to - not only does the server have to do more work, 
   but the client won't cache anything served by HTTPS. There's a neat tool
   out there that does cacheability testing:

      http://www.mnot.net/cacheability/

   Getting caching right in Zope can be quite a challenge - you may wish to
   just skip this and use a method like we did in 6b below.

4. ZEO clients - we have about 9 of them, spread over a number of CPUs. We'll
   be putting in some more real soon. If your Zope servers are CPU-bound, then
   extra ZEO clients will only help.

5. Don't change the ZODB unless it's absolutely necessary. Changes to the ZODB
   things down considerably. Use of a backend high-performance database like
   Oracle is a very good idea.

6. Use a front-end cache for static content. There's two ways to do this:
   
   a. Throw in squid and stuff will be served much much faster. This assumes
      that the content is cacheable - see point 3.

   b. We use apache frontends using FastCGI [*]_ to the ZEO clients. They have
      an on-disk cache which serves up images (using rewrite rules to trap
      image URLs before they are sent to Zope) meaning Zope can do more
      important stuff than serve up static binary files. This halves the
      average page load time - our pages have between 50 and 120 images (yes,
      120 images is a lot and falls under the general rules for optimising
      HTML pages).

7. Python won't make use of multiple processors [*]_ so if you have them
   available, you are strongly advised to set up at least one ZEO client per
   processor (on fast machines, having more than one can sometimes be another
   performance boost).


.. [*] Where a "hit" is a distinct HTTP fetch.
.. [*] FastCGI may be replaced by ProxyPass - whatever you're comfortable
       with.
.. [*] This is due to the `Global Interpreter Lock`_ in Python.

.. _`Global Interpreter Lock`: http://www.python.org/doc/current/api/threads.html