You are not logged in Log in Join
You are here: Home » Members » nexedi » Accelerating Zope » View Document
Log in
Name

Password

 
Link icon Forgot your password?
Print

Accelerating Zope

This document is a copy of http://www.nexedi.org/Members/jp/faq/accelerate-zope.stx please check new versions on the original site until we find a way to mirror it automatically

We are currently hosting at Nexedi a couple of CMF based community sites with Zope as well as a commercial online shop based on CMF, MMMShop and ERP5 extensions . All these sites contain many images and files. Some files can be as big as 256 MB.

Until recently, we have had a lot of trouble with Zope performance: memory explosion (leaks?), poor Web performance, unexpected restarts, etc. In order to circumvent those issues, we first created a static directory in Apache to bypass Zope completely for big files. However, that was very unconvenient for a community site.

We probably could finally solve most issues with the setup described bellow.

What is known not to work

The following Apache config, based on the Debian suggested config for Zope, leads to poor performance: 

    <VirtualHost openbrick.com>
    DocumentRoot /home/jp/public_html/
    ServerName openbrick.com
    ServerAlias www.openbrick.com
    RewriteEngine on
    RewriteCond %{HTTP:Authorization}  ^(.*)
    RewriteRule ^/(.*) /usr/lib/cgi-bin/Zope/VirtualHostBase/http/www.openbrick.com:80/openbrick/VirtualHostRoot/$1  [e=HTTP_CGI_AUTHORIZATION:%1,t=application/x-httpd-cgi,l]
    </VirtualHost>

Each time a large file is being retrieved, the Zope process RAM allocation jumps to 200MB, 300MB, 400MB, 500MB, etc... and takes all the swap until it crashes. Downloading a small file (ex. 8MB) takes up to 30 seconds between the first click on the URL in the navigator and the first answer from the Web server. If the file is bigger (ex. 40MB), it becomes simply impossible to download it. Whenever someone tries to download a big file, other users may have to wait for 1 minute to view any single page of the site.

What works better (1)

As recommended here we set up Apache in order to use ProxyPass and caching. The Apache config looks like this: 

    <VirtualHost openbrick.org>
    DocumentRoot /home/jp/public_html/
    ServerName openbrick.org
    ServerAlias www.openbrick.org
    ProxyRequests On
    ProxyPass / http://localhost:9673/VirtualHostBase/http/www.openbrick.org:80/openbrick/VirtualHostRoot/
    ProxyPassReverse / http://localhost:9673/VirtualHostBase/http/www.openbrick.org:80/openbrick/VirtualHostRoot/
    </VirtualHost>

We have added to each standard_html_header: 

    <dtml-call "RESPONSE.setHeader('Expires', _.DateTime(_.DateTime().timeTime() + 3600).toZone('GMT').rfc822())">
    <dtml-call "RESPONSE.setHeader('Last-Modified', bobobase_modification_time().toZone('GMT').rfc822())">
    <dtml-call "RESPONSE.setHeader('Vary', 'user-Agent; accept-language')">

We also have setup Apache main config with caching: 

    CacheNegotiatedDocs

    CacheRoot "/var/cache/apache/proxy"
    CacheSize 5
    CacheGcInterval 4
    CacheMaxExpire 24
    CacheLastModifiedFactor 0.1
    CacheDefaultExpire 1
    CacheForceCompletion 100

Results

The best result from this setup is that is (finaly) allows to manage large files with Zope. Clicking on a link to a file provides an instant answer from the server.

Performance if also improved a lot thanks to the caching of Zope pages by Apache. We use a 733 MHz Pentium III system (Asus CUSL2-M motherboard) with 256 MB RAM and Maxtor IDE disks. Here are some benchmarks: 

      ab -n1000 http://www.openbrick.org/
           -> 30 rps (requests per second)

      ab -n1000 http://www.openbrick.org/logo.png
           -> 171 rps

With previous setup: 

      ab  -n1000 http://www.openbrick.com/
           -> 25 rps

      ab -n1000 http://www.openbrick.org/logo.png
           -> 30 rps

     The main page (http://www.openbrick.com/) is cached with
     a RAM cache. This explains the performance in both cases.
     Without RAM cache, the performance is closer to 5 rps.

     We tried then to use of Zope http_cache instead of ram_cache (for the 
     front page) and removed any dependency on cookies etc. Here is 
     the result::

      ab  -n1000 http://www.openbrick.org/index2.html
           -> 153 rps

    **Conclusion**: good caching strategy leads to static page
    like performance with Zope.

    **Conclusion 2**: caching is mostly useless for anything which
    is cookie dependent (ie. electronic commerce, personalized content,
    etc.). Only the RAM cache can be used in such case.

    **Conclusion 3**: multilingual web sites should not be
    cookie dependent in order to reach appropriate performance.

What works better (2)

In our case, only certain parts of the site should be cached. The use of rewrite rules on certain file extension (ex. .jpg, .tgz) is a possible solution, combined with 2 virtual sites:

  • www.openbrick.org (main hub with rewrite rules and access to dynamic content)
  • cache.storever.com (cache setup)

Here is an example of config file for Apache to implement this: 

    <VirtualHost openbrick.org>
    DocumentRoot /home/jp/public_html/
    ServerName openbrick.org
    ServerAlias www.openbrick.org
    ServerAdmin [email protected]
    RewriteEngine On
    ProxyVia on
    RewriteRule ^/(.*)jpg$ http://cache.storever.com/openbrick/$1jpg [L,P]
    RewriteRule ^/(.*)png$ http://cache.storever.com/openbrick/$1png [L,P]
    RewriteRule ^/(.*)tgz$ http://cache.storever.com/openbrick/$1tgz [L,P]
    RewriteRule ^/(.*)gif$ http://cache.storever.com/openbrick/$1gif [L,P]
    RewriteRule ^/(.*)pdf$ http://cache.storever.com/openbrick/$1pdf [L,P]
    RewriteRule ^/(.*)gz$ http://cache.storever.com/openbrick/$1gz [L,P]
    RewriteRule ^/(.*)bz2$ http://cache.storever.com/openbrick/$1bz2 [L,P]
    RewriteRule ^/(.*) http://localhost:9673/VirtualHostBase/http/www.openbrick.org:80/openbrick/VirtualHostRoot/$1 [L,P]
    ProxyPassReverse ^/(.*) http://www.openbrick.org/$1
    </VirtualHost>

    <VirtualHost cache.storever.com>
    DocumentRoot /home/jp/public_html/
    ServerName cache.storever.com
    CacheRoot "/var/cache/apache/proxy"
    CacheSize 50
    CacheGcInterval 4
    CacheMaxExpire 24
    CacheLastModifiedFactor 0.1
    CacheDefaultExpire 1
    CacheForceCompletion 100
    ProxyRequests On
    ProxyPass / http://localhost:9673/
    ProxyPassReverse / http://localhost:9673/
    </VirtualHost>

    <VirtualHost openbrick.com>
    DocumentRoot /home/jp/public_html/
    ServerName openbrick.com
    ServerAlias www.openbrick.com
    RewriteEngine on
    RewriteCond %{HTTP:Authorization}  ^(.*)
    RewriteRule ^/(.*) http://www.openbrick.org/$1 [L,P]
    </VirtualHost>

Results

As expected, the results are equivalent to the previous case:

  • images and files are served at the speed ofApache only. Doing an ab bench on a large .tgz (ex. ab -n1000 -c100 http://... ) has no impact on Zope CPU usage (Apache jumps close to 100%).
  • dynamic content is served at the speed of Zope. Doing an ab bench on a large .tgz (ex. ab -n1000 -c100 http://... ) has no impact on Apache CPU usage (Zope jumps close to 100%)

The typical performance can be summarized as follows:

  • Zope 100% dynamic: 5 rps
  • Zope RAM Cache: 30 rps
  • Apache cached file/page: 100-400 rps

Remaining Issues

This setup still has the following issues:

  • image and files are not updated instantly: if an image / file is updated, it will take an hour before it is update by the cache
  • dynamic pages can not be cached by Apache: dynamic content which depends on cookies is not cached by Apache because the standard proxy mechanism (Vary header) does not take into account cookies etc.
  • problems with Konqueror: page content is not always updated as it should with Konqueror. Sometimes, a document which is edited appears just as it was before. This does not happen with Mozilla.
  • Language: cookie-based language selection is expensive. It should be replaced by standard Content negociation built in most browsers and multiple URL access to content. This is mostly in our case a Localizer issue.

Improving this setup

In order to improve this setup, we can think of 2 directions:

1- Better understanding of the Zope HTTP cache manager which should normally allow us to provide same performance and remove

2- Test of the Zope Proxy Cache Manager based on mod_perl which should provide caching of cookie-based dynamic content

References

Rewrite Rules for Debian

Configuring Squid as an Accelerator for Zope

Caching with mod_proxy

Using Apache with ZServer

Zope and Squid with ICP

Squid Cache Accelerator Mode

Zope Proxy Cache Manager based on mod perl

This document is a copy of http://www.nexedi.org/Members/jp/faq/accelerate-zope.stx please check new versions on the original site until we find a way to mirror it automatically
Copyright (c) 2011 Zope Foundation. All rights reserved. Legal | Contact
If you can read this text, it means you are not experiencing the Plone design at its best. Plone makes heavy use of CSS, which means it is accessible to any internet browser, but the design needs a standards-compliant browser to look like we intended it. Just so you know ;)