Maintaining Zope
Keeping a Zope site running smoothly involves a number of administrative tasks. This chapter covers some of these tasks, such as:
- Starting Zope automatically at boot time
- Installing new products
- Setting parameters in the Control Panel
- Monitoring
- Cleaning up log files
- Packing and backing up the database
- Database recovery tools
Maintenance often is a very platform-specific task, and Zope runs on many platforms, so you will find instructions for several different operating systems here. It is not possible to provide specifics for every system; instead, we will supply general instructions which should be modified according to your specific needs and platform.
Starting Zope Automatically at Boot Time
For testing and developing purposes you will start Zope manually most of the time, but for production systems it is necessary to start Zope automatically at boot time. Also, we will want to shut down Zope in an orderly fashion when the system goes down. We will describe the necessary steps for Microsoft Windows and some Linux distributions. Take a look at the Linux section for other Unix-like operating systems. Much of the information presented here also applies to System V like Unices.
Debug Mode and Automatic Startup
If you are planning to run Zope on a Unix production system you
should also disable debug mode. This means removing the -D
option in startup scripts (e.g. the start
script created by
Zope at installation time which calls z2.py with the -D
switch) and if you've manually set it, unsetting the
Z_DEBUG_MODE
environment variable. In debug mode, Zope does
not detach itself from the terminal, which could cause startup
scripts to malfunction.
On Windows, running Zope as a service disables debug mode by
default. You still can run Zope in debug mode by setting the
Z_DEBUG_MODE
environment variable or running Zope manually
from a startup script with the -D
option. Again, this is not
recommended for production systems, since debug mode causes
performance loss.
Linux
Distributions with Prepackaged Zope
For many Linux distributions there are ready-made Zope packages which integrate nicely with the system. For instance, Debian , SuSE , Mandrake and Gentoo usually come with fairly recent Zope packages. Those packages contain ready-made scripts for automatic startup.
Automatic Startup for Custom-Built Zopes
Even if you do not want to use the prepackaged Zope that comes
with your distribution it should be possible to re-use those
startup scripts, eg. by installing the prepackaged Zope and
editing the appropriate files and symlinks in /etc/rc.d
or
by extracting them with a tool like rpm2cpio
.
In the following examples we assume you installed your custom
Zope to a system-wide directory, eg. /usr/local/zope
. If
this is not the case please replace every occurence of
/usr/local/zope
below with your Zope installation directory.
There should also be a separate Zope system user present.
Below we assume that there is a user zope
, group nogroup
present on your system. The user zope
should of course have
read access to the $ZOPE_HOME
directory (the directory which
contains the "top-level" Zope software and the "z2.py" script)
and its descendants, and write access to the contents of the
var
directory.
If you start Zope as root, which is usually the case when
starting Zope automatically on system boot, it is required
that the var
directory belongs to root. Set the ownership
by executing the command chown root var
as root.
As an example, the startup script that comes as a part of the SuSE Linux distribution looks like this:
#! /bin/sh # Copyright (c) 1995-2000 SuSE GmbH Nuernberg, Germany. # # Authors: Kurt Garloff, Vladim�r Linek # <[email protected]> # # init.d/zope # # and symbolic its link # # /usr/sbin/rczope # # System startup script for the Zope server # ### BEGIN INIT INFO # Provides: zope # Required-Start: $remote_fs # Required-Stop: $remote_fs # Default-Start: 3 5 # Default-Stop: 0 1 2 6 # Description: Start Zope server. ### END INIT INFO # Source Zope relevant things . /etc/sysconfig/zope . /etc/sysconfig/apache PYTHON_BIN="/usr/bin/python2.1" test -x $PYTHON_BIN || exit 5 ZOPE_HOME="/opt/zope" test -d $ZOPE_HOME || exit 5 ZOPE_SCRIPT="$ZOPE_HOME/z2.py" test -f $ZOPE_SCRIPT || exit 5 # Shell functions sourced from /etc/rc.status: # rc_check check and set local and overall rc status # rc_status check and set local and overall rc status # rc_status -v ditto but be verbose in local rc status # rc_status -v -r ditto and clear the local rc status # rc_failed set local and overall rc status to failed # rc_failed <num> set local and overall rc status to <num><num> # rc_reset clear local rc status (overall remains) # rc_exit exit appropriate to overall rc status . /etc/rc.status # First reset status of this service rc_reset # Return values acc. to LSB for all commands but status: # 0 - success # 1 - generic or unspecified error # 2 - invalid or excess argument(s) # 3 - unimplemented feature (e.g. "reload") # 4 - insufficient privilege # 5 - program is not installed # 6 - program is not configured # 7 - program is not running # # Note that starting an already running service, stopping # or restarting a not-running service as well as the restart # with force-reload (in case signalling is not supported) are # considered a success. COMMON_PARAMS="-u zope -z $ZOPE_HOME -Z /var/run/zope.pid -l /var/log/zope.log" PCGI_PARAMS="-p $ZOPE_HOME/Zope.cgi" [ -z "$ZOPE_HTTP_PORT" ] && ZOPE_HTTP_PORT="8080" ALONE_PARAMS="-w $ZOPE_HTTP_PORT" # For debugging... #SPECIAL_PARAMS="-D" [ -z "$ZOPE_FTP_PORT" ] && ZOPE_FTP_PORT="8021" if [ "$ZOPE_FTP" == "yes" ]; then SPECIAL_PARAMS="-f $ZOPE_FTP_PORT $SPECIAL_PARAMS" fi if [ "$ZOPE_PCGI" == "yes" ]; then PARAMS="$SPECIAL_PARAMS $PCGI_PARAMS $COMMON_PARAMS" else PARAMS="$SPECIAL_PARAMS $ALONE_PARAMS $COMMON_PARAMS" fi case "$1" in start) echo -n "Starting zope" ## Start daemon with startproc(8). If this fails ## the echo return value is set appropriate. # NOTE: startproc return 0, even if service is # already running to match LSB spec. startproc $PYTHON_BIN $ZOPE_SCRIPT -X $PARAMS # Remember status and be verbose rc_status -v ;; stop) echo -n "Shutting down zope" ## Stop daemon with killproc(8) and if this fails ## set echo the echo return value. killproc -g -p /var/run/zope.pid -TERM $PYTHON_BIN # Remember status and be verbose rc_status -v ;; try-restart) ## Stop the service and if this succeeds (i.e. the ## service was running before), start it again. ## Note: try-restart is not (yet) part of LSB (as of 0.7.5) $0 status >/dev/null && $0 restart # Remember status and be quiet rc_status ;; restart) ## Stop the service and regardless of whether it was ## running or not, start it again. $0 stop $0 start # Remember status and be quiet rc_status ;; force-reload) ## Signal the daemon to reload its config. Most daemons ## do this on signal 1 (SIGHUP). ## If it does not support it, restart. echo -n "Reload service zope" $0 stop && $0 start rc_status ;; reload) ## Like force-reload, but if daemon does not support ## signalling, do nothing (!) rc_failed 3 rc_status -v ;; status) echo -n "Checking for zope: " ## Check status with checkproc(8), if process is running ## checkproc will return with exit status 0. # Status has a slightly different for the status command: # 0 - service running # 1 - service dead, but /var/run/ pid file exists # 2 - service dead, but /var/lock/ lock file exists # 3 - service not running # NOTE: checkproc returns LSB compliant status values. checkproc -p /var/run/zope.pid $PYTHON_BIN rc_status -v ;; probe) ## Optional: Probe for the necessity of a reload, ## give out the argument which is required for a reload. test $ZOPE_HOME/superuser -nt /var/run/zope.pid && echo reload ;; *) echo "Usage: $0 {start|stop|status|try-restart|restart|force-reload|reload|probe}" exit 1 ;; esac rc_exit
You could adapt this script to start your own Zope by modifying the PYTHON_BIN, ZOPE_HOME and COMMON_PARAMS.
To set up a Zope binary package with built-in python situated
in /usr/local/zope
running as user zope
, with a "WebDAV
Source port" set to 8081, you would set:
ZOPE_HOME=/usr/local/zope PYTHON_BIN=$ZOPE_HOME/bin/python COMMON_PARAMS="-u zope -z $ZOPE_HOME -Z /var/run/zope.pid \ -l /var/log/Z2.log -W 8081 "
You can also set up a file /etc/sysconfig/zope
with variables
ZOPE_FTP_PORT, ZOPE_HTTP_PORT:
ZOPE_HTTP_PORT=80 ZOPE_FTP_PORT=21
to set the HTTP and FTP ports. The default is to start them at port 8080 and 8021.
For Red Hat, you can find a start-up script which seems to work here
Unfortunately, all Linux distributions start and stop services a little differently, so it is not possible to write a startup script that integrates well with every distribution. We will try to outline a crude version of a generic startup script which you can refine according to your needs.
To do this some shell scripting knowledge and root system access is required.
Linux startup scripts usually reside in /etc/init.d
or in
/etc/rc.d/init.d
. For our examples we assume the startup
scripts to be in /etc/rc.d/init.d
, adjust if necessary.
To let the boot process call a startup script, you also have
to place a symbolic link to the startup script in the
/etc/rc.d/rc?.d
directories, where '? is a number from 0-6
which stands for the SystemV run levels. You usually will
want to start Zope in run levels 3 and 5 (3 is full multi-user
mode, 5 is multiuser mode with X started, according to the
"Linux Standard Base":http://www.linuxbase.org), so you would
place two links in the
/etc/rc.d' directories. Be warned
that some systems (such as Debian) assume that runlevel 2 is
full multiuser mode. As stated above, we assume the main
startup script to located in /etc/rc.d/init.d/zope
, if your
system puts the init.d
directory somewhere else, you should
accomodate the paths below:
# cd /etc/rc.d/rc3.d # ln -s /etc/rc.d/init.d/zope S99zope # cd /etc/rc.d/rc5.d # ln -s /etc/rc.d/init.d/zope S99zope
Anonymous User - Oct. 19, 2005 12:57 pm: ...and to graciously stop zope when the machine is shutting down: cd /etc/rc.d/rc6.d ln -s /etc/rc.d/init.d/zope K20zope or (Debian GNU/Linux) cd /etc/rc6.d ln -s ../init.d/zope K20zope
The scripts are called by the boot process with an argument
start
when starting up and stop
on shutdown.
A simple generic startup script structure could be something like this:
#!/bin/sh # set paths and startup options ZOPE_HOME=/usr/local/zope PYTHON_BIN=$ZOPE_HOME/bin/python ZOPE_OPTS=" -u zope -P 8000" EVENT_LOG_FILE=$ZOPE_HOME/var/event.log EVENT_LOG_SEVERITY=-300 # define more environment variables ... export EVENT_LOG_FILE EVENT_LOG_SEVERITY # export more environment variables ... umask 077 cd $ZOPE_HOME case "$1" in start) # start service exec $PYTHON_BIN $ZOPE_HOME/z2.py $ZOPE_OPTS # if you want to start in debug mode (not recommended for # production systems): # exec $PYTHON_BIN $ZOPE_HOME/z2.py $ZOPE_OPTS -D & ;; stop) # stop service kill `cat $ZOPE_HOME/var/Z2.pid` ;; restart) # stop service and restart $0 stop $0 start ;; *) echo "Usage: $0 {start|stop|restart}" exit 1 ;; esac
This script lets you perform start / stop / restart operations:
-
start
- Start Zope (and the zdaemon management process)
-
stop
- Stop Zope. Kill Zope and the zdaemon management process
-
restart
- Stop then start Zope
Mac OS X
For Mac OS X, there is a website:http://zope-mosx.zopeonarope.com devoted to running Zope on Mac OS X; you also might want to look here:http://www.zope.org/Members/jens/docs/zope_osx for building / installing Zope on OS X. You might also want to check out this:http://www.zope.org/Members/richard/zope_controller/mac for starting / stopping Zope on Mac OS X, although it is currently unmaintained.
MS Windows
The prevalent way to autostart Zope on MS Windows is to install
Zope as a service. However, since services are not available on
Windows 95, Windows 98, or Windows ME, you will have to resort
to the somewhat crude method of putting a link to Zope's
start.bat
script into the Startup
folder of the Windows
start menu on those platforms.
If you installed Zope on Windows NT/2000/XP to be started manually and later on want it started as a service, perform these steps from the command line to register Zope as a Windows service:
> cd c:\Program Files\zope > bin\lib\win32\PythonService.exe /register > bin\python.exe ZServer\ZService.py --startup auto install
Replace c:\Program Files\zope
with the path to your Zope
installation. Zope should now be installed as a service which
starts automatically on system boot. To start and stop Zope
manually, go to the Windows service administration tool,
right-click the Zope service and select the corresponding
entry.
Installing New Products
Zope is a framework for building websites from new and existing software, known as Zope products. A product is a Python package with special conventions that register with the Zope framework. The primary purpose of a Zope product is to create new kinds of objects that appear in the add list. This extensibility through products has spawned a broad market of add-on software for Zope.
The guidelines for packaging a product are given in the "Packaging Products" section in the Zope Products chapter of the Zope Developer Guide. However, since these guidelines are not enforced, many Zope products adhere to different conventions. This section will discuss the different approaches to installing Zope packages.
To install a Zope product, you first download an archive file from
a website, such as the Downloads
section of zope.org. These archive
files come in several varieties, such as tgz
(gzipped tar
files), zip
(the popular ZIP format common on Windows), and
others.
In general, unpacking these archives will create a subdirectory
containing the Product itself. For instance, the Poll-1.0.tgz
archive file in the "Packaging Products" section mentioned above
contains a subdirectory of Poll
. All the software is contained
in this directory.
To install the product, you unarchive the file in the
lib/python/Products
directory. In the Poll example, this will
create a directory lib/python/Products/Poll
.
Unfortunately not all Zope developers adhere to this convention.
Often the archive file will have the lib/python/Products
part of
the path included. Worse, the archive might contain no directory,
and instead have all the files in the top-level of the archive.
Thus, it is advised to inspect the contents of the archive first.
Once you have the new directory in lib/python/Products
, you need
to tell Zope that a new product has been added. You can do this
by restarting your Zope server through the Control Panel of the
Zope Management Interface (ZMI), or, on POSIX systems, by sending
the Zope process a -HUP
signal. For instance, from the Zope
directory:
kill -HUP `cat var/Z2.pid`
If your Zope server is running in debug mode, a log message will appear indicating a new product has been discovered and registered.
To confirm that your product is installed, log into your Zope site and visit the Control Panel's Products section. You should see the new product appear in the list of installed products.
If there was a problem with the installation, the Control Panel will list it as a "Broken Product". Usually this is because Python had a problem importing a package, or the software had a syntax error. You can visit the broken product in the Control Panel and click on its Traceback tab. You will see the Python traceback generated when the package was imported.
A traceback generally will tell you what went wrong with the import. For instance, a package the software depends on could be missing. To illustrate this take a look at the traceback below - a result of trying to install CMFOODocument:http://www.zope.org/Members/longsleep/CMFOODocument without the (required) CMF package:
Traceback (most recent call last): File "/usr/share/zope/2.6.0/lib/python/OFS/Application.py", line 541, in import_product product=__import__(pname, global_dict, global_dict, silly) File "/usr/share/zope/2.6.0/lib/python/Products/CMFOODocument/__init__.py", line 19, in ? import OODocument File "/usr/share/zope/2.6.0/lib/python/Products/CMFOODocument/OODocument.py", line 31, in ? from Products.CMFCore.PortalContent import NoWL, ResourceLockedError ImportError: No module named CMFCore.PortalContent
Server Settings
The Zope server has a number of settings that can be adjusted for
performance. Unfortunately, performance tuning is not an exact
science, that is, there is no recipe for setting parameters.
Rather, you have to test every change. To load test a site, you
should run a test setup with easily reproducible results. Load
test a few significant spots in your application. The trick is to
identify typical situations while still permitting automated
testing. There are several tools to load test websites. One of
the simple yet surprisingly useful tools is ab
which comes with
Apache distributions. With ab
you can test individual URLs,
optionally providing cookies and POST data. Other tools often
allow one to create or record a user session and playing it back
multiple times. See eg. the Open System Testing
Architecture,
JMeter, or Microsoft's Web
Application Stress
Tool
.
Database Cache
The most important is the database cache setting. To adjust these settings, visit the Control Panel and click on the
Database link.Figure 23-1 Database Cache settings
There are usually seven database connections to the internal Zope database (see Database Connections below for information about how to change the number of connections). Each connection gets its own database cache. The "Target number of objects in memory per cache" setting controls just that - the system will try not to put more than this number of persistent Zope objects into RAM per database connection. So if this number is set to 400 and there are seven database connections configured, there should not be more than 2800 objects sitting in memory. Obviously, this does not say much about memory consumption, since the objects might be anything in size - from a few hundred bytes upwards. The cache favors commonly used objects - it wholly depends on your application and the kind of objects which memory consumption will result from the number set here. As a rule, Zope objects are about as big as the data they contain. There is only little overhead in wrapping data into Zope objects.
Note that only objects residing in the Zope object database are affected - data residing in external files (for instance through the ExtFile or LocalFS add-on products) or in relational databases connected to Zope will not be cached here.
Interpreter Check Intervals
The interpreter check interval determines how often the interpreter stops to execute Zope code and checks for housekeeping things like signal handlers and thread switches. A higher number means it stops less often. The default of 500 should give good performance with most platforms, but you may want to experiment with other values. The general rule is to set it higher for faster machines, so if you have a really fast system you could try setting this higher.
The interpreter check interval is set with the -i
argument to
z2.py
script. This means you should set it in Zopes own start
script if you start Zope manually, or in the system start script.
ZServer Threads
This number determines how many ZServer threads Zope starts to service requests. The default number is four (4). You may try to increase this number if you are running a heavily loaded website. If you want to increase this to more than seven (7) threads, you also should increase the number of database connections (see the next section).
Database Connections
We briefly mentioned Zope's internal database connections in the Database Cache section above. Out of the box, the number of database connections is hardwired to seven (7); but this can be changed. There is no "knob" to change this number so in order to change the number of database connections, you will need to enter quite deep into the systems' bowels. It is probably a wise idea to back up your Zope installation before following any of the instructions below.
Each database connection maintains its own cache (see above, "Database Cache"), so bumping the number of connections up increases memory requirements. Only change this setting if you're sure you have the memory to spare.
To change this setting, create a file called "custom_zodb.py" in your Zope installation directory. In this file, put the following code:
import ZODB.FileStorage import ZODB.DB filename = os.path.join(INSTANCE_HOME, 'var', 'Data.fs') Storage = ZODB.FileStorage.FileStorage(filename) DB = ZODB.DB(Storage, pool_size=25, cache_size=2000)
Anonymous User - May 27, 2004 6:56 pm: You must also import the "os" module for this to work, i.e., import os import ZODB.FileStorage import ZODB.DB filename = os.path.join(INSTANCE_HOME, 'var', 'Data.fs') Storage = ZODB.FileStorage.FileStorage(filename) DB = ZODB.DB(Storage, pool_size=25, cache_size=2000)
This only applies if you are using the standard Zope FileStorage storage.
The "pool_size" parameter is the number of database connections. Note that the number of database connections should always be higher than the number of ZServer threads by a few (it doesn't make sense to have fewer database connections than threads). See above on how to change the number of ZServer threads.
Signals (POSIX only)
Signals are a POSIX inter-process communications mechanism. If you are using Windows then this documentation does not apply.
Zope responds to signals which are sent to the process id specified in the file '$ZOPE_HOME/var/Z2.pid':
SIGHUP - close open database connections, then restart the server process. The common idiom for restarting a Zope server is: kill -HUP `cat $ZOPE_HOME/var/Z2.pid` SIGTERM - close open database connections then shut down. The common idiom for shutting down Zope is: kill -TERM `cat $ZOPE_HOME/var/Z2.pid` SIGINT - same as SIGTERM SIGUSR2 - close and re-open all Zope log files (z2.log, event log, detailed log.) The common idiom after rotating Zope log files is: kill -USR2 `cat $ZOPE_HOME/var/Z2.pid`
The process id written to the Z2.pid
file depends on whether Zope
is run under the zdaemon
management process. If Zope is run under
a management process (as it is by default) then the pid of the
management process is recorded here. Relevant signals sent to the
management process are forwarded on to the server process.
Specifically, it forwards all those signals listed above, plus
SIGQUIT and SIGUSR1. If Zope is not using a management process (-Z0
on the z2.py command line), the server process records its own pid
into z2.pid
, but all signals work the same way.
Monitoring
To detect problems (both present and future) when running Zope on production systems, it is wise to watch a few parameters.
Monitor the Event Log and the Access Log
If you set the EVENT_LOG_FILE (formerly known as the STUPID_LOG_FILE) as an environment variable or a parameter to the startup script, you can find potential problems logged to the file set there. Each log entry is tagged with a severity level, ranging from TRACE (lowest) to PANIC (highest). You can set the verbosity of the event log with the environment variable EVENT_LOG_SEVERITY. You have to set this to an integer value - see below:
TRACE=-300 -- Trace messages DEBUG=-200 -- Debugging messages BLATHER=-100 -- Somebody shut this app up. INFO=0 -- For things like startup and shutdown. PROBLEM=100 -- This isn't causing any immediate problems, but deserves attention. WARNING=100 -- A wishy-washy alias for PROBLEM. ERROR=200 -- This is going to have adverse effects. PANIC=300 -- We're dead!
So, for example setting EVENT_LOG_SEVERITY=-300 should give you all log messages for Zope and Zope applications that use Zopes' logging system.
You also should look at your access log (usually placed in $ZOPE_HOME/var/Z2.log). The Z2.log file is recorded in the Common Log Format. The sixth field of each line contains the HTTP status code. Look out for status codes of 5xx, server error. Server errors often point to performance problems.
Monitor the HTTP Service
You can find several tools on the net which facilitate monitoring of remote services, for example Nagios or VisualPulse.
For a simple "ping" type of HTTP monitoring, you could also try to put a small DTML Method with a known value on your server, for instance only containing the character "1". Then, using something along the line of the shell script below, you could periodically request the URL of this DTML Method, and mail an error report if we are getting some other value (note the script below requires a Un*x-like operating system):
#!/bin/sh # configure the values below URL="http://localhost/ping" EXPECTED_ANSWER="1" MAILTO="[email protected]" SUBJECT="There seems to be a problem with your website" MAIL_BIN="/bin/mail" resp=`wget -O - -q -t 1 -T 1 $URL` if [ "$resp" != "$EXPECTED_ANSWER" ]; then $MAIL_BIN -s "$SUBJECT" $MAILTO <<EOF The URL ---------------------------------------------- $URL ---------------------------------------------- did not respond with the expected value of $EXPECTED_ANSWER. EOF fi;
Run this script eg. every 10 minutes from cron and you
should be set for simple tasks. Be aware though that we do not
handle connections timeouts well here. If the connection hangs,
for instance because of firewall misconfiguration, wget
will
likely wait for quite a while (around 15 minutes) before it
reports an error.
Log Files
There are two main sources of log information in Zope, the access log and the event log.
Access Log
The access log records every request made to the HTTP server. It is recorded in the Common Log Format.
The default target of the access log is the file
$ZOPE_HOME/var/Z2.log. Under Unix it is however possible to
direct this to the syslog by setting the environment variable
ZSYSLOG_ACCESS to the desired domain socket (usually /dev/log
)
Anonymous User - Mar. 7, 2005 7:50 pm: Under Windows distributions the default location of the log files is $ZOPE_INSTANCE\log
If you are using syslog, you can also set a facility name by setting the environment variable ZSYSLOG_FACILITY. It is also possible to log to a remote machine. This is also controlled, you might have guessed it, by an environment variable. The variable is called ZSYSLOG_SERVER and should be set to a string of the form "host:port" where host is the remote logging machine name or IP address and port is the port number the syslog daemon is listening on (usually 514).
Event Log
The event log (formerly also called "stupid log") logs Zope and
third-party application message. The ordinary log method is to
log to a file specified by the EVENT_LOG_FILE,
eg. EVENT_LOG_FILE=$ZOPE_HOME/var/event.log
.
On Unix it is also possible to use the syslog daemon by setting
the environment variable ZSYSLOG to the desired Unix domain
socket, usually /dev/log
. Like with access logs (see above),
it is possible to set a facility name by setting the
ZSYSLOG_FACILITY environment variable, and to log to a remote
logging machine by setting the ZSYSLOG_SERVER variable to a
string of the form "host:port", where port usually should be
514.
You can coarsely control how much logging information you want to get by setting the variable EVENT_LOG_SEVERITY to an integer number - see the section "Monitor the Event Log and the Access Log" above.
Log Rotation
Log files always grow, so it is customary to periodically rotate
logs. This means logfiles are closed, renamed (and optionally
compressed) and new logfiles get created. On Unix, there is the
logrotate
package which traditionally handles this. A sample
configuration might look like this:
compress /usr/local/zope/var/Z2.log { rotate 25 weekly postrotate /sbin/kill -USR2 `cat /usr/local/zope/var/Z2.pid` endscript }
This would tell logrotate to compress all log files (not just
Zope's!), handle Zopes access log file, keep 25 rotated log
files, do a log rotation every week, and send the SIGUSR2 signal
to Zope after rotation. This will cause Zope to close the
logfile and start a new one. See the documentation to
logrotate
for further details.
On Windows there are no widespread tools for log rotation. You might try the KiWi Syslog Daemon and configure Zope to log to it. Also see the sections "Access Log" and "Event Log" above.
Packing and Backing Up the FileStorage Database
The storage used by default by Zope's built-in object database, FileStorage, is an undoable storage. This essentially means changes to Zope objects do not overwrite the old object data, rather the new object gets appended to the database. This makes it possible to recreate an objects previous state, but it also means that the file the objects are kept in (which usually resides in $ZOPE_HOME/var/Data.fs) always keeps growing.
To get rid of obsolete objects, you need to pack
the ZODB. This
can be done manually by opening Zopes Control_Panel and clicking on
the "Database Management" link. Zope offers you the option of
removing only object version older than an adjustable amount of
days.
If you want to automatically pack the ZODB you could tickle the appropriate URL with a small python script (the traditional filesystem based kind, not Zopes "Script (Python)"):
#!/usr/bin/python import sys, urllib host = sys.argv[1] days = sys.argv[2] url = "%s/Control_Panel/Database/manage_pack?days:float=%s" % \ (host, days) try: f = urllib.urlopen(url).read() except IOError: print "Cannot open URL %s, aborting" % url print "Successfully packed ZODB on host %s" % host
Anonymous User - Dec. 4, 2003 6:55 pm: Is it really possible to pack anonymously through-the-web? Shouldn't everything in the control panel require authentication?
Anonymous User - Dec. 6, 2003 12:58 pm: I just checked with zope 2.6.2... authentication is required to tickle that URL.
Anonymous User - Mar. 10, 2004 6:35 am: http://user:password@host:port/...
Anonymous User - Mar. 10, 2004 7:53 am: Automatically pack the ZODB with user/password This works! Please change the script above, because the original script makes no sense. #!/usr/bin/python import sys, urllib host = sys.argv[1] days = sys.argv[2] user = sys.argv[3] pwd = sys.argv[4] class MyUrlOpener(urllib.FancyURLopener): def prompt_user_passwd(self, host, realm): return (user,pwd) def __init__(self, *args): self.version = "Zope Packer" urllib.FancyURLopener.__init__(self, *args) def main(): urllib._urlopener = MyUrlOpener() url = "%s/Control_Panel/Database/manage_pack?days:float=%s" % \ (host, days) try: f = urllib.urlopen(url).read() except IOError: print "Cannot open URL %s, aborting" % url print "Successfully packed ZODB on host %s" % host if __name__ == '__main__': main()
Anonymous User - Mar. 16, 2004 11:35 am: In the example above, you used > url = "%s/Control_panel/Database/manage_pack?days;float=%s" % (host, days) Does this function pack all the DBs? For example I want to use some databases for virtual hosting and I need to pack all of them periodically. What about the time of this operation? I meet some strange feature: little db Test.fs (about 5KB size) was packed during 1 second - browser showed me new size (about 3KB), and file Test.fs.old appeared. Attempt to pack database again raises an error: Error Type: FileStorageError Error Value: The database has already been packed to a later time or no changes have been made since the last pack OK. The same operation with another db Main.fs (about 3MB) leaded to the following: -browser showed page, where size was the same as before starting pack, and no file appeared in var directory. -attempt to repack the db leads to error: Error Type: FileStorageError Error Value: Already packing About 20 minutes passed but nothing changed. So how can I know the moment of packing finish? At what time should I backup the db file?
Anonymous User - May 5, 2004 1:44 pm: The form with .../Database packs the Data.fs I tested with .../Database/mydb/manage_pack?days:float=x and is compresses mydb correctl So: "/usr/bin/curl -s http://user:pwd@localhost:8080/Control_Panel/Database/mydb/manage_pack?days:float=0" is fine
Anonymous User - Sep. 1, 2005 11:45 am: What about zeopack.py? Can anyone explain why a very useful script for maintenance pack, avoiding authentication on a localhost, is broken and nobody talks about it? I explain, running: python /usr/lib/zope2.7/ZODBTools/zeopack.py -d 10 -p 9673 -h localhost results in a CPU loading loop, without the packing done! Any ideas? Help is welcome :-)
The script takes two arguments, the URL of your server (eg. http://mymachine.com) and the number of days old an object version has to be to get discarded.
On Unix, put this in eg. the file /usr/local/sbin/zope_pack
, and
make it executable with chmod +x zope_pack
. Then you can put in
into your crontab with eg.:
5 4 * * sun /usr/local/sbin/zope_pack http://localhost 7
This would instruct your system to pack the ZODB on 4:05 every sunday. It would connect to the local machine, and leave object versions younger than 7 days in the ZODB.
Under Windows, you should use the scheduler to periodically start
the script. Put the above script in eg. c:\Program
Files\zope_pack.py
or whereever you keep custom scripts, and
create a batch file zope_pack.bat
with contents similar to the
following:
"C:\Program Files\zope\bin\python.exe" "C:\Program Files\zope_pack.py" "http://localhost" 7
The first parameter to python is the path to the python script we
just created. The second is the root URL of the machine you want
to pack, and the third is the maximum age of object versions you
want to keep. Now instruct the scheduler to run this .bat
file
every week.
Zope backup is quite straightforward. If you are using the
default storage (FileStorage), all you need to do is to save the
file $ZOPE_HOME/var/Data.fs
. This can be done online, because
Zope only appends to the Data.fs
file - and if a few bytes are
missing at the end of the file due to a copy while the file is
being written to, ZODB is usually capable of repairing that upon
startup. The only thing to worry about would be if someone were
to be using the Undo feature during backup. If you cannot
ensure that this does not happen, you should take one of two
routes. The first is be to shutdown Zope prior to a backup, and
the second is to do a packing operation in combination with
backup. Packing the ZODB leaves a file Data.fs.old
with the
previous contents of the ZODB. Since Zope does not write to that
file anymore after packing, it is safe to backup this file even if
undo operations are performed on the live ZODB.
To backup Data.fs
on Linux, you should not tar
it directly,
because tar
will exit with an error if files change in the
middle of a tar
operation. Simply copying it over first will do
the trick. Another option is to use rsync
like in this shell
script by Jeff Rush:
#!/bin/sh ######################################## # File: /etc/cron.daily/zbackup.cron # # Backup Zope Database Daily ######################################## # # rsync arguments: # -q ::= Quiet operation, for cron logs # -u ::= Update only, don't overwrite newer files # -t ::= Preserve file timestamps # -p ::= Preserve file permissions # -o ::= Preserve file owner # -g ::= Preserve file group # -z ::= Compress during transfer # -e ssh ::= Use the ssh utility to secure the link # ARCHTOP="/archive/zope/" DOW=`date +%A` ARCHDIR="${ARCHTOP}${DOW}" # # Insure Our Day-of-Week Directory Exists [ -d ${ARCHDIR} ] || mkdir ${ARCHDIR} || { echo "Could Not Create Day-of-Week Directory: ${ARCHDIR}" ; exit 1 } # /usr/bin/rsync -q -u -t -p -o -g /var/zope/var/Data.fs ${ARCHDIR} # ln -sf ${ARCHDIR} ${ARCHTOP}Current # exit 0
Anonymous User - Aug. 25, 2004 6:11 am: This should be changed to do a copy. Here is my script. #!/bin/sh ######################################## # File: /etc/cron.daily/zope_data_backup.cron # # Backup Zope Database Daily ######################################## # ARCHTOP="/mnt/drive2/archive/zope/" DOW=`date +%A` ARCHDIR="${ARCHTOP}${DOW}" # # Insure Our Day-of-Week Directory Exists [ -d ${ARCHDIR} ] || mkdir ${ARCHDIR} || { echo "Could Not Create Day-of-Week Directory: ${ARCHDIR}" ; exit 1 } # cp -p /usr/local/zope/var/Data.fs ${ARCHDIR} # # ln -sf ${ARCHDIR} ${ARCHTOP}Current # exit 0
This script should be run daily from cron. It will create
day-of-week subdirectories under /archive/zope
, and update the
Data.fs
file there with the current version. rsync
only
transmits differences between files, so only a minimal amount
of disk copying is needed. This could be advantageous especially
for large ZODB databases.
Anonymous User - Dec. 6, 2003 1:05 pm: what does rsync do if the file is modified during copying? I had the impression from zope list discussions that it's safer to copy Data.fs first, then rsync the copy. A newer option that deserves mention here is using repozo.py for incremental backups.
Database Recovery Tools
To recover data from corrupted ZODB database file (typically
located in $ZOPE_HOME/var/Data.fs
) there is a script
fsrecover.py
located in $ZOPE_HOME/lib/python/ZODB
.
fsrecover.py
has the following help output:
python fsrecover.py [ <options> ] inputfile outputfile Options: -f -- force output even if output file exists -v level -- Set the verbosity level: 0 -- Show progress indicator (default) 1 -- Show transaction times and sizes 2 -- Show transaction times and sizes, and show object (record) ids, versions, and sizes. -p -- Copy partial transactions. If a data record in the middle of a transaction is bad, the data up to the bad data are packed. The output record is marked as packed. If this option is not used, transaction with any bad data are skipped. -P t -- Pack data to t seconds in the past. Note that is the "-p" option is used, then t should be 0.
Anonymous User - Dec. 4, 2003 6:58 pm: This chapter could use some information on database testing tools. fstest.py and fsrefs.py are both useful. I run them both with cron every night. Maybe checkbtrees.py too?
inflatables - Oct. 14, 2005 11:39 pm: Not clear -------------- http://www.inflatables-china.net