'static' DTML documents from database records

Created by . Last modified on 2003/08/05.

No doubt you've built some beautiful, complex web pages out of database backend content. You may also have discovered that rendering those pages takes a while.

For example, I'm building some pages which have between 30 and 50 elements on them (links to curriculum content with appropriate header and descriptive data). The combination of dtml and ZSQL takes about 3 seconds per page on a linux/pII box talking to a nearby sybase database server.

So, it occurs to me that since I don't change the content very often, it might be sensible to generate all the pages one time so they can be served as if they were static Zope DTML documents. The old "train hard, fight easy" approach.

Turns out to be fairly easy to do, so it might be worth documenting here.

I have a dtml method which calls a bunch of other dtml widgets to format database elements grabbed by some Zsql methods. The end result is a dynamically generated page on demand. The dtml which actually pulls the whole mess together (called studentShellGMPID) is included below - not because it's a good example of anything, but it does illustrate two tricks worth knowing about !

1. it sets some request variables if they're not present in the namespace so I can test it - this is a very useful little trick which I recommend - set default parameters while you're building form pages ! Note that the viewer is reminded that the default value was used - also handy discovering that parameters aren't being passed.
2. the form of the calls to one of the widgets includes the use of the two parameters .None and _ plus the setting of some "form variables" which are needed for some zsql calls in the widget

Here's the code :

 <dtml-comment>
 a sample student shell to lay out SMP documents using the SMPdocwidget
 needs GMPID to identify problem 
 defaults supplied for testing in the first few lines
 August 20, 1999. rml
 </dtml-comment>
 <dtml-unless GMPID><dtml-call "REQUEST.set('GMPID','1946')">(default gmpid 1946)</dtml-unless>
 <dtml-var standard_html_header>
 <table border=1 width="95%">
 <tr><td colspan="2" align="center">
 <dtml-var studentproblembanner>
 </td></tr><tr>
 <td align="center" valign="top"><dtml-var "SMPdocwidget(_.None,_,ItemId='LE')"></td>
 <td align="center" valign="top"><dtml-var "SMPdocwidget(_.None,_,ItemId='TS')"></td> 
 </tr><tr>
 <td align="center" valign="top"><dtml-var "SMPdocwidget(_.None,_,ItemId='LT')"></td>
 <td align="center" valign="top"><dtml-var "SMPdocwidget(_.None,_,ItemId='WL')"></td>
 </tr><tr>
 <td align="center" valign="top" colspan="2">
 <dtml-var "showotherSMPlinks(_.None,_)"></td>
 </tr>
 </table>
 <dtml-var standard_html_footer> 

Now, this particular dtml method is called with the parameter GMPID set - that's used for the database lookup to assemble all the relevant database elements.

What I wanted to do was to generate a series of dtml documents which looked like each possible page - so I wrote a little zsql query which returns all of the possible GMPID values (called getSMPProblems) together with a string useful for a title field, and included it in the following method (note that the actual method I use has error checking cruft removed for this brief document !) :

 <dtml-var standard_html_header>
 <dtml-in getSMPProblems>
 <dtml-with test>
 <dtml-call "manage_addDTMLDocument(id=GMPID,title=DName,file=studentShellGMPID(_.None,_))">
 </dtml-with>
 </dtml-in>
 <dtml-var standard_html_footer>

So, for each database record, this will make a nice new dtml document in a folder called "test". The document contains the html which studentShellGMPID generates when fed a GMPID and has an ID equal to GMPID together with a nice meaningful title.

Now I can change the front end so that a user is given the pre-generated "static" version of the page rather than making all those expensive database calls every time. I can regenerate pages dynamically once only whenever any of the content gets changed and my zope server is much less heavily loaded as a result.

One possible extension would be to turn this into a kind of "server side cache" for optionally dynamic (but expensive to generate) documents - Construct a mechanism which returns the last update to any component part of the page to be served, compare this with the time stamp on the document and regenerate it only if needed, otherwise just serve up the static pre-formed one. Best of both worlds ! I wonder how dynamic the site would need to be to gain a performance advantage using this method ? If you expect a large number of page hits between changes to any of the components, this method might be useful.