History for MountedDatabases
??changed:
-
Mounted databases
Problem -- It would be desirable to mix multiple storages or
multiple databases in the same of Zope object space. For example
one might want to mix storages with difference semantics. Objects
for which undo or histories were important would be stored in
storages that supports undo and kept revision information. Objects
that change often or for which undo would be inappropriate might
be stored in a different storage that didn't keep multiple
revisions. We must try to approach this by allowing multiple
storages to be combined in a single storage however accommodating
multiple semantics within the same storage seems problematic. An
alternative approach is to allow multiples of databases with the
road semantics to be combined in a single object system.
Allowing multiple databases within a single object system has been
considered from the beginning of the development of the Zope object
database. From the beginning the transaction system assumed that
there might be multiple databases present within the application.
This has been used to integrate relational transactional semantics
within the larger transaction system. It was envisioned that
multiple object databases would be present and that objects in one
database could refer directly to objects in other databases.
This presents some significant problems however. Maintaining the
necessary database connections when traversing from one object to
another is rather complicated, although doable. Garbage collection in
the presence of multiple databases with arbitrary connections across
databases is extremely difficult. In fact it may be an unsolved
problem.
It was suggested by Philip Eby that a simpler approach might be
taken. Some application-level object might provide a bridge to a
second database in much the same way that SQL methods and SQL
database connections provide a bridge to relational databases.
Recent experiments along these lines have been very promising. On
the basis of these experiments we are moving forward to provide both
the infrastructure support and example products that allow multiple
Zope databases to be used within a single cell process. This
document describes the basic assumptions and requirements of such a
system.
Basic model
The model of the mounted database approach on the mounting of file
systems in Unix system. We will allow an object from one database
to be mounted or inserted into the object space of another
database. We will allow out name references to span multiple
databases in much the same way that symbolic links can span
multiple file systems in a Unix system. We will disallow a direct
object references across databases in much the same way that's
hard links are disallowed across file systems.
Basic approach
We will use mount objects. These small objects will know how to
retrieve an object from a database. They will provide an __of__
method that returns the object retrieved it from the database in
the context of their container::
def __of__(self, parent):
data=getattr(self, '_v_data', None)
if data is None:
db=getattr(self, '_v_db', None)
if db is None: db=self._connect()
j=db.open(version=self._p_jar.getVersion())
self._p_jar.onCloseCallback(self._close)
app=j.root()!['Application']
data=self._v_data=Acquisition.aq_base(
app.unrestrictedTraverse(self._path))
return data.__of__(parent)
def _close(self):
self._v_data._p_jar.close()
del self._v_data
MartijnF: wikis sometimes are a bit tricky with source code; when I read it
as web page, app=jroot()!['Application'] has the square brackets stripped off.
MindLace: corrected
When a mount object retrieves an object from its database it must
open a database connection. It needs to assure that the database
connection is closed at the end of the current request. It does
this by registering a callback with the parent object's database
connection. When the parent object's connection is closest the
callback will be called and the mount object's database connection
will be closed.
Issues
- We need a callback registration mechanism in database
connections. (Update by Shane: Completed.)
- We need to modify the copy and paste mechanisms so that they
use object paths rather than monikers. Monikers use object IDs
from a single object database. They could be extended to record
object database information in addition to object IDs however
this would require a database registry which would be desirable
to avoid. By relying on object paths instead, we leverage the
traversal capabilities of Zope and the ability of mount object's
to participate in traversal. (Update by Shane: Completed.)
- We need to decide what the semantics of ZClasses should be in a
mounted database environment. ZClasses are stored in object
database. If we allow references from an instance in one
database to ZClasses in another database we would be almost violating
the cross-database reference restriction. ZClasses are a little
bit different than other objects in that they don't use direct
references between instances and ZClasses. Rather indirection
is used. A ZClass' instance records a global ID for the ZClass
the global ID is used to look up the ZClasses when the instance
is loaded from the database.
At least initially we will impose a restriction that a ZClass
can be created in a mounted storage only if the mounted storage
has a registration for the ZClass. The easiest way to
accomplish this, would be to use another Zope database as a
mounted storage and to define the ZClass in the control panel
for the database that is mounted.
Update -- With the above restriction, mounted databases **work
with ZClasses**.
- We need to modify the ZODB logic to prevent cross database
references. While we may allow cross-database references some
day, some protocol should be used at that time to prevent
invalid references. In the short term we will simply disallowed
references across databases. (Update by Shane: Jim has been
setting this up and it may be working already.)
- Someone needs to come up with the UI for managing mounted
databases. Some people want to be able to be able to *create*
and mount arbitrary databases through the web. This is controvercial
for security reasons.
anthony - 1 Jun 2000: I'd be curious to know about the security issues here. The only
one I can think of right now is that you could mount an object from a ZODB underneath
a different acl_users where the username would have a different meaning. But I think
that this is handled by the newer security model, where there's a link back to the actual
ZODB user folder(?). It could also be addressed by not allowing mounts to anything but
the root of the new ZODB.
Progress (Shane) - 22 May 2000
A mounted database is now running on zope.org. The updated copy/paste
is not yet installed on zope.org but the patch is in the CVS repository.
Once we got it going, we have not had any problems that we know of.
The mount work is on a branch in CVS, but the changes required are so
small that I wouldn't be surprised if the changes went into the next release.
The UI issue is quite significant. We might consider the possibility of
managing databases from the control panel, in a centralized way, rather
than the normal Zope decentralized way.
Advantages to centralized database management:
- One place to go for packing.
- System administrators would always know the status of the various
databases.
Disadvantages:
- (Add your own comments)
Advantages to decentralized management:
- Creating a separate database for each user is more straightforward.
(Excellent option for ISP's.)
- anthony: it allows you to roll out different chunks of a large ZODB in pieces;
currently you have to push the entire ZODB in one large update (if you're
doing a staged development).
- anthony: I was wondering about how you could mount/unmount various
ZODB's, so I hacked around in the code a bit and figured out the magic. If
you delete the mountitem, and it's the last reference to the FileStorage (or
whatever) it just quietly unmounts it. Very neat. It means you can push out
different pieces on the fly. You can also make multiple mountpoints into the
same FileStorage (symlinks :).
Disadvantages:
- Currently the mounted database on Zope.org is installed in a decentralized
way. I didn't realize this problem until I saw it on the Zope management
screen: there is no hint at all that an object is a mounted object. It just
looks like any other object. That can be confusing!
MountedFileStorage
See the "MountedFileStorage":http://www.zope.org/Members/anthony/software/MountedFileStorage/ for a product that allows you to mount multiple file stores.
MountTab
anthony, 2000-06-05. I'm trying to put together some code to produce
[1 more lines...]