I did a talk on Grails clustering at the Groovy & Grails eXchange 2009 in London last week and wanted to put up the slides and the sample application and clustering scripts. This was the third time I've given a version of this talk (first at a Boston Grails meetup and again at SpringOne 2GX) so I'm way overdue getting this up.

I created a simple Grails app and installed the Spring Security plugin (to test web cluster login failover and to have domain classes for Hibernate 2^nd level caching) and the Quartz plugin to demonstrate support for multiple servers. The basic non-clustered version is here so you can compare it to the version that has the required changes here.

In no particular order, here are the relevant project files:

• There's a simple Quartz job in grails-app/jobs/ClusterTestJob that prints which instance it's running on to stdout every two seconds.
• grails-app/conf/QuartzConfig.groovy specifies jdbcStore=true to have the plugin configure job storage in the database instead of in-memory.
• Externalized configuration is enabled in grails-app/conf/Config.groovy using
  grails.config.locations = [
       "classpath:${appName}-config.groovy",
       "file:./${appName}-config.groovy"]

  which allows local dev environment configuration using a file name clustered-config.groovy in the root directory of the project, or prod configuration (e.g. to keep production database passwords out of source control) using a clustered-config.groovy in the classpath. Tomcat's lib directory is in its classpath, so that's a convenient place to put the file.

• In addition note that the Log4j file loggers are configured using ServerAwareFileAppender. Specifying the location of log files is tricky. If you leave the path out and make it relative, then files get created relative to the directory Tomcat is started from, which might not always be the same. But if you hard-code the path, it won't work cross-platform. So this class figures out if you're running in Tomcat or Jetty and if you're running in dev mode, and is also cluster-aware. Depending on how you're running it knows where the logs directory is and puts logs there.
• grails-app/conf/hibernate/hibernate.cfg.xml loads grails-app/conf/hibernate/Quartz.mysql.innodb.hbm.xml which has DDL for creating the tables needed for Quartz. Use the appropriate DDL for your database from one of the files in the Quartz plugin's src/templates/sql directory.
• Role.groovy has been customized to support the Hibernate 2nd-level cache:
  - implements Serializable
  - read-only cache configured in mapping (along with disabling optimistic locking since it's not needed)
  - custom list() and count() methods that optionally use the cache or bypass it
• User.groovy is also customized for the Hibernate 2nd-level cache:
  - implements Serializable
  - read-write cache configured in mapping
• grails-app/conf/spring/resources.groovy has the fix for GRAILSPLUGINS-1207
• src/java/ehcache.xml has caches configured for the domain classes. It's not distributed though since it'll be used in the development environment.
• scripts/_Events.groovy has code to delete unused jars (unrelated to clustering, just a good idea) and delete the version of ehcache.jar that comes with Grails since the app uses a newer version. It also replaces the non-distributed ehcache.xml with the distributed version from cluster_resources, and copies quartz.properties in case you need to customize those beyond the defaults.
• grails-app/conf/BootStrap.groovy has code to create the admin role and an admin user to test login

The files from Tomcat and the cluster scripts are available here. Untar them somewhere on the server where you want to create a cluster:

You'll see five scripts:

• cleanup.sh
• createCluster.sh
• createInstance.sh
• deploy.sh
• run.sh

These are combinations of bash scripts and an Ant build file (clusterTasks.xml). All of the scripts need a cluster root directory. You can either specify it for each script invocation (run the script without parameters to see the usage) or you can set the CR environment variable

The first script to run is createCluster.sh:

This will create the directory structure and copy the Tomcat jars and other shared files.

There's an empty clustered-config.groovy that you can use to customize the production deployment. One common use for this is to externalize database passwords. This can be done with JNDI, but I prefer a self-contained war file that doesn't require container-specific configuration. This is copied to $CR/shared/lib when you run createCluster.sh since it's shared by all instances.

Next you need to run createInstance.sh once for each cluster node on this server. The syntax is

createInstance.sh [server number] [instance number] [cluster root dir]

and you need to ensure that the server number and instance number combination is unique throughout the cluster. This is simple if you number each server and choose a unique instance number for each instance on that server, e.g.

on server one,

on server two, etc. So go ahead and create at least two nodes on this server:

Some files to note are $CR/shared/conf/server.xml and $CR/instance_X/conf/catalina.properties. Most values in server.xml have been replaced with ${} properties whose values are specified in catalina.properties. These property values are retrieved via System.getProperty() but Tomcat reads catalina.properties and sets system properties from there. Using this approach we can have one parameterized server.xml and allow per-instance customization of ports, etc. Another config file is $CR/instance_X/bin/setenv.sh which configures CATALINA_OPTS (and optionally other environment variables). Edit this file to change the heap or permgen memory for each instance.

Having created the instances, now you need to deploy a war. Download the test project and unpack it:

and run grails compile to trigger installation of the project's plugins. As mentioned in the presentation, there's a bug in the Quartz plugin that causes a problem in clusters since it tries to re-register jobs, so edit plugins/quartz-0.4.1/QuartzGrailsPlugin.groovy and change

to

Now you can build the war:

and use deploy.sh to deploy it to the cluster:

The name of the war isn't important since deploy.sh deploys as the root context (in $CR/shared/webapps/ROOT). Note that you only deploy a war once per server since it's shared by all instances on that box.

Next create the database:

The project has a modified version of the SchemaExport script that fixes a bug reading hbm.xml files, so run

and choose option [2] to use the project version instead of the Grails version (this bug is fixed in Grails 1.2). This will create a file called ddl.sql that you can use to create the tables. The generated script is designed to be used by Hibernate's schema export tool which ignores errors trying to drop foreign keys on non-existent tables but this will fail from the commandline, so edit the file and remove all of the DROP TABLE ... and alter table ... drop foreign key ... statements and run

Now that we've deployed the war and created the database, we can now start the instances using run.sh. The syntax is

run.sh [start|stop|run] [instance number]

so you would run

to start instance #1. Once it has successfully started you can start the other instances:

You should see output indicating that the instances are discovering one another.

The server logs are in a few different directories. $CR/instance_X/logs will contain that instance's catalina.out and localhost_access_log, and the two rolling Log4j logs configured in Config.groovy, X_clusterdemo.log and X_sql.log, prefixed with the instance number.

$CR/shared/logs contains the remaining Tomcat logs; admin, catalina, host-manager, localhost, and manager all prefixed with the instance number. instance_X.pid files are here also, containing the process id for each instance to help with shutting down individual instances.

There is no load balancing configured yet; this can be configured using hardware load balancers, Apache, etc. $CR/shared/conf/server.xml configures the jvmRoute attribute of the <Engine> element (worker${cluster.server.number}_${cluster.instance.number}) for use with mod_jk.

So for now to test the app we'll go to specific instances. Open http://localhost:8091/admin/ and since that controller is secured, it'll prompt you to login. Use the username and password configured in BootStrap.groovy (admin/p4ssw0rd) and you should be allowed to view the page.

To test session replication, kill this instance. Find the pid for instance 1 in $CR/shared/logs/instance_1.pid and run kill -9 to ensure there's no orderly cleanup:

You'll see messages in the logs for instance 2 indicating that instance 1 has disappeared:
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared. You'll also see that if Quartz wasn't running in instance 2 it now is.

Open http://localhost:8092/admin/ and you should still be logged in since your Authentication is stored in the HTTP session and was replicated.

To shutdown an instance, use run.sh with the stop argument:

You can use cleanup.sh to delete log files and temp files in the temp and work directories. This isn't required but it's convenient.

Some further reading:

• Tomcat 6 clustering docs
• Tomcat: The Definitive Guide
• Quartz JDBC information
• Ehcache xml documentation

Links and files:

• The PDF slides from the London talk
• Watch the video of the London talk
• The non-clustered application
• The clustered version
• The cluster files and scripts

In an earlier post I wrote about getting GORM to work outside of Grails. It worked, but wasn't really usable since it could only execute script files, so its usefulness was pretty limited in a real application. Greg Bond replied on the mailing (here and here) with some great enhancements that allowed it to work without script files. So I fleshed that out some more and now have a working implementation and even a Swing application to demonstrate.

The key was that Greg used grails compile to generate his domain class files. I'd just been using the groovyc Ant task, and when I looked at what the Grails compile script the difference turned out to be that Grails uses a Grails-aware subclass of Groovyc, org.codehaus.groovy.grails.compiler.GrailsCompiler. It allows you to specify a resourcePattern attribute to point at the domain class .groovy files for special treatment.

So now instead of one sample project there's three. One is the gorm standalone project, which creates a jar (gorm_standalone.jar) containing GormHelper which bootstraps GORM. The other two are the sample application, split into GORM domain classes and the Swing application. The domain class application contains the domain class .groovy files plus DataSource.groovy, and most importantly an Ant script that builds a usable jar (domainclasses.jar). The Swing application uses gorm_standalone.jar and domainclasses.jar as libraries and displays a simple UI showing the results of database queries.

Here's a quick screen shot:

You can download the sample apps here:

GORM standalone app

Sample app domain class app

Sample Swing app

Powerful features in Ant 1.6 including presetdef, macrodef, and import make it simple to keep common tasks in common locations and simplify the creation of a project's build script. Further, although Ant scripts are not programs in the traditional sense, these features do allow quasi-OO techniques.

These aren't new features - Ant 1.6 was released almost three years ago - but it's always good to be reminded of cool features.

Build files for Java projects tend to contain the same core sets of actions:

• compile source classes
• compile test classes
• run unit tests
• build javadoc
• package files into JARs, WARs, EARs, etc.

I find that using Spring and Hibernate in many projects I also generate Hibernate *.hbm.xml files using XDoclet, and I'm also starting to formalize best practices for running test coverage and generating quality metrics as part of the continuous build process, so I'm adding the common elements of those tasks also.

PresetDef

PresetDefs allow you to set default values and even default child elements in existing targets. For example, here's my PresetDef for the copy target:

I can override either preservelastmodified or includeEmptyDirs when I call def.copy, but if I leave those attributes out the default values are used. This is essentially like having a protected accessor in a base class.

PresetDefs can contain default child elements too, for example:

In this case I'm defaulting to fork the jvm for testing, I'm using build properties to specify whether to fail the build on error or failure, and I'm including two formatters, one to stdout and one to an XML file.

Declaring a PresetDef creates a new pseudo target whose XML element name is the name attribute.¹ Here's a simple target that's used in conjunction with compiling Java source; typically you'll want to copy all non-Java resources from the source path to the class path, e.g. image files, property files, etc. after compiling:

In this example I'm using def.copy primarily because I want to maintain the modification date of the files. This wouldn't make sense if I were changing the file while copying it (e.g. if I were replacing tokens) but I do want to preserve dates in this case so I know that the correct versions of the files were copied.

MacroDef

The primary motivation for adding MacroDef to Ant was to avoid the cost of AntCall. For small projects you probably wouldn't notice the time and resource costs of AntCalls but for a large and complex project, reloading/reparsing the build script and reinitializing everything can slow down a build significantly. So MacroDef was added to address the need to perform common sequences of tasks with parameter passing, similar to calling a method in a programming language, but in a lightweight fashion.

Here's a sample MacroDef for running JUnit tests. This one is fairly simple with just two subtasks - running the tests and generating HTML reports from the XML formatter output. There are two attributes, maxMemory (with a typical default value) and report.dir, and two required child elements filesets and classpathElements:

attributes and elements allow simple and complex 'parameter' passing. An attribute is a string parameter, like a property, except that the syntax for resolving its value uses an '@' instead of '$' to avoid collisions with build properties. elements allow you to pass entire XML elements (arbitrarily complex) into a MacroDef. Note that Ant does validate the usage and the build will fail if non-optional child elements are missing.

Here's a sample usage of this MacroDef:

And to see how Ant inserts default values from the PresetDef and parameters from the MacroDef, here's the equivalent Ant code not using PresetDef or MacroDef:

As with any code reuse strategy (e.g. base classes or JSP includes) the biggest benefit is centralization. If you copy and paste and need to make changes in the common code, you need to find all instances and make multiple identical changes, risking missing some. But with intelligent code reuse, when you change the code in the base class or included file, or for Ant build files in a PresetDef or MacroDef, and all users immediately see the changes.

Import

So you've created a bunch of useful PresetDefs, MacroDefs, Targetss, etc. but how do you use them in your various build scripts? Ant build files are just XML, so the traditional way was to use entity includes. This basically worked but was very inflexible. In Ant 1.6 you can use import to intelligently include a partial or complete build script. It's intelligent inclusion because you can redefine (override) targets or other elements and you can even continue to use or call the overridden versions.

Here's a complete sample build script for a project called 'common' that contains shared code common to many projects. It imports a file called standard.ant² located here, which contains the above examples and a few other targets, paths, and property definitions:

This is a pretty small build file considering what it does. If you run the all target it will do the following:

• delete the previous build directory
• compile Java source
• compile test Java source
• generate Hibernate .hbm.xml descriptors for classes in any package whose name contains "model"
• build a JAR file with source and test classes (to include reusable test base classes), Hibernate descriptors, and all other non-Java resouces in the src and test directories
• run unit tests
• generate HTML reports from JUnit XML formatter output
• build Javadoc

This certainly isn't all that apparent just by looking at the build script, but that's a side effect of any code reuse.

Note that Ant properties cannot be changed once they've been set, so the statement order is important. First a build.properties file is loaded (if it exists) to give the developer a chance to set custom values. Then standard.ant is imported and any property values that weren't set from the command line via -D parameters or in build.properties are set.

So you can see that it's quite simple to extract common build functionality into libraries of reusable components to very quickly create build scripts with significant functionality, with only a minimum of customization. Even the sample build script that I've show could be a lot smaller if all projects have the same basic structure (one Java source directory in src/, one test directory in test/, JARs in lib/, etc.). You could easily push even more than I have into standard.ant (and other included files) and have build scripts that contain little more than the title for your Javadoc and the name of your JAR file.

1. Note that the naming convention of prefixing "def." to the PresetDef is mine and not a requirement of Ant.[back]
2. My naming convention for Ant includes is to use "ant" as the file extension to differentiate between including build.xml files and included *.ant files[back]