Archive for December, 2008

An Alternative Approach for Grails Integration Tests

Tuesday, December 30th, 2008

I like the approach that Grails takes for integration (and unit) testing; it’s similar to the testing strategy that I’ve used in non-Grails apps. Using HSQLDB to create an in-memory test database significantly reduces the time required to run database tests, and Hibernate’s database independence means you don’t need to pollute application code to make it testable.

Thomas Mueller originally wrote Hypersonic SQL but he ended up discontinuing it, and then other developers picked up where he left off, creating HSQLDB. For the past few years Mueller has been developing H2 from scratch and it has benefited from hindsight and lessons learned from Hypersonic and HSQLDB. According to the HSQLDB website there are some cool new features coming in 1.9 and 2.0, but H2 is already significantly more advanced. So I’ve chosen to use H2 here, but it would be pretty much the same with HSQLDB – just change the configuration properties.


There are many options for database testing; the approach Grails takes is to start a transaction for each test and roll it back afterwards. The approach I’d previously used was to rebuild the database between tests. Since the database is in-memory this is very fast and allows what I think is a more natural coding style. Since each test starts with a guaranteed clean database, there’s no need for a single wrapper transaction – I run everything in autocommit mode, similar to the way application code works in the app. If transactional behavior is required, just group multiple statements in a transaction – Grails makes it simple with withTransaction blocks and with transactional Services.

The key to this approach is a base test class that extends GroovyTestCase and which all Domain class test classes (or any test that accesses the database) extend:

import groovy.sql.Sql

import org.apache.log4j.Logger
import org.codehaus.groovy.grails.orm.hibernate.cfg.DefaultGrailsDomainConfiguration
import org.hibernate.cfg.Configuration

import org.hibernate.tool.hbm2ddl.SchemaExport

/**
 * Abstract base class for H2-based integration tests.
 * Rebuilds an in-memory database for each test.
 */
abstract class AbstractIntegrationTestCase extends GroovyTestCase {

  private static Configuration _configuration

  /** Disable Grails transaction around the tests. */
  def transactional = false

  def dataSource
  def grailsApplication
  def messageSource
  def sessionFactory

  /**
   * Replace the base class logger with one that logs using the actual class name.
   */
  protected final Logger log = Logger.getLogger(getClass())

  /**
   * {@inheritDoc}
   * @see junit.framework.TestCase#setUp()
   */
  @Override
  protected void setUp() {
    super.setUp()

    if (!_configuration) {
      // 1-time creation of the configuration

      Properties properties = new Properties()
      properties.setProperty 'hibernate.connection.driver_class', 'org.h2.Driver'
      properties.setProperty 'hibernate.connection.username', 'sa'
      properties.setProperty 'hibernate.connection.password', ''
      properties.setProperty 'hibernate.connection.url', 'jdbc:h2:mem:test'
      properties.setProperty 'hibernate.dialect', 'org.hibernate.dialect.H2Dialect'

      _configuration = new DefaultGrailsDomainConfiguration(
        grailsApplication: grailsApplication,
        properties: properties)
    }

    // rebuild the database before each test.
    new SchemaExport(_configuration).create(false, true)
    loadTestData dataSource

    clearSession()
  }

  protected void loadTestData(dataSource) {
    // override as necessary; by default run GORM create statements,
    // but could also load sql script(s)
  }

  /**
   * Run a JDBC query independent of Hibernate to verify the table count.
   * @param table  the table to query
   * @param where  optional WHERE clause
   * @return  the count
   */
  protected int findTableCount(String table, String where = null)  {

    String sqlSelect = "SELECT COUNT(0) FROM " + table
    if (where) {
      sqlSelect += " WHERE " + where
    }

    Sql.newInstance(dataSource).firstRow(sqlSelect)[0]
  }

  /**
   * Save or create the bean if it validates, and log validation errors otherwise.
   * @param bean  the bean
   * @param flush  if <code>true</code>, flush on save/update
   * @param throwException  if <code>true</code>, throw an exception if
   *    validation fails instead of returning null
   * @return  the saved/updated instance or <code>null</code>
   *    if there was a problem
   */
  protected def saveOrUpdate(bean, flush = false, throwException = true) {
    bean.validate()
    if (bean.hasErrors()) {
      println "problem creating/updating bean: ${bean}"
      def locale = Locale.getDefault()
      bean.errors.each { fieldErrors ->
        fieldErrors.allErrors.each { error ->
          println messageSource.getMessage(error, locale)
        }
      }
      if (throwException) {
        fail 'validation failed in saveOrUpdate'
      }
      return null
    }

    return bean.save(flush: flush)
  }

  protected void clearSession() {
    sessionFactory.currentSession.clear()
  }

  protected def getDomainClass() {
    return grailsApplication.getArtefact('Domain', getDomainClassName())
  }

  protected String getDomainClassName() {
    return getClass().name - 'Tests'
  }
}

It sets transactional = false to keep Grails from starting a transaction around each test. There’s a static variable to ensure that the one-time Configuration creation runs just once for all tests, and it runs SchemaExport.create() to run the drop and create DDL statements to create new empty tables for each test. A protected loadTestData() method is used to populate test data. Individual tests might need specific data but often there’s a core set of data that all tests use – this is good place to populate that shared data and to override as necessary.

Other utility methods include findTableCount() (with an optional WHERE clause) to check table row counts independent of Hibernate, saveOrUpdate() to save or update Domain class instances and log validation errors to help debug data population code issues, and clearSession() to clear the Hibernate Session’s 1st-level cache (to force reloading saved instances).

Here’s some sample test code from a test class:

void testWhatever() {

   Thing.withTransaction { status ->
      saveOrUpdate new Thing(...)
      saveOrUpdate new OtherThing(...)
      ...
   }

   clearSession()

   assertEquals ...
   assertTrue ...
}

but transactions aren’t required:

void testSomethingElse() {

   saveOrUpdate new Thing(...)
   assertEquals 5, Thing.count()
   assertTrue ...
}

For the most part I find that the test code I write using this isn’t much different from what I’d write in a regular Grails app. But it is definitely more flexible, and I find it’s more intuitive.

Performance issues with GString logging

Sunday, December 28th, 2008

Everybody knows about wrapping Log4j log.debug() statements in if (log.isDebugEnabled()) {} blocks to make sure that the message being sent to the log method isn’t evaluated unnecessarily if the DEBUG level isn’t enabled for the logger. This is true in regular Java apps as well as Groovy/Grails apps.

For example, suppose you have large Collection foo and you log it with a logger whose level is set to INFO:

log.debug("foo is " + foo);

This will concatenate "foo is " with foo.toString() but then discard the combined string since the logger isn’t at the DEBUG level. If there are a lot of these debug statements in your code then you’ll waste a lot of cycles concatenating strings, especially if you’re logging in loops. This is trivially fixed with this though:

if (log.isDebugEnabled()) {
   log.debug("foo is " + foo);
}

But if there’s no string concatenation and evaluation, then you don’t need to wrap the call – the cost of a discarded log() invocation is about the same as the isDebugEnabled() test;

log.debug("just a simple string");

incurs next to zero overhead.


Enter Groovy. What happens if you use a GString, e.g.:

log.debug "foo is ${foo}"

It turns out that this needs to be wrapped also, since although GStrings are lazy evaluated, they must be converted to a String for the debug() invocation, so you’re right back where you started. To test this, try out this code in a Groovy/Grails console.

Note that to test whether toString() is called there’s a println statement in toString():

import org.apache.log4j.Level
import org.apache.log4j.Logger

class Thing {
  String toString() {
    println "calling thing tostring"
    return "a thing"
  }
}

def thing = new Thing()

def log = Logger.getLogger('foo')
log.level = Level.INFO
log.debug "Debug: Thing: $thing"
log.error "Error: Thing: $thing"

When running this the output to the shell is

[2344] foo Error: Thing: a thing

and in the console is

calling thing tostring

so it didn’t log the debug message, but it did log the error message since it’s above the INFO level that the logger is set at. But it only printed the message from the toString() method once, meaning it didn’t evaluate the GString in the debug message. Hmmm.

Turns out I was wrong in an earlier assumption – the signatures of the various Logger methods take an Object, not a String, e.g.

public void debug(Object message)

and

public void debug(Object message, Throwable t)

So there’s nothing to evaluate, and the GString is discarded. Cool – Groovy to the rescue, no performance issues with logging GStrings.

I wondered though what it would take to coerce a GString to a String:

This doesn’t; nothing gets printed:

log.debug "Debug: Thing: $thing" + " now"

This also doesn’t, since it’s concatenating two GStrings:

log.debug "Debug: Thing: $thing at " + "${System.currentTimeMillis()}"

This does trigger toString() since it’s no longer concatenating strings:

log.debug "Debug: Thing: $thing at " + System.currentTimeMillis()

and of course

String message = "Debug: Thing: $thing"
log.debug message

triggers toString() since it explicitly creates a String.

So it turns out that you’d pretty much have to work at causing a performance issue with logging GStrings.

Annotations in Grails Controllers

Sunday, December 28th, 2008

Grails has support for enforcing that a controller method is called via GET or POST using the ‘allowedMethods’ map, e.g.

def allowedMethods = [delete: 'POST', save: 'POST', update: 'POST']

This works well but is somewhat restrictive. Grails decides how to handle a caller making a GET request when POST is required and vice versa – it sends a 405 error.

But if you want some other behavior in this case, or want to check other preconditions (other than security checks, those are much better left to proper security frameworks) then you need to implement the checks yourself, and Grails filters make the most sense.

That leaves the question of how to declare the rules. The “Grails Way”tm would be to define a Map, List, or a DSL block in the controller class that contains all of the rules for the class, but I think that annotations are the way to go here. Annotations have limitations; they can get way out of control. I converted a fairly complicated Spring/Hibernate application that used hbm files to be annotation-based, and ended up with about as much code in the annotations as I replaced in the XML – not much of a reduction.


Recently I was working on creating a site map and some automated security testing so I went through all of the controllers and noted their urls. I tested each one with a browser, issuing a GET request for each. Many failed, assuming that they’d be called via POST with required parameters that were missing, or failed less badly assuming they’re be called via Ajax and only sending a partial page or JSON response. So I created some annotations, RequiresPost and RequiresAjax to guard these. I’m not sure yet what the proper handling of using the wrong request method or type should be, but this conveniently puts that handling code in one place (and separates it from the controllers as a cross-cutting concern).

In this case the annotations are pretty simple, and are declared either at the method or class level. And since they’re declared where they’re used, they’re highly self-documenting – it’s clear looking at the method which rules apply, or that none apply.

Here are the annotations:

package com.foo.annotation;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * Indicates that a service method (or entire controller) requires
 * being called using POST.
 */
@Target({ElementType.FIELD, ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface RequiresPost {

  /**
   * Whether to display an error or be silent (default).
   * @return  <code>true</code> if an error should be shown
   */
  boolean error() default false;
}
package com.foo.annotation;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * Indicates that a service method (or entire controller) requires
 * being called by Ajax.
 */
@Target({ElementType.FIELD, ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface RequiresAjax {

  /**
   * Whether to display an error or be silent (default).
   * @return  <code>true</code> if an error should be shown
   */
  boolean error() default false;
}

and a couple of sample controllers using them. First, one that requires that all actions be called via Ajax:

@RequiresAjax
class PopupController {

  def viewFoo = {
    def foo = Foo.get(params.id)
    render foo as JSON
  }

  def viewThing = {
    def thing = Thing.get(params.id)
    [thing: thing]
  }
}

and another that requires Post or Ajax on a per-action basis:

class SignupController {

  def index = {
    ...
  }

  /**
   * Ajax call to monitor upload status.
   */
  @RequiresAjax
  def uploadInfo = {
    def data = ...
    render data as JSON
  }

  /**
   * Ajax post to perform search.
   */
  @RequiresAjax @RequiresPost
  def searchResults = {
    def results = ...
    render results as JSON
  }
}

So how do we enforce the behavior specified by the annotations? With a Grails filter:

import com.foo.annotation.ControllerAnnotationHelper

class SiteFilters {

  def filters = {

    ajaxAndPostRequired(controller: '*', action:'*') {
      before = {
        boolean ok = true

        if (!request.xhr &&
            ControllerAnnotationHelper.requiresAjax(controllerName, actionName)) {
          ok = false
        }

        if (ok) {
          if (!request.post &&
              ControllerAnnotationHelper.requiresPost(controllerName, actionName)) {
            ok = false
          }
        }

        if (!ok) {
          render text: '', template: null, contentType: 'text/plain'
          return false
        }

        return true
      }
    }
  }
}

This checks in the before() closure whether POST is required and used and whether Ajax is required and used. If either condition isn’t met, it renders a blank page and returns false, which indicates that no further processing should happen.

This isn’t necessarily the final implementation of this check, but changing site-wide behavior only requires changing this single file. Also note that the annotations support an error attribute indicating whether an error should be reported; this is currently not implemented.


One other missing piece is how to detect Ajax. Most of the popular Ajax frameworks have standardized on indicating that a request was made via Ajax by setting the X-Requested-With request header to XMLHttpRequest, so we can add an isXhr() method to the HttpServletRequest metaclass:

if (!HttpServletRequest.metaClass.hasMetaMethod('isXhr')) {
  HttpServletRequest.metaClass.isXhr = { ->
    'XMLHttpRequest' == delegate.getHeader('X-Requested-With')
  }
}

in BootStrap.groovy or in a plugin.


Finally, here’s the helper class that scans the controllers looking for annotations. This would be best in a plugin, but we currently have it implemented as a standalone class that’s triggered in BootStrap.groovy, calling init() when the application starts up. I’m intentionally using generics in type declarations, even though Groovy ignores them, to document the collection types. I’m getting increasingly frustrated with the Grails/Groovy “everything is an Object” approach that often makes it hard to figure out what parameters to send and what method return values are, but that’s another blog post …

package com.foo.annotation

import org.apache.commons.lang.WordUtils

import org.codehaus.groovy.grails.commons.ApplicationHolder as AH

class ControllerAnnotationHelper {

  private static Map<String, Map<String, List<Class>>> _actionMap = [:]
  private static Map<String, Class> _controllerAnnotationMap = [:]

  /**
   * Find controller annotation information. Called by BootStrap.init().
   */
  static void init() {

    AH.application.controllerClasses.each { controllerClass ->
      def clazz = controllerClass.clazz
      String controllerName = WordUtils.uncapitalize(controllerClass.name)
      mapClassAnnotation clazz, RequiresAjax, controllerName
      mapClassAnnotation clazz, RequiresPost, controllerName

      Map<String, List<Class>> annotatedClosures = findAnnotatedClosures(
          clazz, RequiresAjax, RequiresPost)
      if (annotatedClosures) {
        _actionMap[controllerName] = annotatedClosures
      }
    }
  }

  // for testing
  static void reset() {
    _actionMap.clear()
    _controllerAnnotationMap.clear()
  }

  // for testing
  static Map<String, Map<String, List<Class>>> getActionMap() {
    return _actionMap
  }

  // for testing
  static Map<String, Class> getControllerAnnotationMap() {
    return _controllerAnnotationMap
  }

  private static void mapClassAnnotation(clazz, annotationClass, controllerName) {
    if (clazz.isAnnotationPresent(annotationClass)) {
      def list = _controllerAnnotationMap[controllerName] ?: []
      list << annotationClass
      _controllerAnnotationMap[controllerName] = list
    }
  }

  /**
   * Check if the specified controller action requires Ajax.
   * @param controllerName  the controller name
   * @param actionName  the action name (closure name)
   */
  static boolean requiresAjax(String controllerName, String actionName) {
    return requiresAnnotation(RequiresAjax, controllerName, actionName)
  }

  /**
   * Check if the specified controller action requires POST.
   * @param controllerName  the controller name
   * @param actionName  the action name (closure name)
   */
  static boolean requiresPost(String controllerName, String actionName) {
    return requiresAnnotation(RequiresPost, controllerName, actionName)
  }

  private static boolean requiresAnnotation(Class annotationClass,
      String controllerName, String actionName) {

    // see if the controller has the annotation
    def annotations = _controllerAnnotationMap[controllerName]
    if (annotations && annotations.contains(annotationClass)) {
      return true
    }

    // otherwise check the action
    Map<String, List<Class>> controllerClosureAnnotations =
            _actionMap[controllerName] ?: [:]
    List<Class> annotationClasses = controllerClosureAnnotations[actionName]
    return annotationClasses && annotationClasses.contains(annotationClass)
  }

  private static Map<String, List<Class>> findAnnotatedClosures(
      Class clazz, Class... annotationClasses) {

    // since action closures are defined as "def foo = ..." they're
    // fields, but they end up private
    def map = [:]
    for (field in clazz.declaredFields) {
      def fieldAnnotations = []
      for (annotationClass in annotationClasses) {
        if (field.isAnnotationPresent(annotationClass)) {
          fieldAnnotations << annotationClass
        }
      }
      if (fieldAnnotations) {
        map[field.name] = fieldAnnotations
      }
    }

    return map
  }
}

This approach can help centralize request method handling, but more generically also provides some template code for defining and processing other annotations in a Grails app.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.