Objectify

The good folks at Packt Publishing, having already realised they can bribe me to mention their promotions, have again come through with another excellent promotion worthy of your attention and hard-earned money.

This time it’s Moodle Month, a fantastic deal that allows you to get 20% off print books and 30% off e-books for the world’s most popular e-learning platform – Moodle! Moodle is, according to the blurb, “a free, open-source PHP web application for producing modular internet-based courses that support a modern social constructionist pedagogy” – a dictionary is available here.

Normally, of course, the mere mention of PHP would have me reaching for my gun but since Moodle Month is tagged as festive fun, this would be a little churlish. Especially, as I already mentioned, I’m getting bribed to write this!*

Clicking around, I also noticed they have a “Save 50%” promotion on book bundles, which is pretty nice.

* Please note – bribing me to promote something generally works, and I make no moral judgements!

I had a very unpleasant experiece on Saturday. I took my daughter to see Sinterklaas arrive in Aarschot (if you don’t live in Belgium, it’s equivalent to seeing Santa Claus come to town).

Sinterklaas arrives (albeit in a car – presumably his horse got sick on the way here from Spain) and gets up on stage. He announces there are no naughty children in Aarschot…much to my daughter’s delight, and my surpirse Everything was good. It was better organised than last year, I met up with a friend and her two children in the queue, and the general feeling was one of conviviality. Apart from being surrounded by other people’s kids, but I did this to myself.

The gates open. Or, more accurately, one of the gates open. There are four gates, which allow everyone in the queue to feel like they’re moving forward. It’s a good idea – in fact, I think it’s the best way to do something like this – but the side-effect is that the individual gates open and close frequently.

My daughter was standing next to my friend’s daughter, and a couple of meters away from me, when the gate opened. They both went through. I started to follow, but the gate closed. I told the guy at the gte my daughter was already in the enclosure and I wanted to be with her. Before he could answer, the lady at the front of the queue started shouting about how her kids were first in line and I should wait.

Hold on, there’s a good point coming eventually.

So, someone told me – a guy who wasn’t bringing any more kids into the enclosure – that I couldn’t be with my daughter when she got her presents from Sinterklaas. I counted to ten, thought of a reply, and then just let it go. I repeated myself to the guy, and he said “Of course!”. In I went. Photos were taken, and smiles were smilen. But I was really pissed off. *Really* pissed off. But, now I’m in a bind. I was with my daughter, as I wanted, and to escalate the issue would be to ruin her kids’ day. Sometimes, I wish I wasn’t so damn nice.

But, of course, being pissed off generally does no good. So, I figured to myself, there’s got to be some way I can write up this whole episode and turn it into a parable for software development. I thought, and then thought some more, but I couldn’t make that leap. Then I thought to myself, why take that approach? I seem to be surrounded by bloggers who are able to apply everything that happens in their life to software development…but maybe they’re just pissed off at something too, and are writing it out. Or, of course, they’re just a lot more talented than I am.

Did I succeed in avenging myself on the person who made the most fatal mistake ever? No. Because at the end of the day, it’s no harm no foul. If I had said nothing to her frankly insane demand that I didn’t join my daughter, I would have been even more pissed off – but at myself. If I had started an argument, I would also been pissed off – again, at myself for ruining her kids’ day – and I imagine, I would also have missed the exact moment I wanted with my daughter. Sometimes option C is the best – if someone is screwing with you, just ignore them. Don’t give in, dont’ blow up…just ignore them.

Hmm, another blog entry on the Play framework. You must be sooo surprised!

Play allows you to suspend requests while doing heavy computation – this frees up your application to continue processing new requests, regardless of how long the computation takes. However, you may be deploying to an environment with a fixed request timeout – in the case that prompted me to come up with this code, I was deploying to a server that required requests to return within 120 seconds.

When generating PDFs, Excel files, images or whatever, that may take longer than the available window I needed to be able to push the computation into the background using a job. This solves the immediate problem, but leads to another issue – what if the job actually completes in a couple of seconds? I would then be forcing the user to then go to another link to download the generated content. Not so good for the user experience.

A third way was clearly needed, and this is the solution I came up with:

• All file generation computation runs in a job. This job is responsible for saving the content into the database.
• A promise is requested from the job, and given a certain amount of time to complete in:

  F.Promise promise = downloadJob.now();
  UserContent userContent = promise.get(5,
                                                        TimeUnit.SECONDS);
  

• If the object returned from the promise is not null, it’s sent to the browser. If it is null, a message is returned to the browser indicating the generation is on-going.

  if (userContent != null)
  {
      ...
      renderBinary(new ByteArrayInputStream(userContent.content.byteContent),
                        userContent.fileName,
                        userContent.content.byteContent.length,
                        MimeTypes.getMimeType(userContent.fileName),
                        false);
  }
  else
  {
      flash.success("Content is being generated in the background");
      index();
  }
  

• A download manager is provided to download any generated content generated (even content that returned a binary response).
• A job returns every night to remove any generated content older than a week

The result is an application that will guarantee to return within the alloted timeframe, and also allows users to keep working while their content is generated asynchronously. It also reduces concurrent load on the server because the generation job is executed from within the jobs pool, which can be limited using the application.conf.

An example implementation can be found here: variable-downloads. The only thing missing (apart from comments – it’s pretty self-explanatory) is polling of the server to update the downloads table when a new download is available.

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
• 
• 
• 
• 
• 
• 
• 
• 

I just re-read my previous post and thought to myself…huh? What’s the point in telling people who’ll probably never even read the blog to make sure they cache in order to improve performance?

The answer came to me in a blaze of comprehension – add a caching feature to Deadbolt, document it and then it’s obvious (to anyone who RTFM of course).

So, Deadbolt 1.4.3 is released tonight with caching. It should also have been the release with FastTags instead of Groovy tags, but that’s a bug for another day. Now it’s just a question of getting time to finish the Scala version – coming soon

If you’re using Deadbolt to provide authorisation support for your Play! app, you need to be aware that DeadboltHandler#getRoleHolder() can potentially be called a lot due to the template tags. A rule of thumb is that for every #{restrict} or #{restrictedResource} tag in your template, getRoleHolder() will be invoked once. If you’re hitting the database to get the user every time, this can have a huge impact on your performance.

Since it’s likely that the your RoleHolder won’t change within a request, you should consider adding the RoleHolder instance to the request argument if you’re not already storing it for the lifespan of the request!

Providing feedback to users following some action is important if you don’t want them confused or – worse – annoyed with your application. You can achieve this by adding something to the view following every action, but the question is where? If you add, for example, the information in a div at the top of your page this is noticable but can only contain so much information before you start screwing up your carefully designed page layout.

Another option is to use dialogs – you can get a lot of information in them and they won’t affect your page payout – but they’re a) annoying and b) blocking if modal.

A third option is to use a notification widget, and that’s what this example does. I’ve chosen Eric Hynds’ excellent jquery.notify plugin – it’s small, fast and easily configurable. Take a look at the demos at http://www.erichynds.com/examples/jquery-notify/ to get a feel for them.

Due to the inheritable structure of Play! views, you can define the entire setup of the notification system in your root view and have it automatically handle success or error messages from your flash context. For notifying of AJAX results, a small utility method can handle the dispatch to the correct notify method.

To get started, you’ll need to have jquery and jquery-ui in your application. Add the javascript and css to your root view (e.g. app/views/main.html)

<link rel="stylesheet" type="text/css"
          href="@{'/public/javascripts/jquery-ui/css/ui-lightness/jquery-ui-1.8.16.custom.css'}" charset="utf-8">
<script src="@{'/public/javascripts/jquery-1.5.2.min.js'}"
              type="text/javascript" charset="${_response_encoding}"></script>
<script src="@{'/public/javascripts/jquery-ui/js/jquery-ui-1.8.16.custom.min.js'}"
              type="text/javascript" charset="${_response_encoding}"></script>

Next, head on over to https://github.com/ehynds/jquery-notify and download the latest version of jquery.notify. Put the .js and .css files in your favoured locations, and reference them from your root view:

<link rel="stylesheet" type="text/css"
          href="@{'/public/stylesheets/ui.notify.css'}" charset="utf-8">
<script src="@{'/public/javascripts/jquery.notify.js'}"
              type="text/javascript" charset="${_response_encoding}"></script>

Next, we need to hook the widget into the view. In the body tag of your root, add the following above the doLayout tag:

<div id="notifyContainer">
    <div id="notifyPopup">
        <a class="ui-notify-close ui-notify-cross" href="#">x</a>
            <div style="float:left;margin:0 10px 0 0"><img src="${'#'}{icon}" alt="${'#'}{notificationType}"/></div>
            <h1>${'#'}{title}</h1>
        <p>${'#'}{text}</p>
    </div>
</div>

Notice one oddity here – because the templating used by the notify plugin conflicts with Play’s syntax, we have to twist things a bit. Instead of writing, for example, src=”#{icon}” we have to use src=”${‘#’}{icon}”. This will write the literal # into the resulting HTML, and match what notify expects without causing a server-side exception from Play’s rendering engine.

Now it’s time to time to add the javascript support that will do the heavy lifting. This goes into the head of your root view, and consists of three things:

• Globally-available functions that allow the notifications to be invoked from anywhere (including iframes)
• A document-ready block that prepares the notification widget for use
• A hook-up for Play’s flash context

See the in-code comments for an explanation of what’s what. Comments use the Play comment syntax of *{}*

<script type="text/javascript">
    *{ Creates the notificaton popup with the given message, icon, etc.  All other methods are convenience methods for this. }*
    function createNotification(notificationContainer, template, vars, opts) {
        return notificationContainer.notify("create", template, vars, opts);
    }

    *{ Convenience method for notifying on AJAX results.  See the AjaxResult class in the downloadble example below for the structure of the result parameter }*
    function ajaxNotify(result) {
        if (result.success) {
            notifySuccess(result.message);
        } else {
            notifyError(result.message);
        }
    }

    *{ Convenience method for creating success dialogs }*
    function notifySuccess(message) {
        createNotification($notifyContainer,
                                   "notifyPopup",
                                   { title:'Success',
                                     text:message,
                                     icon:'@{'/public/images/cool-icon.jpg'}',
                                     notificationType: 'Success'},
                                   {expires:10000});
    }

    *{ Convenience method for creating error dialogs }*
    function notifyError(message) {
        createNotification($notifyContainer,
                                   "notifyPopup",
                                   { title:'Error',
                                     text:message,
                                     icon:'@{'/public/images/shame-icon.jpg'}',
                                     notificationType: 'Success'},
                                   { expires:false });
    }

    *{ Runs when the document is ready }*
    $(function() {
        *{ Prepare the notification widget }*
        $notifyContainer = $("#notifyContainer").notify();

        *{ Create notifications if there is a success or error to report on }*
        #{if flash?.success}
            notifySuccess('${flash.success}');
        #{/if}
        #{if flash?.error}
            notifyError('${flash.error}');
        #{/if}
    });
</script>

Note the images used here can be anything – just put something in your public/images directory and reference it here.

And that’s it! If you have success or error states in your flash, they’ll automatically pop up when the page loads. In the example above, success messages stick around for 10 seconds and then vanish, but error messages have to be manually closed.

As for AJAX calls, you simply need to add a callback function to the call that will invoke the correct method when the call returns. If you call renderJSON() from your controller on an instance of AjaxResult (see the example download), you can use the ajaxNotify() method with the JSON object and it will route to the correct notification automatically.

For example,

$.post('@{Application.ajaxSomethingSuccessful()}',
          function(result) {
              ajaxNotify(result);
          });

You can download the implementation here: notifications. Note that it’s quite a large file because the whole JQuery UI library is included.

The good folks over at Packt Publishing have found a good way to get me back in the blogging habit, which is to say, bribe me with some free books in return for promoting their 2011 Open Source Awards. Sounds like a fair deal to me, so here we go…

The selection stage is now over, so the finalists in the several categories have been chosen. The categories are

• Open Source CMS
• Open Source Mobile Toolkits and Libraries
• Most Promising Open Source Project
• Open Source Business Applications
• Open Source JavaScript Libraries
• Open Source Multimedia Software
• The Open Source CMS subcategory
• The Open Source JavaScript Libraries subcategory

I’m personally using a couple of these projects on a daily basis, so it’s nice to have the chance to vote for them to show my appreciation. Similarly, I’ve never heard of some of these projects (primarily the PHP ones) so it’s also a good opportunity to look at new ideas.

As a further bonus, you have the chance to be the randomly selected voter who wins an Amazon Kindle (I’ve already got one, and it rocks. It also rocks my credit card on a weekly basis, since Amazon have made it bloody easy to buy books through it!)

There’s a rather nice selection of prizes (totalling $24000, if my maths hasn’t deserted me) for the various categories, so this is a chance to help the authors of your favourite open source projects get some payback for all their efforts too!

Voting runs from September 19th through to October 31st, so there’s plenty of time to deliberate and even try out some of the projects you may not be familiar with. So, head on over to the 2011 Open Source Awards and exercise your democratic rights!

End of shameless plug!

Wow, nearly 9 months since my last blog entry! I realised I’ve been pretty busy this year, but this seems like a fairly major gap! I have to get in the habit of blogging small entries, and finish off some of the technical articles that have been sitting in the draft folder for months!

In my previous post, I described an approach for binding data from uploaded files into entity instances. One important part was left out from this post – validation.

Since the play framework supports JPA as a persistence mechanism out of the box, and defining a constraint once beats defining it twice, it makes sense to re-use the JPA annotations to provide on-the-fly validation during the binding process.

First up, we need a class that describes the JPA constraints on a field. Keeping it simple, here we have nullable and maximum length along with the name of the field as given in the @be.objectify.led.Property annotation.

public class FieldDescriptor
{
    public String name;
    public boolean nullable = true;
    public int maxLength = -1;
}

Next, we need something that checks the model for @Property-annotated fields. Again, I’m keeping this simple by only looking at @Column annotations, but you can have @MaxSize, @OneToOne, @ManyToOne, @Required, etc. – any possible annotation. In fact, they don’t even need to be JPA annotations if you have some other way of indicating constraints.

public class ConstraintsParser
{
    public static Map<String, FieldDescriptor> parseConstraints(Class clazz)
    {
        Map<String, FieldDescriptor> fieldDescriptors = new HahsMap<String, FieldDescriptor>();

        for (Field field : clazz.getDeclaredFields())
        {
            if (field.isAnnotationPresent(Property.class))
            {
                FieldDescriptor fieldDescriptor = new FieldDescriptor();
                Property ledProperty = field.getAnnotation(Property.class);
                fieldDescriptor.name = ledProperty.value();

                if (field.isAnnotationPresent(Column.class))
                {
                    Column column = field.getAnnotation(Column.class);
                    fieldDescriptor.nullable = column.nullable();
                    fieldDescriptor.maxLength = column.length();
                }               

                fieldDescriptors.put(fieldDescriptor.name,
                                           fieldDescriptor);
            }
        }

        return fieldDescriptors;
    }
}

In order to validate values during binding, we need to implement a be.objectify.led.factory.ValidationFunction.

    public class MyValidationFunction implements ValidationFunction
    {
        private Map<String, FieldDescriptor> fieldDescriptors;

        public MyValidationFunction(Map<String, FieldDescriptor> fieldDescriptors)
        {
            this.fieldDescriptors = fieldDescriptors;
        }

        public void validate(String propertyName,
                                   String propertyValue) throws ValidationException
        {
            FieldDescriptor fieldDescriptor = fieldDescriptors.get(propertyName);
            if (!fieldDescriptor.nullable && StringUtils.isEmpty(propertyValue))
            {
                throw new ValidationException(propertyName,
                                                          String.format("% is required", propertyName));
            }

            if (fieldDescriptor.maxLength != -1 && propertyValue.length() > fieldDescriptor.maxLength)
            {
                throw new ValidationException(propertyName,
                                                          String.format("%s has a maximum length of [%d] but was [%d]",
                                                                             fieldDescriptor.maxLength,
                                                                             propertyValue.length()));
        }
    }
}

Finally, going back to the ExcelBinder class from the previous post we change the bind method to add the validation.

    public static <T> List bind(File file,
                                              Class<T> modelType) throws ValidationException
        {
            jxl.Workbook workbook = jxl.Workbook.getWorkbook(file);
            jxl.Sheet sheet = workbook.getSheet(0);
            List<String> headerNames = getHeaderNames(workbook);

            Map<String, FieldDescriptor> fieldDescriptorMap = ConstraintsParser.parseConstraints(modelType);
            ValidationFunction validator = new MyValidationFunction(fieldDescriptorMap);

            List objects = new ArrayList();
            // iterate over each non-header row and make it into an object
            for (int i = 1; i < sheet.getRows(); i++)
            {
                PropertyContext propertyContext = getPropertyContext(sheet,
                                                                                           i,
                                                                                           headerNames);
                PropertySetter propertySetter = new PropertySetter(context);
                T t = modelType.newInstance();
                propertySetter.process(t,
                                               validator);
                objects.add(t);
            }
            return objects;
        }

With this, we've added a generic way of validating the input without having to resort to anything as boring as writing it by hand and it will always be in sync with your model!

One of the most useful tools in my collection is the objectify-led library – http://www.objectify.be/objectify-led/ (full disclosure: I wrote it). It allows you to annotate fields in your classes with @Property and then bind information to them automatically. It also allows you to retrieve @Property-annotated fields along with any other annotations that are set on them.

Adding a @Property annotation to Play model entities allows us to add new functionality with minimum effort. In this example, I’m going to load an Excel spreadsheet from the client via Play, parse it into objectify-led property contexts, validate, bind and persist.

First, we define a HTML form to upload the file from. See here for further details.

Orders/upload.html

     <form action="@{Orders.uploadOrders()}" method="POST"
                enctype="multipart/form-data">
        <input type="file" id="ordersFile" name="ordersFile" />
        <input type="submit" value="Upload" />
    </form>

Our Order model class is annotated with JPA and objectify-led annotations.

models.Order

    @Entity
    public class Order extends play.db.jpa.Model
    {
        @Column(nullable = false)
        @be.objectify.led.Property("Order code")
        public String orderCode;

        @Column(length = 255)
        @be.objectify.led.Property("Description")
        public String description;

        @be.objectify.led.Property("Quantity")
        public int quantity;
    }

The Orders controller just passes the file and entity type to the ExcelBinder class, and receives a list of Order instances with bound fields back.

controllers.Orders

    public static void uploadOrders(File ordersFile)
    {
        List<Order> orders = ExcelBinder.bind(ordersFile, Order.class);
        for (Order order : orders)
        {
            order.save();
        }
    }

Each row in the spreadsheet represents one Order instance, with the exception of the first row – we treat this row as the column headers. The trick here is to have the column header match the name given to the @Property on the corresponding field. So, in our spreadsheet we have three columns: “Order code”, “Description” and “Quantity”.

The ExcelBinder class uses an arbitrary library (in most cases JExcelAPI works well) to extract each row from the spreadsheet and

• Get the column headers from the first row
• Create a be.objectify.led.PropertyContext for each subsequent row

The implementation of the PropertyContext is very simple – it just wraps a Map:
utils.MapPropertyContext

    public class MapPropertyContext implements PropertyContext
    {
        private Map<String, String> data = new HashMap<String, String>();

        public void setValue(String name,
                                    String value)
        {
            data.put(name, value);
        }

        public String getValue(String name)
        {
            return data.get(name);
        }
    }

I’m using Workbook, Sheet and Cell based on JExcelApi. If you use a different library, just retool as necessary. Also, I’m not handling exceptions here just to keep the code a bit more readable – if you use this, you’ll need to add in a few try/catch blocks.
utils.ExcelBinder

    public class ExcelBinder
    {
        public static <T> List bind(File file,
                                                  Class<T> modelType)
        {
            jxl.Workbook workbook = jxl.Workbook.getWorkbook(file);
            jxl.Sheet sheet = workbook.getSheet(0);
            List<String> headerNames = getHeaderNames(sheet);

            List<T> objects = new ArrayList<T>();
            // iterate over each non-header row and make it into an object
            for (int i = 1; i < sheet.getRows(); i++)
            {
                PropertyContext propertyContext = getPropertyContext(sheet,
                                                                                           i,
                                                                                           headerNames);
                PropertySetter propertySetter = new PropertySetter(propertyContext);
                T t = modelType.newInstance();
                propertySetter.process(t);
                objects.add(t);
            }
            return objects;
        }

        private static List<String> getHeaderNames(Sheet sheet)
        {
            List<String> headerNames = new ArrayList<String>();
            Cell[] headerCells = sheet.getRow(0);
            for (Cell cell : headerCells)
            {
                headerNames.add(cell.getContents());
            }

            return headerNames;
        }

        private static PropertyContext getPropertyContext(Sheet sheet,
                                                                            int rowIndex,
                                                                            List<String> headerNames)
        {
            Cell[] cells = sheet.getRow(rowIndex);
            MapPropertyContext propertyContext = new MapPropertyContext();
            for (int i = 0; i < headerNames.size(); i++)
            {
                propertyContext.setValue(headerNames.get(i),
                                                   cells[i].getContents());
            }

            return propertyContext;
        }
    }

So, quite a lot of code but the important thing to understand is that if you had various other model types, such as e.g. Customer, ShoppingList, etc that were also annotated with @Property you can reuse all the code above without changing a single line!
For example,

    List<Customer> customers = ExcelBinder.bind(file, Customer.class);

will take a spreadsheet and bind it to customer instances. The only things that are non-reusable are upload.html and the uploadFile method of Orders. Once MapPropertyContext and ExcelBinder have been added to your codebase, you can add Excel-model binding with a single line as above!

Also, this example used Excel but you could also support CSV, XML, YAML – any arbitrary data source in fact. Once your model has been annotated with @Property, all you need to do is load the data into a PropertyContext and the binding will happen.

You can download a complete working example here: uploads