Maven meanderings...

So... as my understanding of how to get my little programming projects rolled out matures, I have discovered Apache Maven.... and it really is quite nice for getting everything I was doing manually for the projects, done automatically. Building, distributing, constructing a project reporting site, running tests (finally have some testing built into my apps), etc. All in all, a very useful all-inclusive tool. It has taken a little work to make everything run the way I was hoping it would run, but little by little, I'm getting there.

So far, I've been using the gwt-syntaxhighlighter project as my testbed for maven, with plans on rolling out the concepts/methods I've used there to also be applied to the gwt-fontface project, and finally the ShowSort app.

Since I decided to go with Google Code for where I have the project hosted, and picked Mercurial as my source control of choice, it has proven to produce some interesting hurdles along the way. First and foremost, getting maven configured to be able to generate a GWT module was interesting, as the "Maven Way" is to keep source code and compiled code separate in two different packaged jar files, but the GWT way is to have them together, so the GWT compiler can do its thing with the source code to generate the JavaScript code for GWT. This proved to be rather simple, with the Maven GWT Plugin as it can handle having the source code in a separate file from the compiled code, and make it work with GWT. However I still needed to build something that would work for non-maven people... so the Maven Assembly Plugin comes to the rescue, making an additional jar file with the necessary source files added into the compiled file. The GWT plugin also makes it so I can separate the "resource" files (the CSS, JavaScript, and .gwt.xml files needed for the final product) separate from the Java files, making it a little cleaner and easier to locate the java files I need to work on without a polluted directory structure.

Tying it all together with Maven's SCM configuration, I was then able to have maven automatically push the new, updated source files to the source code repository whenever I release a new version. I use the SonaType OSS Maven artifact repository to release the jars into the Maven world, which is a nice convenient way for Maven users to be able to locate and use the gwt-syntaxhighlighter project and have it included into their own apps. Finally, I was able to rely on the Maven GWU Plugin to have it automatically upload the released jars up to the Google Code downloads. This means in one simple command, I can completely deploy the gwt-syntaxhighlighter project in an efficient and effective manner, getting the source code checked-in, and releasing the built jars into both the Maven central repository and into the Google Code downloads section for the project.

Finally, I tied in the Maven site generation system, which constructs a nice, simple website for the project. I have two sets of Java API documentation, the change log for the project, and a few other details all integrated into it, and have it all rigged up to be updated when I release a new version of the code.

Another thing Maven allowed for me to do, was to have a simple way to construct and run tests on the gwt-syntaxhighlighter module, checking to ensure that it works how I think it should be working. These tests end up being run automatically whenever I build the project, giving me a nice, comfortable feeling that the project will be running the way I hope it will be. Right now, the tests are probably not completely adequate, but I will be working over time to make them as robust as I can.

Since everything is tied directly into Maven, it should, theoretically, be possible for me to use any development environment (that supports Maven), so I'll no longer need to worry about how the directory structure and set-up is for my project in Eclipse and then need to switch to NetBeans at some point or another... I can use either one at my leisure.

More about the SyntaxHighlighter

A lot has happened to this little sub-project of mine. Originally simply a part of the ShowSort app, after breaking it out into its own project (see previous entry), I eventually moved it over to Google Code, completely segregating it from the ShowSort app into a project of its own.

I then transitioned it over into a maven project, hopefully separating it from the NetBeans-specific structure to something portable (in case I switch to a different IDE or something). This didn't go off without a few bumps along the way, but it is looking like it is getting closer to the mark finally. Maven lets me do some real nice things, like finally having some version management, and setting up a simple repository (which eventually, I'll get this into the main maven repositories).

I also updated the project to use the GWT 2.1.0 release, which finally removes my previous dependence on the aging gwt-incubator, in favor of the new Logging API presented in 2.1.0, which is far more comfortable to work with.

SyntaxHighlighter Widget for GWT 2.0

While working on ShowSort, I split out the code I was using to integrate the SyntaxHighlighter into its own separate project within the ShowSort project on Project Kenai, with the desire to be to create a widget usable within GWT that wraps the capabilities of SyntaxHighlighter into something simple and easy to use. The end result is something that works for my personal needs, but still has some things that could be improved/worked on at a later date.

I really wanted a widget that I could wrap into a UiBinder XML file, something that would be very simple to use. One of the obvious things I felt it would need is for each of the parameters defined for SyntaxHighlighter to be separate parameters in the UiBinder, as well as the brush selection, and for the widget to even be able to have text added that could be embedded straight from the XML or programmatically, and automatically highlight what was needed. The features I ended up with for this widget (so far) are:

• UiBinder support/functionality.
• Theme support with extensibility to add new CSS themes. (Configured at compile-time through the .gwt.xml file)
• Brush support with dynamic brush loading & extensibility to add new brushes.
• Significant concealment of the actual SyntaxHighlighter process (like the SyntaxHighlighter.highlight() JavaScript function call, dynamic loading of brush scripts, etc.)

The API documentation for the SyntaxHighlighter widget (which also includes links for the main, source, and doc jars to use if wanted) is here: http://showsort.darkhelm.org/shdoc/.

Serializing compiled Jython code objects (works on Google AppsEngine for Java)

In the process of working on the ShowSort project, and updating it to some of the more recent API changes for Google Web Toolkit (GWT) and Google AppEngine for Java (GAE/J), I ran into a dilemma -- I wanted the sorting algorithm processing to be handled through a series of Task Queue API tasks, and I wanted to break apart the tasks in such a way to provide the largest amount of time for a sorting algorithm to run -- which meant splitting up the actual execution of the sorting algorithm from the compiling of the sorting algorithm into bytecodes.

Well, as I was first using the JSR-223 implementation of Java scripting support (just for fun), I would make a compiled script using the Compilable interface in one step, save it to the memcache, and then stick a new task on the queue to execute the compiled script. Much to my surprise, this didn't work -- because the compiled script created by Jython isn't Serializable (a requirement for the memcache to work).

So, my next attempt was to use the Jython classes directly, specifically the PythonInterpreter class, and compile my script in the first step (task), save to memcache, execute on the next step (task). Once again, it failed to work, despite the PyCode object claiming to be Serializable, it wasn't able to be serialized, because some of the components of the object did not implement Serializable, specifically since the actual compiled object is of type PyTableCode, which inherits from PyBaseCode, which has a few elements that are not Serializable (like CompilerFlags for example).

Stumped again, I began googling for help (like anyone would do), and got to a lot of dead ends. Some people tried XML Serialization (which the prime Java one to use, XStream, doesn't work out of the box for GAE/J), others offered making secondary version of objects, inherited from the original, non-serializable objects, and constructing some elaborate conversion from a PyTableCode object into something home-brewed that was serializable... but that was a bit crazy.

My final idea, which actually worked, was to take a look at the Jython source code. Wandering a while in the source, I stumbled upon a class called: org.python.modules._py_compile -- the actual class that is used within Jython to do compiler operations. From that class, I derived a process that when it compiles, it doesn't create the PyTableCode object yet, instead it gives me a byte[] array -- which is very serializable. I can store the byte[] array (which is the compiled bytecodes of my script from the Jython compiler) into my object, am able to serialize the object, and then when I want to execute my code, I use another Jython object's method, called  org.python.core.BytecodeLoader.makeCode() to construct the PyTableCode object needed to be executed, using the byte[] array I had saved. The bulk of the time for compilation is still in the compile step, the only thing I can see the makeCode() method doing is wrapping it into something Jython's PythonInterpreter class can use. The final results of my attempt are below:

package org.darkhelm.showsort.server;

import java.io.ByteArrayOutputStream;
import java.io.Serializable;
import java.io.StringReader;
import java.util.Arrays;

import java.util.logging.Level;
import java.util.logging.Logger;

import org.python.compiler.Module;

import org.python.core.BytecodeLoader;
import org.python.core.CompileMode;
import org.python.core.CompilerFlags;
import org.python.core.ParserFacade;
import org.python.core.PyCode;

import org.python.util.PythonInterpreter;

/**
 *
 * @author cliffhill
 */
public class Code implements Serializable {

    private static Logger log = Logger.getLogger("ShowSort.Code");
    private static final String VAR_NAME = "array";
    private static final String LIB_PYTHON_PATH = "WEB-INF/lib-python/Lib.zip";
    private static final String FILENAME = "<script>";
    private static final String NAME = "sort$py";

    private String code;
    private byte[] compiledCode;

    public Code() {
        this(null);
    }

    public Code(String code) {
        setCode(code);
    }

    public String getCode() {
        return code;
    }

    public void setCode(String code) {
        this.code = code;
        this.compiledCode = null;
    }

    public void compile() {
        log.finest("Code to be compiled:\n" + code);
        String realCode = "import sys\n\nsys.path.insert(0, '" + LIB_PYTHON_PATH + "')\n\n" + code + "\nsort(" + VAR_NAME + ")";
        log.finest("Real Code to be compiled:\n" + realCode);

        StringReader codeReader = new StringReader(realCode);
        ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
        try {
            org.python.antlr.base.mod node;
            try {
                node = ParserFacade.parse(codeReader, CompileMode.exec, FILENAME, new CompilerFlags());
            } finally {
                codeReader.close();
            }
            Module.compile(node, byteStream, NAME, FILENAME, true, false, null, org.python.core.imp.NO_MTIME);
            compiledCode = byteStream.toByteArray();

            log.finest("Compiled code:\n" + Arrays.toString(compiledCode));
        } catch(Throwable t) {
            log.log(Level.SEVERE, null, t);
        }
    }

    public void exec(Array array) {
        PythonInterpreter pi = new PythonInterpreter();

        log.finest("Interpreter to use: " + pi);

        log.finest("setting local variable '" + VAR_NAME + "' to: " + array);

        pi.set(VAR_NAME, array);

        PyCode execCode = BytecodeLoader.makeCode(NAME, compiledCode, FILENAME);

        log.finest("Executable compiled code: " + execCode);

        pi.exec(execCode);
    }
}

I hope this helps someone else who has dealt with the limitations of the non-serializable Jython compiled script objects.

The Return of ShowSort.

Ok, so for several months, I've neglected this app, and I finally got around to fixing it. Basically, I updated my appengine libraries and GWT libraries, and it broke the app, preventing it from getting the list of sorting algorithms available to display with the app. Now that it is corrected, this should all work much better. I've also included the (very rudimentary) javadoc for the project at: http://showsort.darkhelm.org/doc/index.html. It is my rather rough attempt at writing documentation for the entire app, so please, bear with me.

The entire project was an attempt to learn about Google App Engine for Java, GWT, Jython, and making it all work together. The concept was an old application that I had to write for a computer class back in high school, a program I've rewritten multiple times to learn the basics of several languages. This is the first time I've tried to separate the code into specific components -- display, algorithm, and processing. The algorithms are written in python, and visible in the app (as well as a brief summary/description of the algorithm), and they are quite literally what is executed to do the sorting of the array. The processing components of the app are the server-side components, which define the array, and execute the sorting algorithm. The display components are the client-side pieces, utilizing GWT for rendering, as well as some pieces of code to generate the graphical aspects with a canvas tag (and the excanvas javascript library so it works in Internet Explorer, that does not natively support the cavas tag). It also handles the syntax highlighting & rendering of the python code, plus the rest of the UI in general.

All in all, it ended up being a rather complex way to do something simple (sort an array of integers). As time progresses, I will most likely add to this app. There are some GWT 2.0 features I'd like to fiddle with, and some further features I'd like to add, not to mention getting a few more sorting algorithms added to the roster.

about ShowSort

This is a Google Apps Engine/Java app I wrote, primarily to become familiar with how Google Apps Engine works, as well as to learn Google Web Toolkit (GWT), scripting in Java with Jython using JSR-223. tag, and tying them all together. It is based on an old programming project I did in high school, implementing a graphical representation of sorting an array of integers. I've expanded it some, added in a whole slew of new sorting algorithms to the process, and refining the concept a lit. For curiosity's sake, it includes the actual source code (visible in the app) for each sorting algorithm, which is written in Python. All and all, I had enjoyed figuring out how to write the app and getting it working like it is now... in the future, I might expand upon the process significantly... There's still a number of algorithms I'd like to add to the app to make it more complete, and maybe some different ways to represent the sorting algorithm's work.

So, now I'm on Blogger...

And I've gotten even more tied into the Google monolith. Well, I guess it was just a matter of time...