Lots of great ideas

I would like to thank everyone that offered suggestions for what they would like to see in the next version of cfcUnit. Many of these are great ideas, and some will certainly be added to the core framework. They have given me something to focus on and a bit more future food for thought.

Something else that is made much clearer is that more documentation, tutorials and examples are needed to get people started more quickly. I will be addressing that.

I would also like to remind folks that there is a cfcUnit mailing list (link below). If you have any ideas or questions about how things work, it is a good way of getting some help.

Your thoughts on unit testing

I have spent these last few months working on the next release of cfcUnit 1.2, so it has been on my mind quite a bit. While I add important functionality to the framework like a remote service for Flash Remoting and a rather robust Ant task and fix some long-standing issues, I can't help wonder if there is something else I need to add. So I pose this question to you: What do you want to see in the next release of cfcUnit?

If you already using cfcUnit for your unit testing needs, what would could be done to make your experience that much better? If you are not writing unit tests right now, what would it take to change your mind?

All this talk of cfcUnit is making me nervous...

There are a number of people writing of the improvements in the latest version of cfcUnit, and I have not been one of them. So, you might be wondering why. Well, this last month since CFUnited, I have been busy making cfcUnit 1.1 ready for full release. That means fixing some bugs, adding some new assertions, and most importantly making sure that there is some better documentation available. On top of all that, though, I have been very busy making sure that you can run cfcUnit from everyone's favorite build tool, Ant.

Apparently, the absence of a cfcUnit Ant task and the existence of one for CFUnit has been a critical distinction between the two CF testing frameworks, giving people a reason not to use cfcUnit. Well, at CFUnited John Paul Ashenfelter took the initial stab at this elusive feature to show that it could be done without too much effort using the existing test runner that produces its results in WDDX.

There were a couple of problems with the first implementation, first among them that JDOM was used to process the XML document. This meant that when using the cfcUnit Ant task, you had to first download and install the JDOM libaray (and any of its subsequent dependencies) somewhere in the Ant classpath. A number of people discovered just how difficult (and not to say tedious) this process was for them just to get going.

Another problem was that the data format produced by the test runner was WDDX, a format that isn't very friendly to someone trying to read using XPath expressions since it really conveys no meaning at all to anything besides another WDDX parser.

So, what did I do about this?

Well, the first thing I did was come up with a new XML grammar that better describes the test results in a much more concise way. Once the format was in nailed down, I wrote a test runner to produce that XML. Once that was done, I needed to fix the problem posed by JDOM and its numerous dependencies.

One thing that is nice about Ant is that it is configured using XML. In fact, it uses Xerces and Xalan to process build configuration files, and I figured the cfcUnit task could take advantage of the fact that by running inside Ant it already has access to these libraries. So, after a bit of work, the cfcUnit Ant task is now a lot easier to install without any external libraries.

Now, some people might be wondering why I have gone to all this trouble of producing XML in the test runner and parsing it in the Ant task. Why not just format the text in ColdFusion and just display it in the console? Well, there is a good reason for this, and its pretty simple really. By sending raw data to the client, I can give the cfcUnit task a lot of options in how it handles errors and failures in both the individual tests and in the test runner itself. It can be configured to halt a build when something fails or to keep going, or show a full formatted stacktrace generated by an error inside a test.

Anyhow, after working with Ant for a little bit and seeing how Eclipse can automatically run cfcUnit whenever a file is changed, I can see why some people have gotten all bent out of shape because cfcUnit did not have an Ant task, but I'm sure that will all change soon.

Anyhow, here are some of the things others have been saying about cfcUnit lately:

cfcUnit at CFUnited

There has been a lot more attention paid to testing CFCs this year at CFUnited, and cfcUnit is right there at the front when discussing unit testing one's CFML. Perhaps the biggest proponent of unit testing has been John Paul Ashenfelter and his talks on Agile Development and Testing ColdFusion. Additionally, I have seen demos of unit testing by Chris Scott when showing off different functionality of the ColdSpring framework.

A side effect of this attention has been that there will soon be a new Ant task written by John Paul that will allow cfcUnit to be run as a part of an automated build, but it could also be run from within Eclipse just as other unit testing frameworks do now.

Unit tests to the rescue!

It's amazing what you discover about your code when you try to write unit tests for it after the fact. When I was developing Tartan some months ago, all the testing I did was manual. I just checked that it acted the way I wanted to under certain configurations, but as I moved forward, some tests would no longer work since the code they were testing no longer existed or worked in a different manner. Really, I was testing the code to validate ideas and ways of doing things.

The biggest problem, though, was that I was not really testing small units of functionality like method calls, but in larger units of general behavior. How then would I know if something broke as I made changes to the core framework code? The simple answer is that it would throw some exception someplace and I might know how to fix it. But really, I wouldn't know since I had nothing in place to do any sort of regression testing. So far, that has worked.

A couple of weeks ago, I decided that this needed to change. As Tartan has moved forward, I have gotten a lot of requests for changes, bug fixes, and enhancements. Many of these changes aren't so simple and require a good idea of how things are put together. Some people have sent me code that makes some of these changes, but it is really difficult to see how these changes affect the rest of the code.

So, I decided that before any important changes were made to the framework, there would be unit tests to verify all of the current code. It was a tough decision since there is so much code to test and there are a lot of things I would like to do with the framework. There are several things that guided my decision. First, I really need to know exactly what the code is doing and how it can break. The only way to do this is through specific and repeatable tests. Second, as more people use the framework and the code evolves backwards-compatibilty becomes more important that ever. The only way to track this is to have tests that guarantee pre-existing interfaces and behavior. Third, I am not the only person that would like to make contributions to the codebase. As I stated earler, several people have expressed interest in improving the code and have submitted code highlighting these changes. Without any formal unit tests in place, it is more difficult for anyone involved (even me) to know if the change interferes with anything else or just doesn't work. Additionally, any new code would be given its own set of tests that run with all the others.

So far, I have written about 45 tests using cfcUnit, and I am not even half way yet. I have however learned quite a bit about some of the current weaknesses that exist in my code, and I have even discovered some rather subtle bugs that I didn't really see in my earlier testing. I have written tests for these and fixed them. I also see a lot of things I really want to change, and that has been the hardest part in a way. Part of me just wants to go ahead and introduce new features and move things forward, but I realize that in order to move forward I have to know where I stand. I can't do that without the tests. So, I make a list and move on to writing more tests.

I have been asked when the next release of Tartan will be coming out, and my answer is after these tests are written. It is an investment in the future of the framework and the community of people use it and want to contribute to it.

BlogCFC was created by Raymond Camden. This blog is running version 5.5.003.