Why automating your Sites tests will REALLY save two thirds of your QA time

Let's face it. The vast majority of Sites project are late already the day when development starts.

As a result, delivery pressure is high, budget is usually under heavy scrutiny and all the “unnecessary” development tasks are cut out, since d no one is allowed to “waste time”.

Under this so common scenario, let see what usually happens

The low-quality, late project: one years instead of 5 months

In the standard scenario (I would say, 80% of the real world projects I worked on) pressured developers rush write as much code as they can quickly, while project managers usually plan skipping altogether the task of writing automated test.

The common motivations are usually:

• “because there is no time”
• “it is redundant since there is the QA team for that”
• “We need to double the amount of code to be writter”

Often there is another motivation, but no one admit it: we do not know how to do it!

Developers are expected to code their part and test it… manually. No test code is asked to be delivered.

The big problem you face skipping test automation is the QA team needs to do a lot work. Lot more than necessary actually. The most important waste of time is the need to test and retest everything at every release cycle.

Why? Because in a manual-test-only environment, once a bug is fixed, it not fixed forever! Experience shows bugs thought to be fixed can pop up again at any moment. Also, new bugs will be introduced in old code, so testers do not trust that old bugs are gone! Since there is no way to know the quality of the whole project, testers end up repeating the whole testing cycle many many time.

If you look to real world numbers you can easily see that in a non-automated test environment, the QA cycle can takes up to 3 times the development time. So in a project initially scheduled to be 3 months of work, with 2 months of testing, you usually end up testing and fixing bugs for six to nine months, delivering the project after one full year!

The unfortunate occurrence is the consequence of cut out any “unnecessary”. Or simply not following a true agile practice (although the rush-to-code development method is often sold as an agile one!).

The high-quality, on-time project: 6 months of build-on-quality project

Test automation however, one of the key features offered by AgileSites, is something that can completely change the landscape of your project, keeping it on track even under heavy pressure constraints.

Assume that, even if you have strong time constraints, you can manage to convince the customer to add to the 3 months of planned development time, and to 2 months of testing planned, one extra month allowing developers to actually write automated tests for their code.

I am well aware that that this is a very difficult task for many project managers! I also understand that normally writing tests in sites is not easy. Actually, allowing code testability is the main reason of the existence of AgileSites ! Someone suggested that AgileSites should be instead named TestDrivenSites.

It is also true that writing tests can be seen as an additional burden to put on the shoulders of the developers. So instead of taking 3 months of developments, with appropriate tools it will take maybe 4 months. This can be seen as a waste. It is not: developers are the best people to write automated tests for their code.

Why developers should write tests

You may ask: why should developer write tests? There are the testers for that! That is, unfortunately, an inaccurate view of the development process. First, developers usually DO test their code. They need to. No developer can write a piece of code and produce instantaneous correct code.

Every developer, after writing the code, they verify if it works. At least if it works at least a little bit… The problem with code manually tested is that the test follows an an ad-hoc, unrepeatable and manual process, and is performed in a limited development environment.

As a result, developers ends up testing their code in an unrealistic environment, without integrating their work with the work of others, often limiting to test just a few use cases, and then they throw away forever this significant amount of work! Not writing automated tests usually means to throw away at least half of the development time.

Also a developer is the only person in the world to know really need to be tested, since they wrote they code. They know what they should check. They can look at their code like in a fishbowl and check the fish is doing what is expected. On the opposite, testers are looking in a black box, they have no clue what the code is doing so much of their tests are an educate guess and they may test what is actually constant, so it cannot change! So they waste a lot of time testing unnecessary parts.

Surprise! The extra investment pays off. Hugely.

If you really allows developers do that, benefits are huge.

First, code delivered to the QA teams is in a much better shape. Basically it is already largely tested and mostly works. So the QA team must check for real bugs, not for integration issues, overwritten code, forgotten features.

Second, because there is a test suite, you do not have to run an entire test cycle for every releases. In such an environment, the QA cycle is very fast. The QA team can test the work, then produce bug reports that are sent to developers. Developers will fix the bugs, but because there is a test suite, they can be sure they are not going to introduce new bugs.

Third, the QA team can be confident of the safety of the fixes, and usually they will run the full test life cycle only twice: the first time, and the last, before the release. Using automated tests, there is confidence that only new things must be tested, and not everything.

So just adding one month of extra work (actually, an additional test-while-you-develop, or build-in-quality step) your project can be actually delivered in 6 months!

Those are not hypothetical numbers, are real numbers, proved in countless projects actually following agile practices. Actually, this is Agile development.

Why it is so uncommon to actually automate tests?

Definitely in the Sites world, the biggest obstacle is the lack of tools allowing you to write real tests for your code. In Sites, JSP does not allow for an easy way to automate tests. You have to resort to tools implementing browser automation like selenium.

The problem with browser automation is you end up testing the full rendered html page, the whole thing, the final deliverable. Experience shows if have to test the whole thing without splitting in parts, it is very inconvenient since the combination of the possibilities is huge.

It is much better if you can test the building blocks in isolation. That is what AgileSites test support allows for. Next week I will write a post explaining how AgileSites actually helps you to separate html from java and write unit tests for every parts, but in the meanwhile you can check the relevant page in the tutorial: http://www.agilesites.org/tutorial/Testing.html