Agile architecture metrics

In developing software there is a need for communicating ideas and chosen directions among team members and customers alike. Some sort of overall view and common vocabulary about the domain provides understanding of the system and is a basis for discussion. Software architecture provides such a “gestalt perspective” and is therefore very useful in any but trivial software development effort. In agile methods there is no BUFD, but how do you know you are doing enough architecture?

Lets take a quick look at what three well known methods have to say about architecture.

eXtreme Programming

In XP architecture is just as important as in any other software project [Beck00: pp 113]. XP used the notion of the system metaphor to cope with software architecture up front. The system metaphor is used to create the big picture of the system, the vision of the system’s components and how they work. About 8 years ago Kent Beck defined it as:

“System Metaphor - A story that everyone – customers, programmers, and managers – can tell about how the system works.” [Beck00: pp. 179]

You can learn about the problem at hand by using spikes. The spikes drive the System Metaphor. So even in XP you can have initial “architectural” work to set the baseline for understanding, communication and vision. Kent Beck was quoted saying “The first iteration must be a functioning skeleton of the system as a whole.” [Wake02: pp78].

Unified Process (they say it’s agile)

The UP creates high level architecture in the inception phase. It then creates an appropriate executable architecture during the elaboration phase. The architecture tackles risks, develops a common understanding and is the basis for identifying reuse and for further development during the construction phase [Kroll03: pp120].

Feature Driven Development

FDD suggests that architecture could evolve in parallel during the Develop an Overall Model, Build a Features List and Plan by Feature processes. The goal is to have enough architecture ready to start iterating though the Design by Feature and Build By Feature processes [Palmer02: pp. 204]. During the first iterations a lot is learned about the system and the architecture keeps evolving until it becomes more or less stable during the project.

The common thing here is that they all do upfront work prior to development to establish common understanding about the system domain and functionality. UP does a lot, FDD does less and XP does the absolute minimum. The effort for getting the big picture right is higher initially and then decreases as iterations and increments pass. In agile projects you keep on learning and adapting and therefore do not stop doing architecture work until the end of the project.

So all this would mean that in a healthy agile project you will see something like depicted in the figure below.

Initially you spend more effort on architecture compared to developing features. The amount of architectural work will decrease over time but will only be done when the project is finished. The effort spend on features will increase. So you should also see a steady increase in the number of Running Tested Features http://www.xprogramming.com/xpmag/jatRtsMetric.htm.

By defining some architecture upfront you optimize your throughput not at the level of the current iteration but at the release level as discussed in YAGNI by the book.

References

[Beck00] Beck, Kent. Extreme Programming explained.

Addison-Wesley. 2000.

[Beck01] Beck, Kent and Martin Fowler. Planning Extreme Programming.

Addison-Wesley. 2001.

[Wake02] Wake, William C. Extreme Programming Explained.

Addison-Wesley. 2002.

[Palmer02] Palmer, Stephan. Feauture Driven Development.

Prinice Hall. 2002.

[Kroll03] Kroll Per and Philippe Kruchten. The Rational Unified Process Made Easy.

Addison-Wesley. 2003.

YAGNI by the Book

I recently attended a workshop on Agile architecture by James Coplien. He made some very interesting remarks. He claims that your domain architecture,”what the system is”, is relatively stable. The domain of a bank or insurance company will not change that rapidly! Things that do change have to do with “what the system does”. The user stories and the user interface just to name two. He reasoned that you are creating “knowledge debt” if you know something but do not act upon it.

He further claims that doing too little architecture up front gets you into trouble around the third month of an agile project. To cope with new requirements and gained knowledge about the problem domain a lot of ‘refactoring’ or re-architecting as he calls it is needed. As a result your overall velocity goes down!

I personally experienced various projects that had to do excessive refactoring as sprints passed. The refactorings had a severe impact on velocity. In every one of these projects the misapplication of the Simple Design and YAGNI principle was part of the cause. The misapplication is in applying it to the letter. If you do that you are saying, “I focus only on what is valuable in the present and I assume I know nothing about the future because nothing is certain”.

The good thing is that you CAN know things about the future pretty accurately. There are always things you know upfront and that are relatively stable. “You know the sun will rise tomorrow”! It may not…. but I’ll bet you it will. If you are building a family car you know that it will require brakes. Putting in a steering wheel is not enough.

So if you apply YAGNI to the letter you are playing stoopid. You should use your knowledge about probable future things and act upon it. If this means putting in flexibility into the system this iteration because in the next you WILL need it, then just do it. I am not saying you should build functionality in the present it if it’s not part of the current iteration. I am just saying that you should always keep the design as good as it can be. Putting in flexibility for a highly probable feature in the near future sounds like good design to me!

Should you put all work on the product backlog?

In a particular project I was involved in the stakeholders and the product owner was not interested in the technical tasks the team was working on. Technical tasks could be anything from setting up environments to software architecture work. The stakeholders were only interested in the user stories the team was working on and the progress in terms of completed user stories. The product owner actually told the team to stop putting technical tasks on the backlog because they could not relate to them and therefore could not assess their value or even prioritize them. Of course you write as much of the technical work in terms of a user story. But there are always technical tasks you need to do that are e.g. needed for multiple user stories.

So in the end the team satisfied the product owner and ended up with a product backlog with just user stories!

So where do you put all this technical work?
First of all, the team estimated the user stories without any assumption that any other stories or any technical tasks are already done. This approach improves user story independence and provides the product owner with the greatest flexibility on prioritizing the backlog. Because there were only user stories on the product backlog it also helped the team focus on developing only that part of the software architecture they actually need in order to realize the user stories at hand. It acts as a natural guidance for the team in applying the principle of Simple Design and YAGNI. So the technical work gets divided over the user stories and emerges as tasks on the sprint backlog only when and to the extend that they are really needed.

You might think that the big downside to this approach is that it hides work and does not provide the most transparency. I would argue that because architectural tasks pop up in the sprint backlog you are being transparent to the people who are interested in technical tasks, without confusing the ones who are not.

As a result of all this the team’s velocity is lower in early sprints and increases and eventually stabilizes over time because it benefits of the architectural work already done. It’s obvious that your velocity will increase once you have stable software architecture because you have to do less work.So it’s probably the case that stable software architecture gives you higher velocity. The interesting question is ‘how much upfront architecture do you need?’

This experience showed me the being transparent as a team is very important but in order to be so you don’t need to write ‘all’ your work on the product backlog.

User story driven performance testing

Quite often performance testing is only done after the system is developed and is in it’s acceptance phase. Of course this can give you some unpleasant surprises. You would like to get feedback on performance issues as quickly as possible.

We could tackle part of this problem by doing performance testing each iteration and by making performance testing part of the daily or continues build. To make the test results most valuable, the test environment should resemble the production environment as much as possible. I know that it’s very hard to realize this especially in big projects at big companies. But even if you cannot have a representative environment, doing performance testing each iteration can give you some good insights into performance trends during development. If some piece of functionality is getting slower and slower it could indicate that there are performance issues. Having insights in these trends gives you the possibility to investigate these possible performance issues early on while fixing them is still cheap.

JUnitPerf is a library for decorating JUnit tests to measure the performance of functionality contained within existing JUnit tests. You can write TimedTests that test if a JUnit test finishes within a certain time. You can also write a LoadTest that lets you measure the elapsed time under concurrent use. Because it is based on the Decorator pattern you can easily combine load en timed tests to create more complicated ones.

Using JUnitPerf can give you trends based on written JUnit tests. But a JUnit test is not that suitable for performance testing. A JUnit test fixture can have a number of various tests including expected failures etc. What would really be interesting is measuring performance of functionality that is valuable to a system user. You would like use your user story scenarios not only for acceptance testing of functionality but also for performance measurement.

The idea is to use a BDD framework like easyb for writing your acceptance test scenarios for a iteration as I explained in Easy Requirements by Example. The scenarios are complete pieces of user valued functionality and so are a perfect candidate for performance testing.

It would be great if you could write something like this in easyb for measuring performance of individual steps;

performancetest {
given “scenario X”, {
Scenario x = new Scenario(“X.groovy”);
}
when “load is generated by 10 persons”, {
x.load 10
}
then “measure individual step performance”, {
x.measureSteps
x.start
}
}

or the following for a timedtest under certain load;

performancetest {
given “scenario X”, {
when “load is generated by 10 persons”, {
x.load 10
}
then “response time should less then 3 seconds”, {
x.start
x.responseTime.lessThan 3
}
}

or just for a timed test as

performancetest {
given“scenario X”, {
when “load is generated by 1 person”, {
x.load 1
}
then “measure end to end performance”, {
x.measureEndToEnd
x.start
}
}

I am currently working on getting this done in easyb. I’ll write about it in the next weeks.

Easy requirements by example

There is this fun framework called easyb http://www.easyb.org/ that could be used to make you stories or requirements by example executable in an easy way.
In the article ‘Introducing BDD’ http://dannorth.net/introducing-bdd Dan North discusses BDD and uses the following user story for explanation.

Story: Customer withdraws cash
As a customer,
I want to withdraw cash from an ATM,
so that I don’t have to wait in line at the bank..

Well, now let’s assume you are in a meeting discussing this user story and as a good team you ask the customer to give some specific examples of it. During the discussion you are of course helped by the testers who help the team also focus on ‘things the system should not do’. At the end of the meeting you have a good understanding of the user story and have a couple of written examples such as the one shown below.

scenario Account is in credit
given the account has 10 dollars in credit
and given the dispenser contains 1000 dollars
and given the card is valid
when the customer requests 5 dollars
then ensure the account is debited and now contains 5 dollars
and then ensure cash is dispensed from the dispenser and the dispenser now has 995 dollars

It would be great if you could translate this easily to an executable specification so that you could start developing your Java or Groovy code against it. The good news is that you can do this easily with easyb. You could write your test as follows:

scenario “Account is in credit”, {
given “the account has 10 dollars in credit”, {
//do some stuff here
}
and
given “the dispenser contains 1000 dollars”, {
//do some stuff here
}
and
given “the card is valid”, {
//do some stuff here
}
when “the customer requests 5 dollars”, {
//do some stuff here
}
then “ensure the account is debited and now contains 5 dollars”, {
//do some stuff here
}
and
then “ensure cash is dispensed from the dispenser and the dispenser now has 995 dollars”, {
//do some stuff here
}

}

This almost the same as the text from the scenario and is easy for non technical people to read and understand! Once you have this you can start developing and could arrive at something like this:

scenario “Account is in credit”, {
given “the account has 10 dollars in credit”, {
accountInCredit = Account.create(10)
}
and
given “the dispenser contains 1000 dollars”, {
dispenser = Dispenser.create(1000)
}
and
given “the card is valid”, {
card = Card.create(accountInCredit)
card.enabled = true
dispenser.insertCard card
}
when “the customer requests 5 dollars”, {
dispenser.requestCash 5
}
then “ensure the account is debited and now contains 5 dollars”, {
accountInCredit.amount.shouldBe 5
}
and
then “ensure cash is dispensed from the dispenser and the dispenser now has 995 dollars”, {
dispenser.amount.shouldBe 995
}
}

Well…. you probably would not arrive at something like this in a real system but this is just an example…
You could use this for unit testing, using the groovy build in possibilities of mocking and stubbing and if that’s not enough you could use Java frameworks like EasyMock, and DbUnit to name a few.

But I see more value for this on integration testing your system and creating a suite of acceptance test scenarios for a iteration. You can easily mail it, print it to discuss and its easy to come up with more test cases once you have a example.

Getting feedback on iterations

An important thing in iterative development is the information you generate during iterations and the feedback you get from an iteration result. Even more important is what you eventually do with it!

There are various practices that guide us in generating and handling this information. In scrum we have e.g. the daily scrum, sprint demo and retrospective. More general we see that successful agile organizations implement patterns like ‘Group Validation’, ‘Engage Customers’ and ‘Producers in the Middle’ as described in Organizational Patterns of Agile Software Development by J. Coplien et al.

One part of the feedback loop that can often be improved has to do with iteration acceptance. It would be great if after each iteration demo the results are accepted as DONE. Also very good would be having iteration results including approved requirements going to the acceptance team and having the verification in parallel with the next iteration.

What I see in projects is that iteration results are often not properly handled due to various organizational issues. This reduces the feedback loop limiting the team in their ability to adapt and improve their work.

How can we improve our iteration acceptance to give us better feedback? You could take a piecemeal growth approach. First do some exercises and get people fit, once they are fit they can become agile!

How to become fit on iteration acceptance?

Break it up!
Instead of trying to get the organization do acceptance after every iteration right from the beginning. Focus on having the organization accept the result of multiple iterations first. It is easier to get the acceptance team to accept a complete feature set of functionality instead of a partial feature set. Make sure that the end result includes approved requirements and any other required artifacts that can go to up for acceptance i.e. installation scripts, training materials etc.

It’s also difficult to go in one step from complete requirements sign off to sign off at the iteration level. Therefore you could begin by having requirements specifications signoff for multiple iterations. This is mostly better understood and not as alienated to people’s currently working approach of bigger signoffs. It helps people getting used to working on smaller batches of work. It also helps them focus on the set of requirements that they want developed first.

No dull demo!
We all know that it’s important to have written acceptance tests or requirements by example see http://www.testing.com/writings/2004-roadmap.html. The benefits here are that most people work easier from examples instead of from general often much more abstract descriptions and that it improves knowledge transfer.

To get to this point you could start with installing the iteration results on a system where the customer and stakeholders can actually use it. Show it around, let them play with it, help them using it and encourage them to test some specific cases they think are important. This generates lots of feedback and is relatively easy to implement. Once people start actively using the demo systems and try it out with some of their own test cases, you can start thinking about asking them to write their acceptance tests before the iteration.

Once people pick this up and they see its benefits you can start working on making the feedback cycle shorter by having the acceptance team accept every iteration result if it passes THE acceptance tests.