Measurements in Agile vs. Predictive Projects (1)–Earned Value Analysis

I am going through the Agile Alliance Guide chapter by chapter, section by section, page by page, in order to understand its contents.   I am blogging about what I’ve read in order to add value for those who may be coming from a traditional project management background.

So, for example, I am starting section 5.4 on p. 60 which deals with measurements in agile projects.   How can you tell whether your project is on track for success or not?

Before I discuss the introduction to this section on p. 60, however, I want in this post to discuss how measurement is done on a traditional project, which usually has a predictive life cycle.   In this way, the contrast to how measurement is done on an agile project can be better appreciated in context.

Let’s review what the characteristics of a predictive life cycle are as compared to the other three types of project life cycle (iterative, incremental, and agile):

  • Requirements–these are fixed at the beginning of the project and change is managed as if it were a necessary evil (the other three life cycles have dynamic requirements and change is managed as if it were a positive good)
  • Activities–these are performed once for the entire project (in incremental, they are performed once for a given increment, and in iterative and agile they are repeated until correct)
  • Delivery–there is a single delivery of the final product (this is true in iterative as well, but in incremental and agile there are frequent smaller deliveries during the course of the project)
  • Main goal–to manage cost (in iterative the main goal is the correctness of the solution, in incremental it is speed, and in agile, it is customer value)

The main characteristic to focus on in our discussion of measurement is the first one, that of fixed requirements.   This allows you to breakdown the work with a tool called the Work Breakdown Structure.   This is then used to create both the schedule and the budget.   These in turn are used as inputs to a data analysis technique called Earned Value Analysis or EVA.   Remember, the goal of measurement is to

  1. compare the actual work done vs. what was supposed to be done and see if there is a variance (I call this the “snapshot” function of measurement)
  2. predict what resources will be required by the end of the project to complete it given the trends of those variances discovered (I call this the “crystal ball” function of measurement.

Okay, let’s go back to how EVA works.   The three building blocks of formulas dealing with variance are the following, which are based on the triple constraints of schedule, scope and cost, respectively.

  • Planned value (PV)–this is the authorized budget assigned to the work that is scheduled to be done.   (The key word there is “scheduled”.)
  • Earned value (EV)–this is the measure of work actually performed.   (The key words there are “actually performed”–it is a a measure of how much scope was accomplished.)  This is expressed in terms of the budget authorized for that work.
  • Actual cost (AC)–this is the measure of the actual cost for work actually performed.  (The key word is obviously “cost”.)

Let’s take a simple example to show how these are used.   Let’s say your project is to have a room painted, assuming that all of the preparation work has already been completed.  It will cost you $1,000 per day to do each of the four walls of the room.

Scenario one:   At the end of day 2, you only have one room painted.   Your gut feeling is that the project is behind.   What does EVA tell you?

The schedule performance index (SPI) is EV/PV.   What is EV?   It is the end of day 2, and you actually did only one wall, which costs $1,000 to do.  So your EV is $1,000.   What is the planned value?   Although you did one wall, you were supposed to (according to the schedule) do two walls, which would cost you $2,000 to do.  So your PV is $2,000.   Plugging these values in the formula, you get 0.5 as the result.   If your result was 1.0, you would be on schedule.   If your result is greater than 1.0, you are ahead of schedule, and if your result is less than 1.0, you are behind schedule.

The cost performance index (CPI) is EV/AC.   Let’s say that, it is the end of day 2, and were able to complete two walls, but on day 2, you realize you needed to add an extra painter (assuming you planned for two painters costing $500 each for labor and materials, but had to add a third painter in order to get the work done on time).   The SPI is going to be 1.0, because you are on schedule.   But what about the CPI?   The EV is $2,000, because two walls were done, but the budget authorized for doing two walls is $2,000.   The AC, however, doesn’t care about the budget authorized the work; it cares about what the actual costs of the work turned out to be.   This is $1,000 for day 1, but $1,500 for day 2 because of the extra painter, so the cumulative AC is $2,500.   Now using the CPI formula you get $2,000/$2,500 or 0.80.   Anything less than 1.0 with either the SPI or CPI is not good.   In this case, it means you are over budget.

So you can see how this works.   Because the schedule and budget are, at any one time, fixed, you can use EVA to create a measurement which is precise.   But it is accurate?  Let’s review what these two words imply.   A measurement is precise if the measurements are closer together (using smaller units).   A measurement is accurate if the measurements are close to the actual value that is being measured.

If I am at a pub playing darts, my precision will decrease as the amount of beer I have increases.   The fine motor control needed to throw the dart exactly where I want is affected by the alcohol, and I end up throwing more wildly as time goes on.   Now the accuracy increases the more times I play in the pub, because my brain learns and causes me to get a bulls-eye more often.

The fact that EVA measurements are precise can fool you into thinking they are accurate.  If someone says they are 90% completed with their work, you can use this data to say that the SPI is 90%.   But what if their own self-assessment is wrong?  Or what if they don’t want to admit that they aren’t as far along as they should be?   If someone tells me that the completion of their work is “just over the horizon”, that might make me feel better–until I look up the word “horizon” in the dictionary and see that one of the definitions is “an imaginary line that gets farther away from you the closer you approach to it.”   Well, that’s not comforting at all, is it?

And it gets worse.   Let’s say that the person IS actually telling the truth and that the work is 90% done.   They turn in the work and then the testing is done, first of the module itself and then of the entire system with the module added (an integration test).   It doesn’t work as it is supposed to, and it has to be reworked.

Okay now it is re-tested and it works fine.   It is delivered to the customer and the customer says “that’s not what I ordered.”   The user story was not clear or objective enough, and what the customer thought was being ordered isn’t what the team thought.  This is why you don’t use subjective acceptance criteria like “I want the product to look nice” because there’s a LOT of room for disagreement about what “looks nice” means.

So whether from your standpoint you are within the budget and/or schedule, in the end run it doesn’t matter if the work that was done does not add value from the customer’s standpoint.

So, as opposed to earned value analysis, which is a more precise measurement tool but may have problems with accuracy, measurement in agile projects uses tools that may not seem at first glance to be precise from the standpoint of traditional project management, because they are dealing with units (user story points) that seem to be more subjective than dollars and cents, but they are more accurate in that they are focused on actual customer value.

With that background of measurement in traditional project management in mind, let’s turn in the next post to characteristics of measurements in agile projects.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: