Monday, November 30, 2009

Services, clients, chickens and eggs

I find it remarkable how some teams manage to invent new and exciting ways to undermine the benefits of agile process and principles. The latest I have come across is what I have started to call the SOA chicken and egg dilemma - how do you develop a service and an associated client?

Here is the wrong (but sadly too common) way to approach this. Mock the client, build the service. Then mock the service and build the client. Then integrate. Each step being presented to the Product Owner as “done” separately. Worse still, some projects are using separate teams to develop each side.

See the problem? It’s mini-waterfall. Make the service, make the client, make it work. It can’t be used until you have developed both sides in isolation and integrated. Each side is based on BUFD assumptions about what it should do, and not what the system needs to do from a user perspective.

OK, so how would I handle this? Slice the functionality, not the components. Pick a nice, easy user oriented function involving the client-service combo, and make it work end-to-end. Then pick the next piece of functionality, make it work, thickening the slice. Iterate until sufficient. By all means use mocks to make life easier, but don't mislead yourself and your customer into thinking that if it works with the mock it’s ready to ship. Only ever demo as complete using a real client and service.

By approaching the problem in this way you get all the usual benefits - ability to ship working functionality as far as has been implemented, fast identification of design problems, ability to finish when there is just enough functionality.

The principle underlying this approach is simple. Done is DONE. No smoke. No mirrors. No excuses. If it can’t ship it ain’t finished.

Tuesday, November 03, 2009

Government IT failures - could we do better?

Once again the UK Government has a failed IT project. Failed as in almost three times over budget (an approximate overspend of a whopping £456 million), 3 years late (and still not delivered) and not satisfying fundamental requirements.

The chairman of the committee commented, "There was not even a minimum level of competence in the planning and execution of this project." Ouch!

So what went wrong? From the Public Accounts Committee report it appears that some of the key issues were:

  • Underestimating technical complexity
  • Underestimating the need to standardise ways of working to avoid excessive customisation
  • Poor planning
  • Poor financial monitoring
  • Poor supplier management
  • Too little control over changes

Costs and progress were not monitored for 3 years?!

Sounds familiar? It does to me - it smells like a typical "throw the requirements over the wall and hope" waterfall failure pattern.

But would a more agile, flexible approach helped us here?

Continuous delivery of requested features that were truly "Done" and ready to ship would have highlighted the project was floundering early. Iteratively delivering real features enables open, honest and irrefutable reporting, including useful data like estimated end dates, and cash burn so providing financial monitoring.

Highlighting a failing project early allows the right conversations to take place, leading to corrective action before the situation is critical. It is highly likely that problems with work practices, excessive customisation and technical difficulties would have been identified and fixed early.

Finally, controlling changes. By making the resistance to change as low as possible and applying a customer focussed "what is important" criteria the system would have had the most valuable features implemented first. This ensures the true nature of the project reveals itself very quickly - so if it is technically complex, it is discovered quickly and either mitigated (maybe the difficult feature is not that important after all?), or the project can be shelved or cancelled completely.

So in my view, using some common sense could have helped this project deliver, or at very least not burn £456 million in a failure.

There are two more points I would make. I don't know who was to blame, nor do I care. But I can see two fundamental behaviours that compounded a bad situation:

  • The Government abdicated its responsibility as Product Owner. Allowing a team to go dark for 3 years, especially when they have effectively a blank cheque, is simply ridiculous.
  • The Supplier abdicated its responsibility as development team. Even if the Product Owner is not engaging fully, I believe that a Team has a professional responsibility to inform them of progress, good or bad.

I see these failure modes all the time. But it does not have to be like this - there are a growing number of genuinely professional companies, teams and developers who could improve on this performance if only they were given the chance.

Tuesday, October 27, 2009

Liskov Ducks

In case you missed it, here is the Liskov Substitution Principle summarised in a handy motivational poster.


(from a set of originals by Derick Bailey, released under Creative Commons)


Burn up? Burn down? Who cares?

It seems that some people have some quite strong views on the "right" way to track progress. Here's my view.

In the red corner, we have the traditional burn-down graph. You start with a certain amount of estimated work, and as you finish it you cross it off the list. You finish when you hit zero.

In the blue corner, we have the burn-up graph. Here you still have a certain amount of estimated work represented as a line across the graph. Work is added cumulatively and you finish when you reach the target amount of work.

OK, so what's the difference? In my opinion, not much. Some suggest that a burn-up graph is psychologically more motivating because it is going up. Others I have spoken to prefer to see completion as zero, and like to see work left heading downwards. Personally I'm with the burn-downers.

Another factor in choosing which way the burn goes might be ease of changing the goalposts - in other words, how easy is it to add or remove from the backlog estimate. Let's throw in 50% more unexpected work into iteration 7:



Again, much the same. Personally I prefer the burn-down since to me it is more obvious that work has been injected, but again, others mileage varies.

So does it make any difference which you use? Is one "better" than the other? In my opinion, no. Having used both, as far as I can tell it makes no difference whether you burn-up or burn-down. Decide which suits you/your team and don't get bogged down in unnecessary debate about which is best. The important thing you are doing is tracking real progress, and adapting to what it is telling you.

That said, I have no doubt that debating the pros and cons of burn chart direction makes for a fascinating late night bar conversation at the conference of your choice....


Monday, October 26, 2009

So you want to be an Software Craftsman?

Dear aspiring Software Craftsman,

Here is my advice.

  • Take whatever courses you think are interesting.
  • Study closely the work of the Old Masters.
  • Stop writing software that was only designed in your own mind.
  • Stick with one technique until you perfect it.
  • Buy a book on software structure. It's the only book you need.
  • Until you can write a program without bugs you don't know how to program.
  • Stay away from Javascript. You'll never master it. Very few ever have.
  • Forget about commercial frameworks. Use Open Source. It's where the action is.
  • Visit an old age home. Talk to the people who remember 8 inch floppy disks and punch cards.
  • Learn to play chess.
  • Take a business course.
  • Do not use an MP3 player.
  • Learn a foreign language. Scala should do it.
  • Learn to cook. Please. Before you get scurvy or rickets. There are more food groups than pizza, coffee and chocolate bars.
  • Learn to play a musical instrument.
  • Learn to swim.
  • Do not litter.
  • Avoid politically correct people.
  • Avoid anyone who says "We have always done it this way".
  • Take away the keyboard of anyone who says "We need a bug database".
  • Remove the Version Control account of anyone who claims to write bug free code without tests. Better still, take away the account, and their PC, and desk, and chair just to be sure.
  • Listen to classical music and jazz. If you are unable to appreciate it at least as much as contemporary music, you lack the sensitivity to develop into an craftsman of any real depth.
  • Keep your word.
  • Do not work with anyone who you do not trust.
  • Never explain yourself. Better yet, never do anything that will, later, require you to explain yourself or to say you're sorry.
  • Always use spell check.
  • Stop aspiring and start doing.

I can't guarantee anything, but this is how you might, just might, become a software craftsman...

(Thanks to HR Giger's agent, Les Barany, for inspiration and some of the ideas that seem to apply to both crafts)


Sunday, September 13, 2009

Congratulations!

My buddies at Energized Work, Gus Power and Simon Baker, have won the Gordon Pask Award for Contributions to Agile Practice.

Their "No Compromise No Excuses" approach has proven incredibly successful, and is helping many in the industry raise their game, myself included.


Congratulations guys!

Of Waterfalls and Ponds


Just found a great metaphor to contrast waterfall development with agile. Agile is more like a pond. And you get a picnic at the end!

The Opposite of Waterfall is Pond

Thanks Alan Atlas for a very clear, beginner-friendly metaphor.

Thursday, June 25, 2009

Agile is Dead. Long live Agile!

It seems that right now "agile" is flavour of the month everywhere.It has hit the mainstream. Everyone has it on their CV, too, so we have a situation where the industry want the skills, and everyone has the skills. Everything is fine and dandy in the software industry. Right?

Dead wrong.

There is a disturbing trend appearing in the industry. It is becoming increasingly difficult to identify anyone who has genuine agile experience. Many developers, BAs, QAs and product owners declaring agile experience on their CVs, when put into a true agile/lean environment struggle to cope (to say the least). To give typical examples, consider the developer who claims "agile" experience, but does not understand (let alone able to write) unit tests or the power of automated build systems. Or the "experienced agile" guy who had a Gantt chart showing the story contents of every iteration. Or the teams who believe it is acceptable to leave the build broken for weeks on end "because it's too difficult to keep it green". I could go on. If this is the level of agile experience on the average CV, it's a joke.

The only explanation I can offer for this is that the cargo cults and corporate pseudo-agile processes are finally strangling the real deal. The team members now being churned out by these behemoths only have experience of a pale imitation of the real thing. The end result? As far as I am concerned, putting "agile" on your CV or job ad is completely worthless. Agile is dead.

Or is it?

There are still islands of excellence out there. There are still people who who actively remove wasteful practices and work leaner, faster, better. There are still companies who are trying to build experienced agile teams in order to build quality products using lightweight processes and have fun while doing it. There are companies and people who demand more, demand better. But how to find them? How can they avoid the wagile hiring trap? Two ways.

Firstly, word of mouth. Personal recommendation by someone you trust is always the best way of doing things. I have a personal network of developers, Product Owners, BAs, QAs etc who I have worked with and rate highly (you know who you are). These are the guys I would work with myself, hire should the need arise, and also recommend to others. If a client asks for recommendations, these are the guys I phone first to see if they are free. Hiring a Coach is a good way of tapping into these kinds of lists (yes, a shameless plug, but true :-) )

But what if you don't have any way into this kind of network? Since having "agile" on a CV is now so devalued, the only choice has to be audition. Get the applicant into the office. Pair with him for an hour or so. Solve a real-world problem with him. He obviously talks the talk to get this far, but can he genuinely walk the walk? If you don't have anyone experienced enough in genuine agile process to do this, don't be afraid to hire someone to do it for you - another example where an experienced Agile Coach can come to the rescue.

So long as you vet your team rigourously, any reports of the death of agile will have been somewhat exaggerated. Although somewhat beaten and battered, lightweight processes are picking themselves up and are faster and stronger than ever before. Whatever form they evolve into (agile, kanban, lean, call them what you will), lightweight processes are still the sensible way to develop software.


Thursday, May 28, 2009

If you want to go faster, raise your internal quality

Mike Hill has written an excellent article titled "How TDD and Pairing Increase Production"

An excerpt:

"If you want more production, look first to raising your internal quality.
...
All day long, every time you make a move, you will be depending on the code that’s already there. Every line of it you have to study will slow you down. Every extra open dependency will slow you down. Every bad variable name will slow you down. Every flawed design decision, be it big or small, will slow you down.

If you want to work as fast as you can, you want to work with clean code. Period."


There are many more thought provoking truths in there. Read it! It might just change the way you write code.

Tuesday, March 17, 2009

Something to ponder

If you get an infinite number of non-test-driven developers programming in a room for an infinite amount of time, would you end up with the works of Shakespeare?

Thursday, March 12, 2009

What is this software craftmanship thing anyway?

There's a buzz in my tiny corner of the industry at the moment. "Software Craftmanship". But what is it exactly?

I for one am not sure. Currently it seems to be all things to all people, but with a common theme of quality running through it. Some think it's fairly hardline, others think it needs to be flexible and all-encompassing. But one thing it's not - it's not just about agile or lean processes and practices (although personally I think they are a significant step in the right direction). So what the hell is it? Here's an attempt at defining what it needs based on existing professional engineering and other disciplines that could be considered craftmanship (for example, carpentry, surgery, architecture/building and so on).

  • It needs a set of ethics. This is what keeps you honest and provides a structure on which you can base decisions. Ethics tell you when to walk away when you are being asked to compromise too far. They provide the line in the sand that you do not cross. In XP, these are the Values.

  • It needs a set of tools and techniques that are the generally accepted best way of doing something. Think about a stonemason, or cabinetmaker - they use specialist tools in specific ways. Also a surgeon has an array of specific techniques to handle different situations. Note that there is absolutely no reason why there should not be multiple ways to achieve the same results here. However, unfortunately for us the software industry is notorious for aggressively defending personal preference and style over and above effectiveness and quality so there are plenty of red herrings to remove.

  • It needs continuous learning. A master craftsman will always be experimenting with things, looking for new tools and techniques to add to the entire craft. Since these new techniques are outside the accepted practices, they can be carefully assessed in controlled conditions in much the same way that, say, a surgeon would try a new procedure with extra precautions, or a master stonemason would try something new on a cheaper, expendable offcut of stone first.

  • It needs apprenticeship. Beginners and journeymen need to teach and be taught. An apprenticeship model works extremely well for this - see something, do something, remember something. Plus by working alongside master craftsmen you learn the nuances that can never be documented.
Once the majority can agree on the values, principles and practices that fit these four things, then - and only then - can we really claim to be software craftsmen. Until then it's simply another forum flame war.

Tuesday, March 10, 2009

Let the Craftsmen Craft Software

Ivy_House,_Runcorn.jpg

There are times when the software industry makes me despair. Making an analogy with someone building a house, you wouldn't:
  • Hire the cheapest cowboy builders you can to build your house. ("'We build you house very cheap. £500', 'OK, you're the cheapest. It's a deal!'")

  • Hire skilled builders but force them to compromise by refusing to supply them with the resources they need ("Oh we can't let you use electricity. Or petrol - it's a fire risk. Or let you move that fence that is in the way. Or..."), restricting their working environment ("You can plaster that room, but you have to do it through the letterbox") or by unreasonably restricting budget ("you can only buy the cheapest bricks").

  • Hire skilled builders but tell them how to do the job, despite not knowing anything about building ("Don't lay bricks like that. Stack them, don't overlap").
Also you wouldn't make the same mistake when building the house again and again when it collapsed, expecting that somehow things will be different this time around.

You absolutely would not build a house like this. So why expect to deliver software successfully like this?

Here's a dangerous idea for you - hire skilled people, provide them with what they need and let them do what they do best.

Thursday, March 05, 2009

Groan

A recent exchange between me and a non-developer, trying to explain that some Scrum Masters are better than others:

Me: "Scrum Mastering is pretty much the same as Team Leading. To find a good one you need to look at their past. They almost need a 'pedigree'."
S: "Oh, you mean Pedigree Scrum?"
Me: "Yes. They need to eat their own dogfood....."

At that point we had to temporarily abort the conversation.

OK, maybe you had to be there....

Scrum needs XP

I stumbled across a blog post about Scrum's origins on Jeff Sutherland's blog recently. The whole thing is worth a read in order to get a perspective on where the methodology came from, but especially interesting was the following comment:

"Few implementations of Scrum achieve the hyperproductive state for which Scrum was designed (5-10 times normal performance). Those that do all implement variations on XP engineering practices..."
This fits with my - and others' - experience over the past few years. Scrum, with a few tweaks based on personal experience, provides a good starting point for a lightweight management framework. But that's just it - it's a management framework. You need wrap something to be managed in it. And if that something is a lardy, slow mini-waterfall process then you will simply end up delivering stale garbage more efficiently.

XP has the opposite problem. It provides tools and techniques to develop quality software efficiently, but it does not provide any guidance on tracking or how to control requirements. It benefits from being wrapped in something to manage its external dependencies.

Combining Scrum and XP may not be the only way to develop software but it does seem to be a damned good starting point for further iteration. Do it - you know it makes sense.

Monday, March 02, 2009

The Right People

I talk a lot about getting the "right people on the bus" when talking about teams, especially agile teams. Quite simply, by ensuring the people involved can all pull together rather than undermine each other makes a critical difference to the success or otherwise of the team. Research suggests that this is more critical than we might think.

But how do you find these people? How can you recognise them? What traits do these people have? I've had a go at nailing down specific behavioural traits based on my experience of working with good - and bad - engineers over the years. In no particular order:
  • Courage. The "right people" that I talk about are courageous. They overcome their fear of being wrong and are prepared to challenge ideas. They are also "doers", people who have the courage and motivation to get up and make a change where it is needed.
  • Pride. Not the damaging kind, but pride as in wanting to do a job well. Professional pride is definitely one trait that separates a money focussed Mortgage Driven Developer from a Software Craftsman. Striving for improvement is always valuable.
  • Respect and Humility. Being humble is hugely important when working closely with others towards a common goal. It is the ability to sit and listen without judgement, then provide feedback and respectful challenge. It is the ability to admit when you are wrong. The ability to resist pointing fingers and scoring political points when others stumble. From humility comes respect.
  • Curiosity. I cannot think of a single successful developer or manager who does not play with new technologies and/or techniques to see if they are useful. If they work, they get added to their toolbox. If they don't work, they get put to on side until they find a use. Active curiosity stops people from getting stuck in a rut and proves they have the ability to learn.
  • Communication. The ability to communicate is essential. Gone are the days when developers can "go dark" for weeks on end. Active, regular communication engenders understanding. Rolling this ability in with respect, humility and courage creates a powerful mix that gets results.
  • Sense of fun. Fun is a useful way of releasing potentially damaging stress. I also think it is massively undervalued in the workplace. In my experience, people who have fun tend to be better motivated and have a better work-life balance than those who are 100% focussed on serious work. The better balance allows them to deliver day in, day out. And sometimes a Nerf duel is simply the only way to settle that pointless design argument....

People who have these traits are rare but worth seeking out - they are definitely out there. If you can build a development team of these kinds of people then you really will have a team to be reckoned with - if you put them on the right bus...but that's another article.

One bad apple spoils the barrel

Via Coding Horror:

"A recent episode of This American Life interviewed Will Felps, a professor who conducted a sociological experiment demonstrating the surprisingly powerful effect of bad apples."

This is not news. In fact it's something that I and other coaches have suspected for many years. But this study provides some more scientific backing for our suspicions. In short, even one misbehaving team member can upset the entire team and endanger delivery.

Getting the right people on the bus makes life much easier, massively reducing the risk of introducing agile thinking to a company and increasing the chance of successful delivery. OK, the study also suggests that a good facilitator can defuse the effect of the Bad Apple, but as the study shows, people capable of doing this are few and far between (only one team out of the entire study had a team lead capable of counteracting the problem).

Tuesday, February 17, 2009

Oxymoron

Just spotted in a job ad for a lead developer:
"You will be expected to implement a rigorous Agile Development process and work with the Technical Architect who will define the technologies and toolsets"

Wednesday, February 04, 2009

Website revamp, stage 1 complete

Finally, after several days of thinking, searching for templates, scratching head over the CMS and rewording, my poor, neglected website has been revamped. Or at least Stage 1 is complete, and it is now in a state where I can iterate further changes over the weeks/months/years. It now reflects more accurately what I actually do these days, which is an added bonus!

Tuesday, February 03, 2009

Don't compromise quality for speed

Ron Jeffries has just blogged about dropping quality in favour of speed. Take a look at Quality-Speed Tradeoff — You’re kidding yourself. Having seen many, many projects fail because of this common fallacy, it is a subject close to my heart (but there goes the draft blog article I was writing....).

In summary, there are two choices. Subjective historical addenda are mine based on my experience:
  • Reduce quality to go faster. You might get ahead short term, and you might even get ahead longer term if you don't add too many bugs and if you don't make the code uninhabitable. [But history is really not on your side here.]
  • Increase quality to go faster. There is a chance that you will over-polish and not add important business functionality. [Historically this will not happen. Especially if you make this possibility an integral part of the quality process by holding regular customer product reviews and focus on delivering important business value first.]

Monday, February 02, 2009

Google-oops!

It would seem that someone at that fine institution Google had a bit of finger trouble over the weekend.

I can't help thinking that some (more?) unit testing might save them a whole world of pain and panic in the future.

'nuff said.

Friday, January 30, 2009

Bletchley Park Needs You!

Bletchley Park is an unremarkable, run-down country estate in South East England. Unremarkable, that is, until you realise that it was where a team of boffins, including geniuses like Alan Turing, broke the German Enigma code in WW2. This allowed the Allies to intercept vital information and save countless lives. It is also likely to have significantly shortened the war - maybe by up to two years and 22 million lives. Its place in history is assured by this alone.

But even more remarkable, in order to break the code the scientists had to invent from scratch the first ever electronic computer - Colussus. Thanks to the efforts of Max Newman and Tommy Flowers and many, many others we have the computers we have today. Through their efforts, we have the software industry - the productivity tools, the games. We have levels of communication no-one could have dreamed of back then - mobile phones, satellites, GPS. We went to the moon, and are now eyeing up Mars and beyond. We have the ability to carry out advanced research, analysis and ultimate cure of crippling diseases. The potential they unleashed is endless.

Now a museum and tourist attraction, Bletchley Park is suffering from a crippling lack of funds. They need a huge amount of money to restore it to its former glory (around £10 milion). Anyone involved with the IT industry should be concerned by this - Bletchley Park represents the birthplace of our industry and as such deserves to be saved. So I urge everyone to donate just £5 to the cause, more if you can.

Also, if you have a blog or website, promote Bletchley Park's cause. Spread the word.

Wednesday, January 28, 2009

Yet another name for compromised agile process

Via Simon Voice:

wagile adj a cross between agile and waterfall process. The mess that's left if agile is only partially implemented without cultural change. See also: failure, compromised, doomed

Wednesday, January 21, 2009

Re-energised


I have just finished doing my own, personal retrospective of 2008 after spending a couple of weeks relaxing and decompressing. Definitely a cathartic experience that has left me re-energised and ready to go in 2009.

Highlights:
  • 2008 saw me go into several flawed companies/teams in succession as well as some well adjusted ones. Although I delivered results, they were nowhere near the level that I am used to getting, and the battles to get even those results were long, protracted and draining. Portia Tung wrote about something she called "The Wall". Well, I found my wall last year and although I managed to scale it, it wasn't elegant. So in 2009 I intend to only work with well behaved clients who genuinely want to change, or have already made the change to agile/lean thinking but need guidance. Life is too short not to. Consulting should not be a battle, it's a co-operation. More of that in a later blog post. But I also need to understand better what my personal "wall" feels like, how to recognise it more quickly and how I don't just scale it but burst through it.

  • One of the related problems in 2008 was isolation. Sometimes when you work as an independent consultant you can feel very alone and begin to lose focus (the ancient wood for trees syndrome). You really need to bounce ideas off like-minded people to identify root causes and figure out new and exciting ways to deal with them. So I intend to kick off a peer review group amongst some trusted friends and colleagues and see how it goes. I have used the peer review approach before but this will be more challenging since the people involved are likely to be on entirely different projects....

  • Last year I neglected the important-but-not-urgent stuff. Things like this blog, going to conferences, networking with people, getting the industry vibe. That sort of thing. So expect more blog posts, and watch out for me at the odd conference. I might even present at one (although it's probably too late for 2009).

  • In 2008 I learned a helluva lot about myself and the state of the agile nation. I am entering 2009 better educated and more aware of the issues which can only make me better prepared to recognise and solve some of them.

  • I had nowhere near enough fun last year. It was far too serious, filled with destructive conflict and injury instead. So I fully intend to make 2009 a better year - both at work and in my personal life.
So here's to a happy and prosperous New Year packed full of learning, delivering top quality software, and fun, regardless of what the current economic situation throws at us. Cheers!

Monday, January 19, 2009

You don't need bug tracking

I have just been reading a thread on an agile group discussing the best practice for bug tracking in an agile team. Almost everyone has immediately jumped in and suggested things like Bugzilla and Jira. At risk of this turning into a rant:

You do not need a formal bug tracking system in a healthy agile development team

Quite a sweeping statement. Let me explain.

The whole problem seems to come from a fundamental misunderstanding of what a 'bug' is. I define a software bug as "undesired or missing functionality". Now let's compare this to a definition of a 'story' - a story is a statement describing the desired functionality - i.e. it is an invitation to correct undesired or missing functionality... Sounds familiar!

In other words bugs are stories. And if they are really stories, then why treat them differently? Simply write them on cards1, throw them onto the backlog and let them be prioritised along with everything else. Bugs that are important to the Customer will always be done first. Unimportant bugs will get fixed eventually/never in the same way as unimportant stories would. 

[1] I use red cards so that bugs are visible in the backlog. That way if they start to multiply the team can stop and fix the problem causing it. 


Tuesday, January 13, 2009

Twitter

I have been playing on Twitter for a month or so to see if it could be useful.

As yet I'm undecided, but it has been useful enough to capture idle thoughts on the run so I'll stick with it for a while longer.

For anyone interested in following my random ramblings I can be found here.

Friday, January 09, 2009

When weekly iterations go bad

Having tried several different iteration lengths over the years, I now tend to recommend iterating weekly. It provides maximum flexibility and adaptability while being extremely intolerant of waste - problems are quickly surfaced with such a tight feedback loop.

But it does not always work as I found out over Christmas.

Our live system broke. Something ate some important files so the app collapsed. This was quickly traced to the Jetty servlet container deploying to the /tmp directory by default. Not a problem in itself, but there was a default cron script that helpfully tidies up /tmp by deleting anything that has not been touched for 10 days.... Of course, the Team was in Christmas shutdown for two weeks, and sure enough 10 days after our last live deploy...<BOOM!>

We missed this because we were using weekly iterations. We were deploying to the various dev, QA, demo and live systems every week so the deployed files were never older than 7 days.

So beware...weekly iterations can bite back!