Sunday, February 06, 2011

Testing for the Super Bowl Ad

I am sitting at HomeAway's Super Bowl Command Center patiently waiting for the game to begin.  I have not posted in awhile due to an enormous amount of testing.

We have been conducting performance tests for almost 4 weeks.  Usually on Sunday we could do our most impacting performance tests.  We ran tests every Tuesday, Thursday and Sunday.  Is our site ready?  We should know in about 5 hours.  Thanks to tools like jMeter and BrowserMob I believe we are ready!

Not only was I involved with a great team for performance testing, I also had to test some new applications developed specifically for the Super Bowl.  This testing effort was exhausting.  The final build was delivered at 1:00 PM Saturday.  Yep, that is right!  Just 28 hours before the Super Bowl.

I am confident in the software, but there are some known issues.  Very difficult to test across the variety of OS and browser configurations.  Throw in different versions of flash and media players you have an infinite amount of testing.  If I had not read Gerald Weinberg's book "Perfect Software", I think I would be freaking out right about now.

So what would be the lessons learned over the past four weeks.
  • You must performance test with a great group of technical people, because collaboration is the key.
  • You must insist on testing third party applications early and often.
  • You must learn to let go, because unfortunately with a small time line and infinite number of test cases, you can only do so much.
  • You must have good performance testing tools and a cloud vendor like Browser Mob who will go the extra mile.
  • You must have a dedicated team who are willing to put in long hours.
  • You must have testers who are willing to report all of their observations.
  • You must not use a Google Shared document to track and collaborate on issues.  Use a web based defect tracking tool. (Primary mistake I made for Super Bowl)
  • You must think about process as soon as you engage a third party vendor
  • You must have network monitoring in place.
I am sure I can think of more things later, but it is now time to enjoy the game and festivities.

Why stay in a hotel when you can HOME AWAY!

Vote for HomeAway's Super Bowl commercial!