Monday, November 29, 2010

Books I am currently reading

All testers MUST read Perfect Software and Other Illusions about Testing by Gerald Weinberg.  I just finished reading this book and I found the content extremely useful and inspiring.  If you test software, read this book.

Now I am reading Lessons Learned in Software Testing by Cem Kaner, James Bach, and Brett Pettichord.  I am only a little ways into this book, but I am finding it to be spot on.

Next in the queue is Kanban - Successful Evolutionary Change for Your Technology Business by David J. Anderson.

If you are reading a great testing or development book, please make a comment so I can get more books into my queue.

Read on!

Sunday, November 21, 2010

Think Times for Performance Testing

After only 10 years in the software industry, I am probably naive, but a recent post by Dan Barstow has caused me to question the use of think times.

Our current performance testing approach does not have think times.  I have used think times in the past for various applications.  I always have found think times to be somewhat arbitrary and subjective.  I can definitely see the value if your objective is to really understand the number of concurrent users the application can handle, but typically our objectives are focused on the number of transactions a system can process.    Also if you are using virtual users you truly are not representing a browser,.  A browser, depending on type, could leverage 6 threads at a time making requests.  So how do you adjust the think time between thread A getting an image and thread B retrieving javascript and firing the script?

Specifically we look for what is the optimal TPS value where 95% of requests latencies are less than 1 second and 99% of all requests are less than 4 seconds.

In my previous career I was a chemist.  On some regular frequency we had to calibrate our analytical instrumentation.  We would generate a 5-point curve using known standards.  Typically we sought a linear relationship of this curve where the low point was near the detection limit and the high point was near to the maximum detection point of the analytical instrumentation (e.g. – Gas Chromatographs).

When performance testing ramping VU in time intervals is similar to generating this calibration curve.  What I am typically looking for is how high can we generate the curve, yet still be below the high point of the instruments capability, which in this case the instrument is a configuration of servers and services.

I guess my philosophy might be that by eliminating think time I can get to that high point a bit faster.  Since we are testing a system, then my criteria is applicable to the total request and response.  So if the request is for an image or for a database object, as long as that request is returned within the criteria stated above, then we are within calibration.

My opinion is that if you are using virtual users then transactions per second is the important metric.  If you are using real browser users, then I think that think times might be important, but still subjective.  Regardless the objective is to do the best job possible to validate that your site gives customers a good experience.  I guess this can be achieved with or without think times.

For the record I have done some performance testing of JSON API where you could not even conduct the test without some level of think time.  So I will refine my conclusion to state that it depends on your objective and the application under test.

Please help educate me, because I am not in the think time camp unless it is necessary.

Wednesday, November 17, 2010

Selenium Fury has been released!

In my humble opinion, the page object pattern can be extremely important for a test framework.

Do you have an ever changing UI that causes your locators to frequently break?

With do your developers frequently move elements around the page?

Do you have a brand new page and you quickly want locators to work with?

Selenium Fury just might be the tool to assist you.

I have posted in the past about generators and validators.  Scott Sims has rolled this functionality up into a nice gem.  The gem is easily extensible.

Check out Scott's post, use the gem and let him know what you think!

Oh, it is so fun being a tester!

Thursday, November 11, 2010

Pirates vs Ninjas make it to Prime Time

I was catching up on some recorded shows.  I was watching my new favorite show, The Defenders.  Funny that it is set in Las Vegas the last location of STPCon.  Any rate last nights episode involved a hacker.  The main characters mentioned pirates and ninjas when talking about the hacker.  Were they observing Adam's and Lanette's Lightning talks at STPCon.  I found it very funny to hear pirates and ninjas mentioned regarding software on a prime time show.

The lightning talks were one of the highlights of the conference.  They were put on as breakfast bytes by Matt Heuser.  I love the format of lightning talks.

Since STPCon I have done some thinking about "pirates being better than Ninjas".  I have to respectively disagree with Adam Goucher.  I think ninjas are better than pirates.  Pirates might support developers in a loud and boisterous manner.  Ninjas would support developers in a silent and stealthy manner.  The good news is that testers exist to support development.

I guess it was obvious that I lean toward ninjas, hence the title of this blog.  That reminds me I need to get my ninjapus image uploaded to this humble blog.  My kids worked really hard prior to STPCon to help me create my logo.  I need to shrink it to appear on the blog, but I can certainly include it in this post.  You have to love the power of a ninja with 8 arms and multiple weapons for testing.  I think my kids 11 & 14 did an excellent job!  Go Ninjas!

Wednesday, November 03, 2010

Tester and Developer Pairing

I know at STPcon Lanette Creamer gave a session on "Pairing with Developers".  In hind site I truly wish I had attended that session. 

Currently one of our development teams is pushing out two releases per week.  There are a couple of issues in that the code is actually complete in the previous sprint and it takes test 1-2 weeks to execute regression.  This is not meeting the needs of the business, so we have added in "quick releases" with minimum code changes and minimal testing.  On top of that we are also injecting releases where we are conducting A/B testing (more on that latter).  In order to keep up an efficient pace, the team is now considering daily releases.  Wow!

This is essentially a shift to Kanban.  If you listen to Kent Beck, moving to this type of SDLC a team would essentially remove the testers.  I respectively disagree.  Testers can still play a critical role in the reporting of quality and making sure the releases are meeting business and customer needs.  As I ponder this paradigm shift, I believe "Pairing with Developers" becomes essential.

Developer grabs the next story and pairs with an experienced tester.  They work together design the unit tests.  As the developer implements the unit tests, the tester designs the acceptance tests (Cucumber) and developer does a code read.  As the developer writes the code, the tester implements the acceptance tests.  The tester should also collaborate with the developer to design and implement any browser facing tests (Selenium).  Integration tests should also be considered.  At the end of the day the code should be complete and all of the tests pass.  If all tests pass, then automated deployment pushes the release for UAT.  UAT Testers evaluate the functionality in a non-customer facing production environment.  If the user acceptance tests pass, then switch the environment to be customer facing.  In the A/B world we can expose only 20% of the customers to the new feature.  If there are no issues raised by the customers, then we push live. 

There may be other ways to complete the test automation, but this paradigm makes sense to me.  Not all testers have automation experience, so this places the emphasis on education and training.  How quickly will we get there? I do not know.  Is it possible?  Absolutely!

I am looking forward to this challenge.  If any of you are actively doing this style of rapid development please share you thoughts.

Gotta LOVE testing!