Continuous Integration

I will have to do a critical analysis as to why I do not blog more often, but I will save that analysis for another time.

STP Conference Fall 2011 is only about a month away.  I am actively trying to prepare.  I thought I would make an attempt to finish out my experiences from STPCon Spring 2011.  This blog post brings me to Session 702 - Turbocharge your Automated Tests with CI presented by Eric Pugh.

An interesting side note is that I mentioned Eric's name last Thursday when I noticed one of our development teams is implementing Solr.  From talking with Eric at the conference I knew he was an expert, so tossed his name out as a potential resource for collaborating on complex implementations and bottlenecks.

I must say as I recall, Eric was a poised presenter and really knows his stuff!

Continuous integration is something I am a novice at so Eric's presentation was a good one to attend.

He defined CI by using a quote from Martin Fowler - "A fully automated and reproducible build, including testing, that runs many times per day".  That makes great sense to me.

During the presentation Eric had the audience do an exercise where in just a few minutes of a discussion with the persons next to you find out how many non-obvious things we had in common.  The sad part is that since March I do not recall the true intent of the exercise, but what I liked was that it got the audience engaged.

Eric then did a live demonstration of his CI system.  Doing a live demonstration is always a challenge at these events and he did a great job.  I will admit I did not understand all of the components, but it did put things more into perspective for me.

Another key takeaway for me is that he mentioned that the results should be readily available and readable at a glance.  He even talked about places where flashing lights and lava lamps would ignite if a build ever failed.  There was even a photo of Agnes showing the team a gesture for breaking the build.

There was a ton of great information in this presentation.  I walked away from the presentation more informed and with a desire to learn more about how to best leverage CI.  Some of the automation that I do takes to long to execute so the tests do not run with every build.  Currently we run those nightly.  So I am encouraged to take a closer look at automation and push into the build process the automation layers that make sense.

You can find Eric's presentation here if you seek greater detail - http://www.opensourceconnections.com/oscshare/eric/Core%20Principles%20of%20CI_Session.pdf

I still have a ton to learn about CI.

One other great side effect of meeting Eric Pugh is that he provided me with some great constructive feedback on my Cucumber presentation.  I am new to giving presentations so I found it admirable that we was willing to kindly give me suggestions for improvement.  Eric did not even know me, but offered mentoring.  "Nice Bike" to Eric Pugh.  It was also nice to have a couple of cold beverages with Eric at the conference party at the WildHorse Saloon!

Thoughts from Agile Austin QA SIG Meeting on August 17

I am taking a slight detour from catching up with my spring STP Conference experience.  Once a month a group of testers meet as part of Agile Austin.  The topic was TDD and BDD.  Jill Ott was the presenter and she did a good job introducing the topic.  For some reason I walked out of that meeting completely unsatisfied.

This is my attempt at a personal retrospective.

One comment was made saying that testers really do not get involved in TDD.  I consider myself a tester and when collaborating on building an open-source testing framework, I found TDD necessary,  TDD is an excellent practice to prove that your ruby methods work.  One person did comment saying that testers could certainly add value by reviewing the tests created through TDD.  I agree with that statement.  The conversation did not advance beyond a couple of comments.  I guess I was not satisfied because I wanted more detail on how developers leveraging TDD create better code and increase tester confidence.  After recently learning TDD, I found it extremely valuable as a tester.  I guess I had expected more testers in the room to be developing automation code and that they would have an opinion on whether TDD was easy to do with automation code.

The conversation on BDD seemed to bounce around two themes: requirements and brittleness of automation.  One person gave a couple of great working examples of BDD.  One example where it appeared to work extremely well and a counter example where things did not go well.  I did have to admit that the reason I considered BDD is because to many legacy requirements were in peoples heads.

I guess I was not satisfied because I wanted to talk details with respect to cucumber implementations and the processes surrounding the implementation.  We never got into that level of detail that I desired.

• When in the process do Cucumbers get written?
• Who is responsible for writing them?
• Who is responsible for building the step definitions?
• How well do teams get reuse out of their step definitions?
• How do you avoid cucumbers being scripts and focus on specification?

There were also threads within the conversation about the cost of automation.  Yes automation can be expensive, but I am of the opinion that automation is extremely necessary.  I tend not to pull cost into the game, but rather focus on the value that automation can deliver.  I concede that I probably should factor in cost a bit more, but I do not permit cost to be a deterrent.

So My conclusion as to why I walked away dissatisfied is because  I did not get the value I had hoped out of this meeting.

I am now pondering what could I have done differently to get more value out of this meeting.  There were 20 persons in the room and I knew 3 or 4.  I think if I attend this meeting again, that I should arrive early and meet more of the attendees.

I think I may have to either speak out more or get my questions on the table early.  I was a bit hesitant to chime in, which is unusual for me.  Approximately 4 -5 of the 20 people tended to dominate the conversation.  I ponder how can we facilitate more participation.  I am wondering if I should have submitted in advance of the meetings the questions I was hoping to address.

Perhaps I should follow up with the user group with the questions posed above.

I think I will do that!

"How to win an Unfair Fight..."

Continuing the efforts to catch up on what I learned at STP Conference in Nashville.

The Tuesday evening key note speaker was Garrison Wynn.  His presentation was entitled, "How to Win an Unfair Fight: Influencing People You Don't Have Authority Over".

• Get an understanding of what people really value and how that impacts agreement
• Develop strategic advocates and create your own personal "influence upward" plan
• How to get people to agree with you
• Why some people disagree with everything and what you can do about it
• How to get people to listen to your ideas

The above bullets were the target points of Garrison's presentation.  He delivered a presentation with great poise, energy and most importantly humor.

Unfortunately I do not remember all of the key lessons, but I do remember laughing.  After the key note I spoke with him in the hallway.  Of course I had to purchase an autographed copy of his new book, The Real Truth about Success.

I have slowly been reading this book and I have found some very insightful items.  Here is a quote testers can apply toward the continuous mission of honing our craft.

"Approach life talent first.  Find and create your personal advantage."

The book takes a close and humorous look at "what the top 1% do differently, why they won't tell you, and how you can do it anyway!"  There are examples of the traditional dressing for success, treating others has you would like to be treated, and shifting a negative attribute into a positive asset.

You too should find your secret weapon and not be afraid to use it.

STP Conference in Nashville Continues - Virtual Systems

Choosing a session for Block 2 was just as difficult as block one.  There were three topics of interest to me and two familiar speakers.  There were sessions on Mobile Testing, Agile Testing, and Performance Testing.  Mobile testing certainly is the newest kid on the block, but I opted for performance testing.

There were three reasons I chose performance testing.  For one my colleagues were going to the Agile Testing session by Bob Galen.  Second I love learning more about performance testing.  And finally the presentation was being given by Dan Downing, who I consider a mentor in the field of performance testing.  The session was entitled "Performance Testing Virtualized Systems".

Dan introduced the topic by highlighting the pace at which organizations are moving to virtualization and the many pain points associated with this movement.  It was the next slide that captured my curiosity.

Dan would venture to explain the six critical factors for Testers.

The first was "anatomy of a Virtual System".  I will be honest in that I really do not know much about virtual systems.  So the key point for me was that I needed to know and understand the system under test.

Mapping workloads was the next topic.  Dan is brilliant enough to do this type of work.  This is something I would have to seek assistance on.  None the less if you are going to do performance testing you must know how data flows through the system and how various conditions can influence the data flow.

Dan's next point was regarding bottlenecks of a virtual system.  My first thought was that is why we are performance testing to find the bottlenecks in the system.  True that is why we would be performance testing, but Dan's point was understand the applications to confirm they are properly distributed based on you knowledge of the dedicated system.  So for me this tied the first to points together in that you need to understand the potential areas of risk and design your performance testing to properly measure the key areas of risk.

The next topic was about testing technique.  Dan talked about executing the tests in parallel.  Do not let time gaps call into question your test results because system performance can have different influences depending on the time of day.  One example might be that late in the evening database backups take place or ETL pulls moving data from system A to system B.  Honestly I do not think in my short performance testing career this technique occurred to me.  It makes sense because I have spent time in the past trying to understand nonsensical data.  If you have the resources to run in parallel, then I think this is a great idea.  If you do not have the resources then it comes back to the first three points about knowing the system under test.  Often performance testing is influenced by many disparate systems so you will still have potential for unexplainable results.

Mr. Downing talked about Test execution.  He split the load into three different phases.  Start with a  "light" load.  His point was to establish a baseline for each system where the system is performing optimally and there is limited competition for resources.  Then progress into a medium load test where everything is scaling properly and their are no bottlenecks, such as a database.  Finally move to the stress mode where you determine failure points and that the systems fail in the appropriate manner.

The next session of the talk was of course regarding the zillion of metrics you should measure and monitor.  I think Dan is spot on mentioning all of the measurements.  And if you have the tools and resources you should absolutely look at every thing.  Unfortunately when I have the time to help with performance testing I am squeezed for time.  I try to pick out maybe 5 key metrics to gather results on and use those results to determine if further monitoring is needed.  Performance results data can be overwhelming.  Dan certainly knows what tools to use and is a wizard at data analysis.  Having tools and systems in place to conduct real time monitoring can certainly ease the data analysis paralysis.

Dan spent some time in this presentation on data analysis.  The key take away for me was he had a focus on comparative data.  The goal of this performance test was to compare a dedicated system to a virtual system.  Assuming you know the dedicated system performance from a historical perspective, then you have the best heuristic.  Show data for both systems in the results.  Differences become more apparent and you then can investigate those differences.

For me it was a thought provoking presentation.  Dan concluded by stating how important it is for testers to keep up with the latest trends and develop the new skills to keep pace with these trends.  The performance testing fundamentals are the same, but testers need to stay ahead of the curve in order to provide value.

If you have never heard Dan speak, you should.  He has a great passion and enthusiasm for performance testing.  More importantly he likes to share this knowledge with everyone.  If you attend STPCon in Dallas and you love performance testing, introduce yourself to Mr. Dan Downing.

Note on Performance Testing from Scott Barber

At the STP Conference in Nashville I had many daily decisions to make.  After James Bach's inspirational key note, came my first important decision.  There were two speakers both of whom wrote featured articles in the the book, "Beautiful Testing".   One of my co-workers was going to listen to Karen Johnson about "The Strategy Part of the Test Strategy", so I chose to listen to Scott Barber.

Scott in his normal flamboyant style delivered  a passionate presentation on performance testing, "A Performance Testing Life Story: From Conception to Headstone".  I also have a passion for performance testing, but I have a ton of learning to do.  Scott's presentation went right to the heart of the performance testing life cycle.  Here is a summary:

1.  Building a software product it is critical to consider performance within the architecture and design phase.  Performance should be part of the DNA by asking performance specific questions as the software concept evolves.

2.  We should set performance targets then profile and test the code at the component level.

3.  We should continue profiling, performance unit testings, but also add in environmental performance testing and load or stress testing.

4.  There should be a tuning phase where we do our best to optimize performance prior to launch.

5.  We should performance test every patch, update, or configuration change.

6.  Even sun setting applications should be monitored for performance.

Scott summarized his presentation by stating "Successful people and applications are tested, monitored and tuned all of their lives.

Most companies I have worked for performance testing is the last thing considered.  To some extent performance is thought about during the design phase, but not actually tested until the end of the life cycle.

Performance testing is hard!  A non-performance site can hurt a reputation so we should fold performance into the corporate DNA.

Bust out the performance tools and "Git Er Done"!

James Bach at STP Conference in Nashville

I can honestly state that the reason I have not been blogging lately is James Bach.  Just kidding of course, but he certainly inspired me with his keynote "Notes from a Testing Coach".  I have been extremely busy mentoring, learning, implementing, collaborating, testing, and innovating.  It has been extremely fun and I owe the energy to Mr. James Bach.

In his key note he opened by explaining the three kinds of practical credentials: portfolio (your past work), performance (demonstrating your ability), and reputation (stories told about you).  All of these things combine to establish credibility.  Testers should actively work on their portfolio.  Testers should consistently demonstrate their skills.  If you do these two things well hopefully "good" stories will be told about you.

James mentioned one of the things that can get in the way of mentoring are feelings.  He is very accurate in this assessment.  Once you go beyond feelings testers have the ability to leap tall buildings with a single bound.

The coaching process involves building relationships, challenges, allowing things to happen, retrospective or diagnosis of the problem, and collaboration.  There will be set backs as well as celebration of success. 

I recently visited London and I found myself hearing the words "Mind the Gap" in my sleep.  I remember Mr. Bach saying "mind the syllabus".  My interpretation of these words months after the keynote is as a mentor you should have a plan for teaching just has you should have a plan for Session Based testing.  I may have this way out of context at this point, so I will need to do some research.

James also talked about as a mentor you must be prepared to demonstrate to the student what you might do.  In other words, you may get to a point in your mentoring where you have to roll up your sleeves and lead by example.

Another huge lesson from this key note was his demonstration of the hidden picture.  I think the main point was to mess with his brother, but the demonstration illustrated how a tester can explore, change focus, change approach or technique, get reasonable coverage rapidly, yet not find a potentially large defect.

James gave an over view of the dice game.  I had inquired via email to James on how to execute the dice game.  Through email collaboration I got a reasonable idea of the intent of the game, but I was extremely fortunate to be able to learn more about this game in person with his brother Jon.  This hands on experience had a huge impact on me.  The conversation and the approaches Jon took clearly illustrated how testers can benefit by rapidly assessing patterns.  I now try to show this game to every tester I encounter.  I think it is fun and most importantly in invokes thought.  I even went to a local game store and bought the Cast Elk puzzle.  I have yet to solve it!  I keep trying but no success.  I know the answer is on You Tube, but I refuse to cave in.  During my travels I now buy puzzle books and attempt puzzles I never thought I could do.  I am extremely amazed at how fun learning and challenging yourself can be.  Thanks James and Jon for this inspiration.  Jon also turned me on to a site, http://www.sporcle.com/.

This is some of the value I took away from this key note presentation.  There was much more content that I do not recall. I can honestly say that this key note presentation was a true inspiration to me.  Not only as a tester, but in my every day life. 

A huge "Nice Bike" to James for the key note and to Jon for taking the time to experience the dice game with me.

Dice game ROCKS!

Workshop at STP Conference

 I attended a workshop prior to the STP Conference in Nashville called "Creating & Leading a High Performance Test Organization".  Bob Galen was the presenter.

Bob did a great job presenting a ton of material.  Honestly much of the material seemed like common sense, however the information was packaged nicely.  The material served as that gentle kick in the pants to remind you that you that as test leaders we need to revisit our foundations.

Tons of things were covered including recruiting, out sourcing, marketing test, defect management, leadership and communication.  I am going to focus in on two topics.


Effective Communication

Mr. Galen talked about knowing your audience and adopting their point of view.  This makes sense if you have some idea of who the person you are talking to is and what they do.  Even in this happy path situation how you phrase your communication can be extremely complex.  What happens if you meet the person for the very first time?  How do you get their point of view into context?

My conclusion to effective communication is that it is really hard.  We must always work on our communication skills.  Somehow in a conversation you must size up the moment.  In other words put some context around the current environment both setting and mood of the participants.

Bob's second point of "active listening" is probably the key to effective communication.  One approach might be to break the ice with an introducing statement, then carefully listen to the response.  Somehow we must grab the clues that help us to know our audience.  Honestly I tend not to be very good at the active listening part.  Just ask my wife!

One other point Bob had was "can your audience handle the truth and how much of the truth".  This is a tough one for me.  I have an opinion and I tend to share it regardless of the impact on the audience.  As a communicator I need to learn to be more judicious with my opinion and determine how much of the truth is appropriate for audience at hand.

Effective communication is critical in everything we do.   It is not easy and communication is something we can always improve upon.

Defects

At this work shop there was some interesting discussion around defects.  When to document defects and when not to document them.  One person in the audience felt it was critical to document every defect.  Others felt that there are times when it is appropriate not to spend the time documenting a defect.  I shall not continue this debate here, but I thought Bob had a couple of great points in his material.

"A good report is written, numbered, simple, understandable, reproducible, legible and non-judgmental."   I agree with this statement, but there is one attribute that stood out for me, non-judgmental.  As testers we need to do our best to remove emotion from a defect report.  We should be concise, state the discovery process, and add supporting material as facts.

Bob also provided a list of styles testers should consider putting together a defect report.

1. Condense–Say it clearly but briefly

2. Accurate–Is it truly a defect?

3. Neutralize–Just the facts

4. Precise–Explicitly, what is the problem?

5. Isolate–What has been done to isolate the problem?

6. Generalize–How general is the problem?

7. Re-create–essential environment, steps, conditions

8. Impact –To the customer, to testing, safety?

9. Debug–Debugging materials (logs, traces, dumps,

environment, etc.)

10.Evidence–Other documentation proving existence

I felt like this was a pretty good reminder of how to write an effective defect report.  Perhaps I can develop this into a little acronym - CAN-PIG-RIDE. 

I need to do a better job of effective communication and part of that effort is making sure defect reports are communicated effectively.

Happy Testing!

Defect Tracking

I still plan to post thoughts on the STP Conference in Nashville, but some recent posts has spawned some thought.

Lisa Crispin gave a presentation at Star East that sparked Gojko to write a post entitled "Bug Statistics are a Waste of Time".

I agree with the notion that we should clearly understand the business objectives and find ways to measure the value the features bring to a customer community.  I do not agree that looking at bug statistics is a waste of time.  History is one of the greatest oracles available to a tester.  How were we doing in the past compared to today?  Are there any lessons to be learned from a software systems past defects?  I certainly think so.

Lets assume that company X provides some value to someone and that company X knows how to measure their business objectives and value of those objectives using things like Net Promoter Score, Google Analytics, Agile Velocity,  and Get Satisfaction.

What can inspecting defect metrics add to the cause of determining value?

I view metrics as flashlights into a dark cave.  How do you know what is there unless you look?

A simple inspection of the total number of defects in the back log implies some level of technical debt.  I agree with Gojko that if defects simply sit in a backlog then we are wasting some time.  Teams should and must proactively triage, fix, or even throw away defects.  But not to document them especially in a non-searchable manner would be detrimental to the team.

Teams should occasionally have retrospectives on their processes.  Finding data regarding defect groupings is a fantastic lever for continuous improvement.  Where are the majority of our bugs historically clustering.  Where you find defect clusters you have the opportunity to change your process to reduce those clusters.  Agile teams especially should do look backs at some frequency.  How did we do last quarter?  How does trending look?

As a tester I have at times had an extremely difficult time advocating for a process change.  In several situations I have found the ammunition to influence change by showing historical trends.  Could we do this carrying around a notebook? Sure we could, but it would be difficult especially as to how time flies.

Let me toss this scenario into the mix.  You are a new tester at a large company or even a consultant.  Your mission is to understand the quality of the software and you have to do it fast.  It would be nice to shine a flash light into the new cave and know the areas of risk.  Yes you could put your hands on the application and start testing, but a sneak peak at the historical defect data could narrow in on the best place to start.

Yes some defect tracking tools really really suck, but our ability as testers to search, learn and educate provides great value even to Company X who knows how to monitor and measure value.

I shared Gojko's link with a large community of developers and testers.  I shared the link not because I agreed with it, but because it made me think.  I received back a quote that really struck a chord with me.

"Sure hope this isn’t the future of QA."

Losing the oracle of history would be a huge mistake.  Using metrics prudently, adapting the metrics to changing business value, and having conversations around the findings are key elements of Continuous Improvement.

If we influence a change for the better using metrics, then we certainly are not wasting our time! 

Read Gojko's post and the associated comments it definitely should spark some thought.

Happy Testing! 

Funny thing Happened at STP in Nashville

Testers, I must find the time to catch up on my back log of posts.  I thought I would start out with an embarrassing tale.

Once upon a time there was a tester, Carl, who had signed up for a work shop at the STP conference in March 2011 at Nashville TN.  It was a beautiful day and the coffee was flowing.  Carl happened to be running late for the workshop.  All Carl knew was that the work shop was labeled Pre-5.

Carl raced up the stairs (very hard to do with a tender knee).  There at the top of the stairs on the right was Pre-5.  The instructors name was Bob.  Carl remembered his instructor's name was Bob.  The title of the work shop was Quality Monitoring and Coaching.  For some reason Carl seemed out of place. 

Everyone introduced themselves.  Of course I introduced myself and why I was there.  What did I say?  "Hi I am Carl Shaulis and I am a Test Manager at HomeAway.  I recently have been working close with Customer Service to facilitate quality and I am here to continue learning on how to bridge the gaps between our two teams."  Wait one second, that does not sound right.  Carl pulls out his conference brochure.  As he thumbs through the pages Fiona walks into the conference room.  Oh Carl knows Fiona from STP Las Vegas.  Carl must be in the right place.  Carl finally finds the work shop he should be in called "Creating and Leading a High Performance Test Organization" by Bob Galen.

Panic sets in!  I am in the wrong room.  I just gave a great introduction on how this presentation could benefit me.  What should Carl do?

Carl quickly packed up his gear and headed out of the room.  Carl found his intended work shop and enjoyed Spring STP Conference 2011.

Carl did get some friendly ribbing and crap from Abbie and Fiona.  Deservedly so!  The secret was out of the bag that there were two concurrent conferences  Contact Center Conference Expo 2011 and Software Test Professionals.

The really fun part is that Carl probably would have learned a hell of a lot if he had stayed in the incorrect work shop.  However, interacting with the testers and learning from Bob Galen was the right move.

Happy Testing!

Defect Severity

I now have a huge backlog of blog posts, but here is one that has me losing some sleep.

Why do developers seem to get their feathers ruffled when we talk about defect severity?  I am not implying that developers are ducks, chickens, or geese.  I could have used the phrase why do developers get their panties in a wad when we talk about defect severity, but that did not seem PC.

First I will give a little back ground.  I was asked to inspect and access team quality.  Hmm, that is a monumental task.  Where shall I start?  Lets take a look at how many severity one defects are in production.  If there are any severity one defects in production, then the quality status is RED.  If there are no severity one defects in production then the quality status is GREEN.  Seemed like a simple assessment, which would require some follow up conversations if the values were red.  By no means do I mean to imply that this is the only indicator of quality, but it was the first flash light used to look into the cave.

But wait we (development) do not care about severity we focus all of our efforts on Priority.  This has been echoed by at least three development leads.

My two brain cells are telling me that severity is the perception of how bad a defect is and priority is the process around getting a defect resolved.  Hmm, do I have this all wrong?  Is priority the only thing of importance.  I read Perfect Software by Jerry Weinberg a few months ago.  Did I not understand his writing?  Well I guess I had better do whatever any internet geek would do, "Google It"!

I found a great post by my friend and colleague Dr. Stan Taylor.  I did not see to much in that post that I could disagree with.  Next I stumbled on a Bug Advocacy video by Professor Cem Kaner.  I did not find a definitive answer there.  I read at least 10 other blog posts and all seemed to have a slightly different spin, but there did seem to be some consistency that severity is the perception of the user and priority is the decision process on when to fix a defect.

I am still perplexed as to why inspecting severity gets such a negative response by development.  Let me attempt to noodle out a couple of basic definitions.  Please note that I did not come up with these definitions, but I tend to agree with these.

What is Quality? -  “Quality is value to some person (that matters)”

What is a bug? – Something that bothers (bugs) someone who matters.

Severity - defines the impact the defect has on the customer and the likelihood of occurrence. 

S1 – Product is on fire.  We are getting sued.  We cannot take money

S2 – Breaking but we can offer a workaround

S3 – Minor functional defect (pain in the butt or poor user experience)

S4 – Cosmetic (we can live with it)

Priority – determines the order in which defects will be fixed/resolved and retested

            What do we fix?

            When do we fix it?

Priority decisions are based on impact on the customer (severity) and resolution difficulty (time & resources)

P1 – Resolve Immediately

P2 – High Attention

P3 – Normal

P4 – low

Those seem reasonable to me and some of the definition came from a Power Point presentation by Dave Whalen called "The Matrix". 

So now I am thinking that priority cannot be established without taking into consideration the severity.

I am still puzzled as to why developers do not want to track defects based on severity.  Somebody perceived the defect as having a negative impact on someone.

I am concluding that I need to have more conversations with developers!  Here is a reasonable post provided by one of the development leads.  One point made in our conversation was that when I used the phrase perception of the customer it inserts emotion into the concern.  The developer preferred impact on the customer. 

I can certainly understand with the pressures of Rapid Software development of today there is little time for fixing defects. So the balancing act of meeting milestones and building great features is extremely priority centric.

However persons documenting the defect have no input with respect to priority at the time of discovery, so they must focus on setting a proper severity.  A well-documented defect can be leveraged as a sales tool to influence priority and the person submitting the defect must be an advocate of the defect in order to see it through to resolution.

When severity and priority do not align, I would suggest it is extremely important to communicate the factors used to set the priority to the person who established the severity.

Hmm, have developers been burned in the past by incorrect values of severity being placed on the defects? Could that be the reason for the ruffled feathers?  I think this might be a good possibility.

Perhaps feathers are not ruffled at all, because based on all of the blog posts the ambiguity of Severity and Priority have been around for a long long time.

Time to read "Perfect Software" again!

Goodwill for Automation Code

I had an interesting dream, thought, or inspiration last night.  It could have been codeine induced based on the horrible cough I have been battling, but no the less it was an important thought.

What happens to legacy code?  Does it simply get deleted and gone eternally?  Does it fall into an abyss?

No!  We take it to Goodwill for Automation Code.

It occurred to me that one of the key takeaways from STP conference was mentoring.  The "Ah Ha moment" was that we could leverage legacy code to educate others.  We have some Software Engineers in test that are struggling with TDD, test driven development.  So my concept is to look at our code repository and find code that has matured over time.  We can then share this evolution of code with the testers that struggle with TDD.

I also struggled a bit with TDD, until I saw the light.  Even for test automation it is important to unit test your code.  How did I learn?  What turned on the light?

Mentoring by my colleague certainly was key. The knowledge of knowing my new methods would potentially be used by many tests forced me to write better code, that could easily and repeatably be tested.  I still struggle because I simply want to get a test implemented quickly, so I admit sometimes I write my unit tests after the fact.  One rule of thumb I do have is that if I struggle at all to write a piece of code, I immediately stop and write the unit tests.  Breaking the complexity into simpler chunks helps me to get to a more robust solution.

So I am going to go into the office today, then search for learning examples of code.  These legacy jewels will then be moved into Goodwill for future education opportunities.

Cast off code will be used to assist others.  Amazing the thoughts that come to you during sleep and I do not recommend the codeine!

Keep on testing!

Complexity

I know I have not posted in quite some time.  Being busy is not a good excuse!

I do have numerous random ramblings floating through my brain with respect to testing.  But my key thought for the day is that testing web applications can be extremely complex.  Besides complexity the simplest deformity can cause the most chaos.

Are web applications similar to macromolecular complex assembly?  Macromolecular complex assembly is the aggregation, arrangement and bonding together of a set of components to form a complex.

I guess I would consider the combining of a bunch of disparate technology stacks an aggregation.  The integration of these technologies certainly must be arranged and bonded.  I believe today that tiny aspects of this complex are under the scrutiny of a microscope.

Someone of importance notices that an implementation in a specific browser causes a line to wrap.  This is a tiny deformity on the surface of a cell causes concern, yet the neighboring system which is about to go into complete cardiac arrest goes unnoticed.

Why is it that the tiniest blemish on a complex system gets noticed first?  Why is it that the most insignificant concern causes the most chaos for a tester?

Unfortunately it becomes difficult for testers to inspect the entire surface of a cell, when the internal organs (mitochondria) are in need of some much attention.  As we inspect and flush out the issues with the key features we get the most demerits for a cosmetic blemish on the user interface.

So I guess the resolution is to perform CPR on the failing feature while fixing the blemish.  The complex will endure!

Testing for the Super Bowl Ad

I am sitting at HomeAway's Super Bowl Command Center patiently waiting for the game to begin.  I have not posted in awhile due to an enormous amount of testing.

We have been conducting performance tests for almost 4 weeks.  Usually on Sunday we could do our most impacting performance tests.  We ran tests every Tuesday, Thursday and Sunday.  Is our site ready?  We should know in about 5 hours.  Thanks to tools like jMeter and BrowserMob I believe we are ready!

Not only was I involved with a great team for performance testing, I also had to test some new applications developed specifically for the Super Bowl.  This testing effort was exhausting.  The final build was delivered at 1:00 PM Saturday.  Yep, that is right!  Just 28 hours before the Super Bowl.

I am confident in the software, but there are some known issues.  Very difficult to test across the variety of OS and browser configurations.  Throw in different versions of flash and media players you have an infinite amount of testing.  If I had not read Gerald Weinberg's book "Perfect Software", I think I would be freaking out right about now.

So what would be the lessons learned over the past four weeks.

• You must performance test with a great group of technical people, because collaboration is the key.
• You must insist on testing third party applications early and often.
• You must learn to let go, because unfortunately with a small time line and infinite number of test cases, you can only do so much.
• You must have good performance testing tools and a cloud vendor like Browser Mob who will go the extra mile.
• You must have a dedicated team who are willing to put in long hours.
• You must have testers who are willing to report all of their observations.
• You must not use a Google Shared document to track and collaborate on issues.  Use a web based defect tracking tool. (Primary mistake I made for Super Bowl)
• You must think about process as soon as you engage a third party vendor
• You must have network monitoring in place.

I am sure I can think of more things later, but it is now time to enjoy the game and festivities.

Why stay in a hotel when you can HOME AWAY!

Vote for HomeAway's Super Bowl commercial!

Presentation sponsored by Thoughtworks and Agile Austin

I am shifting gears a bit here, but I had the absolute pleasure of listening to Jim Highsmith speak here in Austin Texas.  Last Wednesday Thoughtworks and Agile Austin sponsored a talk by Mr. Highsmith on the topic of Agile Leadership.

I could never articulate all of the great content presented by Mr. Highsmith, so I will toss out some key concepts I managed to snag.  I am hoping Thoughtworks will distribute the slide deck so I can remember even more.

Focusing on Organizational Agility I recall one fundamental distinction.  Is your business focus based on Responsiveness or Efficiency?  If it is Responsiveness then it is probably important to learn Agile Management skills.

The core tenants according to Mr. Highsmith were Do less, Quality, Speed to Value, and Engage & Inspire. 

Do less is definitely applicable to my recent experiences with kanban. 

Of course Quality is important to me.  Quality really does matter to management and it is a proven fact that reducing defects reduces cost.

I am not so enthusiastic about increasing speed, but speed to value is a key to success.   We really do not want to be focusing on things that add little value.  In my opinion, we need to find a sustainable pace.

Now Engage and Inspire is a prime theme for me these days.  After attending the STP Conference in the fall of 2010, I certainly became inspired.  I am seeking the opportunity to engage with other testers and be inspired.  Hopefully I can inspire too.

Mr. Highsmith recommended several books.  You can find those recommendations on his blog.  I recall two suggestions: Drive by Dan Pink and Organizing Genius by Warren Bennis.

Software companies should have a focus on value and delivering features.  Wow!  That certainly makes sense to me. An interesting point was that companies should not permit product owners to do all of the prioritization.  If you let product determine the priority, then there is no room to work on technical debt, defects, or useful tools.  His point was many companies suffer from technical debt.

The talk inspired me to do my best to participate more in the Agile community in Austin.  I was speaking with Matt, the president of Agile Austin and I found out there is a group of testers that meet monthly over lunch.   I definitely want to get plugged in to that group.

My final point is if you get a chance to go hear these great speakers, please do so!  There are bits of goodness for everyone to take home.

Happy Testing!

Build Blocked at the end of Week 1

The team implemented their new process and this was the first week in action.  They started off with a large bottleneck in the acceptance test column.  The team quickly moved through the bottleneck.

As an outside observer the testers seemed to be happier.  They were actively involved at the start of the process rather than at the end.  Developers were completing acceptance test quickly and what appeared to be efficiently.  Blocked stories were obviously labeled on the board.  There definitely was a buzz in the air.

I did over hear a couple of developers wondering why the hell are we doing this, "Seems like we are reinventing the wheel!"  There were a couple of developers who were blocked.  So there was more time for foosball.

The teams goal was to have a branched release candidate by the close of business on Thursday.  Unfortunately at the close of business on Friday testers had found enough defects that a clean build did not exist.  The release is due out Tuesday.

It will be very interesting to see how things go on Monday.  Will the testers be able to adjust their regression testing pattern in order to get out a solid release.

As an outside observer I really enjoy collaborating with the test and development leads.  I shared two observations with respect to their kanban board.  Looking at the board there were numerous stories that were marked as blocked.  To me it was not clear from the board who owned the action item to unblock the story.  The other suggestion that I made was that they also need some sort of issue board or list, so everyone on the team knows what things are impeding the process flow.

The new 9 foot magnetic board is in.  The team has created some magnetic buttons with their personal avatars.  Lets just say some of the avatars are quite interesting, but I think it adds a bit of fun to the process.  They also created the yuck face buttons, which they can apply to blocked stories.  I am looking forward to seeing this board in action.

Oh one more very positive observation was that the team was able to move a few more defects through the process than were original scheduled.  I liked the buzz of the new process and testers know what to test.

Several of the team members were able to collaborate over an 18 page Power Point document and condense the material down into a handful of cukes.  Cukes are executable requirements from the open source tool, Cucumber.  Soon we will implement the strategy to automate the cukes.

Laissez les bons temps rouler!

Diving into the Abyss

We had a couple more brain storming sessions on moving a team onto the kanban train.

The biggest concern during these two sessions focused on our build process and software control systems.  We currently use Subversion.  Much of the discussion was around the complexities of checking in and backing out code.  How can our cadence flow in order to give testers builds of high quality and builds of trust.  I believe the final solution discussed was that we need to move from Subversion to GIT.

The development team believes GIT is a more friendly change control system for moving software around.  We shall see, but we do not have GIT today and we are about to jump into the abyss.

From my perspective, there are two potential issues when the team arrives for work on Tuesday.  One concern is that not all of the stakeholders were present during our brainstorming sessions.  I believe some team members will be surprised on Tuesday, but most will jump into the kanban abyss with both feet.  The other concern is that the WIP is already way over the estimated capacity.

Despite the WIP we decided to make the leap.  I stopped by the office on Friday and the Development Lead had a carefully crafted kanban board drawn across a couple of white boards.  He has a 9 foot magnetized board on order, but it has not arrived.  "See the team is already adjusting."  The large portion of the WIP is currently in the completed state and ready for acceptance testing.  The entire team (developers and testers) is going to start with this phase, write acceptance tests when they are not available, actuate the code, and test the pending features.  Viewing the kanban board I also noticed several key stories and defects have been placed in the ready state.

I am looking forward as an observer/consultant to see the giant green start button get pushed on Tuesday and the new process will begin to flow.  During our brain storming sessions to other key concepts were often mentioned.  During daily stand up meetings we have the ability to make minor adjustments.  Communication will be important.  The second concept was the big RED button.  The big RED button is "stop the presses" button.  I am hoping that for the first one week release cycle the red button is not used.

Unfortunately there is a high volume of risk with this release as the team jumps into the abyss.  If the team does a good job conducting thorough code reviews, acceptance tests and the test team stays on top of their game, then we may have the parachutes to float gently to the bottom of the abyss.  If we happen to land hard then we have the retrospective and make the first adjustments to the process.

The team is leaping and I believe 2011 is going to be a great year!

Kanban - Part II

Yesterday we had a brain storming session on what kanban would look like for the team.  I must say it generated a ton of good conversation.  Here are a couple of key points that I remember.

Someone mentioned that a team of testers could spend a majority of their time in the design phase, especially facilitating the development of Cucumber scripts.  It was suggested that one tester each week focus on this area and rotate the responsibility each week.  Another suggestion was to pair a tester and developer for each story.  The primary premise discussed was that if effective Cucumber scripts are developed then further down the process testers would know what to test even though they did not participate in the design phase.

Another key point was with respect to the current defect backlog.  How do we keep the defect backlog from building?  The team concluded that we need a dedicated swim lane in the process dedicated to fixing defects.  Dan Anderson shows in his book a good example where X% of the capacity is dedicated to defect resolution. 

What happens if a story is full of defects?  The conclusion was that the story remains in the process and is not released.  Since the team is inspecting the process daily, they can make adjustments to provide potential solutions.  One approach might be paired-development practices which provides real time training and retrospectives on what is happening with the story.  The ultimate conclusion is that if the team only got one story deployed each week they were adding incremental value to the business.  This should not happen often, but the bottom line is that it is OK to tweak the process on a daily basis and focus on deploying high quality valued features.

The one scary piece that we did not resolve was dealing with complexity.  Occasionally an Epic story comes into the team's backlog.  During the design phase we should be able to dissect and break the Epic into smaller stories, but the concern became how do we make sure the stories get implemented in the right order and that all of the stories that comprise the whole get assembled, tested, and deployed.  Adjusting for complexity will be a fun challenge, but a challenge never the less.

We talked about buffer zones and capacity limits.  We have no clue where to set the capacity limits.  It was suggested that we minimize the WIP so that we can gain confidence in the process and trust in the quality of the features being produced.

I must admit I am excited about this approach.  I know we will fall flat on our face a couple of times, but we will get up, collaborate, and continuous improve the process.

Stay tuned for Part 3.  We plan to walk through some real scenarios.

Kanban

I am wide awake Christmas Eve day at 5:20 AM.  I should be sleeping!  Instead I am reading a book by David Anderson, Kanban.  Why as a tester am I reading Kanban?  Well one of the teams I work closely with is about to implement kanban in 2011 as an attempt to improve their process.  So I ponder how does test have to change on order to fit into this paradigm.

I am not even going to attempt to describe kanban in this post, but I do want to convey I get a bit excited when I see the first step David Anderson recommends for moving to kanban.  He says the first step is to "Focus on Quality".  As a tester you have to like hearing those words.

I also agree with these points.

• Both Agile and traditional development approaches to quality have merit.
• Code inspections if done right can improve quality.
• Collaborative analysis and design improve quality. I believe testers should get involved early in the process!
• Using design patterns improve quality.
• Use of modern development tools improve quality.

As pointed out in the book all of these things improve trust and confidence in the process.

The team using kanban will be doing weekly releases.  They have been doing weekly releases, but development and test cycles were taking four weeks (2 in development and 2 in test).  The product owners were not getting their new features in a timely manner.

In December the team has been focusing on reducing the backlog of defects and discussing the implications of kanban.  In my opinion, the biggest fear of this change is coming from the testers.

I contend there is nothing to fear.  A story will move through this flow:

Idea > Design > Work in Progress (code, unit test, code review, automation) > acceptance test > final test > production

If there are 10 stories in the pipeline and 5 testers.  Then a tester is responsible for 2 stories.  Testers should be able to test 2 stories in a week.  Right?  OK it does depend on the size of the story.  The fear comes from the regression test phase.  My philosophy is automate regression tests where possible and focus on Exploratory Testing.  If testers cannot get over their fear perhaps any manual regression testing should be focused on the area(s) of highest risk.

I predict that the process will be bottle-necked in two areas.  The first bottleneck will be in design.  As part of design the team will be writing Cucumber tests.   Through collaboration the cucumber tests will make sure everyone is understanding the feature and testers will know what to test early in the process.  This process will take time.

I believe acceptance testing will go fast, but the second bottleneck will be the final testing phase.  Unfortunately testers are still focused on "Perfect Software".  I contend that the testers will have to implement test strategies that can be completed in 1-2 days.

I am looking forward to seeing this in action.

The extremely nice thing about kanban is that you know your pain points because you see them on the flow chart.  More importantly kanban expects you to change the process in order to remove the pain points.

Over simplification?  Maybe!  As Larry the Cable Guy might say "Git R done"!

I am definitely looking forward to this challenge in 2011.  I would welcome any comments from testers who are actively doing kanban.  I suspect there are not many.

Waterfall vs Agile

This past week Lanette Creamer posted on her great blog some statements with respect to her current team.  She implies that Waterfall teams are better at communication than Agile teams.

I would contend that the methodology has little to do with it.  You take a motivated and well spoken tester like Lanette, then combine other motivated members to her team and you will get good communication.

Motivated people make communication happen.  I will go one step further and say good communication makes for successful projects.

I have worked on waterfall, v-model, and agile projects.  If given the choice I will choose agile.

With waterfall and v-model I always found the massive amounts of documentation, document review, and the cumbersome process of reviewing changes to the documentation to be tedious.  In my opinion at times the process hindered communication.

With agile, lean, or Kanban we are afforded the opportunity to learn from our mistakes and quickly change the process.

But as to Lanette's point, agile projects can only be successful with excellent team communication.

So everyone who would like their projects to be successful sharpen your communication skills.

Books I am currently reading

All testers MUST read Perfect Software and Other Illusions about Testing by Gerald Weinberg.  I just finished reading this book and I found the content extremely useful and inspiring.  If you test software, read this book.

Now I am reading Lessons Learned in Software Testing by Cem Kaner, James Bach, and Brett Pettichord.  I am only a little ways into this book, but I am finding it to be spot on.

Next in the queue is Kanban - Successful Evolutionary Change for Your Technology Business by David J. Anderson.

If you are reading a great testing or development book, please make a comment so I can get more books into my queue.

Read on!