At the STP Conference in Nashville I had many daily decisions to make. After James Bach's inspirational key note, came my first important decision. There were two speakers both of whom wrote featured articles in the the book, "Beautiful Testing". One of my co-workers was going to listen to Karen Johnson about "The Strategy Part of the Test Strategy", so I chose to listen to Scott Barber.
Scott in his normal flamboyant style delivered a passionate presentation on performance testing, "A Performance Testing Life Story: From Conception to Headstone". I also have a passion for performance testing, but I have a ton of learning to do. Scott's presentation went right to the heart of the performance testing life cycle. Here is a summary:
1. Building a software product it is critical to consider performance within the architecture and design phase. Performance should be part of the DNA by asking performance specific questions as the software concept evolves.
2. We should set performance targets then profile and test the code at the component level.
3. We should continue profiling, performance unit testings, but also add in environmental performance testing and load or stress testing.
4. There should be a tuning phase where we do our best to optimize performance prior to launch.
5. We should performance test every patch, update, or configuration change.
6. Even sun setting applications should be monitored for performance.
Scott summarized his presentation by stating "Successful people and applications are tested, monitored and tuned all of their lives.
Most companies I have worked for performance testing is the last thing considered. To some extent performance is thought about during the design phase, but not actually tested until the end of the life cycle.
Performance testing is hard! A non-performance site can hurt a reputation so we should fold performance into the corporate DNA.
Bust out the performance tools and "Git Er Done"!
Sunday, July 24, 2011
Wednesday, July 20, 2011
James Bach at STP Conference in Nashville
I can honestly state that the reason I have not been blogging lately is James Bach. Just kidding of course, but he certainly inspired me with his keynote "Notes from a Testing Coach". I have been extremely busy mentoring, learning, implementing, collaborating, testing, and innovating. It has been extremely fun and I owe the energy to Mr. James Bach.
In his key note he opened by explaining the three kinds of practical credentials: portfolio (your past work), performance (demonstrating your ability), and reputation (stories told about you). All of these things combine to establish credibility. Testers should actively work on their portfolio. Testers should consistently demonstrate their skills. If you do these two things well hopefully "good" stories will be told about you.
James mentioned one of the things that can get in the way of mentoring are feelings. He is very accurate in this assessment. Once you go beyond feelings testers have the ability to leap tall buildings with a single bound.
The coaching process involves building relationships, challenges, allowing things to happen, retrospective or diagnosis of the problem, and collaboration. There will be set backs as well as celebration of success.
I recently visited London and I found myself hearing the words "Mind the Gap" in my sleep. I remember Mr. Bach saying "mind the syllabus". My interpretation of these words months after the keynote is as a mentor you should have a plan for teaching just has you should have a plan for Session Based testing. I may have this way out of context at this point, so I will need to do some research.
James also talked about as a mentor you must be prepared to demonstrate to the student what you might do. In other words, you may get to a point in your mentoring where you have to roll up your sleeves and lead by example.
Another huge lesson from this key note was his demonstration of the hidden picture. I think the main point was to mess with his brother, but the demonstration illustrated how a tester can explore, change focus, change approach or technique, get reasonable coverage rapidly, yet not find a potentially large defect.
James gave an over view of the dice game. I had inquired via email to James on how to execute the dice game. Through email collaboration I got a reasonable idea of the intent of the game, but I was extremely fortunate to be able to learn more about this game in person with his brother Jon. This hands on experience had a huge impact on me. The conversation and the approaches Jon took clearly illustrated how testers can benefit by rapidly assessing patterns. I now try to show this game to every tester I encounter. I think it is fun and most importantly in invokes thought. I even went to a local game store and bought the Cast Elk puzzle. I have yet to solve it! I keep trying but no success. I know the answer is on You Tube, but I refuse to cave in. During my travels I now buy puzzle books and attempt puzzles I never thought I could do. I am extremely amazed at how fun learning and challenging yourself can be. Thanks James and Jon for this inspiration. Jon also turned me on to a site, http://www.sporcle.com/.
This is some of the value I took away from this key note presentation. There was much more content that I do not recall. I can honestly say that this key note presentation was a true inspiration to me. Not only as a tester, but in my every day life.
A huge "Nice Bike" to James for the key note and to Jon for taking the time to experience the dice game with me.
Dice game ROCKS!
In his key note he opened by explaining the three kinds of practical credentials: portfolio (your past work), performance (demonstrating your ability), and reputation (stories told about you). All of these things combine to establish credibility. Testers should actively work on their portfolio. Testers should consistently demonstrate their skills. If you do these two things well hopefully "good" stories will be told about you.
James mentioned one of the things that can get in the way of mentoring are feelings. He is very accurate in this assessment. Once you go beyond feelings testers have the ability to leap tall buildings with a single bound.
The coaching process involves building relationships, challenges, allowing things to happen, retrospective or diagnosis of the problem, and collaboration. There will be set backs as well as celebration of success.
I recently visited London and I found myself hearing the words "Mind the Gap" in my sleep. I remember Mr. Bach saying "mind the syllabus". My interpretation of these words months after the keynote is as a mentor you should have a plan for teaching just has you should have a plan for Session Based testing. I may have this way out of context at this point, so I will need to do some research.
James also talked about as a mentor you must be prepared to demonstrate to the student what you might do. In other words, you may get to a point in your mentoring where you have to roll up your sleeves and lead by example.
Another huge lesson from this key note was his demonstration of the hidden picture. I think the main point was to mess with his brother, but the demonstration illustrated how a tester can explore, change focus, change approach or technique, get reasonable coverage rapidly, yet not find a potentially large defect.
James gave an over view of the dice game. I had inquired via email to James on how to execute the dice game. Through email collaboration I got a reasonable idea of the intent of the game, but I was extremely fortunate to be able to learn more about this game in person with his brother Jon. This hands on experience had a huge impact on me. The conversation and the approaches Jon took clearly illustrated how testers can benefit by rapidly assessing patterns. I now try to show this game to every tester I encounter. I think it is fun and most importantly in invokes thought. I even went to a local game store and bought the Cast Elk puzzle. I have yet to solve it! I keep trying but no success. I know the answer is on You Tube, but I refuse to cave in. During my travels I now buy puzzle books and attempt puzzles I never thought I could do. I am extremely amazed at how fun learning and challenging yourself can be. Thanks James and Jon for this inspiration. Jon also turned me on to a site, http://www.sporcle.com/.
This is some of the value I took away from this key note presentation. There was much more content that I do not recall. I can honestly say that this key note presentation was a true inspiration to me. Not only as a tester, but in my every day life.
A huge "Nice Bike" to James for the key note and to Jon for taking the time to experience the dice game with me.
Dice game ROCKS!
Monday, May 30, 2011
Workshop at STP Conference
I attended a workshop prior to the STP Conference in Nashville called "Creating & Leading a High Performance Test Organization". Bob Galen was the presenter.
Bob did a great job presenting a ton of material. Honestly much of the material seemed like common sense, however the information was packaged nicely. The material served as that gentle kick in the pants to remind you that you that as test leaders we need to revisit our foundations.
Tons of things were covered including recruiting, out sourcing, marketing test, defect management, leadership and communication. I am going to focus in on two topics.
Effective Communication
Mr. Galen talked about knowing your audience and adopting their point of view. This makes sense if you have some idea of who the person you are talking to is and what they do. Even in this happy path situation how you phrase your communication can be extremely complex. What happens if you meet the person for the very first time? How do you get their point of view into context?
My conclusion to effective communication is that it is really hard. We must always work on our communication skills. Somehow in a conversation you must size up the moment. In other words put some context around the current environment both setting and mood of the participants.
Bob's second point of "active listening" is probably the key to effective communication. One approach might be to break the ice with an introducing statement, then carefully listen to the response. Somehow we must grab the clues that help us to know our audience. Honestly I tend not to be very good at the active listening part. Just ask my wife!
One other point Bob had was "can your audience handle the truth and how much of the truth". This is a tough one for me. I have an opinion and I tend to share it regardless of the impact on the audience. As a communicator I need to learn to be more judicious with my opinion and determine how much of the truth is appropriate for audience at hand.
Effective communication is critical in everything we do. It is not easy and communication is something we can always improve upon.
Defects
At this work shop there was some interesting discussion around defects. When to document defects and when not to document them. One person in the audience felt it was critical to document every defect. Others felt that there are times when it is appropriate not to spend the time documenting a defect. I shall not continue this debate here, but I thought Bob had a couple of great points in his material.
Bob did a great job presenting a ton of material. Honestly much of the material seemed like common sense, however the information was packaged nicely. The material served as that gentle kick in the pants to remind you that you that as test leaders we need to revisit our foundations.
Tons of things were covered including recruiting, out sourcing, marketing test, defect management, leadership and communication. I am going to focus in on two topics.
Effective Communication
Mr. Galen talked about knowing your audience and adopting their point of view. This makes sense if you have some idea of who the person you are talking to is and what they do. Even in this happy path situation how you phrase your communication can be extremely complex. What happens if you meet the person for the very first time? How do you get their point of view into context?
My conclusion to effective communication is that it is really hard. We must always work on our communication skills. Somehow in a conversation you must size up the moment. In other words put some context around the current environment both setting and mood of the participants.
Bob's second point of "active listening" is probably the key to effective communication. One approach might be to break the ice with an introducing statement, then carefully listen to the response. Somehow we must grab the clues that help us to know our audience. Honestly I tend not to be very good at the active listening part. Just ask my wife!
One other point Bob had was "can your audience handle the truth and how much of the truth". This is a tough one for me. I have an opinion and I tend to share it regardless of the impact on the audience. As a communicator I need to learn to be more judicious with my opinion and determine how much of the truth is appropriate for audience at hand.
Effective communication is critical in everything we do. It is not easy and communication is something we can always improve upon.
Defects
At this work shop there was some interesting discussion around defects. When to document defects and when not to document them. One person in the audience felt it was critical to document every defect. Others felt that there are times when it is appropriate not to spend the time documenting a defect. I shall not continue this debate here, but I thought Bob had a couple of great points in his material.
"A good report is written, numbered, simple, understandable, reproducible, legible and non-judgmental." I agree with this statement, but there is one attribute that stood out for me, non-judgmental. As testers we need to do our best to remove emotion from a defect report. We should be concise, state the discovery process, and add supporting material as facts.
Bob also provided a list of styles testers should consider putting together a defect report.
1. Condense–Say it clearly but briefly
2. Accurate–Is it truly a defect?
3. Neutralize–Just the facts
4. Precise–Explicitly, what is the problem?
5. Isolate–What has been done to isolate the problem?
6. Generalize–How general is the problem?
7. Re-create–essential environment, steps, conditions
8. Impact –To the customer, to testing, safety?
9. Debug–Debugging materials (logs, traces, dumps,
environment, etc.)
10.Evidence–Other documentation proving existence
I felt like this was a pretty good reminder of how to write an effective defect report. Perhaps I can develop this into a little acronym - CAN-PIG-RIDE.
I need to do a better job of effective communication and part of that effort is making sure defect reports are communicated effectively.
Happy Testing!
Friday, May 20, 2011
Defect Tracking
I still plan to post thoughts on the STP Conference in Nashville, but some recent posts has spawned some thought.
Lisa Crispin gave a presentation at Star East that sparked Gojko to write a post entitled "Bug Statistics are a Waste of Time".
I agree with the notion that we should clearly understand the business objectives and find ways to measure the value the features bring to a customer community. I do not agree that looking at bug statistics is a waste of time. History is one of the greatest oracles available to a tester. How were we doing in the past compared to today? Are there any lessons to be learned from a software systems past defects? I certainly think so.
Lets assume that company X provides some value to someone and that company X knows how to measure their business objectives and value of those objectives using things like Net Promoter Score, Google Analytics, Agile Velocity, and Get Satisfaction.
What can inspecting defect metrics add to the cause of determining value?
I view metrics as flashlights into a dark cave. How do you know what is there unless you look?
A simple inspection of the total number of defects in the back log implies some level of technical debt. I agree with Gojko that if defects simply sit in a backlog then we are wasting some time. Teams should and must proactively triage, fix, or even throw away defects. But not to document them especially in a non-searchable manner would be detrimental to the team.
Teams should occasionally have retrospectives on their processes. Finding data regarding defect groupings is a fantastic lever for continuous improvement. Where are the majority of our bugs historically clustering. Where you find defect clusters you have the opportunity to change your process to reduce those clusters. Agile teams especially should do look backs at some frequency. How did we do last quarter? How does trending look?
As a tester I have at times had an extremely difficult time advocating for a process change. In several situations I have found the ammunition to influence change by showing historical trends. Could we do this carrying around a notebook? Sure we could, but it would be difficult especially as to how time flies.
Let me toss this scenario into the mix. You are a new tester at a large company or even a consultant. Your mission is to understand the quality of the software and you have to do it fast. It would be nice to shine a flash light into the new cave and know the areas of risk. Yes you could put your hands on the application and start testing, but a sneak peak at the historical defect data could narrow in on the best place to start.
Yes some defect tracking tools really really suck, but our ability as testers to search, learn and educate provides great value even to Company X who knows how to monitor and measure value.
I shared Gojko's link with a large community of developers and testers. I shared the link not because I agreed with it, but because it made me think. I received back a quote that really struck a chord with me.
"Sure hope this isn’t the future of QA."
Losing the oracle of history would be a huge mistake. Using metrics prudently, adapting the metrics to changing business value, and having conversations around the findings are key elements of Continuous Improvement.
If we influence a change for the better using metrics, then we certainly are not wasting our time!
Read Gojko's post and the associated comments it definitely should spark some thought.
Happy Testing!
Saturday, May 14, 2011
Funny thing Happened at STP in Nashville
Testers, I must find the time to catch up on my back log of posts. I thought I would start out with an embarrassing tale.
Once upon a time there was a tester, Carl, who had signed up for a work shop at the STP conference in March 2011 at Nashville TN. It was a beautiful day and the coffee was flowing. Carl happened to be running late for the workshop. All Carl knew was that the work shop was labeled Pre-5.
Carl raced up the stairs (very hard to do with a tender knee). There at the top of the stairs on the right was Pre-5. The instructors name was Bob. Carl remembered his instructor's name was Bob. The title of the work shop was Quality Monitoring and Coaching. For some reason Carl seemed out of place.
Everyone introduced themselves. Of course I introduced myself and why I was there. What did I say? "Hi I am Carl Shaulis and I am a Test Manager at HomeAway. I recently have been working close with Customer Service to facilitate quality and I am here to continue learning on how to bridge the gaps between our two teams." Wait one second, that does not sound right. Carl pulls out his conference brochure. As he thumbs through the pages Fiona walks into the conference room. Oh Carl knows Fiona from STP Las Vegas. Carl must be in the right place. Carl finally finds the work shop he should be in called "Creating and Leading a High Performance Test Organization" by Bob Galen.
Panic sets in! I am in the wrong room. I just gave a great introduction on how this presentation could benefit me. What should Carl do?
Carl quickly packed up his gear and headed out of the room. Carl found his intended work shop and enjoyed Spring STP Conference 2011.
Carl did get some friendly ribbing and crap from Abbie and Fiona. Deservedly so! The secret was out of the bag that there were two concurrent conferences Contact Center Conference Expo 2011 and Software Test Professionals.
The really fun part is that Carl probably would have learned a hell of a lot if he had stayed in the incorrect work shop. However, interacting with the testers and learning from Bob Galen was the right move.
Happy Testing!
Once upon a time there was a tester, Carl, who had signed up for a work shop at the STP conference in March 2011 at Nashville TN. It was a beautiful day and the coffee was flowing. Carl happened to be running late for the workshop. All Carl knew was that the work shop was labeled Pre-5.
Carl raced up the stairs (very hard to do with a tender knee). There at the top of the stairs on the right was Pre-5. The instructors name was Bob. Carl remembered his instructor's name was Bob. The title of the work shop was Quality Monitoring and Coaching. For some reason Carl seemed out of place.
Everyone introduced themselves. Of course I introduced myself and why I was there. What did I say? "Hi I am Carl Shaulis and I am a Test Manager at HomeAway. I recently have been working close with Customer Service to facilitate quality and I am here to continue learning on how to bridge the gaps between our two teams." Wait one second, that does not sound right. Carl pulls out his conference brochure. As he thumbs through the pages Fiona walks into the conference room. Oh Carl knows Fiona from STP Las Vegas. Carl must be in the right place. Carl finally finds the work shop he should be in called "Creating and Leading a High Performance Test Organization" by Bob Galen.
Panic sets in! I am in the wrong room. I just gave a great introduction on how this presentation could benefit me. What should Carl do?
Carl quickly packed up his gear and headed out of the room. Carl found his intended work shop and enjoyed Spring STP Conference 2011.
Carl did get some friendly ribbing and crap from Abbie and Fiona. Deservedly so! The secret was out of the bag that there were two concurrent conferences Contact Center Conference Expo 2011 and Software Test Professionals.
The really fun part is that Carl probably would have learned a hell of a lot if he had stayed in the incorrect work shop. However, interacting with the testers and learning from Bob Galen was the right move.
Happy Testing!
Tuesday, April 26, 2011
Defect Severity
I now have a huge backlog of blog posts, but here is one that has me losing some sleep.
Why do developers seem to get their feathers ruffled when we talk about defect severity? I am not implying that developers are ducks, chickens, or geese. I could have used the phrase why do developers get their panties in a wad when we talk about defect severity, but that did not seem PC.
First I will give a little back ground. I was asked to inspect and access team quality. Hmm, that is a monumental task. Where shall I start? Lets take a look at how many severity one defects are in production. If there are any severity one defects in production, then the quality status is RED. If there are no severity one defects in production then the quality status is GREEN. Seemed like a simple assessment, which would require some follow up conversations if the values were red. By no means do I mean to imply that this is the only indicator of quality, but it was the first flash light used to look into the cave.
But wait we (development) do not care about severity we focus all of our efforts on Priority. This has been echoed by at least three development leads.
My two brain cells are telling me that severity is the perception of how bad a defect is and priority is the process around getting a defect resolved. Hmm, do I have this all wrong? Is priority the only thing of importance. I read Perfect Software by Jerry Weinberg a few months ago. Did I not understand his writing? Well I guess I had better do whatever any internet geek would do, "Google It"!
I found a great post by my friend and colleague Dr. Stan Taylor. I did not see to much in that post that I could disagree with. Next I stumbled on a Bug Advocacy video by Professor Cem Kaner. I did not find a definitive answer there. I read at least 10 other blog posts and all seemed to have a slightly different spin, but there did seem to be some consistency that severity is the perception of the user and priority is the decision process on when to fix a defect.
I am still perplexed as to why inspecting severity gets such a negative response by development. Let me attempt to noodle out a couple of basic definitions. Please note that I did not come up with these definitions, but I tend to agree with these.
Perhaps feathers are not ruffled at all, because based on all of the blog posts the ambiguity of Severity and Priority have been around for a long long time.
Why do developers seem to get their feathers ruffled when we talk about defect severity? I am not implying that developers are ducks, chickens, or geese. I could have used the phrase why do developers get their panties in a wad when we talk about defect severity, but that did not seem PC.
First I will give a little back ground. I was asked to inspect and access team quality. Hmm, that is a monumental task. Where shall I start? Lets take a look at how many severity one defects are in production. If there are any severity one defects in production, then the quality status is RED. If there are no severity one defects in production then the quality status is GREEN. Seemed like a simple assessment, which would require some follow up conversations if the values were red. By no means do I mean to imply that this is the only indicator of quality, but it was the first flash light used to look into the cave.
But wait we (development) do not care about severity we focus all of our efforts on Priority. This has been echoed by at least three development leads.
My two brain cells are telling me that severity is the perception of how bad a defect is and priority is the process around getting a defect resolved. Hmm, do I have this all wrong? Is priority the only thing of importance. I read Perfect Software by Jerry Weinberg a few months ago. Did I not understand his writing? Well I guess I had better do whatever any internet geek would do, "Google It"!
I found a great post by my friend and colleague Dr. Stan Taylor. I did not see to much in that post that I could disagree with. Next I stumbled on a Bug Advocacy video by Professor Cem Kaner. I did not find a definitive answer there. I read at least 10 other blog posts and all seemed to have a slightly different spin, but there did seem to be some consistency that severity is the perception of the user and priority is the decision process on when to fix a defect.
I am still perplexed as to why inspecting severity gets such a negative response by development. Let me attempt to noodle out a couple of basic definitions. Please note that I did not come up with these definitions, but I tend to agree with these.
What is Quality? - “Quality is value to some person (that matters)”
What is a bug? – Something that bothers (bugs) someone who matters.
Severity - defines the impact the defect has on the customer and the likelihood of occurrence.
S1 – Product is on fire. We are getting sued. We cannot take money
S2 – Breaking but we can offer a workaround
S3 – Minor functional defect (pain in the butt or poor user experience)
S4 – Cosmetic (we can live with it)
Priority – determines the order in which defects will be fixed/resolved and retested
What do we fix?
When do we fix it?
Priority decisions are based on impact on the customer (severity) and resolution difficulty (time & resources)
P1 – Resolve Immediately
P2 – High Attention
P3 – Normal
P4 – low
Those seem reasonable to me and some of the definition came from a Power Point presentation by Dave Whalen called "The Matrix".
So now I am thinking that priority cannot be established without taking into consideration the severity.
I am still puzzled as to why developers do not want to track defects based on severity. Somebody perceived the defect as having a negative impact on someone.
I am concluding that I need to have more conversations with developers! Here is a reasonable post provided by one of the development leads. One point made in our conversation was that when I used the phrase perception of the customer it inserts emotion into the concern. The developer preferred impact on the customer.
I can certainly understand with the pressures of Rapid Software development of today there is little time for fixing defects. So the balancing act of meeting milestones and building great features is extremely priority centric.
However persons documenting the defect have no input with respect to priority at the time of discovery, so they must focus on setting a proper severity. A well-documented defect can be leveraged as a sales tool to influence priority and the person submitting the defect must be an advocate of the defect in order to see it through to resolution.
When severity and priority do not align, I would suggest it is extremely important to communicate the factors used to set the priority to the person who established the severity.
Hmm, have developers been burned in the past by incorrect values of severity being placed on the defects? Could that be the reason for the ruffled feathers? I think this might be a good possibility.
Time to read "Perfect Software" again!
Friday, April 01, 2011
Goodwill for Automation Code
I had an interesting dream, thought, or inspiration last night. It could have been codeine induced based on the horrible cough I have been battling, but no the less it was an important thought.
What happens to legacy code? Does it simply get deleted and gone eternally? Does it fall into an abyss?
No! We take it to Goodwill for Automation Code.
It occurred to me that one of the key takeaways from STP conference was mentoring. The "Ah Ha moment" was that we could leverage legacy code to educate others. We have some Software Engineers in test that are struggling with TDD, test driven development. So my concept is to look at our code repository and find code that has matured over time. We can then share this evolution of code with the testers that struggle with TDD.
I also struggled a bit with TDD, until I saw the light. Even for test automation it is important to unit test your code. How did I learn? What turned on the light?
Mentoring by my colleague certainly was key. The knowledge of knowing my new methods would potentially be used by many tests forced me to write better code, that could easily and repeatably be tested. I still struggle because I simply want to get a test implemented quickly, so I admit sometimes I write my unit tests after the fact. One rule of thumb I do have is that if I struggle at all to write a piece of code, I immediately stop and write the unit tests. Breaking the complexity into simpler chunks helps me to get to a more robust solution.
So I am going to go into the office today, then search for learning examples of code. These legacy jewels will then be moved into Goodwill for future education opportunities.
Cast off code will be used to assist others. Amazing the thoughts that come to you during sleep and I do not recommend the codeine!
Keep on testing!
What happens to legacy code? Does it simply get deleted and gone eternally? Does it fall into an abyss?
No! We take it to Goodwill for Automation Code.
It occurred to me that one of the key takeaways from STP conference was mentoring. The "Ah Ha moment" was that we could leverage legacy code to educate others. We have some Software Engineers in test that are struggling with TDD, test driven development. So my concept is to look at our code repository and find code that has matured over time. We can then share this evolution of code with the testers that struggle with TDD.
I also struggled a bit with TDD, until I saw the light. Even for test automation it is important to unit test your code. How did I learn? What turned on the light?
Mentoring by my colleague certainly was key. The knowledge of knowing my new methods would potentially be used by many tests forced me to write better code, that could easily and repeatably be tested. I still struggle because I simply want to get a test implemented quickly, so I admit sometimes I write my unit tests after the fact. One rule of thumb I do have is that if I struggle at all to write a piece of code, I immediately stop and write the unit tests. Breaking the complexity into simpler chunks helps me to get to a more robust solution.
So I am going to go into the office today, then search for learning examples of code. These legacy jewels will then be moved into Goodwill for future education opportunities.
Cast off code will be used to assist others. Amazing the thoughts that come to you during sleep and I do not recommend the codeine!
Keep on testing!
Thursday, March 17, 2011
Complexity
I know I have not posted in quite some time. Being busy is not a good excuse!
I do have numerous random ramblings floating through my brain with respect to testing. But my key thought for the day is that testing web applications can be extremely complex. Besides complexity the simplest deformity can cause the most chaos.
Are web applications similar to macromolecular complex assembly? Macromolecular complex assembly is the aggregation, arrangement and bonding together of a set of components to form a complex.
I guess I would consider the combining of a bunch of disparate technology stacks an aggregation. The integration of these technologies certainly must be arranged and bonded. I believe today that tiny aspects of this complex are under the scrutiny of a microscope.
Someone of importance notices that an implementation in a specific browser causes a line to wrap. This is a tiny deformity on the surface of a cell causes concern, yet the neighboring system which is about to go into complete cardiac arrest goes unnoticed.
Why is it that the tiniest blemish on a complex system gets noticed first? Why is it that the most insignificant concern causes the most chaos for a tester?
Unfortunately it becomes difficult for testers to inspect the entire surface of a cell, when the internal organs (mitochondria) are in need of some much attention. As we inspect and flush out the issues with the key features we get the most demerits for a cosmetic blemish on the user interface.
So I guess the resolution is to perform CPR on the failing feature while fixing the blemish. The complex will endure!
I do have numerous random ramblings floating through my brain with respect to testing. But my key thought for the day is that testing web applications can be extremely complex. Besides complexity the simplest deformity can cause the most chaos.
Are web applications similar to macromolecular complex assembly? Macromolecular complex assembly is the aggregation, arrangement and bonding together of a set of components to form a complex.
I guess I would consider the combining of a bunch of disparate technology stacks an aggregation. The integration of these technologies certainly must be arranged and bonded. I believe today that tiny aspects of this complex are under the scrutiny of a microscope.
Someone of importance notices that an implementation in a specific browser causes a line to wrap. This is a tiny deformity on the surface of a cell causes concern, yet the neighboring system which is about to go into complete cardiac arrest goes unnoticed.
Why is it that the tiniest blemish on a complex system gets noticed first? Why is it that the most insignificant concern causes the most chaos for a tester?
Unfortunately it becomes difficult for testers to inspect the entire surface of a cell, when the internal organs (mitochondria) are in need of some much attention. As we inspect and flush out the issues with the key features we get the most demerits for a cosmetic blemish on the user interface.
So I guess the resolution is to perform CPR on the failing feature while fixing the blemish. The complex will endure!
Sunday, February 06, 2011
Testing for the Super Bowl Ad
I am sitting at HomeAway's Super Bowl Command Center patiently waiting for the game to begin. I have not posted in awhile due to an enormous amount of testing.
We have been conducting performance tests for almost 4 weeks. Usually on Sunday we could do our most impacting performance tests. We ran tests every Tuesday, Thursday and Sunday. Is our site ready? We should know in about 5 hours. Thanks to tools like jMeter and BrowserMob I believe we are ready!
Not only was I involved with a great team for performance testing, I also had to test some new applications developed specifically for the Super Bowl. This testing effort was exhausting. The final build was delivered at 1:00 PM Saturday. Yep, that is right! Just 28 hours before the Super Bowl.
I am confident in the software, but there are some known issues. Very difficult to test across the variety of OS and browser configurations. Throw in different versions of flash and media players you have an infinite amount of testing. If I had not read Gerald Weinberg's book "Perfect Software", I think I would be freaking out right about now.
So what would be the lessons learned over the past four weeks.
Why stay in a hotel when you can HOME AWAY!
Vote for HomeAway's Super Bowl commercial!
We have been conducting performance tests for almost 4 weeks. Usually on Sunday we could do our most impacting performance tests. We ran tests every Tuesday, Thursday and Sunday. Is our site ready? We should know in about 5 hours. Thanks to tools like jMeter and BrowserMob I believe we are ready!
Not only was I involved with a great team for performance testing, I also had to test some new applications developed specifically for the Super Bowl. This testing effort was exhausting. The final build was delivered at 1:00 PM Saturday. Yep, that is right! Just 28 hours before the Super Bowl.
I am confident in the software, but there are some known issues. Very difficult to test across the variety of OS and browser configurations. Throw in different versions of flash and media players you have an infinite amount of testing. If I had not read Gerald Weinberg's book "Perfect Software", I think I would be freaking out right about now.
So what would be the lessons learned over the past four weeks.
- You must performance test with a great group of technical people, because collaboration is the key.
- You must insist on testing third party applications early and often.
- You must learn to let go, because unfortunately with a small time line and infinite number of test cases, you can only do so much.
- You must have good performance testing tools and a cloud vendor like Browser Mob who will go the extra mile.
- You must have a dedicated team who are willing to put in long hours.
- You must have testers who are willing to report all of their observations.
- You must not use a Google Shared document to track and collaborate on issues. Use a web based defect tracking tool. (Primary mistake I made for Super Bowl)
- You must think about process as soon as you engage a third party vendor
- You must have network monitoring in place.
Why stay in a hotel when you can HOME AWAY!
Vote for HomeAway's Super Bowl commercial!
Monday, January 17, 2011
Presentation sponsored by Thoughtworks and Agile Austin
I am shifting gears a bit here, but I had the absolute pleasure of listening to Jim Highsmith speak here in Austin Texas. Last Wednesday Thoughtworks and Agile Austin sponsored a talk by Mr. Highsmith on the topic of Agile Leadership.
I could never articulate all of the great content presented by Mr. Highsmith, so I will toss out some key concepts I managed to snag. I am hoping Thoughtworks will distribute the slide deck so I can remember even more.
Focusing on Organizational Agility I recall one fundamental distinction. Is your business focus based on Responsiveness or Efficiency? If it is Responsiveness then it is probably important to learn Agile Management skills.
The core tenants according to Mr. Highsmith were Do less, Quality, Speed to Value, and Engage & Inspire.
Do less is definitely applicable to my recent experiences with kanban.
Of course Quality is important to me. Quality really does matter to management and it is a proven fact that reducing defects reduces cost.
I am not so enthusiastic about increasing speed, but speed to value is a key to success. We really do not want to be focusing on things that add little value. In my opinion, we need to find a sustainable pace.
Now Engage and Inspire is a prime theme for me these days. After attending the STP Conference in the fall of 2010, I certainly became inspired. I am seeking the opportunity to engage with other testers and be inspired. Hopefully I can inspire too.
Mr. Highsmith recommended several books. You can find those recommendations on his blog. I recall two suggestions: Drive by Dan Pink and Organizing Genius by Warren Bennis.
Software companies should have a focus on value and delivering features. Wow! That certainly makes sense to me. An interesting point was that companies should not permit product owners to do all of the prioritization. If you let product determine the priority, then there is no room to work on technical debt, defects, or useful tools. His point was many companies suffer from technical debt.
The talk inspired me to do my best to participate more in the Agile community in Austin. I was speaking with Matt, the president of Agile Austin and I found out there is a group of testers that meet monthly over lunch. I definitely want to get plugged in to that group.
My final point is if you get a chance to go hear these great speakers, please do so! There are bits of goodness for everyone to take home.
Happy Testing!
I could never articulate all of the great content presented by Mr. Highsmith, so I will toss out some key concepts I managed to snag. I am hoping Thoughtworks will distribute the slide deck so I can remember even more.
Focusing on Organizational Agility I recall one fundamental distinction. Is your business focus based on Responsiveness or Efficiency? If it is Responsiveness then it is probably important to learn Agile Management skills.
The core tenants according to Mr. Highsmith were Do less, Quality, Speed to Value, and Engage & Inspire.
Do less is definitely applicable to my recent experiences with kanban.
Of course Quality is important to me. Quality really does matter to management and it is a proven fact that reducing defects reduces cost.
I am not so enthusiastic about increasing speed, but speed to value is a key to success. We really do not want to be focusing on things that add little value. In my opinion, we need to find a sustainable pace.
Now Engage and Inspire is a prime theme for me these days. After attending the STP Conference in the fall of 2010, I certainly became inspired. I am seeking the opportunity to engage with other testers and be inspired. Hopefully I can inspire too.
Mr. Highsmith recommended several books. You can find those recommendations on his blog. I recall two suggestions: Drive by Dan Pink and Organizing Genius by Warren Bennis.
Software companies should have a focus on value and delivering features. Wow! That certainly makes sense to me. An interesting point was that companies should not permit product owners to do all of the prioritization. If you let product determine the priority, then there is no room to work on technical debt, defects, or useful tools. His point was many companies suffer from technical debt.
The talk inspired me to do my best to participate more in the Agile community in Austin. I was speaking with Matt, the president of Agile Austin and I found out there is a group of testers that meet monthly over lunch. I definitely want to get plugged in to that group.
My final point is if you get a chance to go hear these great speakers, please do so! There are bits of goodness for everyone to take home.
Happy Testing!
Sunday, January 09, 2011
Build Blocked at the end of Week 1
The team implemented their new process and this was the first week in action. They started off with a large bottleneck in the acceptance test column. The team quickly moved through the bottleneck.
As an outside observer the testers seemed to be happier. They were actively involved at the start of the process rather than at the end. Developers were completing acceptance test quickly and what appeared to be efficiently. Blocked stories were obviously labeled on the board. There definitely was a buzz in the air.
I did over hear a couple of developers wondering why the hell are we doing this, "Seems like we are reinventing the wheel!" There were a couple of developers who were blocked. So there was more time for foosball.
The teams goal was to have a branched release candidate by the close of business on Thursday. Unfortunately at the close of business on Friday testers had found enough defects that a clean build did not exist. The release is due out Tuesday.
It will be very interesting to see how things go on Monday. Will the testers be able to adjust their regression testing pattern in order to get out a solid release.
As an outside observer I really enjoy collaborating with the test and development leads. I shared two observations with respect to their kanban board. Looking at the board there were numerous stories that were marked as blocked. To me it was not clear from the board who owned the action item to unblock the story. The other suggestion that I made was that they also need some sort of issue board or list, so everyone on the team knows what things are impeding the process flow.
The new 9 foot magnetic board is in. The team has created some magnetic buttons with their personal avatars. Lets just say some of the avatars are quite interesting, but I think it adds a bit of fun to the process. They also created the yuck face buttons, which they can apply to blocked stories. I am looking forward to seeing this board in action.
Oh one more very positive observation was that the team was able to move a few more defects through the process than were original scheduled. I liked the buzz of the new process and testers know what to test.
Several of the team members were able to collaborate over an 18 page Power Point document and condense the material down into a handful of cukes. Cukes are executable requirements from the open source tool, Cucumber. Soon we will implement the strategy to automate the cukes.
Laissez les bons temps rouler!
As an outside observer the testers seemed to be happier. They were actively involved at the start of the process rather than at the end. Developers were completing acceptance test quickly and what appeared to be efficiently. Blocked stories were obviously labeled on the board. There definitely was a buzz in the air.
I did over hear a couple of developers wondering why the hell are we doing this, "Seems like we are reinventing the wheel!" There were a couple of developers who were blocked. So there was more time for foosball.
The teams goal was to have a branched release candidate by the close of business on Thursday. Unfortunately at the close of business on Friday testers had found enough defects that a clean build did not exist. The release is due out Tuesday.
It will be very interesting to see how things go on Monday. Will the testers be able to adjust their regression testing pattern in order to get out a solid release.
As an outside observer I really enjoy collaborating with the test and development leads. I shared two observations with respect to their kanban board. Looking at the board there were numerous stories that were marked as blocked. To me it was not clear from the board who owned the action item to unblock the story. The other suggestion that I made was that they also need some sort of issue board or list, so everyone on the team knows what things are impeding the process flow.
The new 9 foot magnetic board is in. The team has created some magnetic buttons with their personal avatars. Lets just say some of the avatars are quite interesting, but I think it adds a bit of fun to the process. They also created the yuck face buttons, which they can apply to blocked stories. I am looking forward to seeing this board in action.
Oh one more very positive observation was that the team was able to move a few more defects through the process than were original scheduled. I liked the buzz of the new process and testers know what to test.
Several of the team members were able to collaborate over an 18 page Power Point document and condense the material down into a handful of cukes. Cukes are executable requirements from the open source tool, Cucumber. Soon we will implement the strategy to automate the cukes.
Laissez les bons temps rouler!
Sunday, January 02, 2011
Diving into the Abyss
We had a couple more brain storming sessions on moving a team onto the kanban train.
The biggest concern during these two sessions focused on our build process and software control systems. We currently use Subversion. Much of the discussion was around the complexities of checking in and backing out code. How can our cadence flow in order to give testers builds of high quality and builds of trust. I believe the final solution discussed was that we need to move from Subversion to GIT.
The development team believes GIT is a more friendly change control system for moving software around. We shall see, but we do not have GIT today and we are about to jump into the abyss.
From my perspective, there are two potential issues when the team arrives for work on Tuesday. One concern is that not all of the stakeholders were present during our brainstorming sessions. I believe some team members will be surprised on Tuesday, but most will jump into the kanban abyss with both feet. The other concern is that the WIP is already way over the estimated capacity.
Despite the WIP we decided to make the leap. I stopped by the office on Friday and the Development Lead had a carefully crafted kanban board drawn across a couple of white boards. He has a 9 foot magnetized board on order, but it has not arrived. "See the team is already adjusting." The large portion of the WIP is currently in the completed state and ready for acceptance testing. The entire team (developers and testers) is going to start with this phase, write acceptance tests when they are not available, actuate the code, and test the pending features. Viewing the kanban board I also noticed several key stories and defects have been placed in the ready state.
I am looking forward as an observer/consultant to see the giant green start button get pushed on Tuesday and the new process will begin to flow. During our brain storming sessions to other key concepts were often mentioned. During daily stand up meetings we have the ability to make minor adjustments. Communication will be important. The second concept was the big RED button. The big RED button is "stop the presses" button. I am hoping that for the first one week release cycle the red button is not used.
Unfortunately there is a high volume of risk with this release as the team jumps into the abyss. If the team does a good job conducting thorough code reviews, acceptance tests and the test team stays on top of their game, then we may have the parachutes to float gently to the bottom of the abyss. If we happen to land hard then we have the retrospective and make the first adjustments to the process.
The team is leaping and I believe 2011 is going to be a great year!
The biggest concern during these two sessions focused on our build process and software control systems. We currently use Subversion. Much of the discussion was around the complexities of checking in and backing out code. How can our cadence flow in order to give testers builds of high quality and builds of trust. I believe the final solution discussed was that we need to move from Subversion to GIT.
The development team believes GIT is a more friendly change control system for moving software around. We shall see, but we do not have GIT today and we are about to jump into the abyss.
From my perspective, there are two potential issues when the team arrives for work on Tuesday. One concern is that not all of the stakeholders were present during our brainstorming sessions. I believe some team members will be surprised on Tuesday, but most will jump into the kanban abyss with both feet. The other concern is that the WIP is already way over the estimated capacity.
Despite the WIP we decided to make the leap. I stopped by the office on Friday and the Development Lead had a carefully crafted kanban board drawn across a couple of white boards. He has a 9 foot magnetized board on order, but it has not arrived. "See the team is already adjusting." The large portion of the WIP is currently in the completed state and ready for acceptance testing. The entire team (developers and testers) is going to start with this phase, write acceptance tests when they are not available, actuate the code, and test the pending features. Viewing the kanban board I also noticed several key stories and defects have been placed in the ready state.
I am looking forward as an observer/consultant to see the giant green start button get pushed on Tuesday and the new process will begin to flow. During our brain storming sessions to other key concepts were often mentioned. During daily stand up meetings we have the ability to make minor adjustments. Communication will be important. The second concept was the big RED button. The big RED button is "stop the presses" button. I am hoping that for the first one week release cycle the red button is not used.
Unfortunately there is a high volume of risk with this release as the team jumps into the abyss. If the team does a good job conducting thorough code reviews, acceptance tests and the test team stays on top of their game, then we may have the parachutes to float gently to the bottom of the abyss. If we happen to land hard then we have the retrospective and make the first adjustments to the process.
The team is leaping and I believe 2011 is going to be a great year!
Tuesday, December 28, 2010
Kanban - Part II
Yesterday we had a brain storming session on what kanban would look like for the team. I must say it generated a ton of good conversation. Here are a couple of key points that I remember.
Someone mentioned that a team of testers could spend a majority of their time in the design phase, especially facilitating the development of Cucumber scripts. It was suggested that one tester each week focus on this area and rotate the responsibility each week. Another suggestion was to pair a tester and developer for each story. The primary premise discussed was that if effective Cucumber scripts are developed then further down the process testers would know what to test even though they did not participate in the design phase.
Another key point was with respect to the current defect backlog. How do we keep the defect backlog from building? The team concluded that we need a dedicated swim lane in the process dedicated to fixing defects. Dan Anderson shows in his book a good example where X% of the capacity is dedicated to defect resolution.
What happens if a story is full of defects? The conclusion was that the story remains in the process and is not released. Since the team is inspecting the process daily, they can make adjustments to provide potential solutions. One approach might be paired-development practices which provides real time training and retrospectives on what is happening with the story. The ultimate conclusion is that if the team only got one story deployed each week they were adding incremental value to the business. This should not happen often, but the bottom line is that it is OK to tweak the process on a daily basis and focus on deploying high quality valued features.
The one scary piece that we did not resolve was dealing with complexity. Occasionally an Epic story comes into the team's backlog. During the design phase we should be able to dissect and break the Epic into smaller stories, but the concern became how do we make sure the stories get implemented in the right order and that all of the stories that comprise the whole get assembled, tested, and deployed. Adjusting for complexity will be a fun challenge, but a challenge never the less.
We talked about buffer zones and capacity limits. We have no clue where to set the capacity limits. It was suggested that we minimize the WIP so that we can gain confidence in the process and trust in the quality of the features being produced.
I must admit I am excited about this approach. I know we will fall flat on our face a couple of times, but we will get up, collaborate, and continuous improve the process.
Stay tuned for Part 3. We plan to walk through some real scenarios.
Someone mentioned that a team of testers could spend a majority of their time in the design phase, especially facilitating the development of Cucumber scripts. It was suggested that one tester each week focus on this area and rotate the responsibility each week. Another suggestion was to pair a tester and developer for each story. The primary premise discussed was that if effective Cucumber scripts are developed then further down the process testers would know what to test even though they did not participate in the design phase.
Another key point was with respect to the current defect backlog. How do we keep the defect backlog from building? The team concluded that we need a dedicated swim lane in the process dedicated to fixing defects. Dan Anderson shows in his book a good example where X% of the capacity is dedicated to defect resolution.
What happens if a story is full of defects? The conclusion was that the story remains in the process and is not released. Since the team is inspecting the process daily, they can make adjustments to provide potential solutions. One approach might be paired-development practices which provides real time training and retrospectives on what is happening with the story. The ultimate conclusion is that if the team only got one story deployed each week they were adding incremental value to the business. This should not happen often, but the bottom line is that it is OK to tweak the process on a daily basis and focus on deploying high quality valued features.
The one scary piece that we did not resolve was dealing with complexity. Occasionally an Epic story comes into the team's backlog. During the design phase we should be able to dissect and break the Epic into smaller stories, but the concern became how do we make sure the stories get implemented in the right order and that all of the stories that comprise the whole get assembled, tested, and deployed. Adjusting for complexity will be a fun challenge, but a challenge never the less.
We talked about buffer zones and capacity limits. We have no clue where to set the capacity limits. It was suggested that we minimize the WIP so that we can gain confidence in the process and trust in the quality of the features being produced.
I must admit I am excited about this approach. I know we will fall flat on our face a couple of times, but we will get up, collaborate, and continuous improve the process.
Stay tuned for Part 3. We plan to walk through some real scenarios.
Friday, December 24, 2010
Kanban
I am wide awake Christmas Eve day at 5:20 AM. I should be sleeping! Instead I am reading a book by David Anderson, Kanban. Why as a tester am I reading Kanban? Well one of the teams I work closely with is about to implement kanban in 2011 as an attempt to improve their process. So I ponder how does test have to change on order to fit into this paradigm.
I am not even going to attempt to describe kanban in this post, but I do want to convey I get a bit excited when I see the first step David Anderson recommends for moving to kanban. He says the first step is to "Focus on Quality". As a tester you have to like hearing those words.
I also agree with these points.
The team using kanban will be doing weekly releases. They have been doing weekly releases, but development and test cycles were taking four weeks (2 in development and 2 in test). The product owners were not getting their new features in a timely manner.
In December the team has been focusing on reducing the backlog of defects and discussing the implications of kanban. In my opinion, the biggest fear of this change is coming from the testers.
I contend there is nothing to fear. A story will move through this flow:
Idea > Design > Work in Progress (code, unit test, code review, automation) > acceptance test > final test > production
If there are 10 stories in the pipeline and 5 testers. Then a tester is responsible for 2 stories. Testers should be able to test 2 stories in a week. Right? OK it does depend on the size of the story. The fear comes from the regression test phase. My philosophy is automate regression tests where possible and focus on Exploratory Testing. If testers cannot get over their fear perhaps any manual regression testing should be focused on the area(s) of highest risk.
I predict that the process will be bottle-necked in two areas. The first bottleneck will be in design. As part of design the team will be writing Cucumber tests. Through collaboration the cucumber tests will make sure everyone is understanding the feature and testers will know what to test early in the process. This process will take time.
I believe acceptance testing will go fast, but the second bottleneck will be the final testing phase. Unfortunately testers are still focused on "Perfect Software". I contend that the testers will have to implement test strategies that can be completed in 1-2 days.
I am looking forward to seeing this in action.
The extremely nice thing about kanban is that you know your pain points because you see them on the flow chart. More importantly kanban expects you to change the process in order to remove the pain points.
Over simplification? Maybe! As Larry the Cable Guy might say "Git R done"!
I am definitely looking forward to this challenge in 2011. I would welcome any comments from testers who are actively doing kanban. I suspect there are not many.
I am not even going to attempt to describe kanban in this post, but I do want to convey I get a bit excited when I see the first step David Anderson recommends for moving to kanban. He says the first step is to "Focus on Quality". As a tester you have to like hearing those words.
I also agree with these points.
- Both Agile and traditional development approaches to quality have merit.
- Code inspections if done right can improve quality.
- Collaborative analysis and design improve quality. I believe testers should get involved early in the process!
- Using design patterns improve quality.
- Use of modern development tools improve quality.
The team using kanban will be doing weekly releases. They have been doing weekly releases, but development and test cycles were taking four weeks (2 in development and 2 in test). The product owners were not getting their new features in a timely manner.
In December the team has been focusing on reducing the backlog of defects and discussing the implications of kanban. In my opinion, the biggest fear of this change is coming from the testers.
I contend there is nothing to fear. A story will move through this flow:
Idea > Design > Work in Progress (code, unit test, code review, automation) > acceptance test > final test > production
If there are 10 stories in the pipeline and 5 testers. Then a tester is responsible for 2 stories. Testers should be able to test 2 stories in a week. Right? OK it does depend on the size of the story. The fear comes from the regression test phase. My philosophy is automate regression tests where possible and focus on Exploratory Testing. If testers cannot get over their fear perhaps any manual regression testing should be focused on the area(s) of highest risk.
I predict that the process will be bottle-necked in two areas. The first bottleneck will be in design. As part of design the team will be writing Cucumber tests. Through collaboration the cucumber tests will make sure everyone is understanding the feature and testers will know what to test early in the process. This process will take time.
I believe acceptance testing will go fast, but the second bottleneck will be the final testing phase. Unfortunately testers are still focused on "Perfect Software". I contend that the testers will have to implement test strategies that can be completed in 1-2 days.
I am looking forward to seeing this in action.
The extremely nice thing about kanban is that you know your pain points because you see them on the flow chart. More importantly kanban expects you to change the process in order to remove the pain points.
Over simplification? Maybe! As Larry the Cable Guy might say "Git R done"!
I am definitely looking forward to this challenge in 2011. I would welcome any comments from testers who are actively doing kanban. I suspect there are not many.
Sunday, December 19, 2010
Waterfall vs Agile
This past week Lanette Creamer posted on her great blog some statements with respect to her current team. She implies that Waterfall teams are better at communication than Agile teams.
I would contend that the methodology has little to do with it. You take a motivated and well spoken tester like Lanette, then combine other motivated members to her team and you will get good communication.
Motivated people make communication happen. I will go one step further and say good communication makes for successful projects.
I have worked on waterfall, v-model, and agile projects. If given the choice I will choose agile.
With waterfall and v-model I always found the massive amounts of documentation, document review, and the cumbersome process of reviewing changes to the documentation to be tedious. In my opinion at times the process hindered communication.
With agile, lean, or Kanban we are afforded the opportunity to learn from our mistakes and quickly change the process.
But as to Lanette's point, agile projects can only be successful with excellent team communication.
So everyone who would like their projects to be successful sharpen your communication skills.
I would contend that the methodology has little to do with it. You take a motivated and well spoken tester like Lanette, then combine other motivated members to her team and you will get good communication.
Motivated people make communication happen. I will go one step further and say good communication makes for successful projects.
I have worked on waterfall, v-model, and agile projects. If given the choice I will choose agile.
With waterfall and v-model I always found the massive amounts of documentation, document review, and the cumbersome process of reviewing changes to the documentation to be tedious. In my opinion at times the process hindered communication.
With agile, lean, or Kanban we are afforded the opportunity to learn from our mistakes and quickly change the process.
But as to Lanette's point, agile projects can only be successful with excellent team communication.
So everyone who would like their projects to be successful sharpen your communication skills.
Monday, November 29, 2010
Books I am currently reading
All testers MUST read Perfect Software and Other Illusions about Testing by Gerald Weinberg. I just finished reading this book and I found the content extremely useful and inspiring. If you test software, read this book.
Now I am reading Lessons Learned in Software Testing by Cem Kaner, James Bach, and Brett Pettichord. I am only a little ways into this book, but I am finding it to be spot on.
Next in the queue is Kanban - Successful Evolutionary Change for Your Technology Business by David J. Anderson.
If you are reading a great testing or development book, please make a comment so I can get more books into my queue.
Read on!
Now I am reading Lessons Learned in Software Testing by Cem Kaner, James Bach, and Brett Pettichord. I am only a little ways into this book, but I am finding it to be spot on.
Next in the queue is Kanban - Successful Evolutionary Change for Your Technology Business by David J. Anderson.
If you are reading a great testing or development book, please make a comment so I can get more books into my queue.
Read on!
Sunday, November 21, 2010
Think Times for Performance Testing
After only 10 years in the software industry, I am probably naive, but a recent post by Dan Barstow has caused me to question the use of think times.
Our current performance testing approach does not have think times. I have used think times in the past for various applications. I always have found think times to be somewhat arbitrary and subjective. I can definitely see the value if your objective is to really understand the number of concurrent users the application can handle, but typically our objectives are focused on the number of transactions a system can process. Also if you are using virtual users you truly are not representing a browser,. A browser, depending on type, could leverage 6 threads at a time making requests. So how do you adjust the think time between thread A getting an image and thread B retrieving javascript and firing the script?
Specifically we look for what is the optimal TPS value where 95% of requests latencies are less than 1 second and 99% of all requests are less than 4 seconds.
In my previous career I was a chemist. On some regular frequency we had to calibrate our analytical instrumentation. We would generate a 5-point curve using known standards. Typically we sought a linear relationship of this curve where the low point was near the detection limit and the high point was near to the maximum detection point of the analytical instrumentation (e.g. – Gas Chromatographs).
When performance testing ramping VU in time intervals is similar to generating this calibration curve. What I am typically looking for is how high can we generate the curve, yet still be below the high point of the instruments capability, which in this case the instrument is a configuration of servers and services.
I guess my philosophy might be that by eliminating think time I can get to that high point a bit faster. Since we are testing a system, then my criteria is applicable to the total request and response. So if the request is for an image or for a database object, as long as that request is returned within the criteria stated above, then we are within calibration.
My opinion is that if you are using virtual users then transactions per second is the important metric. If you are using real browser users, then I think that think times might be important, but still subjective. Regardless the objective is to do the best job possible to validate that your site gives customers a good experience. I guess this can be achieved with or without think times.
For the record I have done some performance testing of JSON API where you could not even conduct the test without some level of think time. So I will refine my conclusion to state that it depends on your objective and the application under test.
Please help educate me, because I am not in the think time camp unless it is necessary.
Wednesday, November 17, 2010
Selenium Fury has been released!
In my humble opinion, the page object pattern can be extremely important for a test framework.
Do you have an ever changing UI that causes your locators to frequently break?
With ASP.net do your developers frequently move elements around the page?
Do you have a brand new page and you quickly want locators to work with?
Selenium Fury just might be the tool to assist you.
I have posted in the past about generators and validators. Scott Sims has rolled this functionality up into a nice gem. The gem is easily extensible.
Check out Scott's post, use the gem and let him know what you think!
http://scottcsims.com/wordpress/?p=251
Oh, it is so fun being a tester!
Do you have an ever changing UI that causes your locators to frequently break?
With ASP.net do your developers frequently move elements around the page?
Do you have a brand new page and you quickly want locators to work with?
Selenium Fury just might be the tool to assist you.
I have posted in the past about generators and validators. Scott Sims has rolled this functionality up into a nice gem. The gem is easily extensible.
Check out Scott's post, use the gem and let him know what you think!
http://scottcsims.com/wordpress/?p=251
Oh, it is so fun being a tester!
Thursday, November 11, 2010
Pirates vs Ninjas make it to Prime Time
I was catching up on some recorded shows. I was watching my new favorite show, The Defenders. Funny that it is set in Las Vegas the last location of STPCon. Any rate last nights episode involved a hacker. The main characters mentioned pirates and ninjas when talking about the hacker. Were they observing Adam's and Lanette's Lightning talks at STPCon. I found it very funny to hear pirates and ninjas mentioned regarding software on a prime time show.
The lightning talks were one of the highlights of the conference. They were put on as breakfast bytes by Matt Heuser. I love the format of lightning talks.
Since STPCon I have done some thinking about "pirates being better than Ninjas". I have to respectively disagree with Adam Goucher. I think ninjas are better than pirates. Pirates might support developers in a loud and boisterous manner. Ninjas would support developers in a silent and stealthy manner. The good news is that testers exist to support development.
I guess it was obvious that I lean toward ninjas, hence the title of this blog. That reminds me I need to get my ninjapus image uploaded to this humble blog. My kids worked really hard prior to STPCon to help me create my logo. I need to shrink it to appear on the blog, but I can certainly include it in this post. You have to love the power of a ninja with 8 arms and multiple weapons for testing. I think my kids 11 & 14 did an excellent job! Go Ninjas!
The lightning talks were one of the highlights of the conference. They were put on as breakfast bytes by Matt Heuser. I love the format of lightning talks.
Since STPCon I have done some thinking about "pirates being better than Ninjas". I have to respectively disagree with Adam Goucher. I think ninjas are better than pirates. Pirates might support developers in a loud and boisterous manner. Ninjas would support developers in a silent and stealthy manner. The good news is that testers exist to support development.
I guess it was obvious that I lean toward ninjas, hence the title of this blog. That reminds me I need to get my ninjapus image uploaded to this humble blog. My kids worked really hard prior to STPCon to help me create my logo. I need to shrink it to appear on the blog, but I can certainly include it in this post. You have to love the power of a ninja with 8 arms and multiple weapons for testing. I think my kids 11 & 14 did an excellent job! Go Ninjas!
Wednesday, November 03, 2010
Tester and Developer Pairing
I know at STPcon Lanette Creamer gave a session on "Pairing with Developers". In hind site I truly wish I had attended that session.
Currently one of our development teams is pushing out two releases per week. There are a couple of issues in that the code is actually complete in the previous sprint and it takes test 1-2 weeks to execute regression. This is not meeting the needs of the business, so we have added in "quick releases" with minimum code changes and minimal testing. On top of that we are also injecting releases where we are conducting A/B testing (more on that latter). In order to keep up an efficient pace, the team is now considering daily releases. Wow!
This is essentially a shift to Kanban. If you listen to Kent Beck, moving to this type of SDLC a team would essentially remove the testers. I respectively disagree. Testers can still play a critical role in the reporting of quality and making sure the releases are meeting business and customer needs. As I ponder this paradigm shift, I believe "Pairing with Developers" becomes essential.
Developer grabs the next story and pairs with an experienced tester. They work together design the unit tests. As the developer implements the unit tests, the tester designs the acceptance tests (Cucumber) and developer does a code read. As the developer writes the code, the tester implements the acceptance tests. The tester should also collaborate with the developer to design and implement any browser facing tests (Selenium). Integration tests should also be considered. At the end of the day the code should be complete and all of the tests pass. If all tests pass, then automated deployment pushes the release for UAT. UAT Testers evaluate the functionality in a non-customer facing production environment. If the user acceptance tests pass, then switch the environment to be customer facing. In the A/B world we can expose only 20% of the customers to the new feature. If there are no issues raised by the customers, then we push live.
There may be other ways to complete the test automation, but this paradigm makes sense to me. Not all testers have automation experience, so this places the emphasis on education and training. How quickly will we get there? I do not know. Is it possible? Absolutely!
I am looking forward to this challenge. If any of you are actively doing this style of rapid development please share you thoughts.
Gotta LOVE testing!
Currently one of our development teams is pushing out two releases per week. There are a couple of issues in that the code is actually complete in the previous sprint and it takes test 1-2 weeks to execute regression. This is not meeting the needs of the business, so we have added in "quick releases" with minimum code changes and minimal testing. On top of that we are also injecting releases where we are conducting A/B testing (more on that latter). In order to keep up an efficient pace, the team is now considering daily releases. Wow!
This is essentially a shift to Kanban. If you listen to Kent Beck, moving to this type of SDLC a team would essentially remove the testers. I respectively disagree. Testers can still play a critical role in the reporting of quality and making sure the releases are meeting business and customer needs. As I ponder this paradigm shift, I believe "Pairing with Developers" becomes essential.
Developer grabs the next story and pairs with an experienced tester. They work together design the unit tests. As the developer implements the unit tests, the tester designs the acceptance tests (Cucumber) and developer does a code read. As the developer writes the code, the tester implements the acceptance tests. The tester should also collaborate with the developer to design and implement any browser facing tests (Selenium). Integration tests should also be considered. At the end of the day the code should be complete and all of the tests pass. If all tests pass, then automated deployment pushes the release for UAT. UAT Testers evaluate the functionality in a non-customer facing production environment. If the user acceptance tests pass, then switch the environment to be customer facing. In the A/B world we can expose only 20% of the customers to the new feature. If there are no issues raised by the customers, then we push live.
There may be other ways to complete the test automation, but this paradigm makes sense to me. Not all testers have automation experience, so this places the emphasis on education and training. How quickly will we get there? I do not know. Is it possible? Absolutely!
I am looking forward to this challenge. If any of you are actively doing this style of rapid development please share you thoughts.
Gotta LOVE testing!
Subscribe to:
Posts (Atom)