Friday, January 26, 2007

Trip Report for AWTA

The Austin Workshop for Test Automation, AWTA, took place in Austin Texas January 12 -14, 2007.

The information below outlined italics is intended to be from my perspective, including tangent thoughts during the conference.

The first think worth note is the meeting format. The meeting took on its own flavor of LAWST meeting style.

Meeting began with brief introductions round table, stating who you are and what you hope to gain from workshop.

Next a participant gave an experience report. Each person at the workshop was given a green, purple, and pink index card. During the experience report the only questions permitted are clarifying questions. If it is not a clarifying question the facilitator will move the report forward and table the question. After the experience report it becomes “Open Season”. Here is where the cards come into play. If you have a question or comment regarding the report you hold up your green card and the facilitator puts your name on a list. If a discussion is going on and you want to add input you hold up the purple card. If you think a topic has deviated from the topic or you think it has been beat to death you raise the pink card. This format is a very good means to keep a meeting on topic and permit everyone to participate.

The first experience report was presented by Jean McAuliffe. She previously had worked for Rational and had gotten tired of the rigorous process. She went to work for a start up that used Agile techniques. She fell in love with the approach and began using an open source tool called Fitnesse. This tool is based on an open-source project called Fit.

This is a tool that can allow the team to test without the GUI. Column based data from a Wiki page are fed into fixtures created by the developers. This is a cool way to test, but I am not sure it is appropriate for us. In my opinion we need to have Unit Test coverage and a more robust UI testing suite.

Jean explained how she left this agile team to lead a start up. All of the effort was sweat equity and she found it difficult to implement Fitnesse, because the developers did not want to spend time writing the fixtures. These tools are unique in that there are no tools like it on the open market. Sometimes test work fine in Fitnesse, but then fail at the GUI level. The Fit discussions lead Paul Roger’s and others to begin talking about the difficulties of open source tools. “What would make open-source tools easier to use? Bulleted ListSome responses included:

  • More concrete or real world examples
  • More Peer help
  • Better documentation
  • Documented in different ways to serve a wider audience
  • Identify and bring in Tool Smiths

Brian Marick added that consumer of tools needed

  • Direct communication to the experts
  • Developers to have a good attitude toward Newbies
  • Acknowledgement to members who submit quality defects
  • Perhaps documentation similar to Wikapedia, but can only add content and not edit the original content. A place where clarifications, struggles, and resolutions can be entered.
  • Concept of having a "WATIR buddy" – paired test programming
Brian added that Selenium was geared more toward developers and WATIR received a wider audience.

Danny Faught the context school of testing and the importance of Agile.

It is a real puzzle to the testing community as to what tools should be used. One example is WATIR versus Selenium. Potentially Selenium RC has offered an improvement. Selenium uses HTML tables and is not re-factorable. Selenium RC comes with API’s to plug into using Ruby. There was a ton of conversation around how to make open-source tools better and I unfortunately did not capture it all.

A point of clarification or realization for me was that WATIR was developed for IE only. Teams were leaning toward Selenium where multi-browser support is required. There are new projects Fire WATIR and Safari WATIR to assist with other browsers.

Another observation about this workshop format was the use of GIANT Post it pads. Concepts were written down by scribes and posted all over the room for future elaboration and discussion. This was a pretty cool way to capture the concepts.

The next portion of the meeting was very interesting. They referred to it as “Brain Storming”. This was an hour long whirlwind of problems and ideas around the topic “Costs, Limitations, and Improving Test Frameworks”.

Next experience report came from Bob Cotton of Rally. Bob showed some of the work they were doing using RSpec, which is behavior driven development versus test-driven development. He referenced Mike Cohn’s testing pyramid.

Fit or Fitnesse is typically used to test the middle layer. JUnit is typically used for the Unit Test. And WATIR for the UI portion. I got the sense that the Acceptance portion is expected to be executed without the UI and consists of direct testing of business objects. Testing should be done proportionally with the most being Unit Testing, less Acceptance and even less UI. The other interesting comment was that he drew a tiny circle above the period indicating the amount of manual testing that should take place.

Bob talked about an approach where the things you do not have time to do should be doe first. How are you going to test should be the focus and not how are you going to build. He talked about doing functional test and acceptance testing through the GUI via a layer of abstraction.

He used Ruby to build a set of new methods that would be leveraged in Selenium (i.e. – attr_reader and attr_writer). Test suites then run nightly and they typically take about 3 hours.

He talked about behavior-driven development using RSpec which followed the pattern of state the setup/context (state of the world), specify (action), and teardown/end (close the test).

His code was based on Ruby (which out of the box has an XML parser), Selenium RC, RSpec, Test::Unit, and proprietary code.

There agile approach is to understand the story, then determine how to test the story.

  • What are the acceptance tests?
  • Unit tests should be XYZ
  • Functional tests should be ABC
During open season a question was asked why did the team throw away several frameworks. They first used HTTPUnit and could not efficiently use it because it could not do AJAX type stuff. They tried something called JIFFY, which crashed frequently and provided false positives. They finally settled on Selenium and planned to extend the functionality to meet the needs of the team.

I really liked structuring the test cases with the RSpec format: set up, do, teardown format.

context "BDD framework" do
setup do
@bdd_framework =

specify "should be adopted quickly" do

specify "should be intuitive" do

Fit has the ability to test the acceptance layer of the Testing Pyramid. What would replace Fit?

New tools need to be developed to exercise the middle layer (e.g. – JRuby). Brian had some comments. He stated that he agrees that Domain Specific Language (DSL) is important. DSL is a programming language designed to be useful for a specific set of tBulleted Listasks. DSLs focus on doing one sort of task well (e.g. – Ruby for Test Automation).

Brian discussed Business facing tests.
  • Tests of Business logic
  • Workflows through the system
  • Presentation tests - UI- How does it look to a person?

A Fit table does this by describing the way the system works. He proposed pick up the look and feel of a page and turn it into an automated test using Ruby. He gave a brief demonstration of code and it was a very unique way to test page views.

At this point it occurred to me how important it would be to test PowerSeller in IE7 before our clients ask for it. I set this as an action item for myself.

Next another interesting format to this workshop took place called “Open Spaces”. Basically everyone in the room could toss out ideas of things they wanted more information on or things they wanted to present to those interested. Many topics are place on cards and taped to the wall. After the topic session was complete someone would grab a topic and head to the corner of a room. Essentially people would vote with there feet and go to the topic they were interested in.

I chose to participate in the discussion of creating a Common Browser driver that supported both WATIR and Selenium. The other discussions in the room focused on What it means to be a Tool Smith and a Custom JavaScript project.

Bret Pettichord and Bob Cotton did most of the work and discussion on the common driver. The group split into two groups when Paul Rogers decided to focus on his new approach to test automation called RUMBA.

RUMBA in a nutshell is a framework that places a tiny piece of flash code on every web page, then flash is used as the testing framework. Unfortunately I do not understand the specifics, but I think it has great potential.

Developing the common driver was an interesting topic. From my perspective I wondered why bother since other WATIR projects are moving ahead full steam to test other browsers. The collaboration between Bret and Bob was fun to watch. My take away was a style of coding and two books were recommended, Ruby Cookbook and Programming Ruby. As they were collaborating, Brain Marick stopped by the group and suggested the name for the project, Mineral WATIR.

Miscellaneous tools were mentioned:

SYSTIR – I believe this is a domain specific language using Ruby

WATIR::Simple – I believe this is also intended as a DSL.

Tclockexe – Can change the format of your machine clock. Good for dating screen shots. – Allows you to send a snapshot of your desktop directly to a printer. One advantage would be if your machine freezes and you want to show IT the state.

XPlanner – open source project management tool

We received a summary report of the Tool smith discussion.

Someone mentioned we should read an article by Eric Raymond. Look to works by James Bach on the definition of a good test. Also check out an innovative company called Atomic Object. Carl Erickson and David Crosby were at the AWTA.

Next was another really cool format used during the workshop, “Lightning Talks”. Basic format is that anyone can speak, but they only have 5 minutes to make their point.

Chris McMahon took the lead with his talk on Better Marketing for Open Source tools. High energy pitch on how software should be represented today. Software development is not a factory and not as described by structuralists. Software is not engineering or manufacturing. Software involves unity, complexity, and variety. Today software is a craft and a discipline. Using the language of art, music, and culture is the best way to talk about software.

Al Snow then talk about “User Groups Internal to a Company” introducing FIT at his company. They formed internal user groups and held routine meet ups to discuss FIT. The important take away for me was to get others involved. Al was able to collaborate with other users of FIT within his organization.

Rick Hower talked about How to Find Useful Information about a Tool. Today it is difficult to choose the correct testing tool. Do you do thorough research or do you just go for it and pick one. There are 400 – 500 different tools to choose from. In summary the documentation needs to improve and there needs to be more comparisons across tools. Tools should clearly list their strengths and even their weaknesses

A possible tool for testing web services can be found at

Danny Faught spoke on Tool Review and the OpenSTA exceptions. Is an open source project healthy? He talked about looking at activity on user forums, number of downloads or the number of releases for a product. However even those statistics could be misleading. One example was SCLC which is a valuable tool, but really has had no active development since 2003. Another one mentioned was MWSnap. He talked about the health of JUnit. He ran out of time to contrast OpenSTA. I spoke with him later and he stated that on the surface it appears to be a healthy product, but apparently the tool is not maturing and contains complex defects.

Other possible open source load testing tools for consideration are JMeter and TheGrinder

Martin Taylor illustrated his testing framework that interacts with test director called EPDAV (Environment Preconditions Data Actions Verification. This is a proprietary framework and Martin is attempting to make it open source.

David Whitesell gave a lightning talk on Distributed Execution Monitoring and Resource Management. He hated Winrunner because he could not use his machine while winrunner was executing and he could not run multiple browsers. David switched to selenium and uses a Java based structure to drive and monitor distributed test executions.

Jeff Fry demonstrated some really cool WATIR code. The code essentially generated methods on the fly as it conducted google searches. The code was used to reproduce a random defect in google search algorithm that he discovered.

Jim Mathews (from Austin) gave a talk on a Fitnesse implementation. He quickly illustrated how good testing is fostered by good architecture. He could easily structure Fitnesse to execute object level tests.

Chris McMahon gave another talk, but I did not take good notes. The only thing I noted was that I needed to get a copy of a Better Software article from August 2006 called Old School Meets New Wave.

Martin has a follow up presentation on how EPDAV was leveraged as a self load balancing client. It uses a combination of python and XML.

Al Snow expressed his thoughts on developing a Hybrid framework or putting the glue between all the testing pieces.

David Crosby presented System-Test Driven Development be leveraged at Atomic Object. He stressed the importance of starting with the simplest thing that could possible fail. He sighted an approach called 37 signals, which I need to learn more about.

Brian Marick gave presentation and my two brain cells have no clue what it was about. This could have been his position on how the latest version of Fit broke his new testing framework.

Elisabeth Hendrickson demonstrated here JavaScript site called WebStickies and how to test it using JSUnit and Selenium “Square head” addition.

It will be interesting to see if Easy Test can handle testing an application such as this (Web 2.0).

Brian Marick then gave an experience report with Fit. Fit development was led by Ward Cunningham who made the first Wiki. There were three types of ways to execute Fit: Column based data, Row based data, and Action based data.

SageCRM is a workflow tool. It was tossed out in conversation and I wanted to investigate it. I am not sure if it is even open source. Someone also mentioned tools from Green Pepper and thought they were similar t Fitnesse.

We had an experience report from the founders of WATIR, Brett Pettichord, Paul Rogers, and Charley Baker (new Project Manager). The project started out as IE Controller and the foundation was the test harness. WATIR was only intended to be a library. There was discussion on how to handle the growth of an open source project, manage defects, and manage contributions. It was pointed out that you rarely have the opportunity to hear the story first hand from all of the original stake holders.

In my opinion the project has been successful, because the owners insist on having unit tests for all methods.

In the Open Spaces section I had the opportunity to discuss Chris McMahon’s upcoming article in Better Software, Power of Three – A Trio of Techniques for Testing Databases. Article includes some excellent approaches using Ruby to validate data in a database. Code needs to require ODBC, which are standard drivers in Ruby.

I then looked over the shoulders of Elisabeth and Paul as they worked on test automation for the WebStickies site. My take away was a sound approach to troubleshooting and organizing code. I still do not fully understand JSUnit and Selenium, but it was cool to see the tools in action.

Domain Specific language for the web is Ruby on Rails and for system testing it is WATIR.

Buy the “Everyday Scripting in Ruby” by Brian Marick

Carl and David from Atomic object talked about the concept of Developer Testers and Tester Developers. Chris McMahon offers his perspective on his blog.

We went into a final session of Lightning Talks.

Paul Rogers – Demonstrated his work on finding all objects on a page using Ruby.

I need to ask if this code is available. It is similar to our “print pretty” method for tables.

Bret Pettichord – Demonstrated how he warps time using Ruby.

Pretty cool code if you need to test timeouts and things of that nature and do not want to wait.

Martin Taylor – Discussed how he re-factored his framework to not be tightly coupled to Test Director and has the potential to integrate with other applications including open source.

Bret Pettichord – demonstrated direct interaction with Ruby and Excel. As the tests executed you could visually see the cells in Excel change color based on the result.

I would love to understand how to do this. This would probably be my Mt. Everest.

Paul Rogers – demonstration on why relying on spreadsheets might not be the best solution. Data-driven testing can become a maintenance issue as the number of spreadsheets and test cases increase. His team has 500 spreadsheets and 3500 test cases. One change in the application could result in changing numerous spreadsheets. The challenges are how do you know which spreadsheets are impacted and how do you quickly change the data.

Brian Marick – Follow up demonstration on how to use the Ruby library to change time. Mentioned that he had an example where he used Ruby to redefine plus to be minus.

Andy Tinkham – mentioned that he was working on his thesis. The objective was to build a testing tool kit that could live on an external drive and be carried anywhere by a Tool smith.

Here are my final notes. I need to read, Enterprise Integration Patterns by Hohpe. Check out StoryTeller (note there may be a better link). And in the near future I should check out for a Java controlled testing framework.

Here are some key perspectives and action items.

It was unbelievable to be submersed in the conversation with brilliant open-source personalities who were purpose-driven. The group openly shared there thoughts and opinions while respecting everyone. There was laughter and subtle humor throughout the workshop. The atmosphere was kept light but educational. I left the workshop each evening with my two brain cells going full steam ahead. I was surprised I even got any sleep. There is a wealth of knowledge out there regarding test automation and it is there for the taking. I now find myself every night reading articles, projects, and yes even reviewing code.

Open Source
My action items are to become more involved in the WATIR community, specifically assisting with defect management. Eventually reviewing the methods and assisting with the RDocs. I am determined to focus on learning Ruby and WATIR. I plan to re-factor all of our current WATIR code. I see many flaws in they manner in which I implemented it.

Current Project
I am enlightened as to how little testing we actually do on our customization releases and I am determined to find ways to be smart and improve our testing practices. Our team is not actively engaged in developing unit tests. I hope the team will embrace Unit Tests and extend our test coverage. I think we have been extremely lucky that we have not had any major defects discovered by the client in the past 6 or 7 releases. I want to find ways to test the business objects without the UI and also validate the database.

Corporate Community
I am challenged to get involved on a corporate level with a few things. Reach out to other teams and understand their testing practices. Learn more complicated aspects of our internal tool, Easy Test. How does Easy Test compare to WATIR. I would also like to organize a meeting of testers within our company and hold a brief workshop similar to AWTA. Definitely a challenge but something I plan to explore.