Rapid TPS Results using JMeter

Performance testing can certainly be an art. Or a journey into a deep dark cavern. But sometimes there are very simple ways to get a result. Here is the simplest way ever to determine transactions per second, TPS, if you use JMeter and save your results to a .csv file.

On the command line navigate to the directory where your file exists. Sorry but this magic assumes a unix like operating system.

The magic -

type wc -l {filename}
divide this result by the duration of your test in seconds

Potential magic using Windows OS (Untested by me, but found via a Google search)

dir /b c:\temp | find /c /v "~~~"

I like this approach because it allows you to quickly evaluate changes in your results. As you are optimizing an application configuration for performance you can very quickly approximate the throughput as TPS.

Stay tuned for two future posts:

How to quickly move results to MySQL
How to use Rake tasks to manage your results

FYI - I stole this magic from the magician Scott Sims.

Happy Testing!

Rubymine Rocks

Of course I missed another post last week. Amazing how soccer, work, life in that order controls your time. Any rate I was not sure what to post this week. I have been extremely busy learning more about JMeter, BrowserMon and performance testing and creating new automated regression tests.

I decided to pass on writing about performance testing and make a couple statements as to why Rubymine ROCKS.

Number one reason Rubymine ROCKS is that when you are working with Ruby and XPath you can pause the script at the point of failure. Then you highlight the XPath code that is failing and hit F11. Now edit the day lights out of that XPath until you get the XPath to work.

Typically I have a copy of the target page up in another Firefox instance and viewing the HTML closely.

A second reason I like Rubymine is the ability to quickly see what methods are available and command-B to see what the methods are doing (is the correct term intellisense?).

Probably not the last reason I find, but when you want to refactor a term you can find it and replace it for the entire project. Most IDE do this, but I like how Rubymine does it.

I also like how Rubymine tells me gems are not attached. I like the fact that you can can place code in test mode and test one method at a time. And I also really like the fact that they fixed the rename functionality so the name actually appears and I can make minor changes versus having to type in an entire new name.

I have definitely become a fan of Rubymine. Go Horns! I mean RubyMine!

Rocking the Ruby code!

Selenium Brittleness

This will be a quick post, but I wanted to share a little tip. When using the Selenium IDE you often get xpath that looks similar to this.

browser.click "//div[@id='location-bar']/div[2]/div[4]/a"

Well when attempting to do cross-browser testing this Xpath may not work for all browsers and relying on an index value that may change is a bit brittle.

One way to adjust this Xpath would be to make it look like this.

browser.click "//div[@id='location-bar']/div[@class='this']/div[@class='that']/a"

Of course everything is dependent on how the object on the page is represented. Inspect the HTML closely.

Here are another helpful hints on clicking links.

browser.click "//a[contains(@href,'stuff')]"
or
browser.click "//a[contains(text(),'otherstuff')]"

I hope you find these pointers helpful.

Happy coding!

Using Selenium GRID

For the past 3 months I have been on a journey with a team of extremely sharp developers and test engineers to build out a test automation framework. We settled on using Selenium and the Selenium GRID.

There are so many framework options, but I have to say at this point Selenium GRID rocks.

Here are the main pieces to the puzzle.

• MacBook Pro
• Use Rubymine for development
• Now using Selenium GRID 1.04
• Code base is Ruby leveraging the use of Rake
• VMware Fusion
• Team City
• Rally

Locally we can now develop a script and execute it against Safari on MacOSX, Firefox 3.0, Firefox 3.5, IE 6.0, IE 7.0 & IE 8.0. We can execute tests in parallel. I love watching 6 - 10 firefox icons bouncing up and down on my MacBook.

Tests are launched by the test teams in production environment using a job configured in Team City. One click of a button kicks of the test suite. Soon teams can choose to run only smoke test, only regression tests, or all. In production the tests are executed in parallel for each browser configured, which us currently IE 7.0, IE 8.0, Safari on Mac Mini, Firefox 3.0 and Firefox 3.5. After the test execution the test leads can see results input directly into Rally test cases or they can view a nice HTML report. They can also view the results in Team City.

If you cannot tell, I love this stuff! It is really fun to develop in this environment because you can see immediate cross browser results.

Yes I still struggle as a developer and create some brittle tests. But the good news is collaborating with a great team we quickly re-factor brittle code.

Our automation targets a complex and ever changing environment, but having a dedicated automation team is the key to success.

Happy Testing!

JMeter to the Rescue

Recently I was involved in a high priority performance testing effort. The tools at hand were Webload and WAPT. I have previous experience with Webload so that is the initial path we took. The testing effort is relatively simple. The goal was to have concurrent users doing a series of gets and posts. Clarification the posts are actually http puts. I quickly could not find a way to get Webload to emulate an http put. Further complicating things was the system under test, SUT, requires secure header keys. Information on the internet on Webload seemed very limited and I was not able to rapidly find the answers to these two concerns.

The time line was limited so there was not bandwidth for a coding effort nor to purchase any new tool.

We quickly turned to JMeter. This is going to be a short post, but there are a couple of key advantages we had with JMeter.

Observed advantages:

• JMeter out of the box could handle puts
• JMeter worked well with RAKE
• Fast installation and short learning curve
• Easily extensible
• Ability to run headless (no GUI) via RAKE

Observed dis-advantantages:

• Executing tests with the GUI dropped threads as load increased due to utilization of machine java processes
• graphic listeners are of extremely poor quality
• A single instance appears to be limited to approximately 300 concurrent threads
  

The bottom line to this post is that JMeter came to the rescue and has sound potential for future performance projects.

There are some key action items for me.

• Evaluate the ability to distribute load across several machines
• Evaluate the ability of JMeter to interact with Web-based applications (ajax stuff etc...)
• Develop Ruby scripts to calculate and report out data metrics as the number of transactions increase.
• Understand the limitations of this tool
• Evaluate other open source plugins developed for JMeter

I am looking forward to working more closely with the development teams to expand the use of jMeter for performance testing.

Happy Testing!

Selenium RC plus RSpec

It has been an extremely long time since I posted to this humble blog. I have currently changed jobs and I am newly energized to learn, implement, and build out a test automation framework.

There are a zillion frameworks to choose from. I have been using the WATIR framework for several years as I had bandwidth. I never had the opportunity to learn and execute a full solution. I recently interviewed several teams at the new company and I found one common theme with respect to test automation. Every team has selected Selenium RC as the foundation.

Initially I attempted WATIR to execute the tasks required by the team. With the complexity of the web pages and solutions, I quickly learned that WATIR may not be the best solution.

One of the teams built a Selenium & RSpec framework. Although there is some learning curve for me, I am liking what I see.

As I go on this adventure I will attempt to do a weekly post on the good, the bad, and the ugly.

Keep on Testing!