WebTest Efficient Functional Web Testing with HtmlUnit and Beyond Marc Guillemot RuhrJUG 05.03.2009 © Marc Guillemot, 2009 Your speaker French developer exiled in Germany Independent consultant since 2002 Focus on Java, Groovy, Grails and ... web testing HtmlUnit & WebTest lead developer Committer to Groovy & NekoHTML Blog: http://mguillem.wordpress.com Site: http://www.efficient-webtesting.com © Marc Guillemot, 2009 Agenda Introduction to WebTest Test automation: a software engineering activity Extending WebTest Myths and best practices WebTest and AJAX WebTest's future(s) Conclusion Q&A © Marc Guillemot, 2009 Test automation benefits Reliable: eliminate human errors Repeatable: tests can run often without additional costs Fast: reduced testing time => deploy faster Allows to reach a better coverage Find errors when they are cheap to fix Allows better communication between developers, business owners and QA © Marc Guillemot, 2009 What is WebTest Automated web functional testing tool Simulates a browser Open Source (Apache 2 license) Founded in 2001 Currently 4 committers (Germany, Switzerland, USA, Australia) Built on top of Apache Ant http://webtest.canoo.com © Marc Guillemot, 2009 Who uses WebTest? Open Source project => no idea! Let's look at the mailing list: 760 subscribers ­ ­ ­ *@apache.org *@*.gov Akamai, Axa, Bayer, Boeing, Citigroup, Crédit-Suisse, CSC, DHL, EDS, IBM, Lexmark, Novartis, Oracle, Paypal, Sun, TMobile, Thalesgroup, UBS, Verisign, Zuehlke, ... => everybody from small to huge Browser simulation? Does this make sense? © Marc Guillemot, 2009 demo http://new.ruhrjug.de "Tests ensure that the application works" wrong! Tests allow to find errors. That's all! © Marc Guillemot, 2009 First WebTests (in XML) (demo) (in Groovy) © Marc Guillemot, 2009 Over 100 WebTest Steps General ­ ­ ­ PDF ­ ­ ­ ... ... ... Forms ­ ­ ­ ­ Excel documents ­ ­ ­ ... ... Verification ­ ­ ­ ­ Email, Applets, ... Comprehensive online documentation http://webtest.canoo.com A software engineering activity Apply rules of software engineering Remove duplications Extract modules Use source control Consider modifications and extensions ­ Ant, Java, Groovy XPath Learn XPath Learn XPath!!! ­ ­ Apply it wisely /html/body/div[2]/center/table[3]/tr/td[@id='total'] //*[@id='total'] Write HTML code allowing simple XPath expressions!!! Apply the right testing paradigm Capture / replay Model based testing Data driven testing Scripted automation © Marc Guillemot, 2009 demo WebTestRecorder Model based testing Specification: modelling expected behavior Be tolerant to accidental behavior ­ Example: bad XPath: /html/body/div[2]/center/table[3] better: //*[@id='total'] Fault model: modelling unexpected behavior demo data driven Scripted automation Maximum flexibility and power Maximum responsibility Extending WebTest © Marc Guillemot, 2009 Why? Go beyond built-in features Customize for special needs Make your tests easier to write Make your tests results easier to read © Marc Guillemot, 2009 Building blocks WebTest HtmlUnit Apache ANT ... ... Mozilla Rhino NekoHTML Jakarta commons-httpclient Apache POI Groovy © Marc Guillemot, 2009 WebTest uses HtmlUnit A "browser for Java programms" 100% pure java, headless Able to simulate FF or IE Very fast and leightweight Used in WebTest but as well in JSFUnit, WebDriver, JWebUnit, Cactus, Celerity, Schnell, ... Very good JS support... but not yet perfect © Marc Guillemot, 2009 Go beyond built-in features ... step.context.webClient.credentialsProvider.addNTLMProxyCredentials( "michael", "secret", "testserver", 12345, "BlueGene", "my.windows.domain") ..... ... © Marc Guillemot, 2009 Hack the web connection © Marc Guillemot, 2009 Myths & Best Practices © Marc Guillemot, 2009 "UI tests are brittle by nature" © Marc Guillemot, 2009 Determine what you want to test! How to test addition into the basket? ­ ­ ­ ­ clickLink "Einkaufskorb"? clickLink "Ajax" ? clickLink xpath="//*[text() = 'Top Angebot']//a" ? clickLink xpath="//*[text() = '6.60']/following-sibling::a" ? => it depends! These 4 examples don't test the same thing! © Marc Guillemot, 2009 "UI tests are brittle by nature" © Marc Guillemot, 2009 wrong! "UI tests are slow" © Marc Guillemot, 2009 Speed matters! Execution speed ­ WebTest very fast: no rendering, single process, HtmlUnit's API allows a better control HtmlUnit vs Watir ­ Celerity benchmarks: 69%-99% time reduction Schnell benchmark: 97% time reduction Report analyse speed ­ ­ Very rich reports Debugging (mostly) useless © Marc Guillemot, 2009 demo wt.parallel.nbWorkers "UI tests are slow" © Marc Guillemot, 2009 wrong! WebTest & AJAX © Marc Guillemot, 2009 JS Support: how good? HtmlUnit tries to reproduce browser behavior of the simulated browser (available: IE6, IE7, FF2 and FF3) Rhino JavaScript Engine + custom host objects Many, many, many tests to ensure correctness: ­ ­ ­ ­ tests with custom JUnit runner tests through WebDriver unit tests of AJAX libraries like JQuery, Sarissa, ... functional tests with GWT Not yet perfect © Marc Guillemot, 2009 Example of unit test in HtmlUnit © Marc Guillemot, 2009 JS support not yet perfect HtmlUnit JS support continuously improved HtmlUnit/WebTest already used in numerous projects to test AJAX applications Libraries with 100% support: GWT, JQuery, MochiKit, Sarissa, ... © Marc Guillemot, 2009 Able to find complex JS errors © Marc Guillemot, 2009 AJAX testing AJAX supported ­ ­ XMLHttpRequest supported since 2005 But AJAX = complex JS libraries AJAX = testing challenge WebTest JS support not yet perfect Simulated browser offers testing facilities © Marc Guillemot, 2009 AJAX = testing challenge click Selenium & Co' solution: click "Update" waitForElementPresent "some elt" WebTest's solution: clickButton "Update" nothing here! BUT not yet perfect - works well with XHR and setTimeout - doesn't work well with setInterval ;-( © Marc Guillemot, 2009 Testing Facility example: mashup unavailability www.myserver.com ? © Marc Guillemot, 2009 other.server.com Testing Facility example: mashup unavailability © Marc Guillemot, 2009 WebTest's future(s) © Marc Guillemot, 2009 WebTest's future(s) Integration of WebDriver Integration with JMeter for load testing ... © Marc Guillemot, 2009 Integration with WebDriver N o w WebTest HtmlUnit C o m i n g WebTest HtmlUnit WebDriver Firefox IE ... © Marc Guillemot, 2009 Conclusion © Marc Guillemot, 2009 Lessons learned Use recorder only to give you a jump start Write testable code ­ ­ write coherent html code add nodes, ids, ... to make testing easier Test early Modulize / Refactor / Reuse code Customize your test tool Distinguish guaranted from accidental behavior: ­ ­ which changes in the application should make a step fail which changes in the application should NOT make it fail Marc Guillemot, 2009 WebTest key properties simple fast excellent reporting very low TCO runs everywhere no display needed easy to extend straightforward integration doesn't accept (too) badly formatted html code js support not as good as in "normal" browser doesn't accept (too) badly formatted html code References WebTest http://webtest.canoo.com ­ Groovy http://groovy.codehaus.org/ Mailing list http://lists.canoo.com/mailman/listinfo/webtest AntBuilder ­ http://groovy.codehaus.org/Using+Ant+from WebTest screencasts Watir http://wtr.rubyforge.org/ ­ Creating a first WebTest project http://opensource.basehaus.com/webtest/screencasts/creating-a-first-webtest-project.htm Schnell http://code.google.com/p/schnell-jruby/ ­ Data driven WebTest http://opensource.basehaus.com/webtest/screencasts/data-driven-webtest.htm Celerity http://celerity.rubyforge.org/ WebTestRecorder http://webtestrecorder.canoo.com Why Selenium sucks (Mark Striebeck, Selenium User Meetup) http://br.youtube.com/watch?v=EDb8yOM3Vpw ­ HtmlUnit http://htmlunit.sf.net WebDriver http://code.google.com/p/webdriver/ JMeter http://jakarta.apache.org/jmeter/ JSFUnit http://www.jboss.org/jsfunit/ Happy testing!