Design of Selenium tests for ASP.NET: Finding the source of erratic bugs

After week of running UI tests as a part of continuous integration build without modifications in SUT code I realized that significant part of tests runs is failed! I know that UI tests are erratic by their nature, because they have a lot of dependencies, which you can’t control, but 40% of failures is too much. Also, when i run this tests on local without Selenium Grid cluster, all tests always pass.

This needs to be fixed. Lets check error messages:

1) Test CheckNameOnLoginPage: ERROR: Element xpath=/html/body/form[@id='form1']/div[3]/table/tbody/tr[2]/td[2]/input[@id='txtUser'] not found

2) Test CheckCurrentUserName: System.Exception: Error while executing command AssertUserName ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Selenium.SeleniumException: Timed out after 20000ms

3) Test NewlyAddedUserAppearsInUsersList: Selenium.SeleniumException: Timed out after 20000ms

Notice that always appears in random tests and always connected to absence elements. I observed test runs and found out that sometimes IIS error page with message “403.9 Access Forbidden Too many users are connected” appears in browser.

Here is it. IIS in windows of “professional” version has a limit of 10 users and therefore sometimes whole page is not loaded or just several JavaScript files and this breaks whole client side application. Here is the IIS log:

08:51:30 10.6.9.180 POST /SampleApplication/Login.aspx 403
08:51:30 10.6.9.180 GET /SampleApplication/Login.aspx 403

….

15:10:57 10.6.24.72 POST /SampleApplication/Login.aspx 302
15:10:57 10.6.24.72 POST /SampleApplication/Login.aspx 302
15:10:57 10.6.24.72 GET /SampleApplication/Home.aspx 200
15:10:57 10.6.24.72 GET /SampleApplication/Home.aspx 200
15:10:57 10.6.24.72 GET /SampleApplication/js/Frameworks/jquery-1.3.1.js 200
…
15:10:57 10.6.24.72 GET /SampleApplication/js/Services/Services.js 403

…

It is only needed to upgrade to IIS 7 to fix it. But i already spent a lot of time (about 5 hours) to find out what happened and do not want to deal with this problem in future.

First let’s learn tests to recognize IIS error pages:

private static void AssertErrorPage<TT>(TT target) where TT : PageBase, new()
     {
         var bodyText = target.Selenium.GetBodyText();

         if (bodyText.Contains("Server Error in "))
         {
             Assert.Fail("Server error while navigating\r\n\r\n {0}.", bodyText);
         }

         if (bodyText.Contains("Internet Information Services") && bodyText.Contains("Microsoft Support"))
         {
             Assert.Fail("IIS error while navigating\r\n\r\n {0}.", bodyText);
         }
     }

And include iis log analysis in continuous report. LogParser application can be used for this purposes – it supports SQL – like queries to logs. Here is sql to select log items from time to time into xml:

SELECT *
FROM %logdir%\ex%today%.log
TO %reportsdir%\extracted_iis_log.xml
WHERE TO_TIME(time) BETWEEN TIMESTAMP('%starttime%', 'hh:mm:ss') AND TIMESTAMP('%endtime%', 'hh:mm:ss') 

All details how to run logparser in nant script can be found here: http://code.google.com/p/design-of-selenium-tests-for-asp-net/source/browse/trunk/_build/scripts/continuous/main.build

Xsl to integrate iis logs in CCNET report is here: http://code.google.com/p/design-of-selenium-tests-for-asp-net/source/browse/trunk/_build/ccnet/iisloganalyser.xsl

After this modifications we can review reports for bugs mentioned above:

An assertion failed.

IIS error while navigating

The page cannot be displayed There are too many people accessing the Web site at this time.

Please try the following:

Click the Refresh button, or try again later.

Open the epbyminw0115.minsk.epam.com home page, and then look for links to the information you want. HTTP 403.9 - Access Forbidden: Too many users are connected

Internet Information Services

Technical Information (for support personnel)

Background:

This error can occur if the Web server is busy and cannot process your request due to heavy traffic.

More information:
Microsoft Support.

And IIS logs are saved for us right in the CCNET report:

Now it is possible to identify where the problem is without deep investigation – all tips and keys to solution are available in report, and if problem will appear in future – it most likely will be resolved with low efforts.

So, the moral is: if you experienced the problem in SUT or tests, first learn your tests to find and alarm about this problem, and only after that fix it. This will help you and your colleagues in future.

Actually here we started to implement analysis of how system behaves in run time, and this is only beginning. In future posts i want to describe now to get statistics from memory dumps and performance probes. This will give developers priceless statistics which will help to understand problems with concurrency, performance troubles, memory leaks, “hard to reproduce” bugs.

All sources are available in design-of-selenium-tests-for-asp-net solution.

How to run selenium tests without interfering with desktop in Windows

It is very uncomfortably to run UI tests in parallel with other work. It shows browser window over you working window, steals focus moves the mouse,  and you can affect test results by closing one of the testing windows.

The way out – is to use “Desktops” tool from SysInternals. Now you can run tests in one desktop and work in another:

This is also good option for creation of Selenium Grid cluster with developer computers as remote control runners. Tests do not load memory and processor, but they simply obstruct work by interfering with desktop. “Desktops” tool can resolve this problem and allow you to create big and cheap Selenium Grid cluster.

Design of Selenium tests for ASP.NET: Running UI tests as a part of continuous integration.

To keep UI tests up up to date it is needed to maintain them continuously. Build should be verified against UI tests every day and if something is broken – team should know what happened immediately. Only with such strategy give a chance for UI tests to survive. Just think, you are about to release build, it is late afternoon, it is needed to verify this build by QA and you realize that some of tests are broken (worth – if it broken by with strange errors). Will you fix this tests? NO. And another situation – you have just committed your changes and some automated system sends you e-mail that one use case is broken exactly after your commit. Will you fix it? Yep - it is very easy to find source of the problem.

Given all this, it is have no sense to have UI tests without automated continuous integration. And lucky we are it is possible to set up this procedure with only open source tools.

1. Scheduler and viewer of builds - Cruise Control .NET
2. Console client for Subversion
3. Nant as build system
4. Gallio as test runner
5. PsExec tool to open browsers on remote computers.

Here are the solutions to main issues of running Selenium Grid on CCNET:

1) CCNET is that its worker process is implemented as windows service. Browsers just can't start without desktop. This can be resolved by PsExec tool, which can run processes in desktop session of particular user.

2) CCNET service should be runned under user, which have access to all servers, where remote controls will be executed.

3) PsExec shows window with terms of use first time, so exec it manually before using it with service.

The example of CCNet config file can be found here. And all builds scripts which download latest version of sources, adjust configuration files, build application and tests binaries, starts selenium grid on server and slave computers and run tests are here.

Build in Sample application starts every time after code commit. But if your tests take more then 10 minutes to run – divide your tests into 2 tests suits – one for checking after commit, and one for checking every night – this still will keep your tests up to date.

All sources are available in design-of-selenium-tests-for-asp-net solution.

How to run Selenium Grid hub and remote control through nant or command line

On the official site of Selenium Grid there is no description how to run it without ant. But download this tool just to start grid is not very convenient, so i wrote tasks for nant and found the way to run Grid from command line:

First define properties:

      <property name="selenium.server.file" value="${src.dir}\_tools\selenium\selenium-server.jar" />
    <property name="selenium.grid.hub.file" value="${src.dir}\_tools\selenium\selenium-grid-hub-standalone-1.0.4.jar" />
    <property name="selenium.grid.rc.file" value="${src.dir}\_tools\selenium\selenium-grid-remote-control-standalone-1.0.4.jar" />

task to start hub:

     <target name="start.selenium.grid.hub">
      <exec program="java" verbose="true" failonerror="false">
        <arg value="-jar" />
        <arg value="${selenium.grid.hub.file}" />
      </exec>
    </target>

task to start remote control:

    <target name="start.selenium.grid.rc">
      <exec program="java" verbose="true" failonerror="false">
        <arg value="-classpath" />
        <arg value="${selenium.server.file};${selenium.grid.rc.file}" />
        <arg value="com.thoughtworks.selenium.grid.remotecontrol.SelfRegisteringRemoteControlLauncher" />
      </exec>
    </target>

Or simply from command line:

Hub:

java -jar D:\work\SeleniumDesign\build_artifacts\artifacts\continuous\source\_tools\selenium\selenium-grid-hub-standalone-1.0.4.jar

Remote Control:

java -classpath D:\work\SeleniumDesign\build_artifacts\artifacts\continuous\source\_tools\selenium\selenium-server.jar;D:\work\SeleniumDesign\build_artifacts\artifacts\continuous\source\_tools\selenium\selenium-grid-remote-control-standalone-1.0.4.jar com.thoughtworks.selenium.grid.remotecontrol.SelfRegisteringRemoteControlLauncher

Examples of build script which run selenium grid can be found here: http://code.google.com/p/design-of-selenium-tests-for-asp-net/source/browse/trunk/_build/scripts/continuous/main.build

Design of Selenium tests for ASP.NET: Running Web Tests in parallel using Selenium GRID

One of the biggest problems of Web tests is testing duration. On one of my projects we had fully automated acceptance test suite written on Watin, and it takes more then 6 hours to run it… It was damn frustrating to see night build failed – because it is needed to fix it and run it again and wait for 6 hours again.

The faster smoke test, then you will know about any problems with your application earlier, ideally after each change. In this case it is possible to locate problem in minimal time (it is easier to find problem in one small changeset, but not very easy in daily changeset, which was done by whole team).

Selenium already has a system, which can dramatically speed up web tests – Selenium Grid:

An efficient development cycle requires prompt and reliable feedback. Stop waiting hours to get the results of your web acceptance builds! Selenium Grid transparently distribute your tests on multiple machines so that you can run your tests in parallel, cutting down the time required for running in-browser test suites. This will dramatically speeds up in-browser web testing, giving you quick and accurate feedback you can rely on to improve your web application.

This is amazing tool – i just dreamed to have such system for Watin, luckily we have it for Selenium!

The idea of Grid is simple – it mocks usual Selenium Server interface, but runs a lot of Selenium Server instances behind. This means that tests, which was designed for usual Selenium will be executed on Grid without changes.

To install Grid and run it follow instructions from here. Run hub and two remote controls:

ant.bat launch-hub

ant.bat launch-remote-control -Dport 5555

ant.bat launch-remote-control -Dport 5556

Console will be available by link http://localhost:4444/console:

That is it. Now simply run your test bunch and you will get the result. Notice that tests written on c# are executed one after another.

To gain benefits of parallel executing, all test runners must pass command to Grid in parallel. Unfortunately nunit runner can not run tests written on c# in parallel. This feature has complex PNUnit runner which it is very hard to setup and maintain – this is big disadvantage. From the other side MbUnit framework has simple Parallelizable attribute which gives us exactly what is needed – it runs tests in parallel.

Place next attributes in AssemblyInfo.cs and all tests in this assembly become parallel:

[assembly: DegreeOfParallelism(10)]
[assembly: Parallelizable(TestScope.All)]

The number in “DegreeOfParallelism” attribute shows how many threads will be executed.

The difference between non-parallel run:

And parallel run with 3 remote controls:

It is about 40 % (50s vs 30s) of reduced time. Having 3 servers (or just regular developer workstations) in Grid will cut running time to 10 seconds.

All sources are available in design-of-selenium-tests-for-asp-net sample solution.

Design of Selenium tests for ASP.NET: DSL (Domain Specific Language) for Acceptance Web Tests

A large amount of people (testers, managers and even customers) can contribute Test Cases in acceptance test suite for your project, but not all do have Visual Studio to write and compile pure unit tests like this:

     [Test]
    public void LoginSuccessWithCorrectPassword()
    {
        Start
            .GoToLoginPage()
            .EnterCredentials("admin", "god")
            .Login();
    }

To allow them to create tests it is needed to invent some kind of DSL in the simple to edit format – TXT for example. But also it is needed to be able run these test as regular nunit tests. TestCaseSource attribute is exactly what is needed – it will dynamically create test case for each TXT file in folder.

   [TestFixture]
public class ScriptTest : TestBase
{
    [Test, TestCaseSource("TestCases")]
    public void DivideTest(CommandSet commandSet)
    {
        object currentFlow = Start;

        foreach (var command in commandSet.Commands)
        {

            var method = currentFlow.GetType().GetMethod(command.Name);

            try
            {
                var parameters = command.Parameters.ToArray();

                if(parameters.Length == 0)
                {
                    parameters = null;
                }

                currentFlow = method.Invoke(currentFlow, parameters);
            }
            catch (Exception ex)
            {

                throw new Exception(string.Format("Error while executing command {0}", command.Name), ex);
            }
        
        }
    }

    public IEnumerable TestCases
    {
        get
        {
            return (from fileName in Directory.GetFiles(Configuration.ScriptsPath)
                    let contents = File.ReadAllText(fileName)
                    select new object[] {GetCommand(fileName, contents)})
                    .Cast<object>()
                    .ToList();
        }
    }

    private static CommandSet GetCommand(string fileName, string commands)
    {
        var set = new CommandSet { Name = fileName };

        var lines = commands.Split('\n');

        lines = lines.Select(l => l.Replace("\r", "")).ToArray();

        foreach (var line in lines)
        {
            var commandStrings = line.Split(' ');
       
            var name = commandStrings[0];
            var parameters = commandStrings.Skip(1);
   
            var command = new Command() { Name = name };
            command.Parameters.AddRange(parameters.Cast<object>());

            set.Commands.Add(command);
        }

        return set;
    }
}

Now just it is only needed to place TXT files in Scripts folder. Format of scripts is simple:

CheckNameOnLoginPage.txt

LoginAndGoToHomePage

AssertUserName admin

or

LoginSuccessWithCorrectPassword.txt

GoToLoginPage

EnterCredentials admin god

Login

The result in nunit runner will be like this:

Of course all Flows are needed to be implemented before usage of them in DSL files, but the advantage of such approach is that the people who will write tests do not have to know about aspects of interacting with web page – ids of controls, waiting for ajax requests, URLs of pages etc.

It would be very handful to create "help" system, which will describe all Flows with its parameters. It is easy to generate with help of Reflection.

All sources are available in design-of-selenium-tests-for-asp-net sample solution.

Design of Selenium tests for ASP.NET: Dealing with persistence

Every web site have persisted storage. It can be XML on disk, DB or Key-Value storage, whatever. This aspect brings a lot of troubles while designing Web Tests. The most common example – tests script must login in system with predefined credentials and this credentials should be in storage before test run. Or another example:

 [Test]
      public void NewlyAddedUserCanLoginInSystem()
      {
          Start
              .LoginAndGoToHomePage()
              .AddUser("TestUserForLogin", "TestPwdForLogin")
              .Logout()
              .EnterCredentials("TestUserForLogin", "TestPwdForLogin")
              .Login();
      }

This test will pass only for first time – while second run user with name “TestUserForLogin” will be already in storage and inserting procedure will fail.

To resolve this issues it is needed to:

1. Have some initial data in storage. e.g. test user
2. Clean up data before test suite run.

To perform this actions it is needed to have direct access to storage to the application, which allows to manage storage. Definitely it is not acceptable to insert this functions in system under test – “clean database” button is the grenade waiting for the monkey:)

It is needed to create small standalone admin application:

Next function will be executed when button is clicked:

 protected void ClearStorage(object sender, EventArgs e)
      {
          const string emptyStorage = @"<users><user name=""admin"" password=""god"" /></users>";
          var doc = new XmlDocument();
          doc.LoadXml(emptyStorage);
          doc.Save(new Configuration().Xml);
      }

This function is designed to clean up XML, but if storage is database it will be need to execute simple script to restore database like this:

USE [master]
GO

IF  EXISTS (SELECT name FROM sys.databases WHERE name = N'teststorage')
ALTER DATABASE [teststorage] SET  SINGLE_USER WITH ROLLBACK IMMEDIATE
GO

IF  EXISTS (SELECT name FROM sys.databases WHERE name = N'teststorage')
DROP DATABASE [teststorage]
GO

RESTORE DATABASE [teststorage] FROM  DISK = N'C:\sl\BackUp\teststorage.bak'
WITH  FILE = 1,
MOVE N'teststorage'
TO N'C:\sl\Sites\teststorage.mdf',
MOVE N'teststorage_log'
TO N'C:\sl\Sites\teststorage.ldf',
NOUNLOAD,  REPLACE,  STATS = 10
GO

Now it is only needed to click on this button before any test run. This is possible with nunit “SetupFixture” attribute.

This is the attribute that marks a class that contains the one-time setup or teardown methods for all the test fixtures under a given namespace. The class may contain at most one method marked with the SetUpAttribute and one method marked with the TearDownAttribute.

using NUnit.Framework;
using Tests.SmokeTest.Core;
using Tests.SmokeTest.PageObjects;

namespace Tests.SmokeTest
{
  [SetUpFixture]
  public class SetUpSUT
  {
      [SetUp]
      public void CleanUpStorage()
      {
          CleanUp();
      }

      public static void CleanUp()
      {
          using (var navigator = new Navigator())
          {
              navigator.Start(Configuration.StorageAdminSiteUrl);
              var storageAdminPage = navigator.Open<StorageAdminPage>();

              navigator.Navigate<StorageAdminPage>(storageAdminPage.ClickClear);
          }
      }
  }
}

Of course it is needed to create standalone Storage Sandbox for each developer, which runs web tests and for each continuous integration build to avoid interfering between tests.

Note that both SUT and Storage Admin sites should be deployed and online during the test run. It is very convenient to setup virtual folders for them in IIS, because if you use ASP.NET development server, it will run each site in instance with randomly generated port, and it is needed always fix configs for tests.

All sources are available in design-of-selenium-tests-for-asp-net sample solution.

Design of Selenium tests for ASP.NET: Testing AJAX. Part 2. Wait for JQuery

While testing sites with AJAX it is always hard to catch a moment when asynchronous request is finished. I already described function which can wait for text in label. But this approach fails when it comes to more complex behavior. For example:

On the picture above use case to add users is shown. When user clicks on add button, the system sends data to server and updates grid with users.

HomeMediator.prototype.AddUser = function(name, password)
{
  var me = this;

  this.services.AddUser(name, password,
      function (msg) {
          if(msg.d.Success)
          {
              me.BindUsers();
          }
          else
          {
              me.addUserWidget.SetError(msg.d.Message);            
          }
      });
};

HomeMediator.prototype.BindUsers = function()
{
  var me = this;

  this.services.GetAllUsers(function (users) {
        me.userListWidget.Render(users.d);
  });  
};

Function which executes asynchronous request on server is implemented using JQuery “ajax” function and luckily it has global variable “active” which is set to true until request is done. It is possible to use this variable to force tests to wait AJAX before verification.

 public void ClickAndWaitForJQuery(Action action)
      {
          action();
          WaitForJQuery();
      }

 public void WaitForJQuery()
      {
          _selenium.WaitForCondition("selenium.browserbot.getCurrentWindow().jQuery.active == 0;", Timeout);
      }

Example of usage in tests:

 public HomePageFlow ClickOnAddUser()
      {
          Navigator.ClickAndWaitForJQuery(_home.ClickAddUser);

          return this;
      }

All sources are available in design-of-selenium-tests-for-asp-netsample solution.

Design of Selenium tests for ASP.NET: Autogenerate Page Objects for Selenium tests with T4 templates in .NET. Part 2

In previous post it was described how to generate Page Objects for Selenium test on example with simple page. But what to do with more complex cases, when there are a lot of elements with identical ids or it is not possible to include information about type in id of control?

Lets consider page to manage users from design-of-selenium-tests-for-asp-net sample project:

It contains a form to insert users in system and also list of already inserted users. The main problem is to get values from table – they are not wrapped in any container and therefore do not have any ids. It is only possible to access them by XPath:

Take a look at tr elements, which are rounded by red square. It is only needed to change number next to “td” element and we will get XPath for each element in table.

So, to generate code to access table we need:

1) Xpathes to all elements in one grid:

/html/body/form[@id='form1']/div[3]/div[@id='holderUsers']/table/tbody/tr[1]/td[1]
/html/body/form[@id='form1']/div[3]/div[@id='holderUsers']/table/tbody/tr[1]/td[2]

2) Pattern to identify place where to change row number:

table/tbody/tr[{0}]/td

3) Xpath to identify count of items in grid:

/html/body/form[@id='form1']/div[3]/div[@id='holderUsers']/table/tbody/tr

4) Give meaningful names for each element in row:

/html/body/form[@id='form1']/div[3]/div[@id='holderUsers']/table/tbody/tr[1]/td[1] {Name=lblUser}
/html/body/form[@id='form1']/div[3]/div[@id='holderUsers']/table/tbody/tr[1]/td[2] {Name=lblPassword}

Now id is needed to give this information to generator:

<#@ template inherits="Microsoft.VisualStudio.TextTemplating.VSHost.ModelingTextTransformation" language="C#v3.5" debug="true" hostSpecific="true" #>
<#@ output extension=".cs" #>
<#@ include file="PageObjectGeneratorByXPath.ttinclude" #>
<#
string url = "Home.aspx";

var mainClass = new ClassInfo {
 ClassName = "Home",
 Elements =
@"/html/body/form[@id='form1']/div[3]/span[@id='lblUserName']
/html/body/form[@id='form1']/div[3]/input[@id='btnLogout']
/html/body/form[@id='form1']/div[3]/table/tbody/tr[3]/td[2]/input[@id='txtName']
/html/body/form[@id='form1']/div[3]/table/tbody/tr[4]/td[2]/input[@id='txtPassword']
/html/body/form[@id='form1']/div[3]/table/tbody/tr[5]/td/input[@id='btnAddUser']
/html/body/form[@id='form1']/div[3]/table/tbody/tr[2]/td/span[@id='lblError']"};
var rowClasses = new List<ClassRowInfo>();

rowClasses.Add(
 new ClassRowInfo {
  ClassName = "UserTable",
  CountXpath = "/html/body/form[@id='form1']/div[3]/div[@id='holderUsers']/table/tbody/tr",
  IteratorPattern = "tbody/tr[{0}]/td",
  Elements =
@"/html/body/form[@id='form1']/div[3]/div[@id='holderUsers']/table/tbody/tr[1]/td[1] {Name=lblUser}
/html/body/form[@id='form1']/div[3]/div[@id='holderUsers']/table/tbody/tr[1]/td[2] {Name=lblPassword}"
 });


#>

<#
RenderClasses( mainClass, rowClasses, url);
#>

And from this point we can access values in rows using generated Page Object in the following way:

public HomePageFlow AssertThatUserListContains(string name, string password)
   {
       var isExists = false;

       for (var i = 1; i <= _home.GetUserTableRowsCount(); i++)
       {
           var row = _home.GetUserTableRow(i);

           if(row.User.GetText() == name && row.Password.GetText() == password)
           {
               isExists = true;
           }
       }

       Assert.That(isExists, Is.EqualTo(true), string.Format("User with name '{0}' and password '{1}' doesn't exists in user list table", name, password));

       return this;
   }

All examples from article can be found in the design-of-selenium-tests-for-asp-net sample project.

Design of Selenium tests for ASP.NET: Meaningful failure messages

It is not enough to create unit/acceptance tests and automate them. It is also needed to make them help you to discover issues as fast as it possible. For example lets look at the next test results:

Can you quickly identify the problem? Yes, something is not working, control to enter user name was not found! But why?

1. May be application threw an unhandled exception and we got page with error screen?
2. Login was just unsuccessful and script stayed on login.aspx page, where there is no any controls to add user of course?
3. May be someone changed ids or markup on home.aspx and Selenium can’t find controls to enter values?
4. Script was simply redirected on some random page after login?
5. May be test is just incorrect?

Where exactly problem is?

Of course it is possible to run this test again in slow mode and see what happened, but it takes a lot of time and pretty not effective.

The better approach – identify issue in tests and display in error message. First lets handle 2 and 4 case – just check current page location after each action. We already introduced Navigator pattern for this purposes – so lets implement it:

public TT Navigate<tt>(Action action, bool chooseOkOnConfirmation) where TT : PageBase, new()
      {
          var target = new TT();
          InitPage(target);

          action();

          WaitLoad(target);

          AssertCorrectPageLoaded(target);

          return target;
      }

private void AssertCorrectPageLoaded<TT>(TT target) where TT : PageBase
      {
          var location = _selenium.GetLocation();
          var paramsStart = location.IndexOf('?');

          if (paramsStart >= 0)
          {
              location = location.Substring(0, paramsStart);
          }

          if (!location.EndsWith(target.PageUrl))
          {
              Assert.Fail("Expected URL {0} but was {1}", target.PageUrl, location);
          }
      }

Now the same test run will give us following results:

Aha! Much better! This is definitely second case. But it is still not clear what happened? Let’s look for standard ASP.NET error message and display it:

 public TT Navigate<TT>(Action action, bool chooseOkOnConfirmation) where TT : PageBase, new()
      {
          var target = new TT();
          InitPage(target);

          action();

          WaitLoad(target);

          if (target.Selenium.GetBodyText().Contains("Server Error in "))
          {
              Assert.Fail(String.Format("Server error while navigating\r\n\r\n {0}.", target.Selenium.GetBodyText()));
          }

          AssertCorrectPageLoaded(target);

          return target;
      }

Lets look now:

Here we go… The problem is that application can not find file with user definitions. It is needed to fix configuration file a little bit to resolve this issue. It will take no more then 1 minute to identify where the problem is, fix it and rerun test. Pretty effective, is not it?

Also do not forget to add meaningful messages to asserts, which check business cases.

Assert without message:

Assert with message:

Understanding of what happened is dramatically changed by only one simple message in assert:

 public HomePageFlow AssertThatUserListContains(string name, string password)
      {
          var isExists = false;

          for (var i = 1; i <= _home.GetUserTableRowsCount(); i++)
          {
              var row = _home.GetUserTableRow(i);

              if(row.User.GetText() == name && row.Password.GetText() == password)
              {
                  isExists = true;
              }
          }

          Assert.That(isExists, Is.EqualTo(true), string.Format("User with name '{0}' and password '{1}' doesn't exists in user list table", name, password));

          return this;
      }

All sources from this article can be found in design-of-selenium-tests-for-asp-net sample project.

Design of Selenium tests for ASP.NET: Testing AJAX

When it comes to AJAX, UI tests started to behave unexpectedly. Sometimes they fail, sometimes pass:

The main trick in testing AJAX is to wait until asynchronous request is completed. There is an universal waitForCondition function which gets JavaScript Boolean expression as argument, and waits until this expression returns true.

public HomePageFlow AssertUserName(string userName)
{
   Navigator.WaitForText(_home.UserName.Selector);

   Assert.That(_home.UserName.GetText(), Is.EqualTo(userName));

   return this;
}

public void WaitForText(string selector)
 {
     selector = selector.Replace(@"'", @"\'");

     string script = string.Format("var value = selenium.getText('{0}'); value.length > 0;", selector);

     _selenium.WaitForCondition(script, Timeout);
 }

It was needed to change only Flows to tell them about AJAX behavior. It is not needed to change any tests, and this is reasonable – because actually nothing was changed from user and UI tests prospective in UI. This means that correct level of abstraction was selected.

using NUnit.Framework;
using Tests.SmokeTest.Core;

namespace Tests.SmokeTest.Tests
{
[TestFixture]
public class HomeTest : TestBase
{
[Test]
public void CheckCurrentUserName()
{
   Start
       .LoginAndGoToHomePage()
       .AssertUserName("admin");
}
}
}

All sources are available in design-of-selenium-tests-for-asp-net sample solution.

Integrate Selenium server in Visual studio

It is very useful to run selenium in Visual Studio. All settings are on the picture below:



1) Open Visual Studio
2) On the Tools menu, choose External Tools.
3) In the External Tools dialog box, choose Add, and enter a name "Selenium Server" in the Title box.
4) In the Command box, enter the path to the Java Runtime. For example "C:\Program Files\Java\jre6\bin\java.exe"
5) In the Arguments box, enter arguments in the following format - "-jar selenium-server.jar".
6) In the Initial directory, enter "$(SolutionDir)\_tools\selenium"
7) Select Use output window and then choose OK.

Place selenium-server.jar file in $(SolutionDir)\_tools\selenium directory, commit it in SVN, so every developer will have the same version of Selenium as you.

Now server can be started/stopped through "Tools" menu.

Design of Selenium tests for ASP.NET: Introducing Flow pattern

While acceptance testing through UI it is impossible to test only one aspect of behavior of system like we do usually in unit tests. Acceptance tests asserts use cases, which can be composed of a number of other use cases. One of the frequently encountered examples is login procedure. It is always needed to login first before composing e-mail, see list of incoming messages or to be able to see user specific information.

Flow pattern is intended to encapsulate one use case. It can simply fill two fields:

public LoginPageFlow EnterCredentials(string name, string password)
{
 _login.User.SetText(name);
 _login.Password.SetText(password);

 return this;
}

Or be composed from a number of another complex use cases.

public HomePageFlow LoginAndGoToHomePage()
{
 return GoToLoginPage()
  .EnterValidCredentials()
  .Login();
}

Usually it is implemented using Fluent Interface pattern – this makes consumer code much more clear. Combining different flows it is possible to test something very carefully:

using NUnit.Framework;
using Tests.SmokeTest.Core;

namespace Tests.SmokeTest.Tests
{
   [TestFixture]
   public class LoginTest : TestBase
   {
       [Test]
       public void LoginSuccessWithCorrectPassword()
       {
           Start
               .GoToLoginPage()
               .EnterCredentials("admin", "god")
               .Login();
       }

       [Test]
       public void LoginWrongWithWrongPassword()
       {
           Start
               .GoToLoginPage()
               .EnterCredentials("admin", "incorrectPwd")
               .LoginShouldFail()
               .AssertMessage("Username and password do not match.");
       }
   }
}

Or hide from reader some unimportant steps (login procedure in this case)

using NUnit.Framework;
using Tests.SmokeTest.Core;

namespace Tests.SmokeTest.Tests
{
   [TestFixture]
   public class HomeTest : TestBase
   {
       [Test]
       public void CheckCurrentUserName()
       {
           Start
               .LoginAndGoToHomePage()
               .AssertUserName("admin");
       }
   }
}

One more benefit of Flow pattern is that tests became more declarative, and now it is very easy to extract DSL (domain specific language), which can be used by tester or customer for creating new tests. But this is the topic for another post.

All sources from this post can be found in design-of-selenium-tests-for-asp-net sample project.

Design of Selenium tests for ASP.NET: Autogenerate Page Objects for Selenium tests with T4 templates in .NET

In my previous post Design of Selenium tests for ASP.NET: Page Object and Navigator patterns i described Page Object pattern which helps to improve maintainability of tests through UI. But to maintain these Page Object is still great effort… And good news, everybody! We can easily autogenerate them! How?

Firstly we need a list of all important controls on page. Usually they are inputs and links, but also spans, headers, lists… whatever. Plug-in Xpather for Firefox can easily find them all). Just type “//input | //span” in “XPath” textbox, select needed controls and copy list of XPaths from “XPaths” tab:

You will get list like this:

/html/body/form[@id='form1']/div[3]/table/tbody/tr[1]/td/span[@id='lblMessage']
/html/body/form[@id='form1']/div[3]/table/tbody/tr[2]/td[2]/input[@id='txtUser']
/html/body/form[@id='form1']/div[3]/table/tbody/tr[3]/td[2]/input[@id='txtPassword']
/html/body/form[@id='form1']/div[3]/table/tbody/tr[4]/td/input[@id='btnLogin']

Look at these XPathes – it is possible to extract all information that needed to identify type and name of each control. Also, as you can see i added prefixes to control names in order to recognize whether it is button or textbox. If yours style guidelines are prohibit such names, you can find another workaround – for example add some attributes manually after gaining them from XPather.

The best way to autogenerate code in Visual Studio is T4 templates. It is needed to create simple transformation and insert our Xpathes in it:

<#@ template inherits="Microsoft.VisualStudio.TextTemplating.VSHost.ModelingTextTransformation" language="C#v3.5" debug="true" hostSpecific="true" #>
<#@ output extension=".cs" #>
<#@ include file="PageObjectGeneratorByXPath.ttinclude" #>
<#
string url = "Login.aspx";

var mainClass = new ClassInfo {
ClassName = "Login",
Elements =
@"/html/body/form[@id='form1']/div[3]/table/tbody/tr[1]/td/span[@id='lblMessage']
/html/body/form[@id='form1']/div[3]/table/tbody/tr[2]/td[2]/input[@id='txtUser']
/html/body/form[@id='form1']/div[3]/table/tbody/tr[3]/td[2]/input[@id='txtPassword']
/html/body/form[@id='form1']/div[3]/table/tbody/tr[4]/td/input[@id='btnLogin']"};
var rowClasses = new List<ClassRowInfo>();
#>

<#
RenderClasses( mainClass, rowClasses, url);
#>

Implementation of RenderClasses function can be found in design-of-selenium-tests-for-asp-net sample solution in LoginPage Page Object template.

This template will autogenerate next ready to use code:

using Tests.SmokeTest.Core;
using Tests.SmokeTest.PageObjects.Controls;
using Selenium;

namespace Tests.SmokeTest.PageObjects
{
public partial class LoginPage : PageBase
{
    public LoginPage() : base("Login.aspx") { }

    private Label _message = null;
    public Label Message
    {
        get
        {
            if (_message == null)
            {
                _message = new Label(Selenium, @"xpath=/html/body/form[@id='form1']/div[3]/table/tbody/tr[1]/td/span[@id='lblMessage']");
            }
            return _message;
        }
    }
    private TextField _user = null;
    public TextField User
    {
        get
        {
            if (_user == null)
            {
                _user = new TextField(Selenium, @"xpath=/html/body/form[@id='form1']/div[3]/table/tbody/tr[2]/td[2]/input[@id='txtUser']");
            }
            return _user;
        }
    }
    private TextField _password = null;
    public TextField Password
    {
        get
        {
            if (_password == null)
            {
                _password = new TextField(Selenium, @"xpath=/html/body/form[@id='form1']/div[3]/table/tbody/tr[3]/td[2]/input[@id='txtPassword']");
            }
            return _password;
        }
    }
    public void ClickLogin()
    {
        Selenium.Click(@"xpath=/html/body/form[@id='form1']/div[3]/table/tbody/tr[4]/td/input[@id='btnLogin']");
    }
}
}

What benefits provides autogeneration of Page Objects? first of all it is not longer needed to painstakingly extract these ids from html and hardcode them into code. Secondly, if page is changed, then autogenerated Page Object will contain another set of properties and its types, and this will cause errors in compile time. This errors are pretty easy to fix unlike to runtime errors with hardcoded IDs or Xpathes.

To get know how code generation works for pages with grids and for controls without id please refer part 2 article.

Next post will describe how to improve readability of tests using Flow pattern.