Moles from Microsoft. Just another stubbing framework? Not really…

Recently Microsoft research lab announced Moles stubbing framework that comes tightly with Pex (http://research.microsoft.com/en-us/projects/pex/). I really do not see profits in using Pex on real projects, but Moles is quite interesting.

Features of Moles:
1) It can create all types of test doubles except of mocks. Personally I do not accept mockist style of unit testing, so I consider absence of mocks as advantage.

2) It uses delegates to define behavior of test doubles. Like Moq does. Sweet :)

var fileSystem = new SIFileSystem()
{
    ReadAllTextString = filename => 
        { 
            Assert.AreEqual(fileName, file); 
            return content;
        }
};

3) It uses pre-compile time code generation to create classes for doubles.

4) It can be used with any testing framework, like NUnit, MBUnit, xUnit. Of course it does – in other case nobody will use it!

5) And the last and most significant feature – it can replace any class with test double, even static one! Before this moment only one framework was to able do it – typemock for just 800 $ per license. They call such test doubles “Moles” and mole for DateTime.Now will look like:

MDateTime.NowGet = () => new DateTime(2010, 1, 20);

Finally covering legacy code with unit tests will be not so painful :)

Links for farther reading:
http://angler.wordpress.com/2010/01/21/pex-and-moles-untestable-code-not-really
http://research.microsoft.com/en-us/projects/pex/documentation.aspx
http://research.microsoft.com/en-us/projects/pex/molestutorial.docx
http://research.microsoft.com/en-us/projects/pex/molesmanual.docx


Update: Answer to question "Is Pex (Test generation) really usefull tool?" is here http://stackoverflow.com/questions/2704669/is-pex-test-generation-really-usefull-tool

Solutions for problems of automated testing Web applications through UI

Here is the list of general problems of testing Web applications through UI:

• Maintainability
• Hard to create for non-programmers
• Hard to handle AJAX
• Hard to handle persistence
• Erratic tests
• Testing takes too much time
• Hard to understand test flow
• Hard to understand why test failed

Project design-of-selenium-tests-for-asp-net was created to show how to create automated UI tests for ASP.NET application using Selenium and minify influence of these problems. Description of how each particular issue can be avoided:

Problem: Maintainability

• Abstraction layers (Page Object, Flow)
• Code generation - Part 1 and Part 2
• Continuous integration
• Processes (Do not allow commit when build fails)

Problem: Hard to create for non-programmers

• Domain Specific Language and lightweight editor/runner

Problem: Hard to handle AJAX

• Just tricky technical problem - Part 1 and Part 2

Problem: Hard to handle persistence

• Fresh test fixture
• Use unique data for each test

Problem: Erratic tests

• Continuous integration
• Processes (track such tests, make problem visible and avoid it)

Problem: Testing takes too much time

• Run tests in parallel and distribute tests on several machines

Problem: Hard to understand test flow

• Abstraction layers (Page Object, Flow)
• Think about future generations :)

Problem: Hard to understand why test failed

• Special patterns (Navigator)
• Meaningful messages
• Run time analysis 

Nant task to calculate GMT or UTC timestamp

I was surprised when i realized that standard implementation of <tstamp> task do not support getting time in GMT time zone. I have not found any custom extensions for nant, which will provide such functionality so i was forced to write my own custom task. Fortunately it is really easy:

    <script language="C#" prefix="test" >
 <code>
   <![CDATA[
  [TaskName("gmttimestamp")]
  public class gmttimestamp : Task
  {
   private string _pattern;
   private string _property;

   [TaskAttribute("pattern", Required = true)]
   public string Pattern
   {
    get { return _pattern; }
    set { _pattern = value; }
   }

   [TaskAttribute("property", Required = true)]
   public string Property
   {
    get { return _property; }
    set { _property = value; }
   }

   protected override void ExecuteTask()
   {
    string gmtTime = DateTime.UtcNow.ToString(_pattern);
    Properties.Add(_property, gmtTime);

    Log(Level.Info, "[gmttimestamp] :" + gmtTime);
   }
  }
   ]]>
 </code>
   </script>

Usage in Nant build script:

 <gmttimestamp property='today' pattern="yyMMdd"/>
 <gmttimestamp property='start.selenium.timestamp' pattern="HH:mm:ss"/>

Fully integrated task can be found here: http://code.google.com/p/design-of-selenium-tests-for-asp-net/source/browse/trunk/_build/scripts/continuous/main.build

Design of Selenium tests for ASP.NET: Finding the source of erratic bugs

After week of running UI tests as a part of continuous integration build without modifications in SUT code I realized that significant part of tests runs is failed! I know that UI tests are erratic by their nature, because they have a lot of dependencies, which you can’t control, but 40% of failures is too much. Also, when i run this tests on local without Selenium Grid cluster, all tests always pass.

This needs to be fixed. Lets check error messages:

1) Test CheckNameOnLoginPage: ERROR: Element xpath=/html/body/form[@id='form1']/div[3]/table/tbody/tr[2]/td[2]/input[@id='txtUser'] not found

2) Test CheckCurrentUserName: System.Exception: Error while executing command AssertUserName ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Selenium.SeleniumException: Timed out after 20000ms

3) Test NewlyAddedUserAppearsInUsersList: Selenium.SeleniumException: Timed out after 20000ms

Notice that always appears in random tests and always connected to absence elements. I observed test runs and found out that sometimes IIS error page with message “403.9 Access Forbidden Too many users are connected” appears in browser.

Here is it. IIS in windows of “professional” version has a limit of 10 users and therefore sometimes whole page is not loaded or just several JavaScript files and this breaks whole client side application. Here is the IIS log:

08:51:30 10.6.9.180 POST /SampleApplication/Login.aspx 403
08:51:30 10.6.9.180 GET /SampleApplication/Login.aspx 403

….

15:10:57 10.6.24.72 POST /SampleApplication/Login.aspx 302
15:10:57 10.6.24.72 POST /SampleApplication/Login.aspx 302
15:10:57 10.6.24.72 GET /SampleApplication/Home.aspx 200
15:10:57 10.6.24.72 GET /SampleApplication/Home.aspx 200
15:10:57 10.6.24.72 GET /SampleApplication/js/Frameworks/jquery-1.3.1.js 200
…
15:10:57 10.6.24.72 GET /SampleApplication/js/Services/Services.js 403

…

It is only needed to upgrade to IIS 7 to fix it. But i already spent a lot of time (about 5 hours) to find out what happened and do not want to deal with this problem in future.

First let’s learn tests to recognize IIS error pages:

private static void AssertErrorPage<TT>(TT target) where TT : PageBase, new()
     {
         var bodyText = target.Selenium.GetBodyText();

         if (bodyText.Contains("Server Error in "))
         {
             Assert.Fail("Server error while navigating\r\n\r\n {0}.", bodyText);
         }

         if (bodyText.Contains("Internet Information Services") && bodyText.Contains("Microsoft Support"))
         {
             Assert.Fail("IIS error while navigating\r\n\r\n {0}.", bodyText);
         }
     }

And include iis log analysis in continuous report. LogParser application can be used for this purposes – it supports SQL – like queries to logs. Here is sql to select log items from time to time into xml:

SELECT *
FROM %logdir%\ex%today%.log
TO %reportsdir%\extracted_iis_log.xml
WHERE TO_TIME(time) BETWEEN TIMESTAMP('%starttime%', 'hh:mm:ss') AND TIMESTAMP('%endtime%', 'hh:mm:ss') 

All details how to run logparser in nant script can be found here: http://code.google.com/p/design-of-selenium-tests-for-asp-net/source/browse/trunk/_build/scripts/continuous/main.build

Xsl to integrate iis logs in CCNET report is here: http://code.google.com/p/design-of-selenium-tests-for-asp-net/source/browse/trunk/_build/ccnet/iisloganalyser.xsl

After this modifications we can review reports for bugs mentioned above:

An assertion failed.

IIS error while navigating

The page cannot be displayed There are too many people accessing the Web site at this time.

Please try the following:

Click the Refresh button, or try again later.

Open the epbyminw0115.minsk.epam.com home page, and then look for links to the information you want. HTTP 403.9 - Access Forbidden: Too many users are connected

Internet Information Services

Technical Information (for support personnel)

Background:

This error can occur if the Web server is busy and cannot process your request due to heavy traffic.

More information:
Microsoft Support.

And IIS logs are saved for us right in the CCNET report:

Now it is possible to identify where the problem is without deep investigation – all tips and keys to solution are available in report, and if problem will appear in future – it most likely will be resolved with low efforts.

So, the moral is: if you experienced the problem in SUT or tests, first learn your tests to find and alarm about this problem, and only after that fix it. This will help you and your colleagues in future.

Actually here we started to implement analysis of how system behaves in run time, and this is only beginning. In future posts i want to describe now to get statistics from memory dumps and performance probes. This will give developers priceless statistics which will help to understand problems with concurrency, performance troubles, memory leaks, “hard to reproduce” bugs.

All sources are available in design-of-selenium-tests-for-asp-net solution.

How to run selenium tests without interfering with desktop in Windows

It is very uncomfortably to run UI tests in parallel with other work. It shows browser window over you working window, steals focus moves the mouse,  and you can affect test results by closing one of the testing windows.

The way out – is to use “Desktops” tool from SysInternals. Now you can run tests in one desktop and work in another:

This is also good option for creation of Selenium Grid cluster with developer computers as remote control runners. Tests do not load memory and processor, but they simply obstruct work by interfering with desktop. “Desktops” tool can resolve this problem and allow you to create big and cheap Selenium Grid cluster.

Design of Selenium tests for ASP.NET: Running UI tests as a part of continuous integration.

To keep UI tests up up to date it is needed to maintain them continuously. Build should be verified against UI tests every day and if something is broken – team should know what happened immediately. Only with such strategy give a chance for UI tests to survive. Just think, you are about to release build, it is late afternoon, it is needed to verify this build by QA and you realize that some of tests are broken (worth – if it broken by with strange errors). Will you fix this tests? NO. And another situation – you have just committed your changes and some automated system sends you e-mail that one use case is broken exactly after your commit. Will you fix it? Yep - it is very easy to find source of the problem.

Given all this, it is have no sense to have UI tests without automated continuous integration. And lucky we are it is possible to set up this procedure with only open source tools.

1. Scheduler and viewer of builds - Cruise Control .NET
2. Console client for Subversion
3. Nant as build system
4. Gallio as test runner
5. PsExec tool to open browsers on remote computers.

Here are the solutions to main issues of running Selenium Grid on CCNET:

1) CCNET is that its worker process is implemented as windows service. Browsers just can't start without desktop. This can be resolved by PsExec tool, which can run processes in desktop session of particular user.

2) CCNET service should be runned under user, which have access to all servers, where remote controls will be executed.

3) PsExec shows window with terms of use first time, so exec it manually before using it with service.

The example of CCNet config file can be found here. And all builds scripts which download latest version of sources, adjust configuration files, build application and tests binaries, starts selenium grid on server and slave computers and run tests are here.

Build in Sample application starts every time after code commit. But if your tests take more then 10 minutes to run – divide your tests into 2 tests suits – one for checking after commit, and one for checking every night – this still will keep your tests up to date.

All sources are available in design-of-selenium-tests-for-asp-net solution.

How to run Selenium Grid hub and remote control through nant or command line

On the official site of Selenium Grid there is no description how to run it without ant. But download this tool just to start grid is not very convenient, so i wrote tasks for nant and found the way to run Grid from command line:

First define properties:

      <property name="selenium.server.file" value="${src.dir}\_tools\selenium\selenium-server.jar" />
    <property name="selenium.grid.hub.file" value="${src.dir}\_tools\selenium\selenium-grid-hub-standalone-1.0.4.jar" />
    <property name="selenium.grid.rc.file" value="${src.dir}\_tools\selenium\selenium-grid-remote-control-standalone-1.0.4.jar" />

task to start hub:

     <target name="start.selenium.grid.hub">
      <exec program="java" verbose="true" failonerror="false">
        <arg value="-jar" />
        <arg value="${selenium.grid.hub.file}" />
      </exec>
    </target>

task to start remote control:

    <target name="start.selenium.grid.rc">
      <exec program="java" verbose="true" failonerror="false">
        <arg value="-classpath" />
        <arg value="${selenium.server.file};${selenium.grid.rc.file}" />
        <arg value="com.thoughtworks.selenium.grid.remotecontrol.SelfRegisteringRemoteControlLauncher" />
      </exec>
    </target>

Or simply from command line:

Hub:

java -jar D:\work\SeleniumDesign\build_artifacts\artifacts\continuous\source\_tools\selenium\selenium-grid-hub-standalone-1.0.4.jar

Remote Control:

java -classpath D:\work\SeleniumDesign\build_artifacts\artifacts\continuous\source\_tools\selenium\selenium-server.jar;D:\work\SeleniumDesign\build_artifacts\artifacts\continuous\source\_tools\selenium\selenium-grid-remote-control-standalone-1.0.4.jar com.thoughtworks.selenium.grid.remotecontrol.SelfRegisteringRemoteControlLauncher

Examples of build script which run selenium grid can be found here: http://code.google.com/p/design-of-selenium-tests-for-asp-net/source/browse/trunk/_build/scripts/continuous/main.build