Shai Raiten's talk at the SELA Developer Practice (May 2013) about Advanced Coded UI using Visual Studio 2012. This presentation explain how to work with multiple UI maps, how to use various types of data sources, how to use LINQ and Reflection for project refactoring, and how to create and use logs and traces.
2. Agenda
• Coded UI Testing Overview
• Fast Forward Automation Using MTM
• Understand Coded UI Anatomy
• Getting Started with Coded UI Testing
• Web
• Desktop
• Data Collectors
• Creating Custom Data Collector
• Data Binding
• Advanced Coded UI
• Tips & Tricks
3. What’s Coded UI Test
Coded UI Tests are automated tests that drive your
application through its user interface. These tests include
functional testing of the UI controls. They let you verify
that the whole application, including its user interface, is
functioning correctly. Coded UI Tests are particularly
useful where there is validation or other logic in the user
interface, for example in a web page.
3
4. Coded UI Features
• Functional Testing
• Generate code in C#/VB
• Intent aware recording and resilient playback
• Integrated with ALM story
• Build, deploy & test in lab or as part of build
• Local, remote runs, data collection
• Rich Extensibility
4
5. Intent Aware Recording
Click Start button
Click on the search box
Type keyword in the search box
Click the correct search result
Search for the keyword
Click the correct search result
Launch the application or url
Intent
6. Resilient Playback
Search
• Search for the Control based on the “Query ID” generated during
recording
WFR
• Wait for Control to be Ready: The playback needs to ensure that the
control is ready to be acted upon
Ensure
Visible
• Ensure that the control is visible (i.e. ensure point is clickable and
scrolled into view)
UI Sync
• Playback tries to ensure that the control that was supposed to have
received an action has actually received it
7. Creating Coded UI Test
• Coded UI Test can be created the following ways:
• Fast Forward Automation using MTM
• Generating your Coded UI form existing recoding (Convert the
recorded actions from a manual test)
• Generating a new Coded UI Test from scratch using Coded UI
Test Builder
• Writing your own Coded UI (Advance option)
7
8. Coded UI With MTM
• Microsoft Test Manager is a platform you can use
to create and manager your test plans and test
cases
• We can execute the test case manually, record the
actions on the UI and generate them to a code –
This will be our Coded UI
• We can playback the generated code as an
automatic test
8
14. Generated Coded UI Files
14
CodedUITest1.cs Contains the coded UI test class, test methods and assertions.
UIMap.uitest Contains the XML model for the UIMap class, including all
windows, controls, properties, methods, parameters, actions,
and assertions.
UIMap.Designer.cs Contains the code representation of the XML contained in
the UIMap.uitest file. Do not edit this file.
UIMap.cs Contains more of the code for the UIMap class. You can put
any customizations for the UI map in this file.
15. CodedUITest Class & Test Method
15
[CodedUITest]
public class CodedUITest1
{...
[TestMethod]
public void CodedUITestMethod1()
{
this.UIMap.AddTwoNumbers();
this.UIMap.VerifyResultValue();
// To generate more code for this test, select
// "Generate Code" from the shortcut menu.
{
{
16. UIMap Methods
16
/// <summary>
/// MenuNavigation
/// </summary>
public void MenuNavigation()
}
#region Variable Declarations
HtmlHyperlink GrouHyperlink = this.UIWindow.UISampleDocument.GrouHyperlink;
HtmlHyperlink LisHyperlink = this.UIWindow.UISampleDocument.LisHyperlink;
#endregion
// Click 'DropDownList With Grouping' link
Mouse.Click(GrouHyperlink, new Point(48, 18));
// Click 'Cascading DropDown Lists' link
Mouse.Click(LisHyperlink, new Point(41, 13));
{
17. UIMap Properties
17
public virtual OpenWebSiteParams OpenWebSiteParams
}
get
{
if ((this.mOpenWebSiteParams == null))
{
this.mOpenWebSiteParams = new OpenWebSiteParams();
{
return this.mOpenWebSiteParams;
{
}
18. UIMap Fields
18
[GeneratedCode("Coded UITest Builder", "11.0.60315.1")]
public class OpenWebSiteParams
{
#region Fields
/// <summary>
/// Go to web page 'http://www.outlook.com/' using new
browser instance
/// </summary>
public string UIOutlookWindowsInterneWindowUrl =
"http://www.outlook.com/";
#endregion
}
20. Search Methods
• SearchProperties - Mandatory properties that will
be used to search the control
• FilterProperties – When there is one
SearchProperties match or less, FilterProperties
turns into actions to the stage where if finds one
match.
20
21. Optimize UI Object Finding
21
This is an example of an Search Criteria
public class UIMicrosoftHomePageDevDocument : HtmlDocument
{
public UIMicrosoftHomePageDevDocument(UITestControl searchLimitContainer) :
base(searchLimitContainer)
{
#region Search Criteria
this.SearchProperties[HtmlDocument.PropertyNames.Id] = null;
this.SearchProperties[HtmlDocument.PropertyNames.RedirectingPage] =
"False";
this.SearchProperties[HtmlDocument.PropertyNames.FrameDocument] = "False";
this.FilterProperties[HtmlDocument.PropertyNames.Title] = "Microsoft Home
Page | Devices and Services";
this.FilterProperties[HtmlDocument.PropertyNames.AbsolutePath] = "/en-
us/default.aspx";
this.FilterProperties[HtmlDocument.PropertyNames.PageUrl] =
"http://www.microsoft.com/en-us/default.aspx";
this.WindowTitles.Add("Microsoft Home Page | Devices and Services");
#endregion
{
{
22. Initialize & Cleanup
22
#region Additional test attributes
//Use TestInitialize to run code before running each test
[TestInitialize()]
public void MyTestInitialize()
{
//Run Initialize Code here!
{
//Use TestCleanup to run code after each test has run
[TestCleanup()]
public void MyTestCleanup()
}
//Run Cleanup Code here!
{
#endregion
23. Test Settings
• XML file that provides advanced settings for test execution.
• When should use .testsettings
• Collect diagnostic data to help isolate bugs in your application for
Coded UI Tests
• Run the client, server, and other parts of your application on
different machines (Local or virtual) to verify that it behaves as
expected in a distributed environment.
• Distribute a large number of tests across additional machines on
different configuration (OS/Client Browser)
23
24. Creating .testsettings file
• Right click on your solution in the Solution Explorer
and choose to add new item, pick the Test Settings
template and add new .testsettings file.
• The new .testsettings is now placed in your solution
items
24
25. Test Settings Overview
• Test Setting is compound with the following settings:
• Roles
• Data Collectors & Diagnostics
• Deployment
• Setup and Cleanup Scripts
• Hosts
• Test Timeouts
• Unit Test
• Web Test
25
26. DEMO
Configure Your Test Settings
• To configure the your
test settings, simply
double click on the
.testsettings file
• The Test Setting window
will be display, with the
General section as
default
26
27. Data & Diagnostics
• In this module we will demonstrate how to collect data
and diagnostics data on your local machine
• We will focus on the next data collectors
• Event Log
• IntelliTrace
• System Information
• Video Recorder
• * Custom Collectors
27
28. Event Log
• Enable the Event Log and click to configure it
• The Configuration Dialogue will be presented:
28
29. IntelliTrace
• Enabling this collector during the test can be
helpful for programmers attempting to reproduce a
bug and decreasing the time spent on debugging
the application
29
30. System Information
• The System Information data collectors collects
data about your operating system
• The input is an XML file formatted in this kind of
structure
30
34. Coded UI Test Editor
• Remove unwanted UI actions
• Changes the names for test methods and controls
• View and Open Properties Window for selected item
• Split one action into multiple methods
• Adds custom code to your test methods by moving to
UImap.cs
• Add a pause prior to a UI action specified in milliseconds
• Identifies the location of the control in the UI of application
under test
34
37. What are assertions..?
• Methods that let you validate properties of
your user interface elements
• You typically compare the value of control
properties with an expected value
• Expected value can be hard coded or come
from parameters or external files
• Assertions use the Unit Test assertion
methods
37
38. Assert Types
• They are few types of asserts
Assert.AreEqual - will compare between object property
value to the expected value, if the value is True – pass
the test, otherwise fail the test.
Assert.Fail – Will fail the test with not no
condition if stated for example Assert.Fail();
could be used also with external condition i.e
if (!maxTestRuns == 0) Assert.Fail(“Test failed”)
38
39. Assert Types
• CollectionAssert Class - Verifies that two specified
collections are equal, Two collections are equal if they have
the same elements in the same order and quantity. i.e.
CollectionAssert.AllItemsAreNotNull(actual); - will test if all
items are not null
• StringAssert Class – Compare between strings i.e.
StringAssert.Contains(String Value, String Substring, String
Message) – will test that the first string contains the second
substring, the string must be case sensitive
39
40. Assert Method Code
40
/// <summary>
/// CheckResult - Use 'CheckResultExpectedValues' to pass parameters into
this method.
/// </summary>
public void CheckResult()
{
#region Variable Declarations
WinText uIItem0Text = this.UICalculatorWindow.UIItem0Text;
#endregion
// Verify that the 'Name' property of '0' label equals 'Result'
Assert.AreEqual(this.CheckResultExpectedValues.UIItem0TextName,
uIItem0Text.Name);
// Verify that the 'DisplayText' property of '0' label equals '0'
Assert.AreEqual(this.CheckResultExpectedValues.UIItem0TextDisplayText,
uIItem0Text.DisplayText);
{
42. Create Planned Test Lists – Order
Tests
• An ordered test is a container that holds other tests
and guarantees that tests run in a specific order
• You can add/remove test from an .ordertest file
and run it the Test Explorer or via TFS Build
42
44. Data Driven Tests
• You can run your test case multiple times using
other different sets of data
• You can use MTM Test Case items as data source, as
well as XML, CSV, XSL and SQL Express
44
45. Data Sources Types
Data Source Type Data Source Attribute
Test Case in TFS [DataSource("Microsoft.VisualStudio.TestTools.DataSource.TestCase",
"http://TFSServer:8080/tfs/DefaultCollection;ProjectName", "30",
DataAccessMethod.Sequential), TestMethod]
XML [DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML",
"|DataDirectory|data.xml", "Iterations", DataAccessMethod.Sequential),
DeploymentItem("data.xml"), TestMethod]
CSV [DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV",
"|DataDirectory|data.csv", "data#csv", DataAccessMethod.Sequential),
DeploymentItem("data.csv"), TestMethod]
XLS DataSource("System.Data.Odbc", "Dsn=Excel Files;Driver={Microsoft Excel Driver
(*.xls)};dbq=|DataDirectory|Data.xls;defaultdir=.;driverid=790;maxbuffersize=2048;pageti
meout=5;readonly=true", "Sheet1$", DataAccessMethod.Sequential), TestMethod]
SQL Express [DataSource("System.Data.SqlClient", "Data Source=.sqlexpress;Initial
Catalog=tempdb;Integrated Security=True", "Data", DataAccessMethod.Sequential),
TestMethod]
45
46. Data Driven Tests
46
In Visual Studio 2012 You will have to insert the
DataSource attribute directly in the code in the line
above your testmethod.
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV",
"|DataDirectory|data.csv", "data#csv", DataAccessMethod.Sequential),
DeploymentItem("data.csv"), TestMethod]
public void CodedUITestMethod1()
}
this.UIMap.CheckResultExpectedValues.UIItem0TextName =
TestContext.DataRow["ResultValue"].ToString();
this.UIMap.CheckResult();
{
47. Use XML As Data Source
• Create a new XML file in Visual Studio
• Compose your data in the next order
• Open new root tag under <Item></ Item >
• Add all the other parameters between to first root tag
till his closing:
<URL>http://www.formbreeze.com/demo.htm</URL>
<FirstName>John</FirstName>
<LastName>Doe</LastName>
<Email>john.doe@outlook.com</Email>
<Phone>+972 054 54545454</Phone>
<HeardAboutUs>Other</HeardAboutUs>
47
49. Multiple UIMaps
• Logical partitions when testing large application,
each UIMap can be related to a specific module or
a page, for instance Customer module or login page
• Each tester can work and be responsible on
different sector, which means he will no other
tester working on the same resource, avoiding
multiple check-ins
• Easier to maintain, you “know the ropes”, not
dealing with on file with thousands of code lines
49
51. Extracting Dynamic Data
• Some parts of your test framework will include a
dynamic value you will to use at runtime
• When creating new data it might be rendered and
presented differently or randomly.
• In the following example I’ll show you how get a
text from an alert box -
http://www.w3schools.com/js/tryit.asp?filename=t
ryjs_alert
51
52. Extract property value from an
object (UIMap)
52
The first step is creating a new browser launch within the wanted
URL
public void LaunchPassGen()
{
BrowserWindow PassGenWindow;
PassGenWindow =
BrowserWindow.Launch("http://www.w3schools.com/js/tryit.asp?filen
ame=tryjs_alert");
}
53. Extract property value from an
object (UIMap)
53
Second step is to automate the click on the button that executes
the popup window
You can either record it or hand code it
public void ClickBTN()
{
#region Variable Declarations
HtmlInputButton uIShowalertboxButton =
this.UITryitEditorv17WindowWindow.UITryitEditorv17Document.UIViewFrame.UIHttp
wwww3schoolscomjDocument.UIShowalertboxButton;
#endregion
// Click 'Show alert box' button
Mouse.Click(uIShowalertboxButton, new Point(53, 11));
}
54. Extract property value from an
object (GetPropertyValue)
54
Explore the object and see which property value you can and
want to extract
In this example we will extract the innerText value from the
popup object identifying it by searching for the window name
public static class SharedActions
{
public static string GetPropertyValue()
{
WinWindow popUpWindow = new WinWindow();
popUpWindow.SearchProperties[WinWindow.PropertyNames.Name] = "Message from webpage";
WinText innerText = new WinText(popUpWindow);
string text = innerText.DisplayText;
return text;
}
56. Playback Settings
• In a coded UI test playback, you can instruct a
certain test method to work in a different way, for
instance
• You can postpone the executing of a method until the
involved control will be ready
• You can add a delay time between test methods
• You can decide which methods will not fail the test even
though they failed and more…
56
57. Playback Settings
• Playback is configured by modifying the fields in
Playback.Playbacksettings class.
• Playback.wait - Make playback wait for certain event or
time
• Continue on Error – The engine will continue on to the
next action after an error
57
58. Playback.Wait
• WaitForReadyControl()– This waits for the control to be ready
to accept mouse/keyboard input.
• WaitForControEnabled() – This waits for the control to be
enabled. For example, you can use this wait till the “Next” button of the
wizard is enabled
• WaitForControlExisted() – This waits for the control to exist on
the UI. For example, you are expecting an error dialog after the
application has done the validation of the parameters.
58
Playback Logic: Search
UI can be visualized as a Tree of UI elements rooted at Desktop. QiD is used to search in this tree
RnP engine uses Tree switching across technology boundaries to provide a merged view of the tree
BFS is to search for elements. To improve resiliency intermediate nodes are ignored during search if they are not found.
SmartMatch: If no top-level window is found, a heuristic alogorithm is used to find the nearest match.
Playback logic: WFR/UISync
Wait for Ready: Implemented by using perf counters to check “Thread State” of the UI Element. If it is in wait state and is waiting for user action, then action is performed.
Plug-ins can override by providing plugin specific WFR implementation. (E.g. IE waits for DocumentComplete event)
UI Synchronization: It ensures that the playback action happened as expected. For MSAA, implemented by checking message queue for the control. For WPF, UIA provides a UI sync pattern that can be queried.
Primitive Actions: Playback supports following UI actions
Check, Uncheck, DoubleClick, MouseHover, MouseMove, MouseButtonClick, MouseWheel, SendKeys, SetValueasComboBox, Drag, DragDrop, Scroll, SetFocus
Playback Logic: Search
UI can be visualized as a Tree of UI elements rooted at Desktop. QiD is used to search in this tree
RnP engine uses Tree switching across technology boundaries to provide a merged view of the tree
BFS is to search for elements. To improve resiliency intermediate nodes are ignored during search if they are not found.
SmartMatch: If no top-level window is found, a heuristic alogorithm is used to find the nearest match.
Playback logic: WFR/UISync
Wait for Ready: Implemented by using perf counters to check “Thread State” of the UI Element. If it is in wait state and is waiting for user action, then action is performed.
Plug-ins can override by providing plugin specific WFR implementation. (E.g. IE waits for DocumentComplete event)
UI Synchronization: It ensures that the playback action happened as expected. For MSAA, implemented by checking message queue for the control. For WPF, UIA provides a UI sync pattern that can be queried.
Primitive Actions: Playback supports following UI actions
Check, Uncheck, DoubleClick, MouseHover, MouseMove, MouseButtonClick, MouseWheel, SendKeys, SetValueasComboBox, Drag, DragDrop, Scroll, SetFocus
Open MTM and
Create a manual test case
Execute the created test case using recorder
Save & Close
Generate code from existing recording in VS
To start recording, choose the Record icon. Perform the actions that you want to test in your application, including starting the application if that is required.
For example, if you are testing a web application, you might start a browser, navigate to the web site, and log in to the application.
To pause the recording, for example if the tested application have crashed and you want continue from the last step, simply pause your record, start the application and reach to that certain step, hit back on this button to resume.
To delete actions incase you have made mistakes in recording unnecessary actions, choose the Edit Recorded Steps button
To add Asserts, choose the Add Assertions button in the Coded UI Test Builder, and then choose a UI control in your running application.
When you choose Generate Code, all the recorded actions will be generated to code
You can record actions using UI Test Builder the following ways:
After creating new Coded UI Test in Visual Studio:
choose Record actions, edit UI Map and add assertions
The generate code test builder will appear
A coded UI test class is identified by a CodedUITestAttribute applied to the class.
Each coded UI test is a test method in a coded UI test class. You can add multiple test methods to each coded UI test class and identify each coded UI test method by using the TestMethodAttribute.
Your test method can also add validation code for a UI test control to obtain the value of a property of a UI test control. The test method can use an Assert statement to compare the actual value of the property to an expected value. The result of this comparison determines the outcome of the test. Every time that you run a coded UI test, you can analyze the test result and if the test fails, you can see or store the details of which assertion failed.
When you create a coded UI test, these files are added to your test project:
A coded UI test class is identified by a CodedUITestAttribute applied to the class and automatically applied to the class, which allows the testing framework to recognize it as a testing extension. Also notice that this is not a partial class. All class code is contained in this file.
Each coded UI test is a test method in a coded UI test class. You can add multiple test methods to each coded UI test class and identify each coded UI test method by using the TestMethodAttribute.
Your test method can also add validation code for a UI test control to obtain the value of a property of a UI test control. The test method can use an Assert statement to compare the actual value of the property to an expected value. The result of this comparison determines the outcome of the test. Every time that you run a coded UI test, you can analyze the test result and if the test fails, you can see or store the details of which assertion failed.
When you create a coded UI test, these files are added to your test project:
Each method has a structure that resembles the login() method.
This is explained in more detail under the code, which is presented together with line breaks to add clarity.
The code for each property is also very standard throughout the class. The following code for the loginParams property is used in the login() method.
Notice that the property uses a private local variable that is named mloginParams to hold the value before it returns it. The property name and the class name for the object it returns are the same.
As with all classes in the UIMap.cs file, this class starts with the GeneratedCodeAttribute.
In this small class is a Fields region that defines the strings to use as parameters for the Keyboard.SendKeys method that is used in the UIMap.login () method that was discussed earlier. You can write code to replace the values in these string fields before the method in which these parameters are used is called.
In order to find the control in test execution phase, Coded UI test captures some set of properties on the control and name them as “Search Properties” and “Filter Properties”. Each instance of these is a Name-value pair, with Name being the Property-Name and Value being the actual value of the property during the time of the capture.
First pass of the Search is done for the match of All properties given in SearchProperties list. Basically this can be referred as AND Condition of search properties.
Second is FilterProperties where you can refer as Or Condition.
Each technology has a limited set of properties that can be used for search via Coded UI tests. So, Coded UI tests chooses only the properties exposed by the technology as SearchProperties. These are chosen by targeting a good trade-off between robustness and performance.
For instance, the Web technology exposes many properties including all the attributes of the controls rendered in the page, whereas .Net controls’ accessibility exposes only a sub-set of the properties via accessibility.
From the code presented , we will be able to get all the properties that are associated with the control under test from the control object ( mUIQCustom ).
When the search for the control starts or implicit by any usage of the control in actions or property validations , it starts finding the controls under the Container element of the control ( HtmlDocument ).
Search in Coded UI test is a Breadth-First which compares each of the controls in the traversal for the match of the properties given in the code.
First pass of the Search is done for the match of All properties given in SearchProperties list. Basically this can be referred as AND Condition of search properties. In cases, where the results of the search using SearchProperties finds exactly one control or doesn’t find any control, Coded UI test skips using the Filter Properties (even if is defined in the code) because of the 2 cases,
1. There is exactly one control, so there is no need to apply filters.
2. There are no controls to apply filters.
But, in cases where the list of SearchProperties is not good enough to find the exact control, Coded UI tests uses the filter properties (one by one) till it finds One exact match. Basically this can be referred as Ordered Match.
In remote cases where Search using FilterProperties also results in more than one control, the First match is returned as the resultant control.
This method calls each UIMap method that you specified when you recorded your test, which is described in the section on theUIMap Class.
A region that is titled Additional test attributes, if uncommented, contains two optional methods.
The MyTestInitialize() method has the TestInitializeAttribute applied to it, which tells the testing framework to call this method before any other test methods. Similarly, the MyTestCleanup() method has the TestCleanUpAttribute applied to it, which tells the testing framework to call this method after all other test methods have been called. Use of these methods is optional. For this test, Login() method could be called from MyTestInitialize() and the Logout()method could be called from MyTestCleanup() instead of from CodedUITest1Method1().
To run your unit, coded UI, web performance, or load tests by using Visual Studio, you can add, configure and select the test settings to use when you run your tests. To run your tests, collect data, or affect a test machine remotely, you must specify a test controller to use in your test settings. The test controller will have agents that can be used for each role in your test settings..
To include the data and diagnostics that you want to collect on your local machine, select the diagnostic data adapters according to your testing needs. To configure a diagnostic data adapter that you have included, select the specific diagnostic data adapter and then choose the Configure option above the list of the data and diagnostic adapters.
The Event Log is determined form the Windows Event Viewer that stores event IDs to define the uniquely identifiable events that a Windows computer can encounter. For example, when a user's authentications fails.
For more info regarding the Event Viewer go to: http://windowssecrets.com/top-story/what-you-should-know-about-windows-event-viewer/
This Configuration Dialogue sets what data is gathered from the system Event Log during the test run. In it you can set which Event Logs and Types to collect, and set the Maximum number of entries to log per test.
You can also choose whether you want to Collect data from test cases that are ran as a part of another test (unlike the video recorder log entries do not occupy as much storage space).
You may want to coordinate with the developers before you enable and configure the IntelliTrace, and advise them to look up runtime data from test runs of the application in order to perform historical debugging.
The course of action Recording the relevant events that may cause the bug to occur, this will give the programmers to put the focus on the relevant code and function calls upon these recorded events.
Note: You would want to collect call information in case there is more complicated bug to analyze and user code error and you suspecting the problem lies on a code error.
Keep in mind that collecting call information other than IntelliTrace events only my reduce the tested application performance.
In order to configure the IntelliTrace recorder in a manner that will assist coders to understand and debug the application you may want explore this subject, for more info visit here:
http://msdn.microsoft.com/en-us/library/dd264915.aspx
Since Coded UI is being used at it interacts with the desktop, You can create a video recording of your desktop session when you run an automated test
This data collector can be useful to view the user actions for a coded UI test.
The video can help other team members isolate application issues that are difficult to reproduce.
In Visual Studio 2012 You can enable Audio recording as well while capturing the desktop actions and use it later on as a commentary.
as well as the desktop actions, and set the Frame rate, Bit rate and video quality.
Find your UI control to use as assert
Open your coded UI Test Builder
Drag the assert cross hair to the choose UI Control
An UI Control Map will be display
You can add assertion to any properties shown
Select the property and the value
Click on Add Assertion
Select a comparator and comparison value
Click on OK
Click on the generate button and insert the right name
For the assertion method and click on OK button
When you create a new coded UI test, the Visual Studio testing framework generates code for the test by default in a UIMap class.
The generated code for the UI Map contains a class for each object that the test interacts with. For each generated method, a companion class for method parameters is generated specifically for that method. If there are a large number of objects, pages, and forms and controls in your application, the UI Map can grow very large. Also, if several people are working on tests, the application becomes unwieldy with a single large UI Map file.
One of the examples I give here is when executing a popup window and extracting the inner text
In this example we are going to explore how to extract a text from a popup window