Article on Microsoft Visual Studio 2010 Automated Tests

Microsoft Visual Studio 2010 – Automated Tests – Overview (en-US)

Table of Contents

• Introduction
• Generic Tests
• Unit Tests
• Coded UI Tests
• Web Performance Tests
• Load Tests
• Database Unit Tests
• Ordered Tests
• See Also

Introduction

Automated tests are created using Visual Studio. Visual Studio Ultimate or Visual Studio Premium can be used to create a variety of automated tests, which enable efficient and effective testing of an application.
Automated test runs test steps automatically when executed and also validates the test result against expected result. The tests will pass or fail based on the expected output. Automated tests are quick to run and can be executed more frequently to quickly test if an application is working correctly. Following are the types of tests which can be created:

• Generic Tests 
• Unit Tests 
• Coded UI Tests 
• Web Performance Tests 
• Load Tests 
• Database Unit Tests 
• Ordered Tests

The following section has description of each type of the automated test.

Generic Tests

Generic tests are used to test functionality using API calls or command line tools for the application under test. Use a generic test to wrap an existing test, program, or third-party tool that can: be run from a command line; return a value of Pass or Fail; or optionally, it also return detailed results for 'inner' tests.

Unit Tests

Unit tests are used to test code at the method level. These tests are used to execute other source code by calling the methods of a class directly, passing appropriate parameters, and then including Assert statements; it can test the values that are produced against expected values.

Coded UI Tests

These tests provide functional testing. Coded UI tests are used to test the user interface of an application by performing UI actions and then performing validations. These tests can test whether the user interface of an application functions correctly. These tests perform actions on the user interface controls of an application and verify the correctness of displayed controls along with the correct values.

Web Performance Tests

Web Performance Tests are used as input to load tests for testing responses of web application servers by communicating using http requests. These tests are used for testing the functionality of web applications and testing web applications under load. Web performance tests are used in performance tests, capacity tests and stress tests.

Load Tests

Load tests are used to test application performance, capacity and stress using any of the: unit tests, web performance tests or coded UI tests. The basic load testing is used to determine the application’s behavior under normal and/or anticipated peak load situations. The goal of her is to simulate many users accessing a server at simultaneously.

Database Unit Tests

Database unit tests are used to test a stored procedure, function or trigger in a database. These tests can be used to establish a baseline state for database and then to verify any subsequent changes made to the existing database objects. Tests are used to verify that the changes have not broken any existing functionality.

Ordered Tests

Ordered tests contain other tests that are expected to run in a specific order. This test appears as a single test in the Test List Editor and the Test View window. Its result also appears in a single row in the Test Results window as a single test. An ordered test can contain any type of test except load tests. But, if an ordered test is executed remotely, or from a command line that contains manual tests, then manual tests are temporarily removed from it for that particular test run, and a warning is displayed.

The table below has details of types of automated tests that can be created. Note that Load tests cannot be created unless you have Visual Studio Ultimate. Manual test cases can be converted to automated tests by linking automation routines (methods) to manual test case and can be run using Microsoft Test Manager. The selection of automated test suitable for you depends on what you need to accomplish as part of automation.

Type of test	Run tests from Visual Studio	Run from a test plan by associating a test method with test case	Visual Studio 2010 Premium	Visual Studio 2010 Ultimate
Generic Tests	Yes	Yes		Yes
Unit Tests	Yes	Yes		Yes
Coded UI Tests	Yes	Yes		Yes
Web Performance Tests	Yes	Not recommended because Web Performance tests are primarily intended to be used with load tests. Note: These tests cannot be used to test the UI.	No	Yes
Load tests	Yes	Not recommended because you cannot view the test as it runs.	No	Yes
Database Unit Tests	Yes	Yes		Yes

Microsoft Visual Studio 2010 – Load Test – Overview (en-US)

Table of Contents

• See Also

The key goal of a load test is to imitate many users accessing a server at the same time. When a test (web test, unit test or coded UI test) is added to a load test, multiple users opening simultaneous connections to a server and making multiple HTTP requests are simulated. Properties on load tests can be set that largely apply to the individual tests. For a web application load testing, web tests are used as input to a load test. When unit tests are added to a load test, the performance of non-web based server workings can be exercised. An instance application of a unit test in load is to investigate data access model components.

Load tests facilitate to reproduce many users striking an application at the same time. The chief scenario for load testing is performance testing. How will the server act in response to so many users? Will response times be inaptly slow? Will error rates be tolerable? Another is capacity planning, where it is wanted to understand how large a server will be desirable to support the probable user load. Load tests are used in a number of different types of testing namely: Smoke testing, Stress testing, Performance testing and Capacity Planning:

Type	Description
Smoke Testing	To determine the application performance under load for short durations
Stress Testing	To determine the application sustenance under heavy load
Performance Testing	To determine application response
Capacity Planning	To determine application capacity with benchmark performance or vice versa

Load tests consist of a cycle of web tests or unit tests which operate under multiple simulated users over a phase of time. Load tests are formed with the help of Load Test Wizard. Load test formation can be started by walking through the load test wizard. The wizard guides through creating a scenario, which can be thought of as a particular group of users. Load tests contain one or more scenarios, which are used to reproduce how a group of users interacts with a server application. An individual scenario is prepared of a load pattern, a test mix, a browser mix, and a network mix. Load Test Editor can be used to alter the load test properties. These properties facilitate running the web tests with different user profiles, browser targets, and loading patterns.

Test results are stored in SQL-based Load Test Results Repository. When a load test is executed, information collected during the run is stored in the Load Test Results Repository. The Load Test Results Repository holds performance counter data as well as any information about recorded errors.

The Load Test Monitor is used to observe the load tests when running. To view load test results for completed test runs, the Load Test Analyzer is used. Statistics from the load test runs can be analyzed to trace bottlenecks, recognize errors, and compute improvements in the application. While a load test is running, a smaller set of the performance counter data is maintained in memory. This set of data can be monitored as the load test is running. After a load test has finished, the full set of data from the database can be analyzed. There are some differences in what data can be seen while a load test is running and after a load test has accomplished. There are also some differences in the functionality of the tools offered to analyze the data.

Load tests can be used through a set of computers known as a rig, which consists of agents and a controller. A collection of computers can be set up that generates simulated load for testing. The group consists of a single controller and one or more agents. Collectively, this group is called a rig. An agent is the element of the rig that is used to run tests and generate simulated load. The controller is the element of the rig that is used to organize the agents and gather the test results. A rig is used to generate more load than a single computer can generate alone.

Microsoft Visual Studio 2010 – Coded UI Test – Overview (en-US)

Table of Contents

• See Also

Coded UI tests are innovative capability of Visual Studio 2010. Coded UI tests offer a way to create entirely automated tests which work at user interface level to certify the functionality and behavior of an application. Coded UI Test Framework features of Visual Studio 2010 permit testers to quickly and effortlessly create coded UI tests by recording the actions performed or straight from existing action recordings. Action recordings contain the steps taken during manual testing of an application.

Visual Studio 2010 Ultimate or Visual Studio 2010 Premium can be used to craft Coded UI tests. These tests provide functional testing of the user periphery and validation of user interface controls. Automated UI tests enable testing exactness of the user interface functioning after code changes. Coded UI tests are more rapid to run than manual tests. The coded UI test performs actions on the user interface controls for an application and verifies that the correct controls are displayed with the correct values.

When a coded UI test is created, internally a UIMap object is generated that is definite to the test and represents the windows, controls, parameters, and assertions that are in the UI or that are created through the recording. Once the UIMap object is on hand, then actions can be performed on these UI stuff to automate the user interface. A coded UI test class is recognized by a CodedUITestAttribute applied to the class. Each coded UI test is a test method in a coded UI test class. Multiple test methods can be added to a coded UI test class.

The test method can also add confirmation code for a UI test control to get the value of any of its available property. Test methods use an Assert statement to compare the actual value and expected values. The result of this evaluation determines the outcome of the test. Every time that a coded UI test is run, test results can be analyzed and if the test fails, details of which assertion failed can be seen or stored.

Coded UI Test is a test type which inherits from Unit Test. It can be used anywhere a unit test is used (of course with a small number of catches). Its code is generated in either C# or VB .NET languages. Coded UI Test Framework provides loaded APIs for Web, Win Forms and WPF controls for UI testing. The framework features intent aware recording and resilient playback features also used for fast forwarding at some stage in manual testing and to create actionable bug. It supports both local as well as remote runs. Coded UI has very rich extensibility features to customize recording as well as to provide support for custom controls.

When a Coded UI Test is executed, Playback engine performs all the interactions of the tests. Playback engine has smart match algorithm for searching the controls which applies heuristics. Playback engine supports UI Synchronization (not for all though) which ensures that the control that was supposed to have expected an action has actually received it, and redo in case of failure. Coded UI Test Framework provides many rich features such as smart algorithms to wait for a control to be ready, ensure visibility or existence of a control.

How to Schedule Automated Tests Without Using TFS

Table of Contents

• Introduction
• MSTest.exe
• Windows Task Scheduler
• Scheduling an Automated Test
  • A batch file with MSTest.exe command
  • A Task that contains details of the batch file to execute (.job file)
  • Steps
• See Also

Introduction

A programmed test can be executed through command line program MSTest.exe. Windows Task scheduler can be used to schedule the MSTest.exe to run at particular time or in particular period to allow scheduling the automated tests. This piece of writing provides overview of MSTest.exe which is offered as part of Visual Studio or Test Agent installations and Windows Task Scheduler which is component of various Windows operating systems.

MSTest.exe

MSTest.exe is the command-line command that is used to run tests. This command has a number of options which can be used to tailor the test run. MSTest.exe does neither interpret the options nor the values specified for them in a case-sensitive way.
MSTest.exe program can be used to run automated tests in a test assembly from a command line. It is also feasible to view the test results from these test runs, save the outcome to disk, and save results to Team Foundation Server. See How to: Run Automated Tests from the Command Line Using MSTest  for additional information on the same.
Below is a snapshot displaying a small amount of of the command-line options that can be used to modify the run. Refer MSTest.exe Command-Line Options  for entire list.

Windows Task Scheduler

Task scheduler is a Windows component that provides the facility to schedule the initiation of programs or scripts at pre-defined times or after specified time intervals. The Windows Event Log service must be running before the Task Scheduler starts up.
The Task Scheduler service works by managing tasks; Task refers to an action taken in response to trigger. A task is defined by associating a set of events, which can include launching an application, executing a command or taking some custom-defined action, to a set of triggers, which can either, be time-based or event-based. See How To Schedule Tasks in Windows XP  for more information on the same. Similar information can be found for diverse operating systems.
The picture below gives summary of screens and available options to schedule a task using Windows Task Scheduler in Windows XP operating system.

Scheduling an Automated Test

From the information available in above sections, it is possible to schedule any automated test by creating a parameterized command of MSTest.exe to execute that test with required customization and scheduling that command with the help of Windows Task Scheduler. For the ease of managing the complex commands that have multiple lengthy parameters, it is good to use a batch file to execute that command. Batch file is type of a script, a text file that contains one or more commands to be executed. Thus we can manage the MSTest.exe command and its parameters separately from the task that is created in Windows Task Scheduler.

A batch file with MSTest.exe command

This batch file should have command that shall be executed to run an automated test. The command will be somewhat similar to shown below. It has placeholders for required and commonly used parameters to help user create the command. The command doesn’t contain actual parameters and hence they must be replaced with actual values. The parameters which are not used should be removed or MSTEST.EXE may throw invalid argument error! The output of this command will be saved in a .log file created with date time to see in case of issues.

"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" /testcontainer:[.dll/.loadtest/.webtest] /testsettings:[.testsettings] [/test:[TestMethod names] multiple times] [/testmetadata:.vsmdi file] [/testlist:[testlists] multipletimes]>> %date:~-4,4%-%date:~-7,2%-%date:~-10,2%_%time:~0,2%.%time:~3,2%.%time:~6,2%.log

Below is short description of the parameters provided and the usage:

Parameter Name	Used to specify	Comments
Testsettings	To use the specified test settings file for test run. (provide .testsettings file)
Testcontainer	Load a file that contains tests. Provide .dll (for Coded UI), .webtest or .loadtest file. Multiple files can be provided.
Test	The name of a test to be run. Multiple tests can be provided.	Use with TestContainer to run only selected tests in case .dll is provided that contains multiple tests.
Testmetadata	Load a file that contains test metadata (for .vsmdi file)	Do not use this option along with TestContainer.
testlist	The name of the test list to be run (as specified in metadata file) Multiple lists can be provided.	Use with TestMetadata to run selected tests using test list in case .dll is provided that contains multiple tests.

Since the resultfile parameter is not provided in the command, the results will be generated inside TestResults directory inside the path where batch file is executed (Start in path of the task). This is a better option compared to giving explicit result file because the result file may get overwritten if same test runs multiple times.

A Task that contains details of the batch file to execute (.job file)

This is a task configured to run at desired time or with available triggers. Below is a table explaining the parameters to be set or changed in the task:

Tab / Section	Field	Used to specify	Comments
Task	Run	Path of the file to be run as scheduled task	Give the full path of the batch file here
Task	Start in	Path of the folder where file/program is located	Give path of the folder that contains the batch file
Task	Run as	User account details that runs the scheduled tasks	Give valid user name password that has access to schedule tests
Schedule	Schedule Task	How often the task should run	Select Once/ Daily/ Weekly as required
Schedule	Start Time	Time when scheduled task should start
Schedule	Advanced Options	Additional Scheduling options such as dates
Schedule/ Advanced	Repeat Task	Specify to repeat task at interval specified	Use for executing task multiple times

Steps

• Create a directory where you want to put the batch file. Let us say [WorkingDirectory]. Note that by default, results file and other logs will be generated in this directory.
• Create a batch file MSTest.bat in [WorkingDirectory].
• Create Batch File.
  • Open MSTest.bat with notepad.
  • Right Click the file and select Edit in context menu. OR
  • Open Notepad, and use Open File menu to browse the MSTest.bat file.
  • Avoid double clicking the batch file because that will directly execute the command.
  • Copy and paste the MSTEST command (available in above section) to this file.
  • Modify the parameter values accordingly:
    • Binary/WebTest/LoadTest file path
    • Other parameter values and removed unwanted parameters
  • Save the file.
• Create task.
  • Open Task Scheduler.
  • Click Create New (The actual option will vary based on operating system).
  • Type a name for the task and an optional description.
  • Do one of the following:
    • To select a schedule based on the calendar, click Daily, Weekly, Monthly, or One time, click Next; specify the schedule you want to use, and then click Next.
    • To select a schedule based on common recurring events, click When the computer starts or When I log on, and then click Next.
    • To select a schedule based on specific events, click When a specific event is logged, click Next; specify the event log and other information using the drop-down lists, and Then click Next.
  • Click Browse to find and select the batch file (MSTEST.bat) created in above steps.
  • It’s good to specify the directory to start task in. Set it to [Workingdirectory].
  • Click OK/Finish.
• Done. Your scheduled task will be triggered at specified time and it will execute the batch file to start execution of the automated test(s).