Test automation with Specflow and VSTS
This blog explains in a high overview how Specflow and Visual Studio Team Services (VSTS) can be put to use when it comes to automate regression testing.
When working within agile website projects, incremental deliveries also mean that a lot of regression testing need to be done. Stating that all of the development team members have a T-shaped skill set, therefore spreading the workload between all the team members, is one way to make sure that regression testing is being done properly. However, it still takes a lot of time and effort that could’ve been spent on developing an extra user story.
As said, regression testing is a repetitive job within agile projects, which needs to be done when user stories get delivered. And with many repetitive things, we thought: can’t we automate this? Of course, testing can be automated. Look at our developers creating unit tests, automating tests to check the quality of their code. But to business stakeholders, this is not directly the kind of test that assures them that there is nothing wrong with their website. So we found a way of tackling this; let’s use Specflow to automate our regression tests.
What is Specflow?
In short Specflow is a Cucumber implementation for .Net. This allows us to write scenarios that can be tested in a way our business stakeholders understand what’s being tested.
Writing a scenario
Let’s start with a simple example describing a login scenario:
As you can see, the scenario is written in the Gherkin syntax: Given, When, Then. This describes the behaviour or the process which shall be tested by our automated test. The examples beneath the scenario mean that the test will be run twice. The first time it will check a successful login attempt, the second time an unsuccessful login is being tested. In this way the scenario is readable by any stakeholder and the scenario is put in a structured way so we can program each step of the scenario.
Implementing the behaviour
Now we have the scenario so now we need to do some coding to get things going. First of all Specflow shows us which steps of the scenario have already been bound to code and which are not:
Screenshot of a Specflow feature file
The purple lines tell us that these steps are not implemented yet. Now we can tell Specflow to create these implementations for us. Just select a purple line and open up the menu by right clicking it. In the menu choose for the option “Generate Step Definitions” and follow the wizard which will popup.
Screenshot of the contextual menu of the purple lines
And a step file is generated! If we let Specflow generate the code for the “Given”-step, this piece of code is being generated:
Now a binding has been made between the readable line of the scenario and a method in which C# code is actually running the processes and checking the behaviour of the elements on our website pages. When using frameworks like Selenium you’ll be able to automate browser actions as well, to get the full automated testing experience.
If you want to read more information about setting up and running Specflow tests, please refer to the Specflow documentation page.
Setting up your automated testing process in VSTS
So now we’ve got our scenario’s implemented, it is time to configure when we want the tests to be run. Let’s say that we want to test each delivery to our testing environment. Within VSTS we can setup a release. Now we only have a Specflow sample project created for this blog, so no actual website will be released. Next to that we set up an Azure VM to host my test agent and to be able to physically start up the browsers in which we need to test our scenarios. Just to keep this simple, you’ll need to setup three actions to get your automated tests going on the Azure VM:
- Azure Resource Group Deployment
First select the resource group in which the VM is running on Azure
- Visual Studio Test Agent Deployment
Deploy a test agent to the VM on the selected resource group
- Visual Studio Test
Run the tests, test results will be saved in VSTS
If you already have a website release definition in VSTS, you can add these steps to it after the new version of the website has been deployed. When a test agent is already available on a server whith the necessary browsers present, you can just skip step 1 and 2.
Great! And what about the downsides?
In this blog we gave you a high overview of what we did to automate regression testing. Although this looks great, there also some downsides when it comes to automating regression tests.
- First of all, all of the scenarios and steps need to be maintained and extended with new functionality that need to be tested too. This will cost some time, but it’s time worth spending, because we’re taking a lot of risk out of the development process compared to manual regression testing. When planning your work for the next sprint, take this into account.
- Sometimes automated tests can react very instable to unexpected (but not necessarily wrong) behaviour of the website. For example: if data needs to be retrieved by asynchronous API calls, you will have to make sure that waiting times are set long enough to check the response. If a response comes later than expected, the execution of your scenario will fail.
- Regression testing is not necessarily testing one specific button and check what happens when you click on it. But that is the scope you actually want to test with your coded UI test. When testing forms or processes using Specflow and Selenium you can come across some instability because of long page loading times or changed forms and fields. This will ruin your test run sometimes for sure. But hey! It’s automated, just run your tests again.
- Don’t start writing tests without knowing which scenarios you want to test. Best way to determine what functionality needs to be tested is to let the business stakeholders decide what functionality has the most value to them and therefore needs to be tested with every delivery of the website. The discussion with the stakeholders will take a bit of time, but it’s worth it. Automating tests on functionality that have not that much added value to the stakeholders is a waste of time.
Specflow and VSTS are perfectly compatible with each other and therefore it’s relatively easy to set up automated regression testing combining the power of both technologies. Next to that, thanks to the Gherkin syntax, business stakeholders can also easily be involved in the process of testing the website. It also gives them a better feeling and understanding of what is being tested and why functionalities are being tested.We love these processes in which stakeholders and the development team come together to make pretty things. And that is what makes the downsides as mentioned before acceptable.