Handling test data clean up in SpecFlow

Overview

This post describes how to handle scenario level clean-up in a predictable way.

Note: This blog post was originally written for SpecFlow 1.9 version.
In SpecFlow 2+ version you should use scenario scoped bindings  with  [BeforeScenario] execution order can be now controlled with an Order parameter.

Writing advanced UI tests with SpecFlow and Selenium often requires some test data to be created. During the tests execution this test data is changed. One of requirements for a good test is repeatedness. Test have to be executed multiple  times and should complete with the same results.

So we need some sort of clean-up procedures for our tests in order to satisfy above requirement. Let’s consider what SpecFlow could offer us in this case.

Hooks

Hook is the statement of code that is executed after or before particular event. Here are the most useful from my mind:

  1. [AfterScenario(“MyTag”)] – code is executed after all the scenarios marked with @MyTag.
  2. [BeforeScenario(“MyTag”)] – code is executed before all the scenarios marked with @MyTag

You could find more info about SpecFlow hooks on the project wiki here.

Let’s consider very simple scenario, where we need some sort of clean-up procedures:

Scenario: Price is updated
Given: Price for product foo is 10$
When: Price is changed to 15$
Then: Product foo has price 15$ on the site

What we could expect when test is finished? That the price for product ‘foo’ is 15$.
But what happened when we try to repeat this test? Sure it will fail as now price is 15$. What we need is to organise cleanup action after test execution, so user will be bound to the ABC again.

How could we achieve such behaviour? Well using the hooks in such way.

[AfterScenario]
public void AfterScenario()
{
    var title = ScenarioContext.Current.ScenarioInfo.Title;
    if (title == "Price is updated")
    {
        //do some actions to restore previous state
    }
    if (title == "SomeOtherTest")
    {
        //do some clean-up for that
    }
}

Usually logic inside if statement might be quite complex, as it requires some action to do and reuse the our Page Objects. So AfterScenario method become bloated and hard to read.

Let’s replace above statement with 2 separate methods and mark with our custom attribute:

[ScenarioCleanup("Price is updated")]
public void ResetPrice()
{
    //do some actions to reset the price
}

[ScenarioCleanup("SomeOtherTest")]
public void CleanupAfterOtherTest()
{
    //do some other actions
}

Code for newly created attribute;

public class ScenarioCleanup : Attribute
{
    public string ScenarioTitle { get; set; }

    public ScenarioCleanup(string scenarioTitle)
    {
        ScenarioTitle = scenarioTitle;
    }
}

And finally the code where all parts are gathered;

[Binding]
public class TestBehaviour
{
    public void BeforeTestRun()
    {
        _teardownActions = BuildDictionary(this.GetType());
    }

    [AfterScenario()]
    public void AfterScenario()
    {
        _commonTestBehaviour.ResolveScenarioCleanup(ScenarioContext.Current.ScenarioInfo.Title, this);
        //some other code, like disposing selenium driver or so
    }

    [ScenarioCleanup("User changes company")]
    public void ResetBindedUser()
    {
        //do some actions to rebound user to the initial company
    }

    public void ResolveScenarioCleanup(string title, object instance)
    {
        MethodInfo cleanupMethod;
        if (_teardownActions.TryGetValue(title, out cleanupMethod))
        {
            cleanupMethod.Invoke(instance, null);
        }
    }

    private Dictionary<string, MethodInfo> _teardownActions;

    private Dictionary<string, MethodInfo> BuildDictionary(Type type)
    {
        var withAttrs = type.GetMethods()
            .Where(m => m.GetCustomAttributes(typeof(ScenarioCleanup), false).Length > 0)
            .ToDictionary(key => ((ScenarioCleanup[])key.GetCustomAttributes(typeof(ScenarioCleanup), false))[0].ScenarioTitle, val => val);
        return withAttrs;
    }
}

In short:

  1. Before TestRun via reflection all the cleanup methods init.
  2. When scenario is executed, AfterScenario() method runs
  3. In AfterScenario method, corresponding clean-up method is resolved and executed
Posted in Selenium, SpecFlow | Tagged , , | Leave a comment

ApprovalTests > Customize output file names

By default ApprovalTests generate output file to the same location where test class resides, with the file name as ClassType.MethodName.

This behaviour is configured in `UnitTestFrameworkNamer` class, that is default namer for ApprovalTests.

Long test names issue

Tests are usually named as <SUT>_<Context>_<ExpectedResult>. Such combination might be quite long and could sometimes exceed 100 symbols. If we add complex tree structure of project itself, we could easily go beyond 260 max symbols in file path limit – https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation

In our case in addition to above, we also had TFS branches of different length, and was was possible to comiit in one branch, – was not possible to merge to another one due to path length limitation.

Allow to override output file on test method

Overriding default namer consists of 3 steps:

Creating attribute for test method

[AttributeUsage(AttributeTargets.Method)]
public class ApprovalFileNameAttribute : Attribute
{
	public string Name { get; }

	public ApprovalFileNameAttribute(string name)
	{
		Name = name;
	} 
}

Creating custom Namer for ApprovalTests

/// <summary>
/// Allows to override approved/received file names created by `UnitTestFrameworkNamer`.
/// Could be useful when the test method name is long to reduce the name length of output files.
/// In case attribute is not present on Method name, falls back to `UnitTestFrameworkNamer`
/// </summary>
public class AttributeBasedFileNamer : IApprovalNamer
{
	readonly UnitTestFrameworkNamer _unitTestNamer = new UnitTestFrameworkNamer();

	public string SourcePath => _unitTestNamer.SourcePath;

	public string Name
	{
		get
		{
			var customFileNameAtt = Approvals.CurrentCaller.GetFirstFrameForAttribute<ApprovalFileNameAttribute>();

			return customFileNameAtt == null 
				? _unitTestNamer.Name 
				: customFileNameAtt.Name;
		}
	}
}

In case of attribute does not exist for method, fallback to `UnitTestFrameworkNamer`

Glue it together – Register new namer as default

Approvals.RegisterDefaultNamerCreation(() => new AttributeBasedFileNamer());

You could register it on assembly initialize or in static constructor

Posted in Unit test | Tagged , | Leave a comment

Generate WinMerge report file on CI server when running Approval tests

In our automation tests projects we rely on Approval tests to assert test output files. It could be for example JSON API responses or XML messages generated by integration services.

This approach worked very well for us, in local when framework asserted *.received files vs *.approved files, WinMerge windows popped up and showed the difference.

But it did not work on CI server, if some of tests failed the test report could only indicate that something was wrong. To get the real failure one has to login to CI server and compare received and approved files using manual WinMerge run. WinMerge does provide the GUI for generating compare report, but until the WinMerge 2.15.2 it was not possible to use WinMerge CLI for this.

Generate diff report using WinMerge CLI

Starting from WinMerge 2.15.2 you could execute below command:

"C:\Program Files\WinMerge\WinMergeU.exe" path_to_left_file path_to_right_file 
-minimize -noninteractive -u -or path_to_report_file

The output result:

Generate WinMerge diff file in ApprovalTests

We could add the same functionality to ApprovalTests by creating new Attribute as below:

public class CliWinMergeReporter : WinMergeReporter
{
	public override LaunchArgs GetLaunchArguments(string approved, string received)
	{
		var defaultArgs = base.GetLaunchArguments(approved, received);

		var diffFileName = System.IO.Path.GetFileNameWithoutExtension(received).Replace("received", "diff") + ".html";

		var launchArguments = new LaunchArgs(
			defaultArgs.Program,
			$"{WrapPath(received)} {WrapPath(approved)} -minimize -noninteractive -u -or {WrapPath(diffFileName)}");

		return launchArguments;
	}
}

Then decorate test class with `[UseReporter(typeof(CliWinMergeReporter))]`.

After test run in source folder you will get 3 files:

  1. *.received
  2. *.approved
  3. *.diff

Posted in Uncategorized | Tagged , , | Leave a comment

ASP.NET WEB API documentation using Swagger – Global operation for 500 response code

Some api response codes, such as 500 for Internal server error, are the same for all API endpoints.
Let’s see how to implement such general behaviour using Swashbuckle.

Add below classes:

public class InternalServerErrorModel
{
    public string Message { get; set; }

    public string ExceptionMessage { get; set; }

    public string ExceptionType { get; set; }

    public string StackTrace { get; set; }
}

public class InternalServerErrorResponseFilter : IOperationFilter
{
    public void Apply(Operation operation, SchemaRegistry schemaRegistry, ApiDescription apiDescription)
    {
        var schema = schemaRegistry.GetOrRegister(typeof(InternalServerErrorModel));

        if (!operation.responses.ContainsKey("500"))
        {
            operation.responses.Add(new KeyValuePair<string, Response>("500", new Response()
            {
                description = "Internal server error occured",
                schema = schema
            }));
        }
    }
}

Register new operation filter in SwaggerConfig.cs

  c.OperationFilter<InternalServerErrorResponseFilter>(); 

And after running the application, each endpoint descriptions is decorated with:

Important!

Model above is default for asp.net web api, but it is security issue to expose internal details such Exception stack trace to outside world. Much better is to log exceptions to database using, for example, ELMAH library, and return just Error ID to clients.

Posted in Documentation, WEB API | Tagged , | Leave a comment

ASP.NET WEB API documentation using Swagger – Extend schema generation using SchemaFilters and FluentValidation rules

In this post we will see how to extend schema generation using FluentValidation rules.
We will see how to display min/max constraints for Integer type in Swagger UI and how to extend default examples to show valid email for email attribute.To get overall view about Schema filter, follow the official documentation at: Swashbuckle – Schema filters

Add FluentValidation to project + new API endpoint for demo

Install latest FluentValidation.WebAPI package into SwaggerDemo project.
Now let’s create another API endpoint, add new UsersController, model and validation:

public class UsersController : ApiController
    {
        public IHttpActionResult Post(UserModel model)
        {
            return Ok();//stub
        }
    }

    [Validator(typeof(UserModelValidator))]
    public class UserModel
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public string Email { get; set; }
        public int YearOfBirth { get; set; }
    }

    public class UserModelValidator : AbstractValidator
    {
        public UserModelValidator()
        {
            RuleFor(m => m.Email)
                .NotEmpty()
                .SetValidator(new EmailValidator());

            RuleFor(m => m.FirstName)
                .NotEmpty()
                .Length(3, 25);

            RuleFor(m => m.LastName)
                .Length(3, 25);

            RuleFor(m => m.YearOfBirth)
                .NotEmpty()
                .InclusiveBetween(1991, 2017);
        }
    }

Implement Schema updates based on validation rules

We will create and use IFluentValidatorSchemaUpdater interface and implement it for every suitable validator. With this design, we automatically conform to Single responsibility and Interface segregation principle for each implementation

    public interface IFluentValidatorSchemaUpdater
    {
        void Update(Schema schema, PropertyRule rule);
    }

    public abstract class FluentValidatorSchemaUpdater
        : IFluentValidatorSchemaUpdater where TValidator : IPropertyValidator
    {
        public void Update(Schema schema, PropertyRule rule)
        {
            var validator = rule.Validators.FirstOrDefault(v => v.GetType() == typeof(TValidator));

            if (validator != null)
            {
                Update(schema, rule, (TValidator)validator);
            }
        }

        protected abstract void Update(Schema schema, PropertyRule rule, TValidator validator);
    }

    public class RequriedFieldFluentValidatorSchemaUpdater
        : FluentValidatorSchemaUpdater
    {
        protected override void Update(
            Schema schema,
            PropertyRule rule,
            NotEmptyValidator validator)
        {
            if (schema.required == null)
            {
                schema.required = new List();
            }

            schema.required.Add(rule.PropertyName);
        }
    }

    public class EmailFluentValidatorSchemaUpdater
        : FluentValidatorSchemaUpdater
    {
        protected override void Update(
            Schema schema,
            PropertyRule rule,
            EmailValidator validator)
        {
            schema.properties[rule.PropertyName].example = "string.string@example.com";
        }
    }

    public class StringMinMaxFluentValidatorSchemaUpdater : FluentValidatorSchemaUpdater
    {
        protected override void Update(
            Schema schema,
            PropertyRule rule,
            LengthValidator validator)
        {
            schema.properties[rule.PropertyName].minLength = validator.Min;
            schema.properties[rule.PropertyName].maxLength = validator.Max;
        }
    }

    public class IntMinMaxFluentValidatorSchemaUpdater : FluentValidatorSchemaUpdater
    {
        protected override void Update(
            Schema schema,
            PropertyRule rule,
            InclusiveBetweenValidator validator)
        {
            if (rule.Expression.ReturnType == typeof(int))
            {
                schema.properties[rule.PropertyName].example = (int)validator.From;
                schema.properties[rule.PropertyName].minimum = (int)validator.From;
                schema.properties[rule.PropertyName].maximum = (int)validator.To;
            }
        }
    }

    public class FormatFluentValidatorSchemaUpdater
        : FluentValidatorSchemaUpdater
    {
        protected override void Update(
            Schema schema,
            PropertyRule rule,
            RegularExpressionValidator validator)
        {
            schema.properties[rule.PropertyName].format = validator.Expression;
        }
    }

Register Schema filter to Swagger configuration

Now add new schema filter, here we manually inject all the IFluentValidatorSchemaUpdater implementations, but it could also be done using dependency injection library, such as Ninject.

The implementation is pretty straightforward:

  • For each class decorated with ValidatorAttribute (from FluentValidator namespace), such as UserModel,
  • Take properties with validation rules,
  • Update Swagger schema according to registered schema updaters
 public class FluentValidationSchemaFilter : ISchemaFilter
    {
        public void Apply(Schema schema, SchemaRegistry schemaRegistry, Type type)
        {
            var updaters = new List{
                new StringMinMaxFluentValidatorSchemaUpdater(),
                new FormatFluentValidatorSchemaUpdater(),
                new RequriedFieldFluentValidatorSchemaUpdater(),
                new EmailFluentValidatorSchemaUpdater(),
                new IntMinMaxFluentValidatorSchemaUpdater()
            };

            var customAttribute = type.GetCustomAttribute();

            if (customAttribute == null)
            {
                return;
            }

            var validator = (IValidator)Activator.CreateInstance(customAttribute.ValidatorType);
            var descriptor = validator.CreateDescriptor();
            var membersWithValidations = descriptor.GetMembersWithValidators();

            foreach (var member in membersWithValidations)
            {
                var rules = descriptor.GetRulesForMember(member.Key);

                foreach (var r in rules)
                {
                    updaters.ForEach(u => u.Update(schema, (PropertyRule)r));
                }
            }
        }
    }

The last step is to register SchemaFilter in SwaggerConfig

 c.SchemaFilter();

Now after running project we will see below improvements:

  • Email is displayed not as autogenerated string, but as valid email address
  • YearOfBirth is generated as valid integer based on declared validation rules
  • When hover over the Model field, such as YearOfBirth, in pop-up it is possible to see minimum and maximum values. The same works for strings as well (min/max length)

Summary:

We have seen how to extend Swagger generated schema using FluentValidation rules.
It is a quite common technique to mark particular property/method with custom attribute, that could be used to auto-generate documentation.

Posted in ASP.NET, Documentation, WEB API | Tagged , | Leave a comment

ASP.NET Web API documentation using Swagger – Useful links

Some of good additions to Swagger might be quite difficult to find so I put useful links here:

Generating model examples:

Using markdown in descriptions:

It is possible to render really cool description in Swagger UI using Github markdown, for example nice table:

Open API specification:

Posted in Documentation, WEB API | Tagged , | Leave a comment

ASP.NET Web API documentation using Swagger – Use schema mapper for generic data types

If get back to previous post, you could see that descriptions for StartDate and EndDate are duplicated.
DateTime description shall be consistent across all API, so to not repeat yourself let’s use MapType feature of Swashbuckle configuration.

Open SwaggerConfig and find section with c.MapType (it is commented by default). Now let’s map DateTime to the specific schema:

c.MapType<System.DateTime>(() => new Swashbuckle.Swagger.Schema
{
    type = "string",
    format = "date-time",
    description = 
        @"Date-time string in <a href=""https://en.wikipedia.org/wiki/ISO_8601#UTC\"">ISO 8601 format</a>.
          Date-time without time-zone is treated as UTC."
});

Now, we could remove explicit descriptions from DiscountModel:

[Required]
public DateTime StartDate { get; set; }

[Required]
public DateTime EndDate { get; set; }

If now open Swagger page, you will see consistent descriptions for DateTime field:

One more thing is that you could override the description by using XML comments over the model property:

/// 
<summary>
/// Overriden description
/// </summary>

[Required]
public DateTime StartDate { get; set; }

Now Swagger-UI page looks like this:

Posted in Documentation, WEB API | Tagged , | Leave a comment

ASP.NET Web API documentation using Swagger – Add custom headers

In this post, we will look into how to add custom headers to your Swagger documentation using Swashbuckle.
We will add well-known “Accept-Language” header with conditionally visibility based on API method!

Setup application culture based on Accept-Language header

First, add CultureHandler and enable it into WEB API pipeline (for better understanding on how ASP.NET WEB API pipeline works and what are delegating handler, please refer to the-asp-net-web-api-2-http-message-lifecycle-in-43-easy-steps-2/

public class CultureHandler : DelegatingHandler
{
    private static readonly string[] _acceptedCultures = { "en-GB", "de-DE" };
    private static readonly string _defaultCulture = "en-GB";

    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        Thread.CurrentThread.CurrentCulture = new CultureInfo(_defaultCulture);
            
        if (request.Headers.AcceptLanguage != null)
        {
            foreach (var v in request.Headers.AcceptLanguage.OrderBy(al => al.Quality))
            {
                if (_acceptedCultures.Contains(v.Value))
                {
                    Thread.CurrentThread.CurrentCulture = new CultureInfo(v.Value);
                }
            }
        }

        return base.SendAsync(request, cancellationToken);
    }
}

Next, register CultureHandler in WebApiConfig:

public static class WebApiConfig
{
    public static void Register(HttpConfiguration config)
    {
        config.MapHttpAttributeRoutes();

        config.MessageHandlers.Add(new CultureHandler());

        config.Routes.MapHttpRoute(
            name: "DefaultApi",
            routeTemplate: "api/{controller}/{id}",
            defaults: new { id = RouteParameter.Optional }
        );
    }
}

Display Accept-Language header in Swagger UI

Ok, now it is time to add the possibility to display new header in Swagger UI.
For that, we will extend Swashbuckle behavior by introducing CultureAwareOperationFilter:

public class CultureAwareOperationFilter : IOperationFilter
{
    public void Apply(Operation operation, SchemaRegistry schemaRegistry, ApiDescription apiDescription)
    {
        if (operation.parameters == null)
        {
            operation.parameters = new List<Parameter>();
        }

        operation.parameters.Add(new Parameter
        {
            name = "Accept-Language",
            @in = "header",
            type = "string",
            required = true,
            @enum = new [] { "en-GB", "de-DE"}
        });
    }
}

Register new Operation filter in SwaggerConfig

c.OperationFilter<CultureAwareOperationFilter>();

To test new behavior let’s add new controller with simple localized content

public class ProductController : ApiController
{
    [Route("products")]
    public IHttpActionResult Get()
    {
        if (Thread.CurrentThread.CurrentCulture.Name == "de-DE")
        {
            return Ok("I am German product!");
        }

        return Ok("I am English product!");
    }
}

Now, it is time to run the project, if go by URI: http://localhost:50217/swagger/ui/index#!/Product/Product_Get, you could now see Accept-Language dropdown with English and German culture inside:

Now if select de-DE there, as a response you will get “I am German product!”.

Restricting Accept-Language header for specific APIs

If you now look at Discount API, you will see that it also got new header selection. That is quite misleading as API does not provide localization. Let’s try to fix this by extending CultureAwareOperationFilter. To make OperationFilter configuration decoupled from actual actions we will be using CultureAwareAttribute that will decorate specific methods.

public class CultureAwareAttribute : Attribute
{
}

Extend CultureAwareOperationFilter

public void Apply(Operation operation, SchemaRegistry schemaRegistry, ApiDescription apiDescription)
{
    if (!apiDescription.ActionDescriptor.GetCustomAttributes<CultureAwareAttribute>().Any())
    {
        return;
    }

    ...
}

The last step is to decorate required actions with CultureAwareAttribute,

[Route("products")]
[CultureAware]
public IHttpActionResult Get()

Now if we open Swagger, we will see language header selection only for Products API, but it won’t appear for others.

Another usage example of operation filter

The operation is displayed as dropdown if @enum property is specified.

Another alternative is to specify @defaultValue


operation.parameters.Add(new Parameter
{
name = "X-Custom-Header",
@in = "header",
type = "string",
required = true,
@default = "Default value"
});

In that case operation is displayed as textbox. That representation is often used to supply security key for API usage.

Summary:

We have looked into how to use operation filters to display custom headers in Swagger UI.
For basic usage, remember that @enum option is rendered as dropdown of choices, and as textbox otherwise.

Posted in Documentation, WEB API | Tagged , | Leave a comment