Generate WinMerge report file on CI server when running Approval tests

In our automation tests projects we rely on Approval tests to assert test output files. It could be for example JSON API responses or XML messages generated by integration services.

This approach worked very well for us, in local when framework asserted *.received files vs *.approved files, WinMerge windows popped up and showed the difference.

But it did not work on CI server, if some of tests failed the test report could only indicate that something was wrong. To get the real failure one has to login to CI server and compare received and approved files using manual WinMerge run. WinMerge does provide the GUI for generating compare report, but until the WinMerge 2.15.2 it was not possible to use WinMerge CLI for this.

Generate diff report using WinMerge CLI

Starting from WinMerge 2.15.2 you could execute below command:

"C:\Program Files\WinMerge\WinMergeU.exe" path_to_left_file path_to_right_file 
-minimize -noninteractive -u -or path_to_report_file

The output result:

Generate WinMerge diff file in ApprovalTests

We could add the same functionality to ApprovalTests by creating new Attribute as below:

public class CliWinMergeReporter : WinMergeReporter
	public override LaunchArgs GetLaunchArguments(string approved, string received)
		var defaultArgs = base.GetLaunchArguments(approved, received);

		var diffFileName = System.IO.Path.GetFileNameWithoutExtension(received).Replace("received", "diff") + ".html";

		var launchArguments = new LaunchArgs(
			$"{WrapPath(received)} {WrapPath(approved)} -minimize -noninteractive -u -or {WrapPath(diffFileName)}");

		return launchArguments;

Then decorate test class with `[UseReporter(typeof(CliWinMergeReporter))]`.

After test run in source folder you will get 3 files:

  1. *.received
  2. *.approved
  3. *.diff

This entry was posted in Uncategorized and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.