Compare JSON Response Postman

Introduction

A recent service retirement meant the data our services rely on would migrate to another. The team supporting the new service ensured backward API compatibility (paths, queries, headers, and response schemas) to make the migration seamless. They also migrated the data in batches to the new service’s storage, and validation could begin.

The validation required sending requests to both services and comparing the response bodies to ensure data was not missing or incorrect.

The data migration meant hundreds of values needed validation. With that amount of data, I knew manual testing would take a while and be prone to errors. So, I compiled a CSV file for Postman’s Collection Runner.

Having leveraged Postman’s Collection Runner for other projects, I knew it could send and record hundreds of requests with an imported data file. With that in mind, I sought to find a solution for comparing response bodies.

Comparing JSON files is often needed in software development, and I figured someone had already solved this problem. However, after searching the Internet for some time, I could not find a solution that fit my needs.

This blog aims to share the solution I created, albeit a bit more generic, so that others may learn or remix it for their projects.

Expectations

I needed a tool that could ignore defined keys and values, and one that was flexible to work with a variety of schemas. I could assert each response body contains the same values as the other, but the response body keys change per API and would require a different set of assertions per API.

Assertion

This solution will only test if there’s a difference between the two responses. It will not collect the keys or values that differ between them.

JavaScript and Lodash

Postman’s scripting supports JavaScript and comes with some dependencies, such as Lodash. Ultimately, Lodash will compare the responses, but only after removing ignorable keys and values the responses.

Postman Requests and Tests

Set up two Postman requests with templates mapping to data from the data file for the testable API.

First Request

The first request requires little scripting. Assert the status and assign the body to an environment variable.

In the first request’s Tests tab, verify a successful first response with Postman’s test method.

pm.test("Successful first response", function () {
  pm.response.to.have.status(200);
});

Finally, set the first response, as a JSON string, to an environment variable.

pm.environment.set("firstResponse", pm.response.json());

Second Request

The second request will handle the comparison and define the methods needed to remove ignorable keys and values.

In the second request’s Tests tab, verify a successful second response.

pm.test("Successful second response", function () {
  pm.response.to.have.status(200);
});

Ignorable Arrays

Next, define two string arrays. One for keysToIgnore and the other for valuesToIgnore. Both arrays define the keys or values to filter (or omit) which may differ between responses but do not affect the comparison.

JavasScript’s String.prototype.includes() (includes doc) function will use each value from those arrays to determine if that key or value includes an ignorable value.

For the migration I worked on, both services returned a self-link and links to other resources using their respective domains. Any comparison tool would flag those as different, but links were ignorable for my validation. Therefore, I added link to keysToIgnore and http to valuesToIgnore.

const keysToIgnore = ["link"];
const valuesToIgnore = ["http"];

Functions

Define shouldIgnoreKey(key) method to filter ignorable keys.

function shouldIgnoreKey(key) {
  return keysToIgnore.some((ignorableKey) =>
    key.toLowerCase().includes(ignorableKey.toLowerCase())
  );
}

Define shouldIgnoreValue(value) method to filter ignorable values.

function shouldIgnoreValue(value) {
  if (_.isString(value)) {
    return valuesToIgnore.some((ignorableValue) =>
      value.toLowerCase().includes(ignorableValue.toLowerCase())
    );
  }

  return false;
}

Define sortArrayIteratee(a) as a iteratee for sorting arrays using Lodash’s _.sortBy(collection, [iteratees=[_.identity]])(https://lodash.com/docs/#sortBy).

(If your response does not have arrays, feel free to skip defining sortArrayIteratee and update myOmittedCallback to not reference the method.)

The ordering does not matter as long as it is deterministic; i.e. as long as the iteratee produces the same sorted array for any initial starting state.

The iteratee must produce the same “ordered” array because the final assertion uses Lodash’s _.isEqual(value, other) to determine equality, and it determines array equality by index matching.

Here’s the iteratee approach I wrote for the migration, but yours should require different logic.

function sortArrayIteratee(a) {
  const aEntries = Object.entries(a);

  const [aFirstKey, aFirstValue] = aEntries[0];

  if (_.isString(aFirstValue)) {
    return _.size(aFirstValue);
  }

  // A naive approach but so far it seems to be working.
  return aFirstValue;
}

Define myOmittedCallback(callbackValue) to remove any keys or values that are ignorable. This function uses the should ignore functions defined above (shouldIgnoreKey and shouldIgnoreValue) to delete the key that has an ignorable value or is an ignorable key.

This method uses recursion to traverse the provided object, i.e. callbackValue.

function myOmittedCallback(callbackValue) {
  if (callbackValue instanceof Object) {
    for (var [key, value] of Object.entries(callbackValue)) {
      // Base Case:
      // remove key from callbackValue
      if (shouldIgnoreKey(key) || shouldIgnoreValue(value)) {
        delete callbackValue[`${key}`];
      } else if (value instanceof Object) {
        // Need to sort the array if it contains more than one element
        // because Lodash will compare arrays with index matching
        if (_.isArray(value) && _.size(value) > 1) {
          value = _.sortBy(value, sortArrayIteratee);
          callbackValue[`${key}`] = value;
        }
        // Recursive Call:
        // value is an object; so its children
        // may contain an ignorable key or value
        myOmittedCallback(value);
      }
    }
  }
}

Comparing Responses

Set both responses to local variables. The first response should be in an environment variable from the first request.

const firstResponse = pm.environment.get("firstResponse");
const secondResponse = pm.response.json();

Use Lodash’s omit function to remove the keys and values to ignore for the responses.

const omittedFirstResponse = _.omit(firstResponse, myOmittedCallback);
const omittedSecondResponse = _.omit(secondResponse, myOmittedCallback);

Use Lodash’s .isEqual() method and store result.

const result = _.isEqual(omittedFirstResponse, omittedSecondResponse);

Test Result

Use Postman’s test method to records the comparison’s result.

pm.test("Are there differences?", function () {
  pm.expect(true).to.eql(result);
});

The message (“Are there differences?”) for the test is generic, and that’s by design for this blog. By adding the tested resource’s id with dynamic variables to the message, I was able to see which tests failed from Postman’s results view.

Putting It All Together

First Request Test

pm.test("Successful first response", function () {
  pm.response.to.have.status(200);
});

pm.environment.set("firstResponse", pm.response.json());

Second Request Test

pm.test("Successful second response", function () {
  pm.response.to.have.status(200);
});

const keysToIgnore = ["link"];
const valuesToIgnore = ["http"];

function shouldIgnoreKey(key) {
  return keysToIgnore.some((ignorableKey) =>
    key.toLowerCase().includes(ignorableKey.toLowerCase())
  );
}

function shouldIgnoreValue(value) {
  if (_.isString(value)) {
    return valuesToIgnore.some((ignorableValue) =>
      value.toLowerCase().includes(ignorableValue.toLowerCase())
    );
  }

  return false;
}

function sortArrayIteratee(a) {
  const aEntries = Object.entries(a);

  const [aFirstKey, aFirstValue] = aEntries[0];

  if (_.isString(aFirstValue)) {
    return _.size(aFirstValue);
  }

  // A naive approach but so far it seems to be working.
  return aFirstValue;
}

function myOmittedCallback(callbackValue) {
  if (callbackValue instanceof Object) {
    for (var [key, value] of Object.entries(callbackValue)) {
      // Base Case:
      // remove key from callbackValue
      if (shouldIgnoreKey(key) || shouldIgnoreValue(value)) {
        delete callbackValue[`${key}`];
      } else if (value instanceof Object) {
        // Need to sort the array if it contains more than one element
        // because Lodash will compare arrays with index matching
        if (_.isArray(value) && _.size(value) > 1) {
          value = _.sortBy(value, sortArrayIteratee);
          callbackValue[`${key}`] = value;
        }
        // Recursive Call:
        // value is an object; so its children
        // may contain an ignorable key or value
        myOmittedCallback(value);
      }
    }
  }
}

const firstResponse = pm.environment.get("firstResponse");
const secondResponse = pm.response.json();

const omittedFirstResponse = _.omit(firstResponse, myOmittedCallback);
const omittedSecondResponse = _.omit(secondResponse, myOmittedCallback);

const result = _.isEqual(omittedFirstResponse, omittedSecondResponse);

pm.test("Are there differences?", function () {
  pm.expect(true).to.eql(result);
});