Contract tests for AI-generated API code: check the real JSON response

Call the actual local HTTP handler and assert the parsed response against a written requirement. A direct function test can pass while serialization changes what the client receives. A mocked response can miss the same gap.

This walkthrough answers one narrow question: how do you catch a required nullable JSON field disappearing from an AI-assisted API change? You will run a synthetic two-file example, observe a real assertion failure, make one correction, and rerun it. Nothing here calls a production service or uploads your code.

Use Node.js 22 in a temporary directory. Both files use built-in modules, so there is no package install and no account requirement.

Our invented contract is: GET /profile returns status 200 and JSON containing id and nickname. For this fixed fixture, the ID is u-1. When no nickname exists, the response must include nickname: null; omission has a different meaning to the client.

That is a choice for this example, not a universal API rule. Another API may deliberately omit optional fields. Read its schema and consumer expectations before deciding which behavior is correct.

Save this plausible implementation as profile.mjs:

export function profile() { return { id: 'u-1', nickname: undefined }; }

The object has a nickname property. But JSON cannot represent undefined, and JSON.stringify omits an object property with that value. A consumer cannot see that the property existed before serialization.

Save the following complete file as profile.test.mjs. The first assertion checks the object in memory. The second starts a real loopback server on an available port and checks the response a client reads.

import test from 'node:test';
import assert from 'node:assert/strict';
import {createServer} from 'node:http';
import {profile} from './profile.mjs';

test('the direct object contains a nickname key', () => {
  assert.equal(Object.hasOwn(profile(), 'nickname'), true);
});

test('GET /profile sends an explicit null nickname', async (t) => {
  const server = createServer((req, res) => {
    if (req.method !== 'GET' || req.url !== '/profile') {
      res.writeHead(404).end();
      return;
    }
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify(profile()));
  });
  await new Promise((resolve, reject) => {
    server.once('error', reject);
    server.listen(0, '127.0.0.1', resolve);
  });
  t.after(() => new Promise((resolve, reject) => {
    server.close((error) => error ? reject(error) : resolve());
    server.closeAllConnections();
  }));
  const {port} = server.address();
  const response = await fetch('http://127.0.0.1:' + port + '/profile');
  assert.equal(response.status, 200);
  assert.match(response.headers.get('content-type'), /^application\/json\b/);
  assert.deepEqual(await response.json(), { id: 'u-1', nickname: null });
});

Run node --test profile.test.mjs. The direct test passes because the object owns the key. The HTTP test fails at the deep equality assertion: the actual parsed body is { id: "u-1" }, missing nickname: null. The observed total is one passing test and one failing test.

The server binds only to 127.0.0.1. Port zero asks the operating system for an available port; the test reads that assigned port. Cleanup closes the server and its connections, including when the assertion fails.

We do not replace fetch with a function returning a prepared object. That would bypass the handler and JSON boundary this test is intended to cover.

Replace profile.mjs with:

export function profile() { return { id: 'u-1', nickname: null }; }

Run the same test command again. Both tests pass. The HTTP assertion now receives exactly { id: "u-1", nickname: null }. The fix changes the value sent across the boundary; it does not weaken the expected response.

Check the failure reason before trusting this red-to-green result. A syntax error, missing import, or occupied port would not prove the response contract. Here, the server returned successfully and the client parsed its JSON; the failure concerned the missing field.

This small server is a teaching harness. In your repository, use the framework's supported test server or app entry point, so the test reaches the same routing and serialization code as the application. Rebuilding the handler independently in the test could hide a different production defect.

1. Choose one documented response behavior a consumer depends on.
2. Seed a bounded fixture at the storage boundary when needed. Keep the actual handler and serializer in the path.
3. Assert status, media type, and the relevant parsed payload using expectations derived from the contract.
4. Run on the broken change, confirm an assertion failure, correct the change, and rerun the focused and repository checks.

Do not always assert an entire response. This fixture has an exact two-field contract, so deep equality is appropriate. An extensible API may allow extra fields; assert required keys, types, and meanings without treating a permitted addition as a regression. Avoid brittle string comparisons that fail merely because JSON key order changes.

When reviewing an agent's test, identify every mocked boundary. A mock database can be reasonable for a handler contract test. A mocked HTTP response cannot establish what that handler actually sends.

This example verifies one successful response and its JSON representation. It does not prove authentication, ownership, invalid-input handling, persistence, network retries, browser behavior, or compatibility with another independently deployed service. It is not a consumer-driven contract-testing platform or a complete API test strategy.

Keep a review note with the requirement, failing assertion, correction, checks run, and those remaining gaps. A real integration test with the database and a consumer compatibility check may be the next layer; a passing local server test cannot substitute for them.

For the underlying behavior, see MDN's JSON.stringify reference. The Node.js 22 HTTP documentation covers the server, and the Node.js 22 test runner documentation describes tests and cleanup hooks. Sources checked October 4, 2026.

The useful manual rep is to explain which boundary an assertion exercises, then write an expectation from the contract yourself. Ask a coding agent to pause on that bounded scope while you do the work.

ManualMode's free review exercise uses synthetic patches; it does not inspect your API or verify the example above. For a real repository task, the local test remains your evidence. ManualMode offers three Gym reps and one Project rep free after signup, with raw project source staying local by default.