Skip to main content

Testing Your Policy

Self-hosted testing.

Using Limitr Cloud?

Policy validation is built into the Cloud publishing workflow. Cloud maintains separate test and live environments — make changes in test, verify behavior in your development environment connected to the test policy, publish to live when it's right. You don't need a separate test suite for policy correctness in Cloud. The rest of this guide is for developers using the local open source engine.

What the built-in validator catches​

Limitr.new() validates your policy by default before returning. If the policy is invalid, it throws with the validation error.

try {
  const policy = await Limitr.new(policyYaml, 'yaml');
} catch (err) {
  console.error('Invalid policy:', err.message);
  // e.g. 'A credit named "sonnet_inpput" does not exist in this policy'
}

You can also call valid() explicitly on an already-loaded policy:

const [valid, error] = await policy.valid();
if (!valid) console.error(error);

The validator checks credit references, unit strings, limit and topup values, tier structure, entitlement and topup schema conformance, and trial period configuration.

What the validator does not check:

  • Whether your credit prices and costs make economic sense
  • Whether your limits are appropriate for your product
  • Whether your exchange table chains resolve correctly for your use case
  • Whether enforcement behaves the way you expect at runtime

Those require behavioral tests.

Validation as a CI gate​

The simplest thing you can do: load your policy file in CI and let the built-in validator fail the build on an invalid policy. No test framework required.

scripts/validate-policy.ts
import { Limitr } from '@formata/limitr';
import { readFileSync } from 'fs';

const policyPath = process.argv[2] ?? './policy.yaml';

try {
  const policy = await Limitr.new(readFileSync(policyPath, 'utf-8'), 'yaml');
  const version = await policy.version();
  console.log(`✓ Policy valid (Limitr ${version})`);
  process.exit(0);
} catch (err) {
  console.error(`✗ Invalid policy: ${(err as Error).message}`);
  process.exit(1);
}
package.json
"scripts": {
  "validate": "bun run scripts/validate-policy.ts policy.yaml"
}
.github/workflows/ci.yml
- name: Validate Limitr policy
  run: bun run validate

Writing enforcement tests​

Enforcement tests verify that allow(), increment(), and check() behave correctly for the scenarios your product depends on. Use your existing test framework — Limitr is just TypeScript.

Setup pattern​

import { Limitr } from '@formata/limitr';
import { readFileSync } from 'fs';

async function loadPolicy() {
  return Limitr.new(readFileSync('./policy.yaml', 'utf-8'), 'yaml');
}

Create a fresh policy instance per test or test file. Limitr is in-process — there's no server to reset and no shared state. Each Limitr.new() call is a clean slate.

Testing hard limits​

test('starter hard limit blocks at 500K input tokens', async () => {
  const policy = await loadPolicy();
  await policy.createCustomer('test_user', 'starter');

  const allowed = await policy.allow('test_user', 'chat_input', 500_000);
  expect(allowed).toBe(true);

  const blocked = await policy.allow('test_user', 'chat_input', 1);
  expect(blocked).toBe(false);

  const remaining = await policy.remaining('test_user', 'chat_input');
  expect(remaining).toBe(0);
});

Testing soft limits and events​

test('growth soft limit fires meter-overage event', async () => {
  const policy = await loadPolicy();
  await policy.createCustomer('test_user', 'growth');

  const events: string[] = [];
  policy.addHandler('test', (key) => events.push(key));

  await policy.allow('test_user', 'chat_input', 2_000_000); // at limit
  await policy.allow('test_user', 'chat_input', 1);          // over limit

  expect(events).toContain('meter-overage');
  expect(events).not.toContain('meter-limit');
});

Testing grant coverage​

test('grant covers overage before meter-overage fires', async () => {
  const policy = await loadPolicy();
  await policy.createCustomer('test_user', 'growth');
  await policy.applyCustomerTopup('test_user', 'monthly_credits');

  const overageEvents: unknown[] = [];
  policy.addHandler('test', (key, value) => {
    if (key === 'meter-overage') overageEvents.push(JSON.parse(value as string));
  });

  await policy.allow('test_user', 'chat_input', 2_000_000); // at limit
  await policy.allow('test_user', 'chat_input', 100_000);   // into overage — covered by grant

  expect(overageEvents).toHaveLength(0); // grant covered it

  await policy.allow('test_user', 'chat_input', 10_000_000); // exhaust the grant

  expect(overageEvents.length).toBeGreaterThan(0);
});

Testing boolean feature gates​

test('starter plan does not have advanced analytics', async () => {
  const policy = await loadPolicy();
  await policy.createCustomer('test_starter', 'starter');
  await policy.createCustomer('test_growth', 'growth');

  expect(await policy.check('test_starter', 'advanced_analytics')).toBe(false);
  expect(await policy.check('test_growth', 'advanced_analytics')).toBe(true);
});

Testing customer overrides​

test('customer override replaces plan limit', async () => {
  const policy = await loadPolicy();
  await policy.createCustomer('test_user', 'starter');

  expect(await policy.limit('test_user', 'chat_input', false)).toBe(500_000);

  await policy.createCustomerOverride('test_user', 'chat_input', 1_000_000);
  expect(await policy.limit('test_user', 'chat_input', false)).toBe(1_000_000);

  const allowed = await policy.allow('test_user', 'chat_input', 750_000);
  expect(allowed).toBe(true);

  await policy.removeCustomerOverride('test_user', 'chat_input');
  expect(await policy.limit('test_user', 'chat_input', false)).toBe(500_000);
});

Testing policy changes with difference()​

difference() computes a structured diff between two policy instances — useful for verifying that a policy change affects exactly what you intended and nothing else.

test('adding enterprise plan does not change starter or growth', async () => {
  const before = await Limitr.new(readFileSync('./policy.yaml', 'utf-8'), 'yaml');
  const after  = await Limitr.new(readFileSync('./policy.new.yaml', 'utf-8'), 'yaml');

  // before is treated as the schema for the diff
  const diff = await before.difference(after);

  const changedKeys = Object.keys(diff);
  expect(changedKeys).toContain('enterprise');
  expect(changedKeys).not.toContain('starter');
  expect(changedKeys).not.toContain('growth');
});

difference(other, symmetric?) — when symmetric is false (default), only changes relative to the calling policy's structure are returned. When true, changes in both directions are included.

What to test before shipping a pricing change​

When changing limits, adding a plan, or adjusting credit prices — not just authoring a policy for the first time — test the specific change:

Limit changes — Test that the new limit blocks or allows at the right value. Test that existing customers whose meters are already near the old limit behave correctly under the new one.

New plan — Test that all entitlements gate correctly. Test that a customer on the new plan can't access entitlements from other plans. Test upgrade and downgrade paths to and from the new plan.

Credit price changes — These don't affect enforcement, but verify customerMarginSnapshot() returns the expected margin values if your billing code depends on it.

Exchange table changes — Test that creditExchange() returns the expected conversion values. Test that grants are drawn correctly at the new exchange rate.

New topup — Test that applyCustomerTopup() creates a grant with the right starting value. Test that the grant is drawn before overage fires. Test expiry if expires_after is set.

Last updated on