> ## Documentation Index
> Fetch the complete documentation index at: https://docs.myme.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Conformance

> A black-box test suite that validates any Marfa server implementation against the platform spec — for self-hosters and alternative implementations

The Marfa conformance suite is a standalone test suite that exercises any HTTP-reachable Marfa server. It treats the server as a black box — no implementation imports, no privileged access — and asserts the documented behavior: data integrity, type and permission semantics, error codes, lifecycle invariants, edge handling, and the connections substrate.

Self-hosters point it at their own deployment to validate that their instance behaves like the spec says it should. Alternative implementations of the Marfa contract use it the same way.

## What it covers

| Suite                 | What it asserts                                                                                                                                                                                                                                                                                                                                                                         |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `correctness`         | Data integrity — persistence, versioning, lifecycle, metadata, tombstones, dedup, pagination, blob CRUD, edge CRUD, edge cardinality, cycle rejection, type constraints, backrefs, query filters.                                                                                                                                                                                       |
| `compliance`          | Spec adherence — auth, type validation, permissions, error codes, type registry, namespaces, custom-type semantics, `core.media.*` subtypes, conflict detection, credential-stamped vs client-supplied fields, type inheritance, edge events, library-axis defaults, handle validation, FTS searchable-flag honouring, tenant isolation, quota configuration, cycle metadata threading. |
| `reference-connector` | End-to-end coverage of the connections runtime substrate — Integration manifest validation, inbound webhook receipt with HMAC verification, leased-token issuance/introspection/revocation, OAuth proxy error surfaces. Localhost-only — runs against a developer-machine server because the test runner spins up a mock external service on loopback.                                  |
| `performance`         | Query benchmarks (deferred).                                                                                                                                                                                                                                                                                                                                                            |
| `load`                | Sustained load profiles (deferred).                                                                                                                                                                                                                                                                                                                                                     |

The full set is in [`withmarfa/conformance`](https://github.com/withmarfa/conformance) on GitHub.

## Running

```bash theme={null}
git clone https://github.com/withmarfa/conformance
cd conformance
pnpm install

MARFA_API_URL=https://your-marfa-instance.example MARFA_API_KEY=<admin-key> pnpm test
```

Required environment:

| Variable             | Required | Description                                        |
| -------------------- | -------- | -------------------------------------------------- |
| `MARFA_API_URL`      | yes      | Target server URL.                                 |
| `MARFA_API_KEY`      | yes      | Full-access admin API key on the target.           |
| `MARFA_PERF_SCALE`   | no       | `small` (default), `medium`, `large`.              |
| `MARFA_LOAD_PROFILE` | no       | `light`, `moderate` (default), `heavy`, `extreme`. |

The suite mints its own per-file admin keys against the target, isolated by a credential-stamped `source` field. Tests clean up after themselves; a clean target instance ends the run with no leftover items.

## Capability detection

Some surfaces are gated on platform capabilities the target server may or may not implement (search backends, blob storage backends, hosted-mode-only auth flows). The suite calls a capability-detection probe at startup and skips capability-dependent tests when the target doesn't advertise support. Skipped tests are visible in the run summary; they're not failures.

## Black-box gaps

A small number of behaviors can't be validated black-box and are tracked as documented gaps in the suite:

* **Cross-tenant isolation in `keys` mode.** A keys-mode server has no public path to mint a non-platform key with an arbitrary `tenant_id`; tenant scoping is exercised at the data-plane level but full multi-tenant isolation needs a hosted-mode target.
* **Quota 429 enforcement.** Same root cause — exercising the 429 response requires minting a tenant-bound non-platform key. Configuration CRUD is fully covered.
* **Webhook delivery end-to-end.** The target needs to reach the test runner's HTTP server to deliver a callback; for runs against a remote target this isn't possible. The webhook subscription surface is covered; delivery is documented as a gap.
* **Connector reactive fanout observability.** Needs a connector emitting reactive runs against the target. Header-acceptance and cycle-metadata acceptance are testable shims; live fanout observability is gap-marked.
* **OAuth scope validation on the data plane.** Validated at `/auth/authorize` and `/auth/token`; data-plane drop of malformed scopes is non-observable from a black-box position.

These are documented inline in the relevant test files rather than skipped silently.

## When to run it

* Before deploying a self-hosted Marfa to production — validate the build matches what's documented.
* After upgrading a self-hosted Marfa to a new release — catch regressions in the same window the platform team caught them in upstream.
* During development of an alternative Marfa implementation — the suite is the contract.

The platform team runs it continuously against the canonical implementation as part of release gating.