Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.myme.so/llms.txt

Use this file to discover all available pages before exploring further.

The Marfa conformance suite is a standalone test suite that exercises any HTTP-reachable Marfa server. It treats the server as a black box — no implementation imports, no privileged access — and asserts the documented behavior: data integrity, type and permission semantics, error codes, lifecycle invariants, edge handling, and the connections substrate. Self-hosters point it at their own deployment to validate that their instance behaves like the spec says it should. Alternative implementations of the Marfa contract use it the same way.

​
What it covers

SuiteWhat it asserts
correctnessData integrity — persistence, versioning, lifecycle, metadata, tombstones, dedup, pagination, blob CRUD, edge CRUD, edge cardinality, cycle rejection, type constraints, backrefs, query filters.
complianceSpec adherence — auth, type validation, permissions, error codes, type registry, namespaces, custom-type semantics, core.media.* subtypes, conflict detection, credential-stamped vs client-supplied fields, type inheritance, edge events, library-axis defaults, handle validation, FTS searchable-flag honouring, tenant isolation, quota configuration, cycle metadata threading.
reference-connectorEnd-to-end coverage of the connections runtime substrate — Integration manifest validation, inbound webhook receipt with HMAC verification, leased-token issuance/introspection/revocation, OAuth proxy error surfaces. Localhost-only — runs against a developer-machine server because the test runner spins up a mock external service on loopback.
performanceQuery benchmarks (deferred).
loadSustained load profiles (deferred).
The full set is in withmarfa/conformance on GitHub.

​
Running

git clone https://github.com/withmarfa/conformance
cd conformance
pnpm install

MARFA_API_URL=https://your-marfa-instance.example MARFA_API_KEY=<admin-key> pnpm test
Required environment:
VariableRequiredDescription
MARFA_API_URLyesTarget server URL.
MARFA_API_KEYyesFull-access admin API key on the target.
MARFA_PERF_SCALEnosmall (default), medium, large.
MARFA_LOAD_PROFILEnolight, moderate (default), heavy, extreme.
The suite mints its own per-file admin keys against the target, isolated by a credential-stamped source field. Tests clean up after themselves; a clean target instance ends the run with no leftover items.

​
Capability detection

Some surfaces are gated on platform capabilities the target server may or may not implement (search backends, blob storage backends, hosted-mode-only auth flows). The suite calls a capability-detection probe at startup and skips capability-dependent tests when the target doesn’t advertise support. Skipped tests are visible in the run summary; they’re not failures.

​
Black-box gaps

A small number of behaviors can’t be validated black-box and are tracked as documented gaps in the suite:
  • Cross-tenant isolation in keys mode. A keys-mode server has no public path to mint a non-platform key with an arbitrary tenant_id; tenant scoping is exercised at the data-plane level but full multi-tenant isolation needs a hosted-mode target.
  • Quota 429 enforcement. Same root cause — exercising the 429 response requires minting a tenant-bound non-platform key. Configuration CRUD is fully covered.
  • Webhook delivery end-to-end. The target needs to reach the test runner’s HTTP server to deliver a callback; for runs against a remote target this isn’t possible. The webhook subscription surface is covered; delivery is documented as a gap.
  • Connector reactive fanout observability. Needs a connector emitting reactive runs against the target. Header-acceptance and cycle-metadata acceptance are testable shims; live fanout observability is gap-marked.
  • OAuth scope validation on the data plane. Validated at /auth/authorize and /auth/token; data-plane drop of malformed scopes is non-observable from a black-box position.
These are documented inline in the relevant test files rather than skipped silently.

​
When to run it

  • Before deploying a self-hosted Marfa to production — validate the build matches what’s documented.
  • After upgrading a self-hosted Marfa to a new release — catch regressions in the same window the platform team caught them in upstream.
  • During development of an alternative Marfa implementation — the suite is the contract.
The platform team runs it continuously against the canonical implementation as part of release gating.