M0.5 spike: OpenMLS 0.9 vs mls-rs 0.56

Status: spike report · Date: 2026-09-04 · Evidence lives in spikes/mls; every claim below that says "tested" has a passing test there, run in CI. Answers Q1 and Q2 of the Phase 0 plan and issue #17.

Summary

Both libraries can do what the design needs. Neither blocks ADR-002. The choice is about shape, not capability:

• mls-rs persists only when the application says so, enforces policy through a rules trait on both sending and receiving, and exposes custom proposals and extensions as first-class features. Its stable crypto providers are C-backed (OpenSSL, AWS-LC); its pure-Rust provider is labelled experimental.
• OpenMLS persists through a provider on every operation, hands the application a staged commit to inspect, ships pure-Rust and formally verified crypto providers, and has a production precedent on mobile through Wire's core-crypto. Its own README lists Android, iOS and WASM as compiled but untested.

Proposed decision: integrate mls-rs for the rest of Phase 0, with one named condition to verify before Phase 3 (mobile crypto provider). Reasoning in the last section. The decision is recorded as proposed in ADR-002 and is cheap to reverse: the spike shows the integration surface of the two libraries is equivalent for our purposes.

Q1: group state in the application's transaction

The domain model requires that a send unit (MLS state + event + ciphertext + delivery attempts) and a receive unit (delivery mark + MLS change + event + pending work) each commit as one SQLite transaction (Domain model §5).

	mls-rs	OpenMLS
Write model	Explicit: nothing reaches storage until Group::write_to_storage(). Tested: load_group fails before the call and succeeds after	Write-through: every operation calls the StorageProvider (11 key-value writes for create + add + merge in the test)
Storage trait	GroupStateStorage: 4 methods (state, epoch, write, max_epoch_id), one state blob per group plus epoch records	StorageProvider: 72 methods over typed keys; the reference in-memory implementation reduces them to six helpers, which is what the spike ported to SQLite
Shared transaction	Tested. Provider bound to the application's connection; outbox insert + write_to_storage inside BEGIN: rollback leaves 0/0 rows and no loadable group, commit leaves 1/1 and a loadable group	Tested. Same connection; outbox insert + MlsGroup::new + add_members + merge_pending_commit inside BEGIN: rollback leaves only the pre-existing signer row, commit leaves 12 rows and the group loads at epoch 1
Official SQLite provider	mls-rs-provider-sqlite	openmls_sqlite_storage
Consequence for the core	The core decides the moment of persistence; one write per unit of work	The provider must hold the connection for the duration of each library call; the transaction boundary is set outside the library and every intermediate write lands inside it

Both answer Q1 positively. The mls-rs model matches the domain model's "unit of work" vocabulary directly. The OpenMLS model works but couples atomicity to how the provider is wired, which is easier to get subtly wrong in the real core (a provider on a different connection silently breaks the guarantee).

Q2: refuse a valid commit from an unauthorized committer

The protocol requires a committer policy carried in the authenticated group context and enforced by every client before state changes (Protocol §5).

	mls-rs	OpenMLS
Hook	MlsRules::filter_proposals(direction, source, roster, context, proposals), called for Send and Receive, with CommitSource::ExistingMember { index, signing_identity, .. }	process_message returns ProcessedMessageContent::StagedCommitMessage; the application reads sender, add/remove/update proposals and the group context, then calls merge_staged_commit or drops it
Tested	Leaf 1 commits an Add; leaf 0's client with the policy fails process_incoming_message with our error text; epoch unchanged; leaf 0 can still commit (epoch 2) and leaf 1 follows and decrypts	Leaf 1 commits an Add; leaf 0 inspects the staged commit (sender leaf 1, one Add), does not merge; epoch unchanged
Sender side	The same rule refuses to produce an unauthorized commit, so an honest client cannot violate the policy by mistake	Left to application code around add_members and friends
Where the policy lives	Rules object built into the client; has roster and GroupContext available, so it can read our extension	Application code at the call site; has StagedCommit::group_context()
External commits	Refused by the same hook (CommitSource::NewMember)	Refused by not merging; also gated at join config

Both answer Q2 positively. mls-rs's symmetric enforcement is slightly better aligned with "all clients validate the policy before accepting state"; OpenMLS's staged commit is more transparent when debugging.

Facts that are not tests

Aspect	mls-rs 0.56.0	OpenMLS 0.9.0
License	Apache-2.0 OR MIT	MIT
Maintainer	AWS (awslabs)	Phoenix R&D and Cryspen
Minimum Rust	1.82	1.91
Mandatory suite (MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519)	Yes, all providers	Yes
Crypto providers	OpenSSL and AWS-LC stable; RustCrypto and WebCrypto experimental (README)	RustCrypto and libcrux (formally verified primitives); both first-class
Custom GroupContext extension for the policy	MlsExtension trait, custom_proposal feature	Extension::Unknown(u16, UnknownExtension)
Late/out-of-order messages	out_of_order, prior_epoch features; retention via storage max_epoch_retention	MlsGroupJoinConfig::max_past_epochs
Fork handling helpers	None specific	fork-resolution feature with a manual chapter
Debug features to forbid in release	None equivalent found	content-debug, crypto-debug (transitive check required)
Bindings for Flutter/mobile	mls-rs-uniffi crate on crates.io (0.1.0)	Wire's core-crypto ships OpenMLS on iOS/Android via UniFFI (external precedent)
WASM	Supported (README)	Compiled, untested (README)
Mobile targets	Not stated in README	Android, iOS, WASM compiled in CI, marked unsupported
Security review	"Validated for RFC 9420 conformance, no full third-party audit" (README)	No audit statement in README
Storage of secrets	Zeroizing blobs	Serialized JSON values per key (the provider is responsible for encryption at rest)

Risks by choice

If mls-rs: the mobile crypto provider. The stable providers wrap C libraries. aws-lc-rs cross-compiles for iOS and Android but needs a C toolchain and cmake in the build; the pure-Rust provider is experimental by the maintainers' own label. This must be verified as a Phase 3 gate: either aws-lc-rs builds and passes the MLS test vectors on both mobile targets, or the RustCrypto provider is promoted to stable upstream. If neither, switch to OpenMLS.

If OpenMLS: persistence discipline. Every operation writes through the provider, so the core must guarantee that the provider's connection is the transaction's connection, always, and must tolerate ~10 writes per operation under SQLCipher. Mobile is de-risked by Wire's precedent but not by OpenMLS's own CI. The two debug features must be provably absent from release builds.

Proposed decision and why

Integrate mls-rs for milestones M0.5 step 4 through M0.6.

1. The explicit write model is the cleanest fit for the transactional units in the domain model, and it removes a whole class of "provider on the wrong connection" bugs.
2. MlsRules puts the committer policy in one place, enforced on both send and receive, with the roster and group context in hand. That is exactly how the protocol describes it.
3. Custom proposals and extensions are supported features rather than Unknown escape hatches, which matters for the deterministic-successor design under evaluation for the coordinator.
4. The one serious risk, mobile crypto, is verifiable with a build, and the fallback is a library whose integration surface this spike has already exercised.

What would flip the decision: failure of the Phase 3 mobile-provider gate, or an OpenMLS security audit being published while mls-rs remains unaudited, since parity of review would remove the main argument against OpenMLS's heavier persistence model.

What the spike did not cover

Key package and PSK storage over the same transaction (mls-rs KeyPackageStorage, OpenMLS's remaining labels): the same pattern applies and is part of the integration step. Epoch retention trimming. Performance of either persistence model under SQLCipher. Interop between the two libraries. Official test vectors, which are the first task of the integration step (#18).