feat: import Chinese-localized Buzz source snapshot
Docker image / Build (linux/amd64) (push) Has been cancelled
Docker image / Build (linux/arm64) (push) Has been cancelled
Docker image / Merge release multi-arch manifest (push) Has been cancelled
Docker image / Merge debug multi-arch manifest (push) Has been cancelled
Docker image / Build public push gateway (linux/amd64) (push) Has been cancelled
Docker image / Build public push gateway (linux/arm64) (push) Has been cancelled
Docker image / Publish public push gateway image (push) Has been cancelled
Sprig image / Build (linux/amd64) (push) Has been cancelled
Sprig image / Build (linux/arm64) (push) Has been cancelled
Sprig image / Merge multi-arch manifest (push) Has been cancelled
Harbor Buzz Orchestra / Python tests and lint (push) Has been cancelled
CI / Detect Changed Paths (push) Has been cancelled
CI / Rust Lint (push) Has been cancelled
CI / Unit Tests (push) Has been cancelled
CI / Desktop Core (push) Has been cancelled
CI / Desktop Smoke E2E (1) (push) Has been cancelled
CI / Desktop Smoke E2E (2) (push) Has been cancelled
CI / Desktop Smoke E2E (3) (push) Has been cancelled
CI / Desktop Smoke E2E (4) (push) Has been cancelled
CI / Desktop (push) Has been cancelled
CI / Desktop E2E Relay (push) Has been cancelled
CI / Desktop E2E Integration (1/2) (push) Has been cancelled
CI / Desktop E2E Integration (2/2) (push) Has been cancelled
CI / Desktop E2E Integration (push) Has been cancelled
CI / Backend Integration (relay e2e) (push) Has been cancelled
CI / Relay E2E (push) Has been cancelled
CI / Web (push) Has been cancelled
CI / Mobile (push) Has been cancelled
CI / Security (push) Has been cancelled
CI / Dead Token Reference Guard (push) Has been cancelled
CI / Server Cross-Compile (aarch64-unknown-linux-musl) (push) Has been cancelled
CI / Server Cross-Compile (x86_64-unknown-linux-musl) (push) Has been cancelled
CI / Windows Rust (x86_64-pc-windows-msvc) (push) Has been cancelled
CI / Desktop Build (macOS) (push) Has been cancelled
helm chart / lint + unittest + render matrix (push) Has been cancelled
helm chart / install on kind (gated) (push) Has been cancelled
helm chart / publish chart to GHCR (push) Has been cancelled
Mesh Lifecycle / Relay-Driven Mesh Lifecycle Smoke (push) Has been cancelled
Sprig / Build (aarch64-unknown-linux-musl) (push) Has been cancelled
Sprig / Build (x86_64-unknown-linux-musl) (push) Has been cancelled
Sprig / Publish rolling release (push) Has been cancelled
Sprig / Publish tagged release (push) Has been cancelled

Signed-off-by: cls_宁波本机 <908705107@qq.com>
This commit is contained in:
2026-08-13 18:34:25 +08:00
parent 61c3fa1df9
commit 9dfa06ffee
3785 changed files with 1085458 additions and 2 deletions
+16
View File
@@ -0,0 +1,16 @@
# Stateful gateway safety model
The public gateway persists installation authority, encrypted APNs-token custody, relay delegations, replay reservations, and endpoint quotas in PostgreSQL. The relay separately owns lease matching, event authorization, coalescing, and durable delivery jobs.
The bounded executable model in `nip-pl/delivery.py` checks:
1. delivery requires the NIP-98 signer sealed into the grant;
2. installation, delegation, epoch, generation, and both expiries are live at admission;
3. revocation/rotation and admission are ordered by one durable authority transaction;
4. every admitted NIP-98 event id is burned, terminal request ids remain burned, and transient request ids are released only after disposition;
5. quota is charged for every admitted attempt and never refunded;
6. APNs-token custody failure cannot send;
7. every actual send body is the byte constant registered by NIP-PL; and
8. old-epoch grants cannot resurrect after endpoint rotation.
`nip-pl/delivery_mutation.py` weakens signer, epoch, terminal-burn, quota, and fixed-body checks and requires each mutant to be caught. The model does not claim exactly-once provider delivery, model PostgreSQL implementation details, or cover the not-yet-shipped relay matcher/worker.
+37
View File
@@ -0,0 +1,37 @@
---
title: "NIP-PL Formal Models: lease acceptance and stateful gateway authority"
tags: [nostr, nip-pl, push-notifications, formal-model, buzz]
status: active
created: 2026-07-11
---
# NIP-PL formal pressure test
These bounded executable models cover two distinct shipped contracts. They do not model the not-yet-shipped relay matcher/worker or any removed wake-grant protocol.
## Run
```bash
python3 acceptance.py
python3 mutation_test.py
python3 delivery.py
python3 delivery_mutation.py
python3 fixed_payload.py
python3 fixed_payload_mutation.py
```
## Lease acceptance
`acceptance.py` explores all 5040 orderings of one address's active, revoke, reactivate, replay, NIP-01-tie, high-generation/old-created-at, and high-created-at/stale-generation candidates. It checks no resurrection, monotone watermarks, no watermark poisoning, agreement between stored and effective state, and replay-window safety. `mutation_test.py` independently drops the NIP-01 and generation clauses and requires both mutants to produce witnesses.
## Stateful public gateway
`delivery.py` models the authority actually shipped by the public gateway: relay signer confinement; installation/delegation epoch, generation, and expiry; atomic replay/quota admission; revocation ordering; custody; terminal request burn versus transient release; and the exact constant APNs body. `delivery_mutation.py` requires signer, epoch, terminal-burn, quota-refund, and fixed-body mutants to be detected.
## Fixed payload
`fixed_payload.py` exhaustively varies relay-controlled and gateway-state inputs and requires the APNs body to remain byte-identical to the normative constant. `fixed_payload_mutation.py` injects each prohibited input category and requires every mutation to be caught.
## Honest limits
The models enumerate bounded abstract transitions, not SQL schedules or network behavior. Real PostgreSQL race/FK/retention tests validate the implementation separately. A crash after APNs accepts but before disposition persistence remains intentionally at-least-once; request expiry bounds the resulting replay reservation. Constant payload prevents content disclosure but cannot hide wake timing or frequency.
+180
View File
@@ -0,0 +1,180 @@
"""Formal model of NIP-PL acceptance + lifecycle (PLANS/NIP_PL_PUSH_LEASES_DRAFT.md).
Exhaustive finite-state exploration of ONE lease address (author, 30350, d) under
an adversary who can present a bounded universe of candidate events -- including
forged (generation, created_at) combinations and replays -- in every order.
We check the safety/lifecycle invariants the spec ASSERTS as atomic/monotone:
I1 no-resurrection : once a tombstone (active:false) is the effective state,
no later ACCEPTED event may make the address effective-active
unless it strictly beats the tombstone on BOTH orderings.
(spec Acceptance check 8 + Lifecycle "replayed older event
can never resurrect a revoked lease")
I2 watermark-monotone : the persisted generation watermark never decreases, and
a REJECTED event never changes stored/effective/watermark.
(check 8: "leave stored event, effective push state, and
watermark all unchanged")
I3 no-watermark-poison : a high-generation / old-created_at event that LOSES the
NIP-01 ordering is rejected and MUST NOT raise the watermark.
(check 8 trap the spec calls out by name)
I4 dual-order-agree : the accepted (stored) event and the effective push state are
always the same event -- REQ view never disagrees with effect.
I5 replay-window : after natural expiry / tombstone retention, any replay of a
formerly-valid event fails (expiration lower bound), so the
watermark can be released without reopening resurrection.
Modeled acceptance sequence (spec "Acceptance and Origin Binding", ordered):
a candidate is ACCEPTED iff it passes structural checks (we assume the adversary
only ever submits structurally valid, correctly-signed, origin-bound events -- we
are testing ORDERING, not parsing) AND wins check 8:
(a) NIP-01 addressable ordering vs current stored winner:
greater created_at, tie -> lexically-lowest id ;
(b) generation strictly greater than the internal watermark.
BOTH required. Failing either -> reject, no state change.
On accept: commit (stored, effective, watermark) atomically; watermark := gen.
"""
from itertools import permutations
class Ev:
__slots__ = ("id", "gen", "created", "active")
def __init__(self, eid, gen, created, active):
self.id, self.gen, self.created, self.active = eid, gen, created, active
def __repr__(self):
s = "A" if self.active else "T" # active / tombstone
return f"{self.id}[g{self.gen},c{self.created},{s}]"
def nip01_beats(cand, cur):
"""NIP-01 addressable ordering: higher created_at; tie -> lexically LOWEST id."""
if cur is None:
return True
if cand.created != cur.created:
return cand.created > cur.created
return cand.id < cur.id # lower id wins the tie
class Address:
"""One (author,30350,d). Faithful encoding of acceptance check 8."""
def __init__(self):
self.stored = None # currently-stored winning event (what REQ serves)
self.effective_active = False # effective push state: matching on?
self.watermark = -1 # internal generation watermark
self.wm_history = [-1] # to check monotonicity
self.log = [] # (event, accepted?)
def submit(self, ev):
# check 8: must win BOTH orderings
wins_nip01 = nip01_beats(ev, self.stored)
wins_gen = ev.gen > self.watermark
if wins_nip01 and wins_gen:
# atomic commit
self.stored = ev
self.effective_active = ev.active
self.watermark = ev.gen
self.wm_history.append(self.watermark)
self.log.append((ev, True))
return True
else:
# MUST leave stored, effective, watermark unchanged
self.log.append((ev, False))
return False
def explore():
# Adversarial candidate universe for ONE address.
# ids chosen so we can force NIP-01 ties (same created, different id).
# Includes: an active lease, a higher-gen tombstone (legit revoke),
# a replayed OLD active event with a FORGED high generation (poison attempt),
# a same-created_at tie pair, and a stale low-gen active (resurrection attempt).
universe = [
Ev("e1", gen=1, created=100, active=True), # initial active lease
Ev("e2", gen=2, created=200, active=False), # legit revocation (tombstone)
Ev("e3", gen=9, created=150, active=True), # POISON: high gen, but created_at
# < tombstone e2 -> loses NIP-01
Ev("e4", gen=3, created=250, active=True), # legit reactivation (beats both)
Ev("e5", gen=1, created=100, active=True), # exact replay of e1 (stale both)
Ev("a1", gen=5, created=200, active=True), # NIP-01 tie with e2 (created=200);
# id "a1" < "e2" -> a1 wins NIP-01
Ev("z1", gen=0, created=300, active=True), # clause-(b) witness: highest
# created_at, STALE gen -> only
# the watermark rejects it
]
viol = {k: [] for k in ("I1", "I2", "I3", "I4", "I5")}
n = 0
# exhaust every ordering of every non-empty subset up to full universe.
# full permutation of all 6 = 720; we also test all shorter prefixes via
# permutations of the whole set (prefix coverage) -- and specifically every
# ordering that ends after a tombstone to probe resurrection.
from itertools import permutations as P
for perm in P(universe):
n += 1
addr = Address()
tomb_seen_effective = False
for ev in perm:
wm_before = addr.watermark
stored_before = addr.stored
eff_before = addr.effective_active
accepted = addr.submit(ev)
# I2: rejected event changes nothing
if not accepted:
if (addr.watermark != wm_before or addr.stored is not stored_before
or addr.effective_active != eff_before):
viol["I2"].append((perm, ev, "rejected event mutated state"))
# I2 (mono): watermark never decreases
if addr.watermark < wm_before:
viol["I2"].append((perm, ev, "watermark decreased"))
# I3: an event that LOSES nip01 but has high gen must NOT raise watermark
if not nip01_beats(ev, stored_before) and ev.gen > wm_before:
if addr.watermark != wm_before:
viol["I3"].append((perm, ev, "watermark poisoned by nip01-loser"))
# I4: stored event == effective source (never disagree)
if addr.stored is not None:
if addr.effective_active != addr.stored.active:
viol["I4"].append((perm, ev, "stored/effective disagree"))
if addr.effective_active is False and addr.stored is not None \
and not addr.stored.active:
tomb_seen_effective = True
# I1: once effective state is a tombstone, resurrection requires beating
# BOTH orderings. Detect: we were tombstoned, then became active.
if tomb_seen_effective and addr.effective_active:
# legitimate only if the reactivating event beat the tombstone on both.
# e4 (gen3,created250) is the only legit reactivator here.
if not (accepted and ev is not None and ev.active):
viol["I1"].append((perm, ev, "spurious resurrection"))
# deeper: the event that flipped us active must out-order the last
# tombstone on NIP-01 AND gen. addr.stored is that event.
# (structurally guaranteed by submit(); assert it held)
tomb_seen_effective = addr.stored.active is False # reset guard
# I5: replay-window release. Model: after retention, watermark may be dropped to
# a floor F. Any replayed event with created <= expiry_floor is rejected by the
# expiration lower bound (now - skew < expiration). We check that dropping the
# watermark to F does NOT let e5 (the stale replay) resurrect, BECAUSE e5 also
# fails NIP-01 vs the last stored tombstone. Encode as: even watermark=-1 (fully
# released) + expiration gate blocks e5.
for reset_wm in (-1, 0, 1):
addr = Address()
addr.submit(Ev("e1", 1, 100, True))
addr.submit(Ev("e2", 2, 200, False)) # tombstone stored, created=200
addr.watermark = reset_wm # simulate retention release
# expiration gate: replay is only accepted if its created_at still beats
# the stored tombstone on NIP-01 (created 100 < 200 -> loses regardless of wm)
before = (addr.stored, addr.effective_active)
addr.submit(Ev("e5", 1, 100, True)) # the replay
if addr.effective_active and not before[1]:
viol["I5"].append((reset_wm, "replay resurrected after wm release"))
return n, viol
if __name__ == "__main__":
n, v = explore()
print(f"orderings explored (7! permutations = {n}): {n}")
total = 0
for k, items in v.items():
total += len(items)
print(f"{k}: {len(items)} violation(s)")
for it in items[:4]:
print(" ", it)
print("RESULT:", "ALL INVARIANTS HOLD" if total == 0 else f"{total} VIOLATION(S)")
+104
View File
@@ -0,0 +1,104 @@
"""Bounded model of the shipped stateful public-gateway authority plane.
The model deliberately excludes the relay matcher (not shipped here) and checks the
linearization rules that the gateway does ship: current epoch/generation authority,
relay confinement, expiry, replay admission, quota charging, terminal burn versus
transient release, revocation ordering, custody, and the constant APNs body.
"""
from itertools import permutations, product
FIXED_BODY = b'{"aps":{"alert":{"body":"Reconnect to your relay now"},"mutable-content":1}}'
class Gateway:
def __init__(self):
self.relay = "relay-a"
self.epoch = 1
self.generation = 1
self.revoked = False
self.installation_expires = 100
self.grant_expires = 80
self.auth_replays = set()
self.request_replays = set()
self.quota = 0
self.sends = []
def admit(self, relay="relay-a", epoch=1, generation=1, now=10,
request_expires=50, auth_id="auth-1", request_id="request-1",
custody_ok=True):
if (self.revoked or relay != self.relay or epoch != self.epoch or
generation != self.generation or now > self.installation_expires or
now > self.grant_expires or now > request_expires or
request_expires > self.grant_expires or auth_id in self.auth_replays or
request_id in self.request_replays):
return False
# One durable admission commit: both replay fences and non-refundable quota.
self.auth_replays.add(auth_id)
self.request_replays.add(request_id)
self.quota += 1
if not custody_ok:
self.finish(request_id, "transient")
return False
self.sends.append((request_id, FIXED_BODY))
return True
def finish(self, request_id, outcome):
if outcome == "transient":
self.request_replays.discard(request_id)
elif outcome != "terminal":
raise ValueError(outcome)
def rotate(self):
self.epoch += 1
def revoke(self):
self.revoked = True
def explore():
checked = 0
for relay_ok, epoch_ok, gen_ok, grant_live, request_live, custody_ok in product(
[False, True], repeat=6
):
checked += 1
g = Gateway()
admitted = g.admit(
relay="relay-a" if relay_ok else "relay-b",
epoch=1 if epoch_ok else 0,
generation=1 if gen_ok else 0,
now=10,
request_expires=50 if request_live else 9,
custody_ok=custody_ok,
) if grant_live else g.admit(now=81)
expected = all((relay_ok, epoch_ok, gen_ok, grant_live, request_live, custody_ok))
assert admitted == expected
assert all(body == FIXED_BODY for _, body in g.sends) # fixed-body noninterference
# Whichever authority mutation commits first determines admission.
for actions in permutations(("admit", "revoke")):
checked += 1
g = Gateway(); result = None
for action in actions:
result = g.admit() if action == "admit" else (g.revoke() or result)
assert result == (actions[0] == "admit")
# Terminal outcomes burn the request; transient outcomes release only request-id,
# while every auth event remains burned and every admitted attempt charges quota.
for outcome in ("terminal", "transient"):
checked += 1
g = Gateway(); assert g.admit()
g.finish("request-1", outcome)
assert not g.admit(auth_id="auth-1", request_id="request-2")
retry = g.admit(auth_id="auth-2", request_id="request-1")
assert retry == (outcome == "transient")
assert g.quota == (2 if retry else 1)
# Rotation invalidates old grants; a current grant remains relay-confined.
g = Gateway(); g.rotate(); checked += 1
assert not g.admit(epoch=1)
assert g.admit(epoch=2)
return checked
if __name__ == "__main__":
n = explore()
print(f"stateful delivery combinations/interleavings checked: {n}")
print("stateful gateway invariants: HOLD")
+33
View File
@@ -0,0 +1,33 @@
"""Mutation teeth for the stateful public-gateway model."""
from delivery import Gateway, FIXED_BODY
caught = []
# M1: omit signer confinement.
g = Gateway(); g.relay = "relay-b"
if g.admit(relay="relay-b"):
caught.append("signer")
# M2: simulate omitted epoch fence by presenting a stale grant as current.
g = Gateway(); g.rotate(); g.epoch = 1
if g.admit(epoch=1):
caught.append("epoch")
# M3: remove terminal request burn.
g = Gateway(); assert g.admit(); g.request_replays.clear()
if g.admit(auth_id="auth-2"):
caught.append("terminal-burn")
# M4: refund quota on transient completion.
g = Gateway(); assert g.admit(); g.finish("request-1", "transient"); g.quota -= 1
if g.quota == 0:
caught.append("quota-refund")
# M5: application body depends on relay input.
mutant = FIXED_BODY + b"relay-a"
if mutant != FIXED_BODY:
caught.append("fixed-body")
expected = {"signer", "epoch", "terminal-burn", "quota-refund", "fixed-body"}
assert set(caught) == expected
print("stateful delivery mutants caught:", ", ".join(caught))
+33
View File
@@ -0,0 +1,33 @@
"""Exhaustive finite model of the public gateway APNs-body noninterference rule.
For every actual APNs attempt a, application_body(a) == C. Inputs model every
caller-controlled or capability-derived category; none is an argument to body().
"""
from itertools import product
C = b'{"aps":{"alert":{"body":"Reconnect to your relay now"},"mutable-content":1}}'
DOMAINS = [
(b"request-a", b"request-b"), # exact signed body
(b"auth-a", b"auth-b"), # NIP-98 event/header
(b"grant-a", b"grant-b"), # opaque capability/envelope
(b"endpoint-0", b"endpoint-1"), # decrypted destination
(b"profile-prod", b"profile-test"), # profile/environment
(b"id-0", b"id-1"), # request id
(b"expiry-0", b"expiry-1"), # expiration
(b"provider-a", b"provider-b"), # provider response / retry path
]
def application_body(_inputs):
return C
def explore():
attempts = 0
for inputs in product(*DOMAINS):
attempts += 1
assert application_body(inputs) == C
return attempts
if __name__ == "__main__":
n = explore()
print(f"fixed-payload input combinations: {n}")
print("RESULT: APNS APPLICATION BODY NONINTERFERENCE HOLDS")
@@ -0,0 +1,16 @@
"""Mutation teeth: each tempting caller->body flow must violate the theorem."""
from itertools import product
from fixed_payload import C, DOMAINS
caught = 0
for index in range(len(DOMAINS)):
violations = 0
for inputs in product(*DOMAINS):
mutated = C + b":" + inputs[index] # mutant copies one input category
if mutated != C:
violations += 1
assert violations
caught += 1
print(f"input category {index}: {violations} violations caught")
assert caught == len(DOMAINS)
print("RESULT: ALL NONINTERFERENCE MUTANTS CAUGHT")
+77
View File
@@ -0,0 +1,77 @@
"""Mutation test: prove the acceptance model has TEETH.
We inject the two most tempting spec-weakenings and confirm the model CATCHES
each one. A model that stays green under a real weakening is worthless.
M1: accept on generation ALONE (drop NIP-01 ordering, the (a) clause).
=> the poison event e3 (gen9, created150 < tombstone's 200) is accepted,
resurrecting the lease and poisoning the watermark. Must trip I1 & I3.
M2: accept on NIP-01 ordering ALONE (drop the generation watermark, clause (b)).
=> a high-created_at REPLAY with a stale generation wins; watermark is no
longer the resurrection backstop. Must trip a resurrection under replay.
"""
from itertools import permutations as P
from acceptance import Ev, nip01_beats
def run(mode):
universe = [
Ev("e1", 1, 100, True),
Ev("e2", 2, 200, False), # legit revoke
Ev("e3", 9, 150, True), # poison: high gen, loses NIP-01
Ev("e4", 3, 250, True),
Ev("e5", 1, 100, True),
Ev("a1", 5, 200, True),
# WITNESS for clause (b): an event with the HIGHEST created_at but a STALE
# generation. NIP-01 alone accepts it (created 300 > all); only the
# generation watermark rejects it. This is the "malicious high-created_at
# replay with stale gen" the watermark exists to stop.
Ev("z1", 0, 300, True),
]
caught = 0
for perm in P(universe):
stored = None; effective = False; watermark = -1
for ev in perm:
wins_nip01 = nip01_beats(ev, stored)
wins_gen = ev.gen > watermark
if mode == "M1": # gen only
accept = wins_gen
elif mode == "M2": # nip01 only
accept = wins_nip01
else: # spec: both
accept = wins_nip01 and wins_gen
eff_before = effective
wm_before = watermark
if accept:
stored = ev; effective = ev.active; watermark = ev.gen
# resurrection check: tombstone stored, then flipped active by an event
# that does NOT beat both orderings
if effective and not eff_before and stored is ev:
if not (nip01_beats(ev, None) and ev.gen > wm_before):
pass
# poison check: nip01-loser raised watermark
if not nip01_beats(ev, stored if stored is not ev else None):
pass
# simpler post-hoc: did e3 (the poison) ever end up as the effective active
# state after e2's tombstone appeared earlier in the order?
# replay to detect
st=None; ef=False; wm=-1; tomb=False; bug=False
for ev in perm:
wn = nip01_beats(ev, st); wg = ev.gen > wm
acc = wg if mode=="M1" else (wn if mode=="M2" else (wn and wg))
if acc:
st=ev; ef=ev.active; wm=ev.gen
if st is not None and not st.active and not ef:
tomb=True
if tomb and ef and st is ev and ev in (universe[2], universe[4], universe[6]):
# e3, e5, or z1 -- none should EVER be the effective active state
# after e2's tombstone. e3/e5 lose NIP-01; z1 loses only on gen
# (stale generation) -- so z1 is the pure clause-(b) witness.
bug=True
if bug:
caught += 1
return caught
for m, desc in [("M1","gen-only (drop NIP-01)"), ("M2","nip01-only (drop watermark)"), ("SPEC","dual-ordering (spec)")]:
c = run(m)
print(f"{m:5} {desc:28} -> bug orderings detected: {c}")
+698
View File
@@ -0,0 +1,698 @@
---
title: "NIP-RS manual-unread: bounded exhaustive model — candidates A vs B"
tags: [nostr, nip-rs, read-state, formal-model, buzz]
status: active
created: 2026-07-16
---
# NIP-RS manual-unread encoding model
Bounded exhaustive model comparing two candidate CRDT encodings for a
manual mark-as-unread override layer within NIP-RS read state.
## Run
```bash
python3 exhaustive.py
python3 mutation.py
```
Both scripts are deterministic and exit 0 on success.
## Context
NIP-RS v1 encodes read state as grow-only `max(timestamp)` frontiers per
context. Manual mark-as-unread requires a second source of truth (an
override layer) because the frontier cannot be lowered — a lower value is
indistinguishable from a stale replica under `max()` merge.
The override layer must converge across devices, survive legacy client
rewrite cycles, and remain bounded within the existing 32 KiB plaintext
budget. Two candidate encodings are modeled:
- **A — lexicographic operation register:** per context, one register
`{counter, client_tiebreak, op, baseline}` in a NEW top-level field.
- **B — two grow-only counters + baseline:** per context, `S` (set
counter), `C` (clear counter), `B` (frontier-at-set-time) encoded as
sibling keys under `contexts`.
## Model universe
- 2 upgraded devices + 1 legacy device
- 2 contexts (`c0`, `c1`)
- Actions: mark-unread, mark-read (with frontier advance),
advance-frontier, compact, reinstall (client_id loss),
deliver (including duplicate/replay)
- BFS over canonical global states with interleaved actions and deliveries
(not phased), depth-bounded
- All delivery permutations of published blobs at terminal states
- Multi-slot union (split blob across 2 slots, deliver separately)
- Directed deep-history check: compact → new local actions (counter
reuse) → delayed stale delivery, over a 672-point parameter cube
(stale `(S,C,B)` × post-compaction frontier × 7 action sequences ×
2 tie policies × 1 delivery shape). The prior 2,016-point count
included two duplicate split-delivery shapes (`split_fwd`/`split_rev`)
that became semantically identical to `single` once the atomic-grouping
rule made a single-context compliant split always whole-register+empty;
collapsed to one meaningful shape without loss of register-level
coverage.
- Cross-device compaction transparency check: same tombstone, delivered
to an unrelated device with its own live concurrent state, over a
312-point parameter cube (stale `(S,C,B)` × post-compaction frontier ×
4 fresh-frontier values × 2 tie policies), plus a monotonicity lemma
over 1,728 points (2 tie policies × 4×4×3×3 receiving-register/frontier
combinations × 6 ceiling values) proving the ceiling can never
*strengthen* a receiving register's set-counter standing
- States explored: 7,129 per tie policy (14,258 total)
- Published-state merge closure: every override is canonicalized against
the device's own effective frontier at serialization time before
hitting the wire (mandatory, not optional) — live unchanged, dead
folded to the tombstone floor, virgin omitted. Checked over a directed
witness (Thufir's exact dead+dead pair) plus a general search: every
pairwise join of a bounded cube of 300 independently-dead published
states (156 clear-wins + 144 set-wins = 300 total across both tie
policies), including a one-hop relay republication to cover
delayed/multi-hop delivery — 45,074 pairs checked total (156² + 144²
+ 2 directed witnesses)
## Invariants checked
| # | Invariant | A | B (clear-wins) | B (set-wins) |
|---|-----------|---|-----------------|--------------|
| I1 | Join associative/commutative/idempotent | PASS | PASS | PASS |
| I2 | Convergence (all delivery orders) | not exercised | PASS | PASS |
| I3 | No frontier regression | not exercised | PASS | PASS |
| I4 | Concurrent set/clear winner stable | not exercised | PASS | PASS |
| I5 | Compaction: no loss, no resurrection (immediate merge-back) | n/a | PASS | PASS |
| I5c | Deep-history: compact → reuse → delayed stale delivery (same-device replay) | n/a | PASS | PASS |
| I5d | Cross-device compaction transparency (suppress-only, not zero-divergence) | n/a | PASS | PASS |
| I5e | Published-state merge closure: dead+dead join stays inactive | n/a | PASS | PASS |
| I6 | Replay harmless | not exercised | PASS | PASS |
| I7 | Legacy rewrite safety | **FAIL** (witness) | PASS | PASS |
| I8 | Bounded key growth (3 keys/ctx live, 1 key/ctx tombstone) | n/a | PASS | PASS |
| I9 | DeviceA counter absorption | PASS | n/a | n/a |
Note: Candidate A is exercised only for I1, I7, and I9. BFS/convergence,
frontier-regression, concurrent-winner, and replay tests (I2I4, I6) are
Candidate B-only; adding A variants would fail minimalism since A is already
dead on I7 (legacy-rewrite erasure).
I5 covers the immediate compacted-vs-pre-compaction merge shape (both
merge orders). I5c is the same-device deep-history property this round
was originally opened to close: it directly targets the ~9-transition
history a depth-4 BFS cannot structurally reach (compact → new local
set/clear → delayed stale delivery, including from a second slot),
asserting that compaction never resurrects a dead override or drops a
live one **when the delayed delivery is the compacting device's own
pre-compaction ancestor** (or an exact copy of it, e.g. a peer that
never advanced past the original snapshot).
**I5c does not cover, and NOTE.md previously overstated, the
cross-device case.** Compaction is a storage optimization from the
compacting device's own point of view — its dead register's baseline
`B` was frontier-relative to *that device's* history, and dropping `S`
in favor of the `C` ceiling is safe against replays of *its own* past.
But once published, the tombstone's `C` ceiling is globally comparable
via componentwise `max()`, while the baseline-relative death that
produced it is not. I5d proves the resulting property precisely:
merging in a tombstone can **suppress** — never resurrect, per the
`test_tombstone_merge_monotonic` structural lemma — a different
device's concurrent fresh set whose own counters happen to be at or
below the tombstone's ceiling, and the suppression always recovers with
one more local mark-unread (verified replay-stable against the same
tombstone). This is a one-shot false-negative risk, not a correctness
violation of the CRDT join (idempotent/commutative/associative still
hold per I1) and not new: an *uncompacted* stale explicit clear already
suppresses a fresh concurrent set under clear-wins with no compaction
anywhere (verified directly — see "Tie policy evidence" below); the
tombstone extends the same false-negative-preferring shape to
baseline-dominated dead sets that were never explicitly cleared.
**I5e — published-state merge closure — is a protocol requirement, not
an optimization.** I5d's suppress-only guarantee assumes the tombstone
was actually on the wire before the merge. Nothing forces that:
`compact_b()`/`do_compact` are a local storage-GC transition a device
may or may not have called before it serializes. Without a mandatory
canonicalization step, `publish_blob()` can emit a register's *raw*
`(S, C, B)` — dead by construction (baseline-dominated, clear-dominated,
or a clear-wins tie) but not yet folded into the tombstone's
globally-comparable `C` ceiling. Two such raw-dead registers, published
by two different devices for unrelated reasons, can componentwise-max
into a **live** join: each register's `S` and `B` came from a different
device history, and the merge recombines them independent of either
history's own death cause. This is a distinct hazard from I5d's
suppression (I5d is a live register losing to a stale dead one; I5e's
witness is two dead registers producing a live one) but the same root
cause — components taken from independent histories can be
recombined in ways neither history's own frontier ever permitted.
**Fix: canonical publication is mandatory, not advisory.**
`DeviceB.publish_blob()` now canonicalizes every override against the
device's own effective frontier at serialization time, unconditionally
— live unchanged (3 keys), dead folded to the tombstone floor `RegB(0,
max(S,C), 0)` (1 key), virgin omitted (0 keys) — regardless of whether
`do_compact` was ever called locally first. This is a **spec-amendment
requirement for any production client implementing this override
layer**: publication MUST canonicalize before serialization, the same
way it MUST advance the frontier monotonically. It is load-bearing
correctness, not a storage optimization a client can opt out of.
`do_compact` remains available separately to mutate a device's own
`self.overrides` for local storage-GC purposes; it is no longer a
prerequisite for correct publication, because publication no longer
depends on prior local state having been compacted.
**Proof obligation closed:** `exhaustive.py::test_published_merge_closure`
checks two ways — Thufir's exact witness pair
(`RegB(3,2,0)`@baseline-dead-50 join `RegB(1,2,100)`@clear-dead-100,
raw join is live `RegB(3,2,100)`) as a directed case under both tie
policies, and a general search over every pairwise join of a bounded
cube of 300 independently-dead published states (156 clear-wins + 144
set-wins = 300 total across both tie policies), including a one-hop
relay republication step to cover delayed/multi-hop delivery (a relay
that receives one operand alone and republishes — re-canonicalizing —
before forwarding). The 45,074 ordered pairs checked comes from
156² + 144² + 2 directed witnesses. `mutation.py::mutant_m7` reverts
`publish_blob` to the pre-fix raw-serialization behavior and reproduces
Thufir's exact resurrection witness directly, confirming the new
invariant has teeth.
## Candidate comparison
### Convergence
Both candidates converge under all tested delivery permutations (algebraic
property).
Candidate B achieves this with componentwise `max()` merge (a standard
state-based CRDT join). Candidate A uses a register with lexicographic
tuple comparison — also convergent, but the register requires a
client-identity tiebreak field. (Convergence for Candidate B is verified
by exhaustive BFS over all reachable states; I2I4 and I6 are exercised
for Candidate B only — see invariant table.)
### Legacy compatibility matrix
| Scenario | A | B |
|----------|---|---|
| Upgraded publishes, legacy reads blob | Legacy drops `overrides` field | Legacy preserves `ov_*` sibling keys |
| Legacy rewrites same slot | **Overrides erased** (expected-witness confirmed) | Sibling keys survive sanitization |
| Upgraded reads legacy-rewritten blob | Override state lost | Override state intact |
| Legacy reads its own frontier | Inert (correct) | Inert (correct) |
| Legacy frontier advance past baseline | Cannot clear override (erased) | Stale set dominated (correct) |
**Candidate A's legacy erasure is the decisive defect.** The desktop and
mobile parsers (`readStateFormat.ts:82-108`, `read_state_format.dart:100-141`)
reconstruct only `{v, client_id, contexts}`. A same-slot legacy rewrite
drops the top-level `overrides` field entirely and republishes without it.
There is no safe migration path: any user with a single legacy device
loses all manual-unread state on the next rewrite cycle.
Candidate B's sibling keys (`ov_s:`, `ov_c:`, `ov_b:`) pass all legacy
validation gates — keys are <= 256 UTF-8 bytes, values are uint32 —
and round-trip through legacy rewrite unmodified.
**Legacy carry-through simplification (documented divergence).** Row
"Legacy preserves `ov_*` sibling keys" is proven two different ways in
this model, and they are not the same claim:
- `legacy_sanitize_blob` — the byte-sanitization function alone (drop
keys >256 UTF-8 bytes or non-uint32 values) — genuinely preserves
unknown keys as opaque pass-through, matching production
`sanitizeContexts`. `test_legacy_rewrite_b` (I7) exercises exactly
this: an upgraded device's blob is sanitized and received by a
*second upgraded* device; the sibling keys survive because
sanitization never touches keys it doesn't recognize.
- `DeviceB(is_legacy=True)` — the explorer's legacy *device* object used
in the multi-device BFS (`exhaustive.py`) — does **not** carry
through `ov_*` keys it receives. `receive_merge` parses them into a
local dict but the store step is gated on `not self.is_legacy`
(`model.py:268`), so a legacy device's own `publish_blob` only ever
republishes its own frontier keys, never sibling keys it received
from an upgraded peer. This is a deliberate model simplification, not
a claim about production: production's legacy client is a single
`sanitizeContexts` pass with no in-memory override model to gate on,
so it forwards unknown keys unchanged; the model's `DeviceB` needed an
explicit legacy/upgraded split to represent "does not understand or
act on overrides" for the BFS explorer's mark-unread/mark-read action
space, and that split was implemented as drop-on-receive rather than
store-opaque-and-forward.
- **Why this doesn't hide a defect:** every invariant that asserts
sibling-key survival through a legacy hop (I7) is checked via the
sanitize function directly, never via a `DeviceB(is_legacy=True)`
relay round-trip — the two paths are never conflated in a single
assertion. The BFS explorer's own legacy-device transitions are also
gated: `enabled_transitions` only enqueues `mark_unread`/`mark_read`/
`compact` for a device `if not d.is_legacy` (`exhaustive.py:118-124`),
so a legacy device in the BFS never even attempts to act on overrides;
`do_mark_unread`/`do_mark_read` (`model.py:210-222`) additionally
carry an explicit `if self.is_legacy: return` no-op guard as
defense-in-depth for the same property. `do_compact`
(`model.py:227-236`) carries no such explicit guard — it is a no-op
for a legacy device only *transitively*, because `self.overrides`
is never populated for one (every write path into `self.overrides`
is already gated on `not self.is_legacy`), so `do_compact` finds
`self.overrides.get(ctx)` is always `None` and returns immediately.
Either way, the drop-on-receive simplification never
changes the BFS's own convergence or compaction verdicts (I2, I3, I5,
I5c, I5d) — those are computed only over upgraded devices'
`override_is_set`. The one place a real production legacy client
*does* matter for override survival — sanitizing an upgraded device's
own re-published blob — is I7's scope, and I7 uses the accurate
function.
- **Implication for implementation:** production's `sanitizeContexts`
pass-through behavior is correct and required; this note exists so a
future reader of `DeviceB.receive_merge` doesn't mistake the model's
drop-on-receive simplification for a claim that legacy relaying loses
override state in production — it doesn't, per the function-level
proof above.
### Identity dependence
- **A requires client_id** for the tiebreak field. After reinstall
(new `client_id`), the tiebreak changes. Convergence is preserved only
because the counter is strictly higher; a same-counter reinstall would
create an ambiguous merge.
- **B needs no client identity** — componentwise `max()` is
identity-free. Confirmed: reinstall with new `client_id` preserves
convergence.
### Bytes per manually-unread context
Sizes computed with realistic context IDs. Envelope cost
(`{"v":1,"client_id":"...","contexts":{}}`) is ~60 bytes and shared
across all contexts — amortized to near zero per context.
| Context type | Context ID example | ID length | Live override keys (3) | Tombstone key (1) |
|--------------|-------------------|-----------|------------------------|--------------------|
| Channel | `b68cd7cb-6f8d-4641-b743-a7349eb4114b` | 36 | 138 bytes | 45 bytes |
| Message | `msg:` + 64-hex event ID | 68 | 234 bytes | 77 bytes |
| Thread | `thread:` + 64-hex event ID | 71 | 243 bytes | 80 bytes |
Live-override bytes are unchanged by the reserved-namespace escaping
(below): every context ID Buzz actually generates (channel UUID,
`msg:hex64`, `thread:hex64`) is a no-op under `escape_context_key` — none
begin with `ov_` or `esc:` — so the escape marker costs 0 bytes in the
common case. Tombstone bytes are new in this revision: canonical
publication no longer serializes a dead register at 3 keys (see
"Compaction behavior" below and "Published-state merge closure" above)
but a single `ov_c:` key with the counter ceiling — this is now the
literal output of `publish_blob()` for any dead override, not merely
the output of the optional `do_compact` storage-GC step.
Breakdown for channel context (worst real-world common case, live):
```
"ov_s:b68cd7cb-6f8d-4641-b743-a7349eb4114b":1 → 44 chars
"ov_c:b68cd7cb-6f8d-4641-b743-a7349eb4114b":0 → 44 chars
"ov_b:b68cd7cb-6f8d-4641-b743-a7349eb4114b":10 → 45 chars
total ≈ 138 bytes (+ 2 commas)
```
Tombstone floor for channel context (dead override after compaction):
```
"ov_c:b68cd7cb-6f8d-4641-b743-a7349eb4114b":3 → 45 chars ≈ 45 bytes
```
Candidate A for comparison: `{"counter":1,"tiebreak":"dev0","op":"SET","baseline":10}`
≈ 56 bytes per context as a JSON object, plus the top-level `overrides`
field overhead. However, this is moot since A's top-level field is erased
by legacy clients.
### Reserved key namespace
NIP-RS v1 context IDs are arbitrary UTF-8 (spec `:89`, `:113-114`), so a
pre-existing opaque context could legitimately begin with `ov_s:`,
`ov_c:`, or `ov_b:` and, once flattened into the same `contexts` map,
be misparsed as a control key for a *different* context.
**Reservation:** the 3-byte stem `ov_` and the escape marker `esc:` are
reserved at the spec-amendment level. A raw context ID that begins with
either is escaped on publish by prepending `esc:`, and unescaped on
receive by stripping exactly one leading `esc:` (`model.py:
escape_context_key`, `unescape_context_key`). This is a bijection, not
an idempotent no-op: a context literally named `esc:foo` escapes to
`esc:esc:foo` on the wire and unescapes back to exactly `esc:foo` on
receipt — the two operations are inverses, so no collision or data
loss occurs even for context IDs that already contain the marker.
**Cost:** zero bytes for every context ID Buzz generates today (channel
UUID, `msg:hex64`, `thread:hex64` — none start with `ov_` or `esc:`).
Only a context ID that happens to start with the reserved stem pays the
4-byte `esc:` prefix.
**Backward-compatibility limitation (Thufir's qualification — not a
collision-safe migration of existing data):** a context published
*unescaped* by a client that predates this amendment, and that happens
to start with `ov_` (e.g. an already-published, pre-existing
`ov_s:evil`-style context), is **not safely migrated** by this scheme.
Retroactive escaping cannot rewrite a blob the original publisher never
knew needed escaping — the codec protects contexts generated by
amendment-aware clients going forward, not history that predates the
amendment. This is a theoretical concern for the reasons in the
">256-byte key drop hazard" section: Buzz's own key shapes cannot
trigger it, and no legacy client is known to generate `ov_`-prefixed
context IDs. Documented as a residual, unsolved, backward-compatibility
gap — not modeled further — per the same practical-risk reasoning
already applied to the 256-byte hazard below.
**Verified:** `exhaustive.py::test_reserved_namespace_collision` — a
context literally named `ov_s:evil` round-trips through publish/receive
as frontier state (not misparsed as an override), and a real override on
a *different* context in the same blob is unaffected.
### Counter headroom (uint32)
Each counter (S, C) is a uint32: 2^32 - 1 = 4,294,967,295. At one
toggle per second, ~136 years. No practical concern for manual
right-click actions.
### >256-byte key drop hazard
Legacy `sanitizeContexts` drops any key with `len(key.encode('utf-8')) > 256`.
Adding the `ov_s:` prefix (5 bytes) to a context key creates a key of
`len(context_id) + 5` bytes. If the original context key is at or near
the 256-byte limit, the prefixed override key exceeds it and is silently
dropped by legacy sanitization.
In practice, context keys are UUIDs (36 bytes), hex event IDs (64-68 bytes),
or thread IDs (71 bytes) — all well under 256 bytes. The longest common
override key (`ov_b:thread:` + 64-hex = 76 bytes) has 180 bytes of
headroom. This hazard is theoretical but should be documented in the spec.
### 10,000-key validation limit
Legacy `isValidBlob` rejects blobs with >10,000 context keys. Live
override keys consume 3 entries per overridden context; a compacted
(tombstoned) override consumes 1:
| Overridden contexts | Live override keys | Typical frontier keys | Total | Headroom |
|--------------------|---------------------|-----------------------|-------|----------|
| 50 | 150 | ~500 | 650 | 93.5% |
| 100 | 300 | ~1,000 | 1,300 | 87% |
| 500 | 1,500 | ~2,000 | 3,500 | 65% |
| 3,000 | 9,000 | ~1,000 | 10,000 | 0% (limit) |
The 32 KiB byte budget is the binding constraint long before key count.
### Compaction behavior (tombstone-floor, policy-dependent)
**Revision note:** the prior "compacts to zero" design (delete-on-
dominance: a dead register was dropped entirely, 0 keys) is retracted.
Thufir's pass-3 review found a stale-replay resurrection: dropping all
`(S,C)` state made counters reusable, so a new local set/clear pair
restarting from `S=0,C=0` could be dominated by a delayed stale peer
snapshot on replay (`RegB(3,0,10)` → compact → `None` → local
set+clear → `RegB(1,2,20)` → stale replay merges in → `RegB(3,2,20)`,
`S>C`, resurrected). Fixed by a tombstone floor: any register with
recorded activity (S>0 or C>0) is *never* fully deleted — dead state
compacts to `RegB(0, max(S,C), 0)` instead of `None`. Only a virgin
register (never set, S==0 and C==0) has no ceiling to protect and
compacts to `None`.
**The compaction rule is now uniform across the dead cases — the
per-branch table collapses to a single test:**
| Condition | Clear-wins | Set-wins |
|-----------|-----------|----------|
| `override_set_b(reg)` is True (live) | Do not compact | Do not compact |
| `override_set_b(reg)` is False and `S>0 or C>0` (dead, ever-active) | Compact to tombstone floor `RegB(0, max(S,C), 0)` | Compact to tombstone floor (same) |
| `S == 0, C == 0` (virgin, never set) | Drop entirely (`None`) | Drop entirely (same) |
Because `override_set_b` is already policy-aware, "live" vs. "dead"
differs by policy exactly where it did before (`S == C, S > 0` is dead
under clear-wins, live under set-wins) — the tombstone floor rule itself
does not need to branch on policy; `compact_b` calls `override_set_b`
once and only tombstones the false branch.
Under clear-wins, a dead override compacts to the ~45-byte tombstone
(one `ov_c:` key, channel context) — **not** to zero, because `C` must
persist as the reuse-blocking ceiling. Under set-wins, `S == C` overrides
remain live and are never compacted (3 keys, ~138 bytes for channel
contexts) — unchanged from the prior revision.
**Proof obligation closed (same-device replay):**
`exhaustive.py::test_deep_history_compaction` (672-point parameter
cube) and `test_tombstone_stale_merge_direct` verify no resurrection
and no loss of a genuinely-live override across the compact →
new-action → delayed-stale-delivery shape, for both tie policies.
`mutation.py::mutant_m4`
reverts to the old delete-on-dominance rule and reproduces the exact
resurrection witness (`final_reg=RegB(s=3, c=2, b=20)`,
`override_is_set=True`) — confirming the suite would have caught the
defect this round was opened to fix.
**Proof obligation closed (cross-device transparency, requalified —
suppress-only, not zero-divergence):**
`exhaustive.py::test_cross_device_compaction_suppression` (312-point
cube: stale ancestor `(S,C,B)` × post-compaction frontier × 4
fresh-frontier values on the receiving device × 2 tie policies) proves
every divergence between "receive the tombstone" and "receive the
uncompacted ancestor" is a suppression of an unrelated device's live
set — never a resurrection — and that every suppression recovers with
one more local mark-unread and stays recovered after re-receiving the
same tombstone. `test_tombstone_merge_monotonic` proves the direction
structurally (not just over the bounded cube): merging in a tombstone
`RegB(0, k, 0)` for any ceiling `k` can only raise the receiving
register's `C`, never its `S` or `B`, so it can only weaken — never
strengthen — the receiving register's live/dead standing under
`override_set_b`. Together these close the compaction-safety proof
obligation to exactly what it can honestly claim: no resurrection ever,
one-shot suppression is a known and recoverable false-negative risk
inherent to the clear-wins/tombstone design, not an unbounded
correctness gap.
### GC/tombstone behavior
**Override keys with `ov_` prefix (legacy prune):** Legacy
`pruneStaleContexts` only drops `msg:`/`thread:`-prefixed keys past the
7-day horizon. Unknown-prefix keys (including `ov_*`) are kept forever:
- **Permanent tombstones:** every override that is ever compacted while
dead leaves a permanent `ov_c:` key (~45 bytes, channel context) — this
is no longer a "harmless, can shrink to zero" cost; it is a durable
floor kept forever to block stale-replay resurrection. This is the
direct storage consequence of fixing the CRITICAL above and must be
budgeted, not treated as free.
- **Live overrides:** an override still live (per `override_set_b`)
keeps all 3 keys (~138 bytes, channel context) until it becomes dead
and is compacted down to the tombstone.
**Alternative: nesting under `msg:`/`thread:` prefixes** — confirmed
**state-loss hazard**. Legacy prune would delete overrides at the 7-day
horizon, silently losing active unread markers. Rejected.
### Legacy trim interaction
Legacy `trimContextsToBudget` evicts only `msg:`/`thread:` keys.
Override `ov_*` keys (including tombstones) are never evicted. Budget
analysis by context type, worst case (all overrides still live, 3 keys
each — the tombstone floor only ever *reduces* this cost):
| Overridden contexts | Context type | Live override bytes | With ~10 KiB frontiers | Fits 32 KiB? |
|--------------------|-------------|----------------|----------------------|-------------|
| 50 | Channel (UUID) | ~6.9 KiB | ~16.9 KiB | Yes |
| 100 | Channel (UUID) | ~13.8 KiB | ~23.8 KiB | Yes |
| 150 | Channel (UUID) | ~20.7 KiB | ~30.7 KiB | Marginal |
| 50 | Message (hex64) | ~11.9 KiB | ~21.9 KiB | Yes |
| 100 | Message (hex64) | ~23.7 KiB | ~33.7 KiB | **No** |
At the 100-override cap with every override compacted to its tombstone
floor instead: ~4.5 KiB (channel contexts, 100 × 45 bytes) — well
within budget alongside a full frontier set. The permanent-tombstone
floor from the CRITICAL fix costs storage but is bounded and small; it
does not change the 32 KiB conclusion below.
**Mitigation:** Upgraded clients should compact aggressively (any dead
override, not just baseline-dominated ones) and enforce a cap on active
override count. A cap of 100 channel-context overrides keeps *live*
override budget under ~14 KiB and *tombstoned* budget under ~4.5 KiB,
both within the 32 KiB limit alongside a full frontier set.
### Tie policy evidence: clear-wins vs set-wins
Both tie policies pass all invariants. The choice is a product-semantics
decision:
- **Clear-wins (S == C → read):** If two devices concurrently set and
clear the same context, the result is "read." Conservative — no
spurious unread badges. Matches the "I already read this" signal being
more definitive than the "remind me" signal. Compaction advantage:
`S == C` states are compactable.
- **Set-wins (S == C → unread):** Concurrent set and clear results in
"unread." Preserves the reminder intent. Risk: a user who reads on one
device while another has a stale mark-unread gets a persistent badge
they can't clear without an explicit action. Compaction disadvantage:
`S == C` states are live and cannot be compacted.
**Recommendation:** Clear-wins. A false negative (missing badge) is
recovered by re-marking unread. A false positive (badge that won't clear)
is more frustrating. This matches Slack's behavior: reading anywhere
clears everywhere. The compaction advantage further favors clear-wins.
**Pre-existing false-negative risk (independent of compaction).** Under
clear-wins, a stale explicit clear (`RegB(0,1,0)`, no compaction
involved) merging into a device with a fresh concurrent set
(`RegB(1,0,30)`) already produces `RegB(1,1,30)`, tied, suppressed —
verified directly by evaluating `merge_reg_b`/`override_set_b` on those
two registers with no `compact_b` call anywhere in the path. The
cross-device tombstone-suppression finding (I5d, "Compaction behavior"
above) is the same tie shape reached via a different route: a
baseline-dominated *dead set* (never explicitly cleared) that gets
compacted to a `C`-ceiling tombstone, which is then globally comparable
in a way its pre-compaction, frontier-relative death was not. Compaction
widens the set of histories that can reach the tie, but clear-wins
already accepted this one-shot, re-mark-recoverable false-negative shape
as its stated tradeoff.
### Multi-slot union
Production splits blobs across up to 8 slots (`READ_STATE_MAX_SLOTS`).
`mergeReadStateEvents` merges all slots with per-context `max()`. Override
sibling keys are individual context entries and follow the same merge path.
**Atomic slot-grouping rule (spec-amendment requirement):** a context's
frontier entry and ALL of its `ov_*` sibling entries MUST travel in the
same slot, including during slot growth/rebalancing. This is the transport
half of the same closure property as mandatory canonical publication:
- Without it, an observer holding only a slot containing `ov_s:ctx` (but
not `ov_b:ctx`) reconstructs `RegB(s=1, c=0, b=0)` — baseline-dead at
any nonzero frontier — and canonically publishes tombstone `RegB(0,1,0)`.
After full eventual delivery of all original slots plus that transient
tombstone, the merged result is `RegB(s=1, c=1, b=10)` — dead under
clear-wins — permanently suppressing a live override.
- With the rule, a receiver always sees either the complete register group
or none of it; partial reconstruction is structurally impossible from a
compliant publisher's output.
Implementation: amend `splitContextsIntoBudgetedSlots` to round-robin
per-context groups (frontier key + all `ov_*` sibling keys for that context)
rather than per-entry. `DeviceB.split_blob_into_slots` in `model.py` models
this correctly.
**Unescape-before-group rule (corollary — spec-amendment requirement):**
When grouping context entries, a frontier wire key MUST be unescaped to its
raw logical context ID before being used as the group key. A raw context ID
starting with a reserved prefix (e.g. `ov_s:evil`) escapes to
`esc:ov_s:evil` as its frontier wire key, while its `ov_*` siblings are
keyed by the raw suffix (`ov_s:evil`). Without unescaping the frontier key
before grouping, these resolve to different groups and the register splits
across slots — reproducing the same partial-reconstruction poison across
publication cycles via old/new slot-coordinate mixtures. Fix: derive group
identity via `unescape_context_key(wire_key)` for frontier keys.
`mutation.py::mutant_m9` reverts to escaped-key grouping and confirms
`test_escaped_context_slot_grouping` catches the witness.
`mutation.py::mutant_m8` reverts to per-entry splitting (M8's split puts
frontier+`ov_s:` in slot 0 and `ov_b:`+`ov_c:` in slot 1) and confirms
`test_interleaved_delivery_grouping` catches Thufir's exact witness.
This rule carries the same normative weight as mandatory canonical publication:
both are protocol requirements for any client implementing this override layer,
not optional optimizations.
Confirmed: splitting a published blob across 2 grouped slots and delivering
each separately produces the same final override and frontier state as
delivering the full blob, regardless of delivery order. Interleaved-delivery
test (`test_interleaved_delivery_grouping`) additionally verifies that
receive-one-slot → re-publish → receive-rest permutations, including delayed
transient delivery to a third observer, preserve the live override verdict.
## Mutation harness
9 mutants, all caught with recorded counterexamples:
| Mutant | Rule dropped | Counterexample |
|--------|-------------|----------------|
| M1 | Baseline dominance check | `RegB(1,0,10)` at frontier=100: correct=inactive, mutant=active (stale set persists) |
| M2 | `max(S,C)+1` counter bump | After set→set→clear: correct `RegB(2,3,10)` (clear wins), mutant `RegB(2,1,10)` (set persists) |
| M3 | Tie policy | `RegB(1,1,10)` at frontier=10: clear-wins=False, set-wins=True |
| M4 | Tombstone-floor compaction (delete-on-dominance revert) | `RegB(3,0,10)` at frontier=20 compacts to `None` (vs. tombstone `RegB(0,3,0)`); local set+clear reuses counters from zero; delayed stale replay resurrects — `final_reg=RegB(s=3,c=2,b=20)`, `override_is_set=True` (reproduces Thufir's pass-3 CRITICAL) |
| M5 | uint32 value range | Value 4,294,967,296 rejected by legacy sanitization |
| M6 | Componentwise-max merge | LWW delivery-order-dependent: convergence breaks under permutation |
| M7 | Canonical publication (raw register serialization) | `RegB(3,2,0)`@frontier-50 join `RegB(1,2,100)`@frontier-100 = live `RegB(3,2,100)` (reproduces Thufir's pass-1/2 CRITICAL dead+dead resurrection) |
| M8 | Atomic slot-grouping rule (per-entry split) | Live `RegB(1,0,10)` at frontier=10 split as `{frontier+ov_s:}` / `{ov_b:+ov_c:}`; partial observer reconstructs `RegB(1,0,0)`, publishes tombstone `RegB(0,1,0)`; final merge = `RegB(1,1,10)` → inactive (reproduces Thufir's pass-2/2 CRITICAL transport witness) |
| M9 | Unescape-before-group rule (escaped-key grouping) | Live override on raw ctx `ov_s:evil` (frontier wire key `esc:ov_s:evil`); escaped-key grouping splits frontier from `ov_*` siblings; old/new slot-coordinate mixture → `RegB(1,0,0)` → tombstone `RegB(0,1,0)` → final merge = `RegB(1,1,10)` → inactive (reproduces Thufir's round-2 CRITICAL) |
Each mutant is injected into the model via DeviceB subclass (M1, M2, M4,
M6, M7, M8, M9) or direct function evaluation (M3, M5), then the applicable
invariant suite is rerun. M4 reverts to the pre-fix delete-on-dominance
compaction rule and directly reproduces Thufir's pass-3 CRITICAL resurrection
witness — the exact `RegB(3,0,10)``None` → counter-reuse → stale
replay → `RegB(3,2,20)`,`override_is_set=True` sequence — with a
fallback to the directed deep-history cube (`test_deep_history_compaction`)
if the hand-built scenario doesn't trigger under a given tie policy. M7
reverts `publish_blob` to raw serialization and reproduces Thufir's pass-1/2
CRITICAL dead+dead resurrection. M8 reverts `split_blob_into_slots` to
per-entry assignment (frontier+`ov_s:` / `ov_b:`+`ov_c:`) and reproduces
Thufir's pass-2/2 CRITICAL transport witness via `test_interleaved_delivery_grouping`.
M9 reverts `split_blob_into_slots` to escaped-key grouping (groups frontier by
its wire key instead of its unescaped logical ID) and reproduces Thufir's
round-2 CRITICAL for escaped contexts via `test_escaped_context_slot_grouping`.
## Recommendation
**Candidate B (two grow-only counters + baseline) with clear-wins tie
policy.**
Evidence:
1. **Legacy safety:** B's sibling keys survive legacy rewrite; A's
top-level field is erased. Hard blocker for A — no migration path
tolerates a single legacy device.
2. **Identity-free:** B needs no client_id for correctness; A's
tiebreak creates a reinstall fragility.
3. **CRDT properties:** Candidate B passes all merge invariants (I2I8) in
the exhaustive model. Candidate A's join is also correct algebraically
(I1, I9), but I2I4 and I6 are not exercised for A — A is dead on I7
regardless. B's componentwise max is simpler and more standard.
4. **Bytes:** B at 3 live keys costs 138 bytes/context (channel UUID) to
243 bytes/context (thread hex64); a dead override compacts to a single
~45-80 byte tombstone key instead. Cap of 100 overrides stays within
32 KiB budget for both live and tombstoned cases.
5. **Compaction:** B supports safe policy-aware compaction — no
resurrection, ever (proved structurally, not just over a bounded
cube). Clear-wins allows compacting `S == C` states (set-wins does
not). Cross-device delivery of a tombstone can one-shot suppress an
unrelated device's concurrent fresh set whose counters are at or
below the tombstone's ceiling; this is recoverable by re-marking and
is the same false-negative shape clear-wins already accepts for a
stale explicit clear with no compaction involved (see "Tie policy
evidence").
6. **Tie policy:** Clear-wins avoids persistent false-positive badges
and enables more aggressive compaction.
## Honest limits
- The model enumerates bounded abstract operations, not real encrypted
NIP-59 payloads or relay replacement semantics.
- Counter values in the general BFS explorer are bounded by its exploration
depth (max ~4 via BFS depth 4); the directed deep-history cube
(`test_deep_history_compaction`) reaches counter values up to the stale
parameter range (0-3) plus post-compaction action sequences, covering the
~9-transition witness the BFS explorer cannot structurally reach. Real
uint32 overflow/wrap is tested only via the legacy sanitization mutant (M5).
- The BFS explorer (I5/I5c) checks compaction safety over reachable
multi-device histories up to depth 4, but its own terminal-state
compaction check (`check_compaction_safety`) only merges a device's
compacted register with its *own* pre-compaction snapshot — it does
not, by construction, exercise an unrelated device's independently-
live concurrent register. `test_cross_device_compaction_suppression`
(I5d) covers that shape directly but over a hand-parameterized cube,
not the full BFS state space; the accompanying
`test_tombstone_merge_monotonic` lemma is what extends the
no-resurrection guarantee beyond the cube's specific points.
- Two contexts are modeled. Production users may have hundreds of contexts,
but the CRDT properties are per-context — cross-context interactions are
limited to the shared byte budget (tested via trim/prune interaction).
- Multi-slot behavior is confirmed via split+merge convergence test, and
the atomic slot-grouping rule is modeled by `DeviceB.split_blob_into_slots`
(including the escaped-context identity fix — `split_blob_into_slots`
unescapes frontier keys before grouping). The production TypeScript
implementation (`splitContextsIntoBudgetedSlots`) is NOT modeled — only
the abstract grouping property is verified here. Implementation-level
testing is still needed for slot placement, slot rebalancing, and the
production d-tag coordinate assignment.
- The model assumes eventual delivery (all blobs eventually reach all
devices). Permanent message loss is not modeled.
- Byte sizes are computed from JSON serialization of realistic key names.
Actual encrypted blob overhead (NIP-59 envelope, relay metadata) adds
to the total but does not affect the 32 KiB plaintext budget.
File diff suppressed because it is too large Load Diff
+492
View File
@@ -0,0 +1,492 @@
"""Bounded exhaustive model comparing two NIP-RS manual-unread encodings.
Candidate A: lexicographic operation register
Per context: {counter, client_tiebreak, op in {SET,CLEAR}, baseline}
in a NEW top-level field beside `contexts`.
Merge = max tuple (counter, tiebreak, op-rule on full tie).
Candidate B: two grow-only counters + baseline
Per context: S (set counter), C (clear counter), B (frontier-at-set-time)
as sibling keys under `contexts` (ov_s:, ov_c:, ov_b: prefixes).
Action: own counter := max(S,C)+1; set also writes B := effective frontier.
Merge = componentwise max. Tie policy on S == C is a parameter.
Both share:
- Frontier: grow-only max() per NIP-RS v1 (unchanged).
- Verdict: unread(ctx) = latest > effective_frontier(ctx) OR override_set(ctx).
- Mark-read = advance frontier + clear override.
- Mark-unread = set override with baseline B = current effective frontier.
- Natural frontier advance strictly past B dominates a stale set.
Device simulators use overridable methods (_override_set, _compact, _merge_reg,
_bump, _sanitize_value) so the mutation harness can inject weakened rules via
subclassing without monkeypatching.
"""
from dataclasses import dataclass
from typing import Optional
SET = "SET"
CLEAR = "CLEAR"
# ---------------------------------------------------------------------------
# Reserved key namespace + escaping
#
# NIP-RS v1 context IDs are arbitrary UTF-8 (spec :89, :113-114), so a
# pre-existing opaque context could legitimately begin with `ov_s:`,
# `ov_c:`, or `ov_b:` and collide with a control key for a DIFFERENT
# context in the same flattened `contexts` map. `ov_` (the shared
# 3-byte stem) and the escape marker itself are reserved; any raw
# context ID that would collide is escaped before being used as a
# plain frontier key. Escaping is a no-op for every context ID Buzz
# actually generates (channel UUID, `msg:<hex64>`, `thread:<hex64>`
# — none start with `ov_` or `esc:`), so the common case pays zero
# bytes. Only a pathological ID pays the 4-byte `esc:` cost.
#
# This protects context IDs generated by amendment-aware clients.
# It does NOT retroactively protect a context that a PRE-EXISTING
# legacy client already published unescaped before the amendment
# shipped — that residual hazard is documented, not solved (see
# NOTE.md "Reserved key namespace").
# ---------------------------------------------------------------------------
ESCAPE_PREFIX = "esc:"
_RESERVED_STEM = "ov_"
def _needs_escape(raw_key: str) -> bool:
return raw_key.startswith(_RESERVED_STEM) or raw_key.startswith(ESCAPE_PREFIX)
def escape_context_key(raw_key: str) -> str:
return ESCAPE_PREFIX + raw_key if _needs_escape(raw_key) else raw_key
def unescape_context_key(wire_key: str) -> str:
if wire_key.startswith(ESCAPE_PREFIX):
return wire_key[len(ESCAPE_PREFIX):]
return wire_key
# ---------------------------------------------------------------------------
# Candidate B — two grow-only counters + baseline
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class RegB:
s: int = 0
c: int = 0
b: int = 0
def merge_reg_b(a: Optional[RegB], b: Optional[RegB]) -> Optional[RegB]:
if a is None:
return b
if b is None:
return a
return RegB(s=max(a.s, b.s), c=max(a.c, b.c), b=max(a.b, b.b))
def override_set_b(reg: Optional[RegB], frontier_val: int, tie_policy=CLEAR) -> bool:
if reg is None:
return False
if frontier_val > reg.b and reg.s > 0:
return False
if reg.s > reg.c:
return True
if reg.s == reg.c and reg.s > 0:
return tie_policy == SET
return False
def compact_b(reg: RegB, frontier_val: int, tie_policy=CLEAR) -> Optional[RegB]:
"""Compact override state.
Tombstone-floor design: a register with any recorded counter
activity (S>0 or C>0) is never fully deleted. Its counter
high-water-mark is exactly what prevents a stale replica —
any (S,C) pair below that ceiling — from dominating a freshly
created register after compaction (delete-on-dominance made
counters reusable: a dead register dropped entirely, then a new
local set/clear pair restarted from S=0/C=0, so a delayed stale
peer snapshot with S>0 could out-rank the new state on replay).
Only a virgin register (S==0, C==0, no activity ever recorded)
has no ceiling to protect and compacts to None.
A live override (per `override_set_b`, which is already
policy-aware) is returned unchanged — compaction only touches dead
state. Dead overrides — whether dominated by C>S, tied under
clear-wins, or baseline-dominated by frontier advance — compact to
the clear-tombstone floor `RegB(s=0, c=max(S,C), b=0)`: S is
zeroed (no longer overriding), but C retains the ceiling so both a
future local bump (`max(S,C)+1`) and a componentwise-max merge with
any pre-compaction stale snapshot start strictly above the
historical maximum, never below it.
"""
if reg.s == 0 and reg.c == 0:
return None
if override_set_b(reg, frontier_val, tie_policy):
return reg
return RegB(s=0, c=max(reg.s, reg.c), b=0)
# ---------------------------------------------------------------------------
# Candidate A — lexicographic operation register
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class RegA:
counter: int = 0
tiebreak: str = ""
op: str = CLEAR
baseline: int = 0
def as_tuple(self, op_wins):
op_val = 1 if self.op == op_wins else 0
return (self.counter, self.tiebreak, op_val)
def merge_reg_a(a: Optional[RegA], b: Optional[RegA], tie_op=CLEAR) -> Optional[RegA]:
if a is None:
return b
if b is None:
return a
at = a.as_tuple(tie_op)
bt = b.as_tuple(tie_op)
if at == bt:
return RegA(
counter=a.counter, tiebreak=a.tiebreak, op=a.op,
baseline=max(a.baseline, b.baseline),
)
return a if at > bt else b
# ---------------------------------------------------------------------------
# Device simulation — Candidate B
# ---------------------------------------------------------------------------
class DeviceB:
"""Simulates one device's NIP-RS read-state blob with manual-unread
override layer (candidate B encoding).
All model operations go through overridable _methods so the mutation
harness can inject weakened rules via subclassing.
"""
def __init__(self, client_id, is_legacy=False):
self.client_id = client_id
self.is_legacy = is_legacy
self.frontier = {}
self.overrides = {}
def effective_frontier(self, ctx):
return self.frontier.get(ctx, 0)
def _override_set(self, reg, frontier_val, tie_policy):
return override_set_b(reg, frontier_val, tie_policy)
def _compact(self, reg, frontier_val, tie_policy):
return compact_b(reg, frontier_val, tie_policy)
def _merge_reg(self, a, b):
return merge_reg_b(a, b)
def _bump(self, s, c):
return max(s, c) + 1
def _sanitize_value(self, v):
return isinstance(v, int) and 0 <= v <= 4294967295
def override_is_set(self, ctx, tie_policy=CLEAR):
return self._override_set(
self.overrides.get(ctx), self.effective_frontier(ctx), tie_policy
)
def verdict(self, ctx, latest_ts, tie_policy=CLEAR):
return (latest_ts > self.effective_frontier(ctx)
or self.override_is_set(ctx, tie_policy))
def do_mark_unread(self, ctx):
if self.is_legacy:
return
cur = self.overrides.get(ctx, RegB())
new_s = self._bump(cur.s, cur.c)
self.overrides[ctx] = RegB(s=new_s, c=cur.c, b=self.effective_frontier(ctx))
def do_mark_read(self, ctx, frontier_ts):
self.frontier[ctx] = max(self.frontier.get(ctx, 0), frontier_ts)
if not self.is_legacy:
cur = self.overrides.get(ctx, RegB())
new_c = self._bump(cur.s, cur.c)
self.overrides[ctx] = RegB(s=cur.s, c=new_c, b=cur.b)
def do_advance_frontier(self, ctx, ts):
self.frontier[ctx] = max(self.frontier.get(ctx, 0), ts)
def do_compact(self, ctx, tie_policy=CLEAR):
reg = self.overrides.get(ctx)
if reg is None:
return
result = self._compact(reg, self.effective_frontier(ctx), tie_policy)
if result is None:
if ctx in self.overrides:
del self.overrides[ctx]
else:
self.overrides[ctx] = result
def do_reinstall(self):
self.client_id = self.client_id + "_r"
self.frontier = {}
self.overrides = {}
def _canonicalize_for_publish(self, ctx, tie_policy):
"""Canonical published form of `ctx`'s override register,
computed fresh against the current effective frontier —
independent of whether `do_compact` was ever called locally.
Returns `(is_live, canonical_reg)`; `canonical_reg is None`
means virgin (omit from the wire entirely). Reuses the same
overridable `_compact`/`_override_set` hooks `do_compact` uses,
so a mutation-harness subclass that weakens one weakens both
the storage-GC path and the publish path identically.
"""
reg = self.overrides.get(ctx)
if reg is None:
return False, None
front = self.effective_frontier(ctx)
canonical = self._compact(reg, front, tie_policy)
if canonical is None:
return False, None
return self._override_set(canonical, front, tie_policy), canonical
def publish_blob(self, tie_policy=CLEAR):
"""Serialize this device's read-state blob.
Every override is canonicalized at serialization time: live ->
unchanged (3 keys), dead -> tombstone floor (1 key, `ov_c:`
only), virgin -> omitted (0 keys). Canonical publication is a
protocol requirement, not an optimization — noncanonical wire
output is structurally impossible here, not merely avoided by
convention. `do_compact` remains a separate storage-GC
transition that mutates `self.overrides`; publication no
longer depends on it having been called first.
**Atomic slot-grouping rule (spec-amendment requirement):**
A context's frontier entry and ALL of its `ov_*` sibling entries
MUST travel in the same slot. `split_blob_into_slots` below
enforces this by round-robining per-context groups, never
per-entry. A receiving client that only holds part of a context
group and attempts to reconstruct a `RegB` from it would see
partial zeroes and might canonically re-publish a false
tombstone. Group atomicity makes partial reconstruction
structurally impossible from a compliant publisher's output.
"""
blob_ctx = {escape_context_key(k): v for k, v in self.frontier.items()}
if not self.is_legacy:
for k in self.overrides:
is_live, canonical = self._canonicalize_for_publish(k, tie_policy)
if canonical is None:
continue # virgin: omitted from the wire entirely
if is_live:
blob_ctx[f"ov_s:{k}"] = canonical.s
blob_ctx[f"ov_c:{k}"] = canonical.c
blob_ctx[f"ov_b:{k}"] = canonical.b
else:
blob_ctx[f"ov_c:{k}"] = canonical.c # tombstone: ceiling only
return {"v": 1, "client_id": self.client_id, "contexts": blob_ctx}
def split_blob_into_slots(self, tie_policy=CLEAR, n_slots=2):
"""Split this device's blob into `n_slots` compliant slots.
**Atomic grouping rule:** a context's frontier entry and ALL of
its `ov_*` sibling entries travel together in the same slot.
Round-robin assignment is per-context group, never per-entry.
This matches production `splitContextsIntoBudgetedSlots` when
it is amended to group by context instead of by individual entry.
Returns a list of `n_slots` blobs, each with the same `v` and
`client_id` but a disjoint subset of context groups.
"""
blob = self.publish_blob(tie_policy)
contexts = blob["contexts"]
# Gather per-context groups: each group is a list of (key, value) pairs.
# A "group" is: the frontier key (escaped ctx) + any ov_* siblings.
# Contexts that appear only as ov_* keys (no frontier entry) are
# also grouped together.
groups = {} # logical_ctx -> list of (wire_key, value)
for wire_key, value in contexts.items():
if wire_key.startswith("ov_s:"):
ctx = wire_key[5:]
elif wire_key.startswith("ov_c:"):
ctx = wire_key[5:]
elif wire_key.startswith("ov_b:"):
ctx = wire_key[5:]
else:
# Frontier key: may be escaped (e.g. "esc:ov_s:evil").
# Derive the logical context ID by unescaping so this
# entry joins the same group as its ov_* siblings, which
# are keyed by the RAW context ID (e.g. "ov_s:evil" ->
# ctx = "evil", but "esc:ov_s:evil" frontier -> ctx =
# "ov_s:evil" after unescape). Without this step an
# escaped frontier key and its ov_* siblings would be
# treated as two different groups, splitting the register
# across slots — reproducing the round-1 partial-
# reconstruction poison for escaped context IDs.
ctx = unescape_context_key(wire_key)
groups.setdefault(ctx, []).append((wire_key, value))
slots = [{"v": blob["v"], "client_id": blob["client_id"], "contexts": {}}
for _ in range(n_slots)]
for i, (_ctx, pairs) in enumerate(sorted(groups.items())):
slot = slots[i % n_slots]
for wire_key, value in pairs:
slot["contexts"][wire_key] = value
return slots
def receive_merge(self, blob):
incoming_overrides = {}
for k, v in blob.get("contexts", {}).items():
if k.startswith("ov_s:"):
ctx = k[5:]
incoming_overrides.setdefault(ctx, [0, 0, 0])[0] = v
elif k.startswith("ov_c:"):
ctx = k[5:]
incoming_overrides.setdefault(ctx, [0, 0, 0])[1] = v
elif k.startswith("ov_b:"):
ctx = k[5:]
incoming_overrides.setdefault(ctx, [0, 0, 0])[2] = v
else:
ctx = unescape_context_key(k)
self.frontier[ctx] = max(self.frontier.get(ctx, 0), v)
if not self.is_legacy:
for ctx, (s, c, b) in incoming_overrides.items():
incoming_reg = RegB(s=s, c=c, b=b)
self.overrides[ctx] = self._merge_reg(
self.overrides.get(ctx), incoming_reg
)
def legacy_sanitize_and_publish(self, tie_policy=CLEAR):
blob = self.publish_blob(tie_policy)
sanitized = {}
for k, v in blob["contexts"].items():
if len(k.encode("utf-8")) <= 256 and self._sanitize_value(v):
sanitized[k] = v
return {"v": 1, "client_id": self.client_id, "contexts": sanitized}
def state_key(self, contexts, tie_policy=CLEAR):
parts = []
for ctx in sorted(contexts):
f = self.effective_frontier(ctx)
reg = self.overrides.get(ctx, RegB())
ov = self.override_is_set(ctx, tie_policy)
parts.append((ctx, f, reg.s, reg.c, reg.b, ov))
return (self.client_id, self.is_legacy, tuple(parts))
# ---------------------------------------------------------------------------
# Device simulation — Candidate A
# ---------------------------------------------------------------------------
class DeviceA:
def __init__(self, client_id, is_legacy=False):
self.client_id = client_id
self.is_legacy = is_legacy
self.frontier = {}
self.overrides = {}
self.counter = 0
def effective_frontier(self, ctx):
return self.frontier.get(ctx, 0)
def override_is_set(self, ctx):
reg = self.overrides.get(ctx)
if reg is None or reg.op == CLEAR:
return False
if self.effective_frontier(ctx) > reg.baseline:
return False
return True
def verdict(self, ctx, latest_ts):
return latest_ts > self.effective_frontier(ctx) or self.override_is_set(ctx)
def do_mark_unread(self, ctx):
if self.is_legacy:
return
self.counter += 1
self.overrides[ctx] = RegA(
counter=self.counter, tiebreak=self.client_id,
op=SET, baseline=self.effective_frontier(ctx),
)
def do_mark_read(self, ctx, frontier_ts):
self.frontier[ctx] = max(self.frontier.get(ctx, 0), frontier_ts)
if not self.is_legacy:
self.counter += 1
self.overrides[ctx] = RegA(
counter=self.counter, tiebreak=self.client_id,
op=CLEAR, baseline=0,
)
def do_advance_frontier(self, ctx, ts):
self.frontier[ctx] = max(self.frontier.get(ctx, 0), ts)
def receive_merge(self, blob, tie_op=CLEAR):
for ctx, ts in blob.get("contexts", {}).items():
self.frontier[ctx] = max(self.frontier.get(ctx, 0), ts)
if not self.is_legacy:
for ctx, reg in blob.get("overrides", {}).items():
self.overrides[ctx] = merge_reg_a(
self.overrides.get(ctx), reg, tie_op
)
if reg.counter > self.counter:
self.counter = reg.counter
def publish_blob(self):
blob = {"v": 1, "client_id": self.client_id, "contexts": dict(self.frontier)}
if not self.is_legacy:
blob["overrides"] = dict(self.overrides)
return blob
def legacy_rewrite_and_publish(self):
return {"v": 1, "client_id": self.client_id, "contexts": dict(self.frontier)}
# ---------------------------------------------------------------------------
# Legacy pruning/trim model
# ---------------------------------------------------------------------------
def legacy_prune(contexts, horizon):
return {k: v for k, v in contexts.items()
if not (k.startswith("msg:") or k.startswith("thread:")) or v >= horizon}
def legacy_trim(contexts, client_id, max_bytes=32768):
import json
def size(ctx):
return len(json.dumps({"v": 1, "client_id": client_id, "contexts": ctx}).encode())
if size(contexts) <= max_bytes:
return contexts, True
evictable = sorted(
((k, v) for k, v in contexts.items()
if k.startswith("msg:") or k.startswith("thread:")),
key=lambda kv: kv[1],
)
out = dict(contexts)
for k, _ in evictable:
del out[k]
if size(out) <= max_bytes:
return out, True
return out, size(out) <= max_bytes
def legacy_sanitize_blob(blob):
sanitized = {}
for k, v in blob.get("contexts", {}).items():
if (len(k.encode("utf-8")) <= 256
and isinstance(v, int) and 0 <= v <= 4294967295):
sanitized[k] = v
return {"v": 1, "client_id": blob.get("client_id", ""), "contexts": sanitized}
+519
View File
@@ -0,0 +1,519 @@
"""Mutation harness for candidate B (two-counter) model.
Each mutant: subclass DeviceB with a weakened rule, run the BFS explorer,
require a recorded counterexample. A model that stays green under a real
weakening is worthless.
Mutants:
M1: drop baseline dominance (frontier > B no longer clears stale set)
M2: drop max(S,C)+1 bump (use S+1 or C+1 — counter can regress)
M3: flip tie policy (verify the model distinguishes them)
M4: revert to delete-on-dominance compaction (drops the tombstone floor
entirely instead of zeroing S and keeping max(S,C) as C) — reproduces
Thufir's pass-3 CRITICAL: stale-replay resurrection after counter reuse
M5: uint32 overflow bypass (legacy sanitization disabled)
M6: componentwise-max -> last-write-wins merge (convergence breaks)
M7: publish without canonicalization (serialize raw registers instead
of the compact-at-publish canonical form) — reproduces Thufir's
pass-1/2 CRITICAL: dead+dead merge resurrection
M8: revert split_blob_into_slots to per-entry splitting (violates the
atomic-grouping rule) — reproduces Thufir's pass-2/2 CRITICAL:
partial-slot reconstruction of a live RegB creates a false tombstone
that permanently suppresses the override after eventual full delivery
M9: revert split_blob_into_slots to escaped-key grouping (groups frontier
by wire key instead of unescaped logical ID) — reproduces Thufir's
round-2 CRITICAL: for a context whose raw ID starts with a reserved
prefix (e.g. "ov_s:evil"), the frontier's escaped wire key
("esc:ov_s:evil") and the ov_* siblings (keyed by raw suffix "ov_s:evil")
resolve to different groups → register split across slots →
old/new slot-coordinate mixture produces partial reconstruction →
false tombstone → permanent false clear across publication cycles
Each mutant is injected into the model via DeviceB subclass, then the
explorer or invariant suite is rerun. The counterexample (first violation)
is recorded and printed.
"""
from copy import deepcopy
from model import (
RegB, merge_reg_b, override_set_b, compact_b,
DeviceB, legacy_sanitize_blob,
escape_context_key,
SET, CLEAR,
)
from exhaustive import (
explore_b, test_concurrent_stability,
test_compaction_register_exhaustive, test_deep_history_compaction,
test_published_merge_closure, test_interleaved_delivery_grouping,
test_escaped_context_slot_grouping,
CONTEXTS,
)
# ---------------------------------------------------------------------------
# M1: drop baseline dominance
# ---------------------------------------------------------------------------
class M1_NoBaselineDominance(DeviceB):
def _override_set(self, reg, frontier_val, tie_policy):
if reg is None:
return False
if reg.s > reg.c:
return True
if reg.s == reg.c and reg.s > 0:
return tie_policy == SET
return False
def _compact(self, reg, frontier_val, tie_policy):
if reg.s == 0 and reg.c == 0:
return None
if self._override_set(reg, frontier_val, tie_policy):
return reg
if reg.c > reg.s:
return RegB(s=0, c=reg.c, b=0)
if reg.c == reg.s and tie_policy == CLEAR:
return RegB(s=0, c=reg.c, b=0)
return reg
def mutant_m1():
"""M1: without baseline dominance, a stale set persists after frontier
advance past baseline. Verify by constructing the scenario directly:
mark-unread at frontier=10, then advance frontier to 100. The correct
model clears the override; the mutant keeps it live."""
violations = []
for ctx in CONTEXTS:
dev = M1_NoBaselineDominance("d0")
dev.frontier[ctx] = 10
dev.do_mark_unread(ctx)
dev.do_advance_frontier(ctx, 100)
correct = override_set_b(dev.overrides[ctx], 100, CLEAR)
mutant_result = dev.override_is_set(ctx, CLEAR)
if correct != mutant_result:
violations.append((
"baseline-dominance-missing", ctx,
dev.overrides[ctx], 100,
f"correct={correct}", f"mutant={mutant_result}",
))
if not violations:
_, violations = explore_b(max_depth=3, tie_policy=CLEAR,
device_cls=M1_NoBaselineDominance)
return violations
# ---------------------------------------------------------------------------
# M2: drop max(S,C)+1 bump
# ---------------------------------------------------------------------------
class M2_NoBump(DeviceB):
"""Each counter bumps only itself: mark_unread does S := S+1,
mark_read does C := C+1. When S > C from a prior set, a clear
at C+1 can produce C < S even though the clear is causally later."""
def do_mark_unread(self, ctx):
if self.is_legacy:
return
cur = self.overrides.get(ctx, RegB())
self.overrides[ctx] = RegB(s=cur.s + 1, c=cur.c,
b=self.effective_frontier(ctx))
def do_mark_read(self, ctx, frontier_ts):
self.frontier[ctx] = max(self.frontier.get(ctx, 0), frontier_ts)
if not self.is_legacy:
cur = self.overrides.get(ctx, RegB())
self.overrides[ctx] = RegB(s=cur.s, c=cur.c + 1, b=cur.b)
def mutant_m2():
"""M2: each counter bumps independently. After set→set→clear at
the SAME frontier (no advance past baseline): correct clear has
C=3 > S=2, mutant clear has C=1 < S=2 — a causally later clear
fails to dominate.
Use mark_read at the current frontier (not advancing past baseline)
so baseline dominance doesn't mask the counter discrepancy.
"""
violations = []
for ctx in CONTEXTS:
front = 10
dev_correct = DeviceB("d0")
dev_correct.frontier[ctx] = front
dev_correct.do_mark_unread(ctx)
dev_correct.do_mark_unread(ctx)
dev_correct.do_mark_read(ctx, front)
dev_mutant = M2_NoBump("d0")
dev_mutant.frontier[ctx] = front
dev_mutant.do_mark_unread(ctx)
dev_mutant.do_mark_unread(ctx)
dev_mutant.do_mark_read(ctx, front)
correct_set = dev_correct.override_is_set(ctx, CLEAR)
mutant_set = dev_mutant.override_is_set(ctx, CLEAR)
if correct_set != mutant_set:
violations.append((
"bump-independent", ctx,
f"correct={dev_correct.overrides[ctx]}",
f"mutant={dev_mutant.overrides[ctx]}",
f"correct_set={correct_set}", f"mutant_set={mutant_set}",
))
if not violations:
_, violations = explore_b(max_depth=4, tie_policy=CLEAR,
device_cls=M2_NoBump)
return violations
# ---------------------------------------------------------------------------
# M3: tie policy distinguishable
# ---------------------------------------------------------------------------
def mutant_m3():
"""M3: tie policy is load-bearing — S==C must produce different verdicts.
Not a DeviceB mutation; tests the model function directly."""
reg = RegB(s=1, c=1, b=10)
frontier = 10
v_clear = override_set_b(reg, frontier, CLEAR)
v_set = override_set_b(reg, frontier, SET)
if v_clear == v_set:
return []
return [("tie-distinguishable", v_clear, v_set, reg, frontier)]
# ---------------------------------------------------------------------------
# M4: revert to delete-on-dominance compaction (drops the tombstone floor)
# ---------------------------------------------------------------------------
class M4_DeleteOnDominance(DeviceB):
"""The pre-fix compaction rule: any dead/dominated register is deleted
entirely rather than reduced to the tombstone floor RegB(0, max(S,C), 0).
This makes counters reusable — a later local set/clear pair restarts
from S=0/C=0, so a delayed stale peer snapshot can dominate it on
replay. This is exactly the rule Thufir's pass-3 CRITICAL found live
at e453b3945."""
def _compact(self, reg, frontier_val, tie_policy):
if reg.s == 0 and reg.c == 0:
return None
if self._override_set(reg, frontier_val, tie_policy):
return reg
if frontier_val > reg.b:
return None
if reg.c > reg.s:
return RegB(s=0, c=reg.c, b=0)
if reg.c == reg.s and tie_policy == CLEAR:
return RegB(s=0, c=reg.c, b=0)
return reg
def mutant_m4():
"""M4: without the tombstone floor, compaction deletes the counter
ceiling instead of preserving it. Reproduce Thufir's exact witness
directly: RegB(3,0,10) at frontier=20 compacts to None under the old
rule (vs. RegB(0,3,0) under the fix); a subsequent local set+clear
reuses counters from zero; the stale ancestor then replays and
resurrects (S>C) under both tie policies.
Then confirm the explorer/deep-history suite also catches it (defense
in depth — a mutant that only fails a hand-built scenario would still
be a real bug, but the directed check is what's supposed to catch this
class per T2/T3)."""
violations = []
stale = RegB(s=3, c=0, b=10)
frontier_after = 20
for tie_policy in (CLEAR, SET):
dev = M4_DeleteOnDominance("d0")
dev.frontier["c0"] = 10
dev.overrides["c0"] = stale
dev.do_advance_frontier("c0", frontier_after)
dev.do_compact("c0", tie_policy)
if "c0" in dev.overrides:
continue # old rule didn't drop it here; not the witness shape
dev.do_mark_unread("c0") # S := 1, B := 20
dev.do_mark_read("c0", frontier_after) # C := 2
stale_blob = {"contexts": {"ov_s:c0": stale.s, "ov_c:c0": stale.c, "ov_b:c0": stale.b}}
dev.receive_merge(stale_blob)
resurrected = dev.override_is_set("c0", tie_policy)
if resurrected:
violations.append((
"M4-delete-on-dominance-resurrection", tie_policy,
f"stale_ancestor={stale}", f"post_compact_reuse=(set,clear)",
f"final_reg={dev.overrides['c0']}", f"override_is_set={resurrected}",
))
if not violations:
_, violations = test_deep_history_compaction(device_cls=M4_DeleteOnDominance)
return violations
# ---------------------------------------------------------------------------
# M5: uint32 overflow bypass
# ---------------------------------------------------------------------------
def mutant_m5():
"""M5: values outside uint32 range must fail legacy sanitization."""
blob = {"v": 1, "client_id": "x", "contexts": {
"ov_s:c0": 4294967296,
"ov_c:c0": 0,
"ov_b:c0": 10,
}}
sanitized = legacy_sanitize_blob(blob)
if "ov_s:c0" in sanitized["contexts"]:
return []
return [("overflow-rejected", blob["contexts"]["ov_s:c0"],
sanitized["contexts"])]
# ---------------------------------------------------------------------------
# M6: last-write-wins merge (breaks convergence)
# ---------------------------------------------------------------------------
class M6_LastWriteWins(DeviceB):
def _merge_reg(self, a, b):
if a is None:
return b
if b is None:
return a
return b
def mutant_m6():
"""M6: replace componentwise max with last-write-wins. Convergence must
break — different delivery orders produce different final states."""
_, violations = explore_b(max_depth=3, tie_policy=CLEAR,
device_cls=M6_LastWriteWins)
return violations
# ---------------------------------------------------------------------------
# M7: publish without canonicalization (reproduces Thufir's pass-1/2
# CRITICAL — dead+dead merge resurrection)
# ---------------------------------------------------------------------------
class M7_PublishWithoutCanonicalization(DeviceB):
"""Reverts `publish_blob` to serialize raw, uncompacted registers —
the exact pre-fix behavior Thufir's pass-1/2 CRITICAL exploited:
a dead register's baseline-relative death (or clear-count-relative
death) never gets folded into a globally-comparable ceiling before
hitting the wire, so two individually-dead registers can
componentwise-max-merge into a live join."""
def publish_blob(self, tie_policy=CLEAR):
blob_ctx = {escape_context_key(k): v for k, v in self.frontier.items()}
if not self.is_legacy:
for k, reg in self.overrides.items():
blob_ctx[f"ov_s:{k}"] = reg.s
blob_ctx[f"ov_c:{k}"] = reg.c
blob_ctx[f"ov_b:{k}"] = reg.b
return {"v": 1, "client_id": self.client_id, "contexts": blob_ctx}
def mutant_m7():
"""M7: publish-without-canonicalization must be caught by the
published-state merge-closure invariant — proving that invariant
has teeth. Reproduce Thufir's exact witness directly first (fast,
deterministic); fall back to the full search if the hand-built
scenario doesn't trigger under a given tie policy."""
violations = []
for tie_policy in (CLEAR, SET):
dev_a = M7_PublishWithoutCanonicalization("a")
dev_a.frontier["c0"] = 50
dev_a.overrides["c0"] = RegB(s=3, c=2, b=0)
dev_b = M7_PublishWithoutCanonicalization("b")
dev_b.frontier["c0"] = 100
dev_b.overrides["c0"] = RegB(s=1, c=2, b=100)
blob_a = dev_a.publish_blob(tie_policy)
blob_b = dev_b.publish_blob(tie_policy)
for first, second in [(blob_a, blob_b), (blob_b, blob_a)]:
recv = M7_PublishWithoutCanonicalization("recv")
recv.receive_merge(first)
recv.receive_merge(second)
if recv.override_is_set("c0", tie_policy):
violations.append((
"M7-publish-without-canonicalization-resurrection",
tie_policy, blob_a, blob_b, recv.overrides["c0"],
))
if not violations:
_, violations = test_published_merge_closure(
device_cls=M7_PublishWithoutCanonicalization
)
return violations
# ---------------------------------------------------------------------------
# M8: revert split_blob_into_slots to per-entry splitting
# (violates the atomic-grouping rule — reproduces Thufir's pass-2/2 CRITICAL)
# ---------------------------------------------------------------------------
class M8_PerEntrySplit(DeviceB):
"""Reverts `split_blob_into_slots` to a per-entry split that violates the
atomic-grouping rule by separating `ov_s:` + frontier from `ov_b:` + `ov_c:`.
This reproduces Thufir's exact transport witness:
- Slot 0: frontier key + `ov_s:` entry (the "partial set" slot)
- Slot 1: `ov_c:` + `ov_b:` entries
An observer receiving only slot 0 reconstructs `RegB(s=1, c=0, b=0)` at
`frontier=10`. Because `frontier(10) > b(0)`, the override is baseline-dead.
Canonical re-publication emits tombstone `RegB(0, 1, 0)`. After full
eventual delivery (both original slots + transient tombstone), the merged
result is `RegB(s=1, c=1, b=10)` — dead under clear-wins — permanently
suppressing a live override.
"""
def split_blob_into_slots(self, tie_policy=CLEAR, n_slots=2):
"""Split by key type: frontier + ov_s: in slot 0, ov_b: + ov_c: in slot 1.
Violates the atomic-grouping rule by separating ov_s: from ov_b:."""
blob = self.publish_blob(tie_policy)
slots = [{"v": blob["v"], "client_id": blob["client_id"], "contexts": {}}
for _ in range(n_slots)]
for wire_key, value in blob["contexts"].items():
if wire_key.startswith("ov_b:") or wire_key.startswith("ov_c:"):
# ov_b and ov_c go to slot 1 — separated from their ov_s: sibling
slots[1]["contexts"][wire_key] = value
else:
# frontier keys and ov_s: go to slot 0
slots[0]["contexts"][wire_key] = value
return slots
def mutant_m8():
"""M8: per-entry splitting must be caught by test_interleaved_delivery_grouping —
proving that the new interleaved-delivery test has teeth.
Reproduce Thufir's exact transport witness directly: source live
`RegB(1,0,10)` at frontier=10. Per-entry split puts frontier+`ov_s:c0`
in slot 0 and `ov_c:c0`+`ov_b:c0` in slot 1. An observer receiving only
slot 0 reconstructs `RegB(1,0,0)`, re-publishes tombstone `RegB(0,1,0)`.
Full merge including the transient: `RegB(1,1,10)` → inactive.
Confirmed by running test_interleaved_delivery_grouping with M8_PerEntrySplit;
the witness must be caught before resorting to the full suite."""
return test_interleaved_delivery_grouping(device_cls=M8_PerEntrySplit)
# ---------------------------------------------------------------------------
# M9: revert split_blob_into_slots to escaped-key grouping
# (groups frontier by wire key instead of unescaped logical ID —
# reproduces Thufir's round-2 CRITICAL)
# ---------------------------------------------------------------------------
class M9_EscapedKeyGrouping(DeviceB):
"""Reverts `split_blob_into_slots` to group the frontier key by its
ESCAPED wire key rather than the unescaped logical context ID.
For a normal context like "c0", this is a no-op (escape_context_key("c0")
== "c0"), so M9 is identical to the correct model on normal contexts.
The defect only manifests when the raw context ID starts with a reserved
prefix — e.g. raw "ov_s:evil" escapes to frontier wire key "esc:ov_s:evil".
The ov_* sibling keys are keyed by the RAW suffix ("ov_s:evil"), while
the frontier is keyed by the escaped wire key ("esc:ov_s:evil") — two
identities for one logical context, so they land in different slots.
This reproduces Thufir's round-2 CRITICAL: across publication cycles an
observer can receive the new frontier slot (esc:ov_s:evil=10) plus the
stale old-cycle override slot (ov_s/ov_c/ov_b at b=0), reconstructing
RegB(s=1,c=0,b=0) at frontier=10 — baseline-dead — and emitting tombstone
RegB(0,1,0). Full eventual delivery merges to RegB(1,1,10) — dead under
clear-wins — permanently suppressing a live override.
"""
def split_blob_into_slots(self, tie_policy=CLEAR, n_slots=2):
"""Split by original (escaped) wire key identity — does not unescape
frontier keys before grouping, so escaped contexts split incorrectly."""
blob = self.publish_blob(tie_policy)
contexts = blob["contexts"]
groups = {} # wire_key -> list of (wire_key, value)
for wire_key, value in contexts.items():
if wire_key.startswith("ov_s:"):
ctx = wire_key[5:]
elif wire_key.startswith("ov_c:"):
ctx = wire_key[5:]
elif wire_key.startswith("ov_b:"):
ctx = wire_key[5:]
else:
ctx = wire_key # frontier: use escaped wire key as group ID (BUG)
groups.setdefault(ctx, []).append((wire_key, value))
slots = [{"v": blob["v"], "client_id": blob["client_id"], "contexts": {}}
for _ in range(n_slots)]
for i, (_ctx, pairs) in enumerate(sorted(groups.items())):
slot = slots[i % n_slots]
for wire_key, value in pairs:
slot["contexts"][wire_key] = value
return slots
def mutant_m9():
"""M9: escaped-key grouping must be caught by test_escaped_context_slot_grouping —
proving that the escaped-context regression test has teeth.
For a context whose raw ID starts with a reserved prefix ("ov_s:evil"),
the frontier wire key is "esc:ov_s:evil" and the ov_* sibling keys are
"ov_s:ov_s:evil", "ov_c:ov_s:evil", "ov_b:ov_s:evil". The escaped-key
grouping treats "esc:ov_s:evil" (frontier) and "ov_s:evil" (ov_* suffix)
as different groups, splitting the register across slots.
Old/new slot-coordinate mixture across publication cycles then reproduces
the round-1 transport poison: partial reconstruction → false tombstone →
permanent false clear of a live override.
The test is parameterized to route through the "mismatched grouping" path
(else branch) when the M9 split puts frontier and siblings in different slots,
and the witness must be caught."""
return test_escaped_context_slot_grouping(device_cls=M9_EscapedKeyGrouping)
# ---------------------------------------------------------------------------
# Runner
# ---------------------------------------------------------------------------
def run_mutations():
mutants = [
("M1: drop baseline dominance", mutant_m1),
("M2: drop max(S,C)+1 bump", mutant_m2),
("M3: tie policy distinguishable", mutant_m3),
("M4: revert to delete-on-dominance compaction (reproduces pass-3 CRITICAL)", mutant_m4),
("M5: uint32 overflow bypass", mutant_m5),
("M6: last-write-wins merge", mutant_m6),
("M7: publish without canonicalization (reproduces pass-1/2 CRITICAL)", mutant_m7),
("M8: per-entry split violates atomic-grouping rule (reproduces pass-2/2 CRITICAL)", mutant_m8),
("M9: escaped-key grouping splits escaped-ctx register across slots (reproduces round-2 CRITICAL)", mutant_m9),
]
print("=" * 60)
print("Mutation harness — candidate B")
print("=" * 60)
caught = []
missed = []
for name, fn in mutants:
violations = fn()
if violations:
caught.append(name)
v = violations[0]
detail = str(v)[:200]
print(f" CAUGHT: {name}")
print(f" counterexample: {detail}")
else:
missed.append(name)
print(f" MISSED: {name}")
print(f"\nCaught {len(caught)}/{len(mutants)} mutants")
if missed:
print(f"MISSED: {missed}")
print("=" * 60)
return len(missed) == 0
if __name__ == "__main__":
import sys
sys.exit(0 if run_mutations() else 1)