Event Flow
Olly's asynchronous backbone is Kafka, running as a single broker in docker-compose on dev-2 (there is no Amazon MSK / managed cluster today). Events flow between services using the choreography-based saga pattern - no central orchestrator, just producers and consumers reacting to domain events. Each service publishes via a transactional outbox (or a direct producer for the DB-free services) and every message is the canonical JSON envelope {eventId, eventType, occurredAt, partyLocator, correlationId, causationId, sessionId, activityId, activityName, schemaVersion, payload, state} (packages/go/domain/event_envelope.go). Every event type's contract - payload schema, state subjects, lineage, version, producers and consumers - is defined in the event registry (see Schema governance below and the generated Event Registry catalog).
Kafka Topology
Topics are one per service, named <service>.events (claims.events, billing.events, enrollment.events, eligibility.events, provider.events, care.events, consent.events). Every event type a service emits is written to its single <service>.events topic and discriminated by the envelope eventType field. The 3-segment topic names in the catalog tables below (claims.claim.submitted, etc.) are the logical event classes, not physical Kafka topics.
Topic Configuration Defaults
Unless noted otherwise, topics use these settings:
| Parameter | Default | Notes |
|---|---|---|
replication.factor | 1 | Single broker in docker-compose (dev-2); no multi-broker cluster today |
retention.ms | 604800000 (7 days) | Sufficient for consumer lag recovery |
compression.type | lz4 | Good ratio/speed balance |
max.message.bytes | 1 MB | |
partitions | 6 |
🚧 Target-state - not yet built
Multi-broker durability (replication.factor=3, min.insync.replicas=2, acks=all) and per-topic partition/retention tuning are planned for a managed-Kafka production target but are not provisioned today (infra/terraform and infra/k8s contain only .gitkeep).
Consumer Group Naming
Consumer groups follow the pattern {service}-{domain}-consumer. Examples:
billing-claims-consumer- Billing Service consuming from claims topicsnotifications-enrollment-consumer- Notifications Service consuming from enrollment topicseligibility-enrollment-consumer- Eligibility Service consuming from enrollment topics
Schema governance
As-built: the event registry (packages/go/domain/eventregistry) defines one contract per eventType: payload JSON Schema, required event-carried state subjects, lineage requirements, the contract version (retired versions stay in priorVersions so old events remain processable and are judged by the contract they were written under), producers/consumers as path:func refs, and golden examples that double as its test suite. Enforcement runs in CI (generic golden test), in e2e (tests/e2e/registry_compliance_test.go judges every wire event of a real priced quote), and at runtime (the debugger attaches a contract verdict to every event it ingests - live contract drift is visible on every smoke walk, and the stale-stack events of #1782 show up as precisely-indicted invalids).
🚧 Target-state - not yet built
IDL codegen (buf over the existing api/proto/olly/*/v1/ protos, buf breaking in CI, generated Go/TS/Python payload types) remains planned (#1766). It generates into the same registry layout; the enforcement machinery stays.
Event Catalog
The as-built catalog is generated from the registry: Event Registry - 36 registered types with schemas, state/lineage contracts, producers and consumers. The per-domain tables below are the logical topology (3-segment names are event classes, not physical topics) with target-state partition/retention values; treat the registry as the source of truth for anything a producer or consumer must agree on.
Claims Domain
| Topic | Producer | Consumers | Partitions | Retention |
|---|---|---|---|---|
claims.claim.submitted | Claims | Billing, Notifications | 12 | 14 days |
claims.claim.adjudicated | Claims | Billing, Notifications, Mirth Connect | 12 | 14 days |
claims.claim.paid | Billing (primary), Claims (mirror) | Notifications, OpenSearch | 6 | 7 days |
claims.prior_auth.decision | Claims | Notifications, Eligibility | 6 | 7 days |
claims.appeal.resolved | Claims | Notifications, Billing | 6 | 7 days |
claims.claim.adjudicated event types: CLAIM_APPROVED, CLAIM_DENIED
Key fields on approval: claimId, allowedAmount, paidAmount, memberResponsibility, copayApplied, eobS3Key
Key fields on denial: claimId, denialReasonCode (X12 code, e.g. CO-4), denialReasonText, appealDeadline
Eligibility Domain
| Topic | Producer | Consumers | Retention |
|---|---|---|---|
eligibility.coverage.verified | Eligibility | Claims | 7 days |
eligibility.coverage.terminated | Eligibility | Claims, Notifications | 7 days |
eligibility.coverage.verified carries accumulator balances (deductibleRemaining, oopMaxRemaining), network tier, and coverage dates - the Claims Service uses this to compute member cost-sharing without a synchronous lookup.
Enrollment Domain
| Topic | Producer | Consumers | Retention |
|---|---|---|---|
enrollment.enrollment.submitted | Enrollment | Eligibility, Billing, Notifications | 7 days |
enrollment.enrollment.activated | Enrollment | Eligibility, Billing, Notifications | 7 days |
enrollment.enrollment.terminated | Enrollment | Eligibility, Billing, Notifications | 7 days |
enrollment.enrollment.plan_changed | Enrollment | Eligibility, Billing, Notifications | 7 days |
enrollment.cobra.notice_sent | Enrollment | Notifications | 7 days |
enrollment.cobra.elected | Enrollment | Eligibility, Billing, Notifications | 7 days |
enrollment.enrollment.activated is the key saga trigger: it activates the pending coverage record in Eligibility, starts the recurring invoice schedule in Billing, and sends the enrollment confirmation in Notifications.
Billing Domain
| Topic | Producer | Consumers | Partitions | Retention |
|---|---|---|---|---|
billing.invoice.generated | Billing | Notifications | 12 | 14 days |
billing.payment.received | Billing | Enrollment, Notifications | 6 | 7 days |
billing.payment.missed | Billing | Enrollment, Notifications | 6 | 7 days |
billing.payment.completed | Billing | Claims, Notifications | 6 | 7 days |
billing.payment.missed carries three event types: PAYMENT_MISSED, GRACE_PERIOD_STARTED, and GRACE_PERIOD_EXPIRED. The GRACE_PERIOD_EXPIRED event triggers Enrollment Service to begin the COBRA timeline.
Provider Domain
| Topic | Producer | Consumers | Retention |
|---|---|---|---|
provider.credentialing.status_changed | Provider | Claims, Eligibility, Notifications | 7 days |
provider.network.updated | Provider | Claims, Eligibility | 7 days |
provider.credentialing.status_changed event types: CREDENTIALING_APPROVED, CREDENTIALING_DENIED, CREDENTIALING_EXPIRED, CREDENTIALING_SUSPENDED. Claims and Eligibility update their local projection tables on receipt.
EDI Domain
🚧 Target-state - not yet built (and out of market scope)
Olly operates in the United Kingdom. The X12/EDI clearinghouse model below is a carried-over US plan and is 0% built - there are no edi.* topics, no Mirth X12 pipeline, and no raw-EDI object storage in the running system. Retained as historical reference only.
| Topic | Producer | Consumers | Retention |
|---|---|---|---|
edi.inbound.received | Mirth Connect | Claims, Enrollment, Eligibility | 30 days |
edi.outbound.generated | Mirth Connect | OpenSearch (audit) | 30 days |
Outbox Pattern
Every Kafka publish is made safe against partial failures by the transactional outbox pattern:
The outbox worker runs as a background goroutine in each service and polls SELECT ... WHERE published_at IS NULL ORDER BY created_at ASC LIMIT n (no FOR UPDATE SKIP LOCKED - correct because exactly one replica runs the worker today). If the service restarts between the Kafka produce and the published_at stamp, the message is re-published on restart (at-least-once). Consumers in billing and eligibility dedupe durably on the envelope eventId via a processed_events inbox table (check-before-handle, mark-after-success; billing migration 0025, eligibility 0010); the remaining consumers dedupe opportunistically and rely on idempotent, state-transition-guarded handlers.
🚧 Target-state - not yet built
FOR UPDATE SKIP LOCKED on the outbox fetch (to make multi-replica publishing safe) and extending the processed_events inbox pattern beyond billing/eligibility are planned but not implemented.
Dead Letter Queue (DLQ) Pattern
🚧 Target-state - not yet built
There is no DLQ in the current build. A consumer that fails to process a message retries it on redelivery; there are no <topic>.dlq topics, no bounded-retry-then-route logic, and no DLQ consumer. The design below is the planned target.
The target design has every Kafka consumer implement exponential-backoff retry followed by DLQ routing on persistent failure:
Normal Flow:
Topic → Consumer → Process → Commit offset
Failure Flow (after 3 retries with backoff: 1s, 2s, 4s):
Topic → Consumer → Process FAILS
│
▼
DLQ Topic: {original-topic}.dlq (30-day retention)
│
▼
DLQ Consumer: alert to Grafana OnCall + write to OpenSearch for manual reviewDLQ messages would wrap the original payload with failure metadata: originalTopic, originalPartition, originalOffset, failureReason, retryCount, and consumerGroup, with no DLQ of the DLQ (failures in the DLQ consumer logged to OpenSearch and alerted for manual intervention).
Enrollment Saga
The enrollment saga coordinates new member activation from submission through to active coverage and first invoice:
Compensation path: If activation fails (e.g. validation rejection or lapsed offer), Enrollment Service publishes enrollment.enrollment.cancelled. Eligibility deletes the pending coverage record, Billing cancels the staged invoice, and Notifications sends an error message to the member.
Note
The enrollment.enrollment.submitted → activated choreography and the outbox/Kafka mechanics are real. The earlier "carrier 834 ack / 999 rejection" wording was US-EDI framing and has been removed - there is no EDI carrier handshake in the UK build.