Olly Coding Guidelines & Patterns

1. Architecture Overview

Olly is a health insurance platform built as a Go microservices monorepo. Eight backend services share a PostgreSQL cluster (separate database per service), communicate via Kafka events, and authenticate with Keycloak JWTs.

The domain model and ERD are documented at /erd/.

Further reading:

• Effective Go — The starting point for all Go style
• Standard Go Project Layout — Why we use cmd/, internal/, pkg/

2. Service Structure

Every service follows this layout exactly. Consistency matters — you should be able to navigate any service without re-learning the structure.

services/<name>/
├── cmd/server/main.go          # Startup: config → DB → migrations → deps → serve
├── internal/
│   ├── config/config.go        # Env-var config struct with envOr()
│   ├── handler/
│   │   ├── handler.go          # chi router + middleware chain + Deps struct
│   │   └── claims.go           # Domain-specific handlers (one file per resource)
│   ├── service/
│   │   ├── claim.go            # Business logic, injected repo interfaces
│   │   └── errors.go           # Service-level sentinel errors
│   ├── repository/
│   │   ├── repository.go       # Interfaces + ErrNotFound
│   │   └── gorm_claims.go      # GORM implementations
│   ├── client/                 # HTTP clients to other services
│   ├── kafka/                  # Consumer + event handlers
│   └── outbox/                 # Outbox worker for reliable Kafka publishing
└── migrations/                 # SQL files run by Goose on startup

Shared Packages

3. Startup Sequence

Every service follows this exact sequence. If any step fails, the service exits immediately — no partial startups.

func main() {
    // 1. Load config from env vars
    cfg, err := config.Load()

    // 2. Initialize OpenTelemetry (traces + logs) — returns compound shutdown
    shutdown, err := ollotel.Init(ctx, ollotel.Config{ServiceName: "claims"})
    defer shutdown(ctx)

    // 2a. Wire slog → trace correlation + otelslog bridge fan-out
    logging.SetDefault("claims")
    slog.Info("outbox worker configured",
        "env", cfg.Profile.Env,
        "poll_interval", cfg.Profile.OutboxPollInterval,
        "kafka_batch_timeout", cfg.Profile.KafkaBatchTimeout,
        "kafka_batch_size", cfg.Profile.KafkaBatchSize)

    // 3. Open database connection
    db, err := ollydb.Open(cfg.DatabaseURL)

    // 4. Run Goose migrations
    ollydb.RunMigrations(db, "migrations")

    // 5. Wire repositories (interfaces)
    claimRepo := repository.NewGormClaimRepository(db)

    // 6. Wire HTTP clients for other services
    eligClient := client.NewEligibilityClient(cfg.EligibilityURL)

    // 7. Wire service layer (inject repos + clients)
    claimSvc := service.NewClaimService(service.ClaimServiceDeps{...})

    // 8. Start background workers (outbox, Kafka consumer)
    go outboxWorker.Run(ctx)
    go consumer.Run(ctx)

    // 9. Serve HTTP with graceful shutdown
    srv := &http.Server{Handler: handler.New(handler.Deps{...})}
    // signal.Notify → srv.Shutdown(30s timeout)
}

Further reading:

• How I write HTTP services in Go after 13 years — Mat Ryer's patterns for main() wiring, run() functions, and graceful shutdown

4. Handlers & Routing

Router Setup

func New(deps Deps) http.Handler {
    r := chi.NewRouter()
    r.Use(chiMiddleware.RequestID)
    r.Use(chiMiddleware.Recoverer)

    // Health checks — no auth
    r.Get("/healthz", health.Healthz)
    r.Get("/readyz", health.Readyz(deps.DBPing))

    // Internal routes — service-to-service, no JWT
    r.Group(func(r chi.Router) {
        r.Get("/internal/claims/{locator}", internalGetClaim(...))
    })

    // Protected routes — JWT required
    r.Group(func(r chi.Router) {
        r.Use(auth.JWTMiddleware(auth.Config{JWKSURI: deps.JWKSUri}))
        r.Post("/claims", submitClaim(deps.Claims))
        r.Get("/claims/{locator}", getClaim(deps.Claims))
    })

    return otelhttp.NewHandler(r, "claims-service")
}

Handler Function Pattern

Handlers are closure functions that capture their service dependency and return http.HandlerFunc:

func submitClaim(svc ClaimServiceIface) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // 1. Decode request
        var body SubmitClaimRequest
        if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
            respond(w, 400, map[string]string{"error": "invalid request body"})
            return
        }

        // 2. Call service
        claim, err := svc.SubmitClaim(r.Context(), body)
        if err != nil {
            writeError(w, err) // Maps service errors → HTTP status
            return
        }

        // 3. Respond
        respond(w, 202, claim)
    }
}

Error-to-HTTP Mapping

func writeError(w http.ResponseWriter, err error) {
    var valErr service.ErrValidation
    switch {
    case errors.Is(err, service.ErrNotFound):
        respond(w, 404, map[string]string{"error": err.Error()})
    case errors.As(err, &valErr):
        respond(w, 400, map[string]string{"error": valErr.Error()})
    case errors.Is(err, service.ErrInvalidTransition):
        respond(w, 409, map[string]string{"error": err.Error()})
    default:
        respond(w, 500, map[string]string{"error": "internal server error"})
    }
}

Further reading:

• go-chi/chi documentation — Router, middleware groups, URL params
• Mat Ryer — Closure-based handlers — Why we use func submitClaim(svc) http.HandlerFunc

5. Service Layer

The service layer contains business logic. It receives repository interfaces (not concrete GORM types) and HTTP clients for cross-service calls.

type ClaimService struct {
    db          *gorm.DB
    claims      repository.ClaimRepository      // interface
    lines       repository.ClaimLineRepository   // interface
    outbox      repository.OutboxRepository      // interface
    eligibility *client.EligibilityClient        // HTTP client
    enrollment  *client.EnrollmentClient
    log         *slog.Logger
}

type ClaimServiceDeps struct {
    DB          *gorm.DB
    Claims      repository.ClaimRepository
    Lines       repository.ClaimLineRepository
    Outbox      repository.OutboxRepository
    Eligibility *client.EligibilityClient
    Enrollment  *client.EnrollmentClient
}

func NewClaimService(deps ClaimServiceDeps) *ClaimService {
    return &ClaimService{
        db:          deps.DB,
        claims:      deps.Claims,
        // ... all deps injected
        log:         slog.Default(),
    }
}

6. Repository Layer

Interface Definition

var ErrNotFound = errors.New("not found")

type ClaimRepository interface {
    Create(ctx context.Context, claim *domain.Claim) error
    GetByLocator(ctx context.Context, locator string) (*domain.Claim, error)
    GetByID(ctx context.Context, id uuid.UUID) (*domain.Claim, error)
    UpdateStatus(ctx context.Context, id uuid.UUID, from, to domain.ClaimStatus, event domain.ClaimEvent) error
    ListByPolicy(ctx context.Context, policyID uuid.UUID) ([]domain.Claim, error)
}

GORM Implementation

func (r *GormClaimRepository) GetByLocator(ctx context.Context, locator string) (*domain.Claim, error) {
    var c domain.Claim
    if err := r.db.WithContext(ctx).First(&c, "locator = ?", locator).Error; err != nil {
        if errors.Is(err, gorm.ErrRecordNotFound) {
            return nil, ErrNotFound  // Wrap GORM error as domain error
        }
        return nil, err
    }
    return &c, nil
}

Transactions with Pessimistic Locking

r.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
    var c domain.Claim
    if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).
        First(&c, "id = ?", id).Error; err != nil {
        return err
    }
    // ... mutate and save within the same tx
})

Further reading:

• GORM v2 documentation
• Repository pattern in Go (Three Dots Labs) — Why we use interfaces
• Go Wiki: Interfaces — "Accept interfaces, return structs"

7. Error Handling

The Error Flow

Wrapping Errors

// Good — adds context about what failed
return nil, fmt.Errorf("get policy: %w", err)
return nil, fmt.Errorf("apply accumulators: %w", err)

// Bad — no context
return nil, err

Startup Errors

// Always exit on infrastructure failures at startup
if err != nil {
    slog.Error("failed to open database", "error", err)
    os.Exit(1)
}

Further reading:

• Working with Errors in Go 1.13 — errors.Is(), errors.As(), %w wrapping
• Don't just check errors, handle them gracefully — Dave Cheney

8. Domain Models

All GORM models live in a shared domain package — not in individual services. This is the single source of truth for data structures.

type Claim struct {
    ID              uuid.UUID       `gorm:"type:uuid;primaryKey"`
    Locator         string          `gorm:"uniqueIndex;not null"`
    PolicyID        uuid.UUID       `gorm:"type:uuid;not null;index"`
    Status          ClaimStatus     `gorm:"not null"`
    IncidentDate    time.Time       `gorm:"type:date;not null"`
    Document        []byte          `gorm:"type:jsonb"`
    CreatedAt       time.Time
    UpdatedAt       time.Time
}

func (Claim) TableName() string { return "claims.claims" }

func (c *Claim) BeforeCreate(_ *gorm.DB) error {
    if c.ID == uuid.Nil { c.ID = uuid.New() }
    return nil
}

Conventions

• UUID primary keys — auto-generated in BeforeCreate hook
• Locator fields — human-readable unique IDs (e.g., CLM-2026-000001)
• Schema-prefixed table names — claims.claims, enrollment.policies
• JSONB for flexible data — Document, Data, CoverageTerms stored as []byte
• Pointer types for nullable fields — *string, *time.Time, *uuid.UUID
• decimal.Decimal for money — mapped to numeric(14,4), never float64

9. Configuration

type Config struct {
    Port            string
    DatabaseURL     string
    KeycloakJWKSURI string
    KafkaBrokers    string
    EligibilityURL  string
    // ...
}

func Load() (*Config, error) {
    envfile.Load(envOr("SECRETS_FILE", ""))

    // Validate required vars
    dbURL := os.Getenv("DATABASE_URL")
    if dbURL == "" {
        return nil, fmt.Errorf("config: DATABASE_URL is required")
    }

    return &Config{
        Port:        envOr("PORT", "8080"),
        DatabaseURL: dbURL,
        // ...
    }, nil
}

func envOr(key, fallback string) string {
    if v := os.Getenv(key); v != "" { return v }
    return fallback
}

Environment Profiles

Timing-sensitive knobs (Kafka batching, outbox polling) are bundled into a Profile selected at boot from ENVIRONMENT. Services never tune these individually — they read cfg.Profile and pass the fields through to the Kafka writer and outbox loop. A single env-var override (OUTBOX_POLL_INTERVAL, KAFKA_BATCH_TIMEOUT, KAFKA_BATCH_SIZE) can tune one knob without flipping profiles.

type Profile struct {
    Env                string
    KafkaBatchTimeout  time.Duration
    KafkaBatchSize     int
    OutboxPollInterval time.Duration
}

func ForEnvironment(env string) Profile {
    switch env {
    case "prod":
        return Profile{Env: "prod",
            KafkaBatchTimeout: 1 * time.Second,
            KafkaBatchSize: 100,
            OutboxPollInterval: 500 * time.Millisecond}
    default: // dev / unknown → dev defaults
        return Profile{Env: "dev",
            KafkaBatchTimeout: 10 * time.Millisecond,
            KafkaBatchSize: 1,
            OutboxPollInterval: 50 * time.Millisecond}
    }
}

10. Kafka, Outbox & 2PC Trace Correlation

Services publish events reliably using the transactional outbox pattern: write to the business table and the outbox table in the same DB transaction. A background worker polls the outbox and publishes to Kafka. Every request produces one distributed trace covering HTTP → db.Transaction → outbox.enqueue → async outbox.publish_entry → kafka.write → consumer, with logs correlated in Loki via trace_id.

Outbox Table Schema (uniform across services)

CREATE TABLE <schema>.outbox_entries (
    id             UUID PRIMARY KEY,
    topic          TEXT NOT NULL,
    key            TEXT NOT NULL,
    payload        JSONB NOT NULL,
    trace_context  JSONB,                -- W3C propagation headers
    created_at     TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    published_at   TIMESTAMPTZ
);
CREATE INDEX idx_outbox_unpublished ON <schema>.outbox_entries(created_at)
    WHERE published_at IS NULL;

Phase 1 — Writing to Outbox (service layer, in-tx)

// Same transaction: create claim + enqueue outbox entry.
// runInTx wraps both writes so a crash between them is impossible.
s.db.Transaction(func(tx *gorm.DB) error {
    if err := s.claims.Create(ctx, tx, &claim); err != nil { return err }
    return s.outbox.Enqueue(ctx, tx, repository.OutboxMessage{
        Topic:   "claims.events",
        Key:     claim.Locator,
        Payload: eventJSON,
    })
})

Phase 2 — GormOutboxRepository.Enqueue (explicit span + trace injection)

func (r *GormOutboxRepository) Enqueue(ctx context.Context, tx *gorm.DB, msg OutboxMessage) error {
    ctx, span := tracer.Start(ctx, "outbox.enqueue",
        trace.WithAttributes(
            attribute.String("outbox.topic", msg.Topic),
            attribute.String("outbox.key", msg.Key),
            attribute.String("outbox.service", "claims"),
        ))
    defer span.End()

    // Capture W3C propagation into trace_context JSONB
    carrier := propagation.MapCarrier{}
    otel.GetTextMapPropagator().Inject(ctx, carrier)
    traceCtxJSON, _ := json.Marshal(carrier)

    return tx.Create(&domain.OutboxEntry{
        ID:           uuid.New(),
        Topic:        msg.Topic,
        Key:          msg.Key,
        Payload:      msg.Payload,
        TraceContext: traceCtxJSON,
    }).Error
}

Phase 3 — Outbox Worker (granular spans + cross-process propagation)

// Poll interval comes from cfg.Profile.OutboxPollInterval (50ms dev / 500ms prod)
ticker := time.NewTicker(profile.OutboxPollInterval)
for range ticker.C {
    ctx, tickSpan := tracer.Start(ctx, "outbox.tick")
    entries := fetch(ctx)                   // outbox.fetch child span

    for i, e := range entries {
        // Restore the ORIGINATING request's context from trace_context JSONB
        var carrier propagation.MapCarrier
        json.Unmarshal(e.TraceContext, &carrier)
        parentCtx := otel.GetTextMapPropagator().Extract(ctx, carrier)

        pubCtx, pubSpan := tracer.Start(parentCtx, "outbox.publish_entry",
            trace.WithAttributes(
                attribute.Int64("outbox.age_ms", time.Since(e.CreatedAt).Milliseconds()),
                attribute.Int("outbox.queue_position", i),
                attribute.Int("outbox.queue_total", len(entries)),
                attribute.String("event.type", e.EventType()),
            ))

        // Inject restored context into Kafka headers → consumer continues same trace
        headers := kafkaHeadersFromCtx(pubCtx)
        writer.WriteMessages(pubCtx, kafka.Message{
            Key: []byte(e.Key), Value: e.Payload, Headers: headers,
        }) // wrapped in kafka.write child span

        markPublished(pubCtx, e.ID) // outbox.mark_published child span
        pubSpan.End()
    }
    tickSpan.End()
}

The resulting span tree for every outbox row:

outbox.tick
 ├── outbox.fetch
 └── outbox.publish_entry     ← under originating request's trace_id
      ├── kafka.write
      └── outbox.mark_published

Event Envelope

type EventEnvelope struct {
    ID         string          // UUID
    EventType  string          // "policy.issued", "claim.submitted"
    OccurredAt time.Time
    Payload    json.RawMessage
}

Event Types (Enrollment)

Services currently on the uniform outbox

claims, billing, enrollment, care, consent, document-service. All use GormOutboxRepository with trace_context; all run the granular-span worker under cfg.Profile.

Further reading:

• Transactional Outbox Pattern — Chris Richardson
• segmentio/kafka-go — The Kafka client we use
• W3C Trace Context — The propagation format stored in trace_context

11. Authentication

All service routes except /healthz, /readyz, and /internal/* require a JWT Bearer token validated against Keycloak's JWKS endpoint.

// Middleware caches JWKS keys and validates tokens
r.Group(func(r chi.Router) {
    r.Use(auth.JWTMiddleware(auth.Config{JWKSURI: cfg.KeycloakJWKSURI}))
    // ... protected routes
})

// Extract claims in handlers
claims := auth.ClaimsFromContext(r.Context())
orgLocator := auth.OrgLocatorFromContext(r.Context())  // employer scoping

Key JWT Claims

• party_locator — Member's party identifier (member app)
• org_locator — Employer's scheme identifier (employer app)
• sub — Keycloak user ID

12. Database Migrations

-- +goose Up
CREATE SCHEMA IF NOT EXISTS claims;

-- +goose Up
CREATE TABLE claims.claims (
    id                UUID PRIMARY KEY,
    locator           TEXT NOT NULL,
    policy_id         UUID NOT NULL,
    status            TEXT NOT NULL,
    document          JSONB,
    created_at        TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at        TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    CONSTRAINT claims_locator_key UNIQUE (locator)
);

CREATE INDEX idx_claims_policy ON claims.claims(policy_id);

-- +goose Down
DROP TABLE IF EXISTS claims.claims;

Conventions

• Sequential numbering: 0001_create_schema.sql, 0002_create_claims.sql
• Always include -- +goose Down for rollback
• Schema-prefixed tables: claims.claims, billing.invoices
• Add indexes in the same migration as the table
• Use TIMESTAMPTZ (not TIMESTAMP) for all time columns
• Migrations run automatically on service startup via ollydb.RunMigrations()

Further reading:

• Goose migration tool

13. Testing

Unit Tests

// Standard Go testing with interface-based mocks
func TestGetClaim_NotFound(t *testing.T) {
    mockRepo := &mockClaimRepo{
        getByLocator: func(ctx context.Context, loc string) (*domain.Claim, error) {
            return nil, repository.ErrNotFound
        },
    }
    svc := service.NewClaimService(service.ClaimServiceDeps{Claims: mockRepo})
    // ... test handler returns 404
}

E2E Tests

// E2E tests hit running services over HTTP with real auth tokens
func TestClaimLifecycle(t *testing.T) {
    token, err := setup.FetchToken(cfg, setup.Member)
    if err != nil {
        t.Skip("Keycloak unreachable", err)
    }
    // Submit claim, poll for terminal status...
}

# Run all E2E tests (requires services running)
cd tests/e2e && GOWORK=off go test ./... -v -timeout 90s -count=1

# Run by service
cd tests/e2e && GOWORK=off go test ./claims/... -v -timeout 90s -count=1

Test Conventions

• GOWORK=off is required — prevents workspace interference
• Tests skip gracefully with t.Skip() when services are unreachable
• Auth tokens are cached per persona with sync.Once
• Async operations use polling with deadline (30s timeout)

Further reading:

• Go testing tutorial
• Learn Go with Tests — Test-driven Go development

14. Observability (Traces, Logs, Profiles)

Every request produces one distributed trace and a set of structured logs correlated to that trace by trace_id. The pipeline: service → OTel Collector → Tempo (traces) + Loki (logs) → Grafana, with click-through navigation in both directions.

Structured Logging + Trace Correlation

We use Go's standard log/slog package. A thin slog.Handler wrapper (logging.ContextHandler) reads trace.SpanContextFromContext(ctx) on every record and attaches trace_id / span_id as top-level attributes. A second handler bridges every record to the global otelslog LoggerProvider so it ships via OTLP alongside the stdout JSON.

// main() — installs both the trace-context handler AND the otelslog bridge
logging.SetDefault("claims")

// Hot paths must use *Context variants so trace_id gets attached
slog.InfoContext(ctx, "claim submitted", "locator", claim.Locator, "policyId", claim.PolicyID)
slog.ErrorContext(ctx, "failed to publish event", "error", err)

Tracing

ollotel.Init wires three things and returns a compound shutdown:

• OTLP trace exporter → sdktrace.TracerProvider → global tracer
• OTLP log exporter → sdklog.LoggerProvider → global logger provider (used by the otelslog bridge above)
• W3C trace-context propagator (used by outbox → Kafka header flow)

// chi router wrapped — every HTTP request gets an automatic span
return otelhttp.NewHandler(r, "claims-service")

Log Shipping Pipeline

Two paths, one destination (Loki):

1. Container stdout → Grafana Alloy (auto-discovers every olly-* container via the Docker socket, labels by service name) → Loki on :3101. This is the legacy path and still captures APISIX/infra logs.
2. otelslog bridge → OTel Collector logs pipeline → Loki exporter. This is the new path and is what carries trace_id / span_id as Loki labels via loki.resource.labels / loki.attribute.labels hints. service.name is promoted to the service_name Loki label (underscore — the loki exporter's promotion convention).

Grafana Wiring

• Loki datasource has uid: loki and derivedFields regex "trace_id":"(\w+)" / "span_id":"(\w+)" matching the JSON log format our slog handler produces. Each renders as a clickable link into Tempo.
• Tempo datasource has tracesToLogsV2 with a customQuery filtering by traceID and matching Loki label service_name (not service — without this fix, clicking a span fails to find the logs).

Result: click a log line in Loki → jump to the trace in Tempo; click a span in Tempo → jump to the filtered Loki query.

Profiles (timing knobs)

All timing-sensitive knobs — Kafka batch timeout, batch size, outbox poll interval — are bundled in a Profile selected at boot from the ENVIRONMENT env var. See §9 Configuration. Boot logs emit one slog.Info("outbox worker configured", …) line so operators can confirm the active profile from Loki.

Health Checks

Verifying 2PC trace correlation locally

1. Make a request to a service with an outbox (e.g. POST /claims).
2. In Tempo, search by service and find the request's trace. You should see the span tree HTTP → db.Transaction → outbox.enqueue, then (after OutboxPollInterval + Kafka batch timeout) a later child subtree outbox.tick → outbox.publish_entry → kafka.write → outbox.mark_published — all under the same trace_id.
3. Click any span → Grafana opens Loki filtered to {service_name="claims"} | json | trace_id="...". All logs from the request + the async publish appear together.

15. Common Commands

# Build & test
make build                 # Build all Go services
make test                  # Unit tests across all services
make lint                  # golangci-lint
make fmt                   # gofmt all Go code

# Local stack
make local-up              # Start docker-compose (postgres, keycloak, kafka, etc.)
make local-down            # Stop (preserves volumes)
make local-down-clean      # Stop + wipe volumes
make keycloak-set-passwords # Reset test user passwords to Olly2026

# Run services
make run-all               # Start all Go services in background
make stop-all              # Stop background services

# E2E tests (requires services running)
make test-e2e              # All E2E tests
make test-e2e-claims       # Claims only

# Single service dev
cd services/claims && go build ./...
cd services/claims && go test ./...
cd services/claims && go test ./internal/handler/... -run TestSubmitClaim -v

16. Further Reading

Go Fundamentals

• Effective Go — Style, naming, control structures, concurrency
• Go Code Review Comments — The Go team's review checklist (naming, error strings, receiver types)
• Google Go Style Guide — Google's internal Go conventions (decisions, best practices)
• Go Proverbs — Rob Pike's design principles for Go

Patterns We Use

• How I write HTTP services after 13 years — Mat Ryer, Grafana. Covers our main.go → run() → handler pattern
• Repository Pattern in Go — Three Dots Labs. Why we use interfaces for data access
• Transactional Outbox Pattern — Chris Richardson. Our Kafka reliability strategy
• Don't just check errors, handle them gracefully — Dave Cheney on Go error philosophy
• Go 1.13 Errors — errors.Is(), errors.As(), %w wrapping

Libraries

• go-chi/chi — HTTP router
• GORM v2 — ORM
• Goose — Database migrations
• segmentio/kafka-go — Kafka client
• lestrrat-go/jwx — JWT/JWK handling
• OpenTelemetry Go — Tracing and metrics
• log/slog — Structured logging (stdlib)

Architecture

• Olly Domain Model ERD — Interactive entity relationship diagram
• Standard Go Project Layout — Why cmd/, internal/, pkg/
• The Twelve-Factor App — Config from env vars, port binding, disposability