Saltar a contenido

Target Architecture Specification (v1)

This specification defines the target runtime, concurrency model, and messaging architecture for DKV Pet Flows under the Spring Boot 4 + Apache Pekko stack.

1. System Topology

The system is designed to be decoupled from core servers. Event ingestion is managed asynchronously via HTTP Webhooks and RabbitMQ queues. A local Apache Pekko Actor System orchestrates the flow state machine internally.

Direct synchronous HTTP REST connections are established to dkv-pet-cloud for: - Authorization: Validating client tokens and API keys when webhooks are called. - Queue Recovery: Requesting event playbacks from a specific offset or timestamp during system initialization or recovery phases.

graph TD
    subgraph DKV Core
        Cloud["dkv-pet-cloud"]
        Netcomp["netcomp"]
    end

    subgraph DKV Pet Flows API (Spring Boot 4)
        Webhooks["Webhook Ingestion API (/api/v1/events)"]
        QueueListener["RabbitMQ Event Listener"]
        PekkoBridge["Spring-Pekko Lifecycle Bridge"]
        AuthClient["REST Auth Client"]
        RecoveryClient["REST Queue Recovery Client"]

        subgraph Local Pekko Actor System
            Supervisor["Flow Supervisor Actor"]
            CampaignActor["Campaign Actor (One per Campaign)"]
            FlowActor["User Flow Actor (Stateful)"]
            Dispatcher["Channel Dispatcher Actor"]
        end

        DB[("PostgreSQL (State Store)")]
    end

    subgraph Channels
        APNs["Gorush (Push APNs/FCM)"]
        SMS["SMS Gateway"]
        Email["Email Service (SMTP)"]
    end

    %% Decoupled Ingestion
    Cloud -->|HTTP POST Webhook| Webhooks
    Netcomp -->|AMQP Pub| QueueListener

    %% Synchronous Auth & Recovery
    Webhooks -->|Validate Token| AuthClient
    AuthClient -->|REST GET /auth| Cloud
    PekkoBridge -->|Trigger Event Playback| RecoveryClient
    RecoveryClient -->|REST POST /events/replay| Cloud

    %% In-App Dispatching
    Webhooks -->|Dispatch Command| PekkoBridge
    QueueListener -->|Dispatch Command| PekkoBridge
    PekkoBridge -->|Command| Supervisor

    %% Actor Hierarchy
    Supervisor -->|Creates / Manages| CampaignActor
    CampaignActor -->|Spawns| FlowActor
    FlowActor -->|Send Alert| Dispatcher

    %% Persistence
    FlowActor <-->|JDBC/R2DBC| DB

    %% Outbound
    Dispatcher -->|Push API| APNs
    Dispatcher -->|SMS API| SMS
    Dispatcher -->|SMTP| Email

2. Concurrency Model (Project Loom)

To avoid complex reactive streams (Mutiny or Reactor) while maintaining high throughput, the system combines Project Loom (Virtual Threads) and the Actor Model (Pekko).

  • Edge Layer (Spring MVC + Virtual Threads): Standard thread-per-request model, but execution is delegated to lightweight Virtual Threads. Heavy blocking I/O (handling incoming HTTP webhook requests, database persistence checks, parsing large JSON payloads) is processed without pinning OS platform threads.
  • Orchestration Layer (Pekko Actors): Highly specialized stateful computations, timers, rate limiting, and workflow branching. Actors process messages sequentially inside a dedicated Dispatcher, ensuring lock-free concurrency.
Incoming Request ──> [Spring Web Controller] (Virtual Thread - Blocking I/O OK)
                             │
                             ▼ (Non-blocking Dispatch)
                       [Pekko Actor System] (Sequential processing, Lock-free state)

3. Actor Hierarchy & Protocol

A hierarchical supervisor strategy guarantees high resilience. If a single User Flow Actor fails during execution, the failure is isolated and does not disrupt the entire system.

graph TD
    System["Pekko ActorSystem"]
    Supervisor["/user/flow-supervisor"]
    CampaignA["/user/flow-supervisor/campaign-policy-renewals"]
    CampaignB["/user/flow-supervisor/campaign-welcome-onboarding"]
    UserFlow1["UserFlowActor (user-101)"]
    UserFlow2["UserFlowActor (user-102)"]
    Dispatcher["/user/channel-dispatcher"]

    System --> Supervisor
    Supervisor --> CampaignA
    Supervisor --> CampaignB
    CampaignA --> UserFlow1
    CampaignA --> UserFlow2
    UserFlow1 --> Dispatcher
    UserFlow2 --> Dispatcher

Actor Roles

  1. Flow Supervisor: Lifecycle coordinator. Listens to system startup/shutdown events, parses configuration, and instantiates Campaign Actors.
  2. Campaign Actor: Orchestrator of a specific campaign. Handles batch starts, rate-limiting constraints, and acts as a router/parent to individual User Flow Actors.
  3. User Flow Actor: Stateful executor for a specific user within a campaign. Maintains progress (e.g., "Step 1: Wait 24h", "Step 2: Send Push"). Persists state dynamically.
  4. Channel Dispatcher: Handles integration with external communication systems (Gorush APNs/FCM, SMS gateway, Email). Isolates network I/O timeouts.

4. Messaging Protocol (CBOR Serialization)

To protect the Actor System and allow future cluster extensions safely, all messages (Commands, Events, States) must utilize binary CBOR (Concise Binary Object Representation) serialization instead of standard Java serialization.

Strict CBOR Mapping Rules

  • All message classes must implement a marker interface com.dkv.pet.flows.serialization.CborSerializable.
  • Avoid direct Scala types or Jackson native JsonNode fields inside messages. Use standard immutable Java types (record, List, Map).
  • Serialization config (application.conf):
pekko {
  actor {
    serializers {
      jackson-cbor = "org.apache.pekko.serialization.jackson.JacksonCborSerializer"
    }
    serialization-bindings {
      "com.dkv.pet.flows.serialization.CborSerializable" = jackson-cbor
    }
  }
}

5. Scale & Performance Targets

Handling campaigns with 100,000+ simultaneous actors:

  • Memory Footprint: Active Pekko actors consume ~300 bytes of overhead each. 100k active actors in memory consume less than 40MB RAM.
  • Passivation: Actors that are waiting (e.g., "Wait 3 days before next follow-up") are passivated (state persisted to PostgreSQL, actor stopped) to preserve heap memory. They are re-hydrated dynamically upon receiving a new event or when their timer expires.
  • Database Access: Leverages Spring Data JPA with batch updates and optimized connection pooling (HikariCP).

6. Concurrency Integration & Non-Blocking Persistence

To prevent carrier thread pinning under Project Loom when calling blocking JDBC/JPA libraries, Pekko actors must never execute database I/O directly within their message-handling loop. Instead, database interactions are strictly decoupled.

Asynchronous Write-Behind & Blocking Dispatcher

  1. Dedicated Dispatcher: A custom Pekko dispatcher (db-dispatcher) is configured with a dedicated thread pool for blocking operations.
  2. Persistence Message Protocol: The UserFlowActor does not block. It forwards state snapshots to a thread-safe DatabasePersister actor, which runs exclusively on the db-dispatcher.
  3. Execution Path: 
    [UserFlowActor] ──(Forward State Snapshot)──> [DatabasePersister] 
                                                     │
                                                     ▼ (Runs on db-dispatcher thread)
                                              [Spring Data JPA Write]

7. State Migration (Read-Through Fallback)

To migrate active user flows from the experimental Dapr state store (Redis/PostgreSQL) without breaking active user workflows:

sequenceDiagram
    autonumber
    actor Cloud as dkv-pet-cloud
    participant Controller as Ingestion Controller
    participant Supervisor as Flow Supervisor
    participant FlowActor as User Flow Actor
    database NewDB as New PostgreSQL
    database LegacyDB as Legacy Dapr Store

    Cloud->>Controller: HTTP Webhook Event (user-101)
    Controller->>Supervisor: IngestEventCmd
    Supervisor->>FlowActor: Rehydrate / Start
    FlowActor->>NewDB: Query current state
    NewDB-->>FlowActor: State NOT FOUND (New User/In-Flight Pivot)

    rect rgb(230, 240, 255)
        note right of FlowActor: Read-Through Fallback Triggered
        FlowActor->>LegacyDB: Query legacy Dapr state
        LegacyDB-->>FlowActor: Hydrated State (Step 2: Wait 24h)
        FlowActor->>NewDB: Save hydrated state to new schema
    end

    FlowActor->>FlowActor: Process incoming event with restored state

8. Webhook Ingestion Security (HMAC Validation)

Incoming HTTP Webhooks at /api/v1/events from dkv-pet-cloud must be authenticated using high-speed HMAC-SHA256 signature verification at the edge layer.

  • Header: X-DKV-Signature
  • Mechanism: The Edge Webhook Controller calculates the HMAC of the raw request payload using a shared webhook secret and compares it in constant-time with the header value.
  • Loom Benefit: Payload signature computing and verification are offloaded to Java Virtual Threads, keeping execution non-blocking and highly parallel.

9. Ingestion Backpressure & Fault Tolerance

To handle high traffic surges and downstream system failures, the architecture implements native rate-limiting and isolation boundaries.

A. RabbitMQ Ingestion Backpressure

To prevent memory exhaustion during event storms: - The RabbitMQ consumer is configured with a strict prefetchCount = 200. - Consumers pull events from queues only when local actors have finished processing pending command queues, creating a natural backpressure boundary.

B. Outbound Channel Fault Tolerance (Gorush / SMS Gateway)

The ChannelDispatcher actor utilizes a BackoffSupervisor pattern: - Exponential Backoff: If an outbound dispatch fails due to external API timeouts or 5xx errors, the actor restarts with exponential backoff (minBackoff = 1s, maxBackoff = 30s, randomFactor = 0.2). - Dead Letter Queue (DLQ): After 5 unsuccessful delivery attempts, messages are directed to a Dead Letter Queue for auditing and manual playback.

10. Deployment Canary & Graceful Shutdown

To release new versions of the stateful API without losing active campaigns or causing duplicate actions:

  1. Graceful Passivation: Upon receiving SIGTERM, Spring's lifecycle hooks trigger a coordinated shutdown sequence.
  2. Coordinated Shutdown Phase:
  3. Stop RabbitMQ consumers immediately to halt new ingestion.
  4. Deny new incoming webhook requests (return 503 Service Unavailable with active retry hints).
  5. Allow active actor messages in-flight to finish.
  6. Force passivation of all memory-resident UserFlowActor instances, flushing their state snapshots to PostgreSQL.
  7. Wait for outbound ChannelDispatcher buffers to empty.
  8. Talos Kubernetes Integration: The pod definition configures terminationGracePeriodSeconds = 60 to ensure the coordinated shutdown completes fully.