Skip to main content

29. EventSource & EventSink abstractions

· 4 min read
Elaine Cardenas
Pi Lanningham
Sebastian Nagel

Status

Accepted

Context

  • The Hydra node represents a significant engineering asset, providing layer 1 monitoring, peer to peer consensus, durable persistence, and an isomorphic Cardano ledger. Because of this, it is being eyed as a key building block not just in Hydra based applications, but other protocols as well.

  • Currently the hydra-node uses a very basic persistence mechanism for it's internal HeadState, that is saving StateChanged events to file on disk and reading them back to load and re-aggregate the HeadState upon startup.

    • Some production setups would benefit from storing these events to a service like Amazon Kinesis data stream instead of local files.
  • The hydra-node websocket-based API is the only available event stream right now and might not fit all purposes.

    • See also ADR 3 and 25
    • Internally, this is realized as a single Server handle which can sendOutput :: ServerOutput tx -> m ()
    • These ServerOutputs closely relate to StateChanged events and ClientEffects are yielded by the logic layer often together with the StateChanged. For example:
    onInitialChainAbortTx newChainState committed headId =
    StateChanged HeadAborted{chainState = newChainState}
    <> Effects [ClientEffect $ ServerOutput.HeadIsAborted{headId, utxo = fold committed}]
  • Users of hydra-node are interested to add alternative implementations for storing, loading and consuming events of the Hydra protocol.

Decision

  • We create two new interfaces in the hydra-node architecture:

    • data EventSource e m = EventSource { getEvents :: m [e] }
    • data EventSink e m = EventSink { putEvent :: e -> m () }
  • We realize our current PersistenceIncremental used for persisting StateChanged events is both an EventSource and an EventSink

  • We drop the persistence from the main handle HydraNode tx m, add one EventSource and allow many EventSinks

data HydraNode tx m = HydraNode
{ -- ...
, eventSource :: EventSource (StateEvent tx) m
, eventSinks :: [EventSink (StateEvent tx) m]
}
  • The hydra-node will load events and hydrate its HeadState using getEvents of the single eventSource.

  • The stepHydraNode main loop does call putEvent on all eventSinks in sequence. Any failure will make the hydra-node process terminate and require a restart.

  • When loading events from eventSource on hydra-node startup, it will also re-submit events via putEvent to all eventSinks.

  • The default hydra-node main loop does use the file-based EventSource and a single file-based EventSink (using the same file).

  • We realize that the EventSource and EventSink handles, as well as their aggregation in HydraNode are used as an API by forks of the hydra-node and try to minimize changes to it.

Consequences

  • The default operation of the hyda-node remains unchanged.

  • There are other things called Event and EventQueue(putEvent) right now in the hydra-node. This is getting confusing and when we implement this, we should also rename several things first (tidying).

  • Interface first: Implementations of EventSink should specify their format in a non-ambiguous and versioned way, especially when a corresponding EventSource exists.

  • The API Server can be modelled and refactored as an EventSink.

  • Projects forking the hydra node have dedicated extension points for producing and consuming events.

  • Sundae Labs can build a "Save transaction batches to S3" proof of concept EventSink.

  • Sundae Labs can build a "Scrolls source" EventSink.

  • Sundae Labs can build a "Amazon Kinesis" EventSource and EventSink.

Out of scope / future work

  • Available implementations for EventSource and EventSink could be

    • configured upon hydra-node startup using for example URIs: --event-source file://state or --event-sink s3://some-bucket
    • dynamically loaded as plugins without having to fork hydra-node.
  • The Network and Chain parts qualify as EventSinks as well or shall those be triggered by Effects still?