Operating a Hydra Node
This page aims at helping Hydra users troubleshoot issues when running their own instances of
hydra-node and participate in a Hydra Head.
Following ADR-9 design principles, the
hydra-node provides JSON formatted logs on the
stdout stream, one line per log item. The log items follow a JSON schema. This logging capability is kept voluntarily simple and non configurable in order to ease integration of Hydra logging into more general log analysis infrastructure, whether a custom ELK stack, third-party services, docker sidecars...
There is an unpublished log-filter executable that one can attach to a hydra-node in order to trim down the volume of information in the log stream. This filter provides some filtering features, namely removing transactions bodies and replacing them with transaction ids, but it's not general enough to warrant publication. Similar capabilities can be easily provided with tools like jq.
--monitoring-port PORT argument, the hydra-node executable will expose a Prometheus compatible HTTP
/metrics endpoint on the given port to enable scraping of exposed metrics.
For example, assuming a hydra-node was started with
--monitoring-port 6001, this command
# TYPE hydra_head_confirmed_tx counter
# TYPE hydra_head_events counter
# TYPE hydra_head_requested_tx counter
# TYPE hydra_head_tx_confirmation_time_ms histogram
No Head is observed from the chain
hydra-nodeis connected to a
cardano-nodethat's on the wrong network. Check the
--networkcommand-line argument and the
- Note that the
hydra-nodecannot start if it cannot connect to the
cardano-node, which might require some time as the
cardano-nodeneeds to revalidate its database and possibly even reconstruct its ledger state when it starts and its connections are not open until it's ready. If running as a service or a container, make sure the orchestrator restarts the process when it crashes
- The Scripts transaction identifier is invalid. This transaction id
is available in the
page for the 3 major networks (
hydra-node's Cardano signing key is inconsistent with the Verification key from the
Inittransaction. Check the
--cardano-signing-keyparameter points to the right key, and that peers have the correct
--cardano-verification-keyfor your host.
- The peers' Cardano verification keys are incorrect. This is mirroring the above issue, check parameters on all peers.
Head does not make progress
- Peers are not correctly connected to each others'. Check the
--peerarguments point to the right
host:portfor each peer.
PeerConnectedmessage should be sent to the client (or appears in the logs) and be consistent for all peers involved in a Head.
- The Hydra signing key for our node or the Hydra verification
keys for peers do not match what's expected by each node. Check
AckSnmessages are received by all parties and that the
LogicOutcomelog does not contain any