| Safe Haskell | Safe-Inferred |
|---|---|
| Language | GHC2021 |
Hydra.ModelSpec
Description
Model-Based testing of Hydra Head protocol implementation.
- Troubleshooting
- * Deadlocks
One of the most annoying problems one can face with those very high level properties involving multithreading and a lot of complex moving parts is when the test execution deadlocks. Here is a short guide on what one can do to troubleshoort this kind of issue:
- **Check generators**:
suchThatcombinator from QuickCheck is useful when one wants to refine anotherGenerator's behaviour but it can lead to deadlock if the filtering leads to no value being generated. Avoid it. - **Dump nodes' logs**: In case of a "normal" failure of the tests, the logs from the nodes are dumped. However, if the test does
not even complete then no logs are produced because they are kept in memory. In this case. replacing
traceInIOSimwith `traceInIOSim <> traceDebug` will ensure the logs are dumped on thestderr. It could be a good idea to store them in a file as they can be quite large. - **Use**
traceliberally: Because getting a proper stack trace is hard in Haskell, esp. in pure code, sprinklingtracestatements at key points might help understand what's going on and zoom in on the culprits - **Dump IOSim trace**: In case the deadlock (or race condition) is caused by having two or more concurrent threads competing
to access a resource, dumping the trace of IOSim's runtime scheduleer execution can help. io-sim generate its trace lazily which
means that even when it deadlocks, one can capture at least a significant prefix of the trace and dump it to
stderr. One can `map ( t -> trace (ppEvents t) t) . traceEvents` over theSimTracereturned byrunSimTraceto get some pretty-printed output similar to:
@
Time 380.1s - ThreadId [4] node-94455e3e - EventThrow AsyncCancelled
Time 380.1s - ThreadId [4] node-94455e3e - EventMask MaskedInterruptible
Time 380.1s - ThreadId [4] node-94455e3e - EventMask MaskedInterruptible
Time 380.1s - ThreadId [4] node-94455e3e - EventDeschedule Interruptible
Time 380.1s - ThreadId [4] node-94455e3e - EventTxCommitted [Labelled (TVarId 25) (Just "async-ThreadId [4]")] [] Nothing
Time 380.1s - ThreadId [] main - EventTxWakeup [Labelled (TVarId 25) (Just "async-ThreadId [4]")]
Time 380.1s - ThreadId [4] node-94455e3e - EventUnblocked [ThreadId []]
Time 380.1s - ThreadId [4] node-94455e3e - EventDeschedule Yield
Time 380.1s - ThreadId [] main - EventTxCommitted [] [] Nothing
Time 380.1s - ThreadId [] main - EventUnblocked []
Time 380.1s - ThreadId [] main - EventDeschedule Yield
Time 380.1s - ThreadId [4] node-94455e3e - EventThreadFinished
Time 380.1s - ThreadId [4] node-94455e3e - EventDeschedule Terminated
Time 380.1s - ThreadId [] main - EventThreadFinished
@
- * Recording trace failures
When a property fails it will dump the sequence of actions leading to the failure:
@
do action $ Seed {seedKeys = [("8bbc9f32e4faff669ed1561025f243649f1332902aa79ad7e6e6bbae663f332d",CardanoSigningKey {signingKey = "0400020803030302070808060405040001050408070401040604000005010603"})], seedContestationPeriod = 46s, seedDepositDeadline = 50s}
var2 <- action $ Init (Party {vkey = "b4ea494b4bda6281899727bf4cfef5cdeba8fb3fec4edebc408aa72dfd6ad4f0"})
action $ Deposit {headIdVar = var2, utxoToDeposit = [(CardanoSigningKey {signingKey = "0400020803030302070808060405040001050408070401040604000005010603"},valueFromList [(AdaAssetId,54862683)])], deadline = 1864-06-06 08:24:08.669152896211 UTC}
pure ()
@
Which can be turned into a unit test after resolving most of the imports. Common pitfalls are incorrect show instances (e.g. the UTCTime in deadline above). Should the variables not be bound correctly, double check HasVariables instances. A working example of the above output would be:
@
it "troubleshoot" . withMaxSuccess 1 . flip forAllDL propHydraModel $ do
action $ Seed{seedKeys = [("8bbc9f32e4faff669ed1561025f243649f1332902aa79ad7e6e6bbae663f332d", CardanoSigningKey{signingKey = "0400020803030302070808060405040001050408070401040604000005010603"})], seedContestationPeriod = UnsafeContestationPeriod 46, seedDepositDeadline = UnsafeDepositDeadline 50}
var2 <- action $ Init (Party{vkey = "b4ea494b4bda6281899727bf4cfef5cdeba8fb3fec4edebc408aa72dfd6ad4f0"})
action $ Deposit{headIdVar = var2, utxoToDeposit = [(CardanoSigningKey{signingKey = "0400020803030302070808060405040001050408070401040604000005010603"}, valueFromList [(AdaAssetId, 54862683)])], deadline = read "1864-06-06 08:24:08.669152896211 UTC"}
pure ()
@
Synopsis
- spec :: Spec
- propFanoutLimit :: Int -> Property
- propDL :: DL WorldState () -> Property
- propScripted :: DL WorldState () -> Property
- openHeadWithDepositFuel :: Int -> DL WorldState (Var HeadId, [(Party, UTxOType Payment)])
- twoParties :: [(Party, UTxOType Payment)] -> ((Party, UTxOType Payment), (Party, UTxOType Payment))
- threeParties :: [(Party, UTxOType Payment)] -> ((Party, UTxOType Payment), (Party, UTxOType Payment), (Party, UTxOType Payment))
- decommitFuel :: UTxOType Payment -> Payment
- headStillSettles :: DL WorldState ()
- twoFinalizedIncrementsErased :: DL WorldState ()
- finalizedIncrementErasedWithNextInFlight :: DL WorldState ()
- twoFinalizedDecrementsErased :: DL WorldState ()
- finalizedIncrementAndDecrementErased :: DL WorldState ()
- newSettlementsDuringReplay :: DL WorldState ()
- depositAndIncrementErasedThenRelanded :: DL WorldState ()
- closedHeadWithManyOutputs :: Int -> DL WorldState (UTxOType Payment)
- autoFanoutStepErased :: DL WorldState ()
- autoFanoutTwoStepsErased :: DL WorldState ()
- manualPartialFanout :: DL WorldState ()
- manualFanoutStepErased :: DL WorldState ()
- rePostedIncrementErasedAgain :: DL WorldState ()
- propHydraModel :: Actions WorldState -> Property
- propCheckModelBalances :: Property
- propStressModelBalances :: Property
- concurrentWalk :: DL WorldState ()
- checkModelBalances :: Actions WorldState -> Property
- assertBalancesInOpenHeadAreConsistent :: GlobalState -> UTxOType Payment -> Map Party (TestHydraClient Tx (IOSim s)) -> Party -> PropertyM (RunMonad (IOSim s)) ()
- propIsDistributive :: (Show b, Eq b, Semigroup a, Semigroup b) => (a -> b) -> a -> a -> Property
- partyContestsToWrongClosedSnapshot :: DL WorldState ()
- fanoutContainsWholeConfirmedUTxO :: DL WorldState ()
- settlePending :: DL WorldState ()
- nonConflictingTx :: WorldState -> Quantification (Party, Payment)
- conflictFreeLiveness :: DL WorldState ()
- propDoesNotGenerate0AdaUTxO :: Actions WorldState -> Property
- runIOSimProp :: Testable a => (forall s. PropertyM (RunMonad (IOSim s)) a) -> Property
- runRunMonadIOSimGen :: forall a. Testable a => (forall s. Gen (RunMonad (IOSim s) a)) -> Gen Property
- eventually :: Action WorldState () -> DL WorldState ()
- action_ :: Typeable a => Action WorldState a -> DL WorldState ()
Documentation
propFanoutLimit :: Int -> Property Source #
propDL :: DL WorldState () -> Property Source #
propScripted :: DL WorldState () -> Property Source #
Like propDL for scripted scenarios where only the keys are random.
Shrinking is off: it cannot simplify a fixed script, it only reruns it
hundreds of times with varied values (a failing run takes ~0.3s, a shrunk
one took minutes).
Settlement races under divergent forks
openHeadWithDepositFuel :: Int -> DL WorldState (Var HeadId, [(Party, UTxOType Payment)]) Source #
Open a head of n parties, each owning one UTxO that can be deposited,
and return the head id variable together with each party's deposit fuel.
Funds only enter the head through deposits (it opens empty), so every
scenario starts from here. The Wait leaves room for deep forks that must
stay clear of the head-opening transactions.
twoParties :: [(Party, UTxOType Payment)] -> ((Party, UTxOType Payment), (Party, UTxOType Payment)) Source #
The fuel of a two-party head, see openHeadWithDepositFuel.
threeParties :: [(Party, UTxOType Payment)] -> ((Party, UTxOType Payment), (Party, UTxOType Payment), (Party, UTxOType Payment)) Source #
The fuel of a three-party head, see openHeadWithDepositFuel.
decommitFuel :: UTxOType Payment -> Payment Source #
Payment decommitting a party's deposited fuel back to itself.
headStillSettles :: DL WorldState () Source #
The head must still settle after the fork: confirm an L2 transaction,
close and fan out the whole confirmed UTxO (checked by the Fanout
postcondition). A head wedged on an erased settlement fails here, either
because the close cannot land (its snapshot is ahead of the on-chain
version) or because the fanout does not match.
twoFinalizedIncrementsErased :: DL WorldState () Source #
Scenario 1: two deposits settle back to back, then a fork erases both increments for good (no mempool re-inclusion). Both must be re-posted.
finalizedIncrementErasedWithNextInFlight :: DL WorldState () Source #
Scenario 2: deposit B's transaction is on chain before A's increment lands. Right after A settles, B's snapshot is approved and its increment is in flight; a shallow fork then erases A's increment but keeps B's deposit. Timing dependent by nature; the random walk covers the rest of this space.
twoFinalizedDecrementsErased :: DL WorldState () Source #
Scenario 3: two decommits settle back to back (a second decommit is only accepted once the first is finalized), then a fork erases both decrements.
finalizedIncrementAndDecrementErased :: DL WorldState () Source #
Scenario 4: an increment and then a decrement settle in consecutive versions; a fork erases both. The decrement can only re-land after the increment did.
newSettlementsDuringReplay :: DL WorldState () Source #
While erased settlements are being re-posted, L2 keeps going: a new decommit and a new deposit are requested right after the fork. Their snapshots are signed at the local version, ahead of the chain, so their settlements must queue behind the replayed ones and land in version order.
depositAndIncrementErasedThenRelanded :: DL WorldState () Source #
Scenario 5: a deep fork erases the deposit transaction itself along with its increment; the mempool re-includes the deposit, so the increment must be re-posted once the deposit is observed again.
Partial fanout
closedHeadWithManyOutputs :: Int -> DL WorldState (UTxOType Payment) Source #
Open a single-party head holding n outputs of 1 ADA, each owned by a
different key, and close it. More than fanoutOutputThreshold outputs do
not fit one fanout transaction, so the fanout takes several steps; how many
outputs each step carries is decided by the node from the script budget.
autoFanoutStepErased :: DL WorldState () Source #
The automatic fanout is in progress and a fork erases its latest step. The node must re-post it (the step is expected to be reported a second time) and the head must still be fully fanned out.
autoFanoutTwoStepsErased :: DL WorldState () Source #
Like autoFanoutStepErased but with two partial steps landed, of which
the fork erases the second: the node has to post that step again while its
bookkeeping is already at the final one.
manualPartialFanout :: DL WorldState () Source #
Manual mode: the client hands the node two selections in turn. The second one drains the head, so it ends in the final fanout.
manualFanoutStepErased :: DL WorldState () Source #
Manual mode with a fork erasing the first selection's step before the client hands over the next selection. The node must post the erased step again (the step is expected to be reported a second time) before the head can be drained.
rePostedIncrementErasedAgain :: DL WorldState () Source #
Scenario 6: a fork erases a finalized increment, the re-post lands, and a second fork erases the re-posted increment as well.
propHydraModel :: Actions WorldState -> Property Source #
propCheckModelBalances :: Property Source #
propStressModelBalances :: Property Source #
Same balance consistency assertion as propCheckModelBalances, but over
longer random action sequences: heavier L2 traffic with deposits, decommits,
benign rollbacks and divergent-fork rollbacks interleaved. This is the
property meant to shake out races between settlement, re-posting and
rollbacks: a wedged head surfaces as a waitUntilMatch timeout inside the
failing action, together with the shrunk action sequence and an io-sim
trace to diagnose from.
concurrentWalk :: DL WorldState () Source #
A random walk with WorldState: deposits and decommits may
overlap each other, L2 traffic and forks of every RequeueMode.
checkModelBalances :: Actions WorldState -> Property Source #
assertBalancesInOpenHeadAreConsistent Source #
Arguments
| :: GlobalState | |
| -> UTxOType Payment | Pending (unobserved) commits, see |
| -> Map Party (TestHydraClient Tx (IOSim s)) | |
| -> Party | |
| -> PropertyM (RunMonad (IOSim s)) () |
The node's head UTxO must contain everything the model has as confirmed,
and nothing else except commits still pending in the model: those were
submitted but not observed as finalized (SubmitDeposit), so the node may
or may not have absorbed them yet.
propIsDistributive :: (Show b, Eq b, Semigroup a, Semigroup b) => (a -> b) -> a -> a -> Property Source #
partyContestsToWrongClosedSnapshot :: DL WorldState () Source #
Expect to see contestations when trying to close with an old snapshot
XXX: Since heads open empty (funds only enter via version-bumping
increments), a head with funds always has $sel:onChainVersion:Start > 0 and closing
with the initial snapshot (open version 0) is invalid on-chain — so this
scenario is effectively vacuous under random actions. To stay meaningful it
needs the mock to close with an old confirmed snapshot at the current
version instead of CloseWithInitialSnapshot.
fanoutContainsWholeConfirmedUTxO :: DL WorldState () Source #
Given any random walk of the model, if the Head is open a NewTx getting confirmed must be part of the UTxO after finalization.
settlePending :: DL WorldState () Source #
Observe every settlement the random walk left pending, so that the steps
after it (NewTx, Close, ...) are not blocked by their preconditions.
nonConflictingTx :: WorldState -> Quantification (Party, Payment) Source #
conflictFreeLiveness :: DL WorldState () Source #
propDoesNotGenerate0AdaUTxO :: Actions WorldState -> Property Source #
Utilities
runIOSimProp :: Testable a => (forall s. PropertyM (RunMonad (IOSim s)) a) -> Property Source #
Specialised runner similar to https://hackage.haskell.org/package/QuickCheck-2.14.3/docs/Test-QuickCheck-Monadic.html#v:monadicST.
runRunMonadIOSimGen :: forall a. Testable a => (forall s. Gen (RunMonad (IOSim s) a)) -> Gen Property Source #
Similar to https://hackage.haskell.org/package/QuickCheck-2.14.3/docs/Test-QuickCheck-Monadic.html#v:runSTGen
It returns Property rather than `Gen a`, what allows to enhance the logging
in case of failures.
eventually :: Action WorldState () -> DL WorldState () Source #
action_ :: Typeable a => Action WorldState a -> DL WorldState () Source #
Orphan instances
| HasVariables CardanoSigningKey Source # | |
Methods getAllVariables :: CardanoSigningKey -> Set (Any Var) | |