The coordinator already had one of these, albeit implemented much worse than
the one now properly introduced. It had to either be sending or receiving,
whereas the new one can do both at the same time.
This replaces said instance and enables pleasant patterns when implementing the
processor/coordinator.
* Add a function to deterministically decide which Serai blocks should be co-signed
Has a 5 minute latency between co-signs, also used as the maximal latency
before a co-sign is started.
* Get all active tributaries we're in at a specific block
* Add and route CosignSubstrateBlock, a new provided TX
* Split queued cosigns per network
* Rename BatchSignId to SubstrateSignId
* Add SubstrateSignableId, a meta-type for either Batch or Block, and modularize around it
* Handle the CosignSubstrateBlock provided TX
* Revert substrate_signer.rs to develop (and patch to still work)
Due to SubstrateSigner moving when the prior multisig closes, yet cosigning
occurring with the most recent key, a single SubstrateSigner can be reused.
We could manage multiple SubstrateSigners, yet considering the much lower
specifications for cosigning, I'd rather treat it distinctly.
* Route cosigning through the processor
* Add note to rename SubstrateSigner post-PR
I don't want to do so now in order to preserve the diff's clarity.
* Implement cosign evaluation into the coordinator
* Get tests to compile
* Bug fixes, mark blocks without cosigners available as cosigned
* Correct the ID Batch preprocesses are saved under, add log statements
* Create a dedicated function to handle cosigns
* Correct the flow around Batch verification/queueing
Verifying `Batch`s could stall when a `Batch` was signed before its
predecessors/before the block it's contained in was cosigned (the latter being
inevitable as we can't sign a block containing a signed batch before signing
the batch).
Now, Batch verification happens on a distinct async task in order to not block
the handling of processor messages. This task is the sole caller of verify in
order to ensure last_verified_batch isn't unexpectedly mutated.
When the processor message handler needs to access it, or needs to queue a
Batch, it associates the DB TXN with a lock preventing the other task from
doing so.
This lock, as currently implemented, is a poor and inefficient design. It
should be modified to the pattern used for cosign management. Additionally, a
new primitive of a DB-backed channel may be immensely valuable.
Fixes a standing potential deadlock and a deadlock introduced with the
cosigning protocol.
* Working full-stack tests
After the last commit, this only required extending a timeout.
* Replace "co-sign" with "cosign" to make finding text easier
* Update the coordinator tests to support cosigning
* Inline prior_batch calculation to prevent panic on rotation
Noticed when doing a final review of the branch.
Closes https://github.com/serai-dex/serai/issues/342.
Under ideal network conditions, this is fine. While I won't claim ideal network
conditions will occur IRL, b0fcdd3367 has the
Tributary rebroadcast messages and should brute-force its way into a
functioning system.
* chore: convert nonce_deicer to use create_db macro
* Restore pub NonceDecider
* Remove extraneous comma
I forgot to run git commit --amend on the prior commit :/
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* Have processor report errors during the DKG to the coordinator
* Add RemoveParticipant, InvalidDkgShare to coordinator
* Route DKG blame around coordinator
* Allow public construction of AdditionalBlameMachine
Necessary for upcoming work on handling DKG blame in the processor and
coordinator.
Additionally fixes a publicly reachable panic when commitments parsed with one
ThresholdParams are used in a machine using another set of ThresholdParams.
Renames InvalidProofOfKnowledge to InvalidCommitments.
* Remove unused error from dleq
* Implement support for VerifyBlame in the processor
* Have coordinator send the processor share message relevant to Blame
* Remove desync between processors reporting InvalidShare and ones reporting GeneratedKeyPair
* Route blame on sign between processor and coordinator
Doesn't yet act on it in coordinator.
* Move txn usage as needed for stable Rust to build
* Correct InvalidDkgShare serialization
* add reasons to slash evidence
* fix CI failing
* Remove unnecessary clones
.encode() takes &self
* InvalidVr to InvalidValidRound
* Unrelated to this PR: Clarify reasoning/potentials behind dropping evidence
* Clarify prevotes in SlashEvidence test
* Replace use of read_to_end
* Restore decode_signed_message
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* Update the coordinator to give key shares based on weight, not based on existence
Participants are now identified by their starting index. While this compiles,
the following is unimplemented:
1) A conversion for DKG `i` values. It assumes the threshold `i` values used
will be identical for the MuSig signature used to confirm the DKG.
2) Expansion from compressed values to full values before forwarding to the
processor.
* Add a fn to the DkgConfirmer to convert `i` values as needed
Also removes TODOs regarding Serai ensuring validator key uniqueness +
validity. The current infra achieves both.
* Have the Tributary DB track participation by shares, not by count
* Prevent a node from obtaining 34% of the maximum amount of key shares
This is actually mainly intended to set a bound on message sizes in the
coordinator. Message sizes are amplified by the amount of key shares held, so
setting an upper bound on said amount lets it determine constants. While that
upper bound could be 150, that'd be unreasonable and increase the potential for
DoS attacks.
* Correct the mechanism to detect if sufficient accumulation has occured
It used to check if the latest accumulation hit the required threshold. Now,
accumulations may jump past the required threshold. The required mechanism is
to check the threshold wasn't prior met and is now met.
* Finish updating the coordinator to handle a multiple key share per validator environment
* Adjust stategy re: preventing noce reuse in DKG Confirmer
* Add TODOs regarding dropped transactions, add possible TODO fix
* Update tests/coordinator
This doesn't add new multi-key-share tests, it solely updates the existing
single key-share tests to compile and run, with the necessary fixes to the
coordinator.
* Update processor key_gen to handle generating multiple key shares at once
* Update SubstrateSigner
* Update signer, clippy
* Update processor tests
* Update processor docker tests
If a crate has std set, it should enable std for all dependencies in order to
let them properly select which algorithms to use. Some crates fallback to
slower/worse algorithms on no-std.
Also more aggressively sets default-features = false leading to a *10%*
reduction in the amount of crates coordinator builds.
* initial implementation
* add function to get a balance of an account
* add support for multiple coins
* rename pallet to "coins-pallet"
* replace balances, assets and tokens pallet with coins pallet in runtime
* add total supply info
* update client side for new Coins pallet
* handle fees
* bug fixes
* Update FeeAccount test
* Fmt
* fix pr comments
* remove extraneous Imbalance type
* Minor tweaks
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
Replaces plan IDs with key + ID, letting the coordinator determine the sessions
for the plans.
Properly scopes which plan IDs are set on which tributaries, and ensures we
have the necessary tributaries at time of handling.
Adds Event::SetRetired to validator-sets.
Emit TributaryRetired.
Replaces is_active_set, which made multiple network requests, with
is_retired_tributary, a DB read.
Performs most of the removals necessary upon TributaryRetired.
Still needs to clean up the actual Tributary/Tendermint tasks.
The tests have recently had their timing stilted, causing failures. The tests
are... fine. They're fragile, as obvious, yet they're logical. The simplest fix
is to unstilt their timing rather to make them non-fragile.
The recent change, which presumably caused said stilting, was the the
rebroadcasting added. This de-duplication prevents most of the impact of
rebroadcasting. While there's still the async task, and the lock acquisition on
attempt to rebroadcast, this hopefully is enough.
* fix typos
* remove tributary sleeping
* handle not locally provided txs
* use topic number instead of waiting list
* Clean-up, fixes
1) Uses a single TXN in provided
2) Doesn't continue on non-local provided inside verify_block, skipping further
execution of checks
3) Upon local provision of already on-chain TX, compares
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* Revert "Correct the prior documented TOCTOU"
This reverts commit d50fe87801.
* Correct the prior documented TOCTOU
d50fe87801 edited the challenge for the Batch to
fix it. This won't produce Batch n+1 until Batch n is successfully published
and verified. It's an alternative strategy able to be reviewed, with a much
smaller impact to scope.
Now, if a malicious validator set publishes a malicious `Batch` at the last
moment, it'll cause all future `Batch`s signed by the next validator set to
require a bool being set (yet they never will set it).
This will prevent the handover.
The only overhead is having two distinct `batch_message` calls on-chain.
The new set publishing a `Batch` completes the handover protocol. The new set
should only publish a `Batch` once it believes the old set has completed all of
its on-external-chain activity, marking it honest and finite.
With the handover comes the acceptance of liability, hence the requirement for
all of the on-Serai-chain activity also needing verification. While most
activity would be verified in-real-time (upon ::Batch messages), the new set
will now explicitly verify the complete set of `Batch`s before beginning its
preprocess for its own `Batch` (the one accepting the handover).
* initial staking pallet
* add staking pallet to runtime
* support session rotation for serai
* optimizations & cleaning
* fix deny
* add serai network to initial networks
* a few tweaks & comments
* fix some pr comments
* Rewrite validator-sets with logarithmic algorithms
Uses the fact the underlying DB is sorted to achieve sorting of potential
validators by stake.
Removes release of deallocated stake for now.
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
Renames Update to SignedBatch.
Checks Batch equality via a hash of the InInstructions. That prevents needing
to keep the Batch in node state or TX introspect.
Prior, we only supported a single Tributary per network, and spawned a task to
handled Processor messages per Tributary. Now, we handle Processor messages per
network, yet we still only supported a single Tributary in that handling
function.
Now, when we handle a message, we load the Tributary which is relevant. Once we
know it, we ensure we have it (preventing race conditions), and then proceed.
We do need work to check if we should have a Tributary, or if we're not
participating. We also need to check if a Tributary has been retired, meaning
we shouldn't handle any transactions related to them, and to clean up retired
Tributaries.
This is a possibility under the new deterministic nonce scheme.
While there is a concern of us never creating a transaction with a nonce,
blocking everything, we should always create transactions. We'll always publish
preprocesses, and while we'll only publish shares if everyone else does, we
only allocate for shares once everyone else does.
This isn't an unacceptable timeout. It matches a prior timeout. I'm unsure why
it's now needed to be extended though. My best guess is the test runtime is
single threaded and there's now new overhead in the task management (or perhaps
higher latency now that messages per-tributary is serialized).
* Design and document a multisig rotation flow
* Make Scanner::eventualities a HashMap so it's per-key
* Don't drop eventualities, always follow through on them
Technical improvements made along the way.
* Start creating an isolate object to manage multisigs, which doesn't require being a signer
Removes key from SubstrateBlock.
* Move Scanner/Scheduler under multisigs
* Move Batch construction into MultisigManager
* Clarify "should" in Multisig Rotation docs
* Add block_number to MultisigManager, as it controls the scanner
* Move sign_plans into MultisigManager
Removes ThresholdKeys from prepare_send.
* Make SubstrateMutable an alias for MultisigManager
* Rewrite Multisig Rotation
The prior scheme had an exploit possible where funds were sent to the old
multisig, then burnt on Serai to send from the new multisig, locking liquidity
for 6 hours. While a fee could be applied to stragglers, to make this attack
unprofitable, the newly described scheme avoids all this.
* Add mini
mini is a miniature version of Serai, emphasizing Serai's nature as a
collection of independent clocks. The intended use is to identify race
conditions and prove protocols are comprehensive regarding when certain clocks
tick.
This uses loom, a prior candidate for evaluating the processor/coordinator as
free of race conditions (#361).
* Use mini to prove a race condition in the current multisig rotation docs, and prove safety of alternatives
Technically, the prior commit had mini prove the race condition.
The docs currently say the activation block of the new multisig is the block
after the next Batch's. If the two next Batches had already entered the
mempool, prior to set_keys being called, the second next Batch would be
expected to contain the new key's data yet fail to as the key wasn't public
when the Batch was actually created.
The naive solution is to create a Batch, publish it, wait until it's included,
and only then scan the next block. This sets a bound of
`Batch publication time < block time`. Optimistically, we can publish a Batch
in 24s while our shortest block time is 2m. Accordingly, we should be fine with
the naive solution which doesn't take advantage of throughput. #333 may
significantly change latency however and require an algorithm whose throughput
exceeds the rate of blocks created.
In order to re-introduce parallelization, enabling throughput, we need to
define a safe range of blocks to scan without Serai ordering the first one.
mini demonstrates safety of scanning n blocks Serai hasn't acknowledged, so
long as the first is scanned before block n+1 is (shifting the n-block window).
The docs will be updated next, to reflect this.
* Fix Multisig Rotation
I believe this is finally good enough to be final.
1) Fixes the race condition present in the prior document, as demonstrated by
mini.
`Batch`s for block `n` and `n+1`, may have been in the mempool when a
multisig's activation block was set to `n`. This would cause a potentially
distinct `Batch` for `n+1`, despite `n+1` already having a signed `Batch`.
2) Tightens when UIs should use the new multisig to prevent eclipse attacks,
and protection against `Batch` publication delays.
3) Removes liquidity fragmentation by tightening flow/handling of latency.
4) Several clarifications and documentation of reasoning.
5) Correction of "prior multisig" to "all prior multisigs" regarding historical
verification, with explanation why.
* Clarify terminology in mini
Synchronizes it from my original thoughts on potential schema to the design
actually created.
* Remove most of processor's README for a reference to docs/processor
This does drop some misc commentary, though none too beneficial. The section on
scanning, deemed sufficiently beneficial, has been moved to a document and
expanded on.
* Update scanner TODOs in line with new docs
* Correct documentation on Bitcoin::Block::time, and Block::time
* Make the scanner in MultisigManager no longer public
* Always send ConfirmKeyPair, regardless of if in-set
* Cargo.lock changes from a prior commit
* Add a policy document on defining a Canonical Chain
I accidentally committed a version of this with a few headers earlier, and this
is a proper version.
* Competent MultisigManager::new
* Update processor's comments
* Add mini to copied files
* Re-organize Scanner per multisig rotation document
* Add RUST_LOG trace targets to e2e tests
* Have the scanner wait once it gets too far ahead
Also bug fixes.
* Add activation blocks to the scanner
* Split received outputs into existing/new in MultisigManager
* Select the proper scheduler
* Schedule multisig activation as detailed in documentation
* Have the Coordinator assert if multiple `Batch`s occur within a block
While the processor used to have ack_up_to_block, enabling skips in the block
acked, support for this was removed while reworking it for multiple multisigs.
It should happen extremely infrequently.
While it would still be beneficial to have, if multiple `Batch`s could occur
within a block (with the complexity here not being worth adding that ban as a
policy), multiple `Batch`s were blocked for DoS reasons.
* Schedule payments to the proper multisig
* Correct >= to <
* Use the new multisig's key for change on schedule
* Don't report External TXs to prior multisig once deprecated
* Forward from the old multisig to the new one at all opportunities
* Move unfulfilled payments in queue from prior to new multisig
* Create MultisigsDb, splitting it out of MainDb
Drops the call to finish_signing from the Signer. While this will cause endless
re-attempts, the Signer will still consider them completed and drop them,
making this an O(n) cost at boot even if we did nothing from here.
The MultisigManager should call finish_signing once the Scanner completes the
Eventuality.
* Don't check Scanner-emitted completions, trust they are completions
Prevents needing to use async code to mark the completion and creates a
fault-free model. The current model, on fault, would cause a lack of marked
completion in the signer.
* Fix a possible panic in the processor
A shorter-chain reorg could cause this assert to trip. It's fixed by
de-duplicating the data, as the assertion checked consistency. Without the
potential for inconsistency, it's unnecessary.
* Document why an existing TODO isn't valid
* Change when we drop payments for being to the change address
The earlier timing prevents creating Plans solely to the branch address,
causing the payments to be dropped, and the TX to become an effective
aggregation TX.
* Extensively document solutions to Eventualities being potentially created after having already scanned their resolutions
* When closing, drop External/Branch outputs which don't cause progress
* Properly decide if Change outputs should be forward or not when closing
This completes all code needed to make the old multisig have a finite lifetime.
* Commentary on forwarding schemes
* Provide a 1 block window, with liquidity fragmentation risks, due to latency
On Bitcoin, this will be 10 minutes for the relevant Batch to be confirmed. On
Monero, 2 minutes. On Ethereum, ~6 minutes.
Also updates the Multisig Rotation document with the new forwarding plan.
* Implement transaction forwarding from old multisig to new multisig
Identifies a fault where Branch outputs which shouldn't be dropped may be, if
another output fulfills their next step. Locking Branch fulfillment down to
only Branch outputs is not done in this commit, but will be in the next.
* Only let Branch outputs fulfill branches
* Update TODOs
* Move the location of handling signer events to avoid a race condition
* Avoid a deadlock by using a RwLock on a single txn instead of two txns
* Move Batch ID out of the Scanner
* Increase from one block of latency on new keys activation to two
For Monero, this offered just two minutes when our latency to publish a Batch
is around a minute already. This does increase the time our liquidity can be
fragmented by up to 20 minutes (Bitcoin), yet it's a stupid attack only
possible once a week (when we rotate). Prioritizing normal users' transactions
not being subject to forwarding is more important here.
Ideally, we'd not do +2 blocks yet plus `time`, such as +10 minutes, making
this agnostic of the underlying network's block scheduling. This is a
complexity not worth it.
* Split MultisigManager::substrate_block into multiple functions
* Further tweaks to substrate_block
* Acquire a lock on all Scanner operations after calling ack_block
Gives time to call register_eventuality and initiate signing.
* Merge sign_plans into substrate_block
Also ensure the Scanner's lock isn't prematurely released.
* Use a HashMap to pass to-be-forwarded instructions, not the DB
* Successfully determine in ClosingExisting
* Move from 2 blocks of latency when rotating to 10 minutes
Superior as noted in 6d07af92ce10cfd74c17eb3400368b0150eb36d7, now trivial to
implement thanks to prior commit.
* Add note justifying measuring time in blocks when rotating
* Implement delaying of outputs received early to the new multisig per specification
* Documentation on why Branch outputs don't have the race condition concerns Change do
Also ensures 6 hours is at least N::CONFIRMATIONS, for sanity purposes.
* Remove TODO re: sanity checking Eventualities
We sanity check the Plan the Eventuality is derived from, and the Eventuality
is handled moments later (in the same file, with a clear call path). There's no
reason to add such APIs to Eventualities for a sanity check given that.
* Add TODO(now) for TODOs which must be done in this branch
Also deprecates a pair of TODOs to TODO2, and accepts the flow of the Signer
having the Eventuality.
* Correct errors in potential/future flow descriptions
* Accept having a single Plan Vec
Per the following code consuming it, there's no benefit to bifurcating it by
key.
* Only issue sign_transaction on boot for the proper signer
* Only set keys when participating in their construction
* Misc progress
Only send SubstrateBlockAck when we have a signer, as it's only used to tell
the Tributary of what Plans are being signed in response to this block.
Only immediately sets substrate_signer if session is 0.
On boot, doesn't panic if we don't have an active key (as we wouldn't if only
joining the next multisig). Continues.
* Correctly detect and set retirement block
Modifies the retirement block from first block meeting requirements to block
CONFIRMATIONS after.
Adds an ack flow to the Scanner's Confirmed event and Block event to accomplish
this, which may deadlock at this time (will be fixed shortly).
Removes an invalid await (after a point declared unsafe to use await) from
MultisigsManager::next_event.
* Remove deadlock in multisig_completed and document alternative
The alternative is simpler, albeit less efficient. There's no reason to adopt
it now, yet perhaps if it benefits modeling?
* Handle the final step of retirement, dropping the old key and setting new to existing
* Remove TODO about emitting a Block on every step
If we emit on NewAsChange, we lose the purpose of the NewAsChange period.
The only concern is if we reach ClosingExisting, and nothing has happened, then
all coins will still be in the old multisig until something finally does. This
isn't a problem worth solving, as it's latency under exceptional dead time.
* Add TODO about potentially not emitting a Block event for the reitrement block
* Restore accidentally deleted CI file
* Pair of slight tweaks
* Add missing if statement
* Disable an assertion when testing
One of the test flows currently abuses the Scanner in a way triggering it.
It was a piece of duplicated data used to achieve context-less
de)serialization. This new Vec code is a bit tricker to first read, yet overall
clean and removes a potential fault.
Saves 2 bytes from DkgShares messages.
See prior commit message for more info.
With the plan for the batch sign ID to be just 5 bytes (potentially 4), this
does incur a +5 bytes cost compared to the ExternalBlock system *even in the
standard case*. The simplicity remains preferred at this time.
The initial TODO was simply to use one ExternalBlock per all batches in the
block. This would require publishing ExternalBlock after the last batch,
requiring knowing the last batch. While we could add such a pipeline, it'd
require:
1) Initial preprocesses using a distinct message from BatchPreprocess
2) An additional message sent after all BatchPreprocess are sent
Unfortunately, both would require tweaks to the SubstrateSigner which aren't
worth the complexity compared to the solution here, at least, not at this time.
While this will cause, if a Tributary is signing a block whose total batch data
exceeds 25 kB, to use multiple transactions which could be optimized out by
'better' local data pipelining, that's an extreme edge case. Given the temporal
nature of each Tributary, it's also an acceptable edge.
This does no longer achieve synchrony over external blocks accordingly. While
signed batches have synchrony, as they embed their block hash, batches being
signed don't have cryptographic synchrony on their contents. This means
validators who are eclipsed may produce invalid shares, as they sign a
different batch. This will be introduced in a follow-up commit.
Updates Tributary values to allow 999ms for block processing (from 2s) and
1667ms for latency (up from 1s).
The intent is to resolve#365. I don't know if this will, but it increases the
chances of success and these values should be fine in prod since Tributary is a
post-execution chain (making block procesisng time minimal).
Does embed the dagger of N::block_time() panicking if the block time in ms
doesn't cleanly divide by 1000.
Fixes where ram_scanned is updated in processor. The prior version, while safe,
would redo massive amounts of work during periods of inactivity. It also hit an
undocumented invariant where get_eventuality_completions assumes new blocks,
yet redone work wouldn't have new blocks.
Modifies Monero's generate_blocks to return the hashes of the generated blocks.
We only expect processor messages when we have the relevant Tributary. We
queued Tributary creation, yet then kicked off processor messages. We need to
wait until the Tributary is actually created to kick off processor messages.
Prior to the previous commit, whatever async scheduling occurred caused them to
all have the same tip. Now, some are one block ahead of others. This adds
tolerance for that, as it's an acceptable variance, so long as it's solely one
block.
They used &mut self to prevent execution at the same time. This uses a lock
over the channel to achieve the same security, without requiring a lock over
the entire tributary.
This fixes post-provided Provided transactions. sync_block waited for the TX to
be provided, yet it never would as sync_block held a mutable reference over the
entire Tributary, preventing any other read/write operations of any scope.
A timeout increased (bc2f23f72b) due to this bug
not being identified has been decreased back, thankfully.
Also shims in basic support for Completed, which was the WIP before this bug
was identified.
The Heartbeat was meant to serve for this, yet no Heartbeats are fired when we
don't have active tributaries.
libp2p does offer an explicit KeepAlive protocol, yet it's not recommended in
prod. While this likely has the same pit falls as LibP2p's KeepAlive protocol,
it's at least tailored to our timing.
* dalek 4.0
* cargo update
Moves to a version of Substrate which uses curve25519-dalek 4.0 (not a rc).
Doesn't yet update the repo to curve25519-dalek 4.0 (as a branch does) due
to the official schnorrkel using a conflicting curve25519-dalek. This would
prevent installation of frost-schnorrkel without a patch.
* use half-aggregation for tm messages
* fmt
* fix pr comments
* cargo update
Achieves three notable updates.
1) Resolves RUSTSEC-2022-0093 by updating libp2p-identity.
2) Removes 3 old rand crates via updating ed25519-dalek (a dependency of
libp2p-identity).
3) Sets serde_derive to 1.0.171 via updating to time 0.3.26 which pins at up to
1.0.171.
The last one is the most important. The former two are niceties.
serde_derive, since 1.0.171, ships a non-reproducible binary blob in what's a
complete compromise of supply chain security. This is done in order to reduce
compile times, yet also for the maintainer of serde (dtolnay) to leverage
serde's position as the 8th most downloaded crate to attempt to force changes
to the Rust build pipeline.
While dtolnay's contributions to Rust are respectable, being behind syn, quote,
and proc-macro2 (the top three crates by downloads), along with thiserror,
anyhow, async-trait, and more (I believe also being part of the Rust project),
they have unfortunately decided to refuse to listen to the community on this
issue (or even engage with counter-commentary). Given their political agenda
they seem to try to be accomplishing with force, I'd go as far as to call their
actions terroristic (as they're using the threat of the binary blob as
justification for cargo to ship 'proper' support for binary blobs).
This is arguably representative of dtolnay's past work on watt. watt was a wasm
interpreter to execute a pre-compiled proc macro. This would save the compile
time of proc macros, yet sandbox it so a full binary did not have to be run.
Unfortunately, watt (while decreasing compile times) fails to be a valid
solution to supply chain security (without massive ecosystem changes). It never
implemented reproducible builds for its wasm blobs, and a malicious wasm blob
could still fundamentally compromise a project. The only solution for an end
user to achieve a secure pipeline would be to locally build the project,
verifying the blob aligns, yet doing so would negate all advantages of the
blob.
dtolnay also seems to be giving up their role as a FOSS maintainer given that
serde no longer works in several environments. While FOSS maintainers are not
required to never implement breaking changes, the version number is still 1.0.
While FOSS maintainers are not required to follow semver, releasing a very
notable breaking change *without a new version number* in an ecosystem which
*does follow semver*, then refusing to acknowledge bugs as bugs with their work
does meet my personal definition of "not actively maintaining their existing
work". Maintenance would be to fix bugs, not introduce and ignore.
For now, serde > 1.0.171 has been banned. In the future, we may host a fork
without the blobs (yet with the patches). It may be necessary to ban all of
dtolnay's maintained crates, if they continue to force their agenda as such,
yet I hope this may be resolved within the next week or so.
Sources:
https://github.com/serde-rs/serde/issues/2538 - Binary blob discussion
This includes several reports of various workflows being broken.
https://github.com/serde-rs/serde/issues/2538#issuecomment-1682519944
dtolnay commenting that security should be resolved via Rust toolchain edits,
not via their own work being secure. This is why I say they're trying to
leverage serde in a political game.
https://github.com/serde-rs/serde/issues/2526 - Usage via git broken
dtolnay explicitly asks the submitting user if they'd be willing to advocate
for changes to Rust rather than actually fix the issue they created. This is
further political arm wrestling.
https://github.com/serde-rs/serde/issues/2530 - Usage via Bazel broken
https://github.com/serde-rs/serde/issues/2575 - Unverifiable binary blob
https://github.com/dtolnay/watt - dtolnay's prior work on precompilation
* add Rs() api to SchnorrAggregate
* Correct serai-processor-tests to dalek 4
* fmt + deny
* Slash malevolent validators (#294)
* add slash tx
* ignore unsigned tx replays
* verify that provided evidence is valid
* fix clippy + fmt
* move application tx handling to another module
* partially handle the tendermint txs
* fix pr comments
* support unsigned app txs
* add slash target to the votes
* enforce provided, unsigned, signed tx ordering within a block
* bug fixes
* add unit test for tendermint txs
* bug fixes
* update tests for tendermint txs
* add tx ordering test
* tidy up tx ordering test
* cargo +nightly fmt
* Misc fixes from rebasing
* Finish resolving clippy
* Remove sha3 from tendermint-machine
* Resolve a DoS in SlashEvidence's read
Also moves Evidence from Vec<Message> to (Message, Option<Message>). That
should meet all requirements while being a bit safer.
* Make lazy_static a dev-depend for tributary
* Various small tweaks
One use of sort was inefficient, sorting unsigned || signed when unsigned was
already properly sorted. Given how the unsigned TXs were given a nonce of 0, an
unstable sort may swap places with an unsigned TX and a signed TX with a nonce
of 0 (leading to a faulty block).
The extra protection added here sorts signed, then concats.
* Fix Tributary tests I broke, start review on tendermint/tx.rs
* Finish reviewing everything outside tests and empty_signature
* Remove empty_signature
empty_signature led to corrupted local state histories. Unfortunately, the API
is only sane with a signature.
We now use the actual signature, which risks creating a signature over a
malicious message if we have ever have an invariant producing malicious
messages. Prior, we only signed the message after the local machine confirmed
it was okay per the local view of consensus.
This is tolerated/preferred over a corrupt state history since production of
such messages is already an invariant. TODOs are added to make handling of this
theoretical invariant further robust.
* Remove async_sequential for tokio::test
There was no competition for resources forcing them to be run sequentially.
* Modify block order test to be statistically significant without multiple runs
* Clean tests
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* Add DSTs to Tributary TX sig_hash functions
Prevents conflicts with other systems/other parts of the Tributary.
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* add slash tx
* ignore unsigned tx replays
* verify that provided evidence is valid
* fix clippy + fmt
* move application tx handling to another module
* partially handle the tendermint txs
* fix pr comments
* support unsigned app txs
* add slash target to the votes
* enforce provided, unsigned, signed tx ordering within a block
* bug fixes
* add unit test for tendermint txs
* bug fixes
* update tests for tendermint txs
* add tx ordering test
* tidy up tx ordering test
* cargo +nightly fmt
* Misc fixes from rebasing
* Finish resolving clippy
* Remove sha3 from tendermint-machine
* Resolve a DoS in SlashEvidence's read
Also moves Evidence from Vec<Message> to (Message, Option<Message>). That
should meet all requirements while being a bit safer.
* Make lazy_static a dev-depend for tributary
* Various small tweaks
One use of sort was inefficient, sorting unsigned || signed when unsigned was
already properly sorted. Given how the unsigned TXs were given a nonce of 0, an
unstable sort may swap places with an unsigned TX and a signed TX with a nonce
of 0 (leading to a faulty block).
The extra protection added here sorts signed, then concats.
* Fix Tributary tests I broke, start review on tendermint/tx.rs
* Finish reviewing everything outside tests and empty_signature
* Remove empty_signature
empty_signature led to corrupted local state histories. Unfortunately, the API
is only sane with a signature.
We now use the actual signature, which risks creating a signature over a
malicious message if we have ever have an invariant producing malicious
messages. Prior, we only signed the message after the local machine confirmed
it was okay per the local view of consensus.
This is tolerated/preferred over a corrupt state history since production of
such messages is already an invariant. TODOs are added to make handling of this
theoretical invariant further robust.
* Remove async_sequential for tokio::test
There was no competition for resources forcing them to be run sequentially.
* Modify block order test to be statistically significant without multiple runs
* Clean tests
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* restrict batch size to ~25kb
* add batch size check to node
* rate limit batches to 1 per serai block
* add support for multiple batches for block
* fix review comments
* Misc fixes
Doesn't yet update tests/processor until data flow is inspected.
* Move the block from SignId to ProcessorMessage::BatchPreprocesses
* Misc clean up
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
For logging purposes, I added code to handle negative time till start. I forgot
to only sleep on positive time till start.
Should fix the recent CI failure.
It was improperly implemented, as it assumed rounds had a constant time
interval, which they do not. It also is against the spec and was meant to
absolve us of issues with poor performance when post-starting blockchains. The
new, and much more proper, workaround for the latter is a 120-second delay
between the Substrate time and the Tributary start time.
By default, tokio-spawned worker panics will only kill the task, not the
program. Due to our extensive use of panicking on invariants, we should ensure
the program exits.
It's largely unoptimized, and not yet exclusive to validators, yet has basic
sanity (using message content for ID instead of sender + index).
Fixes bugs as found. Notably, we used a time in milliseconds where the
Tributary expected seconds.
Also has Tributary::new jump to the presumed round number. This reduces slashes
when starting new chains (whose times will be before the current time) and was
the only way I was able to observe successful confirmations given current
surrounding infrastructure.
The Processor's coins folder referred to the networks it could process, as did
its Coin trait. This, and other similar cases throughout the codebase, have now
been corrected.
Also corrects dated documentation for a key pair is confirmed under the
validator-sets pallet.
