Commit graph

59 commits

Author SHA1 Message Date
Luke Parker
a0a7d63dad
bitcoin 0.32 2024-05-21 05:27:01 -04:00
Luke Parker
a41329c027
Update clippy now that redundant imports has been reverted 2024-04-23 04:31:27 -04:00
akildemir
627e7e6210
Add validator set rotation test for the node side (#532)
* add node side unit test

* complete rotation test for all networks

* set up the fast-epoch docker file

* fix pr comments
2024-02-24 14:51:06 -05:00
Luke Parker
cc75b52a43
Don't allow constructing unusable serai_client::bitcoin::Address es 2024-01-31 17:54:43 -05:00
Luke Parker
4913873b10
Slash reports (#523)
* report_slashes plumbing in Substrate

Notably delays the SetRetired event until it provides a slash report or the set
after it becomes the set to report its slashes.

* Add dedicated AcceptedHandover event

* Add SlashReport TX to Tributary

* Create SlashReport TXs

* Handle SlashReport TXs

* Add logic to generate a SlashReport to the coordinator

* Route SlashReportSigner into the processor

* Finish routing the SlashReport signing/TX publication

* Add serai feature to processor's serai-client
2024-01-29 03:48:53 -05:00
Luke Parker
b493e3e31f
Validator DHT (#494)
* Route validators for any active set through sc-authority-discovery

Additionally adds an RPC route to retrieve their P2P addresses.

* Have the coordinator get peers from substrate

* Have the RPC return one address, not up to 3

Prevents the coordinator from believing it has 3 peers when it has one.

* Add missing feature to serai-client

* Correct network argument in serai-client for p2p_validators call

* Add a test in serai-client to check DHT population with a much quicker failure than the coordinator tests

* Update to latest Substrate

Removes distinguishing BABE/AuthorityDiscovery keys which causes
sc_authority_discovery to populate as desired.

* Update to a properly tagged substrate commit

* Add all dialed to peers to GossipSub

* cargo fmt

* Reduce common code in serai-coordinator-tests with amore involved new_test

* Use a recursive async function to spawn `n` DockerTests with the necessary networking configuration

* Merge UNIQUE_ID and ONE_AT_A_TIME

* Tidy up the new recursive code in tests/coordinator

* Use a Mutex in CONTEXT to let it be set multiple times

* Make complimentary edits to full-stack tests

* Augment coordinator P2p connection logs

* Drop lock acquisitions before recursing

* Better scope lock acquisitions in full-stack, preventing a deadlock

* Ensure OUTER_OPS is reset across the test boundary

* Add cargo deny allowance for dockertest fork
2023-12-22 21:09:18 -05:00
Luke Parker
ea3af28139
Add workspace lints 2023-12-17 00:04:47 -05:00
Luke Parker
2532423d42
Remove the RemoveParticipant protocol for having new DKGs specify the participants which were removed
Obvious code cleanup is obvious.
2023-12-14 23:51:57 -05:00
Luke Parker
6a172825aa
Reattempts (#483)
* Schedule re-attempts and add a (not filled out) match statement to actually execute them

A comment explains the methodology. To copy it here:

"""
This is because we *always* re-attempt any protocol which had participation. That doesn't
mean we *should* re-attempt this protocol.

The alternatives were:
1) Note on-chain we completed a protocol, halting re-attempts upon 34%.
2) Vote on-chain to re-attempt a protocol.

This schema doesn't have any additional messages upon the success case (whereas
alternative #1 does) and doesn't have overhead (as alternative #2 does, sending votes and
then preprocesses. This only sends preprocesses).
"""

Any signing protocol which reaches sufficient participation will be
re-attempted until it no longer does.

* Have the Substrate scanner track DKG removals/completions for the Tributary code

* Don't keep trying to publish a participant removal if we've already set keys

* Pad out the re-attempt match a bit more

* Have CosignEvaluator reload from the DB

* Correctly schedule cosign re-attempts

* Actuall spawn new DKG removal attempts

* Use u32 for Batch ID in SubstrateSignableId, finish Batch re-attempt routing

The batch ID was an opaque [u8; 5] which also included the network, yet that's
redundant and unhelpful.

* Clarify a pair of TODOs in the coordinator

* Remove old TODO

* Final comment cleanup

* Correct usage of TARGET_BLOCK_TIME in reattempt scheduler

It's in ms and I assumed it was in s.

* Have coordinator tests drop BatchReattempts which aren't relevant yet may exist

* Bug fix and pointless oddity removal

We scheduled a re-attempt upon receiving 2/3rds of preprocesses and upon
receiving 2/3rds of shares, so any signing protocol could cause two re-attempts
(not one more).

The coordinator tests randomly generated the Batch ID since it was prior an
opaque byte array. While that didn't break the test, it was pointless and did
make the already-succeeded check before re-attempting impossible to hit.

* Add log statements, correct dead-lock in coordinator tests

* Increase pessimistic timeout on recv_message to compensate for tighter best-case timeouts

* Further bump timeout by a minute

AFAICT, GH failed by just a few seconds.

This also is worst-case in a single instance, making it fine to be decently long.

* Further further bump timeout due to lack of distinct error
2023-12-12 12:28:53 -05:00
Luke Parker
7122e0faf4
Cache the block's events within TemporalSerai
Event retrieval was prior:
- Retrieve all events in the block, which may be hundreds of KB
- Filter to just a few

Since it's frequent to want multiple sets of events, each filtered in their own
way, this caused the retrieval to happen multiple times. Now, it only will
happen once.

Also has the scoped clients take a reference, not an owned TemporalSerai.
2023-12-08 10:46:10 -05:00
Luke Parker
c511a54d18
Move serai-client off serai-runtime, MIT licensing it
Uses a full-fledged serai-abi to do so.

Removes use of UncheckedExtrinsic as a pointlessly (for us) length-prefixed
block with a more complicated signing algorithm than advantageous.

In the future, we should considering consolidating the various primitives
crates. I'm not convinced we benefit from one primitives crate per pallet.
2023-12-07 02:30:09 -05:00
Luke Parker
797ed49e7b
DKG Removals (#467)
* Update ValidatorSets with a remove_participant call

* Add DkgRemoval, a sign machine for producing the relevant MuSig signatures

* Don't use position-dependent u8s yet Public when removing validators from the DKG

* Add DkgRemovalPreprocess, DkgRemovalShares

Implementation is via a new publish_tributary_tx lambda.

This is code is a copy-pasted mess which will need to be cleaned up.

* Only allow non-removed validators to vote for removals

Otherwise, it's risked that the remaining validators fall below 67% of the
original set.

* Correct publish_serai_tx, which was prior publish_set_keys in practice
2023-12-04 07:04:44 -05:00
Luke Parker
695d1f0ecf
Remove subxt (#460)
* Remove subxt

Removes ~20 crates from our Cargo.lock.

Removes downloading the metadata and enables removing the getMetadata RPC route
(relevant to #379).

Moves forward #337.

Done now due to distinctions in the subxt 0.32 API surface which make it
justifiable to not update.

* fmt, update due to deny triggering on a yanked crate

* Correct the handling of substrate_block_notifier now that it's ephemeral, not long-lived

* Correct URL in tests/coordinator from ws to http
2023-11-28 02:29:50 -05:00
Luke Parker
d60e007126
Add a binaries feature to the processor to reduce dependencies when used as a lib
processor isn't intended to be used as a library, yet serai-processor-tests
does pull it in as a lib. This caused serai-processor-tests to need to compile
rocksdb, which added multiple minutes to the compilation time.
2023-11-25 04:04:52 -05:00
Luke Parker
372149c2cc
Various simplifications re: Serai transactions
Removes PairSigner for the pair directly.

Resets spec_version to 1.

Defines the extrinsic without its length prefix, only prefixing during publish.
2023-11-23 00:02:01 -05:00
Luke Parker
f58478ad87
Add hex as a dependency to serai-client 2023-11-22 18:06:10 -05:00
Luke Parker
88a1726399
Fixes for prior commit 2023-11-22 16:24:50 -05:00
Luke Parker
08e6669403
Replace substrate/client's use of Payload with usage of RuntimeCall
Gains explicit typing.
2023-11-22 11:23:04 -05:00
akildemir
fcfdadc791
Integrate session pallet into validator-sets pallet (#440)
* remove pallet-session

* Store key shares in InSet

* integrate grandpa to vs-pallet

* integrate pallet babe

* remove pallet-session & authority discovery from runtime

* update the grandpa pallet path

* cargo update grandpa

* cargo update substrate

* Misc tweaks

Sets validators for BABE/GRANDPA in chain_spec, per Akil's realization that was
the missing piece.

* fix pr comments

* bug fix & tidy up

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2023-11-22 06:22:46 -05:00
Luke Parker
96f1d26f7a
Add a cosigning protocol to ensure finalizations are unique (#433)
* Add a function to deterministically decide which Serai blocks should be co-signed

Has a 5 minute latency between co-signs, also used as the maximal latency
before a co-sign is started.

* Get all active tributaries we're in at a specific block

* Add and route CosignSubstrateBlock, a new provided TX

* Split queued cosigns per network

* Rename BatchSignId to SubstrateSignId

* Add SubstrateSignableId, a meta-type for either Batch or Block, and modularize around it

* Handle the CosignSubstrateBlock provided TX

* Revert substrate_signer.rs to develop (and patch to still work)

Due to SubstrateSigner moving when the prior multisig closes, yet cosigning
occurring with the most recent key, a single SubstrateSigner can be reused.
We could manage multiple SubstrateSigners, yet considering the much lower
specifications for cosigning, I'd rather treat it distinctly.

* Route cosigning through the processor

* Add note to rename SubstrateSigner post-PR

I don't want to do so now in order to preserve the diff's clarity.

* Implement cosign evaluation into the coordinator

* Get tests to compile

* Bug fixes, mark blocks without cosigners available as cosigned

* Correct the ID Batch preprocesses are saved under, add log statements

* Create a dedicated function to handle cosigns

* Correct the flow around Batch verification/queueing

Verifying `Batch`s could stall when a `Batch` was signed before its
predecessors/before the block it's contained in was cosigned (the latter being
inevitable as we can't sign a block containing a signed batch before signing
the batch).

Now, Batch verification happens on a distinct async task in order to not block
the handling of processor messages. This task is the sole caller of verify in
order to ensure last_verified_batch isn't unexpectedly mutated.

When the processor message handler needs to access it, or needs to queue a
Batch, it associates the DB TXN with a lock preventing the other task from
doing so.

This lock, as currently implemented, is a poor and inefficient design. It
should be modified to the pattern used for cosign management. Additionally, a
new primitive of a DB-backed channel may be immensely valuable.

Fixes a standing potential deadlock and a deadlock introduced with the
cosigning protocol.

* Working full-stack tests

After the last commit, this only required extending a timeout.

* Replace "co-sign" with "cosign" to make finding text easier

* Update the coordinator tests to support cosigning

* Inline prior_batch calculation to prevent panic on rotation

Noticed when doing a final review of the branch.
2023-11-15 16:57:21 -05:00
akildemir
d015ee96a3
Dex improvements (#422)
* remove dex traits&balance types

* remove liq tokens pallet in favor of coins-pallet instance

* fix tests & benchmarks

* remove liquidity tokens trait

* fix CI

* fix pr comments

* Slight renamings

* Add burn_with_instruction as a negative to LiquidityTokens CallFilter

* Remove use of One, Zero, Saturating taits in dex pallet

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2023-11-12 06:37:31 -05:00
Luke Parker
06e627a562
Support refunds as possible for invalidly received outputs on Serai 2023-11-08 11:26:28 -05:00
akildemir
899a9604e1
Add Dex pallet (#407)
* Move pallet-asset-conversion

* update licensing

* initial integration

* Integrate Currency & Assets types

* integrate liquidity tokens

* fmt

* integrate dex pallet tests

* fmt

* compilation error fixes

* integrate dex benchmarks

* fmt

* cargo clippy

* replace all occurrences of "asset" with "coin"

* add the actual add liq/swap logic to in-instructions

* add client side & tests

* fix deny

* Lint and changes

- Renames InInstruction::AddLiquidity to InInstruction::SwapAndAddLiquidity
- Makes create_pool an internal function
- Makes dex-pallet exclusively create pools against a native coin
- Removes various fees
- Adds new crates to GH workflow

* Fix rebase artifacts

* Correct other rebase artifact

* Correct CI specification for liquidity-tokens

* Correct primitives' test to the standardized pallet account scheme

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2023-11-05 12:02:34 -05:00
Luke Parker
34bcb9eb01 bitcoin 0.31 2023-10-31 03:47:45 -04:00
Luke Parker
a1b2bdf0a2
clippy fixes 2023-10-19 06:30:58 -04:00
akildemir
fdfce9e207
Coins pallet (#399)
* initial implementation

* add function to get a balance of an account

* add support for multiple coins

* rename pallet to "coins-pallet"

* replace balances, assets and tokens pallet with coins pallet in runtime

* add total supply info

* update client side for new Coins pallet

* handle fees

* bug fixes

* Update FeeAccount test

* Fmt

* fix pr comments

* remove extraneous Imbalance type

* Minor tweaks

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2023-10-19 06:22:21 -04:00
Luke Parker
3b3fdd104b
Most of coordinator Tributary retiry
Adds Event::SetRetired to validator-sets.

Emit TributaryRetired.

Replaces is_active_set, which made multiple network requests, with
is_retired_tributary, a DB read.

Performs most of the removals necessary upon TributaryRetired.

Still needs to clean up the actual Tributary/Tendermint tasks.
2023-10-14 16:47:25 -04:00
Luke Parker
5c5c097da9 Tweaks for processor to work with the new serai-client 2023-10-14 15:26:36 -04:00
Luke Parker
7d4e8b59db Update dockertests to new serai-client 2023-10-14 15:26:36 -04:00
Luke Parker
cb61c9052a Reorganize serai-client
Instead of functions taking a block hash, has a scope to a block hash before
functions can be called.

Separates functions by pallets.
2023-10-14 15:26:36 -04:00
Luke Parker
02edfd2935
Verify all Batchs published by the prior set
The new set publishing a `Batch` completes the handover protocol. The new set
should only publish a `Batch` once it believes the old set has completed all of
its on-external-chain activity, marking it honest and finite.

With the handover comes the acceptance of liability, hence the requirement for
all of the on-Serai-chain activity also needing verification. While most
activity would be verified in-real-time (upon ::Batch messages), the new set
will now explicitly verify the complete set of `Batch`s before beginning its
preprocess for its own `Batch` (the one accepting the handover).
2023-10-13 04:12:21 -04:00
Luke Parker
9aeece5bf6
Give one weight per key share to validators in Tributary 2023-10-13 02:29:22 -04:00
Luke Parker
1a0b4198ba
Correct the check for if we still need to set keys
The prior check had an edge case where once keys were pruned, it'd believe the
keys needed to be set ad infinitum.
2023-10-10 22:53:15 -04:00
akildemir
98190b7b83
Staking pallet (#373)
* initial staking pallet

* add staking pallet to runtime

* support session rotation for serai

* optimizations & cleaning

* fix deny

* add serai network to initial networks

* a few tweaks & comments

* fix some pr comments

* Rewrite validator-sets with logarithmic algorithms

Uses the fact the underlying DB is sorted to achieve sorting of potential
validators by stake.

Removes release of deallocated stake for now.

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2023-10-10 06:53:24 -04:00
Luke Parker
f7e49e1f90
Update Rust nightly
Supersedes #368.

Adds exceptions for unwrap_or_default due to preference against Default's
ambiguity.
2023-09-02 01:24:09 -04:00
Luke Parker
d5a19eca8c
Add a notification system for finalizations to serai-client, use in coordinator 2023-08-30 17:25:04 -04:00
Luke Parker
285422f71a
Add a full-stack mint and burn test for Bitcoin and Monero
Fixes where ram_scanned is updated in processor. The prior version, while safe,
would redo massive amounts of work during periods of inactivity. It also hit an
undocumented invariant where get_eventuality_completions assumes new blocks,
yet redone work wouldn't have new blocks.

Modifies Monero's generate_blocks to return the hashes of the generated blocks.
2023-08-28 21:17:22 -04:00
Luke Parker
37b6de9c0c
Add SRI balanace/transfer functions to serai_client 2023-08-26 21:37:54 -04:00
Luke Parker
22f3c9e58f
Stop trying to publish a Batch if another node does 2023-08-26 21:37:21 -04:00
akildemir
e680eabb62
Improve batch handling (#316)
* restrict batch size to ~25kb

* add batch size check to node

* rate limit batches to 1 per serai block

* add support for multiple batches for block

* fix review comments

* Misc fixes

Doesn't yet update tests/processor until data flow is inspected.

* Move the block from SignId to ProcessorMessage::BatchPreprocesses

* Misc clean up

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2023-08-14 11:57:38 -04:00
Luke Parker
a3441a6871
Don't have publish return the 'hash' 2023-08-14 08:18:19 -04:00
Luke Parker
5e02f936e4
Perform MuSig signing of generated keys 2023-08-14 06:08:55 -04:00
Luke Parker
f6f945e747
Add a LibP2P instantiation to coordinator
It's largely unoptimized, and not yet exclusive to validators, yet has basic
sanity (using message content for ID instead of sender + index).

Fixes bugs as found. Notably, we used a time in milliseconds where the
Tributary expected  seconds.

Also has Tributary::new jump to the presumed round number. This reduces slashes
when starting new chains (whose times will be before the current time) and was
the only way I was able to observe successful confirmations given current
surrounding infrastructure.
2023-08-08 15:12:47 -04:00
Luke Parker
3c38a0ec11
cargo +nightly fmt 2023-08-01 00:47:36 -04:00
Luke Parker
9f143a9742
Replace "coin" with "network"
The Processor's coins folder referred to the networks it could process, as did
its Coin trait. This, and other similar cases throughout the codebase, have now
been corrected.

Also corrects dated documentation for a key pair is confirmed under the
validator-sets pallet.
2023-07-30 16:11:30 -04:00
Luke Parker
1b57d655ed
Update to subxt 0.29 2023-07-18 23:01:51 -04:00
Luke Parker
f6a497f3ac
Slight terminology correction in sync test
Also correct a mistake from merging the most recent polkadot version.
2023-06-28 15:20:50 -04:00
Luke Parker
40b2920412
Remove signed Substrate TXs from Coordinator 2023-05-13 22:43:13 -04:00
Luke Parker
f069567f12
Use a MuSig signature to publish validator set key pairs to Serai
The processor/coordinator flow still has to be rewritten.
2023-05-13 02:15:41 -04:00
Luke Parker
f55e9b40e6
Have coordinator publish batches to Substrate 2023-05-10 01:46:20 -04:00