It's still unclear how we'll handle refunding failed InInstructions at this
time. Presumably, extending the InInstruction channel with the associated
output ID?
Has fetched blocks checked to be the indexed blocks. Has scanned outputs be
sorted, meaning they aren't subject to implicit order/may be non-deterministic
(such as if handled by a threadpool).
Scan now only handles External outputs, with an associated essay going over
why. Scan directly creates the InInstruction (prior planned to be done in
Report), and Eventuality is declared to end up yielding the outputs.
That will require making the Eventuality flow two-stage. One stage to evaluate
existing Eventualities and yield outputs, and one stage to incorporate new
Eventualities before advancing the scan window.
Abstract, to be done for the transactions, the batches, the cosigns, the slash
reports, everything. It has a minimal API itself, intending to be as clear as
possible.
* Upstream GBP, divisor, circuit abstraction, and EC gadgets from FCMP++
* Initial eVRF implementation
Not quite done yet. It needs to communicate the resulting points and proofs to
extract them from the Pedersen Commitments in order to return those, and then
be tested.
* Add the openings of the PCs to the eVRF as necessary
* Add implementation of secq256k1
* Make DKG Encryption a bit more flexible
No longer requires the use of an EncryptionKeyMessage, and allows pre-defined
keys for encryption.
* Make NUM_BITS an argument for the field macro
* Have the eVRF take a Zeroizing private key
* Initial eVRF-based DKG
* Add embedwards25519 curve
* Inline the eVRF into the DKG library
Due to how we're handling share encryption, we'd either need two circuits or to
dedicate this circuit to the DKG. The latter makes sense at this time.
* Add documentation to the eVRF-based DKG
* Add paragraph claiming robustness
* Update to the new eVRF proof
* Finish routing the eVRF functionality
Still needs errors and serialization, along with a few other TODOs.
* Add initial eVRF DKG test
* Improve eVRF DKG
Updates how we calculcate verification shares, improves performance when
extracting multiple sets of keys, and adds more to the test for it.
* Start using a proper error for the eVRF DKG
* Resolve various TODOs
Supports recovering multiple key shares from the eVRF DKG.
Inlines two loops to save 2**16 iterations.
Adds support for creating a constant time representation of scalars < NUM_BITS.
* Ban zero ECDH keys, document non-zero requirements
* Implement eVRF traits, all the way up to the DKG, for secp256k1/ed25519
* Add Ristretto eVRF trait impls
* Support participating multiple times in the eVRF DKG
* Only participate once per key, not once per key share
* Rewrite processor key-gen around the eVRF DKG
Still a WIP.
* Finish routing the new key gen in the processor
Doesn't touch the tests, coordinator, nor Substrate yet.
`cargo +nightly fmt && cargo +nightly-2024-07-01 clippy --all-features -p serai-processor`
does pass.
* Deduplicate and better document in processor key_gen
* Update serai-processor tests to the new key gen
* Correct amount of yx coefficients, get processor key gen test to pass
* Add embedded elliptic curve keys to Substrate
* Update processor key gen tests to the eVRF DKG
* Have set_keys take signature_participants, not removed_participants
Now no one is removed from the DKG. Only `t` people publish the key however.
Uses a BitVec for an efficient encoding of the participants.
* Update the coordinator binary for the new DKG
This does not yet update any tests.
* Add sensible Debug to key_gen::[Processor, Coordinator]Message
* Have the DKG explicitly declare how to interpolate its shares
Removes the hack for MuSig where we multiply keys by the inverse of their
lagrange interpolation factor.
* Replace Interpolation::None with Interpolation::Constant
Allows the MuSig DKG to keep the secret share as the original private key,
enabling deriving FROST nonces consistently regardless of the MuSig context.
* Get coordinator tests to pass
* Update spec to the new DKG
* Get clippy to pass across the repo
* cargo machete
* Add an extra sleep to ensure expected ordering of `Participation`s
* Update orchestration
* Remove bad panic in coordinator
It expected ConfirmationShare to be n-of-n, not t-of-n.
* Improve documentation on functions
* Update TX size limit
We now no longer have to support the ridiculous case of having 49 DKG
participations within a 101-of-150 DKG. It does remain quite high due to
needing to _sign_ so many times. It'd may be optimal for parties with multiple
key shares to independently send their preprocesses/shares (despite the
overhead that'll cause with signatures and the transaction structure).
* Correct error in the Processor spec document
* Update a few comments in the validator-sets pallet
* Send/Recv Participation one at a time
Sending all, then attempting to receive all in an expected order, wasn't working
even with notable delays between sending messages. This points to the mempool
not working as expected...
* Correct ThresholdKeys serialization in modular-frost test
* Updating existing TX size limit test for the new DKG parameters
* Increase time allowed for the DKG on the GH CI
* Correct construction of signature_participants in serai-client tests
Fault identified by akil.
* Further contextualize DkgConfirmer by ValidatorSet
Caught by a safety check we wouldn't reuse preprocesses across messages. That
raises the question of we were prior reusing preprocesses (reusing keys)?
Except that'd have caused a variety of signing failures (suggesting we had some
staggered timing avoiding it in practice but yes, this was possible in theory).
* Add necessary calls to set_embedded_elliptic_curve_key in coordinator set rotation tests
* Correct shimmed setting of a secq256k1 key
* cargo fmt
* Don't use `[0; 32]` for the embedded keys in the coordinator rotation test
The key_gen function expects the random values already decided.
* Big-endian secq256k1 scalars
Also restores the prior, safer, Encryption::register function.
* Rename the coins folder to networks
Ethereum isn't a coin. It's a network.
Resolves#357.
* More renames of coins -> networks in orchestration
* Correct paths in tests/
* cargo fmt
* Remove unsafe creation of dalek_ff_group::EdwardsPoint in BP+
* Rename Bulletproofs to Bulletproof, since they are a single Bulletproof
Also bifurcates prove with prove_plus, and adds a few documentation items.
* Make CLSAG signing private
Also adds a bit more documentation and does a bit more tidying.
* Remove the distribution cache
It's a notable bandwidth/performance improvement, yet it's not ready. We need a
dedicated Distribution struct which is managed by the wallet and passed in.
While we can do that now, it's not currently worth the effort.
* Tidy Borromean/MLSAG a tad
* Remove experimental feature from monero-serai
* Move amount_decryption into EncryptedAmount::decrypt
* Various RingCT doc comments
* Begin crate smashing
* Further documentation, start shoring up API boundaries of existing crates
* Document and clean clsag
* Add a dedicated send/recv CLSAG mask struct
Abstracts the types used internally.
Also moves the tests from monero-serai to monero-clsag.
* Smash out monero-bulletproofs
Removes usage of dalek-ff-group/multiexp for curve25519-dalek.
Makes compiling in the generators an optional feature.
Adds a structured batch verifier which should be notably more performant.
Documentation and clean up still necessary.
* Correct no-std builds for monero-clsag and monero-bulletproofs
* Tidy and document monero-bulletproofs
I still don't like the impl of the original Bulletproofs...
* Error if missing documentation
* Smash out MLSAG
* Smash out Borromean
* Tidy up monero-serai as a meta crate
* Smash out RPC, wallet
* Document the RPC
* Improve docs a bit
* Move Protocol to monero-wallet
* Incomplete work on using Option to remove panic cases
* Finish documenting monero-serai
* Remove TODO on reading pseudo_outs for AggregateMlsagBorromean
* Only read transactions with one Input::Gen or all Input::ToKey
Also adds a helper to fetch a transaction's prefix.
* Smash out polyseed
* Smash out seed
* Get the repo to compile again
* Smash out Monero addresses
* Document cargo features
Credit to @hinto-janai for adding such sections to their work on documenting
monero-serai in #568.
* Fix deserializing v2 miner transactions
* Rewrite monero-wallet's send code
I have yet to redo the multisig code and the builder. This should be much
cleaner, albeit slower due to redoing work.
This compiles with clippy --all-features. I have to finish the multisig/builder
for --all-targets to work (and start updating the rest of Serai).
* Add SignableTransaction Read/Write
* Restore Monero multisig TX code
* Correct invalid RPC type def in monero-rpc
* Update monero-wallet tests to compile
Some are _consistently_ failing due to the inputs we attempt to spend being too
young. I'm unsure what's up with that. Most seem to pass _consistently_,
implying it's not a random issue yet some configuration/env aspect.
* Clean and document monero-address
* Sync rest of repo with monero-serai changes
* Represent height/block number as a u32
* Diversify ViewPair/Scanner into ViewPair/GuaranteedViewPair and Scanner/GuaranteedScanner
Also cleans the Scanner impl.
* Remove non-small-order view key bound
Guaranteed addresses are in fact guaranteed even with this due to prefixing key
images causing zeroing the ECDH to not zero the shared key.
* Finish documenting monero-serai
* Correct imports for no-std
* Remove possible panic in monero-serai on systems < 32 bits
This was done by requiring the system's usize can represent a certain number.
* Restore the reserialize chain binary
* fmt, machete, GH CI
* Correct misc TODOs in monero-serai
* Have Monero test runner evaluate an Eventuality for all signed TXs
* Fix a pair of bugs in the decoy tests
Unfortunately, this test is still failing.
* Fix remaining bugs in monero-wallet tests
* Reject torsioned spend keys to ensure we can spend the outputs we scan
* Tidy inlined epee code in the RPC
* Correct the accidental swap of stagenet/testnet address bytes
* Remove unused dep from processor
* Handle Monero fee logic properly in the processor
* Document v2 TX/RCT output relation assumed when scanning
* Adjust how we mine the initial blocks due to some CI test failures
* Fix weight estimation for RctType::ClsagBulletproof TXs
* Again increase the amount of blocks we mine prior to running tests
* Correct the if check about when to mine blocks on start
Finally fixes the lack of decoy candidates failures in CI.
* Run Monero on Debian, even for internal testnets
Change made due to a segfault incurred when locally testing.
https://github.com/monero-project/monero/issues/9141 for the upstream.
* Don't attempt running tests on the verify-chain binary
Adds a minimum XMR fee to the processor and runs fmt.
* Increase minimum Monero fee in processor
I'm truly unsure why this is required right now.
* Distinguish fee from necessary_fee in monero-wallet
If there's no change, the fee is difference of the inputs to the outputs. The
prior code wouldn't check that amount is greater than or equal to the necessary
fee, and returning the would-be change amount as the fee isn't necessarily
helpful.
Now the fee is validated in such cases and the necessary fee is returned,
enabling operating off of that.
* Restore minimum Monero fee from develop
* Clean up Ethereum
* Consistent contract address for deployed contracts
* Flesh out Router a bit
* Add a Deployer for DoS-less deployment
* Implement Router-finding
* Use CREATE2 helper present in ethers
* Move from CREATE2 to CREATE
Bit more streamlined for our use case.
* Document ethereum-serai
* Tidy tests a bit
* Test updateSeraiKey
* Use encodePacked for updateSeraiKey
* Take in the block hash to read state during
* Add a Sandbox contract to the Ethereum integration
* Add retrieval of transfers from Ethereum
* Add inInstruction function to the Router
* Augment our handling of InInstructions events with a check the transfer event also exists
* Have the Deployer error upon failed deployments
* Add --via-ir
* Make get_transaction test-only
We only used it to get transactions to confirm the resolution of Eventualities.
Eventualities need to be modularized. By introducing the dedicated
confirm_completion function, we remove the need for a non-test get_transaction
AND begin this modularization (by no longer explicitly grabbing a transaction
to check with).
* Modularize Eventuality
Almost fully-deprecates the Transaction trait for Completion. Replaces
Transaction ID with Claim.
* Modularize the Scheduler behind a trait
* Add an extremely basic account Scheduler
* Add nonce uses, key rotation to the account scheduler
* Only report the account Scheduler empty after transferring keys
Also ban payments to the branch/change/forward addresses.
* Make fns reliant on state test-only
* Start of an Ethereum integration for the processor
* Add a session to the Router to prevent updateSeraiKey replaying
This would only happen if an old key was rotated to again, which would require
n-of-n collusion (already ridiculous and a valid fault attributable event). It
just clarifies the formal arguments.
* Add a RouterCommand + SignMachine for producing it to coins/ethereum
* Ethereum which compiles
* Have branch/change/forward return an option
Also defines a UtxoNetwork extension trait for MAX_INPUTS.
* Make external_address exclusively a test fn
* Move the "account" scheduler to "smart contract"
* Remove ABI artifact
* Move refund/forward Plan creation into the Processor
We create forward Plans in the scan path, and need to know their exact fees in
the scan path. This requires adding a somewhat wonky shim_forward_plan method
so we can obtain a Plan equivalent to the actual forward Plan for fee reasons,
yet don't expect it to be the actual forward Plan (which may be distinct if
the Plan pulls from the global state, such as with a nonce).
Also properly types a Scheduler addendum such that the SC scheduler isn't
cramming the nonce to use into the N::Output type.
* Flesh out the Ethereum integration more
* Two commits ago, into the **Scheduler, not Processor
* Remove misc TODOs in SC Scheduler
* Add constructor to RouterCommandMachine
* RouterCommand read, pairing with the prior added write
* Further add serialization methods
* Have the Router's key included with the InInstruction
This does not use the key at the time of the event. This uses the key at the
end of the block for the event. Its much simpler than getting the full event
streams for each, checking when they interlace.
This does not read the state. Every block, this makes a request for every
single key update and simply chooses the last one. This allows pruning state,
only keeping the event tree. Ideally, we'd also introduce a cache to reduce the
cost of the filter (small in events yielded, long in blocks searched).
Since Serai doesn't have any forwarding TXs, nor Branches, nor change, all of
our Plans should solely have payments out, and there's no expectation of a Plan
being made under one key broken by it being received by another key.
* Add read/write to InInstruction
* Abstract the ABI for Call/OutInstruction in ethereum-serai
* Fill out signable_transaction for Ethereum
* Move ethereum-serai to alloy
Resolves#331.
* Use the opaque sol macro instead of generated files
* Move the processor over to the now-alloy-based ethereum-serai
* Use the ecrecover provided by alloy
* Have the SC use nonce for rotation, not session (an independent nonce which wasn't synchronized)
* Always use the latest keys for SC scheduled plans
* get_eventuality_completions for Ethereum
* Finish fleshing out the processor Ethereum integration as needed for serai-processor tests
This doesn't not support any actual deployments, not even the ones simulated by
serai-processor-docker-tests.
* Add alloy-simple-request-transport to the GH workflows
* cargo update
* Clarify a few comments and make one check more robust
* Use a string for 27.0 in .github
* Remove optional from no-longer-optional dependencies in processor
* Add alloy to git deny exception
* Fix no longer optional specification in processor's binaries feature
* Use a version of foundry from 2024
* Correct fetching Bitcoin TXs in the processor docker tests
* Update rustls to resolve RUSTSEC warnings
* Use the monthly nightly foundry, not the deleted daily nightly
* Monero: fix decoy selection algo and add test for latest spendable
- DSA only selected coinbase outputs and didn't match the wallet2
implementation
- Added test to make sure DSA will select a decoy output from the
most recent unlocked block
- Made usage of "height" in DSA consistent with other usage of
"height" in Monero code (height == num blocks in chain)
- Rely on monerod RPC response for output's unlocked status
* xmr runner tests mine until outputs are unlocked
* fingerprintable canoncial select decoys
* Separate fingerprintable canonical function
Makes it simpler for callers who are unconcered with consistent
canonical output selection across multiple clients to rely on
the simpler Decoy::select and not worry about fingerprintable
canonical
* fix merge conflicts
* Put back TODO for issue #104
* Fix incorrect check on distribution len
The RingCT distribution on mainnet doesn't start until well after
genesis, so the distribution length is expected to be < height.
To be clear, this was my mistake from this series of changes
to the DSA. I noticed this mistake because the DSA would error
when running on mainnet.
* monero: Use fee priority enums from monero repo CLI/RPC wallets
* Update processor for fee priority change
* Remove FeePriority::Default
Done in consultation with @j-berman.
The RPC/CLI/GUI almost always adjust up except barring very explicit commands,
hence why FeePriority 0 is now only exposed via the explicit command of
FeePriority::Custom { priority: 0 }.
Also helps with terminology.
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* add input script check
* add test
* optimizations
* bug fix
* fix pr comments
* Test SegWit-encoded data using a single output (not two)
* Remove TODO used as a question, document origins when SegWit encoding
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
Moves from concatted Dockerfiles to pseudo-templated Dockerfiles via a dedicated Rust program.
Removes the unmaintained kubernetes, not because we shouldn't have/use it, but because it's unmaintained and needs to be reworked before it's present again.
Replaces the compose with the work in the new orchestrator binary which spawns everything as expected. While this arguably re-invents the wheel, it correctly manages secrets and handles the variadic Dockerfiles.
Also adds an unrelated patch for zstd and simplifies running services a bit by greater utilizing the existing infrastructure.
---
* Delete all Dockerfile fragments, add new orchestator to generate Dockerfiles
Enables greater templating.
Also delete the unmaintained kubernetes folder *for now*. This should be
restored in the future.
* Use Dockerfiles from the orchestator
* Ignore Dockerfiles in the git repo
* Remove CI job to check Dockerfiles are as expected now that they're no longer committed
* Remove old Dockerfiles from repo
* Use Debian for monero-wallet-rpc
* Remove replace_cmds for proper usage of entry-dev
Consolidates ports a bit.
Updates serai-docker-tests from "compose" to "build".
* Only write a new dockerfile if it's distinct
Preserves the updated time metadata.
* Update serai-docker-tests
* Correct the path Dockerfiles are built from
* Correct inclusion of orchestration folder in Docker builds
* Correct debug/release flagging in the cargo command
Apparently, --debug isn't an effective NOP yet an error.
* Correct path used to run the Serai node within a Dockerfile
* Correct path in Monero Dockerfile
* Attempt storing monerod in /usr/bin
* Use sudo to move into /usr/bin in CI
* Correct 18.3.0 to 18.3.1
* Escape * with quotes
* Update deny.toml, ADD orchestration in runtime Dockerfile
* Add --detach to the Monero GH CI
* Diversify dockerfiles by network
* Fixes to network-diversified orchestration
* Bitcoin and Monero testnet scripts
* Permissions and tweaks
* Flatten scripts folders
* Add missing folder specification to Monero Dockerfile
* Have monero-wallet-rpc specify the monerod login
* Have the Docker CMD specify env variables inserted at time of Dockerfile generation
They're overrideable with the global enviornment as for tests. This enables
variable generation in orchestrator and output to productionized Docker files
without creating a life-long file within the Docker container.
* Don't add Dockerfiles into Docker containers now that they have secrets
Solely add the source code for them as needed to satisfy the workspace bounds.
* Download arm64 Monero on arm64
* Ensure constant host architecture when reproducibly building the wasm
Host architecture, for some reason, can effect the generated code despite the
target architecture always being foreign to the host architecture.
* Randomly generate infrastructure keys
* Have orchestrator generate a key, be able to create/start containers
* Ensure bash is used over sh
* Clean dated docs
* Change how quoting occurs
* Standardize to sh
* Have Docker test build the dev Dockerfiles
* Only key_gen once
* cargo update
Adds a patch for zstd and reconciles the breaking nightly change which just
occurred.
* Use a dedicated network for Serai
Also fixes SERAI_HOSTNAME passed to coordinator.
* Support providing a key over the env for the Serai node
* Enable and document running daemons for tests via serai-orchestrator
Has running containers under the dev network port forward the RPC ports.
* Use volumes for bitcoin/monero
* Use bitcoin's run.sh in GH CI
* Only use the volume for testnet (not dev)
* report_slashes plumbing in Substrate
Notably delays the SetRetired event until it provides a slash report or the set
after it becomes the set to report its slashes.
* Add dedicated AcceptedHandover event
* Add SlashReport TX to Tributary
* Create SlashReport TXs
* Handle SlashReport TXs
* Add logic to generate a SlashReport to the coordinator
* Route SlashReportSigner into the processor
* Finish routing the SlashReport signing/TX publication
* Add serai feature to processor's serai-client
* Use an extended timeout for DKGs specifically
* Add a log statement when message-queue connection fails
* Add a 60 second keep-alive to connections
* Use zalloc for processor/message-queue/coordinator
An additional layer which protects us against edge cases with Zeroizing
(objects which don't support it or don't miss it).
* Add further logs to message-queue
* Further increase re-attempt timeouts in CI
* Remove misplaced continue inmessage-queue client
Fixes observed CI failures.
* Revert "Further increase re-attempt timeouts in CI"
This reverts commit 3723530cf6.
With a DKG removal comes a reduction in the amount of participants which was
ignored by re-attempts.
Now, we determine n/i based on the parties removed, and deterministically
obtain the context of who was removd.
* Schedule re-attempts and add a (not filled out) match statement to actually execute them
A comment explains the methodology. To copy it here:
"""
This is because we *always* re-attempt any protocol which had participation. That doesn't
mean we *should* re-attempt this protocol.
The alternatives were:
1) Note on-chain we completed a protocol, halting re-attempts upon 34%.
2) Vote on-chain to re-attempt a protocol.
This schema doesn't have any additional messages upon the success case (whereas
alternative #1 does) and doesn't have overhead (as alternative #2 does, sending votes and
then preprocesses. This only sends preprocesses).
"""
Any signing protocol which reaches sufficient participation will be
re-attempted until it no longer does.
* Have the Substrate scanner track DKG removals/completions for the Tributary code
* Don't keep trying to publish a participant removal if we've already set keys
* Pad out the re-attempt match a bit more
* Have CosignEvaluator reload from the DB
* Correctly schedule cosign re-attempts
* Actuall spawn new DKG removal attempts
* Use u32 for Batch ID in SubstrateSignableId, finish Batch re-attempt routing
The batch ID was an opaque [u8; 5] which also included the network, yet that's
redundant and unhelpful.
* Clarify a pair of TODOs in the coordinator
* Remove old TODO
* Final comment cleanup
* Correct usage of TARGET_BLOCK_TIME in reattempt scheduler
It's in ms and I assumed it was in s.
* Have coordinator tests drop BatchReattempts which aren't relevant yet may exist
* Bug fix and pointless oddity removal
We scheduled a re-attempt upon receiving 2/3rds of preprocesses and upon
receiving 2/3rds of shares, so any signing protocol could cause two re-attempts
(not one more).
The coordinator tests randomly generated the Batch ID since it was prior an
opaque byte array. While that didn't break the test, it was pointless and did
make the already-succeeded check before re-attempting impossible to hit.
* Add log statements, correct dead-lock in coordinator tests
* Increase pessimistic timeout on recv_message to compensate for tighter best-case timeouts
* Further bump timeout by a minute
AFAICT, GH failed by just a few seconds.
This also is worst-case in a single instance, making it fine to be decently long.
* Further further bump timeout due to lack of distinct error
* Move logic for evaluating if a cosign should occur to its own file
Cleans it up and makes it more robust.
* Have expected_next_batch return an error instead of retrying
While convenient to offer an error-free implementation, it potentially caused
very long lived lock acquisitions in handle_processor_message.
* Unify and clean DkgConfirmer and DkgRemoval
Does so via adding a new file for the common code, SigningProtocol.
Modifies from_cache to return the preprocess with the machine, as there's no
reason not to. Also removes an unused Result around the type.
Clarifies the security around deterministic nonces, removing them for
saved-to-disk cached preprocesses. The cached preprocesses are encrypted as the
DB is not a proper secret store.
Moves arguments always present in the protocol from function arguments into the
struct itself.
Removes the horribly ugly code in DkgRemoval, fixing multiple issues present
with it which would cause it to fail on use.
* Set SeraiBlockNumber in cosign.rs as it's used by the cosigning protocol
* Remove unnecessary Clone from lambdas in coordinator
* Remove the EventDb from Tributary scanner
We used per-Transaction DB TXNs so on error, we don't have to rescan the entire
block yet only the rest of it. We prevented scanning multiple transactions by
tracking which we already had.
This is over-engineered and not worth it.
* Implement borsh for HasEvents, removing the manual encoding
* Merge DkgConfirmer and DkgRemoval into signing_protocol.rs
Fixes a bug in DkgConfirmer which would cause it to improperly handle indexes
if any validator had multiple key shares.
* Strictly type DataSpecification's Label
* Correct threshold_i_map_to_keys_and_musig_i_map
It didn't include the participant's own index and accordingly was offset.
* Create TributaryBlockHandler
This struct contains all variables prior passed to handle_block and stops them
from being passed around again and again.
This also ensures fatal_slash is only called while handling a block, as needed
as it expects to operate under perfect consensus.
* Inline accumulate, store confirmation nonces with shares
Inlining accumulate makes sense due to the amount of data accumulate needed to
be passed.
Storing confirmation nonces with shares ensures that both are available or
neither. Prior, one could be yet the other may not have been (requiring an
assert in runtime to ensure we didn't bungle it somehow).
* Create helper functions for handling DkgRemoval/SubstrateSign/Sign Tributary TXs
* Move Label into SignData
All of our transactions which use SignData end up with the same common usage
pattern for Label, justifying this.
Removes 3 transactions, explicitly de-duplicating their handlers.
* Remove CurrentlyCompletingKeyPair for the non-contextual DkgKeyPair
* Remove the manual read/write for TributarySpec for borsh
This struct doesn't have any optimizations booned by the manual impl. Using
borsh reduces our scope.
* Use temporary variables to further minimize LoC in tributary handler
* Remove usage of tuples for non-trivial Tributary transactions
* Remove serde from dkg
serde could be used to deserialize intenrally inconsistent objects which could
lead to panics or faults.
The BorshDeserialize derives have been replaced with a manual implementation
which won't produce inconsistent objects.
* Abstract Future generics using new trait definitions in coordinator
* Move published_signed_transaction to tributary/mod.rs to reduce the size of main.rs
* Split coordinator/src/tributary/mod.rs into spec.rs and transaction.rs
* Make it clear not providing a change address is fingerprintable
When no change address is provided, all change is shunted to the
fee. This PR makes it clear to the caller that it is fingerprintable
when the caller does this.
* Review comments
They're a bit more binding, smaller, provided by the Rust bitcoin library,
sane, and we don't have to worry about malleability since all of our inputs are
SegWit.
* Use redb and in Dockerfiles
The motivation for redb was to remove the multiple rocksdb compile times from
CI.
* Correct feature flagging of coordinator and message-queue in Dockerfiles
* Correct message-queue DB type alias
* Use consistent table typing in redb
* Correct rebase artifacts
* Correct removal of binaries feature from message-queue
* Correct processor feature flagging
* Replace redb with parity-db
It still has much better compile times yet doesn't block when creating multiple
transactions. It also is actively maintained and doesn't grow our tree. The MPT
aspects are irrelevant.
* Correct stray Redb
* clippy warning
* Correct txn get
* Use debug builds in our Dockerfiles to reduce CI times
Also enables only spawning the mdns service when debug in the coordinator.
* Correct underflow in processor
Prior undetected due to relase builds not having bounds checks enabled.
* Restore Serai release due to CI/RPC failures caused by compiling it in debug mode
This is *probably* worth an issue filed upstream, if it can be tracked down.
* Correct failing debug asserts in Monero
These debug asserts assumed there was a change address to take the remainder.
If there's no change address, the remainder is shunted to the fee, causing the
fee to be distinct from the estimate.
We presumably need to modify monero-serai such that change: None isn't valid,
and users must use Change::Fingerprintable(None).
* Remove NetworkId from processor-messages
Because intent binds to the sender/receiver, it's not needed for intent.
The processor knows what the network is.
The coordinator knows which to use because it's sending this message to the
processor for that network.
Also removes the unused zeroize.
* ProcessorMessage::Completed use Session instead of key
* Move SubstrateSignId to Session
* Finish replacing key with session
Monero doesn't assert the time increases with each block, solely that it
doesn't decrease. Now, the block number is added to the time to ensure it
increases.
processor isn't intended to be used as a library, yet serai-processor-tests
does pull it in as a lib. This caused serai-processor-tests to need to compile
rocksdb, which added multiple minutes to the compilation time.
* Add SignalsConfig to chain_spec
* Correct multiexp feature flagging for rand_core std
* Remove bincode for borsh
Replaces a non-canonical encoding with a canonical encoding which additionally
should be faster.
Also fixes an issue where we used bincode in transcripts where it cannot be
trusted.
This ended up fixing a myriad of other bugs observed, unfortunately.
Accordingly, it either has to be merged or the bug fixes from it must be ported
to a new PR.
* Make serde optional, minimize usage
* Make borsh an optional dependency of substrate/ crates
* Remove unused dependencies
* Use [u8; 64] where possible in the processor messages
* Correct borsh feature flagging
* implement db macro for processor/substrate_signer
* Use ()
* Correct AttemptDb usage of ()
* () -> &()
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* Add a function to deterministically decide which Serai blocks should be co-signed
Has a 5 minute latency between co-signs, also used as the maximal latency
before a co-sign is started.
* Get all active tributaries we're in at a specific block
* Add and route CosignSubstrateBlock, a new provided TX
* Split queued cosigns per network
* Rename BatchSignId to SubstrateSignId
* Add SubstrateSignableId, a meta-type for either Batch or Block, and modularize around it
* Handle the CosignSubstrateBlock provided TX
* Revert substrate_signer.rs to develop (and patch to still work)
Due to SubstrateSigner moving when the prior multisig closes, yet cosigning
occurring with the most recent key, a single SubstrateSigner can be reused.
We could manage multiple SubstrateSigners, yet considering the much lower
specifications for cosigning, I'd rather treat it distinctly.
* Route cosigning through the processor
* Add note to rename SubstrateSigner post-PR
I don't want to do so now in order to preserve the diff's clarity.
* Implement cosign evaluation into the coordinator
* Get tests to compile
* Bug fixes, mark blocks without cosigners available as cosigned
* Correct the ID Batch preprocesses are saved under, add log statements
* Create a dedicated function to handle cosigns
* Correct the flow around Batch verification/queueing
Verifying `Batch`s could stall when a `Batch` was signed before its
predecessors/before the block it's contained in was cosigned (the latter being
inevitable as we can't sign a block containing a signed batch before signing
the batch).
Now, Batch verification happens on a distinct async task in order to not block
the handling of processor messages. This task is the sole caller of verify in
order to ensure last_verified_batch isn't unexpectedly mutated.
When the processor message handler needs to access it, or needs to queue a
Batch, it associates the DB TXN with a lock preventing the other task from
doing so.
This lock, as currently implemented, is a poor and inefficient design. It
should be modified to the pattern used for cosign management. Additionally, a
new primitive of a DB-backed channel may be immensely valuable.
Fixes a standing potential deadlock and a deadlock introduced with the
cosigning protocol.
* Working full-stack tests
After the last commit, this only required extending a timeout.
* Replace "co-sign" with "cosign" to make finding text easier
* Update the coordinator tests to support cosigning
* Inline prior_batch calculation to prevent panic on rotation
Noticed when doing a final review of the branch.
* Have processor report errors during the DKG to the coordinator
* Add RemoveParticipant, InvalidDkgShare to coordinator
* Route DKG blame around coordinator
* Allow public construction of AdditionalBlameMachine
Necessary for upcoming work on handling DKG blame in the processor and
coordinator.
Additionally fixes a publicly reachable panic when commitments parsed with one
ThresholdParams are used in a machine using another set of ThresholdParams.
Renames InvalidProofOfKnowledge to InvalidCommitments.
* Remove unused error from dleq
* Implement support for VerifyBlame in the processor
* Have coordinator send the processor share message relevant to Blame
* Remove desync between processors reporting InvalidShare and ones reporting GeneratedKeyPair
* Route blame on sign between processor and coordinator
Doesn't yet act on it in coordinator.
* Move txn usage as needed for stable Rust to build
* Correct InvalidDkgShare serialization
If a user transferred in without an InInstruction, and the amount exactly
matched a forwarded output, the user's output would fulfill the
forwarding. Then the forwarded output would come along, have no InInstruction,
and be refunded (to the prior multisig) when the user should've been refunded.
Adding this new address type resolves such concerns.
The higher-level scanner code in multisigs/mod.rs now creates a series of plans
with limited context. These include forwarding and refunding plans, moving all
handling of forwarding flags on the scanner's clock and therefore safe.
Also simplifies the refunding a decent bit.
This code is still largely designed around the idea a payment for a network is
fungible with any other, which isn't true. This starts moving past that.
Asserts are added to ensure the integrity of coin to the scheduler (which is
now per key per coin, not per key alone) and in Bitcoin/Monero prepare_send.
The prior system spawned a new connection per request to enable parallelism,
yet kept hitting hyper::IncompleteMessages I couldn't track down. This
attempts to resolve those by a long-lived socket.
Halves the amount of requests per-authenticated RPC call, and accordingly is
likely still better overall.
I don't believe this is resolved yet but this is still worth pushing.
Removes bitcoin-serai's usage of sha2 for bitcoin-hashes. While sha2 is still
in play due to modular-frost (more specifically, due to ciphersuite), this
offers a bit more performance (assuming equivalency between sha2 and
bitcoin-hashes' impl) due to removing a static for a const.
Makes secp256k1 a dev dependency for bitcoin-serai. While secp256k1 is still
pulled in via bitcoin, it's hopefully slightly better to compile now and makes
usage of secp256k1 an implementation detail of bitcoin (letting it change it
freely).
Also offers slightly more efficient signing as we don't decode to a signature
just to re-encode for the transaction.
Removes a 20s sleep for a check every second, up to 20 times, for reduced test
times in the processor.
* Move pallet-asset-conversion
* update licensing
* initial integration
* Integrate Currency & Assets types
* integrate liquidity tokens
* fmt
* integrate dex pallet tests
* fmt
* compilation error fixes
* integrate dex benchmarks
* fmt
* cargo clippy
* replace all occurrences of "asset" with "coin"
* add the actual add liq/swap logic to in-instructions
* add client side & tests
* fix deny
* Lint and changes
- Renames InInstruction::AddLiquidity to InInstruction::SwapAndAddLiquidity
- Makes create_pool an internal function
- Makes dex-pallet exclusively create pools against a native coin
- Removes various fees
- Adds new crates to GH workflow
* Fix rebase artifacts
* Correct other rebase artifact
* Correct CI specification for liquidity-tokens
* Correct primitives' test to the standardized pallet account scheme
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* db_macro
* wip: converted prcessor/key_gen to use create_db macro
* wip: converted prcessor/key_gen to use create_db macro
* wip: formatting
* fix: added no_run to doc
* fix: documentation example had extra parenths
* fix: ignore doc test entirely
* Corrections from rebasing
* Misc lint
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* Update the coordinator to give key shares based on weight, not based on existence
Participants are now identified by their starting index. While this compiles,
the following is unimplemented:
1) A conversion for DKG `i` values. It assumes the threshold `i` values used
will be identical for the MuSig signature used to confirm the DKG.
2) Expansion from compressed values to full values before forwarding to the
processor.
* Add a fn to the DkgConfirmer to convert `i` values as needed
Also removes TODOs regarding Serai ensuring validator key uniqueness +
validity. The current infra achieves both.
* Have the Tributary DB track participation by shares, not by count
* Prevent a node from obtaining 34% of the maximum amount of key shares
This is actually mainly intended to set a bound on message sizes in the
coordinator. Message sizes are amplified by the amount of key shares held, so
setting an upper bound on said amount lets it determine constants. While that
upper bound could be 150, that'd be unreasonable and increase the potential for
DoS attacks.
* Correct the mechanism to detect if sufficient accumulation has occured
It used to check if the latest accumulation hit the required threshold. Now,
accumulations may jump past the required threshold. The required mechanism is
to check the threshold wasn't prior met and is now met.
* Finish updating the coordinator to handle a multiple key share per validator environment
* Adjust stategy re: preventing noce reuse in DKG Confirmer
* Add TODOs regarding dropped transactions, add possible TODO fix
* Update tests/coordinator
This doesn't add new multi-key-share tests, it solely updates the existing
single key-share tests to compile and run, with the necessary fixes to the
coordinator.
* Update processor key_gen to handle generating multiple key shares at once
* Update SubstrateSigner
* Update signer, clippy
* Update processor tests
* Update processor docker tests
If a crate has std set, it should enable std for all dependencies in order to
let them properly select which algorithms to use. Some crates fallback to
slower/worse algorithms on no-std.
Also more aggressively sets default-features = false leading to a *10%*
reduction in the amount of crates coordinator builds.
Even though the intent was to test against 0.17.3.2, and a Monero 0.17.3.2 node
was running, the processor now uses docker which will always use 0.18.
Accordingly, while the intent was valid, it was pointless.
This is unfortunate, as testing against 0.17 helped protect against edge cases.
The infra to preserve their tests isn't worth the benefit we'd gain from said
tests however.
The lack of locking the connection when making an authenticated request, which
is actually two sequential requests, risked another caller making a request in
between, invalidating the state.
Now, only unauthenticated connections share a connection object.
Disables the unused zmq RPC.
Removes authentication which seems to be unstable as hell when under load
(see #351).
No longer use Network::Isolated as it's not needed here (the Monero nodes run
with `--offline`).
Also halves the minimum fee policy, which still may be 2x-4x higher than
necessary due to API limitations within bitcoin-serai (which we can fix as it's
within our scope).
Monero would select decoys with a new RNG seed, which may have used more bytes,
increasing the fee.
There's a few comments here.
1) Non-determinism wasn't removed via distinguishing the edits. It was done by
removing part of the transcript. A TODO exists to improve this.
2) Distinct TX fees is a test failure, not an issue in prod *unless* the distinct
fee is greater. So long as the distinct fee is lesser, it's fine.
3) Removing outputs is expected to only decrease fees.
The existing code should've mostly handled this fine. Only a single edge case
(TX fee reduction on no-change Plans) would cause an improper increase in
operating costs.
* initial implementation
* add function to get a balance of an account
* add support for multiple coins
* rename pallet to "coins-pallet"
* replace balances, assets and tokens pallet with coins pallet in runtime
* add total supply info
* update client side for new Coins pallet
* handle fees
* bug fixes
* Update FeeAccount test
* Fmt
* fix pr comments
* remove extraneous Imbalance type
* Minor tweaks
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
Implements most of #297 to the point I'm fine closing it. The solution
implemented is distinct than originally designed, yet much simpler.
Since we have a fully-linear view of created transactions, we don't have to
per-output track operating costs incurred by that output. We can track it
across the entire Serai system, without hooking into the Eventuality system.
Also updates documentation.
