Commit graph

417 commits

Author SHA1 Message Date
Luke Parker
0d5756ffcf
cargo update, upgrade alloy
Some checks failed
Coordinator Tests / build (push) Waiting to run
Full Stack Tests / build (push) Waiting to run
Lint / clippy (macos-13) (push) Waiting to run
Lint / clippy (macos-14) (push) Waiting to run
Lint / clippy (ubuntu-latest) (push) Waiting to run
Lint / clippy (windows-latest) (push) Waiting to run
Lint / deny (push) Waiting to run
Lint / fmt (push) Waiting to run
Lint / machete (push) Waiting to run
Message Queue Tests / build (push) Waiting to run
Monero Tests / unit-tests (push) Waiting to run
Monero Tests / integration-tests (v0.17.3.2) (push) Waiting to run
Monero Tests / integration-tests (v0.18.2.0) (push) Waiting to run
networks/ Tests / test-networks (push) Waiting to run
no-std build / build (push) Waiting to run
Processor Tests / build (push) Waiting to run
Reproducible Runtime / build (push) Waiting to run
Tests / test-infra (push) Waiting to run
Tests / test-substrate (push) Waiting to run
Tests / test-serai-client (push) Waiting to run
common/ Tests / test-common (push) Has been cancelled
crypto/ Tests / test-crypto (push) Has been cancelled
Removes a dated proc-macro-crate patch.
2024-09-05 17:03:23 -04:00
Luke Parker
ac7b98daac
Remove tokio dependency from tendermint-machine
Indirects it via a minimal wrapper which can be trivially patched.
2024-09-05 16:30:27 -04:00
akildemir
a506d74d69
move economic security into it's own pallet (#596)
Some checks are pending
Full Stack Tests / build (push) Waiting to run
Lint / machete (push) Waiting to run
Lint / clippy (macos-14) (push) Waiting to run
Lint / clippy (ubuntu-latest) (push) Waiting to run
Lint / clippy (windows-latest) (push) Waiting to run
Lint / deny (push) Waiting to run
Lint / clippy (macos-13) (push) Waiting to run
Lint / fmt (push) Waiting to run
Reproducible Runtime / build (push) Waiting to run
Tests / test-infra (push) Waiting to run
Tests / test-substrate (push) Waiting to run
Tests / test-serai-client (push) Waiting to run
* move economic security into it's own pallet

* fix deny

* Update Cargo.toml, .github for the new crates

* Remove unused import

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2024-08-31 18:55:42 -04:00
akildemir
cccc1fc7e6
Implement block emissions (#551)
Some checks are pending
Full Stack Tests / build (push) Waiting to run
Lint / fmt (push) Waiting to run
Lint / machete (push) Waiting to run
Lint / clippy (macos-13) (push) Waiting to run
Lint / clippy (macos-14) (push) Waiting to run
Lint / clippy (ubuntu-latest) (push) Waiting to run
Lint / clippy (windows-latest) (push) Waiting to run
Lint / deny (push) Waiting to run
Reproducible Runtime / build (push) Waiting to run
Tests / test-infra (push) Waiting to run
Tests / test-substrate (push) Waiting to run
Tests / test-serai-client (push) Waiting to run
* add genesis liquidity implementation

* add missing deposit event

* fix CI issues

* minor fixes

* make math safer

* fix fmt

* implement block emissions

* make remove liquidity an authorized call

* implement setting initial values for coins

* add genesis liquidity test & misc fixes

* updato develop latest

* fix rotation test

* fix licencing

* add fast-epoch feature

* only create the pool when adding liquidity first time

* add initial reward era test

* test whole pre ec security emissions

* fix clippy

* add swap-to-staked-sri feature

* rebase changes

* fix tests

* Remove accidentally commited ETH ABI files

* fix some pr comments

* Finish up fixing pr comments

* exclude SRI from is_allowed check

* Misc changes

---------

Co-authored-by: akildemir <aeg_asd@hotmail.com>
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2024-08-14 23:12:04 -04:00
Luke Parker
d5205ce231
Update dependencies
Some checks failed
Full Stack Tests / build (push) Waiting to run
Lint / clippy (windows-latest) (push) Waiting to run
Lint / deny (push) Waiting to run
Lint / fmt (push) Waiting to run
Lint / machete (push) Waiting to run
Lint / clippy (macos-13) (push) Waiting to run
Lint / clippy (macos-14) (push) Waiting to run
Lint / clippy (ubuntu-latest) (push) Waiting to run
Reproducible Runtime / build (push) Waiting to run
Tests / test-infra (push) Waiting to run
Tests / test-substrate (push) Waiting to run
Tests / test-serai-client (push) Waiting to run
Monero Tests / unit-tests (push) Has been cancelled
Coordinator Tests / build (push) Has been cancelled
Monero Tests / integration-tests (v0.17.3.2) (push) Has been cancelled
Monero Tests / integration-tests (v0.18.2.0) (push) Has been cancelled
networks/ Tests / test-networks (push) Has been cancelled
no-std build / build (push) Has been cancelled
Processor Tests / build (push) Has been cancelled
Resolves a yanked version of bytemuck.
2024-08-01 04:06:09 -04:00
Luke Parker
0f6878567f
Remove a pair of unused structs/deps
Caught by the most recent nightly.
2024-08-01 01:36:10 -04:00
akildemir
1493f49416
Implement genesis liquidity protocol (#545)
Some checks failed
Full Stack Tests / build (push) Waiting to run
Lint / fmt (push) Waiting to run
Lint / machete (push) Waiting to run
Lint / clippy (macos-13) (push) Waiting to run
Lint / clippy (macos-14) (push) Waiting to run
Lint / clippy (ubuntu-latest) (push) Waiting to run
Lint / clippy (windows-latest) (push) Waiting to run
Lint / deny (push) Waiting to run
Reproducible Runtime / build (push) Has been cancelled
Tests / test-infra (push) Has been cancelled
Tests / test-substrate (push) Has been cancelled
Tests / test-serai-client (push) Has been cancelled
* add genesis liquidity implementation

* add missing deposit event

* fix CI issues

* minor fixes

* make math safer

* fix fmt

* make remove liquidity an authorized call

* implement setting initial values for coins

* add genesis liquidity test & misc fixes

* updato develop latest

* fix rotation test

* Finish merging develop

* Remove accidentally committed ETH files

* fix pr comments

* further bug fixes

* fix last pr comments

* tidy up

* Misc

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2024-07-18 19:30:19 -04:00
Luke Parker
d9f85fab26
Update lockfiles
Resolves a dependabot alert about the Ruby used to generate the docs site.
2024-07-18 15:18:08 -04:00
akildemir
40cc180853 add transaction and crypto unit tests
Some checks failed
Tests / test-substrate (push) Waiting to run
Coordinator Tests / build (push) Waiting to run
Full Stack Tests / build (push) Waiting to run
Lint / clippy (macos-13) (push) Waiting to run
Lint / clippy (macos-14) (push) Waiting to run
Lint / clippy (ubuntu-latest) (push) Waiting to run
Lint / clippy (windows-latest) (push) Waiting to run
Lint / deny (push) Waiting to run
Lint / fmt (push) Waiting to run
Lint / machete (push) Waiting to run
Monero Tests / unit-tests (push) Waiting to run
Monero Tests / integration-tests (v0.17.3.2) (push) Waiting to run
Monero Tests / integration-tests (v0.18.2.0) (push) Waiting to run
no-std build / build (push) Waiting to run
Processor Tests / build (push) Waiting to run
Reproducible Runtime / build (push) Waiting to run
Tests / test-infra (push) Waiting to run
Tests / test-serai-client (push) Waiting to run
coins/ Tests / test-coins (push) Has been cancelled
2024-07-17 16:26:31 -07:00
Luke Parker
0cb24dde02
cargo update
Resolves failing deny.
2024-07-14 20:27:36 -04:00
Luke Parker
c59be46e2f
Optimize Monero BPs 2024-07-12 02:18:57 -04:00
Luke Parker
d7f7f69738
Remove the DecoySelection trait 2024-07-08 00:30:42 -04:00
Luke Parker
a2c3aba82b
Clean the Monero lib for auditing (#577)
Some checks failed
coins/ Tests / test-coins (push) Waiting to run
Coordinator Tests / build (push) Waiting to run
Full Stack Tests / build (push) Waiting to run
Lint / clippy (macos-13) (push) Waiting to run
Lint / clippy (macos-14) (push) Waiting to run
Lint / clippy (ubuntu-latest) (push) Waiting to run
Lint / clippy (windows-latest) (push) Waiting to run
Lint / deny (push) Waiting to run
Lint / fmt (push) Waiting to run
Lint / machete (push) Waiting to run
Monero Tests / unit-tests (push) Waiting to run
Monero Tests / integration-tests (v0.17.3.2) (push) Waiting to run
Monero Tests / integration-tests (v0.18.2.0) (push) Waiting to run
no-std build / build (push) Waiting to run
Processor Tests / build (push) Waiting to run
Reproducible Runtime / build (push) Waiting to run
Tests / test-infra (push) Waiting to run
Tests / test-substrate (push) Waiting to run
Tests / test-serai-client (push) Waiting to run
common/ Tests / test-common (push) Has been cancelled
crypto/ Tests / test-crypto (push) Has been cancelled
Message Queue Tests / build (push) Has been cancelled
* Remove unsafe creation of dalek_ff_group::EdwardsPoint in BP+

* Rename Bulletproofs to Bulletproof, since they are a single Bulletproof

Also bifurcates prove with prove_plus, and adds a few documentation items.

* Make CLSAG signing private

Also adds a bit more documentation and does a bit more tidying.

* Remove the distribution cache

It's a notable bandwidth/performance improvement, yet it's not ready. We need a
dedicated Distribution struct which is managed by the wallet and passed in.
While we can do that now, it's not currently worth the effort.

* Tidy Borromean/MLSAG a tad

* Remove experimental feature from monero-serai

* Move amount_decryption into EncryptedAmount::decrypt

* Various RingCT doc comments

* Begin crate smashing

* Further documentation, start shoring up API boundaries of existing crates

* Document and clean clsag

* Add a dedicated send/recv CLSAG mask struct

Abstracts the types used internally.

Also moves the tests from monero-serai to monero-clsag.

* Smash out monero-bulletproofs

Removes usage of dalek-ff-group/multiexp for curve25519-dalek.

Makes compiling in the generators an optional feature.

Adds a structured batch verifier which should be notably more performant.

Documentation and clean up still necessary.

* Correct no-std builds for monero-clsag and monero-bulletproofs

* Tidy and document monero-bulletproofs

I still don't like the impl of the original Bulletproofs...

* Error if missing documentation

* Smash out MLSAG

* Smash out Borromean

* Tidy up monero-serai as a meta crate

* Smash out RPC, wallet

* Document the RPC

* Improve docs a bit

* Move Protocol to monero-wallet

* Incomplete work on using Option to remove panic cases

* Finish documenting monero-serai

* Remove TODO on reading pseudo_outs for AggregateMlsagBorromean

* Only read transactions with one Input::Gen or all Input::ToKey

Also adds a helper to fetch a transaction's prefix.

* Smash out polyseed

* Smash out seed

* Get the repo to compile again

* Smash out Monero addresses

* Document cargo features

Credit to @hinto-janai for adding such sections to their work on documenting
monero-serai in #568.

* Fix deserializing v2 miner transactions

* Rewrite monero-wallet's send code

I have yet to redo the multisig code and the builder. This should be much
cleaner, albeit slower due to redoing work.

This compiles with clippy --all-features. I have to finish the multisig/builder
for --all-targets to work (and start updating the rest of Serai).

* Add SignableTransaction Read/Write

* Restore Monero multisig TX code

* Correct invalid RPC type def in monero-rpc

* Update monero-wallet tests to compile

Some are _consistently_ failing due to the inputs we attempt to spend being too
young. I'm unsure what's up with that. Most seem to pass _consistently_,
implying it's not a random issue yet some configuration/env aspect.

* Clean and document monero-address

* Sync rest of repo with monero-serai changes

* Represent height/block number as a u32

* Diversify ViewPair/Scanner into ViewPair/GuaranteedViewPair and Scanner/GuaranteedScanner

Also cleans the Scanner impl.

* Remove non-small-order view key bound

Guaranteed addresses are in fact guaranteed even with this due to prefixing key
images causing zeroing the ECDH to not zero the shared key.

* Finish documenting monero-serai

* Correct imports for no-std

* Remove possible panic in monero-serai on systems < 32 bits

This was done by requiring the system's usize can represent a certain number.

* Restore the reserialize chain binary

* fmt, machete, GH CI

* Correct misc TODOs in monero-serai

* Have Monero test runner evaluate an Eventuality for all signed TXs

* Fix a pair of bugs in the decoy tests

Unfortunately, this test is still failing.

* Fix remaining bugs in monero-wallet tests

* Reject torsioned spend keys to ensure we can spend the outputs we scan

* Tidy inlined epee code in the RPC

* Correct the accidental swap of stagenet/testnet address bytes

* Remove unused dep from processor

* Handle Monero fee logic properly in the processor

* Document v2 TX/RCT output relation assumed when scanning

* Adjust how we mine the initial blocks due to some CI test failures

* Fix weight estimation for RctType::ClsagBulletproof TXs

* Again increase the amount of blocks we mine prior to running tests

* Correct the if check about when to mine blocks on start

Finally fixes the lack of decoy candidates failures in CI.

* Run Monero on Debian, even for internal testnets

Change made due to a segfault incurred when locally testing.

https://github.com/monero-project/monero/issues/9141 for the upstream.

* Don't attempt running tests on the verify-chain binary

Adds a minimum XMR fee to the processor and runs fmt.

* Increase minimum Monero fee in processor

I'm truly unsure why this is required right now.

* Distinguish fee from necessary_fee in monero-wallet

If there's no change, the fee is difference of the inputs to the outputs. The
prior code wouldn't check that amount is greater than or equal to the necessary
fee, and returning the would-be change amount as the fee isn't necessarily
helpful.

Now the fee is validated in such cases and the necessary fee is returned,
enabling operating off of that.

* Restore minimum Monero fee from develop
2024-07-07 06:57:18 -04:00
Luke Parker
8ab6f9c36e
alloy 0.1
Some checks failed
Lint / clippy (macos-14) (push) Has been cancelled
Lint / clippy (ubuntu-latest) (push) Has been cancelled
coins/ Tests / test-coins (push) Has been cancelled
Coordinator Tests / build (push) Has been cancelled
Full Stack Tests / build (push) Has been cancelled
Lint / clippy (macos-13) (push) Has been cancelled
Lint / clippy (windows-latest) (push) Has been cancelled
Lint / deny (push) Has been cancelled
Lint / fmt (push) Has been cancelled
Lint / machete (push) Has been cancelled
no-std build / build (push) Has been cancelled
Processor Tests / build (push) Has been cancelled
Reproducible Runtime / build (push) Has been cancelled
Tests / test-infra (push) Has been cancelled
Tests / test-substrate (push) Has been cancelled
Tests / test-serai-client (push) Has been cancelled
2024-06-19 12:39:47 -04:00
Luke Parker
03445b3020
Update httparse, as 1.9.2 was yanked 2024-06-13 16:49:58 -04:00
Luke Parker
9af111b4aa
Rust 1.79, cargo update 2024-06-13 15:57:08 -04:00
Luke Parker
41ce5b1738 Use the serai_abi::Call in the actual Transaction type
We prior required they had the same encoding, yet this ensures they do by
making them one and the same. This does require an large, ugly, From/TryInto
block which is deemed preferable for moving this more and more into syntax
(from semantics).

Further improvements (notably re: Extra) is possible, and this already lets us
strip some members from the Call enum.
2024-06-03 23:38:22 -04:00
Luke Parker
2a05cf3225
June 2024 nightly update
Replaces #571.
2024-06-01 21:46:49 -04:00
Luke Parker
f4147c39b2
bitcoin 0.32.1 2024-05-31 01:02:43 -04:00
rlking
cd69f3b9d6
Check if wasm was built by container exit code and state instead of local mountpoint (#570)
* Check if the serai wasm was built successfully by verifying the build container's status code and state, instead of checking the volume mountpoint locally

* Use a log statement for which wasm is used

* Minor typo fix

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2024-05-25 20:33:23 -04:00
Luke Parker
1d2beb3ee4
Ethereum relayer server
Causes send test to pass for the processor.
2024-05-22 18:50:11 -04:00
Luke Parker
a473800c26
More aggressive cargo update
Adds a few deps which are fine. Patches an old parking_lot(_core) version.
2024-05-21 08:07:32 -04:00
Luke Parker
400319cd29
cargo update
Also updates our gems
2024-05-21 06:09:04 -04:00
Luke Parker
a0a7d63dad
bitcoin 0.32 2024-05-21 05:27:01 -04:00
Luke Parker
11ec9e3535
Ethereum processor docker tests, barring send
We need the TX publication relay thingy for send to work (though that is the
point the test fails at).
2024-05-21 00:29:33 -04:00
Luke Parker
ae8a27b876
Add our own alloy meta module to deduplicate alloy prefixes 2024-05-14 01:42:18 -04:00
Luke Parker
79a79db399
Update dockertest specification 2024-05-10 15:50:07 -04:00
Luke Parker
5501de1f3a
Update to the latest alloy
Also makes various tweaks as necessary.
2024-05-10 14:06:43 -04:00
Luke Parker
95591218bb
Remove cbor 2024-04-23 07:01:07 -04:00
Luke Parker
8cef9eff6f
Move keep alive, heartbeat, block to request/response 2024-04-23 05:44:58 -04:00
Luke Parker
a25e6330bd
Remove DLEq proofs from CLSAG multisig
1) Removes the key image DLEq on the Monero side of things, as the produced
   signature share serves as a DLEq for it.
2) Removes the nonce DLEqs from modular-frost as they're unnecessary for
   monero-serai. Updates documentation accordingly.

Without the proof the nonces are internally consistent, the produced signatures
from modular-frost can be argued as a batch-verifiable CP93 DLEq (R0, R1, s),
or as a GSP for the CP93 DLEq statement (which naturally produces (R0, R1, s)).

The lack of proving the nonces consistent does make the process weaker, yet
it's also unnecessary for the class of protocols this is intended to service.
To provide DLEqs for the nonces would be to provide PoKs for the nonce
commitments (in the traditional Schnorr case).
2024-04-21 23:01:32 -04:00
Luke Parker
0f0db14f05
Ethereum Integration (#557)
* Clean up Ethereum

* Consistent contract address for deployed contracts

* Flesh out Router a bit

* Add a Deployer for DoS-less deployment

* Implement Router-finding

* Use CREATE2 helper present in ethers

* Move from CREATE2 to CREATE

Bit more streamlined for our use case.

* Document ethereum-serai

* Tidy tests a bit

* Test updateSeraiKey

* Use encodePacked for updateSeraiKey

* Take in the block hash to read state during

* Add a Sandbox contract to the Ethereum integration

* Add retrieval of transfers from Ethereum

* Add inInstruction function to the Router

* Augment our handling of InInstructions events with a check the transfer event also exists

* Have the Deployer error upon failed deployments

* Add --via-ir

* Make get_transaction test-only

We only used it to get transactions to confirm the resolution of Eventualities.
Eventualities need to be modularized. By introducing the dedicated
confirm_completion function, we remove the need for a non-test get_transaction
AND begin this modularization (by no longer explicitly grabbing a transaction
to check with).

* Modularize Eventuality

Almost fully-deprecates the Transaction trait for Completion. Replaces
Transaction ID with Claim.

* Modularize the Scheduler behind a trait

* Add an extremely basic account Scheduler

* Add nonce uses, key rotation to the account scheduler

* Only report the account Scheduler empty after transferring keys

Also ban payments to the branch/change/forward addresses.

* Make fns reliant on state test-only

* Start of an Ethereum integration for the processor

* Add a session to the Router to prevent updateSeraiKey replaying

This would only happen if an old key was rotated to again, which would require
n-of-n collusion (already ridiculous and a valid fault attributable event). It
just clarifies the formal arguments.

* Add a RouterCommand + SignMachine for producing it to coins/ethereum

* Ethereum which compiles

* Have branch/change/forward return an option

Also defines a UtxoNetwork extension trait for MAX_INPUTS.

* Make external_address exclusively a test fn

* Move the "account" scheduler to "smart contract"

* Remove ABI artifact

* Move refund/forward Plan creation into the Processor

We create forward Plans in the scan path, and need to know their exact fees in
the scan path. This requires adding a somewhat wonky shim_forward_plan method
so we can obtain a Plan equivalent to the actual forward Plan for fee reasons,
yet don't expect it to be the actual forward Plan (which may be distinct if
the Plan pulls from the global state, such as with a nonce).

Also properly types a Scheduler addendum such that the SC scheduler isn't
cramming the nonce to use into the N::Output type.

* Flesh out the Ethereum integration more

* Two commits ago, into the **Scheduler, not Processor

* Remove misc TODOs in SC Scheduler

* Add constructor to RouterCommandMachine

* RouterCommand read, pairing with the prior added write

* Further add serialization methods

* Have the Router's key included with the InInstruction

This does not use the key at the time of the event. This uses the key at the
end of the block for the event. Its much simpler than getting the full event
streams for each, checking when they interlace.

This does not read the state. Every block, this makes a request for every
single key update and simply chooses the last one. This allows pruning state,
only keeping the event tree. Ideally, we'd also introduce a cache to reduce the
cost of the filter (small in events yielded, long in blocks searched).

Since Serai doesn't have any forwarding TXs, nor Branches, nor change, all of
our Plans should solely have payments out, and there's no expectation of a Plan
being made under one key broken by it being received by another key.

* Add read/write to InInstruction

* Abstract the ABI for Call/OutInstruction in ethereum-serai

* Fill out signable_transaction for Ethereum

* Move ethereum-serai to alloy

Resolves #331.

* Use the opaque sol macro instead of generated files

* Move the processor over to the now-alloy-based ethereum-serai

* Use the ecrecover provided by alloy

* Have the SC use nonce for rotation, not session (an independent nonce which wasn't synchronized)

* Always use the latest keys for SC scheduled plans

* get_eventuality_completions for Ethereum

* Finish fleshing out the processor Ethereum integration as needed for serai-processor tests

This doesn't not support any actual deployments, not even the ones simulated by
serai-processor-docker-tests.

* Add alloy-simple-request-transport to the GH workflows

* cargo update

* Clarify a few comments and make one check more robust

* Use a string for 27.0 in .github

* Remove optional from no-longer-optional dependencies in processor

* Add alloy to git deny exception

* Fix no longer optional specification in processor's binaries feature

* Use a version of foundry from 2024

* Correct fetching Bitcoin TXs in the processor docker tests

* Update rustls to resolve RUSTSEC warnings

* Use the monthly nightly foundry, not the deleted daily nightly
2024-04-21 06:02:12 -04:00
Luke Parker
33dd412e67
Add bootnode code prior used in testnet-internal (#554)
* Add bootnode code prior used in testnet-internal

Also performs the devnet/testnet differentation done since the testnet branch.

* Fixes

* fmt
2024-04-12 00:38:40 -04:00
Luke Parker
fcad402186
cargo update
Resolves deny error caused by h2.
2024-04-10 06:34:01 -04:00
Luke Parker
93be7a3067
Latest hyper-rustls, remove async-recursion
I didn't remove async-recursion when I updated the repo to 1.77 as I forgot we
used it in the tests. I still had to add some Box::pins, which may have been a
valid option, on the prior Rust version, yet at least resolves everything now.

Also updates everything which doesn't introduce further depends.
2024-03-27 00:17:04 -04:00
Luke Parker
bca3728a10
Randomly select an addr from the authority discovery 2024-03-23 00:09:23 -04:00
Luke Parker
1f2b9376f9
zstd 0.13 2024-03-20 21:53:57 -04:00
Luke Parker
ae0ecf9efe
Disable jemalloc for rocksdb 0.22 to fix windows builds 2024-03-09 04:26:24 -05:00
Luke Parker
68dc872b88
sync every txn 2024-03-09 03:18:52 -05:00
Luke Parker
97f433c694
Redo how WAL/logs are limited by the DB
Adds a patch to the latest rocksdb.
2024-03-09 02:20:14 -05:00
Luke Parker
0d569ff7a3
cargo update
Resolves the current deny warning.
2024-03-07 23:23:48 -05:00
Luke Parker
480acfd430
Fix machete 2024-03-07 23:00:17 -05:00
Luke Parker
e266bc2e32
Stop validators from equivocating on reboot
Part of https://github.com/serai-dex/serai/issues/345.

The lack of full DB persistence does mean enough nodes rebooting at the same
time may cause a halt. This will prevent slashes.
2024-03-07 22:56:35 -05:00
Luke Parker
b427f4b8ab
Revert "Add bootnodes"
This reverts commit 1096ddb7ea.

This commit was intended for the testnet branch alone.
2024-02-26 10:47:47 -05:00
Luke Parker
1096ddb7ea
Add bootnodes 2024-02-26 10:46:16 -05:00
Luke Parker
5487844b9e
clippy && cargo update 2024-02-25 18:37:15 -05:00
Luke Parker
6f5d794f10
Median by Position (#533)
* use median price instead of the highest sustained

* add test for lexicographically reversing a byte slice

* fix pr comments

* fix CI fail

* fix dex tests

* Use a fuzz-tested list of prices

* Working median algorithm based on position + lints

---------

Co-authored-by: akildemir <aeg_asd@hotmail.com>
2024-02-19 20:50:04 -05:00
Luke Parker
c41a8ac8f2
Revert "rocksdb 0.22 via a patch"
This reverts commit c05c511938.

rocksdb 0.22 does not work on Windows at this time.
2024-02-18 08:17:26 -05:00
Luke Parker
c05c511938
rocksdb 0.22 via a patch
patch is removeable once https://github.com/paritytech/parity-common/pull/828
is merged and released.
2024-02-18 05:32:04 -05:00
Justin Berman
df85c09435
monero: match monero's stricter check when decompressing points (#515)
* monero: match monero's stricter check when decompressing points

* Reverted type change for output key
2024-02-17 23:16:16 -05:00
Luke Parker
337e54c672
Redo Dockerfile generation (#530)
Moves from concatted Dockerfiles to pseudo-templated Dockerfiles via a dedicated Rust program.

Removes the unmaintained kubernetes, not because we shouldn't have/use it, but because it's unmaintained and needs to be reworked before it's present again.

Replaces the compose with the work in the new orchestrator binary which spawns everything as expected. While this arguably re-invents the wheel, it correctly manages secrets and handles the variadic Dockerfiles.

Also adds an unrelated patch for zstd and simplifies running services a bit by greater utilizing the existing infrastructure.

---

* Delete all Dockerfile fragments, add new orchestator to generate Dockerfiles

Enables greater templating.

Also delete the unmaintained kubernetes folder *for now*. This should be
restored in the future.

* Use Dockerfiles from the orchestator

* Ignore Dockerfiles in the git repo

* Remove CI job to check Dockerfiles are as expected now that they're no longer committed

* Remove old Dockerfiles from repo

* Use Debian for monero-wallet-rpc

* Remove replace_cmds for proper usage of entry-dev

Consolidates ports a bit.

Updates serai-docker-tests from "compose" to "build".

* Only write a new dockerfile if it's distinct

Preserves the updated time metadata.

* Update serai-docker-tests

* Correct the path Dockerfiles are built from

* Correct inclusion of orchestration folder in Docker builds

* Correct debug/release flagging in the cargo command

Apparently, --debug isn't an effective NOP yet an error.

* Correct path used to run the Serai node within a Dockerfile

* Correct path in Monero Dockerfile

* Attempt storing monerod in /usr/bin

* Use sudo to move into /usr/bin in CI

* Correct 18.3.0 to 18.3.1

* Escape * with quotes

* Update deny.toml, ADD orchestration in runtime Dockerfile

* Add --detach to the Monero GH CI

* Diversify dockerfiles by network

* Fixes to network-diversified orchestration

* Bitcoin and Monero testnet scripts

* Permissions and tweaks

* Flatten scripts folders

* Add missing folder specification to Monero Dockerfile

* Have monero-wallet-rpc specify the monerod login

* Have the Docker CMD specify env variables inserted at time of Dockerfile generation

They're overrideable with the global enviornment as for tests. This enables
variable generation in orchestrator and output to productionized Docker files
without creating a life-long file within the Docker container.

* Don't add Dockerfiles into Docker containers now that they have secrets

Solely add the source code for them as needed to satisfy the workspace bounds.

* Download arm64 Monero on arm64

* Ensure constant host architecture when reproducibly building the wasm

Host architecture, for some reason, can effect the generated code despite the
target architecture always being foreign to the host architecture.

* Randomly generate infrastructure keys

* Have orchestrator generate a key, be able to create/start containers

* Ensure bash is used over sh

* Clean dated docs

* Change how quoting occurs

* Standardize to sh

* Have Docker test build the dev Dockerfiles

* Only key_gen once

* cargo update

Adds a patch for zstd and reconciles the breaking nightly change which just
occurred.

* Use a dedicated network for Serai

Also fixes SERAI_HOSTNAME passed to coordinator.

* Support providing a key over the env for the Serai node

* Enable and document running daemons for tests via serai-orchestrator

Has running containers under the dev network port forward the RPC ports.

* Use volumes for bitcoin/monero

* Use bitcoin's run.sh in GH CI

* Only use the volume for testnet (not dev)
2024-02-09 02:48:44 -05:00
Luke Parker
af12cec3b9
cargo update
Resolves deny warning around unintended behavior change (without semver bump).
2024-02-04 03:50:48 -05:00
Luke Parker
508f7eb23a
cargo update
Pseudo-resolves shlex advisory (due to the deprecation of the vulnerable
functions, which hopefully should prevent their use). shlex is only used by
bindgen, a sufficiently trusted dependency.
2024-01-22 22:08:37 -05:00
Luke Parker
90df391170
cargo update
Resolves h2 disclosure (which shouldn't have affected us).
2024-01-19 11:44:49 -05:00
Luke Parker
9d3d47fc9f
hyper-rustls 0.26 (#519)
* hyper-rustls 0.25

This isn't worth it until our dependencies update to rustls 0.22 as well.

* hyper-rustls 0.26 and hyper 1.0
2024-01-16 19:32:30 -05:00
akildemir
6691f16292
remove mach patch 2024-01-16 12:06:50 -05:00
Luke Parker
1cff9b4264
Patch proc-macro-crate 2 to proc-macro-crate 3
Updates toml_edit to 0.21.
2024-01-05 23:40:15 -05:00
Luke Parker
617ec604ee
cargo update
Resolves the deny CI failure.
2024-01-04 01:46:26 -05:00
Luke Parker
7eb388e546
PR to track down CI failures (#501)
* Use an extended timeout for DKGs specifically

* Add a log statement when message-queue connection fails

* Add a 60 second keep-alive to connections

* Use zalloc for processor/message-queue/coordinator

An additional layer which protects us against edge cases with Zeroizing
(objects which don't support it or don't miss it).

* Add further logs to message-queue

* Further increase re-attempt timeouts in CI

* Remove misplaced continue inmessage-queue client

Fixes observed CI failures.

* Revert "Further increase re-attempt timeouts in CI"

This reverts commit 3723530cf6.
2024-01-04 01:08:13 -05:00
Luke Parker
890b70212a
Patch matches, mach 2023-12-30 02:52:05 -05:00
Luke Parker
9f7140c3db
Patch is-terminal to the std-included IsTerminal 2023-12-30 02:48:26 -05:00
Luke Parker
8b26a85faa
Add patches for directories-next/option-ext
The rational is detailed in the root Cargo.toml.

While I don't personally mind MPL dependencies, even if I don't prefer them
(they're allowed in the deny.toml for a reason), I do mind the pointless scope
creep and wish to highlight how little it actually used from the crate by
re-defining it as the single function.

We could also fork directories-next, or directories, and remove the usage of
option-ext per https://github.com/dirs-dev/dirs-sys-rs/issues/24, yet that'd be
a much larger task than what was done here.

In the future, it may be beneficial to submit a PR to wasmtime replacing
directories-next with home, a cargo-team maintained library to get the home
directory and associated folders. An example migration can be found at
https://github.com/harryfei/which-rs/pull/80.
2023-12-30 02:44:33 -05:00
Luke Parker
24ea65eae9
cargo update 2023-12-30 02:36:51 -05:00
Luke Parker
b493e3e31f
Validator DHT (#494)
* Route validators for any active set through sc-authority-discovery

Additionally adds an RPC route to retrieve their P2P addresses.

* Have the coordinator get peers from substrate

* Have the RPC return one address, not up to 3

Prevents the coordinator from believing it has 3 peers when it has one.

* Add missing feature to serai-client

* Correct network argument in serai-client for p2p_validators call

* Add a test in serai-client to check DHT population with a much quicker failure than the coordinator tests

* Update to latest Substrate

Removes distinguishing BABE/AuthorityDiscovery keys which causes
sc_authority_discovery to populate as desired.

* Update to a properly tagged substrate commit

* Add all dialed to peers to GossipSub

* cargo fmt

* Reduce common code in serai-coordinator-tests with amore involved new_test

* Use a recursive async function to spawn `n` DockerTests with the necessary networking configuration

* Merge UNIQUE_ID and ONE_AT_A_TIME

* Tidy up the new recursive code in tests/coordinator

* Use a Mutex in CONTEXT to let it be set multiple times

* Make complimentary edits to full-stack tests

* Augment coordinator P2p connection logs

* Drop lock acquisitions before recursing

* Better scope lock acquisitions in full-stack, preventing a deadlock

* Ensure OUTER_OPS is reset across the test boundary

* Add cargo deny allowance for dockertest fork
2023-12-22 21:09:18 -05:00
Luke Parker
00774c29d7
Replace remaining direct uses of futures with futures_util
Slight downscope which helps combat the antipattern which is the futures glob
crate. While futures_util is still a large crate, it has better defaults and
is smaller by virtue of not pulling the executor.
2023-12-18 19:45:08 -05:00
akildemir
c40ce00955
Slash bad validators (#468)
* implement general design

* add slashing

* bug fixes

* fix pr comments

* misc fixes

* fix grandpa abi call type

* Correct rebase artifacts I introduced

* Cleanups and corrections

1) Uses vec![] for the OpaqueKeyProof as there's no value to passing it around
2) Remove usage of Babe/Grandpa Offences for tracking if an offence is known
   for checking if can slash. If can slash, no prior offence must have been
   known.
3) Rename DisabledIndices to SeraiDisabledIndices, drop historical data for
   current session only.
4) Doesn't remove from the pre-declared upcoming Serai set upon slash due to
   breaking light clients.
5) Into/From instead of AsRef for KeyOwnerProofSystem's generic to ensure
   safety of the conversion.

* Correct deduction from TotalAllocatedStake on slash

It should only be done if in set and only with allocations contributing to
TotalAllocatedStake (Allocation + latest session's PendingDeallocation).

* Changes meant for prior commit

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2023-12-16 17:44:08 -05:00
Luke Parker
74a68c6f68
cargo update
Resolves vulnerably in zerocopy.
2023-12-15 00:54:39 -05:00
Luke Parker
2532423d42
Remove the RemoveParticipant protocol for having new DKGs specify the participants which were removed
Obvious code cleanup is obvious.
2023-12-14 23:51:57 -05:00
Luke Parker
11fdb6da1d
Coordinator Cleanup (#481)
* Move logic for evaluating if a cosign should occur to its own file

Cleans it up and makes it more robust.

* Have expected_next_batch return an error instead of retrying

While convenient to offer an error-free implementation, it potentially caused
very long lived lock acquisitions in handle_processor_message.

* Unify and clean DkgConfirmer and DkgRemoval

Does so via adding a new file for the common code, SigningProtocol.

Modifies from_cache to return the preprocess with the machine, as there's no
reason not to. Also removes an unused Result around the type.

Clarifies the security around deterministic nonces, removing them for
saved-to-disk cached preprocesses. The cached preprocesses are encrypted as the
DB is not a proper secret store.

Moves arguments always present in the protocol from function arguments into the
struct itself.

Removes the horribly ugly code in DkgRemoval, fixing multiple issues present
with it which would cause it to fail on use.

* Set SeraiBlockNumber in cosign.rs as it's used by the cosigning protocol

* Remove unnecessary Clone from lambdas in coordinator

* Remove the EventDb from Tributary scanner

We used per-Transaction DB TXNs so on error, we don't have to rescan the entire
block yet only the rest of it. We prevented scanning multiple transactions by
tracking which we already had.

This is over-engineered and not worth it.

* Implement borsh for HasEvents, removing the manual encoding

* Merge DkgConfirmer and DkgRemoval into signing_protocol.rs

Fixes a bug in DkgConfirmer which would cause it to improperly handle indexes
if any validator had multiple key shares.

* Strictly type DataSpecification's Label

* Correct threshold_i_map_to_keys_and_musig_i_map

It didn't include the participant's own index and accordingly was offset.

* Create TributaryBlockHandler

This struct contains all variables prior passed to handle_block and stops them
from being passed around again and again.

This also ensures fatal_slash is only called while handling a block, as needed
as it expects to operate under perfect consensus.

* Inline accumulate, store confirmation nonces with shares

Inlining accumulate makes sense due to the amount of data accumulate needed to
be passed.

Storing confirmation nonces with shares ensures that both are available or
neither. Prior, one could be yet the other may not have been (requiring an
assert in runtime to ensure we didn't bungle it somehow).

* Create helper functions for handling DkgRemoval/SubstrateSign/Sign Tributary TXs

* Move Label into SignData

All of our transactions which use SignData end up with the same common usage
pattern for Label, justifying this.

Removes 3 transactions, explicitly de-duplicating their handlers.

* Remove CurrentlyCompletingKeyPair for the non-contextual DkgKeyPair

* Remove the manual read/write for TributarySpec for borsh

This struct doesn't have any optimizations booned by the manual impl. Using
borsh reduces our scope.

* Use temporary variables to further minimize LoC in tributary handler

* Remove usage of tuples for non-trivial Tributary transactions

* Remove serde from dkg

serde could be used to deserialize intenrally inconsistent objects which could
lead to panics or faults.

The BorshDeserialize derives have been replaced with a manual implementation
which won't produce inconsistent objects.

* Abstract Future generics using new trait definitions in coordinator

* Move published_signed_transaction to tributary/mod.rs to reduce the size of main.rs

* Split coordinator/src/tributary/mod.rs into spec.rs and transaction.rs
2023-12-10 20:21:44 -05:00
Luke Parker
6caf45ea1d
Downscope usage of futures 2023-12-10 19:32:52 -05:00
Luke Parker
7122e0faf4
Cache the block's events within TemporalSerai
Event retrieval was prior:
- Retrieve all events in the block, which may be hundreds of KB
- Filter to just a few

Since it's frequent to want multiple sets of events, each filtered in their own
way, this caused the retrieval to happen multiple times. Now, it only will
happen once.

Also has the scoped clients take a reference, not an owned TemporalSerai.
2023-12-08 10:46:10 -05:00
Luke Parker
c511a54d18
Move serai-client off serai-runtime, MIT licensing it
Uses a full-fledged serai-abi to do so.

Removes use of UncheckedExtrinsic as a pointlessly (for us) length-prefixed
block with a more complicated signing algorithm than advantageous.

In the future, we should considering consolidating the various primitives
crates. I'm not convinced we benefit from one primitives crate per pallet.
2023-12-07 02:30:09 -05:00
Luke Parker
6416e0079b
Add ABI crate
Call and Event are both from the pallets, which are AGPL licensed. Accordingly,
they make serai-client AGPL licensed when serai-client must end up MIT
licensed. This creates a MIT-licensed variant of Calls and Events such that
they can be used by serai-client, enabling transitioning it to MIT.

Relevant to https://github.com/serai-dex/serai/issues/337.
2023-12-06 09:56:43 -05:00
Luke Parker
82f7342372
cargo update
Updates off a yanked version of zerocopy, fixing the failing deny CI.

Bites the bullet on windows-sys 0.52. While I was hoping to update everything
at once, unfortunately tokio won't update until March (see
https://github.com/tokio-rs/mio/pull/1725). I don't want to withold these
updates for that long.
2023-12-06 05:01:08 -05:00
akildemir
4ebfae0b63
Ensure economic security on validator sets (#459)
* add price oracle

* tidy up

* add todo

* bug fixes

* fix pr comments

* Use spot price, tweak some formulas

Also cleans nits.

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2023-12-05 08:52:50 -05:00
Luke Parker
6e8a5f9cb1
cargo update, remove unneeded dependencies from the processor 2023-12-03 00:05:03 -05:00
Luke Parker
b823413c9b
Use parity-db in current Dockerfiles (#455)
* Use redb and in Dockerfiles

The motivation for redb was to remove the multiple rocksdb compile times from
CI.

* Correct feature flagging of coordinator and message-queue in Dockerfiles

* Correct message-queue DB type alias

* Use consistent table typing in redb

* Correct rebase artifacts

* Correct removal of binaries feature from message-queue

* Correct processor feature flagging

* Replace redb with parity-db

It still has much better compile times yet doesn't block when creating multiple
transactions. It also is actively maintained and doesn't grow our tree. The MPT
aspects are irrelevant.

* Correct stray Redb

* clippy warning

* Correct txn get
2023-11-30 04:22:37 -05:00
Luke Parker
695d1f0ecf
Remove subxt (#460)
* Remove subxt

Removes ~20 crates from our Cargo.lock.

Removes downloading the metadata and enables removing the getMetadata RPC route
(relevant to #379).

Moves forward #337.

Done now due to distinctions in the subxt 0.32 API surface which make it
justifiable to not update.

* fmt, update due to deny triggering on a yanked crate

* Correct the handling of substrate_block_notifier now that it's ephemeral, not long-lived

* Correct URL in tests/coordinator from ws to http
2023-11-28 02:29:50 -05:00
Luke Parker
571195bfda
Resolve #360 (#456)
* Remove NetworkId from processor-messages

Because intent binds to the sender/receiver, it's not needed for intent.

The processor knows what the network is.

The coordinator knows which to use because it's sending this message to the
processor for that network.

Also removes the unused zeroize.

* ProcessorMessage::Completed use Session instead of key

* Move SubstrateSignId to Session

* Finish replacing key with session
2023-11-26 12:14:23 -05:00
Luke Parker
b79cf8abde
Move message-queue to a fully binary representation (#454)
* Move message-queue to a fully binary representation

Additionally adds a timeout to the message queue test.

* coordinator clippy

* Remove contention for the message-queue socket by using per-request sockets

* clippy
2023-11-26 11:22:18 -05:00
Luke Parker
d60e007126
Add a binaries feature to the processor to reduce dependencies when used as a lib
processor isn't intended to be used as a library, yet serai-processor-tests
does pull it in as a lib. This caused serai-processor-tests to need to compile
rocksdb, which added multiple minutes to the compilation time.
2023-11-25 04:04:52 -05:00
Luke Parker
b296be8515
Replace bincode with borsh (#452)
* Add SignalsConfig to chain_spec

* Correct multiexp feature flagging for rand_core std

* Remove bincode for borsh

Replaces a non-canonical encoding with a canonical encoding which additionally
should be faster.

Also fixes an issue where we used bincode in transcripts where it cannot be
trusted.

This ended up fixing a myriad of other bugs observed, unfortunately.
Accordingly, it either has to be merged or the bug fixes from it must be ported
to a new PR.

* Make serde optional, minimize usage

* Make borsh an optional dependency of substrate/ crates

* Remove unused dependencies

* Use [u8; 64] where possible in the processor messages

* Correct borsh feature flagging
2023-11-25 04:01:11 -05:00
Luke Parker
0ea90d054d
Enable the signals pallet to halt networks' publication of Batchs
Relevant to #394.

Prevents hand-over due to hand-over occurring via a `Batch` publication.

Expects a new protocol to restore functionality (after a retirement of the
current protocol).
2023-11-24 19:56:57 -05:00
akildemir
fcfdadc791
Integrate session pallet into validator-sets pallet (#440)
* remove pallet-session

* Store key shares in InSet

* integrate grandpa to vs-pallet

* integrate pallet babe

* remove pallet-session & authority discovery from runtime

* update the grandpa pallet path

* cargo update grandpa

* cargo update substrate

* Misc tweaks

Sets validators for BABE/GRANDPA in chain_spec, per Akil's realization that was
the missing piece.

* fix pr comments

* bug fix & tidy up

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2023-11-22 06:22:46 -05:00
Luke Parker
a315040aee
if-watch 3.2.0 2023-11-22 01:24:17 -05:00
Luke Parker
942873f99d Non-intrusive cargo update 2023-11-20 02:31:47 -05:00
Luke Parker
db49a63c2b
Bump zeroize to appease cargo deny 2023-11-16 14:18:00 -05:00
Luke Parker
c03afbe03e
Downgrade RustCrypto packages which violate Rust's interpretation of semver 2023-11-15 21:24:52 -05:00
Luke Parker
af611a39bf
Restore runtime feature to hyper 2023-11-15 20:40:24 -05:00
Luke Parker
0d080e6d34
Remove unused dependency from dex-pallet 2023-11-15 20:24:33 -05:00
Luke Parker
d25e3d86a2
Make TLS an optional feature of simple-request
Removes 14 crates from the tree when compiling the message-queue client.

Also performs a non-intrusive cargo update.
2023-11-15 17:24:11 -05:00
David Bell
c328e5ea68
Convert coordinator/tributary/nonce_decider to use create_db macro (#423)
* chore: convert nonce_deicer to use create_db macro

* Restore pub NonceDecider

* Remove extraneous comma

I forgot to run git commit --amend on the prior commit :/

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2023-11-12 12:04:34 -05:00
akildemir
d015ee96a3
Dex improvements (#422)
* remove dex traits&balance types

* remove liq tokens pallet in favor of coins-pallet instance

* fix tests & benchmarks

* remove liquidity tokens trait

* fix CI

* fix pr comments

* Slight renamings

* Add burn_with_instruction as a negative to LiquidityTokens CallFilter

* Remove use of One, Zero, Saturating taits in dex pallet

---------

Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
2023-11-12 06:37:31 -05:00
Luke Parker
7f1732c8c0
cargo update to snow 0.9.4 2023-11-12 00:40:32 -05:00
Luke Parker
b51204a4eb
Replace usage of ethers-signers with 11 lines of ECDSA code 2023-11-09 13:22:43 -05:00
Luke Parker
11f66c741d
Remove ethers-middleware 2023-11-08 08:19:12 -05:00
Luke Parker
a0a2ef22e4
Remove ethers-solc
ethers-solc was used for a type (now manually specified) and to call out to
solc. Since Foundry was already a documented dependency, a call to it now
handles building.

Removing this single crate removes a total of 17 crates from our dependency
tree. While these may still be around due to Foundry, they at least may not
be.

Further work to remove the requirement on Foundry for solc alone would be
appreciated.
2023-11-08 06:25:35 -05:00
Luke Parker
5e290a29d9
Remove frame-benchmarking-cli
Not currently used, notably increases our dependency tree.

I wouldn't remove it if we planned to use it. From my understanding, all
benchmarking will be per pallet, voiding our need to have this for the node.
2023-11-08 05:59:56 -05:00
Luke Parker
bc07e14b1e
Remove async_recursion for a for loop 2023-11-07 23:07:26 -05:00
Luke Parker
56fd11ab8d
Use a single long-lived RPC connection when authenticated
The prior system spawned a new connection per request to enable parallelism,
yet kept hitting hyper::IncompleteMessages I couldn't track down. This
attempts to resolve those by a long-lived socket.

Halves the amount of requests per-authenticated RPC call, and accordingly is
likely still better overall.

I don't believe this is resolved yet but this is still worth pushing.
2023-11-07 17:42:19 -05:00