Fast sync part 2 (#156)

* boilerplate

* Cargo.lock

* Stub of block validation

* Block validation (diff by @dllud)

* Cargo.lock

* Complete implementation of block validation request

* Apply suggestions from code review

Co-authored-by: Boog900 <boog900@tutanota.com>

* More suggestions

* Update consensus/fast-sync/src/fast_sync.rs

* Update consensus/fast-sync/src/fast_sync.rs

---------

Co-authored-by: Boog900 <boog900@tutanota.com>

CONTRIBUTING.md

@@ -1,39 +1,93 @@
-# Contributing to Cuprate
+## Contributing to Cuprate
+Thank you for wanting to help out!
 
-## Introduction
+Cuprate is in the stage where things are likely to change quickly, so it's recommended
+you ask questions in our public [Matrix room](https://matrix.to/#/#cuprate:monero.social).
 
-Thank you for wanting to help out! Cuprate is in the stage where things are likely to change quickly, so it's recommend
+- [1. Submitting a pull request](#1-submitting-a-pull-request)
-you join our [Matrix room](https://matrix.to/#/#cuprate:monero.social).
+	- [1.1 Rust toolchain](#11-rust-toolchain)
+	- [1.2 Draft PR](#12-draft-pr)
+	- [1.3 Passing CI](#13-passing-ci)
+	- [1.4 Ready for review](#14-ready-for-review)
+- [2. Crate names](#2-crate-names)
+- [3. Coding guidelines](#3-coding-guidelines)
+- [4. Keeping track of issues and PRs](#4-keeping-track-of-issues-and-prs)
+- [5. Documentation](#5-documentation)
+- [6. Books](#6-books)
+	- [6.1 Architecture book](#61-architecture-book)
+	- [6.2 Protocol book](#62-protocol-book)
+	- [6.3 User book](#63-user-book)
 
-## Making a PR
+## 1. Submitting a pull request
+Once you have found something you would like to work on by:
+- Looking at the [open issues](https://github.com/Cuprate/cuprate/issues)
+- Looking at issues with the [`A-help-wanted`](https://github.com/Cuprate/cuprate/issues?q=is%3Aissue+is%3Aopen+label%3AE-help-wanted) label
+- or joining Cuprate's [Matrix room](https://matrix.to/#/#cuprate:monero.social) and asking
 
-Once you have found something you would like to work on by either looking at the open issues or joining Cuprate's [Matrix room](https://matrix.to/#/#cuprate:monero.social)
+it is recommended to make your interest on working on that thing known so people don't duplicate work.
-and asking it's recommended to make your interest on working on that thing known so people don't duplicate work.
 
-When you are at a stage where you would like feedback you can open a draft PR, keep in mind that feedback may take time especially if the change is large.
+Before starting, consider reading/using Cuprate's:
-Once your PR is at the stage where you feel it's ready to go, open it for review.
+- [`Documentation`](#5-documentation) (practical `cargo` docs)
+- [`Books`](#6-books) (Cuprate's architecture and protocol)
 
-## Passing CI
+These may answer some questions you have, or may confirm an issue you would like to fix.
-The first 3 steps to CI are formatting, typo, and documentation checking.
 
-Check if your changes are formatted, typo-free, and documented correctly by running:
+_Note: Cuprate is currently a work-in-progress; documentation will be changing/unfinished._
-- `cargo fmt --all --check`
+
-- `typos`
+### 1.1 Rust toolchain
-- `RUSTDOCFLAGS='-D warnings' cargo doc --workspace --all-features`
+Cuprate is written in [Rust](https://rust-lang.org).
+
+If you are editing code, you will need Rust's toolchain and package manager,
+[`cargo`](https://doc.rust-lang.org/cargo/index.html), to develop and submit PRs effectively.
+
+Get started with Rust here: <https://www.rust-lang.org/learn/get-started>.
+
+### 1.2 Draft PR
+Consider opening a draft PR until you have passed all CI.
+
+This is also the stage where you can ask for feedback from others. Keep in mind that feedback may take time especially if the change is large.
+
+### 1.3 Passing CI
+Each commit pushed in a PR will trigger our [lovely, yet pedantic CI](https://github.com/Cuprate/cuprate/blob/main/.github/workflows/ci.yml).
+
+It currently:
+- Checks code formatting
+- Checks documentation
+- Looks for typos
+- Runs [`clippy`](https://github.com/rust-lang/rust-clippy) (and fails on warnings)
+- Runs all tests
+- Builds all targets
+- Automatically add approriate [labels](#4-keeping-track-of-issues-and-prs) to your PR
+
+Before pushing your code, please run the following at the root of the repository:
+
+| Command           | Does what |
+|-------------------|-----------|
+| `cargo fmt --all` | Formats code
+| `typos -w`        | Fixes typos
 
 `typos` can be installed with `cargo` from: https://github.com/crate-ci/typos.
 
-After that, ensure all lints, tests, and builds are successful by running:
+After that, ensure all other CI passes by running:
 
-- `cargo clippy --workspace --all-features --all-targets -- -D warnings`
+| Command                                                                | Does what |
-- `cargo fmt --all`
+|------------------------------------------------------------------------|-----------|
-- `cargo test --all-features --workspace`
+| `RUSTDOCFLAGS='-D warnings' cargo doc --workspace --all-features`      | Checks documentation is OK
-- `cargo build --all-features --all-targets --workspace`
+| `cargo clippy --workspace --all-features --all-targets -- -D warnings` | Checks clippy lints are satisfied
+| `cargo test --all-features --workspace`                                | Runs all tests
+| `cargo build --all-features --all-targets --workspace`                 | Builds all code
 
-## Crate names
+**Note: in order for some tests to work, you will need to place a [`monerod`](https://www.getmonero.org/downloads/) binary at the root of the repository.**
-All of Cuprate's crates (libraries) are prefixed with `cuprate-`.
 
-All directories containing crates however, are not. For example:
+### 1.4 Ready for review
+Once your PR has passed all CI and is ready to go, open it for review. Others will leave their thoughts and may ask for changes to be made.
+
+Finally, if everything looks good, we will merge your code! Thank you for contributing!
+
+## 2. Crate names
+All of Cuprate's crates (libraries) are prefixed with `cuprate-`. All directories containing crates however, are not.
+
+For example:
 
 | Crate Directory    | Crate Name         |
 |--------------------|--------------------|
@@ -41,16 +95,21 @@ All directories containing crates however, are not. For example:
 | `net/levin`        | `cuprate-levin`    |
 | `net/wire`         | `cuprate-wire`     |
 
-## Coding guidelines
+## 3. Coding guidelines
+This is a list of rules that are not mandated by any automation, but contributors generally follow.
 
+You should keep these in mind when submitting code:
+
+- Separate and sort imports as core, std, third-party, Cuprate crates, current crate
+- Follow the [Rust API Guidelines](https://rust-lang.github.io/api-guidelines)
 - `// Comment like this.` and not `//like this`
 - Use `TODO` instead of `FIXME`
 - Avoid `unsafe`
-- Sort imports as core, std, third-party, Cuprate crates, current crate.
-- Follow the [Rust API Guidelines](https://rust-lang.github.io/api-guidelines)
-- Break the above rules when it makes sense
 
-## Keeping track of issues and PRs
+And the most important rule:
+- Break any and all of the above rules when it makes sense
+
+## 4. Keeping track of issues and PRs
 The Cuprate GitHub repository has a lot of issues and PRs to keep track of. Cuprate makes use of generic labels and labels grouped by a prefixes to help with this.
 
 Some labels will be [automatically added/removed](https://github.com/Cuprate/cuprate/tree/main/.github/labeler.yml) if certain file paths have been changed in a PR.
@@ -76,7 +135,45 @@ This section is primarily targeted at maintainers. Most contributors aren't able
 [O-]: https://github.com/Cuprate/cuprate/labels?q=O
 [P-]: https://github.com/Cuprate/cuprate/labels?q=P
 
-## Books
+## 5. Documentation
+Cuprate's crates (libraries) have inline documentation.
+
+These can be built and viewed using the `cargo` tool. For example, to build and view a specific crate's documentation, run the following command at the repository's root:
+```bash
+cargo doc --open --package $CRATE
+```
+`$CRATE` can be any package listed in the [root `Cargo.toml`](https://github.com/Cuprate/cuprate/tree/main/Cargo.toml)'s workspace members list, for example, `cuprate-blockchain`.
+
+You can also build all documentation at once:
+```bash
+cargo doc
+```
+and view by using a web-browser to open the `index.html` file within the build directory: `target/doc/$CRATE/index.html`, for example, `target/doc/cuprate_blockchain/index.html`.
+
+## 6. Books
 Cuprate has various documentation books whose source files live in [`books/`](https://github.com/Cuprate/cuprate/tree/main/books).
 
 Please contribute if you found a mistake! The files are mostly [markdown](https://wikipedia.org/wiki/Markdown) files and can be easily edited. See the `books/` directory for more information.
+
+These books are also good resources to understand how Cuprate and Monero work.
+
+### 6.1 Architecture book
+This book documents Cuprate's architecture and implementation.
+
+- <https://architecture.cuprate.org>
+- <https://github.com/Cuprate/architecture-book>
+- <https://github.com/Cuprate/cuprate/tree/main/books/architecture>
+
+### 6.2 Protocol book
+This book documents the Monero protocol.
+
+- <https://monero-book.cuprate.org>
+- <https://github.com/Cuprate/monero-book>
+- <https://github.com/Cuprate/cuprate/tree/main/books/protocol>
+
+### 6.3 User book
+This book is a user-guide for using Cuprate.
+
+- <https://user.cuprate.org>
+- <https://github.com/Cuprate/user-book>
+- <https://github.com/Cuprate/cuprate/tree/main/books/user>

Cargo.lock

@@ -565,11 +565,15 @@ version = "0.1.0"
 dependencies = [
  "clap",
  "cuprate-blockchain",
+ "cuprate-consensus",
+ "cuprate-consensus-rules",
  "cuprate-types",
  "hex",
  "hex-literal",
+ "monero-serai",
  "rayon",
  "sha3",
+ "thiserror",
  "tokio",
  "tokio-test",
  "tower",

consensus/fast-sync/Cargo.toml

@@ -11,11 +11,15 @@ path = "src/create.rs"
 [dependencies]
 clap = { workspace = true, features = ["derive", "std"] }
 cuprate-blockchain = { path = "../../storage/cuprate-blockchain" }
+cuprate-consensus = { path = ".." }
+cuprate-consensus-rules = { path = "../rules" }
 cuprate-types = { path = "../../types" }
 hex.workspace = true
 hex-literal.workspace = true
+monero-serai.workspace = true
 rayon.workspace = true
 sha3 = "0.10.8"
+thiserror.workspace = true
 tokio = { workspace = true, features = ["full"] }
 tower.workspace = true
 

consensus/fast-sync/src/fast_sync.rs

@@ -1,5 +1,6 @@
 use std::{
     cmp,
+    collections::HashMap,
     future::Future,
     pin::Pin,
     task::{Context, Poll},
@@ -7,9 +8,21 @@ use std::{
 
 #[allow(unused_imports)]
 use hex_literal::hex;
-use tower::Service;
+use monero_serai::{
+    block::Block,
+    transaction::{Input, Transaction},
+};
+use tower::{Service, ServiceExt};
+
+use cuprate_consensus::{
+    context::{BlockChainContextRequest, BlockChainContextResponse},
+    transactions::TransactionVerificationData,
+};
+use cuprate_consensus_rules::{miner_tx::MinerTxError, ConsensusError};
+use cuprate_types::{VerifiedBlockInformation, VerifiedTransactionInformation};
 
 use crate::{hash_of_hashes, BlockId, HashOfHashes};
+
 #[cfg(not(test))]
 static HASHES_OF_HASHES: &[HashOfHashes] = &include!("./data/hashes_of_hashes");
 
@@ -31,13 +44,6 @@ fn max_height() -> u64 {
     (HASHES_OF_HASHES.len() * BATCH_SIZE) as u64
 }
 
-pub enum FastSyncRequest {
-    ValidateHashes {
-        start_height: u64,
-        block_ids: Vec<BlockId>,
-    },
-}
-
 #[derive(Debug, PartialEq)]
 pub struct ValidBlockId(BlockId);
 
@@ -45,31 +51,79 @@ fn valid_block_ids(block_ids: &[BlockId]) -> Vec<ValidBlockId> {
     block_ids.iter().map(|b| ValidBlockId(*b)).collect()
 }
 
+#[allow(clippy::large_enum_variant)]
+pub enum FastSyncRequest {
+    ValidateHashes {
+        start_height: u64,
+        block_ids: Vec<BlockId>,
+    },
+    ValidateBlock {
+        block: Block,
+        txs: HashMap<[u8; 32], Transaction>,
+        token: ValidBlockId,
+    },
+}
+
+#[allow(clippy::large_enum_variant)]
 #[derive(Debug, PartialEq)]
 pub enum FastSyncResponse {
     ValidateHashes {
         validated_hashes: Vec<ValidBlockId>,
         unknown_hashes: Vec<BlockId>,
     },
+    ValidateBlock(VerifiedBlockInformation),
 }
 
-#[derive(Debug, PartialEq)]
+#[derive(thiserror::Error, Debug, PartialEq)]
 pub enum FastSyncError {
-    InvalidStartHeight, // start_height not a multiple of BATCH_SIZE
+    #[error("Block does not match its expected hash")]
-    Mismatch,           // hash does not match
+    BlockHashMismatch,
-    NothingToDo,        // no complete batch to check
+
-    OutOfRange,         // start_height too high
+    #[error("Start height must be a multiple of the batch size")]
+    InvalidStartHeight,
+
+    #[error("Hash of hashes mismatch")]
+    Mismatch,
+
+    #[error("Given range too small for fast sync (less than one batch)")]
+    NothingToDo,
+
+    #[error("Start height too high for fast sync")]
+    OutOfRange,
+
+    #[error("Block does not have the expected height entry")]
+    BlockHeightMismatch,
+
+    #[error("Block does not contain the expected transaction list")]
+    TxsIncludedWithBlockIncorrect,
+
+    #[error(transparent)]
+    Consensus(#[from] ConsensusError),
+
+    #[error(transparent)]
+    MinerTx(#[from] MinerTxError),
+
+    #[error("Database error: {0}")]
+    DbErr(String),
+}
+
+impl From<tower::BoxError> for FastSyncError {
+    fn from(error: tower::BoxError) -> Self {
+        Self::DbErr(error.to_string())
+    }
 }
 
-#[allow(dead_code)]
 pub struct FastSyncService<C> {
     context_svc: C,
 }
 
 impl<C> FastSyncService<C>
 where
-    C: Service<FastSyncRequest, Response = FastSyncResponse, Error = FastSyncError>
+    C: Service<
-        + Clone
+            BlockChainContextRequest,
+            Response = BlockChainContextResponse,
+            Error = tower::BoxError,
+        > + Clone
         + Send
         + 'static,
 {
@@ -81,8 +135,11 @@ where
 
 impl<C> Service<FastSyncRequest> for FastSyncService<C>
 where
-    C: Service<FastSyncRequest, Response = FastSyncResponse, Error = FastSyncError>
+    C: Service<
-        + Clone
+            BlockChainContextRequest,
+            Response = BlockChainContextResponse,
+            Error = tower::BoxError,
+        > + Clone
         + Send
         + 'static,
     C::Future: Send + 'static,
@@ -97,12 +154,17 @@ where
     }
 
     fn call(&mut self, req: FastSyncRequest) -> Self::Future {
+        let context_svc = self.context_svc.clone();
+
         Box::pin(async move {
             match req {
                 FastSyncRequest::ValidateHashes {
                     start_height,
                     block_ids,
                 } => validate_hashes(start_height, &block_ids).await,
+                FastSyncRequest::ValidateBlock { block, txs, token } => {
+                    validate_block(context_svc, block, txs, token).await
+                }
             }
         })
     }
@@ -149,6 +211,91 @@ async fn validate_hashes(
     })
 }
 
+async fn validate_block<C>(
+    mut context_svc: C,
+    block: Block,
+    mut txs: HashMap<[u8; 32], Transaction>,
+    token: ValidBlockId,
+) -> Result<FastSyncResponse, FastSyncError>
+where
+    C: Service<
+            BlockChainContextRequest,
+            Response = BlockChainContextResponse,
+            Error = tower::BoxError,
+        > + Send
+        + 'static,
+    C::Future: Send + 'static,
+{
+    let BlockChainContextResponse::Context(checked_context) = context_svc
+        .ready()
+        .await?
+        .call(BlockChainContextRequest::GetContext)
+        .await?
+    else {
+        panic!("Context service returned wrong response!");
+    };
+
+    let block_chain_ctx = checked_context.unchecked_blockchain_context().clone();
+
+    let block_hash = block.hash();
+    if block_hash != token.0 {
+        return Err(FastSyncError::BlockHashMismatch);
+    }
+
+    let block_blob = block.serialize();
+
+    let Some(Input::Gen(height)) = block.miner_tx.prefix.inputs.first() else {
+        return Err(FastSyncError::MinerTx(MinerTxError::InputNotOfTypeGen));
+    };
+    if *height != block_chain_ctx.chain_height {
+        return Err(FastSyncError::BlockHeightMismatch);
+    }
+
+    let mut verified_txs = Vec::with_capacity(txs.len());
+    for tx in &block.txs {
+        let tx = txs
+            .remove(tx)
+            .ok_or(FastSyncError::TxsIncludedWithBlockIncorrect)?;
+
+        let data = TransactionVerificationData::new(tx)?;
+        verified_txs.push(VerifiedTransactionInformation {
+            tx_blob: data.tx_blob,
+            tx_weight: data.tx_weight,
+            fee: data.fee,
+            tx_hash: data.tx_hash,
+            tx: data.tx,
+        });
+    }
+
+    let total_fees = verified_txs.iter().map(|tx| tx.fee).sum::<u64>();
+    let total_outputs = block
+        .miner_tx
+        .prefix
+        .outputs
+        .iter()
+        .map(|output| output.amount.unwrap_or(0))
+        .sum::<u64>();
+
+    let generated_coins = total_outputs - total_fees;
+
+    let weight =
+        block.miner_tx.weight() + verified_txs.iter().map(|tx| tx.tx_weight).sum::<usize>();
+
+    Ok(FastSyncResponse::ValidateBlock(VerifiedBlockInformation {
+        block_blob,
+        txs: verified_txs,
+        block_hash,
+        pow_hash: [0u8; 32],
+        height: *height,
+        generated_coins,
+        weight,
+        long_term_weight: block_chain_ctx.next_block_long_term_weight(weight),
+        cumulative_difficulty: block_chain_ctx.cumulative_difficulty
+            + block_chain_ctx.next_difficulty,
+        block,
+    }))
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;