* impl async buffer
* clippy
* p2p changes
* clippy
* a few more docs
* init cuprate-p2p
* remove some unrelated code and add some docs
* start documenting client_pool.rs
* add more docs
* typo
* fix docs
* use JoinSet in connection maintainer
* small changes
* add peer sync state svc
* add broadcast svc
* add more docs
* add some tests
* add a test
* fix merge
* add another test
* unify PeerDisconnectFut and add more docs
* start network init
* add an inbound connection server
* remove crate doc for now
* fix address book docs
* fix leak in client pool
* correct comment
* fix merge + add some docs
* review comments
* init block downloader
* fix doc
* initial chain search
* add chain_tracker
* move block downloader to struct
* spawn task whe getting blocks
* check for free peers and handle batch response
* add test bin
* working block downloader
* dynamic batch sizes
* dandelion_tower -> dandelion-tower
* fix async-buffer builds
* check if incoming peers are banned
* add interface methods
* update docs
* use a JoinSet for background network tasks
* dynamic batch size changes
* Keep a longer of queue of blocks to get
* more checks on incoming data
* fix merge
* fix imports
* add more docs
* add some limits on messages
* keep peers that dont have the current need data
* fix clippy
* fix .lock
* fix stopping the block downloader
* clean up API and add more docs
* tracing + bug fixes
* fix panic
* doc changes
* remove test_init
* remove spammy log
* fix previous merge
* add a test
* fix test
* remove test unwrap
* order imports correctly
* clean up test
* add a timeout
* fix tests
* review fixes
* make `BlockDownloader` pub
* make `initial_chain_search` pub
* make `block_downloader` private
* Apply suggestions from code review
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
* split some sections into separate modules
* split chain requests
* sort imports
* check previous ID is correct
* fix typos
* Apply suggestions from code review
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
---------
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
* change monero-consensus to cuprate-consensus-rules
* document the context service
* remove the mutex on blockchain context
* comment the context caches
* add back tokio
* document block checks
* typo
* keep tha amount of outputs with a certain amount in the output cache
* typo
* nuke cross-block batch verification
* remove RPC scanner
* change how contextual data is got.
* fmt & clippy fixes
* typo
* cargo update
* restore Cargo.lock
* add a verify tx test.
+ fixes an issue with verifying signatures after BPs
* clippy
* remove bad test
* add mores tests and fix a couple bugs
* typos
* move tests and add some more
* typo
* remove scan_chain docs
* fix check for duplicate txs when duplicates are not sequential
* add a proptest for dup txs
* cache tx verification state
* doc updates + move `Vec` to `Arc<[]>`
* clippy
* misc changes
* Apply suggestions from code review
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
Co-authored-by: SyntheticBird <118022351+SyntheticBird45@users.noreply.github.com>
* fix fmt
* review changes
---------
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
Co-authored-by: SyntheticBird <118022351+SyntheticBird45@users.noreply.github.com>
* `database/` -> `storage/`, create skeleton `storage/` crates
* add skeleton root directories
* books: add skeleton
* ci: fix labeler.yml
* fix CONTRIBUTING.md
* fix workspace Cargo.toml
* fix `storage/`
* rename helper fs fn, `cuprate_database` -> `cuprate_blockchain`
* fix Cargo.toml
* cuprate-blockchain: revert find-and-replace
Since it has to be ported to the book anyway, there's not much
reason to update this and create review diffs.
* labeler.yml: remove `A-docs` from `books/` changes
Although books are documentation,
the `A-books` label already exists for this
* cargo: add `pretty_assertions`
* test-utils: add tests to compare data against RPC data, and fix
* small fixes
* database: fix `cumulative_generated_coins` from test data
block had wrong data which offset all of the asserts
* ops: add `trait MoneroR{o,w}`
* update `trait MoneroR{o,w}` bounds
* types: add `BlockInfoLatest` type alias
* block: impl most core functions
* types: fix https://github.com/Cuprate/cuprate/pull/91#discussion_r1527668916
* fix table type test
* cargo.toml: add `{cuprate-types, monero-serai}`
* add_block: add all other block data
* ops: remove unneeded `block` functions
* env: add `EnvInner::open_db_rw_all()`
* types: fix test
* block: `&mut TxRw` -> `&TxRw`, use `open_db_rw_all()`
* add `trait Tables[Mut]` and use it in `EnvInner`
* block: use `TablesMut`
* tables: replace manual impl with `define_trait_tables!()`
* tables: docs for `trait Tables[Mut]`
* tables: doc functions
* create `call_fn_on_all_tables_or_early_return!()` macro
* block: cleanup signatures + bodies
* block: more fn's, docs
* block: add `doc_{error,single,bulk}!()`
* remove `ops/monero.rs`
* move `height()` to `ops/blockchain.rs`
* add `ops/macros.rs`
* tx: add fn signatures
* output: fix fn signatures
* ops: expose `_inner()` functions
* block: add `add_block_header{_bulk, _inner}()`
* ops: remove doc_{fn,inner}!()`
* ops: remove `_{inner,bulk}()`, lifetime + generics
* update lib/mod docs
* ops: add and use `doc_add_block_inner_invariant!()`
* ops: add docs/return to inner `add_block()` functions
* add_block(): extract and use fn for {key_image, output}
* ops: more fn body impl + `add_block()`
* cargo: add `monero-pruning`
* ops: extract out `tx` functions from `add_block()`
* property: add `db_version()`
* ops: `pop_block()` body, remove other `pop_block` fn's
* types: add `block_blob: Vec<u8>` to `VerifiedBlockInformation`
* block: put `block_blob`, pass `Tables` to sub-functions
`impl TablesMut` can't mutably pass multiple tables since
it takes `&mut self`, so all functions unfortunately have to
take a full `&mut impl TablesMut` even though they only need
a few tables.
* database: add `DatabaseRw::take()`
useful for `pop_block()` where we need the value afterwards
* block: deserialize tx's from `block_blobs` in `pop_block()`
* blockchain: `height()` -> `chain_height()`
* output: fix `amount_index`
* ops: fix unlock_time, chain_height
* `BlockInfoV{1,2,3}` -> `BlockInfo`
* constants: add `DATABASE_VERSION`
* database: add `DatabaseRw::update()`
* output: use `DatabaseRw::update()` in `remove_output()`
* add `TxBlobs` table, ignore pruning tables
* block: mostly impl `add_block()` body
* ops: comments
* add_block: miner v2 tx commitment, height cast
* block: impl `pop_block()`
* block: mostly impl `get_block()`
* block: impl `get_block_{from_height,header,header_from_height}`
* add `OutputFlags` bitflags
* add_block: u32::try_into(height: u32), use `OutputFlags`
* tx: impl `get_{tx,tx_from_id}()`
* tx: move docs tests to `#[test]`
testing everything in 1 go is more natural since e.g:
`add_tx()` is followed by `get_tx()`
* tables: add `trait TablesIter`, `all_tables_empty()`
This allows `TablesMut` to be a superset of `Tables`
and use all its accessor functions.
* use cuprate-test-utils, fix tx tests
* block: `add_block()` take block by ref
* tx: use all txs in tests
* output: add `all_tx_functions()` test
* add_block: check current height against input
* block: map more fields in `get_block()`
* block: remove `get_block()`, doc tests, fix `get_block_header()`
* block: dummy values in test
* heed: use `last/first()` instead of `unsafe` cursors
We no longer have DUP semantics and also hard to debug errors
were popping up on `del_current()`...
* heed: fix `DatabaseRw::delete`
Ok(true) means the key did not exist, so we must return
Err(RuntimeError::KeyNotFound)
* block: `add_block()` (dummy value) test
* ops: `key_image` tests
* cleanup, docs, tests
* backend: test `take()` & `update()`
* docs
* remove `OutputFlags::NONE`
* add_block(): add asserts, panic docs, `should_panic` tests
* backend: remove `Ok(())` in `Database::delete` if already deleted
redb already does this, so heed so match
* block: move block operations after tx/outputs
* `amount == 0` -> `amount == 1`
* Nit: StorableVec::wrap_ref
* `saturating_sub(1)` -> `- 1`
* add `TxOutputs` table
* add_block(): add to `tx_outputs` table
* fix `DatabaseRo::update`
* add_tx(): take `block_height` as input
* tx: add/remove from `TxOutputs` table
* output: remove if `amount == 1` -> `amount_index == 0`
* output: fix `add_output()`'s `amount_index` calculation
* output: fix `add_output()`'s `amount_index` calculation again
* output: tests for `amount_index/num_outputs`
* block: `num_outputs - 1` and `take()` -> `get()`
We don't need to `take()` since the call afterwards to
`remove_output()` removes the entry
* block: swap `get_block_extended_header[_from_height]()`
* move `{key_image,output}` handling `add_block()` -> `add_tx()`
* blockchain: add doc to `top_block_height()`
* block: manual panic -> `assert_eq!()`
* test-utils: add `block_blob` to `VerifiedBlockInformation`
field introduced in this PR
* ops: use real block/tx data in tests
* block: `total_generated_coins` -> `cumulative_generated_coins`
* fix clippy, docs, TODOs
* `cumulative_generated_coins()`: `block/` -> `blockchain/`
* blockchain: add `cumulative_generated_coins()` tests
* Update database/src/ops/block.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* `cumulative_generated_coins()` docs for pre-block-0 special case
---------
Co-authored-by: Boog900 <boog900@tutanota.com>
* add `cuprate-types`
* remove `cuprate_database::service::{request,response}`
* use `cuprate_types::service::{request,response}`
* service: fix `Request` `match`'s
* service: create `ReadRequest` function mappings
* service: create `WriteRequest` function mappings
* service: add rough `WriteRequest` retry loop
* service: handle `RuntimeError::ResizeNeeded` in writer
* add `{R,r}o` exception to typos
* docs
* env: make `resize_map()` return new memory map byte size
* write: proactively handle resizes
`add_block()` takes `VerifiedBlockInformation` such
that it can just take the inner blobs of data.
This is a problem when reactively resizing since we no longer
have the block struct we just gave away so we're forced to `.clone()`
each retry.
Instead of that - we will proactively resize so the resize error
will never occur in the first place.
* read: use type aliases
* docs
* fix import
* write: handle resizes reactively
* service: panic if response can't be sent back
* write: add loop unreachable asserts
* service: print and drop error instead of panic
* write: fix retry loop off-by-1
* write: fix docs
* review changes
* update readme
* remove `BlockBatchInRange` request/response
* Update database/README.md
Co-authored-by: Boog900 <boog900@tutanota.com>
---------
Co-authored-by: Boog900 <boog900@tutanota.com>
* use tokio's delay queue for bans
* document handles
* remove peers from address book when retrieving
* ping inbound peers during handshakes
* support receiving pings during handshakes
* add peer to anchor before reducing whit list
* clippy
* comment handshakes
* typos
* sort `use`
* use `rand::prelude::*`
* review comments
* update macro
* levin: fix fragmented messages & use bitflags
* levin: add a method to fragment a message
* levin: add tests for fragmented messages and fix issues
* fix docs
* tests: don't include bytes length
* levin: add support for sending fragmented
/ dummy messages
* fmt
* add fragmented handshake tests.
* fix handshake detection when fragmented
and alt (non-monero) protocol info
* add tracing logs
* remove `already_built`, this was an old way I was thinking of sending raw buckets
* clippy
* clippy 2
* Update net/levin/src/message.rs
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
* review comments
* add timeout to tests
* Update net/levin/src/header.rs
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
---------
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
* monerod: set a random DB and remove dir when done
* add a test for an inbound monerod handshake
* don't fail if can't remove dir on windows
* Update test-utils/src/monerod.rs
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
* use `temp_dir`
* use `tempfile`
---------
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
* error: add `NeedsResize`
accidently removed, was `MapFull` before.
We need this because we as the writer thread must
react to this error in order to resize.
The writer thread doesn't have access to `heed::Error`, but
`Runtime::Error`, so this variant must exist
* env/backend: add `MANUAL_RESIZE` and `resize()`
* heed: respect `ReadersFull`, comment errors
* free: add `resize_memory_map()`
* env: add `Env::current_map_size`
* service: add `resize_map()`
* database: make `Env` itself cheaply clonable + threadsafe
`heed::Env` already uses `Arc` internally, but `sanakirja` does
not, so abstract this at the `Env` level instead of wrapping
`ConcreteEnv` in `Arc` ourselves.
* service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone`
* env: add `SYNCS_PER_TX`, `path()`, `shutdown()`
* database: add `src/service/state.rs`
* service: add to `state.rs`, add `DatabaseState` to readers/writer
* add `parking_lot`
Okay, turns out we need to take locks in
`database/src/service/state.rs`...
`std`'s lock fairness policies are not well defined and
depend on the OS implementation, `parking_lot` on the other hand
has a fairness policy which is important when the writer needs
the lock but readers keep pouring in, essentially never letting
the writer do stuff.
* state: use `crossbeam::atomic::AtomicCell`
We have crossbeam as a dep anyway.
* heed: `heed::Env` -> `Arc<RwLock<heed::Env>>`
* service: add reader shutdown handle, use `Select` for msgs
* service: remove `state.rs`
We don't need this, we will represent shutdowns with channel
messages and `Select`, and mutual exclusion with a `RwLock`.
* service: fix misc reader/writer stuff
* database: add `config.rs`
* service: use `ReaderThreads` when spawning readers
* service: impl `shutdown()` for readers/writer
* service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle`
Solves the issue of unfortunately timed `Request`s
that come in _right_ as we are shutting down.
If we (Cuprate) drop the database channels too early the requesting
thread will probably panic as they probably use `.unwrap()`,
never expecting a channel failure.
Returning this structure containing all the channels allows the
final shutdown code to carry these channels until the very end
of the program, at which point, all threads exit - no panics.
* remove `parking_lot`
Could be used as the database mutual exclusion lock.
Needs to be tested against `std`.
* config: add `path`
* env: `path()` -> `config()`, backend: impl `Drop`
* `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect
* backend: add `Config` to `ConcreteEnv`
* service: use `std:🧵:Builder` for readers/writer
* config: `PathBuf` -> `Cow<'static, Path>`
* misc docs, remove `RuntimeError::ShuttingDown`
* service: init & shutdown docs
* heed: impl `Env::resize_map()`
* heed: impl `Env::current_map_size()`
* lib.rs: add example crate usage test
* heed: `RwLock` comment
* helper: add `cuprate_database_dir()`
* config: use `cuprate_database_dir()`
* lib.rs: TODO example test
* database: add `page_size`
The database memory map size must be a multiple of
the OS page size. Why doesn't `heed` re-expose this?
It calls it when checking anyway...
https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772
* free: impl `resize_memory_map()`
* free: docs
* env: add `disk_size_bytes()`
* config: impl `From<$num>` for `ReaderThreads`
* move `fs`-related constants to `cuprate_helper::fs`
* docs
* add `resize.rs`, `ResizeAlgorithm`
* env: use `ResizeAlgorithm` in `resize_map()`
* TODO: account for LMDB reader limit
* resize: docs, add `page_size()`, impl `fixed_bytes()`
* resize: impl `percent()`
* resize: docs
* env: take `ResizeAlgorithm` by value (it impls `Copy`)
* heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED`
* config: `From<Into<usize>>` for `ReaderThreads`
Co-authored-by: Boog900 <boog900@tutanota.com>
* env: move mutual exclusion doc to backend
* free: update invariant doc
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/service/mod.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* fix `[allow(unused_imports)] // docs`
* move DB filename from `helper/` -> `database/`
* config: use `DATABASE_FILENAME`
* config: add `db_file_path()`
* lib: add non-`service` usage invariant docs
* table: seal `Table` trait, impl for all `crate::tables`
* fix docs
* fix docs pt.2
---------
Co-authored-by: Boog900 <boog900@tutanota.com>
