SyntheticBird/cuprate-hinto-janai
1
0
Fork 0
mirror of https://github.com/hinto-janai/cuprate.git synced 2024-12-23 03:59:37 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
cuprate-hinto-janai/database/Cargo.toml

58 lines
2.4 KiB
TOML
Raw Normal View History

initial database code (#6) * commit to start the draft pull request. added a space * Please don't look to close.It might hurt your eyes * impl associated types * errors, docs & divided ro/rw tx Added some more errors to DB_FAILURES, rewrited crates docs, and specified WriteTransaction subtype which implement write mode method. * more changes see description changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface. Also rewrited a declarative macro for tables. Will have to add Dummy Tables later. * small changes * Organized modules & implemented get_block_hash * write prototype & error Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode errors. Still have to rethink about it to resend a reference to the data that can't be encoded. * Multiple changes - hse.rs Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe code, the wrapper will be written outside of the project. - lib.rs Added a specific FailedToCommit error. (will investigate if really necessary). Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro. Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it. Added WriteCursor trait & methods, which is basically put & del. Added mandatory type for Cursors in Transaction & WriteTransactions Refactored get_block_hash interface method. - mdbx.rs Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented Added implementation of get & commit for Transaction * put mdbx as features with its dependency * save * refactored some method with macros * more mdbx errors, docs correction, moved to error.rs * finish nodup mdbx impl, errors.rs, macros, tables Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity. * bye bye match statement in mdbx.rs * defined all blockchain tables (not txpool) * dupsort/fixed support, dupcursor, basic block interface * tables, types, encoding and documentations Redefined all the database types from @Boog900's monero-rs db branch and added the needed implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was 5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations to modules and types. * replaced macros and added hfversion table * save * multiple changes * modified database schema. deprecated output global index and splited up pre-rct from rct output. * Fixed DupCursor function to return subkey (thx to rust turbofish inference). * Added some output functions * Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed. * fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index. * i hate lifetimes * read-only method now use read-only tx * initial output fn * some tx functions. Yes I'll refactor them * moved interface in a module * redefined errors, more tx fn, None->error * corrected a table + started blk fns * save * fixed TxOutputIdx + pop_block * IIRC I finished initial interface fns * fixed table name const + db build/check/open fn * switched important tables to dummy keys + rm blockhfversion * minor docs correction * fixed mentioned issues * make a test bin, just for fun * fixed issues + cargo fmt * removed monerod part * fixed a comment
2023-04-20 17:20:32 +00:00
[package]
Database (#35) * rename `database` -> `old_database` Keeping it around for reference until new implementation is complete. * create new `database/` skeleton * add `DATABASE.md` design doc skeleton * move design doc to `database/README.md` * add rough code * `lib.rs` -> `gist.rs` * database: use workspace deps * workspace: include `database/` as member CI will now include this crate. * cargo fmt * database: `AGPL` -> `MIT` * readme: add `TODO`s * add base files * cargo.toml: add `heed` feature * fix clippy * lib.rs: add extremely pedantic lints * readme: add `# Backends` * cargo.toml: add `cfg-if` * add `backend/` structure * base `database.rs` * cargo.toml: add `borsh` * backend: add `DATABASE_BACKEND` * base `error.rs` * base `database.rs` * base `transaction.rs` * base `table.rs` * lib.rs: add imports * add `pod.rs` * pod: use `Read/Write`, add tests for all primitive numbers * pod: impl Pod for `Vec<u8>`, `[u8; N]` * pod: add docs, add `private::Sealed` * pod: `to_writer`, `from_reader` The new `as_bytes` + `from_bytes` now allows (de)serializing from bytes directly instead of going through Read/Write. Different array return sizes are abstracted away with `-> impl AsRef<[u8]>` * pod: impl Pod for `Box<[u8]>` * pod: return `Err` on incorrect length in `from_bytes()` * pod: docs * pod: impl Pod for `Arc<[u8]>` * readme: docs * database: add `create_table()`, `get_table()` * table: `Pod` bound * backend: move into directories * pod: add `into_bytes()` * heed: impl `BytesEncode`, `BytesDecode` * add `actor`, `service` features The thread/actor system used will be gated behind `actor`, and the `tower/tokio` integration will be gated behind `service`. * add `lib.rs` docs * service: add `service.rs` * service: add `reader.rs` * service: add `writer.rs` * service: add `request.rs` & `response.rs` * cargo.toml: add `crossbeam` * service: add/use `enum Request`, `enum Response` * service: basic signatures for thread-pools, `Writer` -> `Writers` * service: split `tower::Service<ReadRequest/WriteRequest>` * service: impl `tower::Service` for `Database(Reader|Writer)` * service: add `init()`, impl basic `Reader/Writer` pools * service: add example `Request`'s * service: add example `ReadRequest` handling * temporarily allow clippy lints * readme: update file structure * transaction: add `RoTx::get_range()` * service: module docs * cargo.toml: add `cuprate-helper` * service: scale readers/writers based on thread count * database: change lifetimes * heed: impl Database for `heed` * heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits * docs * service: `read.rs` docs * service: `write.rs` docs * service: request/response docs * service: use `OnceLock` in `init()`, add `db_read()`, `db_write()` * service: leak database into `&'static` * database: add `#[inline]`, `#[cold]` * service: `free.rs` docs, more `#[inline]` + `#[cold]` * service: add `shutdown()` * service: add `Request::Shutdown` * service: `shutdown()` docs * heed: hide concrete tx types * lib.rs: add terms * split `Env` <-> `Database` * cargo.toml: add `paste` * database: add `tables/` * impl `serde/borsh` where possible * tables: add `Tables`, add test docs * make db backend mutually exclusive to fix `--all-features` * tables: use `$()*` in `tables!()` * cargo.toml: add `sanakirja 1.4.0` * sanakirja: impl `Env` * sanakirja: impl `Database` * sanakirja: impl `Transaction` * table: temporarily fix `sanakirja` K/V bounds * table: fix `#[cfg]` * cargo.toml: fix deps * lib.rs: docs * service: docs * readme: add files, update `# Documentation`, add `# Layers` * readme: `src/` file purpose * readme: `src/service/` file purpose * readme: `src/backend/` file purpose * fix `Cargo.lock` merge conflict * database: remove `gist.rs` * add to `constants.rs` * add top `//! comments` for files/modules * constants: add sanity-check test * service: add `only_one_database` test in `free.rs` * service: add `tests.rs` * remove unneeded markers + imports * backend: fix `get_range()`'s trait `impl` return * env: add `create_tables_if_needed()`, don't return `Option<Db>` * sort imports by `CONTRIBUTING.md` rules oops sorry boog * add `monero.rs` * monero: docs * database: add missing `RoTx/RwTx` inputs * backend: add missing `RoTx/RwTx` inputs * `monero.rs` trait -> free functions in `ops/` * pod: make methods infallible * ci: add `rustup update` step * service: use `Arc` instead of leaking, remove `db_read/db_write` * service: use `InfallibleOneshotReceiver` for readers * service: shutdown on error, add todos * service: remove `Request` * service: combine `ReadResponse` and `WriteResponse` * service: use `InfallibleOneshotReceiver` for writer * service: only spawn 1 writer, don't allow cloning writer handle * table: add associated `const CONSTANT_SIZE` * add `key.rs` + `trait Key`, add bound to `Table` * fix typos
2024-02-13 17:43:25 +00:00
name = "cuprate-database"
version = "0.0.0"
edition = "2021"
description = "Cuprate's database abstraction"
license = "MIT"
authors = ["hinto-janai"]
repository = "https://github.com/Cuprate/cuprate/tree/main/database"
keywords = ["cuprate", "database"]
initial database code (#6) * commit to start the draft pull request. added a space * Please don't look to close.It might hurt your eyes * impl associated types * errors, docs & divided ro/rw tx Added some more errors to DB_FAILURES, rewrited crates docs, and specified WriteTransaction subtype which implement write mode method. * more changes see description changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface. Also rewrited a declarative macro for tables. Will have to add Dummy Tables later. * small changes * Organized modules & implemented get_block_hash * write prototype & error Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode errors. Still have to rethink about it to resend a reference to the data that can't be encoded. * Multiple changes - hse.rs Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe code, the wrapper will be written outside of the project. - lib.rs Added a specific FailedToCommit error. (will investigate if really necessary). Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro. Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it. Added WriteCursor trait & methods, which is basically put & del. Added mandatory type for Cursors in Transaction & WriteTransactions Refactored get_block_hash interface method. - mdbx.rs Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented Added implementation of get & commit for Transaction * put mdbx as features with its dependency * save * refactored some method with macros * more mdbx errors, docs correction, moved to error.rs * finish nodup mdbx impl, errors.rs, macros, tables Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity. * bye bye match statement in mdbx.rs * defined all blockchain tables (not txpool) * dupsort/fixed support, dupcursor, basic block interface * tables, types, encoding and documentations Redefined all the database types from @Boog900's monero-rs db branch and added the needed implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was 5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations to modules and types. * replaced macros and added hfversion table * save * multiple changes * modified database schema. deprecated output global index and splited up pre-rct from rct output. * Fixed DupCursor function to return subkey (thx to rust turbofish inference). * Added some output functions * Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed. * fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index. * i hate lifetimes * read-only method now use read-only tx * initial output fn * some tx functions. Yes I'll refactor them * moved interface in a module * redefined errors, more tx fn, None->error * corrected a table + started blk fns * save * fixed TxOutputIdx + pop_block * IIRC I finished initial interface fns * fixed table name const + db build/check/open fn * switched important tables to dummy keys + rm blockhfversion * minor docs correction * fixed mentioned issues * make a test bin, just for fun * fixed issues + cargo fmt * removed monerod part * fixed a comment
2023-04-20 17:20:32 +00:00
[features]
database: implement `ops/` (#102) * ops: add `trait MoneroR{o,w}` * update `trait MoneroR{o,w}` bounds * types: add `BlockInfoLatest` type alias * block: impl most core functions * types: fix https://github.com/Cuprate/cuprate/pull/91#discussion_r1527668916 * fix table type test * cargo.toml: add `{cuprate-types, monero-serai}` * add_block: add all other block data * ops: remove unneeded `block` functions * env: add `EnvInner::open_db_rw_all()` * types: fix test * block: `&mut TxRw` -> `&TxRw`, use `open_db_rw_all()` * add `trait Tables[Mut]` and use it in `EnvInner` * block: use `TablesMut` * tables: replace manual impl with `define_trait_tables!()` * tables: docs for `trait Tables[Mut]` * tables: doc functions * create `call_fn_on_all_tables_or_early_return!()` macro * block: cleanup signatures + bodies * block: more fn's, docs * block: add `doc_{error,single,bulk}!()` * remove `ops/monero.rs` * move `height()` to `ops/blockchain.rs` * add `ops/macros.rs` * tx: add fn signatures * output: fix fn signatures * ops: expose `_inner()` functions * block: add `add_block_header{_bulk, _inner}()` * ops: remove doc_{fn,inner}!()` * ops: remove `_{inner,bulk}()`, lifetime + generics * update lib/mod docs * ops: add and use `doc_add_block_inner_invariant!()` * ops: add docs/return to inner `add_block()` functions * add_block(): extract and use fn for {key_image, output} * ops: more fn body impl + `add_block()` * cargo: add `monero-pruning` * ops: extract out `tx` functions from `add_block()` * property: add `db_version()` * ops: `pop_block()` body, remove other `pop_block` fn's * types: add `block_blob: Vec<u8>` to `VerifiedBlockInformation` * block: put `block_blob`, pass `Tables` to sub-functions `impl TablesMut` can't mutably pass multiple tables since it takes `&mut self`, so all functions unfortunately have to take a full `&mut impl TablesMut` even though they only need a few tables. * database: add `DatabaseRw::take()` useful for `pop_block()` where we need the value afterwards * block: deserialize tx's from `block_blobs` in `pop_block()` * blockchain: `height()` -> `chain_height()` * output: fix `amount_index` * ops: fix unlock_time, chain_height * `BlockInfoV{1,2,3}` -> `BlockInfo` * constants: add `DATABASE_VERSION` * database: add `DatabaseRw::update()` * output: use `DatabaseRw::update()` in `remove_output()` * add `TxBlobs` table, ignore pruning tables * block: mostly impl `add_block()` body * ops: comments * add_block: miner v2 tx commitment, height cast * block: impl `pop_block()` * block: mostly impl `get_block()` * block: impl `get_block_{from_height,header,header_from_height}` * add `OutputFlags` bitflags * add_block: u32::try_into(height: u32), use `OutputFlags` * tx: impl `get_{tx,tx_from_id}()` * tx: move docs tests to `#[test]` testing everything in 1 go is more natural since e.g: `add_tx()` is followed by `get_tx()` * tables: add `trait TablesIter`, `all_tables_empty()` This allows `TablesMut` to be a superset of `Tables` and use all its accessor functions. * use cuprate-test-utils, fix tx tests * block: `add_block()` take block by ref * tx: use all txs in tests * output: add `all_tx_functions()` test * add_block: check current height against input * block: map more fields in `get_block()` * block: remove `get_block()`, doc tests, fix `get_block_header()` * block: dummy values in test * heed: use `last/first()` instead of `unsafe` cursors We no longer have DUP semantics and also hard to debug errors were popping up on `del_current()`... * heed: fix `DatabaseRw::delete` Ok(true) means the key did not exist, so we must return Err(RuntimeError::KeyNotFound) * block: `add_block()` (dummy value) test * ops: `key_image` tests * cleanup, docs, tests * backend: test `take()` & `update()` * docs * remove `OutputFlags::NONE` * add_block(): add asserts, panic docs, `should_panic` tests * backend: remove `Ok(())` in `Database::delete` if already deleted redb already does this, so heed so match * block: move block operations after tx/outputs * `amount == 0` -> `amount == 1` * Nit: StorableVec::wrap_ref * `saturating_sub(1)` -> `- 1` * add `TxOutputs` table * add_block(): add to `tx_outputs` table * fix `DatabaseRo::update` * add_tx(): take `block_height` as input * tx: add/remove from `TxOutputs` table * output: remove if `amount == 1` -> `amount_index == 0` * output: fix `add_output()`'s `amount_index` calculation * output: fix `add_output()`'s `amount_index` calculation again * output: tests for `amount_index/num_outputs` * block: `num_outputs - 1` and `take()` -> `get()` We don't need to `take()` since the call afterwards to `remove_output()` removes the entry * block: swap `get_block_extended_header[_from_height]()` * move `{key_image,output}` handling `add_block()` -> `add_tx()` * blockchain: add doc to `top_block_height()` * block: manual panic -> `assert_eq!()` * test-utils: add `block_blob` to `VerifiedBlockInformation` field introduced in this PR * ops: use real block/tx data in tests * block: `total_generated_coins` -> `cumulative_generated_coins` * fix clippy, docs, TODOs * `cumulative_generated_coins()`: `block/` -> `blockchain/` * blockchain: add `cumulative_generated_coins()` tests * Update database/src/ops/block.rs Co-authored-by: Boog900 <boog900@tutanota.com> * `cumulative_generated_coins()` docs for pre-block-0 special case --------- Co-authored-by: Boog900 <boog900@tutanota.com>
2024-04-23 01:53:29 +00:00
default = ["heed", "redb", "service"]
database: return owned `T` in `Storable` (#92) * add `storable_slice.rs` * storable: `from_bytes(&[u8]) -> Self` * heed: return `T` in `Storable`, use `StorableSlice` in tests * remove `value_guard.rs` * redb: update `Storable` to return owned `T` * database: return `T` directly, remove `'a` lifetime * replace `[]` with `StorableSlice` * tables: add lifetime to `tables!()` and generated structs * backend: fix tests * heed: update fn signatures * storable: add `StorableVec`, remove `Storable::ALIGN` * tables/types: remove slice lifetimes, use `StorableVec` * backend: use `StorableVec`, fix tests * types: fix tests to use owned returned value * heed: fix `Storable` impl and tests * storable: `StorableVec` docs + tests * backend: assert values are detached from `get_range()` iterator * database: add docs * udpate docs, remove unneeded clippy `allow` * replace `ToOwnedDebug` with `std::fmt::Debug` * impl `StorableBytes` * backend: add `StorableBytes` to tests * readme: remove `to_owned_debug.rs` reference
2024-03-22 21:11:48 +00:00
# default = ["redb", "service"]
database: implement `ops/` (#102) * ops: add `trait MoneroR{o,w}` * update `trait MoneroR{o,w}` bounds * types: add `BlockInfoLatest` type alias * block: impl most core functions * types: fix https://github.com/Cuprate/cuprate/pull/91#discussion_r1527668916 * fix table type test * cargo.toml: add `{cuprate-types, monero-serai}` * add_block: add all other block data * ops: remove unneeded `block` functions * env: add `EnvInner::open_db_rw_all()` * types: fix test * block: `&mut TxRw` -> `&TxRw`, use `open_db_rw_all()` * add `trait Tables[Mut]` and use it in `EnvInner` * block: use `TablesMut` * tables: replace manual impl with `define_trait_tables!()` * tables: docs for `trait Tables[Mut]` * tables: doc functions * create `call_fn_on_all_tables_or_early_return!()` macro * block: cleanup signatures + bodies * block: more fn's, docs * block: add `doc_{error,single,bulk}!()` * remove `ops/monero.rs` * move `height()` to `ops/blockchain.rs` * add `ops/macros.rs` * tx: add fn signatures * output: fix fn signatures * ops: expose `_inner()` functions * block: add `add_block_header{_bulk, _inner}()` * ops: remove doc_{fn,inner}!()` * ops: remove `_{inner,bulk}()`, lifetime + generics * update lib/mod docs * ops: add and use `doc_add_block_inner_invariant!()` * ops: add docs/return to inner `add_block()` functions * add_block(): extract and use fn for {key_image, output} * ops: more fn body impl + `add_block()` * cargo: add `monero-pruning` * ops: extract out `tx` functions from `add_block()` * property: add `db_version()` * ops: `pop_block()` body, remove other `pop_block` fn's * types: add `block_blob: Vec<u8>` to `VerifiedBlockInformation` * block: put `block_blob`, pass `Tables` to sub-functions `impl TablesMut` can't mutably pass multiple tables since it takes `&mut self`, so all functions unfortunately have to take a full `&mut impl TablesMut` even though they only need a few tables. * database: add `DatabaseRw::take()` useful for `pop_block()` where we need the value afterwards * block: deserialize tx's from `block_blobs` in `pop_block()` * blockchain: `height()` -> `chain_height()` * output: fix `amount_index` * ops: fix unlock_time, chain_height * `BlockInfoV{1,2,3}` -> `BlockInfo` * constants: add `DATABASE_VERSION` * database: add `DatabaseRw::update()` * output: use `DatabaseRw::update()` in `remove_output()` * add `TxBlobs` table, ignore pruning tables * block: mostly impl `add_block()` body * ops: comments * add_block: miner v2 tx commitment, height cast * block: impl `pop_block()` * block: mostly impl `get_block()` * block: impl `get_block_{from_height,header,header_from_height}` * add `OutputFlags` bitflags * add_block: u32::try_into(height: u32), use `OutputFlags` * tx: impl `get_{tx,tx_from_id}()` * tx: move docs tests to `#[test]` testing everything in 1 go is more natural since e.g: `add_tx()` is followed by `get_tx()` * tables: add `trait TablesIter`, `all_tables_empty()` This allows `TablesMut` to be a superset of `Tables` and use all its accessor functions. * use cuprate-test-utils, fix tx tests * block: `add_block()` take block by ref * tx: use all txs in tests * output: add `all_tx_functions()` test * add_block: check current height against input * block: map more fields in `get_block()` * block: remove `get_block()`, doc tests, fix `get_block_header()` * block: dummy values in test * heed: use `last/first()` instead of `unsafe` cursors We no longer have DUP semantics and also hard to debug errors were popping up on `del_current()`... * heed: fix `DatabaseRw::delete` Ok(true) means the key did not exist, so we must return Err(RuntimeError::KeyNotFound) * block: `add_block()` (dummy value) test * ops: `key_image` tests * cleanup, docs, tests * backend: test `take()` & `update()` * docs * remove `OutputFlags::NONE` * add_block(): add asserts, panic docs, `should_panic` tests * backend: remove `Ok(())` in `Database::delete` if already deleted redb already does this, so heed so match * block: move block operations after tx/outputs * `amount == 0` -> `amount == 1` * Nit: StorableVec::wrap_ref * `saturating_sub(1)` -> `- 1` * add `TxOutputs` table * add_block(): add to `tx_outputs` table * fix `DatabaseRo::update` * add_tx(): take `block_height` as input * tx: add/remove from `TxOutputs` table * output: remove if `amount == 1` -> `amount_index == 0` * output: fix `add_output()`'s `amount_index` calculation * output: fix `add_output()`'s `amount_index` calculation again * output: tests for `amount_index/num_outputs` * block: `num_outputs - 1` and `take()` -> `get()` We don't need to `take()` since the call afterwards to `remove_output()` removes the entry * block: swap `get_block_extended_header[_from_height]()` * move `{key_image,output}` handling `add_block()` -> `add_tx()` * blockchain: add doc to `top_block_height()` * block: manual panic -> `assert_eq!()` * test-utils: add `block_blob` to `VerifiedBlockInformation` field introduced in this PR * ops: use real block/tx data in tests * block: `total_generated_coins` -> `cumulative_generated_coins` * fix clippy, docs, TODOs * `cumulative_generated_coins()`: `block/` -> `blockchain/` * blockchain: add `cumulative_generated_coins()` tests * Update database/src/ops/block.rs Co-authored-by: Boog900 <boog900@tutanota.com> * `cumulative_generated_coins()` docs for pre-block-0 special case --------- Co-authored-by: Boog900 <boog900@tutanota.com>
2024-04-23 01:53:29 +00:00
# default = ["redb-memory", "service"]
database: add `redb-memory` backend (#97) * add `redb-memory` feature * redb: use `redb::backend::InMemoryBackend` if enabled * readme: add `redb-memory` section * ci: add `redb-memory` testing * ci: remove `redb-memory` testing probably not worth adding time to CI for this
2024-03-27 00:50:10 +00:00
heed = ["dep:heed"]
redb = ["dep:redb"]
redb-memory = ["redb"]
service = ["dep:crossbeam", "dep:futures", "dep:tokio", "dep:tokio-util", "dep:tower", "dep:rayon"]
initial database code (#6) * commit to start the draft pull request. added a space * Please don't look to close.It might hurt your eyes * impl associated types * errors, docs & divided ro/rw tx Added some more errors to DB_FAILURES, rewrited crates docs, and specified WriteTransaction subtype which implement write mode method. * more changes see description changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface. Also rewrited a declarative macro for tables. Will have to add Dummy Tables later. * small changes * Organized modules & implemented get_block_hash * write prototype & error Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode errors. Still have to rethink about it to resend a reference to the data that can't be encoded. * Multiple changes - hse.rs Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe code, the wrapper will be written outside of the project. - lib.rs Added a specific FailedToCommit error. (will investigate if really necessary). Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro. Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it. Added WriteCursor trait & methods, which is basically put & del. Added mandatory type for Cursors in Transaction & WriteTransactions Refactored get_block_hash interface method. - mdbx.rs Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented Added implementation of get & commit for Transaction * put mdbx as features with its dependency * save * refactored some method with macros * more mdbx errors, docs correction, moved to error.rs * finish nodup mdbx impl, errors.rs, macros, tables Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity. * bye bye match statement in mdbx.rs * defined all blockchain tables (not txpool) * dupsort/fixed support, dupcursor, basic block interface * tables, types, encoding and documentations Redefined all the database types from @Boog900's monero-rs db branch and added the needed implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was 5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations to modules and types. * replaced macros and added hfversion table * save * multiple changes * modified database schema. deprecated output global index and splited up pre-rct from rct output. * Fixed DupCursor function to return subkey (thx to rust turbofish inference). * Added some output functions * Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed. * fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index. * i hate lifetimes * read-only method now use read-only tx * initial output fn * some tx functions. Yes I'll refactor them * moved interface in a module * redefined errors, more tx fn, None->error * corrected a table + started blk fns * save * fixed TxOutputIdx + pop_block * IIRC I finished initial interface fns * fixed table name const + db build/check/open fn * switched important tables to dummy keys + rm blockhfversion * minor docs correction * fixed mentioned issues * make a test bin, just for fun * fixed issues + cargo fmt * removed monerod part * fixed a comment
2023-04-20 17:20:32 +00:00
[dependencies]
database: implement `ops/` (#102) * ops: add `trait MoneroR{o,w}` * update `trait MoneroR{o,w}` bounds * types: add `BlockInfoLatest` type alias * block: impl most core functions * types: fix https://github.com/Cuprate/cuprate/pull/91#discussion_r1527668916 * fix table type test * cargo.toml: add `{cuprate-types, monero-serai}` * add_block: add all other block data * ops: remove unneeded `block` functions * env: add `EnvInner::open_db_rw_all()` * types: fix test * block: `&mut TxRw` -> `&TxRw`, use `open_db_rw_all()` * add `trait Tables[Mut]` and use it in `EnvInner` * block: use `TablesMut` * tables: replace manual impl with `define_trait_tables!()` * tables: docs for `trait Tables[Mut]` * tables: doc functions * create `call_fn_on_all_tables_or_early_return!()` macro * block: cleanup signatures + bodies * block: more fn's, docs * block: add `doc_{error,single,bulk}!()` * remove `ops/monero.rs` * move `height()` to `ops/blockchain.rs` * add `ops/macros.rs` * tx: add fn signatures * output: fix fn signatures * ops: expose `_inner()` functions * block: add `add_block_header{_bulk, _inner}()` * ops: remove doc_{fn,inner}!()` * ops: remove `_{inner,bulk}()`, lifetime + generics * update lib/mod docs * ops: add and use `doc_add_block_inner_invariant!()` * ops: add docs/return to inner `add_block()` functions * add_block(): extract and use fn for {key_image, output} * ops: more fn body impl + `add_block()` * cargo: add `monero-pruning` * ops: extract out `tx` functions from `add_block()` * property: add `db_version()` * ops: `pop_block()` body, remove other `pop_block` fn's * types: add `block_blob: Vec<u8>` to `VerifiedBlockInformation` * block: put `block_blob`, pass `Tables` to sub-functions `impl TablesMut` can't mutably pass multiple tables since it takes `&mut self`, so all functions unfortunately have to take a full `&mut impl TablesMut` even though they only need a few tables. * database: add `DatabaseRw::take()` useful for `pop_block()` where we need the value afterwards * block: deserialize tx's from `block_blobs` in `pop_block()` * blockchain: `height()` -> `chain_height()` * output: fix `amount_index` * ops: fix unlock_time, chain_height * `BlockInfoV{1,2,3}` -> `BlockInfo` * constants: add `DATABASE_VERSION` * database: add `DatabaseRw::update()` * output: use `DatabaseRw::update()` in `remove_output()` * add `TxBlobs` table, ignore pruning tables * block: mostly impl `add_block()` body * ops: comments * add_block: miner v2 tx commitment, height cast * block: impl `pop_block()` * block: mostly impl `get_block()` * block: impl `get_block_{from_height,header,header_from_height}` * add `OutputFlags` bitflags * add_block: u32::try_into(height: u32), use `OutputFlags` * tx: impl `get_{tx,tx_from_id}()` * tx: move docs tests to `#[test]` testing everything in 1 go is more natural since e.g: `add_tx()` is followed by `get_tx()` * tables: add `trait TablesIter`, `all_tables_empty()` This allows `TablesMut` to be a superset of `Tables` and use all its accessor functions. * use cuprate-test-utils, fix tx tests * block: `add_block()` take block by ref * tx: use all txs in tests * output: add `all_tx_functions()` test * add_block: check current height against input * block: map more fields in `get_block()` * block: remove `get_block()`, doc tests, fix `get_block_header()` * block: dummy values in test * heed: use `last/first()` instead of `unsafe` cursors We no longer have DUP semantics and also hard to debug errors were popping up on `del_current()`... * heed: fix `DatabaseRw::delete` Ok(true) means the key did not exist, so we must return Err(RuntimeError::KeyNotFound) * block: `add_block()` (dummy value) test * ops: `key_image` tests * cleanup, docs, tests * backend: test `take()` & `update()` * docs * remove `OutputFlags::NONE` * add_block(): add asserts, panic docs, `should_panic` tests * backend: remove `Ok(())` in `Database::delete` if already deleted redb already does this, so heed so match * block: move block operations after tx/outputs * `amount == 0` -> `amount == 1` * Nit: StorableVec::wrap_ref * `saturating_sub(1)` -> `- 1` * add `TxOutputs` table * add_block(): add to `tx_outputs` table * fix `DatabaseRo::update` * add_tx(): take `block_height` as input * tx: add/remove from `TxOutputs` table * output: remove if `amount == 1` -> `amount_index == 0` * output: fix `add_output()`'s `amount_index` calculation * output: fix `add_output()`'s `amount_index` calculation again * output: tests for `amount_index/num_outputs` * block: `num_outputs - 1` and `take()` -> `get()` We don't need to `take()` since the call afterwards to `remove_output()` removes the entry * block: swap `get_block_extended_header[_from_height]()` * move `{key_image,output}` handling `add_block()` -> `add_tx()` * blockchain: add doc to `top_block_height()` * block: manual panic -> `assert_eq!()` * test-utils: add `block_blob` to `VerifiedBlockInformation` field introduced in this PR * ops: use real block/tx data in tests * block: `total_generated_coins` -> `cumulative_generated_coins` * fix clippy, docs, TODOs * `cumulative_generated_coins()`: `block/` -> `blockchain/` * blockchain: add `cumulative_generated_coins()` tests * Update database/src/ops/block.rs Co-authored-by: Boog900 <boog900@tutanota.com> * `cumulative_generated_coins()` docs for pre-block-0 special case --------- Co-authored-by: Boog900 <boog900@tutanota.com>
2024-04-23 01:53:29 +00:00
bitflags = { workspace = true, features = ["serde", "bytemuck"] }
database: replace `Pod` with `bytemuck` (#81) * database: add `bytemuck` * database: add `types` module, replace `Pod` with `bytemuck` traits * types: docs * types: more docs * types: align safety msg * types: docs * misc docs * add `storable.rs` * add `slice.rs` * storable: impl `impl_storable_checked_bit_pattern!()` * database: TODO: fix `DatabaseRo::get_range` lifetimes * key/table: trait bound fixes * misc fixes * remove `borsh` - Doesn't work on must types - Probably won't use it anyway - Most things impl `serde` * key: add `new_with_max_secondary()` * key: add `new_with_max_secondary()` * heed: add `StorableHeed` for `Storable` compat * redb: add `StorableRedb{Key,Value}` for `Storable` compat * storable: add `Debug` bound and `fixed_width()` * redb: fix `'static` bound * storable: docs * `pod.rs` tests -> `storable.rs` * redb: add `Storable` tests * storable: add doc tests * redb: simplify `Storable` tests * heed: add `Storable` tests * misc docs/fixes * cargo: switch from forked `heed` -> `heed 0.20.0-alpha.9` * update readme * docs * fix README * table: remove `CONSTANT_SIZE` * database: `get()/delete() -> Err(KeyNotFound)` instead of `Option`
2024-03-03 22:26:39 +00:00
bytemuck = { version = "1.14.3", features = ["must_cast", "derive", "min_const_generics", "extern_crate_alloc"] }
database: return owned `T` in `Storable` (#92) * add `storable_slice.rs` * storable: `from_bytes(&[u8]) -> Self` * heed: return `T` in `Storable`, use `StorableSlice` in tests * remove `value_guard.rs` * redb: update `Storable` to return owned `T` * database: return `T` directly, remove `'a` lifetime * replace `[]` with `StorableSlice` * tables: add lifetime to `tables!()` and generated structs * backend: fix tests * heed: update fn signatures * storable: add `StorableVec`, remove `Storable::ALIGN` * tables/types: remove slice lifetimes, use `StorableVec` * backend: use `StorableVec`, fix tests * types: fix tests to use owned returned value * heed: fix `Storable` impl and tests * storable: `StorableVec` docs + tests * backend: assert values are detached from `get_range()` iterator * database: add docs * udpate docs, remove unneeded clippy `allow` * replace `ToOwnedDebug` with `std::fmt::Debug` * impl `StorableBytes` * backend: add `StorableBytes` to tests * readme: remove `to_owned_debug.rs` reference
2024-03-22 21:11:48 +00:00
bytes = { workspace = true }
database: replace `Pod` with `bytemuck` (#81) * database: add `bytemuck` * database: add `types` module, replace `Pod` with `bytemuck` traits * types: docs * types: more docs * types: align safety msg * types: docs * misc docs * add `storable.rs` * add `slice.rs` * storable: impl `impl_storable_checked_bit_pattern!()` * database: TODO: fix `DatabaseRo::get_range` lifetimes * key/table: trait bound fixes * misc fixes * remove `borsh` - Doesn't work on must types - Probably won't use it anyway - Most things impl `serde` * key: add `new_with_max_secondary()` * key: add `new_with_max_secondary()` * heed: add `StorableHeed` for `Storable` compat * redb: add `StorableRedb{Key,Value}` for `Storable` compat * storable: add `Debug` bound and `fixed_width()` * redb: fix `'static` bound * storable: docs * `pod.rs` tests -> `storable.rs` * redb: add `Storable` tests * storable: add doc tests * redb: simplify `Storable` tests * heed: add `Storable` tests * misc docs/fixes * cargo: switch from forked `heed` -> `heed 0.20.0-alpha.9` * update readme * docs * fix README * table: remove `CONSTANT_SIZE` * database: `get()/delete() -> Err(KeyNotFound)` instead of `Option`
2024-03-03 22:26:39 +00:00
cfg-if = { workspace = true }
database: Resizes, Shutdown, Flushing (#68) * error: add `NeedsResize` accidently removed, was `MapFull` before. We need this because we as the writer thread must react to this error in order to resize. The writer thread doesn't have access to `heed::Error`, but `Runtime::Error`, so this variant must exist * env/backend: add `MANUAL_RESIZE` and `resize()` * heed: respect `ReadersFull`, comment errors * free: add `resize_memory_map()` * env: add `Env::current_map_size` * service: add `resize_map()` * database: make `Env` itself cheaply clonable + threadsafe `heed::Env` already uses `Arc` internally, but `sanakirja` does not, so abstract this at the `Env` level instead of wrapping `ConcreteEnv` in `Arc` ourselves. * service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone` * env: add `SYNCS_PER_TX`, `path()`, `shutdown()` * database: add `src/service/state.rs` * service: add to `state.rs`, add `DatabaseState` to readers/writer * add `parking_lot` Okay, turns out we need to take locks in `database/src/service/state.rs`... `std`'s lock fairness policies are not well defined and depend on the OS implementation, `parking_lot` on the other hand has a fairness policy which is important when the writer needs the lock but readers keep pouring in, essentially never letting the writer do stuff. * state: use `crossbeam::atomic::AtomicCell` We have crossbeam as a dep anyway. * heed: `heed::Env` -> `Arc<RwLock<heed::Env>>` * service: add reader shutdown handle, use `Select` for msgs * service: remove `state.rs` We don't need this, we will represent shutdowns with channel messages and `Select`, and mutual exclusion with a `RwLock`. * service: fix misc reader/writer stuff * database: add `config.rs` * service: use `ReaderThreads` when spawning readers * service: impl `shutdown()` for readers/writer * service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle` Solves the issue of unfortunately timed `Request`s that come in _right_ as we are shutting down. If we (Cuprate) drop the database channels too early the requesting thread will probably panic as they probably use `.unwrap()`, never expecting a channel failure. Returning this structure containing all the channels allows the final shutdown code to carry these channels until the very end of the program, at which point, all threads exit - no panics. * remove `parking_lot` Could be used as the database mutual exclusion lock. Needs to be tested against `std`. * config: add `path` * env: `path()` -> `config()`, backend: impl `Drop` * `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect * backend: add `Config` to `ConcreteEnv` * service: use `std::thread::Builder` for readers/writer * config: `PathBuf` -> `Cow<'static, Path>` * misc docs, remove `RuntimeError::ShuttingDown` * service: init & shutdown docs * heed: impl `Env::resize_map()` * heed: impl `Env::current_map_size()` * lib.rs: add example crate usage test * heed: `RwLock` comment * helper: add `cuprate_database_dir()` * config: use `cuprate_database_dir()` * lib.rs: TODO example test * database: add `page_size` The database memory map size must be a multiple of the OS page size. Why doesn't `heed` re-expose this? It calls it when checking anyway... https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772 * free: impl `resize_memory_map()` * free: docs * env: add `disk_size_bytes()` * config: impl `From<$num>` for `ReaderThreads` * move `fs`-related constants to `cuprate_helper::fs` * docs * add `resize.rs`, `ResizeAlgorithm` * env: use `ResizeAlgorithm` in `resize_map()` * TODO: account for LMDB reader limit * resize: docs, add `page_size()`, impl `fixed_bytes()` * resize: impl `percent()` * resize: docs * env: take `ResizeAlgorithm` by value (it impls `Copy`) * heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED` * config: `From<Into<usize>>` for `ReaderThreads` Co-authored-by: Boog900 <boog900@tutanota.com> * env: move mutual exclusion doc to backend * free: update invariant doc Co-authored-by: Boog900 <boog900@tutanota.com> * Update database/src/service/mod.rs Co-authored-by: Boog900 <boog900@tutanota.com> * fix `[allow(unused_imports)] // docs` * move DB filename from `helper/` -> `database/` * config: use `DATABASE_FILENAME` * config: add `db_file_path()` * lib: add non-`service` usage invariant docs * table: seal `Table` trait, impl for all `crate::tables` * fix docs * fix docs pt.2 --------- Co-authored-by: Boog900 <boog900@tutanota.com>
2024-02-25 19:46:36 +00:00
# FIXME:
# We only need the `thread` feature if `service` is enabled.
# Figure out how to enable features of an already pulled in dependency conditionally.
database: split `cumulative_difficulty` into low/high bits (#114) * types: split `cumulative_difficulty` into low/high bits * helper: add `map` module * database: use `helper`'s cumulative_diff functions * helper: rename functions splitting bits isn't necessarily `cumulative_difficulty` specific * database: fix tests * helper: docs * helper: test output is low-endian bits * helper: docs * Update helper/src/map.rs Co-authored-by: Boog900 <boog900@tutanota.com> * Update helper/src/map.rs Co-authored-by: Boog900 <boog900@tutanota.com> --------- Co-authored-by: Boog900 <boog900@tutanota.com>
2024-04-24 20:47:48 +00:00
cuprate-helper = { path = "../helper", features = ["fs", "thread", "map"] }
database: implement `ops/` (#102) * ops: add `trait MoneroR{o,w}` * update `trait MoneroR{o,w}` bounds * types: add `BlockInfoLatest` type alias * block: impl most core functions * types: fix https://github.com/Cuprate/cuprate/pull/91#discussion_r1527668916 * fix table type test * cargo.toml: add `{cuprate-types, monero-serai}` * add_block: add all other block data * ops: remove unneeded `block` functions * env: add `EnvInner::open_db_rw_all()` * types: fix test * block: `&mut TxRw` -> `&TxRw`, use `open_db_rw_all()` * add `trait Tables[Mut]` and use it in `EnvInner` * block: use `TablesMut` * tables: replace manual impl with `define_trait_tables!()` * tables: docs for `trait Tables[Mut]` * tables: doc functions * create `call_fn_on_all_tables_or_early_return!()` macro * block: cleanup signatures + bodies * block: more fn's, docs * block: add `doc_{error,single,bulk}!()` * remove `ops/monero.rs` * move `height()` to `ops/blockchain.rs` * add `ops/macros.rs` * tx: add fn signatures * output: fix fn signatures * ops: expose `_inner()` functions * block: add `add_block_header{_bulk, _inner}()` * ops: remove doc_{fn,inner}!()` * ops: remove `_{inner,bulk}()`, lifetime + generics * update lib/mod docs * ops: add and use `doc_add_block_inner_invariant!()` * ops: add docs/return to inner `add_block()` functions * add_block(): extract and use fn for {key_image, output} * ops: more fn body impl + `add_block()` * cargo: add `monero-pruning` * ops: extract out `tx` functions from `add_block()` * property: add `db_version()` * ops: `pop_block()` body, remove other `pop_block` fn's * types: add `block_blob: Vec<u8>` to `VerifiedBlockInformation` * block: put `block_blob`, pass `Tables` to sub-functions `impl TablesMut` can't mutably pass multiple tables since it takes `&mut self`, so all functions unfortunately have to take a full `&mut impl TablesMut` even though they only need a few tables. * database: add `DatabaseRw::take()` useful for `pop_block()` where we need the value afterwards * block: deserialize tx's from `block_blobs` in `pop_block()` * blockchain: `height()` -> `chain_height()` * output: fix `amount_index` * ops: fix unlock_time, chain_height * `BlockInfoV{1,2,3}` -> `BlockInfo` * constants: add `DATABASE_VERSION` * database: add `DatabaseRw::update()` * output: use `DatabaseRw::update()` in `remove_output()` * add `TxBlobs` table, ignore pruning tables * block: mostly impl `add_block()` body * ops: comments * add_block: miner v2 tx commitment, height cast * block: impl `pop_block()` * block: mostly impl `get_block()` * block: impl `get_block_{from_height,header,header_from_height}` * add `OutputFlags` bitflags * add_block: u32::try_into(height: u32), use `OutputFlags` * tx: impl `get_{tx,tx_from_id}()` * tx: move docs tests to `#[test]` testing everything in 1 go is more natural since e.g: `add_tx()` is followed by `get_tx()` * tables: add `trait TablesIter`, `all_tables_empty()` This allows `TablesMut` to be a superset of `Tables` and use all its accessor functions. * use cuprate-test-utils, fix tx tests * block: `add_block()` take block by ref * tx: use all txs in tests * output: add `all_tx_functions()` test * add_block: check current height against input * block: map more fields in `get_block()` * block: remove `get_block()`, doc tests, fix `get_block_header()` * block: dummy values in test * heed: use `last/first()` instead of `unsafe` cursors We no longer have DUP semantics and also hard to debug errors were popping up on `del_current()`... * heed: fix `DatabaseRw::delete` Ok(true) means the key did not exist, so we must return Err(RuntimeError::KeyNotFound) * block: `add_block()` (dummy value) test * ops: `key_image` tests * cleanup, docs, tests * backend: test `take()` & `update()` * docs * remove `OutputFlags::NONE` * add_block(): add asserts, panic docs, `should_panic` tests * backend: remove `Ok(())` in `Database::delete` if already deleted redb already does this, so heed so match * block: move block operations after tx/outputs * `amount == 0` -> `amount == 1` * Nit: StorableVec::wrap_ref * `saturating_sub(1)` -> `- 1` * add `TxOutputs` table * add_block(): add to `tx_outputs` table * fix `DatabaseRo::update` * add_tx(): take `block_height` as input * tx: add/remove from `TxOutputs` table * output: remove if `amount == 1` -> `amount_index == 0` * output: fix `add_output()`'s `amount_index` calculation * output: fix `add_output()`'s `amount_index` calculation again * output: tests for `amount_index/num_outputs` * block: `num_outputs - 1` and `take()` -> `get()` We don't need to `take()` since the call afterwards to `remove_output()` removes the entry * block: swap `get_block_extended_header[_from_height]()` * move `{key_image,output}` handling `add_block()` -> `add_tx()` * blockchain: add doc to `top_block_height()` * block: manual panic -> `assert_eq!()` * test-utils: add `block_blob` to `VerifiedBlockInformation` field introduced in this PR * ops: use real block/tx data in tests * block: `total_generated_coins` -> `cumulative_generated_coins` * fix clippy, docs, TODOs * `cumulative_generated_coins()`: `block/` -> `blockchain/` * blockchain: add `cumulative_generated_coins()` tests * Update database/src/ops/block.rs Co-authored-by: Boog900 <boog900@tutanota.com> * `cumulative_generated_coins()` docs for pre-block-0 special case --------- Co-authored-by: Boog900 <boog900@tutanota.com>
2024-04-23 01:53:29 +00:00
cuprate-types = { path = "../types", features = ["service"] }
curve25519-dalek = { workspace = true }
monero-pruning = { path = "../pruning" }
monero-serai = { workspace = true, features = ["std"] }
paste = { workspace = true }
page_size = { version = "0.6.0" } # Needed for database resizes, they must be a multiple of the OS page size.
thiserror = { workspace = true }
Database (#35) * rename `database` -> `old_database` Keeping it around for reference until new implementation is complete. * create new `database/` skeleton * add `DATABASE.md` design doc skeleton * move design doc to `database/README.md` * add rough code * `lib.rs` -> `gist.rs` * database: use workspace deps * workspace: include `database/` as member CI will now include this crate. * cargo fmt * database: `AGPL` -> `MIT` * readme: add `TODO`s * add base files * cargo.toml: add `heed` feature * fix clippy * lib.rs: add extremely pedantic lints * readme: add `# Backends` * cargo.toml: add `cfg-if` * add `backend/` structure * base `database.rs` * cargo.toml: add `borsh` * backend: add `DATABASE_BACKEND` * base `error.rs` * base `database.rs` * base `transaction.rs` * base `table.rs` * lib.rs: add imports * add `pod.rs` * pod: use `Read/Write`, add tests for all primitive numbers * pod: impl Pod for `Vec<u8>`, `[u8; N]` * pod: add docs, add `private::Sealed` * pod: `to_writer`, `from_reader` The new `as_bytes` + `from_bytes` now allows (de)serializing from bytes directly instead of going through Read/Write. Different array return sizes are abstracted away with `-> impl AsRef<[u8]>` * pod: impl Pod for `Box<[u8]>` * pod: return `Err` on incorrect length in `from_bytes()` * pod: docs * pod: impl Pod for `Arc<[u8]>` * readme: docs * database: add `create_table()`, `get_table()` * table: `Pod` bound * backend: move into directories * pod: add `into_bytes()` * heed: impl `BytesEncode`, `BytesDecode` * add `actor`, `service` features The thread/actor system used will be gated behind `actor`, and the `tower/tokio` integration will be gated behind `service`. * add `lib.rs` docs * service: add `service.rs` * service: add `reader.rs` * service: add `writer.rs` * service: add `request.rs` & `response.rs` * cargo.toml: add `crossbeam` * service: add/use `enum Request`, `enum Response` * service: basic signatures for thread-pools, `Writer` -> `Writers` * service: split `tower::Service<ReadRequest/WriteRequest>` * service: impl `tower::Service` for `Database(Reader|Writer)` * service: add `init()`, impl basic `Reader/Writer` pools * service: add example `Request`'s * service: add example `ReadRequest` handling * temporarily allow clippy lints * readme: update file structure * transaction: add `RoTx::get_range()` * service: module docs * cargo.toml: add `cuprate-helper` * service: scale readers/writers based on thread count * database: change lifetimes * heed: impl Database for `heed` * heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits * docs * service: `read.rs` docs * service: `write.rs` docs * service: request/response docs * service: use `OnceLock` in `init()`, add `db_read()`, `db_write()` * service: leak database into `&'static` * database: add `#[inline]`, `#[cold]` * service: `free.rs` docs, more `#[inline]` + `#[cold]` * service: add `shutdown()` * service: add `Request::Shutdown` * service: `shutdown()` docs * heed: hide concrete tx types * lib.rs: add terms * split `Env` <-> `Database` * cargo.toml: add `paste` * database: add `tables/` * impl `serde/borsh` where possible * tables: add `Tables`, add test docs * make db backend mutually exclusive to fix `--all-features` * tables: use `$()*` in `tables!()` * cargo.toml: add `sanakirja 1.4.0` * sanakirja: impl `Env` * sanakirja: impl `Database` * sanakirja: impl `Transaction` * table: temporarily fix `sanakirja` K/V bounds * table: fix `#[cfg]` * cargo.toml: fix deps * lib.rs: docs * service: docs * readme: add files, update `# Documentation`, add `# Layers` * readme: `src/` file purpose * readme: `src/service/` file purpose * readme: `src/backend/` file purpose * fix `Cargo.lock` merge conflict * database: remove `gist.rs` * add to `constants.rs` * add top `//! comments` for files/modules * constants: add sanity-check test * service: add `only_one_database` test in `free.rs` * service: add `tests.rs` * remove unneeded markers + imports * backend: fix `get_range()`'s trait `impl` return * env: add `create_tables_if_needed()`, don't return `Option<Db>` * sort imports by `CONTRIBUTING.md` rules oops sorry boog * add `monero.rs` * monero: docs * database: add missing `RoTx/RwTx` inputs * backend: add missing `RoTx/RwTx` inputs * `monero.rs` trait -> free functions in `ops/` * pod: make methods infallible * ci: add `rustup update` step * service: use `Arc` instead of leaking, remove `db_read/db_write` * service: use `InfallibleOneshotReceiver` for readers * service: shutdown on error, add todos * service: remove `Request` * service: combine `ReadResponse` and `WriteResponse` * service: use `InfallibleOneshotReceiver` for writer * service: only spawn 1 writer, don't allow cloning writer handle * table: add associated `const CONSTANT_SIZE` * add `key.rs` + `trait Key`, add bound to `Table` * fix typos
2024-02-13 17:43:25 +00:00
# `service` feature.
database: use `rayon` for `service`'s reader thread-pool (#93) * add `rayon 1.9.0` * service: re-impl reader threadpool with `rayon` * service: impl `tower::Service` for writer * backend: create db dir in `Env::open` * service: read + write request/response tests * docs, name changes * service: always return `Poll::Ready` in writer * service: use `spawn()` instead of `install()` * service: replace `DatabaseReader` with free functions * cargo: add `tokio-utils` * service: acquire permit before `call()` for read requests * service: acquire permit in tests * docs * service: use loop for write request tests * service: use `ready!()`
2024-03-21 20:16:12 +00:00
crossbeam = { workspace = true, features = ["std"], optional = true }
futures = { workspace = true, optional = true }
tokio = { workspace = true, features = ["full"], optional = true }
tokio-util = { workspace = true, features = ["full"], optional = true }
tower = { workspace = true, features = ["full"], optional = true }
rayon = { workspace = true, optional = true }
initial database code (#6) * commit to start the draft pull request. added a space * Please don't look to close.It might hurt your eyes * impl associated types * errors, docs & divided ro/rw tx Added some more errors to DB_FAILURES, rewrited crates docs, and specified WriteTransaction subtype which implement write mode method. * more changes see description changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface. Also rewrited a declarative macro for tables. Will have to add Dummy Tables later. * small changes * Organized modules & implemented get_block_hash * write prototype & error Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode errors. Still have to rethink about it to resend a reference to the data that can't be encoded. * Multiple changes - hse.rs Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe code, the wrapper will be written outside of the project. - lib.rs Added a specific FailedToCommit error. (will investigate if really necessary). Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro. Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it. Added WriteCursor trait & methods, which is basically put & del. Added mandatory type for Cursors in Transaction & WriteTransactions Refactored get_block_hash interface method. - mdbx.rs Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented Added implementation of get & commit for Transaction * put mdbx as features with its dependency * save * refactored some method with macros * more mdbx errors, docs correction, moved to error.rs * finish nodup mdbx impl, errors.rs, macros, tables Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity. * bye bye match statement in mdbx.rs * defined all blockchain tables (not txpool) * dupsort/fixed support, dupcursor, basic block interface * tables, types, encoding and documentations Redefined all the database types from @Boog900's monero-rs db branch and added the needed implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was 5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations to modules and types. * replaced macros and added hfversion table * save * multiple changes * modified database schema. deprecated output global index and splited up pre-rct from rct output. * Fixed DupCursor function to return subkey (thx to rust turbofish inference). * Added some output functions * Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed. * fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index. * i hate lifetimes * read-only method now use read-only tx * initial output fn * some tx functions. Yes I'll refactor them * moved interface in a module * redefined errors, more tx fn, None->error * corrected a table + started blk fns * save * fixed TxOutputIdx + pop_block * IIRC I finished initial interface fns * fixed table name const + db build/check/open fn * switched important tables to dummy keys + rm blockhfversion * minor docs correction * fixed mentioned issues * make a test bin, just for fun * fixed issues + cargo fmt * removed monerod part * fixed a comment
2023-04-20 17:20:32 +00:00
Database (#35) * rename `database` -> `old_database` Keeping it around for reference until new implementation is complete. * create new `database/` skeleton * add `DATABASE.md` design doc skeleton * move design doc to `database/README.md` * add rough code * `lib.rs` -> `gist.rs` * database: use workspace deps * workspace: include `database/` as member CI will now include this crate. * cargo fmt * database: `AGPL` -> `MIT` * readme: add `TODO`s * add base files * cargo.toml: add `heed` feature * fix clippy * lib.rs: add extremely pedantic lints * readme: add `# Backends` * cargo.toml: add `cfg-if` * add `backend/` structure * base `database.rs` * cargo.toml: add `borsh` * backend: add `DATABASE_BACKEND` * base `error.rs` * base `database.rs` * base `transaction.rs` * base `table.rs` * lib.rs: add imports * add `pod.rs` * pod: use `Read/Write`, add tests for all primitive numbers * pod: impl Pod for `Vec<u8>`, `[u8; N]` * pod: add docs, add `private::Sealed` * pod: `to_writer`, `from_reader` The new `as_bytes` + `from_bytes` now allows (de)serializing from bytes directly instead of going through Read/Write. Different array return sizes are abstracted away with `-> impl AsRef<[u8]>` * pod: impl Pod for `Box<[u8]>` * pod: return `Err` on incorrect length in `from_bytes()` * pod: docs * pod: impl Pod for `Arc<[u8]>` * readme: docs * database: add `create_table()`, `get_table()` * table: `Pod` bound * backend: move into directories * pod: add `into_bytes()` * heed: impl `BytesEncode`, `BytesDecode` * add `actor`, `service` features The thread/actor system used will be gated behind `actor`, and the `tower/tokio` integration will be gated behind `service`. * add `lib.rs` docs * service: add `service.rs` * service: add `reader.rs` * service: add `writer.rs` * service: add `request.rs` & `response.rs` * cargo.toml: add `crossbeam` * service: add/use `enum Request`, `enum Response` * service: basic signatures for thread-pools, `Writer` -> `Writers` * service: split `tower::Service<ReadRequest/WriteRequest>` * service: impl `tower::Service` for `Database(Reader|Writer)` * service: add `init()`, impl basic `Reader/Writer` pools * service: add example `Request`'s * service: add example `ReadRequest` handling * temporarily allow clippy lints * readme: update file structure * transaction: add `RoTx::get_range()` * service: module docs * cargo.toml: add `cuprate-helper` * service: scale readers/writers based on thread count * database: change lifetimes * heed: impl Database for `heed` * heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits * docs * service: `read.rs` docs * service: `write.rs` docs * service: request/response docs * service: use `OnceLock` in `init()`, add `db_read()`, `db_write()` * service: leak database into `&'static` * database: add `#[inline]`, `#[cold]` * service: `free.rs` docs, more `#[inline]` + `#[cold]` * service: add `shutdown()` * service: add `Request::Shutdown` * service: `shutdown()` docs * heed: hide concrete tx types * lib.rs: add terms * split `Env` <-> `Database` * cargo.toml: add `paste` * database: add `tables/` * impl `serde/borsh` where possible * tables: add `Tables`, add test docs * make db backend mutually exclusive to fix `--all-features` * tables: use `$()*` in `tables!()` * cargo.toml: add `sanakirja 1.4.0` * sanakirja: impl `Env` * sanakirja: impl `Database` * sanakirja: impl `Transaction` * table: temporarily fix `sanakirja` K/V bounds * table: fix `#[cfg]` * cargo.toml: fix deps * lib.rs: docs * service: docs * readme: add files, update `# Documentation`, add `# Layers` * readme: `src/` file purpose * readme: `src/service/` file purpose * readme: `src/backend/` file purpose * fix `Cargo.lock` merge conflict * database: remove `gist.rs` * add to `constants.rs` * add top `//! comments` for files/modules * constants: add sanity-check test * service: add `only_one_database` test in `free.rs` * service: add `tests.rs` * remove unneeded markers + imports * backend: fix `get_range()`'s trait `impl` return * env: add `create_tables_if_needed()`, don't return `Option<Db>` * sort imports by `CONTRIBUTING.md` rules oops sorry boog * add `monero.rs` * monero: docs * database: add missing `RoTx/RwTx` inputs * backend: add missing `RoTx/RwTx` inputs * `monero.rs` trait -> free functions in `ops/` * pod: make methods infallible * ci: add `rustup update` step * service: use `Arc` instead of leaking, remove `db_read/db_write` * service: use `InfallibleOneshotReceiver` for readers * service: shutdown on error, add todos * service: remove `Request` * service: combine `ReadResponse` and `WriteResponse` * service: use `InfallibleOneshotReceiver` for writer * service: only spawn 1 writer, don't allow cloning writer handle * table: add associated `const CONSTANT_SIZE` * add `key.rs` + `trait Key`, add bound to `Table` * fix typos
2024-02-13 17:43:25 +00:00
# Optional features.
database: update backends (#119) * `heed 0.20.0-alpha.9 -> 0.20.0`, `redb 2.0.0 -> 2.1.0` * heed: fix * redb: fix
2024-04-29 00:12:42 +00:00
heed = { version = "0.20.0", optional = true }
redb = { version = "2.1.0", optional = true }
database: replace `sanakirja` with `redb` (#80) * cargo: replace `sanakirja` with `redb` * database: update docs `sanakirja` -> `redb` * lib: add TODO for `ConcreteEnv` generic replacement * database: split `trait Database` -> `trait Database{Read,Write}` * heed: add `struct HeedTable{Ro,Rw}` to match `redb` behavior * ops: remove imports for now * env: fix `&mut` bound on RwTx * database: impl `redb`, type-checks * fix heed trait impls, `Database{Read,Write}` -> `Database{Ro,Rw}` * redb: impl `From<_>` for `RuntimeError` * update readme * heed: document `HeedTableR{o,w}` types * env: doc `sync()` invariant * database: document data & lock filenames * misc docs, `redb` durability impl, `'db` -> `'env` * redb: fixes * misc docs and fixes * Update database/README.md Co-authored-by: Boog900 <boog900@tutanota.com> --------- Co-authored-by: Boog900 <boog900@tutanota.com>
2024-02-29 17:40:15 +00:00
serde = { workspace = true, optional = true }
database: Resizes, Shutdown, Flushing (#68) * error: add `NeedsResize` accidently removed, was `MapFull` before. We need this because we as the writer thread must react to this error in order to resize. The writer thread doesn't have access to `heed::Error`, but `Runtime::Error`, so this variant must exist * env/backend: add `MANUAL_RESIZE` and `resize()` * heed: respect `ReadersFull`, comment errors * free: add `resize_memory_map()` * env: add `Env::current_map_size` * service: add `resize_map()` * database: make `Env` itself cheaply clonable + threadsafe `heed::Env` already uses `Arc` internally, but `sanakirja` does not, so abstract this at the `Env` level instead of wrapping `ConcreteEnv` in `Arc` ourselves. * service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone` * env: add `SYNCS_PER_TX`, `path()`, `shutdown()` * database: add `src/service/state.rs` * service: add to `state.rs`, add `DatabaseState` to readers/writer * add `parking_lot` Okay, turns out we need to take locks in `database/src/service/state.rs`... `std`'s lock fairness policies are not well defined and depend on the OS implementation, `parking_lot` on the other hand has a fairness policy which is important when the writer needs the lock but readers keep pouring in, essentially never letting the writer do stuff. * state: use `crossbeam::atomic::AtomicCell` We have crossbeam as a dep anyway. * heed: `heed::Env` -> `Arc<RwLock<heed::Env>>` * service: add reader shutdown handle, use `Select` for msgs * service: remove `state.rs` We don't need this, we will represent shutdowns with channel messages and `Select`, and mutual exclusion with a `RwLock`. * service: fix misc reader/writer stuff * database: add `config.rs` * service: use `ReaderThreads` when spawning readers * service: impl `shutdown()` for readers/writer * service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle` Solves the issue of unfortunately timed `Request`s that come in _right_ as we are shutting down. If we (Cuprate) drop the database channels too early the requesting thread will probably panic as they probably use `.unwrap()`, never expecting a channel failure. Returning this structure containing all the channels allows the final shutdown code to carry these channels until the very end of the program, at which point, all threads exit - no panics. * remove `parking_lot` Could be used as the database mutual exclusion lock. Needs to be tested against `std`. * config: add `path` * env: `path()` -> `config()`, backend: impl `Drop` * `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect * backend: add `Config` to `ConcreteEnv` * service: use `std::thread::Builder` for readers/writer * config: `PathBuf` -> `Cow<'static, Path>` * misc docs, remove `RuntimeError::ShuttingDown` * service: init & shutdown docs * heed: impl `Env::resize_map()` * heed: impl `Env::current_map_size()` * lib.rs: add example crate usage test * heed: `RwLock` comment * helper: add `cuprate_database_dir()` * config: use `cuprate_database_dir()` * lib.rs: TODO example test * database: add `page_size` The database memory map size must be a multiple of the OS page size. Why doesn't `heed` re-expose this? It calls it when checking anyway... https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772 * free: impl `resize_memory_map()` * free: docs * env: add `disk_size_bytes()` * config: impl `From<$num>` for `ReaderThreads` * move `fs`-related constants to `cuprate_helper::fs` * docs * add `resize.rs`, `ResizeAlgorithm` * env: use `ResizeAlgorithm` in `resize_map()` * TODO: account for LMDB reader limit * resize: docs, add `page_size()`, impl `fixed_bytes()` * resize: impl `percent()` * resize: docs * env: take `ResizeAlgorithm` by value (it impls `Copy`) * heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED` * config: `From<Into<usize>>` for `ReaderThreads` Co-authored-by: Boog900 <boog900@tutanota.com> * env: move mutual exclusion doc to backend * free: update invariant doc Co-authored-by: Boog900 <boog900@tutanota.com> * Update database/src/service/mod.rs Co-authored-by: Boog900 <boog900@tutanota.com> * fix `[allow(unused_imports)] // docs` * move DB filename from `helper/` -> `database/` * config: use `DATABASE_FILENAME` * config: add `db_file_path()` * lib: add non-`service` usage invariant docs * table: seal `Table` trait, impl for all `crate::tables` * fix docs * fix docs pt.2 --------- Co-authored-by: Boog900 <boog900@tutanota.com>
2024-02-25 19:46:36 +00:00
[dev-dependencies]
database: replace `Pod` with `bytemuck` (#81) * database: add `bytemuck` * database: add `types` module, replace `Pod` with `bytemuck` traits * types: docs * types: more docs * types: align safety msg * types: docs * misc docs * add `storable.rs` * add `slice.rs` * storable: impl `impl_storable_checked_bit_pattern!()` * database: TODO: fix `DatabaseRo::get_range` lifetimes * key/table: trait bound fixes * misc fixes * remove `borsh` - Doesn't work on must types - Probably won't use it anyway - Most things impl `serde` * key: add `new_with_max_secondary()` * key: add `new_with_max_secondary()` * heed: add `StorableHeed` for `Storable` compat * redb: add `StorableRedb{Key,Value}` for `Storable` compat * storable: add `Debug` bound and `fixed_width()` * redb: fix `'static` bound * storable: docs * `pod.rs` tests -> `storable.rs` * redb: add `Storable` tests * storable: add doc tests * redb: simplify `Storable` tests * heed: add `Storable` tests * misc docs/fixes * cargo: switch from forked `heed` -> `heed 0.20.0-alpha.9` * update readme * docs * fix README * table: remove `CONSTANT_SIZE` * database: `get()/delete() -> Err(KeyNotFound)` instead of `Option`
2024-03-03 22:26:39 +00:00
bytemuck = { version = "1.14.3", features = ["must_cast", "derive", "min_const_generics", "extern_crate_alloc"] }
database: docs & small fixes for `Config` (#77) * config: `ReaderThreads::from(0)` -> `ReaderThreads::OnePerThread` * config: all docs, impl `as_threads()` * config: fix doc link * config: add `db_directory()` and make field private * config: remove `P: AsRef<Path>` We need a `PathBuf` anyway, make the caller create one. * config: impl `Debug`, add `Config::default()` test * config: more docs * fix doc links * config: review fix * Update database/src/config.rs --------- Co-authored-by: Boog900 <boog900@tutanota.com>
2024-02-26 23:40:13 +00:00
cuprate-helper = { path = "../helper", features = ["thread"] }
database: implement `ops/` (#102) * ops: add `trait MoneroR{o,w}` * update `trait MoneroR{o,w}` bounds * types: add `BlockInfoLatest` type alias * block: impl most core functions * types: fix https://github.com/Cuprate/cuprate/pull/91#discussion_r1527668916 * fix table type test * cargo.toml: add `{cuprate-types, monero-serai}` * add_block: add all other block data * ops: remove unneeded `block` functions * env: add `EnvInner::open_db_rw_all()` * types: fix test * block: `&mut TxRw` -> `&TxRw`, use `open_db_rw_all()` * add `trait Tables[Mut]` and use it in `EnvInner` * block: use `TablesMut` * tables: replace manual impl with `define_trait_tables!()` * tables: docs for `trait Tables[Mut]` * tables: doc functions * create `call_fn_on_all_tables_or_early_return!()` macro * block: cleanup signatures + bodies * block: more fn's, docs * block: add `doc_{error,single,bulk}!()` * remove `ops/monero.rs` * move `height()` to `ops/blockchain.rs` * add `ops/macros.rs` * tx: add fn signatures * output: fix fn signatures * ops: expose `_inner()` functions * block: add `add_block_header{_bulk, _inner}()` * ops: remove doc_{fn,inner}!()` * ops: remove `_{inner,bulk}()`, lifetime + generics * update lib/mod docs * ops: add and use `doc_add_block_inner_invariant!()` * ops: add docs/return to inner `add_block()` functions * add_block(): extract and use fn for {key_image, output} * ops: more fn body impl + `add_block()` * cargo: add `monero-pruning` * ops: extract out `tx` functions from `add_block()` * property: add `db_version()` * ops: `pop_block()` body, remove other `pop_block` fn's * types: add `block_blob: Vec<u8>` to `VerifiedBlockInformation` * block: put `block_blob`, pass `Tables` to sub-functions `impl TablesMut` can't mutably pass multiple tables since it takes `&mut self`, so all functions unfortunately have to take a full `&mut impl TablesMut` even though they only need a few tables. * database: add `DatabaseRw::take()` useful for `pop_block()` where we need the value afterwards * block: deserialize tx's from `block_blobs` in `pop_block()` * blockchain: `height()` -> `chain_height()` * output: fix `amount_index` * ops: fix unlock_time, chain_height * `BlockInfoV{1,2,3}` -> `BlockInfo` * constants: add `DATABASE_VERSION` * database: add `DatabaseRw::update()` * output: use `DatabaseRw::update()` in `remove_output()` * add `TxBlobs` table, ignore pruning tables * block: mostly impl `add_block()` body * ops: comments * add_block: miner v2 tx commitment, height cast * block: impl `pop_block()` * block: mostly impl `get_block()` * block: impl `get_block_{from_height,header,header_from_height}` * add `OutputFlags` bitflags * add_block: u32::try_into(height: u32), use `OutputFlags` * tx: impl `get_{tx,tx_from_id}()` * tx: move docs tests to `#[test]` testing everything in 1 go is more natural since e.g: `add_tx()` is followed by `get_tx()` * tables: add `trait TablesIter`, `all_tables_empty()` This allows `TablesMut` to be a superset of `Tables` and use all its accessor functions. * use cuprate-test-utils, fix tx tests * block: `add_block()` take block by ref * tx: use all txs in tests * output: add `all_tx_functions()` test * add_block: check current height against input * block: map more fields in `get_block()` * block: remove `get_block()`, doc tests, fix `get_block_header()` * block: dummy values in test * heed: use `last/first()` instead of `unsafe` cursors We no longer have DUP semantics and also hard to debug errors were popping up on `del_current()`... * heed: fix `DatabaseRw::delete` Ok(true) means the key did not exist, so we must return Err(RuntimeError::KeyNotFound) * block: `add_block()` (dummy value) test * ops: `key_image` tests * cleanup, docs, tests * backend: test `take()` & `update()` * docs * remove `OutputFlags::NONE` * add_block(): add asserts, panic docs, `should_panic` tests * backend: remove `Ok(())` in `Database::delete` if already deleted redb already does this, so heed so match * block: move block operations after tx/outputs * `amount == 0` -> `amount == 1` * Nit: StorableVec::wrap_ref * `saturating_sub(1)` -> `- 1` * add `TxOutputs` table * add_block(): add to `tx_outputs` table * fix `DatabaseRo::update` * add_tx(): take `block_height` as input * tx: add/remove from `TxOutputs` table * output: remove if `amount == 1` -> `amount_index == 0` * output: fix `add_output()`'s `amount_index` calculation * output: fix `add_output()`'s `amount_index` calculation again * output: tests for `amount_index/num_outputs` * block: `num_outputs - 1` and `take()` -> `get()` We don't need to `take()` since the call afterwards to `remove_output()` removes the entry * block: swap `get_block_extended_header[_from_height]()` * move `{key_image,output}` handling `add_block()` -> `add_tx()` * blockchain: add doc to `top_block_height()` * block: manual panic -> `assert_eq!()` * test-utils: add `block_blob` to `VerifiedBlockInformation` field introduced in this PR * ops: use real block/tx data in tests * block: `total_generated_coins` -> `cumulative_generated_coins` * fix clippy, docs, TODOs * `cumulative_generated_coins()`: `block/` -> `blockchain/` * blockchain: add `cumulative_generated_coins()` tests * Update database/src/ops/block.rs Co-authored-by: Boog900 <boog900@tutanota.com> * `cumulative_generated_coins()` docs for pre-block-0 special case --------- Co-authored-by: Boog900 <boog900@tutanota.com>
2024-04-23 01:53:29 +00:00
cuprate-test-utils = { path = "../test-utils" }
database: Resizes, Shutdown, Flushing (#68) * error: add `NeedsResize` accidently removed, was `MapFull` before. We need this because we as the writer thread must react to this error in order to resize. The writer thread doesn't have access to `heed::Error`, but `Runtime::Error`, so this variant must exist * env/backend: add `MANUAL_RESIZE` and `resize()` * heed: respect `ReadersFull`, comment errors * free: add `resize_memory_map()` * env: add `Env::current_map_size` * service: add `resize_map()` * database: make `Env` itself cheaply clonable + threadsafe `heed::Env` already uses `Arc` internally, but `sanakirja` does not, so abstract this at the `Env` level instead of wrapping `ConcreteEnv` in `Arc` ourselves. * service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone` * env: add `SYNCS_PER_TX`, `path()`, `shutdown()` * database: add `src/service/state.rs` * service: add to `state.rs`, add `DatabaseState` to readers/writer * add `parking_lot` Okay, turns out we need to take locks in `database/src/service/state.rs`... `std`'s lock fairness policies are not well defined and depend on the OS implementation, `parking_lot` on the other hand has a fairness policy which is important when the writer needs the lock but readers keep pouring in, essentially never letting the writer do stuff. * state: use `crossbeam::atomic::AtomicCell` We have crossbeam as a dep anyway. * heed: `heed::Env` -> `Arc<RwLock<heed::Env>>` * service: add reader shutdown handle, use `Select` for msgs * service: remove `state.rs` We don't need this, we will represent shutdowns with channel messages and `Select`, and mutual exclusion with a `RwLock`. * service: fix misc reader/writer stuff * database: add `config.rs` * service: use `ReaderThreads` when spawning readers * service: impl `shutdown()` for readers/writer * service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle` Solves the issue of unfortunately timed `Request`s that come in _right_ as we are shutting down. If we (Cuprate) drop the database channels too early the requesting thread will probably panic as they probably use `.unwrap()`, never expecting a channel failure. Returning this structure containing all the channels allows the final shutdown code to carry these channels until the very end of the program, at which point, all threads exit - no panics. * remove `parking_lot` Could be used as the database mutual exclusion lock. Needs to be tested against `std`. * config: add `path` * env: `path()` -> `config()`, backend: impl `Drop` * `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect * backend: add `Config` to `ConcreteEnv` * service: use `std::thread::Builder` for readers/writer * config: `PathBuf` -> `Cow<'static, Path>` * misc docs, remove `RuntimeError::ShuttingDown` * service: init & shutdown docs * heed: impl `Env::resize_map()` * heed: impl `Env::current_map_size()` * lib.rs: add example crate usage test * heed: `RwLock` comment * helper: add `cuprate_database_dir()` * config: use `cuprate_database_dir()` * lib.rs: TODO example test * database: add `page_size` The database memory map size must be a multiple of the OS page size. Why doesn't `heed` re-expose this? It calls it when checking anyway... https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772 * free: impl `resize_memory_map()` * free: docs * env: add `disk_size_bytes()` * config: impl `From<$num>` for `ReaderThreads` * move `fs`-related constants to `cuprate_helper::fs` * docs * add `resize.rs`, `ResizeAlgorithm` * env: use `ResizeAlgorithm` in `resize_map()` * TODO: account for LMDB reader limit * resize: docs, add `page_size()`, impl `fixed_bytes()` * resize: impl `percent()` * resize: docs * env: take `ResizeAlgorithm` by value (it impls `Copy`) * heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED` * config: `From<Into<usize>>` for `ReaderThreads` Co-authored-by: Boog900 <boog900@tutanota.com> * env: move mutual exclusion doc to backend * free: update invariant doc Co-authored-by: Boog900 <boog900@tutanota.com> * Update database/src/service/mod.rs Co-authored-by: Boog900 <boog900@tutanota.com> * fix `[allow(unused_imports)] // docs` * move DB filename from `helper/` -> `database/` * config: use `DATABASE_FILENAME` * config: add `db_file_path()` * lib: add non-`service` usage invariant docs * table: seal `Table` trait, impl for all `crate::tables` * fix docs * fix docs pt.2 --------- Co-authored-by: Boog900 <boog900@tutanota.com>
2024-02-25 19:46:36 +00:00
page_size = { version = "0.6.0" }
database: replace `Pod` with `bytemuck` (#81) * database: add `bytemuck` * database: add `types` module, replace `Pod` with `bytemuck` traits * types: docs * types: more docs * types: align safety msg * types: docs * misc docs * add `storable.rs` * add `slice.rs` * storable: impl `impl_storable_checked_bit_pattern!()` * database: TODO: fix `DatabaseRo::get_range` lifetimes * key/table: trait bound fixes * misc fixes * remove `borsh` - Doesn't work on must types - Probably won't use it anyway - Most things impl `serde` * key: add `new_with_max_secondary()` * key: add `new_with_max_secondary()` * heed: add `StorableHeed` for `Storable` compat * redb: add `StorableRedb{Key,Value}` for `Storable` compat * storable: add `Debug` bound and `fixed_width()` * redb: fix `'static` bound * storable: docs * `pod.rs` tests -> `storable.rs` * redb: add `Storable` tests * storable: add doc tests * redb: simplify `Storable` tests * heed: add `Storable` tests * misc docs/fixes * cargo: switch from forked `heed` -> `heed 0.20.0-alpha.9` * update readme * docs * fix README * table: remove `CONSTANT_SIZE` * database: `get()/delete() -> Err(KeyNotFound)` instead of `Option`
2024-03-03 22:26:39 +00:00
tempfile = { version = "3.10.0" }
test-utils: compare data against RPC (#115) * cargo: add `pretty_assertions` * test-utils: add tests to compare data against RPC data, and fix * small fixes * database: fix `cumulative_generated_coins` from test data block had wrong data which offset all of the asserts
2024-04-25 18:58:45 +00:00
pretty_assertions = { workspace = true }
database: implement `ops/` (#102) * ops: add `trait MoneroR{o,w}` * update `trait MoneroR{o,w}` bounds * types: add `BlockInfoLatest` type alias * block: impl most core functions * types: fix https://github.com/Cuprate/cuprate/pull/91#discussion_r1527668916 * fix table type test * cargo.toml: add `{cuprate-types, monero-serai}` * add_block: add all other block data * ops: remove unneeded `block` functions * env: add `EnvInner::open_db_rw_all()` * types: fix test * block: `&mut TxRw` -> `&TxRw`, use `open_db_rw_all()` * add `trait Tables[Mut]` and use it in `EnvInner` * block: use `TablesMut` * tables: replace manual impl with `define_trait_tables!()` * tables: docs for `trait Tables[Mut]` * tables: doc functions * create `call_fn_on_all_tables_or_early_return!()` macro * block: cleanup signatures + bodies * block: more fn's, docs * block: add `doc_{error,single,bulk}!()` * remove `ops/monero.rs` * move `height()` to `ops/blockchain.rs` * add `ops/macros.rs` * tx: add fn signatures * output: fix fn signatures * ops: expose `_inner()` functions * block: add `add_block_header{_bulk, _inner}()` * ops: remove doc_{fn,inner}!()` * ops: remove `_{inner,bulk}()`, lifetime + generics * update lib/mod docs * ops: add and use `doc_add_block_inner_invariant!()` * ops: add docs/return to inner `add_block()` functions * add_block(): extract and use fn for {key_image, output} * ops: more fn body impl + `add_block()` * cargo: add `monero-pruning` * ops: extract out `tx` functions from `add_block()` * property: add `db_version()` * ops: `pop_block()` body, remove other `pop_block` fn's * types: add `block_blob: Vec<u8>` to `VerifiedBlockInformation` * block: put `block_blob`, pass `Tables` to sub-functions `impl TablesMut` can't mutably pass multiple tables since it takes `&mut self`, so all functions unfortunately have to take a full `&mut impl TablesMut` even though they only need a few tables. * database: add `DatabaseRw::take()` useful for `pop_block()` where we need the value afterwards * block: deserialize tx's from `block_blobs` in `pop_block()` * blockchain: `height()` -> `chain_height()` * output: fix `amount_index` * ops: fix unlock_time, chain_height * `BlockInfoV{1,2,3}` -> `BlockInfo` * constants: add `DATABASE_VERSION` * database: add `DatabaseRw::update()` * output: use `DatabaseRw::update()` in `remove_output()` * add `TxBlobs` table, ignore pruning tables * block: mostly impl `add_block()` body * ops: comments * add_block: miner v2 tx commitment, height cast * block: impl `pop_block()` * block: mostly impl `get_block()` * block: impl `get_block_{from_height,header,header_from_height}` * add `OutputFlags` bitflags * add_block: u32::try_into(height: u32), use `OutputFlags` * tx: impl `get_{tx,tx_from_id}()` * tx: move docs tests to `#[test]` testing everything in 1 go is more natural since e.g: `add_tx()` is followed by `get_tx()` * tables: add `trait TablesIter`, `all_tables_empty()` This allows `TablesMut` to be a superset of `Tables` and use all its accessor functions. * use cuprate-test-utils, fix tx tests * block: `add_block()` take block by ref * tx: use all txs in tests * output: add `all_tx_functions()` test * add_block: check current height against input * block: map more fields in `get_block()` * block: remove `get_block()`, doc tests, fix `get_block_header()` * block: dummy values in test * heed: use `last/first()` instead of `unsafe` cursors We no longer have DUP semantics and also hard to debug errors were popping up on `del_current()`... * heed: fix `DatabaseRw::delete` Ok(true) means the key did not exist, so we must return Err(RuntimeError::KeyNotFound) * block: `add_block()` (dummy value) test * ops: `key_image` tests * cleanup, docs, tests * backend: test `take()` & `update()` * docs * remove `OutputFlags::NONE` * add_block(): add asserts, panic docs, `should_panic` tests * backend: remove `Ok(())` in `Database::delete` if already deleted redb already does this, so heed so match * block: move block operations after tx/outputs * `amount == 0` -> `amount == 1` * Nit: StorableVec::wrap_ref * `saturating_sub(1)` -> `- 1` * add `TxOutputs` table * add_block(): add to `tx_outputs` table * fix `DatabaseRo::update` * add_tx(): take `block_height` as input * tx: add/remove from `TxOutputs` table * output: remove if `amount == 1` -> `amount_index == 0` * output: fix `add_output()`'s `amount_index` calculation * output: fix `add_output()`'s `amount_index` calculation again * output: tests for `amount_index/num_outputs` * block: `num_outputs - 1` and `take()` -> `get()` We don't need to `take()` since the call afterwards to `remove_output()` removes the entry * block: swap `get_block_extended_header[_from_height]()` * move `{key_image,output}` handling `add_block()` -> `add_tx()` * blockchain: add doc to `top_block_height()` * block: manual panic -> `assert_eq!()` * test-utils: add `block_blob` to `VerifiedBlockInformation` field introduced in this PR * ops: use real block/tx data in tests * block: `total_generated_coins` -> `cumulative_generated_coins` * fix clippy, docs, TODOs * `cumulative_generated_coins()`: `block/` -> `blockchain/` * blockchain: add `cumulative_generated_coins()` tests * Update database/src/ops/block.rs Co-authored-by: Boog900 <boog900@tutanota.com> * `cumulative_generated_coins()` docs for pre-block-0 special case --------- Co-authored-by: Boog900 <boog900@tutanota.com>
2024-04-23 01:53:29 +00:00
hex = { workspace = true }
hex-literal = { workspace = true }
Reference in a new issue Copy permalink