* add `redb-memory` feature
* redb: use `redb::backend::InMemoryBackend` if enabled
* readme: add `redb-memory` section
* ci: add `redb-memory` testing
* ci: remove `redb-memory` testing
probably not worth adding time to CI for this
* error: add `NeedsResize`
accidently removed, was `MapFull` before.
We need this because we as the writer thread must
react to this error in order to resize.
The writer thread doesn't have access to `heed::Error`, but
`Runtime::Error`, so this variant must exist
* env/backend: add `MANUAL_RESIZE` and `resize()`
* heed: respect `ReadersFull`, comment errors
* free: add `resize_memory_map()`
* env: add `Env::current_map_size`
* service: add `resize_map()`
* database: make `Env` itself cheaply clonable + threadsafe
`heed::Env` already uses `Arc` internally, but `sanakirja` does
not, so abstract this at the `Env` level instead of wrapping
`ConcreteEnv` in `Arc` ourselves.
* service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone`
* env: add `SYNCS_PER_TX`, `path()`, `shutdown()`
* database: add `src/service/state.rs`
* service: add to `state.rs`, add `DatabaseState` to readers/writer
* add `parking_lot`
Okay, turns out we need to take locks in
`database/src/service/state.rs`...
`std`'s lock fairness policies are not well defined and
depend on the OS implementation, `parking_lot` on the other hand
has a fairness policy which is important when the writer needs
the lock but readers keep pouring in, essentially never letting
the writer do stuff.
* state: use `crossbeam::atomic::AtomicCell`
We have crossbeam as a dep anyway.
* heed: `heed::Env` -> `Arc<RwLock<heed::Env>>`
* service: add reader shutdown handle, use `Select` for msgs
* service: remove `state.rs`
We don't need this, we will represent shutdowns with channel
messages and `Select`, and mutual exclusion with a `RwLock`.
* service: fix misc reader/writer stuff
* database: add `config.rs`
* service: use `ReaderThreads` when spawning readers
* service: impl `shutdown()` for readers/writer
* service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle`
Solves the issue of unfortunately timed `Request`s
that come in _right_ as we are shutting down.
If we (Cuprate) drop the database channels too early the requesting
thread will probably panic as they probably use `.unwrap()`,
never expecting a channel failure.
Returning this structure containing all the channels allows the
final shutdown code to carry these channels until the very end
of the program, at which point, all threads exit - no panics.
* remove `parking_lot`
Could be used as the database mutual exclusion lock.
Needs to be tested against `std`.
* config: add `path`
* env: `path()` -> `config()`, backend: impl `Drop`
* `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect
* backend: add `Config` to `ConcreteEnv`
* service: use `std:🧵:Builder` for readers/writer
* config: `PathBuf` -> `Cow<'static, Path>`
* misc docs, remove `RuntimeError::ShuttingDown`
* service: init & shutdown docs
* heed: impl `Env::resize_map()`
* heed: impl `Env::current_map_size()`
* lib.rs: add example crate usage test
* heed: `RwLock` comment
* helper: add `cuprate_database_dir()`
* config: use `cuprate_database_dir()`
* lib.rs: TODO example test
* database: add `page_size`
The database memory map size must be a multiple of
the OS page size. Why doesn't `heed` re-expose this?
It calls it when checking anyway...
https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772
* free: impl `resize_memory_map()`
* free: docs
* env: add `disk_size_bytes()`
* config: impl `From<$num>` for `ReaderThreads`
* move `fs`-related constants to `cuprate_helper::fs`
* docs
* add `resize.rs`, `ResizeAlgorithm`
* env: use `ResizeAlgorithm` in `resize_map()`
* TODO: account for LMDB reader limit
* resize: docs, add `page_size()`, impl `fixed_bytes()`
* resize: impl `percent()`
* resize: docs
* env: take `ResizeAlgorithm` by value (it impls `Copy`)
* heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED`
* config: `From<Into<usize>>` for `ReaderThreads`
Co-authored-by: Boog900 <boog900@tutanota.com>
* env: move mutual exclusion doc to backend
* free: update invariant doc
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/service/mod.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* fix `[allow(unused_imports)] // docs`
* move DB filename from `helper/` -> `database/`
* config: use `DATABASE_FILENAME`
* config: add `db_file_path()`
* lib: add non-`service` usage invariant docs
* table: seal `Table` trait, impl for all `crate::tables`
* fix docs
* fix docs pt.2
---------
Co-authored-by: Boog900 <boog900@tutanota.com>
* error: add variants to `RuntimeError`
* error: remove `<BackendError>` generic
we can just map each backend error variant <-> our error as needed
* backend: impl `From<heed::Error>` for `RuntimeError`
* add `Never` type to allow foreign trait implementations
This is a newtype to workaround `sanakirja::Storable` not being
able to be implemented on `std::convert::Infallible` due to
"foreign trait on foreign type" rules.
* revert 0342848, fix `sanakirja` trait bounds
K/V will always be `[u8]`, not the concrete type
so it does not need to be bounded.
* backend: fix `sanakijra` K/V generics
* sanakirja: add `error.rs`
* error: remove serde traits
* heed: add `todo!()` for error mappings
* error: add `Corrupt` variant
* sanakirja: finish error mappings
* heed: finish error mappings
* error: add to error types
* env: use `InitError` in `Env::open()`
* error: docs
* heed: remove `serde.rs`
Not needed if all K/V's stored are `[u8]`.
* heed: use `heed::types::Bytes` as K/V
* database: docs
* heed: impl `From<heed::Error>` for `InitError`
* sanakirja: impl `From<sanakirja::Error>` for `InitError`
* error: fix doc warnings
* heed: fix `clippy::match_same_arms` in error match
* readme: add TODO
* error: add `BoxError`, and fatal/unknown variants
* heed: use `Fatal/Unknown` variants in errors
* sanakirja: use `Fatal/Unknown` variants in errors
* clippy
* sanakijra: remove `fallible_impl_from`
* error: remove `RuntimeError::InvalidVersion`
* error: remove `RuntimeError` variants that should panic
* error: remove `InitError::Fatal`
We will exit on all errors regardless.
Any non-enumrated variants will use `InitError::Unknown`.
* heed: fix error mappings
* constants: add `CUPRATE_DATABASE_CORRUPT_MSG`
* sanakijra: fix error mappings
* heed: fix import
* comments/docs
* key: fix docs
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
