Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
//! Database abstraction and utilities.
|
|
|
|
//!
|
|
|
|
//! This crate does 3 things:
|
|
|
|
//! 1. Abstracts various databases with the [`Env`], [`Database`], [`Table`], [`Key`], [`RoTx`], and [`RwTx`] traits
|
|
|
|
//! 2. Implements various `Monero` related [functions](ops) & [`tables`]
|
|
|
|
//! 3. Exposes a [`tower::Service`] backed by a thread-pool
|
|
|
|
//!
|
|
|
|
//! # Terminology
|
|
|
|
//! To be more clear on some terms used in this crate:
|
|
|
|
//!
|
|
|
|
//! | Term | Meaning |
|
|
|
|
//! |------------|--------------------------------------|
|
|
|
|
//! | `Env` | The 1 database environment, the "whole" thing
|
|
|
|
//! | `Database` | A `key/value` store
|
|
|
|
//! | `Table` | Solely the metadata of a `Database` (the `key` and `value` types, and the name)
|
|
|
|
//! | `RoTx` | Read only transaction
|
|
|
|
//! | `RwTx` | Read/write transaction
|
|
|
|
//!
|
|
|
|
//! The dataflow is `Env` -> `Tx` -> `Database`
|
|
|
|
//!
|
|
|
|
//! Which reads as:
|
|
|
|
//! 1. You have a database `Environment`
|
|
|
|
//! 2. You open up a `Transaction`
|
|
|
|
//! 2. You get a particular `Database` from that `Environment`
|
|
|
|
//! 3. You can now read/write data from/to that `Database`
|
|
|
|
//!
|
|
|
|
//! # `ConcreteEnv`
|
|
|
|
//! This crate exposes [`ConcreteEnv`], which is a non-generic/non-dynamic,
|
|
|
|
//! concrete object representing a database [`Env`]ironment.
|
|
|
|
//!
|
|
|
|
//! The actual backend for this type is determined via feature flags.
|
|
|
|
//!
|
|
|
|
//! This object existing means `E: Env` doesn't need to be spread all through the codebase,
|
|
|
|
//! however, it also means some small invariants should be kept in mind.
|
|
|
|
//!
|
|
|
|
//! As `ConcreteEnv` is just a re-exposed type which has varying inner types,
|
|
|
|
//! it means some properties will change depending on the backend used.
|
|
|
|
//!
|
|
|
|
//! For example:
|
|
|
|
//! - [`std::mem::size_of::<ConcreteEnv>`]
|
|
|
|
//! - [`std::mem::align_of::<ConcreteEnv>`]
|
|
|
|
//! - [`Drop::<ConcreteEnv>::drop`]
|
|
|
|
//!
|
|
|
|
//! Things like these functions are affected by the backend and inner data,
|
|
|
|
//! and should not be relied upon. This extends to any `struct/enum` that contains `ConcreteEnv`.
|
|
|
|
//!
|
|
|
|
//! The only thing about `ConcreteEnv` that should
|
|
|
|
//! be relied upon is that it implements [`Env`].
|
|
|
|
//!
|
|
|
|
//! TODO: we could also expose `ConcreteDatabase` if we're
|
|
|
|
//! going to be storing any databases in structs, to lessen
|
|
|
|
//! the generic `<D: Database>` pain.
|
|
|
|
//!
|
|
|
|
//! # Feature flags
|
|
|
|
//! The `service` module requires the `service` feature to be enabled.
|
|
|
|
//! See the module for more documentation.
|
|
|
|
//!
|
|
|
|
//! Different database backends are enabled by the feature flags:
|
|
|
|
//! - `heed`
|
|
|
|
//! - `sanakirja`
|
|
|
|
//!
|
|
|
|
//! The default is `heed`.
|
|
|
|
|
|
|
|
//---------------------------------------------------------------------------------------------------- Lints
|
|
|
|
// Forbid lints.
|
|
|
|
// Our code, and code generated (e.g macros) cannot overrule these.
|
|
|
|
#![forbid(
|
|
|
|
// Never.
|
|
|
|
unsafe_code,
|
|
|
|
unused_unsafe,
|
|
|
|
redundant_semicolons,
|
|
|
|
unused_allocation,
|
|
|
|
coherence_leak_check,
|
|
|
|
single_use_lifetimes,
|
|
|
|
while_true,
|
|
|
|
clippy::missing_docs_in_private_items,
|
|
|
|
|
|
|
|
// Maybe can be put into `#[deny]`.
|
|
|
|
unconditional_recursion,
|
|
|
|
for_loops_over_fallibles,
|
|
|
|
unused_braces,
|
|
|
|
unused_doc_comments,
|
|
|
|
unused_labels,
|
|
|
|
keyword_idents,
|
|
|
|
non_ascii_idents,
|
|
|
|
variant_size_differences,
|
|
|
|
unused_mut, // Annoying when debugging, maybe put in allow.
|
|
|
|
|
|
|
|
// Probably can be put into `#[deny]`.
|
|
|
|
future_incompatible,
|
|
|
|
let_underscore,
|
|
|
|
break_with_label_and_loop,
|
|
|
|
duplicate_macro_attributes,
|
|
|
|
exported_private_dependencies,
|
|
|
|
large_assignments,
|
|
|
|
overlapping_range_endpoints,
|
|
|
|
semicolon_in_expressions_from_macros,
|
|
|
|
noop_method_call,
|
|
|
|
unreachable_pub,
|
|
|
|
)]
|
|
|
|
// Deny lints.
|
|
|
|
// Some of these are `#[allow]`'ed on a per-case basis.
|
|
|
|
#![deny(
|
|
|
|
clippy::all,
|
|
|
|
clippy::correctness,
|
|
|
|
clippy::suspicious,
|
|
|
|
clippy::style,
|
|
|
|
clippy::complexity,
|
|
|
|
clippy::perf,
|
|
|
|
clippy::pedantic,
|
|
|
|
clippy::nursery,
|
|
|
|
clippy::cargo,
|
|
|
|
missing_docs,
|
|
|
|
deprecated,
|
|
|
|
unused_comparisons,
|
|
|
|
nonstandard_style
|
|
|
|
)]
|
|
|
|
#![allow(unreachable_code, unused_variables, dead_code)] // TODO: remove
|
|
|
|
#![allow(
|
|
|
|
// FIXME: this lint affects crates outside of
|
|
|
|
// `database/` for some reason, allow for now.
|
|
|
|
clippy::cargo_common_metadata,
|
|
|
|
|
|
|
|
// FIXME: adding `#[must_use]` onto everything
|
|
|
|
// might just be more annoying than useful...
|
|
|
|
// although it is sometimes nice.
|
|
|
|
clippy::must_use_candidate,
|
|
|
|
|
|
|
|
// TODO: should be removed after all `todo!()`'s are gone.
|
|
|
|
clippy::diverging_sub_expression,
|
|
|
|
|
|
|
|
// FIXME:
|
|
|
|
// If #[deny(clippy::restriction)] is used, it
|
|
|
|
// enables a whole bunch of very subjective lints.
|
|
|
|
// The below disables most of the ones that are
|
|
|
|
// a bit too unwieldy.
|
|
|
|
//
|
|
|
|
// Figure out if if `clippy::restriction` should be
|
|
|
|
// used (it enables a bunch of good lints but has
|
|
|
|
// many false positives).
|
|
|
|
|
|
|
|
// clippy::single_char_lifetime_names,
|
|
|
|
// clippy::implicit_return,
|
|
|
|
// clippy::std_instead_of_alloc,
|
|
|
|
// clippy::std_instead_of_core,
|
|
|
|
// clippy::unwrap_used,
|
|
|
|
// clippy::min_ident_chars,
|
|
|
|
// clippy::absolute_paths,
|
|
|
|
// clippy::missing_inline_in_public_items,
|
|
|
|
// clippy::shadow_reuse,
|
|
|
|
// clippy::shadow_unrelated,
|
|
|
|
// clippy::missing_trait_methods,
|
|
|
|
// clippy::pub_use,
|
|
|
|
// clippy::pub_with_shorthand,
|
|
|
|
// clippy::blanket_clippy_restriction_lints,
|
|
|
|
// clippy::exhaustive_structs,
|
|
|
|
// clippy::exhaustive_enums,
|
|
|
|
// clippy::unsafe_derive_deserialize,
|
|
|
|
// clippy::multiple_inherent_impl,
|
|
|
|
// clippy::unreadable_literal,
|
|
|
|
// clippy::indexing_slicing,
|
|
|
|
// clippy::float_arithmetic,
|
|
|
|
// clippy::cast_possible_truncation,
|
|
|
|
// clippy::as_conversions,
|
|
|
|
// clippy::cast_precision_loss,
|
|
|
|
// clippy::cast_sign_loss,
|
|
|
|
// clippy::missing_asserts_for_indexing,
|
|
|
|
// clippy::default_numeric_fallback,
|
|
|
|
// clippy::module_inception,
|
|
|
|
// clippy::mod_module_files,
|
|
|
|
// clippy::multiple_unsafe_ops_per_block,
|
|
|
|
// clippy::too_many_lines,
|
|
|
|
// clippy::missing_assert_message,
|
|
|
|
// clippy::len_zero,
|
|
|
|
// clippy::separated_literal_suffix,
|
|
|
|
// clippy::single_call_fn,
|
|
|
|
// clippy::unreachable,
|
|
|
|
// clippy::many_single_char_names,
|
|
|
|
// clippy::redundant_pub_crate,
|
|
|
|
// clippy::decimal_literal_representation,
|
|
|
|
// clippy::option_if_let_else,
|
|
|
|
// clippy::lossy_float_literal,
|
|
|
|
// clippy::modulo_arithmetic,
|
|
|
|
// clippy::print_stdout,
|
|
|
|
// clippy::module_name_repetitions,
|
|
|
|
// clippy::no_effect,
|
|
|
|
// clippy::semicolon_outside_block,
|
|
|
|
// clippy::panic,
|
|
|
|
// clippy::question_mark_used,
|
|
|
|
// clippy::expect_used,
|
|
|
|
// clippy::integer_division,
|
|
|
|
// clippy::type_complexity,
|
|
|
|
// clippy::pattern_type_mismatch,
|
|
|
|
// clippy::arithmetic_side_effects,
|
|
|
|
// clippy::default_trait_access,
|
|
|
|
// clippy::similar_names,
|
|
|
|
// clippy::needless_pass_by_value,
|
|
|
|
// clippy::inline_always,
|
|
|
|
// clippy::if_then_some_else_none,
|
|
|
|
// clippy::arithmetic_side_effects,
|
|
|
|
// clippy::float_cmp,
|
|
|
|
// clippy::items_after_statements,
|
|
|
|
// clippy::use_debug,
|
|
|
|
// clippy::mem_forget,
|
|
|
|
// clippy::else_if_without_else,
|
|
|
|
// clippy::str_to_string,
|
|
|
|
// clippy::branches_sharing_code,
|
|
|
|
// clippy::impl_trait_in_params,
|
|
|
|
// clippy::struct_excessive_bools,
|
|
|
|
// clippy::exit,
|
|
|
|
// // This lint is actually good but
|
|
|
|
// // it sometimes hits false positive.
|
|
|
|
// clippy::self_named_module_files
|
|
|
|
|
|
|
|
clippy::module_name_repetitions,
|
|
|
|
clippy::module_inception,
|
|
|
|
clippy::redundant_pub_crate,
|
|
|
|
)]
|
|
|
|
// Allow some lints when running in debug mode.
|
|
|
|
#![cfg_attr(debug_assertions, allow(clippy::todo, clippy::multiple_crate_versions))]
|
|
|
|
|
|
|
|
// Only allow building 64-bit targets.
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
//
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
// This allows us to assume 64-bit
|
|
|
|
// invariants in code, e.g. `usize as u64`.
|
|
|
|
#[cfg(not(target_pointer_width = "64"))]
|
|
|
|
compile_error!("Cuprate is only compatible with 64-bit CPUs");
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
//---------------------------------------------------------------------------------------------------- Public API
|
|
|
|
// Import private modules, export public types.
|
|
|
|
//
|
|
|
|
// Documentation for each module is
|
|
|
|
// located in the respective file.
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
mod backend;
|
|
|
|
pub use backend::ConcreteEnv;
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
mod constants;
|
|
|
|
pub use constants::{CUPRATE_DATABASE_DIR, CUPRATE_DATABASE_FILE, DATABASE_BACKEND};
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
mod database;
|
|
|
|
pub use database::Database;
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
mod env;
|
|
|
|
pub use env::Env;
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
mod error;
|
|
|
|
pub use error::{InitError, RuntimeError};
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
mod free;
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
mod key;
|
|
|
|
pub use key::{DupKey, Key};
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
mod macros;
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
pub mod ops;
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
mod pod;
|
|
|
|
pub use pod::Pod;
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
mod table;
|
|
|
|
pub use table::Table;
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
pub mod tables;
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
mod transaction;
|
|
|
|
pub use transaction::{RoTx, RwTx};
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
//---------------------------------------------------------------------------------------------------- Feature-gated
|
|
|
|
#[cfg(feature = "service")]
|
|
|
|
pub mod service;
|
initial database code (#6)
* commit to start the draft pull request.
added a space
* Please don't look to close.It might hurt your eyes
* impl associated types
* errors, docs & divided ro/rw tx
Added some more errors to DB_FAILURES, rewrited crates docs, and specified
WriteTransaction subtype which implement write mode method.
* more changes see description
changed blockchain_db folder by database. Implemented (just for test) get_block_hash, open, from to Interface.
Also rewrited a declarative macro for tables. Will have to add Dummy Tables later.
* small changes
* Organized modules & implemented get_block_hash
* write prototype & error
Added prototype functions for clear(), put() & delete() in mdbx implementation. They still don't
consider table flags. Also added a temporary DB_FAILURES::EncodingError for monero-rs consensus_encode
errors. Still have to rethink about it to resend a reference to the data that can't be encoded.
* Multiple changes
- hse.rs
Added hse.rs that will contain db implementations for HSE. Since the codebase can't welcome unsafe
code, the wrapper will be written outside of the project.
- lib.rs
Added a specific FailedToCommit error. (will investigate if really necessary).
Added DupTable trait, which is a Table with DUPSORT/DUPFIXED support and its declarative macro.
Added two other tables, blockheaders that give block's header with specified hash & blockbody that give block's body with specified hash
Added Cursor methods, that are likely to be deprecated if I found a way to implemen Iterator on top of it.
Added WriteCursor trait & methods, which is basically put & del.
Added mandatory type for Cursors in Transaction & WriteTransactions
Refactored get_block_hash interface method.
- mdbx.rs
Added partial implementation of Cursor & WriteCursor trait for libmdbx::Cursor. Only the first() & get() methods are implemented
Added implementation of get & commit for Transaction
* put mdbx as features with its dependency
* save
* refactored some method with macros
* more mdbx errors, docs correction, moved to error.rs
* finish nodup mdbx impl, errors.rs, macros, tables
Finished the initial implementation of Cursor, WriteCursor, Transaction and WriteTransaction in mdbx.rs. Corrected some macros in mdbx.rs to simplify the implementations. There is certainly rooms to more flexible macros. Also added 3 other tables. I started to divide errors into category to more easily handle them at higher-level. Due to the large number of errors i just moved them into another file. There is know DB_SERIAL enum for errors relating of decoding/encoding error & DB_FULL enum for every errors relating a component being overeaching its capacity.
* bye bye match statement in mdbx.rs
* defined all blockchain tables (not txpool)
* dupsort/fixed support, dupcursor, basic block interface
* tables, types, encoding and documentations
Redefined all the database types from @Boog900's monero-rs db branch and added the needed
implementations. The database now use bincode2 for encoding and decoding. We observe that bincode was
5 times faster at serializing than monero::consensus_encode. Since we still use monero-rs types but can't implement
foreign trait to them, the encoding module contain a compatibility layer, the time we switch from monero-rs to properly
implement it. All the tables are now defined. (can be subject to change if there is good reason for). added documentations
to modules and types.
* replaced macros and added hfversion table
* save
* multiple changes
* modified database schema. deprecated output global index and splited up pre-rct from rct output.
* Fixed DupCursor function to return subkey (thx to rust turbofish inference).
* Added some output functions
* Added two new DB_FAILURES, one to handle a prohibited None case and one for undefined case where a dev msg is needed.
* fixed TxOutputIdx, previously used global index, now is a tuple of amount/amount_index.
* i hate lifetimes
* read-only method now use read-only tx
* initial output fn
* some tx functions. Yes I'll refactor them
* moved interface in a module
* redefined errors, more tx fn, None->error
* corrected a table + started blk fns
* save
* fixed TxOutputIdx + pop_block
* IIRC I finished initial interface fns
* fixed table name const + db build/check/open fn
* switched important tables to dummy keys + rm blockhfversion
* minor docs correction
* fixed mentioned issues
* make a test bin, just for fun
* fixed issues + cargo fmt
* removed monerod part
* fixed a comment
2023-04-20 17:20:32 +00:00
|
|
|
|
Database (#35)
* rename `database` -> `old_database`
Keeping it around for reference until new implementation is complete.
* create new `database/` skeleton
* add `DATABASE.md` design doc skeleton
* move design doc to `database/README.md`
* add rough code
* `lib.rs` -> `gist.rs`
* database: use workspace deps
* workspace: include `database/` as member
CI will now include this crate.
* cargo fmt
* database: `AGPL` -> `MIT`
* readme: add `TODO`s
* add base files
* cargo.toml: add `heed` feature
* fix clippy
* lib.rs: add extremely pedantic lints
* readme: add `# Backends`
* cargo.toml: add `cfg-if`
* add `backend/` structure
* base `database.rs`
* cargo.toml: add `borsh`
* backend: add `DATABASE_BACKEND`
* base `error.rs`
* base `database.rs`
* base `transaction.rs`
* base `table.rs`
* lib.rs: add imports
* add `pod.rs`
* pod: use `Read/Write`, add tests for all primitive numbers
* pod: impl Pod for `Vec<u8>`, `[u8; N]`
* pod: add docs, add `private::Sealed`
* pod: `to_writer`, `from_reader`
The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.
Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`
* pod: impl Pod for `Box<[u8]>`
* pod: return `Err` on incorrect length in `from_bytes()`
* pod: docs
* pod: impl Pod for `Arc<[u8]>`
* readme: docs
* database: add `create_table()`, `get_table()`
* table: `Pod` bound
* backend: move into directories
* pod: add `into_bytes()`
* heed: impl `BytesEncode`, `BytesDecode`
* add `actor`, `service` features
The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.
* add `lib.rs` docs
* service: add `service.rs`
* service: add `reader.rs`
* service: add `writer.rs`
* service: add `request.rs` & `response.rs`
* cargo.toml: add `crossbeam`
* service: add/use `enum Request`, `enum Response`
* service: basic signatures for thread-pools, `Writer` -> `Writers`
* service: split `tower::Service<ReadRequest/WriteRequest>`
* service: impl `tower::Service` for `Database(Reader|Writer)`
* service: add `init()`, impl basic `Reader/Writer` pools
* service: add example `Request`'s
* service: add example `ReadRequest` handling
* temporarily allow clippy lints
* readme: update file structure
* transaction: add `RoTx::get_range()`
* service: module docs
* cargo.toml: add `cuprate-helper`
* service: scale readers/writers based on thread count
* database: change lifetimes
* heed: impl Database for `heed`
* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits
* docs
* service: `read.rs` docs
* service: `write.rs` docs
* service: request/response docs
* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`
* service: leak database into `&'static`
* database: add `#[inline]`, `#[cold]`
* service: `free.rs` docs, more `#[inline]` + `#[cold]`
* service: add `shutdown()`
* service: add `Request::Shutdown`
* service: `shutdown()` docs
* heed: hide concrete tx types
* lib.rs: add terms
* split `Env` <-> `Database`
* cargo.toml: add `paste`
* database: add `tables/`
* impl `serde/borsh` where possible
* tables: add `Tables`, add test docs
* make db backend mutually exclusive to fix `--all-features`
* tables: use `$()*` in `tables!()`
* cargo.toml: add `sanakirja 1.4.0`
* sanakirja: impl `Env`
* sanakirja: impl `Database`
* sanakirja: impl `Transaction`
* table: temporarily fix `sanakirja` K/V bounds
* table: fix `#[cfg]`
* cargo.toml: fix deps
* lib.rs: docs
* service: docs
* readme: add files, update `# Documentation`, add `# Layers`
* readme: `src/` file purpose
* readme: `src/service/` file purpose
* readme: `src/backend/` file purpose
* fix `Cargo.lock` merge conflict
* database: remove `gist.rs`
* add to `constants.rs`
* add top `//! comments` for files/modules
* constants: add sanity-check test
* service: add `only_one_database` test in `free.rs`
* service: add `tests.rs`
* remove unneeded markers + imports
* backend: fix `get_range()`'s trait `impl` return
* env: add `create_tables_if_needed()`, don't return `Option<Db>`
* sort imports by `CONTRIBUTING.md` rules
oops sorry boog
* add `monero.rs`
* monero: docs
* database: add missing `RoTx/RwTx` inputs
* backend: add missing `RoTx/RwTx` inputs
* `monero.rs` trait -> free functions in `ops/`
* pod: make methods infallible
* ci: add `rustup update` step
* service: use `Arc` instead of leaking, remove `db_read/db_write`
* service: use `InfallibleOneshotReceiver` for readers
* service: shutdown on error, add todos
* service: remove `Request`
* service: combine `ReadResponse` and `WriteResponse`
* service: use `InfallibleOneshotReceiver` for writer
* service: only spawn 1 writer, don't allow cloning writer handle
* table: add associated `const CONSTANT_SIZE`
* add `key.rs` + `trait Key`, add bound to `Table`
* fix typos
2024-02-13 17:43:25 +00:00
|
|
|
//---------------------------------------------------------------------------------------------------- Private
|