cuprate/database/src/tables.rs
hinto-janai 51d9ccd02d
database: Resizes, Shutdown, Flushing (#68)
* error: add `NeedsResize`

accidently removed, was `MapFull` before.

We need this because we as the writer thread must
react to this error in order to resize.

The writer thread doesn't have access to `heed::Error`, but
`Runtime::Error`, so this variant must exist

* env/backend: add `MANUAL_RESIZE` and `resize()`

* heed: respect `ReadersFull`, comment errors

* free: add `resize_memory_map()`

* env: add `Env::current_map_size`

* service: add `resize_map()`

* database: make `Env` itself cheaply clonable + threadsafe

`heed::Env` already uses `Arc` internally, but `sanakirja` does
not, so abstract this at the `Env` level instead of wrapping
`ConcreteEnv` in `Arc` ourselves.

* service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone`

* env: add `SYNCS_PER_TX`, `path()`, `shutdown()`

* database: add `src/service/state.rs`

* service: add to `state.rs`, add `DatabaseState` to readers/writer

* add `parking_lot`

Okay, turns out we need to take locks in
`database/src/service/state.rs`...

`std`'s lock fairness policies are not well defined and
depend on the OS implementation, `parking_lot` on the other hand
has a fairness policy which is important when the writer needs
the lock but readers keep pouring in, essentially never letting
the writer do stuff.

* state: use `crossbeam::atomic::AtomicCell`

We have crossbeam as a dep anyway.

* heed: `heed::Env` -> `Arc<RwLock<heed::Env>>`

* service: add reader shutdown handle, use `Select` for msgs

* service: remove `state.rs`

We don't need this, we will represent shutdowns with channel
messages and `Select`, and mutual exclusion with a `RwLock`.

* service: fix misc reader/writer stuff

* database: add `config.rs`

* service: use `ReaderThreads` when spawning readers

* service: impl `shutdown()` for readers/writer

* service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle`

Solves the issue of unfortunately timed `Request`s
that come in _right_ as we are shutting down.

If we (Cuprate) drop the database channels too early the requesting
thread will probably panic as they probably use `.unwrap()`,
never expecting a channel failure.

Returning this structure containing all the channels allows the
final shutdown code to carry these channels until the very end
of the program, at which point, all threads exit - no panics.

* remove `parking_lot`

Could be used as the database mutual exclusion lock.

Needs to be tested against `std`.

* config: add `path`

* env: `path()` -> `config()`, backend: impl `Drop`

* `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect

* backend: add `Config` to `ConcreteEnv`

* service: use `std:🧵:Builder` for readers/writer

* config: `PathBuf` -> `Cow<'static, Path>`

* misc docs, remove `RuntimeError::ShuttingDown`

* service: init & shutdown docs

* heed: impl `Env::resize_map()`

* heed: impl `Env::current_map_size()`

* lib.rs: add example crate usage test

* heed: `RwLock` comment

* helper: add `cuprate_database_dir()`

* config: use `cuprate_database_dir()`

* lib.rs: TODO example test

* database: add `page_size`

The database memory map size must be a multiple of
the OS page size. Why doesn't `heed` re-expose this?
It calls it when checking anyway...
https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772

* free: impl `resize_memory_map()`

* free: docs

* env: add `disk_size_bytes()`

* config: impl `From<$num>` for `ReaderThreads`

* move `fs`-related constants to `cuprate_helper::fs`

* docs

* add `resize.rs`, `ResizeAlgorithm`

* env: use `ResizeAlgorithm` in `resize_map()`

* TODO: account for LMDB reader limit

* resize: docs, add `page_size()`, impl `fixed_bytes()`

* resize: impl `percent()`

* resize: docs

* env: take `ResizeAlgorithm` by value (it impls `Copy`)

* heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED`

* config: `From<Into<usize>>` for `ReaderThreads`

Co-authored-by: Boog900 <boog900@tutanota.com>

* env: move mutual exclusion doc to backend

* free: update invariant doc

Co-authored-by: Boog900 <boog900@tutanota.com>

* Update database/src/service/mod.rs

Co-authored-by: Boog900 <boog900@tutanota.com>

* fix `[allow(unused_imports)] // docs`

* move DB filename from `helper/` -> `database/`

* config: use `DATABASE_FILENAME`

* config: add `db_file_path()`

* lib: add non-`service` usage invariant docs

* table: seal `Table` trait, impl for all `crate::tables`

* fix docs

* fix docs pt.2

---------

Co-authored-by: Boog900 <boog900@tutanota.com>
2024-02-25 19:46:36 +00:00

144 lines
4.7 KiB
Rust

//! Database tables.
//!
//! This module contains all the table definitions used by `cuprate-database`
//! and [`Tables`], an `enum` containing all [`Table`]s.
//---------------------------------------------------------------------------------------------------- Import
use crate::table::Table;
//---------------------------------------------------------------------------------------------------- Tables
/// Private module, should not be accessible outside this crate.
///
/// Used to block outsiders implementing [`Table`].
/// All [`Table`] types must also implement [`Sealed`].
pub(super) mod private {
/// Private sealed trait.
///
/// Cannot be implemented outside this crate.
pub trait Sealed {}
}
//---------------------------------------------------------------------------------------------------- Tables
/// An enumeration of _all_ database tables.
///
/// TODO: I don't think we need this.
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
#[cfg_attr(
feature = "borsh",
derive(borsh::BorshSerialize, borsh::BorshDeserialize)
)]
#[derive(Copy, Clone, Debug, PartialEq, PartialOrd, Eq, Ord, Hash)]
#[allow(missing_docs)]
pub enum Tables {
TestTable(TestTable),
TestTable2(TestTable2),
}
impl Tables {
/// Get the [`Table::NAME`].
pub const fn name(&self) -> &'static str {
/// Hack to access associated trait constant via a variable.
const fn get<T: Table>(t: &T) -> &'static str {
T::NAME
}
match self {
Self::TestTable(t) => get(t),
Self::TestTable2(t) => get(t),
}
}
/// Get the [`Table::CONSTANT_SIZE`].
pub const fn constant_size(&self) -> bool {
/// Hack to access associated trait constant via a variable.
const fn get<T: Table>(t: &T) -> bool {
T::CONSTANT_SIZE
}
match self {
Self::TestTable(t) => get(t),
Self::TestTable2(t) => get(t),
}
}
}
//---------------------------------------------------------------------------------------------------- Table macro
/// Create all tables, should be used _once_.
///
/// Generating this macro once and using `$()*` is probably
/// faster for compile times than calling the macro _per_ table.
///
/// All tables are zero-sized table structs, and implement the `Table` trait.
///
/// Table structs are automatically `CamelCase`,
/// and their static string names are automatically `snake_case`.
macro_rules! tables {
(
$(
$(#[$attr:meta])* // Documentation and any `derive`'s.
$table:ident, // The table name + doubles as the table struct name.
$size:literal, // Are the table's values all the same size?
$key:ty => // Key type.
$value:ty // Value type.
),* $(,)?
) => {
paste::paste! { $(
// Table struct.
$(#[$attr])*
// The below test show the `snake_case` table name in cargo docs.
/// ## Table Name
/// ```rust
/// # use cuprate_database::{*,tables::*};
#[doc = concat!(
"assert_eq!(",
stringify!([<$table:camel>]),
"::NAME, \"",
stringify!([<$table:snake>]),
"\");",
)]
/// ```
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
#[cfg_attr(feature = "borsh", derive(borsh::BorshSerialize, borsh::BorshDeserialize))]
#[derive(Copy,Clone,Debug,PartialEq,PartialOrd,Eq,Ord,Hash)]
pub struct [<$table:camel>];
// Implement the `Sealed` in this file.
// Required by `Table`.
impl private::Sealed for [<$table:camel>] {}
// Table trait impl.
impl Table for [<$table:camel>] {
const NAME: &'static str = stringify!([<$table:snake>]);
const CONSTANT_SIZE: bool = $size;
type Key = $key;
type Value = $value;
}
// Table enum.
impl From<[<$table:camel>]> for Tables {
fn from(table: [<$table:camel>]) -> Self {
Self::[<$table:camel>](table)
}
}
)* }
};
}
//---------------------------------------------------------------------------------------------------- Tables
tables! {
/// Test documentation.
TestTable,
true,
i64 => u64,
/// Test documentation 2.
TestTable2,
true,
u8 => i8,
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}