mirror of https://github.com/Cuprate/cuprate.git synced 2024-11-17 00:07:55 +00:00

* rename `database` -> `old_database`

Keeping it around for reference until new implementation is complete.

* create new `database/` skeleton

* add `DATABASE.md` design doc skeleton

* move design doc to `database/README.md`

* add rough code

* `lib.rs` -> `gist.rs`

* database: use workspace deps

* workspace: include `database/` as member

CI will now include this crate.

* cargo fmt

* database: `AGPL` -> `MIT`

* readme: add `TODO`s

* add base files

* cargo.toml: add `heed` feature

* fix clippy

* lib.rs: add extremely pedantic lints

* readme: add `# Backends`

* cargo.toml: add `cfg-if`

* add `backend/` structure

* base `database.rs`

* cargo.toml: add `borsh`

* backend: add `DATABASE_BACKEND`

* base `error.rs`

* base `database.rs`

* base `transaction.rs`

* base `table.rs`

* lib.rs: add imports

* add `pod.rs`

* pod: use `Read/Write`, add tests for all primitive numbers

* pod: impl Pod for `Vec<u8>`, `[u8; N]`

* pod: add docs, add `private::Sealed`

* pod: `to_writer`, `from_reader`

The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.

Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`

* pod: impl Pod for `Box<[u8]>`

* pod: return `Err` on incorrect length in `from_bytes()`

* pod: docs

* pod: impl Pod for `Arc<[u8]>`

* readme: docs

* database: add `create_table()`, `get_table()`

* table: `Pod` bound

* backend: move into directories

* pod: add `into_bytes()`

* heed: impl `BytesEncode`, `BytesDecode`

* add `actor`, `service` features

The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.

* add `lib.rs` docs

* service: add `service.rs`

* service: add `reader.rs`

* service: add `writer.rs`

* service: add `request.rs` & `response.rs`

* cargo.toml: add `crossbeam`

* service: add/use `enum Request`, `enum Response`

* service: basic signatures for thread-pools, `Writer` -> `Writers`

* service: split `tower::Service<ReadRequest/WriteRequest>`

* service: impl `tower::Service` for `Database(Reader|Writer)`

* service: add `init()`, impl basic `Reader/Writer` pools

* service: add example `Request`'s

* service: add example `ReadRequest` handling

* temporarily allow clippy lints

* readme: update file structure

* transaction: add `RoTx::get_range()`

* service: module docs

* cargo.toml: add `cuprate-helper`

* service: scale readers/writers based on thread count

* database: change lifetimes

* heed: impl Database for `heed`

* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits

* docs

* service: `read.rs` docs

* service: `write.rs` docs

* service: request/response docs

* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`

* service: leak database into `&'static`

* database: add `#[inline]`, `#[cold]`

* service: `free.rs` docs, more `#[inline]` + `#[cold]`

* service: add `shutdown()`

* service: add `Request::Shutdown`

* service: `shutdown()` docs

* heed: hide concrete tx types

* lib.rs: add terms

* split `Env` <-> `Database`

* cargo.toml: add `paste`

* database: add `tables/`

* impl `serde/borsh` where possible

* tables: add `Tables`, add test docs

* make db backend mutually exclusive to fix `--all-features`

* tables: use `$()*` in `tables!()`

* cargo.toml: add `sanakirja 1.4.0`

* sanakirja: impl `Env`

* sanakirja: impl `Database`

* sanakirja: impl `Transaction`

* table: temporarily fix `sanakirja` K/V bounds

* table: fix `#[cfg]`

* cargo.toml: fix deps

* lib.rs: docs

* service: docs

* readme: add files, update `# Documentation`, add `# Layers`

* readme: `src/` file purpose

* readme: `src/service/` file purpose

* readme: `src/backend/` file purpose

* fix `Cargo.lock` merge conflict

* database: remove `gist.rs`

* add to `constants.rs`

* add top `//! comments` for files/modules

* constants: add sanity-check test

* service: add `only_one_database` test in `free.rs`

* service: add `tests.rs`

* remove unneeded markers + imports

* backend: fix `get_range()`'s trait `impl` return

* env: add `create_tables_if_needed()`, don't return `Option<Db>`

* sort imports by `CONTRIBUTING.md` rules

oops sorry boog

* add `monero.rs`

* monero: docs

* database: add missing `RoTx/RwTx` inputs

* backend: add missing `RoTx/RwTx` inputs

* `monero.rs` trait -> free functions in `ops/`

* pod: make methods infallible

* ci: add `rustup update` step

* service: use `Arc` instead of leaking, remove `db_read/db_write`

* service: use `InfallibleOneshotReceiver` for readers

* service: shutdown on error, add todos

* service: remove `Request`

* service: combine `ReadResponse` and `WriteResponse`

* service: use `InfallibleOneshotReceiver` for writer

* service: only spawn 1 writer, don't allow cloning writer handle

* table: add associated `const CONSTANT_SIZE`

* add `key.rs` + `trait Key`, add bound to `Table`

* fix typos

2024-02-13 17:43:25 +00:00

5.1 KiB

Raw Blame History

Database

Cuprate's database implementation.

Documentation
File Structure
Backends
- heed
- sanakirja
Layers

Documentation

In general, documentation for database/ is split into 3:

Documentation location	Purpose
`database/README.md`	High level design of `cuprate-database`
`cuprate-database`	Practical usage documentation/warnings/notes/etc
Source file `// comments`	Implementation-specific details (e.g, how many reader threads to spawn?)

This README serves as the overview/design document.

For actual practical usage, cuprate-database's types and general usage are documented via standard Rust tooling.

Run:

cargo doc --package cuprate-database --open

at the root of the repo to open/read the documentation.

If this documentation is too abstract, refer to any of the source files, they are heavily commented. There are many // Regular comments that explain more implementation specific details that aren't present here or in the docs. Use the file reference below to find what you're looking for.

The code within src/ is also littered with some grep-able comments containing some keywords:

Word	Meaning
`INVARIANT`	This code makes an assumption that must be upheld for correctness
`SAFETY`	This `unsafe` code is okay, for `x,y,z` reasons
`FIXME`	This code works but isn't ideal
`HACK`	This code is a brittle workaround
`PERF`	This code is weird for performance reasons
`TODO`	This has to be implemented
`SOMEDAY`	This should be implemented... someday

File Structure

A quick reference of the structure of the folders & files in cuprate-database.

Note that lib.rs/mod.rs files are purely for re-exporting/visibility/lints, and contain no code. Each sub-directory has a corresponding mod.rs.

`src/`

The top-level src/ files.

File/Folder	Purpose
`constants.rs`	General constants used throughout `cuprate-database`
`database.rs`	Abstracted database; `trait Database`
`env.rs`	Abstracted database environment; `trait Env`
`error.rs`	Database error types
`free.rs`	General free functions (related to the database)
`key.rs`	Abstracted database keys; `trait Key`
`pod.rs`	Data (de)serialization; `trait Pod`
`table.rs`	Database table abstraction; `trait Table`
`tables.rs`	All the table definitions used by `cuprate-database`
`transaction.rs`	Database transaction abstraction; `trait RoTx`, `trait RwTx`

`src/service/`

This folder contains the cupate_database::service module.

File/Folder	Purpose
`free.rs`	General free functions used (related to `cuprate_database::service`)
`read.rs`	Read thread-pool definitions and logic
`request.rs`	Read/write `Request`s to the database
`response.rs`	Read/write `Response`'s from the database
`write.rs`	Write thread-pool definitions and logic

`src/backend/`

This folder contains the actual database crates used as the backend for cuprate-database.

Each backend has its own folder.

File/Folder	Purpose
`heed/`	Backend using using forked `heed`
`sanakirja/`	Backend using `sanakirja`

`src/backend/heed/`

File/Folder	Purpose
`database.rs`	Implementation of `trait Database`
`env.rs`	Implementation of `trait Env`
`serde.rs`	Data (de)serialization implementations
`transaction.rs`	Implementation of `trait RoTx/RwTx`

`src/backend/sanakirja/`

File/Folder	Purpose
`database.rs`	Implementation of `trait Database`
`env.rs`	Implementation of `trait Env`
`transaction.rs`	Implementation of `trait RoTx/RwTx`

Backends

cuprate-database's traits abstract over various actual databases.

Each database's implementation is located in its respective file in src/backend/${DATABASE_NAME}.rs.

`heed`

The default database used is a modified fork of heed, located at Cuprate/heed.

To generate documentation of the fork for local use:

git clone --recursive https://github.com/Cuprate/heed
cargo doc

LMDB should not need to be installed as heed has a build script that pulls it in automatically.

`sanakirja`

TODO

Layers

TODO: update with accurate information when ready, update image.

5.1 KiB

Raw Blame History

Database

Documentation

File Structure

`src/`

`src/service/`

`src/backend/`

`src/backend/heed/`

`src/backend/sanakirja/`

Backends

`heed`

`sanakirja`

Layers

Database

Trait

ConcreteDatabase

Thread

Service

5.1 KiB Raw Blame History

Database

Documentation

File Structure

src/

src/service/

src/backend/

src/backend/heed/

src/backend/sanakirja/

Backends

heed

sanakirja

Layers

Database

Trait

ConcreteDatabase

Thread

Service

5.1 KiB

Raw Blame History

`src/`

`src/service/`

`src/backend/`

`src/backend/heed/`

`src/backend/sanakirja/`

`heed`

`sanakirja`