SyntheticBird/cuprate-hinto-janai
1
0
Fork 0
mirror of https://github.com/hinto-janai/cuprate.git synced 2024-12-23 03:59:37 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Database (#35)

Browse source 
* rename `database` -> `old_database`

Keeping it around for reference until new implementation is complete.

* create new `database/` skeleton

* add `DATABASE.md` design doc skeleton

* move design doc to `database/README.md`

* add rough code

* `lib.rs` -> `gist.rs`

* database: use workspace deps

* workspace: include `database/` as member

CI will now include this crate.

* cargo fmt

* database: `AGPL` -> `MIT`

* readme: add `TODO`s

* add base files

* cargo.toml: add `heed` feature

* fix clippy

* lib.rs: add extremely pedantic lints

* readme: add `# Backends`

* cargo.toml: add `cfg-if`

* add `backend/` structure

* base `database.rs`

* cargo.toml: add `borsh`

* backend: add `DATABASE_BACKEND`

* base `error.rs`

* base `database.rs`

* base `transaction.rs`

* base `table.rs`

* lib.rs: add imports

* add `pod.rs`

* pod: use `Read/Write`, add tests for all primitive numbers

* pod: impl Pod for `Vec<u8>`, `[u8; N]`

* pod: add docs, add `private::Sealed`

* pod: `to_writer`, `from_reader`

The new `as_bytes` + `from_bytes` now allows (de)serializing from
bytes directly instead of going through Read/Write.

Different array return sizes are abstracted away with `-> impl AsRef<[u8]>`

* pod: impl Pod for `Box<[u8]>`

* pod: return `Err` on incorrect length in `from_bytes()`

* pod: docs

* pod: impl Pod for `Arc<[u8]>`

* readme: docs

* database: add `create_table()`, `get_table()`

* table: `Pod` bound

* backend: move into directories

* pod: add `into_bytes()`

* heed: impl `BytesEncode`, `BytesDecode`

* add `actor`, `service` features

The thread/actor system used will be gated behind `actor`,
and the `tower/tokio` integration will be gated behind `service`.

* add `lib.rs` docs

* service: add `service.rs`

* service: add `reader.rs`

* service: add `writer.rs`

* service: add `request.rs` & `response.rs`

* cargo.toml: add `crossbeam`

* service: add/use `enum Request`, `enum Response`

* service: basic signatures for thread-pools, `Writer` -> `Writers`

* service: split `tower::Service<ReadRequest/WriteRequest>`

* service: impl `tower::Service` for `Database(Reader|Writer)`

* service: add `init()`, impl basic `Reader/Writer` pools

* service: add example `Request`'s

* service: add example `ReadRequest` handling

* temporarily allow clippy lints

* readme: update file structure

* transaction: add `RoTx::get_range()`

* service: module docs

* cargo.toml: add `cuprate-helper`

* service: scale readers/writers based on thread count

* database: change lifetimes

* heed: impl Database for `heed`

* heed: add `ConcreteRoTx`, `ConcreteRwTx`, impl Tx traits

* docs

* service: `read.rs` docs

* service: `write.rs` docs

* service: request/response docs

* service: use `OnceLock` in `init()`, add `db_read()`, `db_write()`

* service: leak database into `&'static`

* database: add `#[inline]`, `#[cold]`

* service: `free.rs` docs, more `#[inline]` + `#[cold]`

* service: add `shutdown()`

* service: add `Request::Shutdown`

* service: `shutdown()` docs

* heed: hide concrete tx types

* lib.rs: add terms

* split `Env` <-> `Database`

* cargo.toml: add `paste`

* database: add `tables/`

* impl `serde/borsh` where possible

* tables: add `Tables`, add test docs

* make db backend mutually exclusive to fix `--all-features`

* tables: use `$()*` in `tables!()`

* cargo.toml: add `sanakirja 1.4.0`

* sanakirja: impl `Env`

* sanakirja: impl `Database`

* sanakirja: impl `Transaction`

* table: temporarily fix `sanakirja` K/V bounds

* table: fix `#[cfg]`

* cargo.toml: fix deps

* lib.rs: docs

* service: docs

* readme: add files, update `# Documentation`, add `# Layers`

* readme: `src/` file purpose

* readme: `src/service/` file purpose

* readme: `src/backend/` file purpose

* fix `Cargo.lock` merge conflict

* database: remove `gist.rs`

* add to `constants.rs`

* add top `//! comments` for files/modules

* constants: add sanity-check test

* service: add `only_one_database` test in `free.rs`

* service: add `tests.rs`

* remove unneeded markers + imports

* backend: fix `get_range()`'s trait `impl` return

* env: add `create_tables_if_needed()`, don't return `Option<Db>`

* sort imports by `CONTRIBUTING.md` rules

oops sorry boog

* add `monero.rs`

* monero: docs

* database: add missing `RoTx/RwTx` inputs

* backend: add missing `RoTx/RwTx` inputs

* `monero.rs` trait -> free functions in `ops/`

* pod: make methods infallible

* ci: add `rustup update` step

* service: use `Arc` instead of leaking, remove `db_read/db_write`

* service: use `InfallibleOneshotReceiver` for readers

* service: shutdown on error, add todos

* service: remove `Request`

* service: combine `ReadResponse` and `WriteResponse`

* service: use `InfallibleOneshotReceiver` for writer

* service: only spawn 1 writer, don't allow cloning writer handle

* table: add associated `const CONSTANT_SIZE`

* add `key.rs` + `trait Key`, add bound to `Table`

* fix typos
This commit is contained in:
hinto-janai 2024-02-13 12:43:25 -05:00 committed by GitHub
parent b7df133175
commit 331d4d3e7f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
52 changed files with 3596 additions and 455 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
.github/workflows/ci.yml vendored
View file

@ -88,6 +88,9 @@ jobs:
if: matrix.os == 'windows-latest' if: matrix.os == 'windows-latest'
run: rustup toolchain install stable-x86_64-pc-windows-gnu -c clippy && rustup set default-host x86_64-pc-windows-gnu && rustup default stable-x86_64-pc-windows-gnu run: rustup toolchain install stable-x86_64-pc-windows-gnu -c clippy && rustup set default-host x86_64-pc-windows-gnu && rustup default stable-x86_64-pc-windows-gnu
- name: Update Rust
run: rustup update
- name: Clippy (fail on warnings) - name: Clippy (fail on warnings)
run: cargo clippy --workspace --all-targets --all-features -- -D warnings run: cargo clippy --workspace --all-targets --all-features -- -D warnings

298
Cargo.lock generated
View file

@ -168,6 +168,15 @@ version = "1.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8c3c1a368f70d6cf7302d78f8f7093da241fb8e8807c05cc9e51a125895a6d5b" checksum = "8c3c1a368f70d6cf7302d78f8f7093da241fb8e8807c05cc9e51a125895a6d5b"
[[package]]
name = "bincode"
version = "1.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1f45e9417d87227c7a56d22e471c6206462cba514c7590c09aff4cf6d1ddcad"
dependencies = [
"serde",
]
[[package]] [[package]]
name = "bit-set" name = "bit-set"
version = "0.5.3" version = "0.5.3"
@ -194,6 +203,9 @@ name = "bitflags"
version = "2.4.2" version = "2.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed570934406eb16438a4e976b1b4500774099c13b8cb96eec99f620f05090ddf" checksum = "ed570934406eb16438a4e976b1b4500774099c13b8cb96eec99f620f05090ddf"
dependencies = [
"serde",
]
[[package]] [[package]]
name = "bitvec" name = "bitvec"
@ -255,6 +267,12 @@ version = "3.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f30e7476521f6f8af1a1c4c0b8cc94f0bee37d91763d0ca2665f299b6cd8aec" checksum = "7f30e7476521f6f8af1a1c4c0b8cc94f0bee37d91763d0ca2665f299b6cd8aec"
[[package]]
name = "bytemuck"
version = "1.14.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2ef034f05691a48569bd920a96c81b9d91bbad1ab5ac7c4616c1f6ef36cb79f"
[[package]] [[package]]
name = "byteorder" name = "byteorder"
version = "1.5.0" version = "1.5.0"
@ -290,11 +308,10 @@ dependencies = [
[[package]] [[package]]
name = "cc" name = "cc"
version = "1.0.83" version = "1.0.85"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1174fb0b6ec23863f8b971027804a42614e347eafb0a95bf0b12cdae21fc4d0" checksum = "9b918671670962b48bc23753aef0c51d072dca6f52f01f800854ada6ddb7f7d3"
dependencies = [ dependencies = [
"jobserver",
"libc", "libc",
] ]
@ -432,6 +449,19 @@ name = "crossbeam"
version = "0.8.4" version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1137cd7e7fc0fb5d3c5a8678be38ec56e819125d8d7907411fe24ccb943faca8" checksum = "1137cd7e7fc0fb5d3c5a8678be38ec56e819125d8d7907411fe24ccb943faca8"
dependencies = [
"crossbeam-channel",
"crossbeam-deque",
"crossbeam-epoch",
"crossbeam-queue",
"crossbeam-utils",
]
[[package]]
name = "crossbeam-channel"
version = "0.5.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "176dc175b78f56c0f321911d9c8eb2b77a78a4860b9c19db83835fea1a46649b"
dependencies = [ dependencies = [
"crossbeam-utils", "crossbeam-utils",
] ]
@ -455,6 +485,15 @@ dependencies = [
"crossbeam-utils", "crossbeam-utils",
] ]
[[package]]
name = "crossbeam-queue"
version = "0.3.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df0346b5d5e76ac2fe4e327c5fd1118d6be7c51dfb18f9b7922923f287471e35"
dependencies = [
"crossbeam-utils",
]
[[package]] [[package]]
name = "crossbeam-utils" name = "crossbeam-utils"
version = "0.8.19" version = "0.8.19"
@ -528,6 +567,23 @@ dependencies = [
"tracing-subscriber", "tracing-subscriber",
] ]
[[package]]
name = "cuprate-database"
version = "0.0.0"
dependencies = [
"borsh",
"cfg-if",
"crossbeam",
"cuprate-helper",
"heed",
"paste",
"sanakirja",
"serde",
"thiserror",
"tokio",
"tower",
]
[[package]] [[package]]
name = "cuprate-helper" name = "cuprate-helper"
version = "0.1.0" version = "0.1.0"
@ -674,6 +730,15 @@ dependencies = [
"zeroize", "zeroize",
] ]
[[package]]
name = "doxygen-rs"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "415b6ec780d34dcf624666747194393603d0373b7141eef01d12ee58881507d9"
dependencies = [
"phf",
]
[[package]] [[package]]
name = "either" name = "either"
version = "1.10.0" version = "1.10.0"
@ -770,7 +835,7 @@ checksum = "1ee447700ac8aa0b2f2bd7bc4462ad686ba06baa6727ac149a2d6277f0d240fd"
dependencies = [ dependencies = [
"cfg-if", "cfg-if",
"libc", "libc",
"redox_syscall", "redox_syscall 0.4.1",
"windows-sys 0.52.0", "windows-sys 0.52.0",
] ]
@ -835,6 +900,16 @@ dependencies = [
"percent-encoding", "percent-encoding",
] ]
[[package]]
name = "fs2"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9564fc758e15025b46aa6643b1b77d047d1a56a1aea6e01002ac0c7026876213"
dependencies = [
"libc",
"winapi",
]
[[package]] [[package]]
name = "funty" name = "funty"
version = "2.0.0" version = "2.0.0"
@ -990,12 +1065,58 @@ dependencies = [
"ahash", "ahash",
] ]
[[package]]
name = "hdrhistogram"
version = "7.5.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "765c9198f173dd59ce26ff9f95ef0aafd0a0fe01fb9d72841bc5066a4c06511d"
dependencies = [
"byteorder",
"num-traits",
]
[[package]] [[package]]
name = "heck" name = "heck"
version = "0.4.1" version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "95505c38b4572b2d910cecb0281560f54b440a19336cbbcb27bf6ce6adc6f5a8" checksum = "95505c38b4572b2d910cecb0281560f54b440a19336cbbcb27bf6ce6adc6f5a8"
[[package]]
name = "heed"
version = "0.20.0-alpha.9"
source = "git+https://github.com/Cuprate/heed?rev=5aa75b7#5aa75b7a44c8e572cf4957c35526e604e7a692ac"
dependencies = [
"bitflags 2.4.2",
"bytemuck",
"byteorder",
"heed-traits",
"heed-types",
"libc",
"lmdb-master-sys",
"once_cell",
"page_size",
"serde",
"synchronoise",
"url",
]
[[package]]
name = "heed-traits"
version = "0.20.0-alpha.9"
source = "git+https://github.com/Cuprate/heed?rev=5aa75b7#5aa75b7a44c8e572cf4957c35526e604e7a692ac"
[[package]]
name = "heed-types"
version = "0.20.0-alpha.9"
source = "git+https://github.com/Cuprate/heed?rev=5aa75b7#5aa75b7a44c8e572cf4957c35526e604e7a692ac"
dependencies = [
"bincode",
"byteorder",
"heed-traits",
"serde",
"serde_json",
]
[[package]] [[package]]
name = "hermit-abi" name = "hermit-abi"
version = "0.3.5" version = "0.3.5"
@ -1246,6 +1367,15 @@ dependencies = [
"generic-array", "generic-array",
] ]
[[package]]
name = "instant"
version = "0.1.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a5bbe824c507c5da5956355e86a746d82e0e1464f65d862cc5e71da70e94b2c"
dependencies = [
"cfg-if",
]
[[package]] [[package]]
name = "ipnet" name = "ipnet"
version = "2.9.0" version = "2.9.0"
@ -1258,15 +1388,6 @@ version = "1.0.10"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1a46d1a171d865aa5f83f92695765caa047a9b4cbae2cbf37dbd613a793fd4c" checksum = "b1a46d1a171d865aa5f83f92695765caa047a9b4cbae2cbf37dbd613a793fd4c"
[[package]]
name = "jobserver"
version = "0.1.28"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab46a6e9526ddef3ae7f787c06f0f2600639ba80ea3eade3d8e670a2230f51d6"
dependencies = [
"libc",
]
[[package]] [[package]]
name = "js-sys" name = "js-sys"
version = "0.3.68" version = "0.3.68"
@ -1320,7 +1441,7 @@ checksum = "85c833ca1e66078851dba29046874e38f08b2c883700aa29a03ddd3b23814ee8"
dependencies = [ dependencies = [
"bitflags 2.4.2", "bitflags 2.4.2",
"libc", "libc",
"redox_syscall", "redox_syscall 0.4.1",
] ]
[[package]] [[package]]
@ -1329,6 +1450,16 @@ version = "0.4.13"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "01cda141df6706de531b6c46c3a33ecca755538219bd484262fa09410c13539c" checksum = "01cda141df6706de531b6c46c3a33ecca755538219bd484262fa09410c13539c"
[[package]]
name = "lmdb-master-sys"
version = "0.1.0"
source = "git+https://github.com/Cuprate/heed?rev=5aa75b7#5aa75b7a44c8e572cf4957c35526e604e7a692ac"
dependencies = [
"cc",
"doxygen-rs",
"libc",
]
[[package]] [[package]]
name = "lock_api" name = "lock_api"
version = "0.4.11" version = "0.4.11"
@ -1361,6 +1492,16 @@ version = "2.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "523dc4f511e55ab87b694dc30d0f820d60906ef06413f93d4d7a1385599cc149" checksum = "523dc4f511e55ab87b694dc30d0f820d60906ef06413f93d4d7a1385599cc149"
[[package]]
name = "memmap"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6585fd95e7bb50d6cc31e20d4cf9afb4e2ba16c5846fc76793f11218da9c475b"
dependencies = [
"libc",
"winapi",
]
[[package]] [[package]]
name = "merlin" name = "merlin"
version = "3.0.0" version = "3.0.0"
@ -1666,12 +1807,33 @@ version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b15813163c1d831bf4a13c3610c05c0d03b39feb07f7e09fa234dac9b15aaf39" checksum = "b15813163c1d831bf4a13c3610c05c0d03b39feb07f7e09fa234dac9b15aaf39"
[[package]]
name = "page_size"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "30d5b2194ed13191c1999ae0704b7839fb18384fa22e49b57eeaa97d79ce40da"
dependencies = [
"libc",
"winapi",
]
[[package]] [[package]]
name = "parking" name = "parking"
version = "2.2.0" version = "2.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bb813b8af86854136c6922af0598d719255ecb2179515e6e7730d468f05c9cae" checksum = "bb813b8af86854136c6922af0598d719255ecb2179515e6e7730d468f05c9cae"
[[package]]
name = "parking_lot"
version = "0.11.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d17b78036a60663b797adeaee46f5c9dfebb86948d1255007a1d6be0271ff99"
dependencies = [
"instant",
"lock_api",
"parking_lot_core 0.8.6",
]
[[package]] [[package]]
name = "parking_lot" name = "parking_lot"
version = "0.12.1" version = "0.12.1"
@ -1679,7 +1841,21 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3742b2c103b9f06bc9fff0a37ff4912935851bee6d36f3c02bcc755bcfec228f" checksum = "3742b2c103b9f06bc9fff0a37ff4912935851bee6d36f3c02bcc755bcfec228f"
dependencies = [ dependencies = [
"lock_api", "lock_api",
"parking_lot_core", "parking_lot_core 0.9.9",
]
[[package]]
name = "parking_lot_core"
version = "0.8.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "60a2cfe6f0ad2bfc16aefa463b497d5c7a5ecd44a23efa72aa342d90177356dc"
dependencies = [
"cfg-if",
"instant",
"libc",
"redox_syscall 0.2.16",
"smallvec",
"winapi",
] ]
[[package]] [[package]]
@ -1690,7 +1866,7 @@ checksum = "4c42a9226546d68acdd9c0a280d17ce19bfe27a46bf68784e4066115788d008e"
dependencies = [ dependencies = [
"cfg-if", "cfg-if",
"libc", "libc",
"redox_syscall", "redox_syscall 0.4.1",
"smallvec", "smallvec",
"windows-targets 0.48.5", "windows-targets 0.48.5",
] ]
@ -1753,6 +1929,48 @@ version = "2.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3148f5046208a5d56bcfc03053e3ca6334e51da8dfb19b6cdc8b306fae3283e" checksum = "e3148f5046208a5d56bcfc03053e3ca6334e51da8dfb19b6cdc8b306fae3283e"
[[package]]
name = "phf"
version = "0.11.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ade2d8b8f33c7333b51bcf0428d37e217e9f32192ae4772156f65063b8ce03dc"
dependencies = [
"phf_macros",
"phf_shared",
]
[[package]]
name = "phf_generator"
version = "0.11.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "48e4cc64c2ad9ebe670cb8fd69dd50ae301650392e81c05f9bfcb2d5bdbc24b0"
dependencies = [
"phf_shared",
"rand",
]
[[package]]
name = "phf_macros"
version = "0.11.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3444646e286606587e49f3bcf1679b8cef1dc2c5ecc29ddacaffc305180d464b"
dependencies = [
"phf_generator",
"phf_shared",
"proc-macro2",
"quote",
"syn 2.0.48",
]
[[package]]
name = "phf_shared"
version = "0.11.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "90fcb95eef784c2ac79119d1dd819e162b5da872ce6f3c3abe1e8ca1c082f72b"
dependencies = [
"siphasher",
]
[[package]] [[package]]
name = "pin-project" name = "pin-project"
version = "1.1.4" version = "1.1.4"
@ -1981,6 +2199,15 @@ dependencies = [
"crossbeam-utils", "crossbeam-utils",
] ]
[[package]]
name = "redox_syscall"
version = "0.2.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb5a58c1855b4b6819d59012155603f0b22ad30cad752600aadfcb695265519a"
dependencies = [
"bitflags 1.3.2",
]
[[package]] [[package]]
name = "redox_syscall" name = "redox_syscall"
version = "0.4.1" version = "0.4.1"
@ -2195,6 +2422,27 @@ version = "1.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f98d2aa92eebf49b69786be48e4477826b256916e84a57ff2a4f21923b48eb4c" checksum = "f98d2aa92eebf49b69786be48e4477826b256916e84a57ff2a4f21923b48eb4c"
[[package]]
name = "sanakirja"
version = "1.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c385eb43079aa7dc6204e473b68b4305ceaea8048dda3a985a339bbb57cde72"
dependencies = [
"fs2",
"log",
"memmap",
"parking_lot 0.11.2",
"sanakirja-core",
"serde",
"thiserror",
]
[[package]]
name = "sanakirja-core"
version = "1.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8376db34ae3eac6e7bd91168bc638450073b708ce9fb46940de676f552238bf5"
[[package]] [[package]]
name = "schannel" name = "schannel"
version = "0.1.23" version = "0.1.23"
@ -2357,6 +2605,12 @@ dependencies = [
"tower-service", "tower-service",
] ]
[[package]]
name = "siphasher"
version = "0.3.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "38b58827f4464d87d377d175e90bf58eb00fd8716ff0a62f80356b5e61555d0d"
[[package]] [[package]]
name = "slab" name = "slab"
version = "0.4.9" version = "0.4.9"
@ -2449,6 +2703,15 @@ version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2047c6ded9c721764247e62cd3b03c09ffc529b2ba5b10ec482ae507a4a70160" checksum = "2047c6ded9c721764247e62cd3b03c09ffc529b2ba5b10ec482ae507a4a70160"
[[package]]
name = "synchronoise"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3dbc01390fc626ce8d1cffe3376ded2b72a11bb70e1c75f404a210e4daa4def2"
dependencies = [
"crossbeam-queue",
]
[[package]] [[package]]
name = "system-configuration" name = "system-configuration"
version = "0.5.1" version = "0.5.1"
@ -2583,7 +2846,7 @@ dependencies = [
"libc", "libc",
"mio", "mio",
"num_cpus", "num_cpus",
"parking_lot", "parking_lot 0.12.1",
"pin-project-lite", "pin-project-lite",
"signal-hook-registry", "signal-hook-registry",
"socket2", "socket2",
@ -2674,6 +2937,7 @@ checksum = "b8fa9be0de6cf49e536ce1851f987bd21a43b771b09473c3549a6c853db37c1c"
dependencies = [ dependencies = [
"futures-core", "futures-core",
"futures-util", "futures-util",
"hdrhistogram",
"indexmap 1.9.3", "indexmap 1.9.3",
"pin-project", "pin-project",
"pin-project-lite", "pin-project-lite",

4
Cargo.toml
View file

@ -5,6 +5,7 @@ members = [
"consensus", "consensus",
"consensus/rules", "consensus/rules",
"cryptonight", "cryptonight",
"database",
"helper", "helper",
"net/epee-encoding", "net/epee-encoding",
"net/fixed-bytes", "net/fixed-bytes",
@ -38,6 +39,7 @@ opt-level = 3
async-trait = { version = "0.1.74", default-features = false } async-trait = { version = "0.1.74", default-features = false }
borsh = { version = "1.2.1", default-features = false } borsh = { version = "1.2.1", default-features = false }
bytes = { version = "1.5.0", default-features = false } bytes = { version = "1.5.0", default-features = false }
cfg-if = { version = "1.0.0", default-features = false }
clap = { version = "4.4.7", default-features = false } clap = { version = "4.4.7", default-features = false }
chrono = { version = "0.4.31", default-features = false } chrono = { version = "0.4.31", default-features = false }
crypto-bigint = { version = "0.5.5", default-features = false } crypto-bigint = { version = "0.5.5", default-features = false }
@ -50,6 +52,7 @@ hex = { version = "0.4.3", default-features = false }
hex-literal = { version = "0.4", default-features = false } hex-literal = { version = "0.4", default-features = false }
monero-serai = { git = "https://github.com/Cuprate/serai.git", rev = "347d4cf", default-features = false } monero-serai = { git = "https://github.com/Cuprate/serai.git", rev = "347d4cf", default-features = false }
multiexp = { git = "https://github.com/Cuprate/serai.git", rev = "347d4cf", default-features = false } multiexp = { git = "https://github.com/Cuprate/serai.git", rev = "347d4cf", default-features = false }
paste = { version = "1.0.14", default-features = false }
pin-project = { version = "1.1.3", default-features = false } pin-project = { version = "1.1.3", default-features = false }
randomx-rs = { git = "https://github.com/Cuprate/randomx-rs.git", rev = "0028464", default-features = false } randomx-rs = { git = "https://github.com/Cuprate/randomx-rs.git", rev = "0028464", default-features = false }
rand = { version = "0.8.5", default-features = false } rand = { version = "0.8.5", default-features = false }
@ -79,7 +82,6 @@ proptest-derive = { version = "0.4.0" }
# notify = { version = "6.1.1" } # Filesystem watching | https://github.com/notify-rs/notify # notify = { version = "6.1.1" } # Filesystem watching | https://github.com/notify-rs/notify
# once_cell = { version = "1.18.0" } # Lazy/one-time initialization | https://github.com/matklad/once_cell # once_cell = { version = "1.18.0" } # Lazy/one-time initialization | https://github.com/matklad/once_cell
# open = { version = "5.0.0" } # Open PATH/URL, probably for binaries | https://github.com/byron/open-rs # open = { version = "5.0.0" } # Open PATH/URL, probably for binaries | https://github.com/byron/open-rs
# paste = { version = "1.0.14" } # Macro `ident` pasting | https://github.com/dtolnay/paste
# regex = { version = "1.10.2" } # Regular expressions | https://github.com/rust-lang/regex # regex = { version = "1.10.2" } # Regular expressions | https://github.com/rust-lang/regex
# ryu = { version = "1.0.15" } # Fast float to string formatting | https://github.com/dtolnay/ryu # ryu = { version = "1.0.15" } # Fast float to string formatting | https://github.com/dtolnay/ryu
# strum = { version = "0.25.0" } # Enum macros/traits | https://github.com/Peternator7/strum # strum = { version = "0.25.0" } # Enum macros/traits | https://github.com/Peternator7/strum

50
database/Cargo.toml
View file

@ -1,33 +1,33 @@
[package] [package]
name = "cuprate-database" name = "cuprate-database"
version = "0.0.1" version = "0.0.0"
edition = "2021" edition = "2021"
license = "AGPL-3.0-only" description = "Cuprate's database abstraction"
license = "MIT"
# All Contributors on github authors = ["hinto-janai"]
authors=[ repository = "https://github.com/Cuprate/cuprate/tree/main/database"
"SyntheticBird45 <@someoneelse495495:matrix.org>", keywords = ["cuprate", "database"]
"Boog900"
]
[features] [features]
mdbx = ["dep:libmdbx"] default = ["heed", "service"]
hse = [] # default = ["sanakirja", "service"] # For testing `sanakirja`.
heed = ["dep:heed"]
sanakirja = ["dep:sanakirja"]
service = ["dep:cuprate-helper", "dep:crossbeam", "dep:tokio", "dep:tower"]
[dependencies] [dependencies]
monero = {workspace = true, features = ["serde"]} cfg-if = { workspace = true }
tiny-keccak = { version = "2.0", features = ["sha3"] } paste = { workspace = true }
serde = { workspace = true} thiserror = { workspace = true }
thiserror = {workspace = true }
bincode = { workspace = true }
libmdbx = { version = "0.3.1", optional = true }
[build] # `service` feature.
linker="clang" cuprate-helper = { path = "../helper", features = ["thread"], optional = true }
rustflags=[ crossbeam = { workspace = true, features = ["std"], optional = true }
"-Clink-arg=-fuse-ld=mold", tokio = { workspace = true, features = ["full"], optional = true }
"-Zcf-protection=full", tower = { workspace = true, features = ["full"], optional = true }
"-Zsanitizer=cfi",
"-Crelocation-model=pie", # Optional features.
"-Cstack-protector=all", borsh = { workspace = true, optional = true }
] heed = { git = "https://github.com/Cuprate/heed", rev = "5aa75b7", optional = true }
sanakirja = { version = "1.4.0", optional = true }
serde = { workspace = true, optional = true }

137
database/README.md Normal file
View file

@ -0,0 +1,137 @@
# Database
Cuprate's database implementation.
<!-- Did you know markdown automatically increments number lists, even if they are all 1...? -->
1. [Documentation](#documentation)
1. [File Structure](#file-structure)
- [`src/`](#src)
- [`src/service/`](#src-service)
- [`src/backend/`](#src-backend)
1. [Backends](#backends)
- [`heed`](#heed)
- [`sanakirja`](#sanakirja)
1. [Layers](#layers)
- [Database](#database)
- [Trait](#trait)
- [ConcreteDatabase](#concretedatabase)
- [Thread-pool](#thread-pool)
- [Service](#service)
---
# Documentation
In general, documentation for `database/` is split into 3:
| Documentation location | Purpose |
|---------------------------|---------|
| `database/README.md` | High level design of `cuprate-database`
| `cuprate-database` | Practical usage documentation/warnings/notes/etc
| Source file `// comments` | Implementation-specific details (e.g, how many reader threads to spawn?)
This README serves as the overview/design document.
For actual practical usage, `cuprate-database`'s types and general usage are documented via standard Rust tooling.
Run:
```bash
cargo doc --package cuprate-database --open
```
at the root of the repo to open/read the documentation.
If this documentation is too abstract, refer to any of the source files, they are heavily commented. There are many `// Regular comments` that explain more implementation specific details that aren't present here or in the docs. Use the file reference below to find what you're looking for.
The code within `src/` is also littered with some `grep`-able comments containing some keywords:
| Word | Meaning |
|-------------|---------|
| `INVARIANT` | This code makes an _assumption_ that must be upheld for correctness
| `SAFETY` | This `unsafe` code is okay, for `x,y,z` reasons
| `FIXME` | This code works but isn't ideal
| `HACK` | This code is a brittle workaround
| `PERF` | This code is weird for performance reasons
| `TODO` | This has to be implemented
| `SOMEDAY` | This should be implemented... someday
# File Structure
A quick reference of the structure of the folders & files in `cuprate-database`.
Note that `lib.rs/mod.rs` files are purely for re-exporting/visibility/lints, and contain no code. Each sub-directory has a corresponding `mod.rs`.
## `src/`
The top-level `src/` files.
| File/Folder | Purpose |
|------------------|---------|
| `constants.rs` | General constants used throughout `cuprate-database`
| `database.rs` | Abstracted database; `trait Database`
| `env.rs` | Abstracted database environment; `trait Env`
| `error.rs` | Database error types
| `free.rs` | General free functions (related to the database)
| `key.rs` | Abstracted database keys; `trait Key`
| `pod.rs` | Data (de)serialization; `trait Pod`
| `table.rs` | Database table abstraction; `trait Table`
| `tables.rs` | All the table definitions used by `cuprate-database`
| `transaction.rs` | Database transaction abstraction; `trait RoTx`, `trait RwTx`
## `src/service/`
This folder contains the `cupate_database::service` module.
| File/Folder | Purpose |
|----------------|---------|
| `free.rs` | General free functions used (related to `cuprate_database::service`)
| `read.rs` | Read thread-pool definitions and logic
| `request.rs` | Read/write `Request`s to the database
| `response.rs` | Read/write `Response`'s from the database
| `write.rs` | Write thread-pool definitions and logic
## `src/backend/`
This folder contains the actual database crates used as the backend for `cuprate-database`.
Each backend has its own folder.
| File/Folder | Purpose |
|--------------|---------|
| `heed/` | Backend using using forked [`heed`](https://github.com/Cuprate/heed)
| `sanakirja/` | Backend using [`sanakirja`](https://docs.rs/sanakirja)
### `src/backend/heed/`
| File/Folder | Purpose |
|------------------|---------|
| `database.rs` | Implementation of `trait Database`
| `env.rs` | Implementation of `trait Env`
| `serde.rs` | Data (de)serialization implementations
| `transaction.rs` | Implementation of `trait RoTx/RwTx`
### `src/backend/sanakirja/`
| File/Folder | Purpose |
|------------------|---------|
| `database.rs` | Implementation of `trait Database`
| `env.rs` | Implementation of `trait Env`
| `transaction.rs` | Implementation of `trait RoTx/RwTx`
# Backends
`cuprate-database`'s `trait`s abstract over various actual databases.
Each database's implementation is located in its respective file in `src/backend/${DATABASE_NAME}.rs`.
## `heed`
The default database used is a modified fork of [`heed`](https://github.com/meilisearch/heed), located at [`Cuprate/heed`](https://github.com/Cuprate/heed).
To generate documentation of the fork for local use:
```bash
git clone --recursive https://github.com/Cuprate/heed
cargo doc
```
`LMDB` should not need to be installed as `heed` has a build script that pulls it in automatically.
## `sanakirja`
TODO
# Layers
TODO: update with accurate information when ready, update image.
## Database
## Trait
## ConcreteDatabase
## Thread
## Service

47
database/src/backend/heed/database.rs Normal file
View file

@ -0,0 +1,47 @@
//! Implementation of `trait Database` for `heed`.
//---------------------------------------------------------------------------------------------------- Import
use crate::{database::Database, error::RuntimeError, table::Table};
//---------------------------------------------------------------------------------------------------- Database Impls
impl<T: Table> Database<T> for heed::Database<T::Key, T::Value> {
type RoTx<'db> = heed::RoTxn<'db>;
type RwTx<'db> = heed::RwTxn<'db>;
fn get(&self, ro_tx: &Self::RoTx<'_>, key: &T::Key) -> Result<Option<T::Value>, RuntimeError> {
todo!()
}
fn get_range(
&self,
ro_tx: &Self::RoTx<'_>,
key: &T::Key,
amount: usize,
) -> Result<impl Iterator<Item = T::Value>, RuntimeError> {
let iter: std::vec::Drain<'_, T::Value> = todo!();
Ok(iter)
}
fn put(
&mut self,
rw_tx: &mut Self::RwTx<'_>,
key: &T::Key,
value: &T::Value,
) -> Result<(), RuntimeError> {
todo!()
}
fn clear(&mut self, rw_tx: &mut Self::RwTx<'_>) -> Result<(), RuntimeError> {
todo!()
}
fn delete(&mut self, rw_tx: &mut Self::RwTx<'_>, key: &T::Key) -> Result<bool, RuntimeError> {
todo!()
}
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

82
database/src/backend/heed/env.rs Normal file
View file

@ -0,0 +1,82 @@
//! Implementation of `trait Env` for `heed`.
//---------------------------------------------------------------------------------------------------- Import
use std::path::Path;
use crate::{database::Database, env::Env, error::RuntimeError, table::Table};
//---------------------------------------------------------------------------------------------------- Env
/// A strongly typed, concrete database environment, backed by `heed`.
pub struct ConcreteEnv(heed::Env);
//---------------------------------------------------------------------------------------------------- Env Impl
impl Env for ConcreteEnv {
/// TODO
type RoTx<'db> = heed::RoTxn<'db>;
/// TODO
type RwTx<'db> = heed::RwTxn<'db>;
//------------------------------------------------ Required
#[cold]
#[inline(never)] // called once.
/// TODO
/// # Errors
/// TODO
fn open<P: AsRef<Path>>(path: P) -> Result<Self, RuntimeError> {
todo!()
}
/// TODO
/// # Errors
/// TODO
fn sync(&self) -> Result<(), RuntimeError> {
todo!()
}
#[inline]
/// TODO
/// # Errors
/// TODO
fn ro_tx(&self) -> Result<Self::RoTx<'_>, RuntimeError> {
todo!()
}
#[inline]
/// TODO
/// # Errors
/// TODO
fn rw_tx(&self) -> Result<Self::RwTx<'_>, RuntimeError> {
todo!()
}
#[cold]
#[inline(never)] // called infrequently?.
/// TODO
/// # Errors
/// TODO
fn create_tables_if_needed<T: Table>(
&self,
tx_rw: &mut Self::RwTx<'_>,
) -> Result<(), RuntimeError> {
todo!()
}
#[inline]
/// TODO
/// # Errors
/// TODO
fn open_database<T: Table>(
&self,
to_rw: &Self::RoTx<'_>,
) -> Result<impl Database<T>, RuntimeError> {
let tx: heed::Database<T::Key, T::Value> = todo!();
Ok(tx)
}
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

9
database/src/backend/heed/mod.rs Normal file
View file

@ -0,0 +1,9 @@
//! Database backend implementation backed by `heed`.
mod env;
pub use env::ConcreteEnv;
mod database;
mod serde;
mod transaction;

53
database/src/backend/heed/serde.rs Normal file
View file

@ -0,0 +1,53 @@
//! (De)serialization trait implementations for `heed`.
//---------------------------------------------------------------------------------------------------- Import
use std::borrow::Cow;
use crate::pod::Pod;
//---------------------------------------------------------------------------------------------------- Serde
/// Implement `heed` (de)serialization traits
/// for anything that implements [`crate::pod::Pod`].
///
/// Blanket implementation breaks orphan impl rules, so this is used instead.
macro_rules! impl_heed {
($(
$name:ident => // The name that implements [`crate::pod::Pod`]
$t:ident // The type to (de)serialize into/from
),* $(,)?) => {
$(
// `heed` Encode.
impl<'a> heed::BytesEncode<'a> for $name {
type EItem = $t;
#[inline]
fn bytes_encode(item: &'a Self::EItem) -> Result<Cow<'a, [u8]>, heed::BoxedError> {
Ok(item.into_bytes())
}
}
// `heed` Decode.
impl<'a> heed::BytesDecode<'a> for $name {
type DItem = $t;
#[inline]
fn bytes_decode(bytes: &'a [u8]) -> Result<Self::DItem, heed::BoxedError> {
Ok(Pod::from_bytes(bytes))
}
}
)*
};
}
/// TODO
struct Test;
impl_heed! {
Test => u8,
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

35
database/src/backend/heed/transaction.rs Normal file
View file

@ -0,0 +1,35 @@
//! Implementation of `trait RoTx/RwTx` for `heed`.
//---------------------------------------------------------------------------------------------------- Import
use crate::{
error::RuntimeError,
transaction::{RoTx, RwTx},
};
//---------------------------------------------------------------------------------------------------- RoTx
impl RoTx<'_> for heed::RoTxn<'_> {
fn commit(self) -> Result<(), RuntimeError> {
todo!()
}
}
//---------------------------------------------------------------------------------------------------- RwTx
impl RwTx<'_> for heed::RwTxn<'_> {
/// TODO
/// # Errors
/// TODO
fn commit(self) -> Result<(), RuntimeError> {
todo!()
}
/// TODO
fn abort(self) {
todo!()
}
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

22
database/src/backend/mod.rs Normal file
View file

@ -0,0 +1,22 @@
//! Database backends.
//!
//! TODO:
//! Create a test backend backed by `std::collections::HashMap`.
//!
//! The full type could be something like `HashMap<&'static str, HashMap<K, V>>`.
//! where the `str` is the table name, and the containing hashmap are are the
//! key and values.
//!
//! Not sure how duplicate keys will work.
cfg_if::cfg_if! {
// If both backends are enabled, fallback to `heed`.
// This is useful when using `--all-features`.
if #[cfg(all(feature = "sanakirja", not(feature = "heed")))] {
mod sanakirja;
pub use sanakirja::ConcreteEnv;
} else {
mod heed;
pub use heed::ConcreteEnv;
}
}

47
database/src/backend/sanakirja/database.rs Normal file
View file

@ -0,0 +1,47 @@
//! Implementation of `trait Database` for `sanakirja`.
//---------------------------------------------------------------------------------------------------- Import
use crate::{database::Database, error::RuntimeError, table::Table};
//---------------------------------------------------------------------------------------------------- Database Impls
impl<T: Table> Database<T> for sanakirja::btree::Db<T::Key, T::Value> {
type RoTx<'db> = sanakirja::Txn<&'_ sanakirja::Env>;
type RwTx<'db> = sanakirja::MutTxn<&'_ sanakirja::Env, ()>;
fn get(&self, ro_tx: &Self::RoTx<'_>, key: &T::Key) -> Result<Option<T::Value>, RuntimeError> {
todo!()
}
fn get_range(
&self,
ro_tx: &Self::RoTx<'_>,
key: &T::Key,
amount: usize,
) -> Result<impl Iterator<Item = T::Value>, RuntimeError> {
let iter: std::vec::Drain<'_, T::Value> = todo!();
Ok(iter)
}
fn put(
&mut self,
rx_tx: &mut Self::RwTx<'_>,
key: &T::Key,
value: &T::Value,
) -> Result<(), RuntimeError> {
todo!()
}
fn clear(&mut self, rx_tx: &mut Self::RwTx<'_>) -> Result<(), RuntimeError> {
todo!()
}
fn delete(&mut self, rx_tx: &mut Self::RwTx<'_>, key: &T::Key) -> Result<bool, RuntimeError> {
todo!()
}
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

85
database/src/backend/sanakirja/env.rs Normal file
View file

@ -0,0 +1,85 @@
//! Implementation of `trait Env` for `sanakirja`.
//---------------------------------------------------------------------------------------------------- Import
use std::path::Path;
use crate::{database::Database, env::Env, error::RuntimeError, table::Table};
//---------------------------------------------------------------------------------------------------- ConcreteEnv
/// A strongly typed, concrete database environment, backed by `sanakirja`.
pub struct ConcreteEnv(sanakirja::Env);
//---------------------------------------------------------------------------------------------------- Env Impl
impl Env for ConcreteEnv {
/// TODO
///
/// We could also implement `Borrow<sanakirja::Env> for ConcreteEnv`
/// instead of this reference.
type RoTx<'db> = sanakirja::Txn<&'db sanakirja::Env>;
/// TODO
type RwTx<'db> = sanakirja::MutTxn<&'db sanakirja::Env, ()>;
//------------------------------------------------ Required
#[cold]
#[inline(never)] // called once.
/// TODO
/// # Errors
/// TODO
fn open<P: AsRef<Path>>(path: P) -> Result<Self, RuntimeError> {
todo!()
}
/// TODO
/// # Errors
/// TODO
fn sync(&self) -> Result<(), RuntimeError> {
todo!()
}
#[inline]
/// TODO
/// # Errors
/// TODO
fn ro_tx(&self) -> Result<Self::RoTx<'_>, RuntimeError> {
todo!()
}
#[inline]
/// TODO
/// # Errors
/// TODO
fn rw_tx(&self) -> Result<Self::RwTx<'_>, RuntimeError> {
todo!()
}
#[cold]
#[inline(never)] // called infrequently?.
/// TODO
/// # Errors
/// TODO
fn create_tables_if_needed<T: Table>(
&self,
tx_rw: &mut Self::RwTx<'_>,
) -> Result<(), RuntimeError> {
todo!()
}
#[inline]
/// TODO
/// # Errors
/// TODO
fn open_database<T: Table>(
&self,
to_rw: &Self::RoTx<'_>,
) -> Result<Option<impl Database<T>>, RuntimeError> {
let tx: sanakirja::btree::Db<T::Key, T::Value> = todo!();
Ok(Some(tx))
}
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

8
database/src/backend/sanakirja/mod.rs Normal file
View file

@ -0,0 +1,8 @@
//! Database backend implementation backed by `sanakirja`.
mod env;
pub use env::ConcreteEnv;
mod database;
mod transaction;

35
database/src/backend/sanakirja/transaction.rs Normal file
View file

@ -0,0 +1,35 @@
//! Implementation of `trait RoTx/RwTx` for `sanakirja`.
//---------------------------------------------------------------------------------------------------- Import
use crate::{
error::RuntimeError,
transaction::{RoTx, RwTx},
};
//---------------------------------------------------------------------------------------------------- RoTx
impl RoTx<'_> for sanakirja::Txn<&'_ sanakirja::Env> {
fn commit(self) -> Result<(), RuntimeError> {
todo!()
}
}
//---------------------------------------------------------------------------------------------------- RwTx
impl RwTx<'_> for sanakirja::MutTxn<&'_ sanakirja::Env, ()> {
/// TODO
/// # Errors
/// TODO
fn commit(self) -> Result<(), RuntimeError> {
todo!()
}
/// TODO
fn abort(self) {
todo!()
}
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

51
database/src/constants.rs Normal file
View file

@ -0,0 +1,51 @@
//! General constants used throughout `cuprate-database`.
//---------------------------------------------------------------------------------------------------- Import
//---------------------------------------------------------------------------------------------------- Constants
/// The directory that contains database-related files.
///
/// This is a sub-directory within the Cuprate folder, e.g:
/// ```txt
/// ~/.local/share/cuprate/
/// ├─ database/ # <-
/// ├─ data.mdb
/// ├─ lock.mdb
/// ```
pub const CUPRATE_DATABASE_DIR: &str = "database";
/// The actual database file name.
///
/// This is a _file_ within [`CUPRATE_DATABASE_DIR`], e.g:
/// ```txt
/// ~/.local/share/cuprate/
/// ├─ database/
/// ├─ data.mdb # <-
/// ├─ lock.mdb
/// ```
pub const CUPRATE_DATABASE_FILE: &str = "data";
cfg_if::cfg_if! {
// If both backends are enabled, fallback to `heed`.
// This is useful when using `--all-features`.
if #[cfg(all(feature = "sanakirja", not(feature = "heed")))] {
/// Static string of the `crate` being used as the database backend.
pub const DATABASE_BACKEND: &str = "sanakirja";
} else {
/// Static string of the `crate` being used as the database backend.
pub const DATABASE_BACKEND: &str = "heed";
}
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
use super::*;
#[test]
/// Sanity check that our PATHs aren't empty... (will cause disaster).
fn non_empty_path() {
assert!(!CUPRATE_DATABASE_DIR.is_empty());
assert!(!CUPRATE_DATABASE_FILE.is_empty());
}
}

58
database/src/database.rs Normal file
View file

@ -0,0 +1,58 @@
//! Abstracted database; `trait Database`.
//---------------------------------------------------------------------------------------------------- Import
use crate::{
error::RuntimeError,
table::Table,
transaction::{RoTx, RwTx},
};
//---------------------------------------------------------------------------------------------------- Database
/// Database (key-value store) abstraction.
///
/// TODO
pub trait Database<T: Table> {
//------------------------------------------------ Types
/// TODO
type RoTx<'db>: RoTx<'db>;
/// TODO
type RwTx<'db>: RwTx<'db>;
//-------------------------------------------------------- Read
/// TODO
/// # Errors
/// TODO
fn get(&self, ro_tx: &Self::RoTx<'_>, key: &T::Key) -> Result<Option<T::Value>, RuntimeError>;
/// TODO
/// # Errors
/// TODO
fn get_range(
&self,
ro_tx: &Self::RoTx<'_>,
key: &T::Key,
amount: usize,
) -> Result<impl Iterator<Item = T::Value>, RuntimeError>;
//-------------------------------------------------------- Write
/// TODO
/// # Errors
/// TODO
fn put(
&mut self,
rw_tx: &mut Self::RwTx<'_>,
key: &T::Key,
value: &T::Value,
) -> Result<(), RuntimeError>;
/// TODO
/// # Errors
/// TODO
fn clear(&mut self, rw_tx: &mut Self::RwTx<'_>) -> Result<(), RuntimeError>;
/// TODO
/// # Errors
/// TODO
fn delete(&mut self, rw_tx: &mut Self::RwTx<'_>, key: &T::Key) -> Result<bool, RuntimeError>;
}

71
database/src/env.rs Normal file
View file

@ -0,0 +1,71 @@
//! Abstracted database environment; `trait Env`.
//---------------------------------------------------------------------------------------------------- Import
use std::path::Path;
use crate::{
database::Database,
error::RuntimeError,
table::Table,
transaction::{RoTx, RwTx},
};
//---------------------------------------------------------------------------------------------------- Env
/// Database environment abstraction.
///
/// TODO
pub trait Env: Sized {
//------------------------------------------------ Types
/// TODO
type RoTx<'db>: RoTx<'db>;
/// TODO
type RwTx<'db>: RwTx<'db>;
//------------------------------------------------ Required
/// TODO
/// # Errors
/// TODO
fn open<P: AsRef<Path>>(path: P) -> Result<Self, RuntimeError>;
/// TODO
/// # Errors
/// TODO
fn sync(&self) -> Result<(), RuntimeError>;
/// TODO
/// # Errors
/// TODO
fn ro_tx(&self) -> Result<Self::RoTx<'_>, RuntimeError>;
/// TODO
/// # Errors
/// TODO
fn rw_tx(&self) -> Result<Self::RwTx<'_>, RuntimeError>;
/// TODO
/// # Errors
/// TODO
fn create_tables_if_needed<T: Table>(
&self,
rw_tx: &mut Self::RwTx<'_>,
) -> Result<(), RuntimeError>;
/// TODO
///
/// # TODO: Invariant
/// This should never panic the database because the table doesn't exist.
///
/// Opening/using the database env should have an invariant
/// that it creates all the tables we need, such that this
/// never returns `None`.
///
/// # Errors
/// TODO
fn open_database<T: Table>(
&self,
ro_tx: &Self::RoTx<'_>,
) -> Result<impl Database<T>, RuntimeError>;
//------------------------------------------------ Provided
}

97
database/src/error.rs
View file

@ -1,53 +1,62 @@
//! ### Error module //! Database error types.
//! This module contains all errors abstraction used by the database crate. By implementing [`From<E>`] to the specific errors of storage engine crates, it let us //! TODO: `InitError/RuntimeError` are maybe bad names.
//! handle more easily any type of error that can happen. This module does **NOT** contain interpretation of these errors, as these are defined for Blockchain abstraction. This is another difference
//! from monerod which interpret these errors directly in its database functions:
//! ```cpp
//! /**
//! * @brief A base class for BlockchainDB exceptions
//! */
//! class DB_EXCEPTION : public std::exception
//! ```
//! see `blockchain_db/blockchain_db.h` in monerod `src/` folder for more details.
//---------------------------------------------------------------------------------------------------- Import
use std::{borrow::Cow, fmt::Debug};
use crate::constants::DATABASE_BACKEND;
//---------------------------------------------------------------------------------------------------- InitError
/// Database errors that occur during initialization.
///
/// `BackendError` is an error specifically from the
/// database backend being used. TODO: this may not
/// be needed if we can convert all error types into
/// "generic" database errors.
#[derive(thiserror::Error, Debug)] #[derive(thiserror::Error, Debug)]
/// `DB_FAILURES` is an enum for backend-agnostic, internal database errors. The `From` Trait must be implemented to the specific backend errors to match DB_FAILURES. pub enum InitError<BackendError: Debug> {
pub enum DB_FAILURES { /// TODO
#[error("MDBX returned an error {0}")] #[error("database PATH is inaccessible: {0}")]
MDBX_Error(#[from] libmdbx::Error), Path(std::io::Error),
#[error("\n<DB_FAILURES::EncodingError> Failed to encode some data : `{0}`")] /// TODO
SerializeIssue(DB_SERIAL), #[error("{DATABASE_BACKEND} error: {0}")]
Backend(BackendError),
#[error("\nObject already exist in the database : {0}")] /// TODO
AlreadyExist(&'static str), ///
/// An unknown error occurred.
#[error("NotFound? {0}")] #[error("unknown error: {0}")]
NotFound(&'static str), Unknown(Cow<'static, str>),
#[error("\n<DB_FAILURES::Other> `{0}`")]
Other(&'static str),
#[error(
"\n<DB_FAILURES::FailedToCommit> A transaction tried to commit to the db, but failed."
)]
FailedToCommit,
} }
#[derive(thiserror::Error, Debug)] //---------------------------------------------------------------------------------------------------- RuntimeError
pub enum DB_SERIAL { /// Database errors that occur _after_ successful initialization.
#[error("An object failed to be serialized into bytes. It is likely an issue from monero-rs library. Please report this error on cuprate's github : https://github.com/Cuprate/cuprate/issues")] #[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
ConsensusEncode, #[cfg_attr(
feature = "borsh",
derive(borsh::BorshSerialize, borsh::BorshDeserialize)
)]
#[derive(thiserror::Error, Debug, Clone, PartialEq, PartialOrd, Eq, Ord, Hash)]
pub enum RuntimeError {
// TODO: replace string with actual error type.
///
/// An error occurred when attempting to
/// serialize the key data into bytes.
#[error("serialize error: {0}")]
Serialize(String),
#[error("Bytes failed to be deserialized into the requested object. It is likely an issue from monero-rs library. Please report this error on cuprate's github : https://github.com/Cuprate/cuprate/issues")] // TODO: replace string with actual error type.
ConsensusDecode(Vec<u8>), ///
/// An error occurred when attempting to
/// deserialize the response value from
/// the database.
#[error("deserialize error: {0}")]
Deserialize(String),
#[error("monero-rs encoding|decoding logic failed : {0}")] /// TODO
MoneroEncode(#[from] monero::consensus::encode::Error), ///
/// An unknown error occurred.
#[error("Bincode failed to decode a type from the database : {0}")] #[error("unknown error: {0}")]
BincodeDecode(#[from] bincode::error::DecodeError), Unknown(Cow<'static, str>),
#[error("Bincode failed to encode a type for the database : {0}")]
BincodeEncode(#[from] bincode::error::EncodeError),
} }

17
database/src/free.rs Normal file
View file

@ -0,0 +1,17 @@
//! General free functions (related to the database).
//---------------------------------------------------------------------------------------------------- Import
//---------------------------------------------------------------------------------------------------- Constants
//---------------------------------------------------------------------------------------------------- TYPE
//---------------------------------------------------------------------------------------------------- IMPL
//---------------------------------------------------------------------------------------------------- Trait Impl
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

127
database/src/key.rs Normal file
View file

@ -0,0 +1,127 @@
//! Database key abstraction; `trait Key`.
//---------------------------------------------------------------------------------------------------- Import
#[allow(unused_imports)] // docs
use crate::table::Table;
//---------------------------------------------------------------------------------------------------- Table
/// Database [`Table`] key metadata.
///
/// Purely compile time information for database table keys, supporting duplicate keys.
pub trait Key {
/// Does this [`Key`] require multiple keys to reach a value?
///
/// If [`Key::DUPLICATE`] is `true`, [`Key::Secondary`] will contain
/// the "subkey", or secondary key needed to access the actual value.
///
/// If [`Key::DUPLICATE`] is `false`, [`Key::Secondary`]
/// will just be the same type as [`Key::Primary`].
const DUPLICATE: bool;
/// The primary key type.
type Primary;
/// The secondary key type.
///
/// Only needs to be different than [`Key::Primary`]
/// if [`Key::DUPLICATE`] is `true`.
type Secondary;
/// Acquire [`Key::Primary`].
fn primary(self) -> Self::Primary;
/// Acquire [`Self::Primary`] & [`Self::Secondary`].
///
/// This only needs to be implemented on types that are [`Self::DUPLICATE`].
///
/// It is `unreachable!()` on non-duplicate key tables.
fn primary_secondary(self) -> (Self::Primary, Self::Secondary);
}
/// Duplicate key container.
///
/// This is a generic container to use alongside [`Key`] to support
/// tables that require more than 1 key to access the value.
#[derive(Copy, Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
#[cfg_attr(
feature = "borsh",
derive(borsh::BorshSerialize, borsh::BorshDeserialize)
)]
pub struct DupKey<P, S> {
/// Primary key type.
pub primary: P,
/// Secondary key type.
pub secondary: S,
}
//---------------------------------------------------------------------------------------------------- Impl
/// Implement `Key` on most primitive types.
///
/// `Key::DUPLICATE` is always `false`.
macro_rules! impl_key {
(
$(
$t:ident // Key type.
),* $(,)?
) => {
$(
impl Key for $t {
const DUPLICATE: bool = false;
type Primary = $t;
type Secondary = $t;
#[inline(always)]
fn primary(self) -> Self::Primary {
self
}
#[cold] #[inline(never)]
fn primary_secondary(self) -> (Self::Primary, Self::Secondary) {
unreachable!();
}
}
)*
};
}
// Implement `Key` for primitives.
impl_key! {
u8,
u16,
u32,
u64,
i8,
i16,
i32,
i64,
}
// Implement `Key` for any [`DupKey`] using [`Copy`] types.
impl<P, S> Key for DupKey<P, S>
where
P: Key + Copy,
S: Key + Copy,
{
const DUPLICATE: bool = true;
type Primary = Self;
type Secondary = S;
#[inline]
fn primary(self) -> Self::Primary {
self
}
#[inline]
fn primary_secondary(self) -> (Self::Primary, Self::Secondary) {
(self, self.secondary)
}
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

441
database/src/lib.rs
View file

@ -1,221 +1,274 @@
// Copyright (C) 2023 Cuprate Contributors //! Database abstraction and utilities.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <https://www.gnu.org/licenses/>.
//! The cuprate-db crate implement (as its name suggests) the relations between the blockchain/txpool objects and their databases.
//! `lib.rs` contains all the generics, trait and specification for interfaces between blockchain and a backend-agnostic database
//! Every other files in this folder are implementation of these traits/methods to real storage engine.
//! //!
//! At the moment, the only storage engine available is MDBX. //! This crate does 3 things:
//! The next storage engine planned is HSE (Heteregeonous Storage Engine) from Micron. //! 1. Abstracts various databases with the [`Env`], [`Database`], [`Table`], [`Key`], [`RoTx`], and [`RwTx`] traits
//! 2. Implements various `Monero` related [functions](ops) & [`tables`]
//! 3. Exposes a [`tower::Service`] backed by a thread-pool
//! //!
//! For more informations, please consult this docs: //! # Terminology
//! To be more clear on some terms used in this crate:
//!
//! | Term | Meaning |
//! |------------|--------------------------------------|
//! | `Env` | The 1 database environment, the "whole" thing
//! | `Database` | A `key/value` store
//! | `Table` | Solely the metadata of a `Database` (the `key` and `value` types, and the name)
//! | `RoTx` | Read only transaction
//! | `RwTx` | Read/write transaction
//!
//! The dataflow is `Env` -> `Tx` -> `Database`
//!
//! Which reads as:
//! 1. You have a database `Environment`
//! 2. You open up a `Transaction`
//! 2. You get a particular `Database` from that `Environment`
//! 3. You can now read/write data from/to that `Database`
//!
//! # `ConcreteEnv`
//! This crate exposes [`ConcreteEnv`], which is a non-generic/non-dynamic,
//! concrete object representing a database [`Env`]ironment.
//!
//! The actual backend for this type is determined via feature flags.
//!
//! This object existing means `E: Env` doesn't need to be spread all through the codebase,
//! however, it also means some small invariants should be kept in mind.
//!
//! As `ConcreteEnv` is just a re-exposed type which has varying inner types,
//! it means some properties will change depending on the backend used.
//!
//! For example:
//! - [`std::mem::size_of::<ConcreteEnv>`]
//! - [`std::mem::align_of::<ConcreteEnv>`]
//! - [`Drop::<ConcreteEnv>::drop`]
//!
//! Things like these functions are affected by the backend and inner data,
//! and should not be relied upon. This extends to any `struct/enum` that contains `ConcreteEnv`.
//!
//! The only thing about `ConcreteEnv` that should
//! be relied upon is that it implements [`Env`].
//!
//! TODO: we could also expose `ConcreteDatabase` if we're
//! going to be storing any databases in structs, to lessen
//! the generic `<D: Database>` pain.
//!
//! # Feature flags
//! The `service` module requires the `service` feature to be enabled.
//! See the module for more documentation.
//!
//! Different database backends are enabled by the feature flags:
//! - `heed`
//! - `sanakirja`
//!
//! The default is `heed`.
#![deny(unused_attributes)] //---------------------------------------------------------------------------------------------------- Lints
#![forbid(unsafe_code)] // Forbid lints.
#![allow(non_camel_case_types)] // Our code, and code generated (e.g macros) cannot overrule these.
#![deny(clippy::expect_used, clippy::panic)] #![forbid(
#![allow(dead_code, unused_macros)] // temporary // Never.
unsafe_code,
unused_unsafe,
redundant_semicolons,
unused_allocation,
coherence_leak_check,
single_use_lifetimes,
while_true,
clippy::missing_docs_in_private_items,
use monero::{util::ringct::RctSig, Block, BlockHeader, Hash}; // Maybe can be put into `#[deny]`.
use std::ops::Range; unconditional_recursion,
use thiserror::Error; for_loops_over_fallibles,
unused_braces,
unused_doc_comments,
unused_labels,
keyword_idents,
non_ascii_idents,
variant_size_differences,
unused_mut, // Annoying when debugging, maybe put in allow.
#[cfg(feature = "mdbx")] // Probably can be put into `#[deny]`.
pub mod mdbx; future_incompatible,
//#[cfg(feature = "hse")] let_underscore,
//pub mod hse; break_with_label_and_loop,
duplicate_macro_attributes,
exported_private_dependencies,
large_assignments,
overlapping_range_endpoints,
semicolon_in_expressions_from_macros,
noop_method_call,
unreachable_pub,
)]
// Deny lints.
// Some of these are `#[allow]`'ed on a per-case basis.
#![deny(
clippy::all,
clippy::correctness,
clippy::suspicious,
clippy::style,
clippy::complexity,
clippy::perf,
clippy::pedantic,
clippy::nursery,
clippy::cargo,
missing_docs,
deprecated,
unused_comparisons,
nonstandard_style
)]
#![allow(unreachable_code, unused_variables, dead_code)] // TODO: remove
#![allow(
// FIXME: this lint affects crates outside of
// `database/` for some reason, allow for now.
clippy::cargo_common_metadata,
pub mod encoding; // FIXME: adding `#[must_use]` onto everything
pub mod error; // might just be more annoying than useful...
pub mod interface; // although it is sometimes nice.
pub mod table; clippy::must_use_candidate,
pub mod types;
const DEFAULT_BLOCKCHAIN_DATABASE_DIRECTORY: &str = "blockchain"; // TODO: should be removed after all `todo!()`'s are gone.
const DEFAULT_TXPOOL_DATABASE_DIRECTORY: &str = "txpool_mem"; clippy::diverging_sub_expression,
const BINCODE_CONFIG: bincode::config::Configuration<
bincode::config::LittleEndian,
bincode::config::Fixint,
> = bincode::config::standard().with_fixed_int_encoding();
// ------------------------------------------| Database |------------------------------------------ // FIXME:
// If #[deny(clippy::restriction)] is used, it
// enables a whole bunch of very subjective lints.
// The below disables most of the ones that are
// a bit too unwieldy.
//
// Figure out if if `clippy::restriction` should be
// used (it enables a bunch of good lints but has
// many false positives).
pub mod database { // clippy::single_char_lifetime_names,
//! This module contains the Database abstraction trait. Any key/value storage engine implemented need // clippy::implicit_return,
//! to fullfil these associated types and functions, in order to be usable. This module also contains the // clippy::std_instead_of_alloc,
//! Interface struct which is used by the DB Reactor to interact with the database. // clippy::std_instead_of_core,
// clippy::unwrap_used,
// clippy::min_ident_chars,
// clippy::absolute_paths,
// clippy::missing_inline_in_public_items,
// clippy::shadow_reuse,
// clippy::shadow_unrelated,
// clippy::missing_trait_methods,
// clippy::pub_use,
// clippy::pub_with_shorthand,
// clippy::blanket_clippy_restriction_lints,
// clippy::exhaustive_structs,
// clippy::exhaustive_enums,
// clippy::unsafe_derive_deserialize,
// clippy::multiple_inherent_impl,
// clippy::unreadable_literal,
// clippy::indexing_slicing,
// clippy::float_arithmetic,
// clippy::cast_possible_truncation,
// clippy::as_conversions,
// clippy::cast_precision_loss,
// clippy::cast_sign_loss,
// clippy::missing_asserts_for_indexing,
// clippy::default_numeric_fallback,
// clippy::module_inception,
// clippy::mod_module_files,
// clippy::multiple_unsafe_ops_per_block,
// clippy::too_many_lines,
// clippy::missing_assert_message,
// clippy::len_zero,
// clippy::separated_literal_suffix,
// clippy::single_call_fn,
// clippy::unreachable,
// clippy::many_single_char_names,
// clippy::redundant_pub_crate,
// clippy::decimal_literal_representation,
// clippy::option_if_let_else,
// clippy::lossy_float_literal,
// clippy::modulo_arithmetic,
// clippy::print_stdout,
// clippy::module_name_repetitions,
// clippy::no_effect,
// clippy::semicolon_outside_block,
// clippy::panic,
// clippy::question_mark_used,
// clippy::expect_used,
// clippy::integer_division,
// clippy::type_complexity,
// clippy::pattern_type_mismatch,
// clippy::arithmetic_side_effects,
// clippy::default_trait_access,
// clippy::similar_names,
// clippy::needless_pass_by_value,
// clippy::inline_always,
// clippy::if_then_some_else_none,
// clippy::arithmetic_side_effects,
// clippy::float_cmp,
// clippy::items_after_statements,
// clippy::use_debug,
// clippy::mem_forget,
// clippy::else_if_without_else,
// clippy::str_to_string,
// clippy::branches_sharing_code,
// clippy::impl_trait_in_params,
// clippy::struct_excessive_bools,
// clippy::exit,
// // This lint is actually good but
// // it sometimes hits false positive.
// clippy::self_named_module_files
use crate::{ clippy::module_name_repetitions,
error::DB_FAILURES, clippy::module_inception,
transaction::{Transaction, WriteTransaction}, clippy::redundant_pub_crate,
}; )]
use std::{ops::Deref, path::PathBuf, sync::Arc}; // Allow some lints when running in debug mode.
#![cfg_attr(debug_assertions, allow(clippy::todo, clippy::multiple_crate_versions))]
/// `Database` Trait implement all the methods necessary to generate transactions as well as execute specific functions. It also implement generic associated types to identify the // Only allow building 64-bit targets.
/// different transaction modes (read & write) and it's native errors. //
pub trait Database<'a> { // This allows us to assume 64-bit
type TX: Transaction<'a>; // invariants in code, e.g. `usize as u64`.
type TXMut: WriteTransaction<'a>; #[cfg(not(target_pointer_width = "64"))]
type Error: Into<DB_FAILURES>; compile_error!("Cuprate is only compatible with 64-bit CPUs");
// Create a transaction from the database //---------------------------------------------------------------------------------------------------- Public API
fn tx(&'a self) -> Result<Self::TX, Self::Error>; // Import private modules, export public types.
//
// Documentation for each module is
// located in the respective file.
// Create a mutable transaction from the database mod backend;
fn tx_mut(&'a self) -> Result<Self::TXMut, Self::Error>; pub use backend::ConcreteEnv;
// Open a database from the specified path mod constants;
fn open(path: PathBuf) -> Result<Self, Self::Error> pub use constants::{CUPRATE_DATABASE_DIR, CUPRATE_DATABASE_FILE, DATABASE_BACKEND};
where
Self: std::marker::Sized;
// Check if the database is built. mod database;
fn check_all_tables_exist(&'a self) -> Result<(), Self::Error>; pub use database::Database;
// Build the database mod env;
fn build(&'a self) -> Result<(), Self::Error>; pub use env::Env;
}
/// `Interface` is a struct containing a shared pointer to the database and transaction's to be used for the implemented method of Interface. mod error;
pub struct Interface<'a, D: Database<'a>> { pub use error::{InitError, RuntimeError};
pub db: Arc<D>,
pub tx: Option<<D as Database<'a>>::TXMut>,
}
// Convenient implementations for database mod free;
impl<'service, D: Database<'service>> Interface<'service, D> {
fn from(db: Arc<D>) -> Result<Self, DB_FAILURES> {
Ok(Self { db, tx: None })
}
fn open(&'service mut self) -> Result<(), DB_FAILURES> { mod key;
let tx = self.db.tx_mut().map_err(Into::into)?; pub use key::{DupKey, Key};
self.tx = Some(tx);
Ok(())
}
}
impl<'service, D: Database<'service>> Deref for Interface<'service, D> { mod macros;
type Target = <D as Database<'service>>::TXMut;
fn deref(&self) -> &Self::Target { pub mod ops;
return self.tx.as_ref().unwrap();
}
}
}
// ------------------------------------------| DatabaseTx |------------------------------------------ mod pod;
pub use pod::Pod;
pub mod transaction { mod table;
//! This module contains the abstractions of Transactional Key/Value database functions. pub use table::Table;
//! Any key/value database/storage engine can be implemented easily for Cuprate as long as
//! these functions or equivalent logic exist for it.
use crate::{ pub mod tables;
error::DB_FAILURES,
table::{DupTable, Table},
};
// Abstraction of a read-only cursor, for simple tables mod transaction;
#[allow(clippy::type_complexity)] pub use transaction::{RoTx, RwTx};
pub trait Cursor<'t, T: Table> {
fn first(&mut self) -> Result<Option<(T::Key, T::Value)>, DB_FAILURES>;
fn get_cursor(&mut self) -> Result<Option<(T::Key, T::Value)>, DB_FAILURES>; //---------------------------------------------------------------------------------------------------- Feature-gated
#[cfg(feature = "service")]
pub mod service;
fn last(&mut self) -> Result<Option<(T::Key, T::Value)>, DB_FAILURES>; //---------------------------------------------------------------------------------------------------- Private
fn next(&mut self) -> Result<Option<(T::Key, T::Value)>, DB_FAILURES>;
fn prev(&mut self) -> Result<Option<(T::Key, T::Value)>, DB_FAILURES>;
fn set(&mut self, key: &T::Key) -> Result<Option<T::Value>, DB_FAILURES>;
}
// Abstraction of a read-only cursor with support for duplicated tables. DupCursor inherit Cursor methods as
// a duplicated table can be treated as a simple table.
#[allow(clippy::type_complexity)]
pub trait DupCursor<'t, T: DupTable>: Cursor<'t, T> {
fn first_dup(&mut self) -> Result<Option<(T::SubKey, T::Value)>, DB_FAILURES>;
fn get_dup(
&mut self,
key: &T::Key,
subkey: &T::SubKey,
) -> Result<Option<T::Value>, DB_FAILURES>;
fn last_dup(&mut self) -> Result<Option<(T::SubKey, T::Value)>, DB_FAILURES>;
fn next_dup(&mut self) -> Result<Option<(T::Key, (T::SubKey, T::Value))>, DB_FAILURES>;
fn prev_dup(&mut self) -> Result<Option<(T::Key, (T::SubKey, T::Value))>, DB_FAILURES>;
}
// Abstraction of a read-write cursor, for simple tables. WriteCursor inherit Cursor methods.
pub trait WriteCursor<'t, T: Table>: Cursor<'t, T> {
fn put_cursor(&mut self, key: &T::Key, value: &T::Value) -> Result<(), DB_FAILURES>;
fn del(&mut self) -> Result<(), DB_FAILURES>;
}
// Abstraction of a read-write cursor with support for duplicated tables. DupWriteCursor inherit DupCursor and WriteCursor methods.
pub trait DupWriteCursor<'t, T: DupTable>: WriteCursor<'t, T> {
fn put_cursor_dup(
&mut self,
key: &T::Key,
subkey: &T::SubKey,
value: &T::Value,
) -> Result<(), DB_FAILURES>;
/// Delete all data under associated to its key
fn del_nodup(&mut self) -> Result<(), DB_FAILURES>;
}
// Abstraction of a read-only transaction.
pub trait Transaction<'a>: Send + Sync {
type Cursor<T: Table>: Cursor<'a, T>;
type DupCursor<T: DupTable>: DupCursor<'a, T> + Cursor<'a, T>;
fn get<T: Table>(&self, key: &T::Key) -> Result<Option<T::Value>, DB_FAILURES>;
fn commit(self) -> Result<(), DB_FAILURES>;
fn cursor<T: Table>(&self) -> Result<Self::Cursor<T>, DB_FAILURES>;
fn cursor_dup<T: DupTable>(&self) -> Result<Self::DupCursor<T>, DB_FAILURES>;
fn num_entries<T: Table>(&self) -> Result<usize, DB_FAILURES>;
}
// Abstraction of a read-write transaction. WriteTransaction inherits Transaction methods.
pub trait WriteTransaction<'a>: Transaction<'a> {
type WriteCursor<T: Table>: WriteCursor<'a, T>;
type DupWriteCursor<T: DupTable>: DupWriteCursor<'a, T> + DupCursor<'a, T>;
fn put<T: Table>(&self, key: &T::Key, value: &T::Value) -> Result<(), DB_FAILURES>;
fn delete<T: Table>(
&self,
key: &T::Key,
value: &Option<T::Value>,
) -> Result<(), DB_FAILURES>;
fn clear<T: Table>(&self) -> Result<(), DB_FAILURES>;
fn write_cursor<T: Table>(&self) -> Result<Self::WriteCursor<T>, DB_FAILURES>;
fn write_cursor_dup<T: DupTable>(&self) -> Result<Self::DupWriteCursor<T>, DB_FAILURES>;
}
}

17
database/src/macros.rs Normal file
View file

@ -0,0 +1,17 @@
//! General macros used throughout `cuprate-database`.
//---------------------------------------------------------------------------------------------------- Import
//---------------------------------------------------------------------------------------------------- Constants
//---------------------------------------------------------------------------------------------------- TYPE
//---------------------------------------------------------------------------------------------------- IMPL
//---------------------------------------------------------------------------------------------------- Trait Impl
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

37
database/src/ops/alt_block.rs Normal file
View file

@ -0,0 +1,37 @@
//! Alternative blocks.
//---------------------------------------------------------------------------------------------------- Import
#[allow(unused_imports)] // FIXME: these traits will be eventually in the function impls.
use crate::{
database::Database,
env::Env,
table::Table,
transaction::{RoTx, RwTx},
ConcreteEnv,
};
//---------------------------------------------------------------------------------------------------- Free Functions
/// TODO
pub fn add_alt_block() {
todo!()
}
/// TODO
pub fn get_alt_block() {
todo!()
}
/// TODO
pub fn remove_alt_block() {
todo!()
}
/// TODO
pub fn get_alt_block_count() {
todo!()
}
/// TODO
pub fn drop_alt_blocks() {
todo!()
}

97
database/src/ops/block.rs Normal file
View file

@ -0,0 +1,97 @@
//! Blocks.
//---------------------------------------------------------------------------------------------------- Import
#[allow(unused_imports)] // FIXME: these traits will be eventually in the function impls.
use crate::{
database::Database,
env::Env,
table::Table,
transaction::{RoTx, RwTx},
ConcreteEnv,
};
//---------------------------------------------------------------------------------------------------- Free Functions
/// TODO
pub fn add_block() {
todo!()
}
/// TODO
pub fn add_block_data() {
todo!()
}
/// TODO
pub fn pop_block() {
todo!()
}
/// TODO
pub fn block_exists() {
todo!()
}
/// TODO
pub fn get_block_hash() {
todo!()
}
/// TODO
pub fn get_block_height() {
todo!()
}
/// TODO
pub fn get_block_weight() {
todo!()
}
/// TODO
pub fn get_block_already_generated_coins() {
todo!()
}
/// TODO
pub fn get_block_long_term_weight() {
todo!()
}
/// TODO
pub fn get_block_timestamp() {
todo!()
}
/// TODO
pub fn get_block_cumulative_rct_outputs() {
todo!()
}
/// TODO
pub fn get_block() {
todo!()
}
/// TODO
pub fn get_block_from_height() {
todo!()
}
/// TODO
pub fn get_block_header() {
todo!()
}
/// TODO
pub fn get_block_header_from_height() {
todo!()
}
/// TODO
pub fn get_top_block() {
todo!()
}
/// TODO
pub fn get_top_block_hash() {
todo!()
}

17
database/src/ops/blockchain.rs Normal file
View file

@ -0,0 +1,17 @@
//! Blockchain.
//---------------------------------------------------------------------------------------------------- Import
#[allow(unused_imports)] // FIXME: these traits will be eventually in the function impls.
use crate::{
database::Database,
env::Env,
table::Table,
transaction::{RoTx, RwTx},
ConcreteEnv,
};
//---------------------------------------------------------------------------------------------------- Free Functions
/// TODO
pub fn height() {
todo!()
}

21
database/src/ops/mod.rs Normal file
View file

@ -0,0 +1,21 @@
//! Abstracted Monero database operations.
//!
//! This module contains many free functions that use the
//! traits in this crate to generically call Monero-related
//! database operations.
//!
//! # TODO
//! TODO: These functions should pretty much map 1-1 to the `Request` enum.
//!
//! TODO: These are function names from `old_database/` for now.
//! The actual underlying functions (e.g `get()`) aren't implemented.
//!
//! TODO: All of these functions need to take in generic
//! database trait parameters (and their actual inputs).
pub mod alt_block;
pub mod block;
pub mod output;
pub mod property;
pub mod spent_key;
pub mod tx;

42
database/src/ops/output.rs Normal file
View file

@ -0,0 +1,42 @@
//! Outputs.
//---------------------------------------------------------------------------------------------------- Import
#[allow(unused_imports)] // FIXME: these traits will be eventually in the function impls.
use crate::{
database::Database,
env::Env,
table::Table,
transaction::{RoTx, RwTx},
ConcreteEnv,
};
//---------------------------------------------------------------------------------------------------- Free Functions
/// TODO
pub fn add_output() {
todo!()
}
/// TODO
pub fn remove_output() {
todo!()
}
/// TODO
pub fn get_output() {
todo!()
}
/// TODO
pub fn get_output_list() {
todo!()
}
/// TODO
pub fn get_rct_num_outputs() {
todo!()
}
/// TODO
pub fn get_pre_rct_num_outputs() {
todo!()
}

17
database/src/ops/property.rs Normal file
View file

@ -0,0 +1,17 @@
//! Properties.
//---------------------------------------------------------------------------------------------------- Import
#[allow(unused_imports)] // FIXME: these traits will be eventually in the function impls.
use crate::{
database::Database,
env::Env,
table::Table,
transaction::{RoTx, RwTx},
ConcreteEnv,
};
//---------------------------------------------------------------------------------------------------- Free Functions
/// TODO
pub fn get_blockchain_pruning_seed() {
todo!()
}

27
database/src/ops/spent_key.rs Normal file
View file

@ -0,0 +1,27 @@
//! Spent keys.
//---------------------------------------------------------------------------------------------------- Import
#[allow(unused_imports)] // FIXME: these traits will be eventually in the function impls.
use crate::{
database::Database,
env::Env,
table::Table,
transaction::{RoTx, RwTx},
ConcreteEnv,
};
//---------------------------------------------------------------------------------------------------- Free Functions
/// TODO
pub fn add_spent_key() {
todo!()
}
/// TODO
pub fn remove_spent_key() {
todo!()
}
/// TODO
pub fn is_spent_key_recorded() {
todo!()
}

72
database/src/ops/tx.rs Normal file
View file

@ -0,0 +1,72 @@
//! Transactions.
//---------------------------------------------------------------------------------------------------- Import
#[allow(unused_imports)] // FIXME: these traits will be eventually in the function impls.
use crate::{
database::Database,
env::Env,
table::Table,
transaction::{RoTx, RwTx},
ConcreteEnv,
};
//---------------------------------------------------------------------------------------------------- Free Functions
/// TODO
pub fn add_transaction() {
todo!()
}
/// TODO
pub fn add_transaction_data() {
todo!()
}
/// TODO
pub fn remove_transaction() {
todo!()
}
/// TODO
pub fn remove_transaction_data() {
todo!()
}
/// TODO
pub fn remove_tx_outputs() {
todo!()
}
/// TODO
pub fn get_num_tx() {
todo!()
}
/// TODO
pub fn tx_exists() {
todo!()
}
/// TODO
pub fn get_tx_unlock_time() {
todo!()
}
/// TODO
pub fn get_tx() {
todo!()
}
/// TODO
pub fn get_tx_list() {
todo!()
}
/// TODO
pub fn get_pruned_tx() {
todo!()
}
/// TODO
pub fn get_tx_block_height() {
todo!()
}

481
database/src/pod.rs Normal file
View file

@ -0,0 +1,481 @@
//! (De)serialization for table keys & values.
//!
//! All keys and values in database tables must be able
//! to be (de)serialized into/from raw bytes ([u8]).
//---------------------------------------------------------------------------------------------------- Import
use std::{
borrow::Cow,
io::{Read, Write},
sync::Arc,
};
//---------------------------------------------------------------------------------------------------- Pod
/// Plain Old Data.
///
/// Trait representing very simple types that can be
/// (de)serialized into/from bytes.
///
/// Reference: <https://docs.rs/bytemuck/latest/bytemuck/trait.Pod.html>
///
/// ## Endianness
/// As `bytemuck` provides everything needed here + more, it could be used,
/// _but_, its `Pod` is endian dependant. We need to ensure bytes are the
/// exact same such that the database stores the same bytes on different machines;
/// so we use little endian functions instead, e.g. [`u8::to_le_bytes`].
///
/// This also means an `INVARIANT` of this trait is that
/// implementors must use little endian bytes when applicable.
///
/// Slice types (just raw `[u8]` bytes) are (de)serialized as-is.
///
/// ## Sealed
/// This trait is [`Sealed`](https://rust-lang.github.io/api-guidelines/future-proofing.html#sealed-traits-protect-against-downstream-implementations-c-sealed).
///
/// It cannot be implemented outside this crate,
/// and is only implemented on specific types.
///
/// # TODO
/// This could be implemented on `bytes::Bytes` if needed.
///
/// Maybe under a `bytes` feature flag.
pub trait Pod: Sized + private::Sealed {
/// Return `self` in byte form.
///
/// The returned bytes can be any form of array,
/// - `[u8]`
/// - `[u8; N]`
/// - [`Vec<u8>`]
///
/// ..etc.
///
/// This is used on slice types (`Vec<u8>`, `[u8; N]`, etc) for cheap conversions.
///
/// Integer types ([`u8`], [`f32`], [`i8`], etc) return a fixed-sized array.
fn as_bytes(&self) -> impl AsRef<[u8]>;
/// TODO
fn into_bytes(self) -> Cow<'static, [u8]>;
/// Create [`Self`] from bytes.
///
/// # Panics
/// This function should be infallible.
///
/// If `bytes` is invalid, this should panic.
fn from_bytes(bytes: &[u8]) -> Self;
/// Convert [`Self`] into bytes, and write those bytes into a [`Write`]r.
///
/// The `usize` returned should be how many bytes were written.
///
/// TODO: do we ever actually need how many bytes were written?
///
/// # Panics
/// This function should be infallible.
///
/// If the `writer` errors, this should panic.
fn to_writer<W: Write>(self, writer: &mut W) -> usize;
/// Create [`Self`] by reading bytes from a [`Read`]er.
///
/// # Panics
/// This function should be infallible.
///
/// If the `reader` errors, this should panic.
fn from_reader<R: Read>(reader: &mut R) -> Self;
}
/// Private module, should not be accessible outside this crate.
///
/// Used to block outsiders implementing [`Pod`].
/// All [`Pod`] types must also implement [`Sealed`].
mod private {
/// Private sealed trait.
///
/// Cannot be implemented outside this crate.
pub trait Sealed {}
/// Implement `Sealed`.
macro_rules! impl_sealed {
($(
$t:ty // The type to implement for.
),* $(,)?) => {
$(
impl Sealed for $t {}
)*
};
}
// Special case cause of generic.
impl<const N: usize> Sealed for [u8; N] {}
impl_sealed! {
Vec<u8>,
Box<[u8]>,
std::sync::Arc<[u8]>,
f32,
f64,
u8,
u16,
u32,
u64,
u128,
usize,
i8,
i16,
i32,
i64,
i128,
isize,
}
}
//---------------------------------------------------------------------------------------------------- Pod Impl (bytes)
// Implement for owned `Vec` bytes.
impl Pod for Vec<u8> {
#[inline]
fn as_bytes(&self) -> impl AsRef<[u8]> {
self
}
#[inline]
fn into_bytes(self) -> Cow<'static, [u8]> {
Cow::Owned(self)
}
#[inline]
fn from_bytes(bytes: &[u8]) -> Self {
bytes.to_vec()
}
#[inline]
fn from_reader<R: Read>(reader: &mut R) -> Self {
// FIXME: Could be `Vec::with_capacity(likely_size)`?
let mut vec = vec![];
reader
.read_to_end(&mut vec)
.expect("Pod::<Vec<u8>>::read_to_end() failed");
vec
}
#[inline]
fn to_writer<W: Write>(self, writer: &mut W) -> usize {
writer
.write_all(&self)
.expect("Pod::<Vec<u8>>::write_all() failed");
self.len()
}
}
// Implement for any sized stack array.
impl<const N: usize> Pod for [u8; N] {
#[inline]
fn as_bytes(&self) -> impl AsRef<[u8]> {
self
}
#[inline]
fn into_bytes(self) -> Cow<'static, [u8]> {
Cow::Owned(self.to_vec())
}
#[inline]
fn from_bytes(bytes: &[u8]) -> Self {
// Return if the bytes are too short/long.
let bytes_len = bytes.len();
assert_eq!(
bytes_len, N,
"Pod::<[u8; {N}]>::from_bytes() failed, expected_len: {N}, found_len: {bytes_len}",
);
let mut array = [0_u8; N];
// INVARIANT: we checked the length is valid above.
array.copy_from_slice(bytes);
array
}
#[inline]
fn from_reader<R: Read>(reader: &mut R) -> Self {
let mut bytes = [0_u8; N];
reader
.read_exact(&mut bytes)
.expect("Pod::<[u8; {N}]>::read_exact() failed");
bytes
}
#[inline]
fn to_writer<W: Write>(self, writer: &mut W) -> usize {
writer
.write_all(&self)
.expect("Pod::<[u8; {N}]>::write_all() failed");
self.len()
}
}
// Implement for any sized boxed array.
//
// In-case `[u8; N]` is too big and would
// overflow the stack, this can be used.
//
// The benefit over `Vec<u8>` is that the capacity & length are static.
//
// The weird constructions of `Box` below are on purpose to avoid this:
// <https://github.com/rust-lang/rust/issues/53827>
impl Pod for Box<[u8]> {
#[inline]
fn as_bytes(&self) -> impl AsRef<[u8]> {
self
}
#[inline]
fn into_bytes(self) -> Cow<'static, [u8]> {
Cow::Owned(self.into())
}
#[inline]
fn from_bytes(bytes: &[u8]) -> Self {
Self::from(bytes)
}
#[inline]
fn from_reader<R: Read>(reader: &mut R) -> Self {
let mut bytes = vec![];
reader
.read_to_end(bytes.as_mut())
.expect("Pod::<Box<[u8]>>::read_to_end() failed");
bytes.into_boxed_slice()
}
#[inline]
fn to_writer<W: Write>(self, writer: &mut W) -> usize {
writer
.write_all(&self)
.expect("Pod::<Box<[u8]>>::write_all() failed");
self.len()
}
}
// Implement for any Arc bytes.
impl Pod for Arc<[u8]> {
#[inline]
fn as_bytes(&self) -> impl AsRef<[u8]> {
self
}
#[inline]
fn into_bytes(self) -> Cow<'static, [u8]> {
Cow::Owned(self.to_vec())
}
#[inline]
fn from_bytes(bytes: &[u8]) -> Self {
Self::from(bytes)
}
#[inline]
fn from_reader<R: Read>(reader: &mut R) -> Self {
let mut bytes = vec![];
reader
.read_to_end(bytes.as_mut())
.expect("Pod::<Arc<[u8]>>::read_to_end() failed");
Self::from(bytes)
}
#[inline]
fn to_writer<W: Write>(self, writer: &mut W) -> usize {
writer
.write_all(&self)
.expect("Pod::<Arc<[u8]>>::write_all() failed");
self.len()
}
}
//---------------------------------------------------------------------------------------------------- Pod Impl (numbers)
/// Implement `Pod` on primitive numbers.
///
/// This will always use little endian representations.
macro_rules! impl_pod_le_bytes {
($(
$number:ident => // The number type.
$length:literal // The length of `u8`'s this type takes up.
),* $(,)?) => {
$(
impl Pod for $number {
#[inline]
fn as_bytes(&self) -> impl AsRef<[u8]> {
$number::to_le_bytes(*self)
}
#[inline]
fn into_bytes(self) -> Cow<'static, [u8]> {
Cow::Owned(self.as_bytes().as_ref().to_vec())
}
#[inline]
/// This function returns [`Err`] if `bytes`'s length is not
#[doc = concat!(" ", stringify!($length), ".")]
fn from_bytes(bytes: &[u8]) -> Self {
// Return if the bytes are too short/long.
let bytes_len = bytes.len();
assert_eq!(
bytes_len, $length,
"Pod::<[u8; {0}]>::from_bytes() failed, expected_len: {0}, found_len: {bytes_len}",
$length,
);
let mut array = [0_u8; $length];
// INVARIANT: we checked the length is valid above.
array.copy_from_slice(bytes);
$number::from_le_bytes(array)
}
#[inline]
fn to_writer<W: Write>(self, writer: &mut W) -> usize {
writer.write_all(self.as_bytes().as_ref()).expect(concat!(
"Pod::<",
stringify!($number),
">::write_all() failed",
));
$length
}
#[inline]
fn from_reader<R: Read>(reader: &mut R) -> Self {
let mut bytes = [0_u8; $length];
// Read exactly the bytes required.
reader.read_exact(&mut bytes).expect(concat!(
"Pod::<",
stringify!($number),
">::react_exact() failed",
));
// INVARIANT: we checked the length is valid above.
$number::from_le_bytes(bytes)
}
}
)*
};
}
impl_pod_le_bytes! {
f32 => 4,
f64 => 8,
u8 => 1,
u16 => 2,
u32 => 4,
u64 => 8,
usize => 8,
u128 => 16,
i8 => 1,
i16 => 2,
i32 => 4,
i64 => 8,
isize => 8,
i128 => 16,
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
use super::*;
/// Serialize, deserialize, and compare that
/// the intermediate/end results are correct.
fn test_serde<const LEN: usize, T: Pod + Copy + PartialEq + std::fmt::Debug>(
// The primitive number function that
// converts the number into little endian bytes,
// e.g `u8::to_le_bytes`.
to_le_bytes: fn(T) -> [u8; LEN],
// A `Vec` of the numbers to test.
t: Vec<T>,
) {
for t in t {
let expected_bytes = to_le_bytes(t);
println!("testing: {t:?}, expected_bytes: {expected_bytes:?}");
let mut bytes = vec![];
// (De)serialize.
let se: usize = t.to_writer::<Vec<u8>>(bytes.as_mut());
let de: T = T::from_reader::<&[u8]>(&mut bytes.as_slice());
println!("written: {se}, deserialize_t: {de:?}, bytes: {bytes:?}\n");
// Assert we wrote correct amount of bytes
// and deserialized correctly.
assert_eq!(se, expected_bytes.len());
assert_eq!(de, t);
}
}
/// Create all the float tests.
macro_rules! test_float {
($(
$float:ident // The float type.
),* $(,)?) => {
$(
#[test]
fn $float() {
test_serde(
$float::to_le_bytes,
vec![
-1.0,
0.0,
1.0,
$float::MIN,
$float::MAX,
$float::INFINITY,
$float::NEG_INFINITY,
],
);
}
)*
};
}
test_float! {
f32,
f64,
}
/// Create all the (un)signed number tests.
/// u8 -> u128, i8 -> i128.
macro_rules! test_unsigned {
($(
$number:ident // The integer type.
),* $(,)?) => {
$(
#[test]
fn $number() {
test_serde($number::to_le_bytes, vec![$number::MIN, 0, 1, $number::MAX]);
}
)*
};
}
test_unsigned! {
u8,
u16,
u32,
u64,
u128,
usize,
i8,
i16,
i32,
i64,
i128,
isize,
}
}

79
database/src/service/free.rs Normal file
View file

@ -0,0 +1,79 @@
//! General free functions used (related to `cuprate_database::service`).
//---------------------------------------------------------------------------------------------------- Import
use std::sync::Arc;
use crate::{
service::read::DatabaseReader,
service::write::DatabaseWriter,
service::{DatabaseReadHandle, DatabaseWriteHandle},
ConcreteEnv,
};
//---------------------------------------------------------------------------------------------------- Init
#[cold]
#[inline(never)] // Only called once.
/// Initialize a database & thread-pool, and return a read/write handle to it.
///
/// The returned handles are cheaply [`Clone`]able.
///
/// TODO: add blocking behavior docs.
pub fn init() -> (DatabaseReadHandle, DatabaseWriteHandle) {
// TODO:
// This should only ever be called once?
// We could `panic!()` if called twice.
// Initialize the database itself.
// TODO: there's probably shutdown code we have to run.
let db: Arc<ConcreteEnv> = Arc::new(todo!());
// Spawn the `Reader/Writer` thread pools.
let readers = DatabaseReader::init(&db);
let writers = DatabaseWriter::init(&db);
// Return the handles to those pools.
(readers, writers)
}
#[cold]
#[inline(never)] // Only called once.
/// Sync/flush all data, and shutdown the database thread-pool.
///
/// This function **blocks**, waiting until:
/// 1. All database transactions are complete
/// 2. All data has been flushed to disk
/// 3. All database threads have exited
///
/// The database being shutdown is the one started in [`init()`],
/// aka, the single program global database.
///
/// # TODO
/// Maybe the visibility/access of this function should somehow be
/// limited such that only certain parts of `cuprate` can actually
/// call this function.
///
/// Anyone/everyone being able to shutdown the database seems dangerous.
///
/// Counter-argument: we can just CTRL+F to see who calls this i guess.
pub fn shutdown(db: Arc<ConcreteEnv>) {
// Not sure how this function is going
// to work on a `&'static` database, but:
// 1. Send a shutdown message to all database threads, maybe `Request::Shutdown`
// 2. Wait on barrier until all threads are "ready" (all tx's are done)
// 3. Writer thread will flush all data to disk
// 4. All threads exit, 1 of them sends us back an OK
// 5. We don't need to reclaim ownership of `&'static ConcreteEnv` because...
// 5a) a bunch of threads have a `&` to it, so this is hard (impossible?)
// 5b) as along as data is flushed, we can just `std::process::exit`
// and there's no need to (manually) drop the actual database
drop(db);
todo!();
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

61
database/src/service/mod.rs Normal file
View file

@ -0,0 +1,61 @@
//! [`tower::Service`] integeration + thread-pool.
//!
//! ## `service`
//! The `service` module implements the [`tower`] integration,
//! along with the reader/writer thread-pool system.
//!
//! The thread-pool allows outside crates to communicate with it by
//! sending database [`Request`](ReadRequest)s and receiving [`Response`]s `async`hronously -
//! without having to actually worry and handle the database themselves.
//!
//! The system is managed by this crate, and only
//! requires [`init`] and [`shutdown`] by the user.
//!
//! This module must be enabled with the `service` feature.
//!
//! ## Initialization
//! The database & thread-pool system can be initialized with [`init()`].
//!
//! This causes the underlying database/threads to be setup
//! and returns a read/write handle to that database.
//!
//! ## Handles
//! The 2 handles to the database are:
//! - [`DatabaseReadHandle`]
//! - [`DatabaseWriteHandle`]
//!
//! The 1st allows any caller to send [`ReadRequest`]s.
//!
//! The 2nd allows any caller to send [`WriteRequest`]s.
//!
//! The `DatabaseReadHandle` can be shared as it is cheaply [`Clone`]able, however,
//! the `DatabaseWriteHandle` cannot be cloned. There is only 1 place in Cuprate that
//! writes, so it is passed there and used.
//!
//! ## Request and Response
//! To interact with the database (whether reading or writing data),
//! a `Request` can be sent using one of the above handles.
//!
//! Both the handles implement `tower::Service`, so they can be [`tower::Service::call`]ed.
//!
//! An `async`hronous channel will be returned from the call.
//! This channel can be `.await`ed upon to (eventually) receive
//! corresponding `Response` to your `Request`.
mod read;
pub use read::DatabaseReadHandle;
mod write;
pub use write::DatabaseWriteHandle;
mod free;
pub use free::{init, shutdown};
mod request;
pub use request::{ReadRequest, WriteRequest};
mod response;
pub use response::Response;
#[cfg(test)]
mod tests;

191
database/src/service/read.rs Normal file
View file

@ -0,0 +1,191 @@
//! Database read thread-pool definitions and logic.
//---------------------------------------------------------------------------------------------------- Import
use std::{
sync::Arc,
task::{Context, Poll},
};
use cuprate_helper::asynch::InfallibleOneshotReceiver;
use crate::{
error::RuntimeError,
service::{request::ReadRequest, response::Response},
ConcreteEnv,
};
//---------------------------------------------------------------------------------------------------- Types
/// The actual type of the response.
///
/// Either our [Response], or a database error occurred.
type ResponseResult = Result<Response, RuntimeError>;
/// The `Receiver` channel that receives the read response.
///
/// This is owned by the caller (the reader)
/// who `.await`'s for the response.
///
/// The channel itself should never fail,
/// but the actual database operation might.
type ResponseRecv = InfallibleOneshotReceiver<ResponseResult>;
/// The `Sender` channel for the response.
///
/// The database reader thread uses this to send
/// the database result to the caller.
type ResponseSend = tokio::sync::oneshot::Sender<ResponseResult>;
//---------------------------------------------------------------------------------------------------- DatabaseReadHandle
/// Read handle to the database.
///
/// This is cheaply [`Clone`]able handle that
/// allows `async`hronously reading from the database.
///
/// Calling [`tower::Service::call`] with a [`DatabaseReadHandle`] & [`ReadRequest`]
/// will return an `async`hronous channel that can be `.await`ed upon
/// to receive the corresponding [`Response`].
#[derive(Clone, Debug)]
pub struct DatabaseReadHandle {
/// Sender channel to the database read thread-pool.
///
/// We provide the response channel for the thread-pool.
pub(super) sender: crossbeam::channel::Sender<(ReadRequest, ResponseSend)>,
}
impl tower::Service<ReadRequest> for DatabaseReadHandle {
type Response = Response;
type Error = RuntimeError;
type Future = ResponseRecv;
#[inline]
fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
todo!()
}
#[inline]
fn call(&mut self, _req: ReadRequest) -> Self::Future {
todo!()
}
}
//---------------------------------------------------------------------------------------------------- DatabaseReader Impl
/// Database reader thread.
///
/// This struct essentially represents a thread.
///
/// Each reader thread is spawned with access to this struct (self).
pub(super) struct DatabaseReader {
/// Receiver side of the database request channel.
///
/// Any caller can send some requests to this channel.
/// They send them alongside another `Response` channel,
/// which we will eventually send to.
///
/// We (the database reader thread) are not responsible
/// for creating this channel, the caller provides it.
///
/// SOMEDAY: this struct itself could cache a return channel
/// instead of creating a new `oneshot` each request.
receiver: crossbeam::channel::Receiver<(ReadRequest, ResponseSend)>,
/// Access to the database.
db: Arc<ConcreteEnv>,
}
impl DatabaseReader {
/// Initialize the `DatabaseReader` thread-pool.
///
/// This spawns `N` amount of `DatabaseReader`'s
/// attached to `db` and returns a handle to the pool.
///
/// Should be called _once_ per actual database.
#[cold]
#[inline(never)] // Only called once.
pub(super) fn init(db: &Arc<ConcreteEnv>) -> DatabaseReadHandle {
// Initialize `Request/Response` channels.
let (sender, receiver) = crossbeam::channel::unbounded();
// TODO: slightly _less_ readers per thread may be more ideal.
// We could account for the writer count as well such that
// readers + writers == total_thread_count
//
// TODO: take in a config option that allows
// manually adjusting this thread-count.
let readers = cuprate_helper::thread::threads().get();
// Spawn pool of readers.
for _ in 0..readers {
let receiver = receiver.clone();
let db = db.clone();
std::thread::spawn(move || {
let this = Self { receiver, db };
Self::main(this);
});
}
// Return a handle to the pool.
DatabaseReadHandle { sender }
}
/// The `DatabaseReader`'s main function.
///
/// Each thread just loops in this function.
#[cold]
#[inline(never)] // Only called once.
fn main(mut self) {
loop {
// 1. Hang on request channel
// 2. Map request to some database function
// 3. Execute that function, get the result
// 4. Return the result via channel
let (request, response_send) = match self.receiver.recv() {
Ok((r, c)) => (r, c),
// Shutdown on error.
Err(e) => {
Self::shutdown(self);
return;
}
};
// Map [`Request`]'s to specific database functions.
match request {
ReadRequest::Example1 => self.example_handler_1(response_send),
ReadRequest::Example2(_x) => self.example_handler_2(response_send),
ReadRequest::Example3(_x) => self.example_handler_3(response_send),
ReadRequest::Shutdown => {
/* TODO: run shutdown code */
Self::shutdown(self);
// Return, exiting the thread.
return;
}
}
}
}
/// TODO
fn example_handler_1(&mut self, response_send: ResponseSend) {
let db_result = todo!();
response_send.send(db_result).unwrap();
}
/// TODO
fn example_handler_2(&mut self, response_send: ResponseSend) {
let db_result = todo!();
response_send.send(db_result).unwrap();
}
/// TODO
fn example_handler_3(&mut self, response_send: ResponseSend) {
let db_result = todo!();
response_send.send(db_result).unwrap();
}
/// TODO
fn shutdown(self) {
todo!()
}
}

45
database/src/service/request.rs Normal file
View file

@ -0,0 +1,45 @@
//! Read/write `Request`s to the database.
//!
//! TODO: could add `strum` derives.
//---------------------------------------------------------------------------------------------------- Import
//---------------------------------------------------------------------------------------------------- Constants
//---------------------------------------------------------------------------------------------------- ReadRequest
#[derive(Debug)]
/// A read request to the database.
pub enum ReadRequest {
/// TODO
Example1,
/// TODO
Example2(usize),
/// TODO
Example3(String),
/// TODO
Shutdown,
}
//---------------------------------------------------------------------------------------------------- WriteRequest
#[derive(Debug)]
/// A write request to the database.
pub enum WriteRequest {
/// TODO
Example1,
/// TODO
Example2(usize),
/// TODO
Example3(String),
///
Shutdown,
}
//---------------------------------------------------------------------------------------------------- IMPL
//---------------------------------------------------------------------------------------------------- Trait Impl
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

38
database/src/service/response.rs Normal file
View file

@ -0,0 +1,38 @@
//! Read/write `Response`'s from the database.
//!
//! TODO: could add `strum` derives.
//---------------------------------------------------------------------------------------------------- Import
//---------------------------------------------------------------------------------------------------- Constants
//---------------------------------------------------------------------------------------------------- Response
#[derive(Debug)]
/// A response from the database.
///
/// TODO
pub enum Response {
//-------------------------------------------------------- Read responses
/// TODO
Example1,
/// TODO
Example2(usize),
/// TODO
Example3(String),
//-------------------------------------------------------- Write responses
/// The response
///
/// TODO
ExampleWriteResponse, // Probably will be just `Ok`
}
//---------------------------------------------------------------------------------------------------- IMPL
//---------------------------------------------------------------------------------------------------- Trait Impl
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

14
database/src/service/tests.rs Normal file
View file

@ -0,0 +1,14 @@
//! `crate::service` tests.
//!
//! This module contains general tests for the `service` implementation.
//!
//! Testing a thread-pool is slightly more complicated,
//! so this file provides TODO.
// This is only imported on `#[cfg(test)]` in `mod.rs`.
#[test]
const fn test() {
// TODO: remove me.
// Just to see if the module gets imported correctly on test mode.
}

157
database/src/service/write.rs Normal file
View file

@ -0,0 +1,157 @@
//! Database write thread-pool definitions and logic.
//---------------------------------------------------------------------------------------------------- Import
use std::{
sync::Arc,
task::{Context, Poll},
};
use cuprate_helper::asynch::InfallibleOneshotReceiver;
use crate::{
error::RuntimeError,
service::{request::WriteRequest, response::Response},
ConcreteEnv,
};
//---------------------------------------------------------------------------------------------------- Types
/// The actual type of the response.
///
/// Either our [Response], or a database error occurred.
type ResponseResult = Result<Response, RuntimeError>;
/// The `Receiver` channel that receives the write response.
///
/// The channel itself should never fail,
/// but the actual database operation might.
type ResponseRecv = InfallibleOneshotReceiver<ResponseResult>;
/// The `Sender` channel for the response.
type ResponseSend = tokio::sync::oneshot::Sender<ResponseResult>;
//---------------------------------------------------------------------------------------------------- DatabaseWriteHandle
/// Write handle to the database.
///
/// This is handle that allows `async`hronously writing to the database,
/// it is not [`Clone`]able as there is only ever 1 place within Cuprate
/// that writes.
///
/// Calling [`tower::Service::call`] with a [`DatabaseWriteHandle`] & [`WriteRequest`]
/// will return an `async`hronous channel that can be `.await`ed upon
/// to receive the corresponding [`Response`].
#[derive(Debug)]
pub struct DatabaseWriteHandle {
/// Sender channel to the database write thread-pool.
///
/// We provide the response channel for the thread-pool.
pub(super) sender: crossbeam::channel::Sender<(WriteRequest, ResponseSend)>,
}
impl tower::Service<WriteRequest> for DatabaseWriteHandle {
type Response = Response;
type Error = RuntimeError;
type Future = ResponseRecv;
#[inline]
fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
todo!()
}
#[inline]
fn call(&mut self, _req: WriteRequest) -> Self::Future {
todo!()
}
}
//---------------------------------------------------------------------------------------------------- DatabaseWriter
/// The single database writer thread.
pub(super) struct DatabaseWriter {
/// Receiver side of the database request channel.
///
/// Any caller can send some requests to this channel.
/// They send them alongside another `Response` channel,
/// which we will eventually send to.
receiver: crossbeam::channel::Receiver<(WriteRequest, ResponseSend)>,
/// Access to the database.
db: Arc<ConcreteEnv>,
}
impl DatabaseWriter {
/// Initialize the single `DatabaseWriter` thread.
#[cold]
#[inline(never)] // Only called once.
pub(super) fn init(db: &Arc<ConcreteEnv>) -> DatabaseWriteHandle {
// Initialize `Request/Response` channels.
let (sender, receiver) = crossbeam::channel::unbounded();
// Spawn the writer.
let db = Arc::clone(db);
std::thread::spawn(move || {
let this = Self { receiver, db };
Self::main(this);
});
// Return a handle to the pool.
DatabaseWriteHandle { sender }
}
/// The `DatabaseWriter`'s main function.
///
/// The writer just loops in this function.
#[cold]
#[inline(never)] // Only called once.
fn main(mut self) {
loop {
// 1. Hang on request channel
// 2. Map request to some database function
// 3. Execute that function, get the result
// 4. Return the result via channel
let (request, response_send) = match self.receiver.recv() {
Ok(tuple) => tuple,
Err(e) => {
// TODO: what to do with this channel error?
todo!();
}
};
// Map [`Request`]'s to specific database functions.
match request {
WriteRequest::Example1 => self.example_handler_1(response_send),
WriteRequest::Example2(_x) => self.example_handler_2(response_send),
WriteRequest::Example3(_x) => self.example_handler_3(response_send),
WriteRequest::Shutdown => {
/* TODO: run shutdown code */
Self::shutdown(self);
// Return, exiting the thread.
return;
}
}
}
}
/// TODO
fn example_handler_1(&mut self, response_send: ResponseSend) {
let db_result = todo!();
response_send.send(db_result).unwrap();
}
/// TODO
fn example_handler_2(&mut self, response_send: ResponseSend) {
let db_result = todo!();
response_send.send(db_result).unwrap();
}
/// TODO
fn example_handler_3(&mut self, response_send: ResponseSend) {
let db_result = todo!();
response_send.send(db_result).unwrap();
}
/// TODO
fn shutdown(self) {
todo!()
}
}

210
database/src/table.rs
View file

@ -1,181 +1,49 @@
//! ### Table module //! Database table abstraction; `trait Table`.
//! This module contains the definition of the [`Table`] and [`DupTable`] trait, and the actual tables used in the database.
//! [`DupTable`] are just a trait used to define that they support DUPSORT|DUPFIXED operation (as of now we don't know the equivalent for HSE).
//! All tables are defined with docs explaining its purpose, what types are the key and data.
//! For more details please look at Cuprate's book : <link to cuprate book>
use crate::{ //---------------------------------------------------------------------------------------------------- Import
encoding::Compat, use crate::{key::Key, pod::Pod};
types::{
/*OutTx,*/ AltBlock, BlockMetadata, /*RctOutkey,*/ OutputMetadata,
TransactionPruned, TxIndex, /*OutAmountIdx,*/ /*KeyImage,*/ TxOutputIdx,
},
};
use bincode::{de::Decode, enc::Encode};
use monero::{blockdata::transaction::KeyImage, Block, Hash};
/// A trait implementing a table interaction for the database. It is implemented to an empty struct to specify the name and table's associated types. These associated //---------------------------------------------------------------------------------------------------- Table
/// types are used to simplify deserialization process. /// Database table metadata.
pub trait Table: Send + Sync + 'static + Clone { ///
// name of the table /// Purely compile time information for database tables.
const TABLE_NAME: &'static str; /// Not really an accurate name for `K/V` database but
/// this represents the metadata of a `K/V` storing object.
pub trait Table {
// TODO:
//
// Add K/V comparison `type`s that define
// how this table will be stored.
//
// type KeyComparator: fn(&Self::Key, &Self::Key) -> Ordering;
// type ValueComparator: fn(&Self::Value, &Self::Value) -> Ordering;
// Definition of a key & value types of the database /// Name of the database table.
type Key: Encode + Decode; const NAME: &'static str;
type Value: Encode + Decode;
}
/// A trait implementing a table with duplicated data support. /// Whether the table's values are all the same size or not.
pub trait DupTable: Table { const CONSTANT_SIZE: bool;
// Subkey of the table (prefix of the data)
type SubKey: Encode + Decode;
}
/// This declarative macro declare a new empty struct and impl the specified name, and corresponding types. // TODO: fix this sanakirja bound.
macro_rules! impl_table { cfg_if::cfg_if! {
( $(#[$docs:meta])* $table:ident , $key:ty , $value:ty ) => { if #[cfg(all(feature = "sanakirja", not(feature = "heed")))] {
#[derive(Clone)] /// Primary key type.
$(#[$docs])* type Key: Key + Pod + sanakirja::Storable;
pub(crate) struct $table;
impl Table for $table { /// Value type.
const TABLE_NAME: &'static str = stringify!($table); type Value: Pod + sanakirja::Storable;
type Key = $key; } else {
type Value = $value; /// Primary key type.
type Key: Key + Pod;
/// Value type.
type Value: Pod;
} }
};
}
/// This declarative macro declare extend the original impl_table! macro by implementy DupTable trait.
macro_rules! impl_duptable {
($(#[$docs:meta])* $table:ident, $key:ty, $subkey:ty, $value:ty) => {
impl_table!($(#[$docs])* $table, $key, $value);
impl DupTable for $table {
type SubKey = $subkey;
} }
};
} }
// ------------------------------------------| Tables definition |------------------------------------------ //---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
// ----- BLOCKS ----- mod test {
// use super::*;
impl_duptable!( }
/// `blockhash` is table defining a relation between the hash of a block and its height. Its primary use is to quickly find block's hash by its height.
blockhash,
(),
Compat<Hash>,
u64
);
impl_duptable!(
/// `blockmetadata` store block metadata alongside their corresponding Hash. The blocks metadata can contains the total_coins_generated, weight, long_term_block_weight & cumulative RingCT
blockmetadata,
(),
u64,
BlockMetadata
);
impl_table!(
/// `blockbody` store blocks' bodies along their Hash. The blocks body contains the coinbase transaction and its corresponding mined transactions' hashes.
blocks,
u64,
Compat<Block>
);
/*
impl_table!(
/// `blockhfversion` keep track of block's hard fork version. If an outdated node continue to run after a hard fork, it needs to know, after updating, what blocks needs to be update.
blockhfversion, u64, u8);
*/
impl_table!(
/// `altblock` is a table that permits the storage of blocks from an alternative chain, which may cause a re-org. These blocks can be fetch by their corresponding hash.
altblock,
Compat<Hash>,
AltBlock
);
// ------- TXNs -------
impl_table!(
/// `txspruned` is table storing TransactionPruned (or Pruned Tx). These can be fetch by the corresponding Transaction ID.
txspruned,
u64,
TransactionPruned
);
impl_table!(
/// `txsprunable` is a table storing the Prunable part of transactions (Signatures and RctSig), stored as raw bytes. These can be fetch by the corresponding Transaction ID.
txsprunable,
u64,
Vec<u8>
);
impl_duptable!(
/// `txsprunablehash` is a table storing hashes of prunable part of transactions. These hash can be fetch by the corresponding Transaction ID.
txsprunablehash,
u64,
(),
Compat<Hash>
);
impl_table!(
/// `txsprunabletip` is a table used for optimization purpose. It defines at which block's height this transaction belong as long as the block is with Tip blocks. These can be fetch by the corresponding Transaction ID.
txsprunabletip,
u64,
u64
);
impl_duptable!(
/// `txsoutputs` is a table storing output indices used in a transaction. These can be fetch by the corresponding Transaction ID.
txsoutputs,
u64,
(),
TxOutputIdx
);
impl_duptable!(
/// `txsidentifier` is a table defining a relation between the hash of a transaction and its transaction Indexes. Its primarly used to quickly find tx's ID by its hash.
txsidentifier,
Compat<Hash>,
(),
TxIndex
);
// ---- OUTPUTS ----
impl_duptable!(
/// `prerctoutputmetadata` is a duplicated table storing Pre-RingCT output's metadata. The key is the amount of this output, and the subkey is its amount idx.
prerctoutputmetadata,
u64,
u64,
OutputMetadata
);
impl_duptable!(
/// `prerctoutputmetadata` is a table storing RingCT output's metadata. The key is the amount idx of this output since amount is always 0 for RingCT outputs.
outputmetadata,
(),
u64,
OutputMetadata
);
// ---- SPT KEYS ----
impl_duptable!(
/// `spentkeys`is a table storing every KeyImage that have been used to create decoys input. As these KeyImage can't be re used they need to marked.
spentkeys,
(),
Compat<KeyImage>,
()
);
// ---- PROPERTIES ----
impl_table!(
/// `spentkeys`is a table storing every KeyImage that have been used to create decoys input. As these KeyImage can't be re used they need to marked.
properties,
u32,
u32
);

128
database/src/tables.rs Normal file
View file

@ -0,0 +1,128 @@
//! Database tables.
//!
//! This module contains all the table definitions used by `cuprate-database`
//! and [`Tables`], an `enum` containing all [`Table`]s.
//---------------------------------------------------------------------------------------------------- Import
use crate::table::Table;
//---------------------------------------------------------------------------------------------------- Tables
/// An enumeration of _all_ database tables.
///
/// TODO: I don't think we need this.
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
#[cfg_attr(
feature = "borsh",
derive(borsh::BorshSerialize, borsh::BorshDeserialize)
)]
#[derive(Copy, Clone, Debug, PartialEq, PartialOrd, Eq, Ord, Hash)]
#[allow(missing_docs)]
pub enum Tables {
TestTable(TestTable),
TestTable2(TestTable2),
}
impl Tables {
/// Get the [`Table::NAME`].
pub const fn name(&self) -> &'static str {
/// Hack to access associated trait constant via a variable.
const fn get<T: Table>(t: &T) -> &'static str {
T::NAME
}
match self {
Self::TestTable(t) => get(t),
Self::TestTable2(t) => get(t),
}
}
/// Get the [`Table::CONSTANT_SIZE`].
pub const fn constant_size(&self) -> bool {
/// Hack to access associated trait constant via a variable.
const fn get<T: Table>(t: &T) -> bool {
T::CONSTANT_SIZE
}
match self {
Self::TestTable(t) => get(t),
Self::TestTable2(t) => get(t),
}
}
}
//---------------------------------------------------------------------------------------------------- Table macro
/// Create all tables, should be used _once_.
///
/// Generating this macro once and using `$()*` is probably
/// faster for compile times than calling the macro _per_ table.
///
/// All tables are zero-sized table structs, and implement the `Table` trait.
///
/// Table structs are automatically `CamelCase`,
/// and their static string names are automatically `snake_case`.
macro_rules! tables {
(
$(
$(#[$attr:meta])* // Documentation and any `derive`'s.
$table:ident, // The table name + doubles as the table struct name.
$size:literal, // Are the table's values all the same size?
$key:ty => // Key type.
$value:ty // Value type.
),* $(,)?
) => {
paste::paste! { $(
// Table struct.
$(#[$attr])*
// The below test show the `snake_case` table name in cargo docs.
/// ## Table Name
/// ```rust
/// # use cuprate_database::{*,tables::*};
#[doc = concat!(
"assert_eq!(",
stringify!([<$table:camel>]),
"::NAME, \"",
stringify!([<$table:snake>]),
"\");",
)]
/// ```
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
#[cfg_attr(feature = "borsh", derive(borsh::BorshSerialize, borsh::BorshDeserialize))]
#[derive(Copy,Clone,Debug,PartialEq,PartialOrd,Eq,Ord,Hash)]
pub struct [<$table:camel>];
// Table trait impl.
impl Table for [<$table:camel>] {
const NAME: &'static str = stringify!([<$table:snake>]);
const CONSTANT_SIZE: bool = $size;
type Key = $key;
type Value = $value;
}
// Table enum.
impl From<[<$table:camel>]> for Tables {
fn from(table: [<$table:camel>]) -> Self {
Self::[<$table:camel>](table)
}
}
)* }
};
}
//---------------------------------------------------------------------------------------------------- Tables
tables! {
/// Test documentation.
TestTable,
true,
i64 => u64,
/// Test documentation 2.
TestTable2,
true,
u8 => i8,
}
//---------------------------------------------------------------------------------------------------- Tests
#[cfg(test)]
mod test {
// use super::*;
}

29
database/src/transaction.rs Normal file
View file

@ -0,0 +1,29 @@
//! Database transaction abstraction; `trait RoTx`, `trait RwTx`.
//---------------------------------------------------------------------------------------------------- Import
use crate::error::RuntimeError;
//---------------------------------------------------------------------------------------------------- RoTx
/// Read-only database transaction.
///
/// TODO
pub trait RoTx<'db> {
/// TODO
/// # Errors
/// TODO
fn commit(self) -> Result<(), RuntimeError>;
}
//---------------------------------------------------------------------------------------------------- RwTx
/// Read/write database transaction.
///
/// TODO
pub trait RwTx<'db> {
/// TODO
/// # Errors
/// TODO
fn commit(self) -> Result<(), RuntimeError>;
/// TODO
fn abort(self);
}

33
old_database/Cargo.toml Normal file
View file

@ -0,0 +1,33 @@
[package]
name = "cuprate-database"
version = "0.0.1"
edition = "2021"
license = "AGPL-3.0-only"
# All Contributors on github
authors=[
"SyntheticBird45 <@someoneelse495495:matrix.org>",
"Boog900"
]
[features]
mdbx = ["dep:libmdbx"]
hse = []
[dependencies]
monero = {workspace = true, features = ["serde"]}
tiny-keccak = { version = "2.0", features = ["sha3"] }
serde = { workspace = true}
thiserror = {workspace = true }
bincode = { workspace = true }
libmdbx = { version = "0.3.1", optional = true }
[build]
linker="clang"
rustflags=[
"-Clink-arg=-fuse-ld=mold",
"-Zcf-protection=full",
"-Zsanitizer=cfi",
"-Crelocation-model=pie",
"-Cstack-protector=all",
]

0
database/LICENSE → old_database/LICENSE
View file

0
database/src/encoding.rs → old_database/src/encoding.rs
View file

53
old_database/src/error.rs Normal file
View file

@ -0,0 +1,53 @@
//! ### Error module
//! This module contains all errors abstraction used by the database crate. By implementing [`From<E>`] to the specific errors of storage engine crates, it let us
//! handle more easily any type of error that can happen. This module does **NOT** contain interpretation of these errors, as these are defined for Blockchain abstraction. This is another difference
//! from monerod which interpret these errors directly in its database functions:
//! ```cpp
//! /**
//! * @brief A base class for BlockchainDB exceptions
//! */
//! class DB_EXCEPTION : public std::exception
//! ```
//! see `blockchain_db/blockchain_db.h` in monerod `src/` folder for more details.
#[derive(thiserror::Error, Debug)]
/// `DB_FAILURES` is an enum for backend-agnostic, internal database errors. The `From` Trait must be implemented to the specific backend errors to match DB_FAILURES.
pub enum DB_FAILURES {
#[error("MDBX returned an error {0}")]
MDBX_Error(#[from] libmdbx::Error),
#[error("\n<DB_FAILURES::EncodingError> Failed to encode some data : `{0}`")]
SerializeIssue(DB_SERIAL),
#[error("\nObject already exist in the database : {0}")]
AlreadyExist(&'static str),
#[error("NotFound? {0}")]
NotFound(&'static str),
#[error("\n<DB_FAILURES::Other> `{0}`")]
Other(&'static str),
#[error(
"\n<DB_FAILURES::FailedToCommit> A transaction tried to commit to the db, but failed."
)]
FailedToCommit,
}
#[derive(thiserror::Error, Debug)]
pub enum DB_SERIAL {
#[error("An object failed to be serialized into bytes. It is likely an issue from monero-rs library. Please report this error on cuprate's github : https://github.com/Cuprate/cuprate/issues")]
ConsensusEncode,
#[error("Bytes failed to be deserialized into the requested object. It is likely an issue from monero-rs library. Please report this error on cuprate's github : https://github.com/Cuprate/cuprate/issues")]
ConsensusDecode(Vec<u8>),
#[error("monero-rs encoding|decoding logic failed : {0}")]
MoneroEncode(#[from] monero::consensus::encode::Error),
#[error("Bincode failed to decode a type from the database : {0}")]
BincodeDecode(#[from] bincode::error::DecodeError),
#[error("Bincode failed to encode a type for the database : {0}")]
BincodeEncode(#[from] bincode::error::EncodeError),
}

0
database/src/hse.rs → old_database/src/hse.rs
View file

0
database/src/interface.rs → old_database/src/interface.rs
View file

221
old_database/src/lib.rs Normal file
View file

@ -0,0 +1,221 @@
// Copyright (C) 2023 Cuprate Contributors
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <https://www.gnu.org/licenses/>.
//! The cuprate-db crate implement (as its name suggests) the relations between the blockchain/txpool objects and their databases.
//! `lib.rs` contains all the generics, trait and specification for interfaces between blockchain and a backend-agnostic database
//! Every other files in this folder are implementation of these traits/methods to real storage engine.
//!
//! At the moment, the only storage engine available is MDBX.
//! The next storage engine planned is HSE (Heteregeonous Storage Engine) from Micron.
//!
//! For more informations, please consult this docs:
#![deny(unused_attributes)]
#![forbid(unsafe_code)]
#![allow(non_camel_case_types)]
#![deny(clippy::expect_used, clippy::panic)]
#![allow(dead_code, unused_macros)] // temporary
use monero::{util::ringct::RctSig, Block, BlockHeader, Hash};
use std::ops::Range;
use thiserror::Error;
#[cfg(feature = "mdbx")]
pub mod mdbx;
//#[cfg(feature = "hse")]
//pub mod hse;
pub mod encoding;
pub mod error;
pub mod interface;
pub mod table;
pub mod types;
const DEFAULT_BLOCKCHAIN_DATABASE_DIRECTORY: &str = "blockchain";
const DEFAULT_TXPOOL_DATABASE_DIRECTORY: &str = "txpool_mem";
const BINCODE_CONFIG: bincode::config::Configuration<
bincode::config::LittleEndian,
bincode::config::Fixint,
> = bincode::config::standard().with_fixed_int_encoding();
// ------------------------------------------| Database |------------------------------------------
pub mod database {
//! This module contains the Database abstraction trait. Any key/value storage engine implemented need
//! to fullfil these associated types and functions, in order to be usable. This module also contains the
//! Interface struct which is used by the DB Reactor to interact with the database.
use crate::{
error::DB_FAILURES,
transaction::{Transaction, WriteTransaction},
};
use std::{ops::Deref, path::PathBuf, sync::Arc};
/// `Database` Trait implement all the methods necessary to generate transactions as well as execute specific functions. It also implement generic associated types to identify the
/// different transaction modes (read & write) and it's native errors.
pub trait Database<'a> {
type TX: Transaction<'a>;
type TXMut: WriteTransaction<'a>;
type Error: Into<DB_FAILURES>;
// Create a transaction from the database
fn tx(&'a self) -> Result<Self::TX, Self::Error>;
// Create a mutable transaction from the database
fn tx_mut(&'a self) -> Result<Self::TXMut, Self::Error>;
// Open a database from the specified path
fn open(path: PathBuf) -> Result<Self, Self::Error>
where
Self: std::marker::Sized;
// Check if the database is built.
fn check_all_tables_exist(&'a self) -> Result<(), Self::Error>;
// Build the database
fn build(&'a self) -> Result<(), Self::Error>;
}
/// `Interface` is a struct containing a shared pointer to the database and transaction's to be used for the implemented method of Interface.
pub struct Interface<'a, D: Database<'a>> {
pub db: Arc<D>,
pub tx: Option<<D as Database<'a>>::TXMut>,
}
// Convenient implementations for database
impl<'service, D: Database<'service>> Interface<'service, D> {
fn from(db: Arc<D>) -> Result<Self, DB_FAILURES> {
Ok(Self { db, tx: None })
}
fn open(&'service mut self) -> Result<(), DB_FAILURES> {
let tx = self.db.tx_mut().map_err(Into::into)?;
self.tx = Some(tx);
Ok(())
}
}
impl<'service, D: Database<'service>> Deref for Interface<'service, D> {
type Target = <D as Database<'service>>::TXMut;
fn deref(&self) -> &Self::Target {
return self.tx.as_ref().unwrap();
}
}
}
// ------------------------------------------| DatabaseTx |------------------------------------------
pub mod transaction {
//! This module contains the abstractions of Transactional Key/Value database functions.
//! Any key/value database/storage engine can be implemented easily for Cuprate as long as
//! these functions or equivalent logic exist for it.
use crate::{
error::DB_FAILURES,
table::{DupTable, Table},
};
// Abstraction of a read-only cursor, for simple tables
#[allow(clippy::type_complexity)]
pub trait Cursor<'t, T: Table> {
fn first(&mut self) -> Result<Option<(T::Key, T::Value)>, DB_FAILURES>;
fn get_cursor(&mut self) -> Result<Option<(T::Key, T::Value)>, DB_FAILURES>;
fn last(&mut self) -> Result<Option<(T::Key, T::Value)>, DB_FAILURES>;
fn next(&mut self) -> Result<Option<(T::Key, T::Value)>, DB_FAILURES>;
fn prev(&mut self) -> Result<Option<(T::Key, T::Value)>, DB_FAILURES>;
fn set(&mut self, key: &T::Key) -> Result<Option<T::Value>, DB_FAILURES>;
}
// Abstraction of a read-only cursor with support for duplicated tables. DupCursor inherit Cursor methods as
// a duplicated table can be treated as a simple table.
#[allow(clippy::type_complexity)]
pub trait DupCursor<'t, T: DupTable>: Cursor<'t, T> {
fn first_dup(&mut self) -> Result<Option<(T::SubKey, T::Value)>, DB_FAILURES>;
fn get_dup(
&mut self,
key: &T::Key,
subkey: &T::SubKey,
) -> Result<Option<T::Value>, DB_FAILURES>;
fn last_dup(&mut self) -> Result<Option<(T::SubKey, T::Value)>, DB_FAILURES>;
fn next_dup(&mut self) -> Result<Option<(T::Key, (T::SubKey, T::Value))>, DB_FAILURES>;
fn prev_dup(&mut self) -> Result<Option<(T::Key, (T::SubKey, T::Value))>, DB_FAILURES>;
}
// Abstraction of a read-write cursor, for simple tables. WriteCursor inherit Cursor methods.
pub trait WriteCursor<'t, T: Table>: Cursor<'t, T> {
fn put_cursor(&mut self, key: &T::Key, value: &T::Value) -> Result<(), DB_FAILURES>;
fn del(&mut self) -> Result<(), DB_FAILURES>;
}
// Abstraction of a read-write cursor with support for duplicated tables. DupWriteCursor inherit DupCursor and WriteCursor methods.
pub trait DupWriteCursor<'t, T: DupTable>: WriteCursor<'t, T> {
fn put_cursor_dup(
&mut self,
key: &T::Key,
subkey: &T::SubKey,
value: &T::Value,
) -> Result<(), DB_FAILURES>;
/// Delete all data under associated to its key
fn del_nodup(&mut self) -> Result<(), DB_FAILURES>;
}
// Abstraction of a read-only transaction.
pub trait Transaction<'a>: Send + Sync {
type Cursor<T: Table>: Cursor<'a, T>;
type DupCursor<T: DupTable>: DupCursor<'a, T> + Cursor<'a, T>;
fn get<T: Table>(&self, key: &T::Key) -> Result<Option<T::Value>, DB_FAILURES>;
fn commit(self) -> Result<(), DB_FAILURES>;
fn cursor<T: Table>(&self) -> Result<Self::Cursor<T>, DB_FAILURES>;
fn cursor_dup<T: DupTable>(&self) -> Result<Self::DupCursor<T>, DB_FAILURES>;
fn num_entries<T: Table>(&self) -> Result<usize, DB_FAILURES>;
}
// Abstraction of a read-write transaction. WriteTransaction inherits Transaction methods.
pub trait WriteTransaction<'a>: Transaction<'a> {
type WriteCursor<T: Table>: WriteCursor<'a, T>;
type DupWriteCursor<T: DupTable>: DupWriteCursor<'a, T> + DupCursor<'a, T>;
fn put<T: Table>(&self, key: &T::Key, value: &T::Value) -> Result<(), DB_FAILURES>;
fn delete<T: Table>(
&self,
key: &T::Key,
value: &Option<T::Value>,
) -> Result<(), DB_FAILURES>;
fn clear<T: Table>(&self) -> Result<(), DB_FAILURES>;
fn write_cursor<T: Table>(&self) -> Result<Self::WriteCursor<T>, DB_FAILURES>;
fn write_cursor_dup<T: DupTable>(&self) -> Result<Self::DupWriteCursor<T>, DB_FAILURES>;
}
}

0
database/src/mdbx.rs → old_database/src/mdbx.rs
View file

181
old_database/src/table.rs Normal file
View file

@ -0,0 +1,181 @@
//! ### Table module
//! This module contains the definition of the [`Table`] and [`DupTable`] trait, and the actual tables used in the database.
//! [`DupTable`] are just a trait used to define that they support DUPSORT|DUPFIXED operation (as of now we don't know the equivalent for HSE).
//! All tables are defined with docs explaining its purpose, what types are the key and data.
//! For more details please look at Cuprate's book : <link to cuprate book>
use crate::{
encoding::Compat,
types::{
/*OutTx,*/ AltBlock, BlockMetadata, /*RctOutkey,*/ OutputMetadata,
TransactionPruned, TxIndex, /*OutAmountIdx,*/ /*KeyImage,*/ TxOutputIdx,
},
};
use bincode::{de::Decode, enc::Encode};
use monero::{blockdata::transaction::KeyImage, Block, Hash};
/// A trait implementing a table interaction for the database. It is implemented to an empty struct to specify the name and table's associated types. These associated
/// types are used to simplify deserialization process.
pub trait Table: Send + Sync + 'static + Clone {
// name of the table
const TABLE_NAME: &'static str;
// Definition of a key & value types of the database
type Key: Encode + Decode;
type Value: Encode + Decode;
}
/// A trait implementing a table with duplicated data support.
pub trait DupTable: Table {
// Subkey of the table (prefix of the data)
type SubKey: Encode + Decode;
}
/// This declarative macro declare a new empty struct and impl the specified name, and corresponding types.
macro_rules! impl_table {
( $(#[$docs:meta])* $table:ident , $key:ty , $value:ty ) => {
#[derive(Clone)]
$(#[$docs])*
pub(crate) struct $table;
impl Table for $table {
const TABLE_NAME: &'static str = stringify!($table);
type Key = $key;
type Value = $value;
}
};
}
/// This declarative macro declare extend the original impl_table! macro by implementy DupTable trait.
macro_rules! impl_duptable {
($(#[$docs:meta])* $table:ident, $key:ty, $subkey:ty, $value:ty) => {
impl_table!($(#[$docs])* $table, $key, $value);
impl DupTable for $table {
type SubKey = $subkey;
}
};
}
// ------------------------------------------| Tables definition |------------------------------------------
// ----- BLOCKS -----
impl_duptable!(
/// `blockhash` is table defining a relation between the hash of a block and its height. Its primary use is to quickly find block's hash by its height.
blockhash,
(),
Compat<Hash>,
u64
);
impl_duptable!(
/// `blockmetadata` store block metadata alongside their corresponding Hash. The blocks metadata can contains the total_coins_generated, weight, long_term_block_weight & cumulative RingCT
blockmetadata,
(),
u64,
BlockMetadata
);
impl_table!(
/// `blockbody` store blocks' bodies along their Hash. The blocks body contains the coinbase transaction and its corresponding mined transactions' hashes.
blocks,
u64,
Compat<Block>
);
/*
impl_table!(
/// `blockhfversion` keep track of block's hard fork version. If an outdated node continue to run after a hard fork, it needs to know, after updating, what blocks needs to be update.
blockhfversion, u64, u8);
*/
impl_table!(
/// `altblock` is a table that permits the storage of blocks from an alternative chain, which may cause a re-org. These blocks can be fetch by their corresponding hash.
altblock,
Compat<Hash>,
AltBlock
);
// ------- TXNs -------
impl_table!(
/// `txspruned` is table storing TransactionPruned (or Pruned Tx). These can be fetch by the corresponding Transaction ID.
txspruned,
u64,
TransactionPruned
);
impl_table!(
/// `txsprunable` is a table storing the Prunable part of transactions (Signatures and RctSig), stored as raw bytes. These can be fetch by the corresponding Transaction ID.
txsprunable,
u64,
Vec<u8>
);
impl_duptable!(
/// `txsprunablehash` is a table storing hashes of prunable part of transactions. These hash can be fetch by the corresponding Transaction ID.
txsprunablehash,
u64,
(),
Compat<Hash>
);
impl_table!(
/// `txsprunabletip` is a table used for optimization purpose. It defines at which block's height this transaction belong as long as the block is with Tip blocks. These can be fetch by the corresponding Transaction ID.
txsprunabletip,
u64,
u64
);
impl_duptable!(
/// `txsoutputs` is a table storing output indices used in a transaction. These can be fetch by the corresponding Transaction ID.
txsoutputs,
u64,
(),
TxOutputIdx
);
impl_duptable!(
/// `txsidentifier` is a table defining a relation between the hash of a transaction and its transaction Indexes. Its primarly used to quickly find tx's ID by its hash.
txsidentifier,
Compat<Hash>,
(),
TxIndex
);
// ---- OUTPUTS ----
impl_duptable!(
/// `prerctoutputmetadata` is a duplicated table storing Pre-RingCT output's metadata. The key is the amount of this output, and the subkey is its amount idx.
prerctoutputmetadata,
u64,
u64,
OutputMetadata
);
impl_duptable!(
/// `prerctoutputmetadata` is a table storing RingCT output's metadata. The key is the amount idx of this output since amount is always 0 for RingCT outputs.
outputmetadata,
(),
u64,
OutputMetadata
);
// ---- SPT KEYS ----
impl_duptable!(
/// `spentkeys`is a table storing every KeyImage that have been used to create decoys input. As these KeyImage can't be re used they need to marked.
spentkeys,
(),
Compat<KeyImage>,
()
);
// ---- PROPERTIES ----
impl_table!(
/// `spentkeys`is a table storing every KeyImage that have been used to create decoys input. As these KeyImage can't be re used they need to marked.
properties,
u32,
u32
);

0
database/src/types.rs → old_database/src/types.rs
View file