* add `cuprate-types`
* remove `cuprate_database::service::{request,response}`
* use `cuprate_types::service::{request,response}`
* service: fix `Request` `match`'s
* service: create `ReadRequest` function mappings
* service: create `WriteRequest` function mappings
* service: add rough `WriteRequest` retry loop
* service: handle `RuntimeError::ResizeNeeded` in writer
* add `{R,r}o` exception to typos
* docs
* env: make `resize_map()` return new memory map byte size
* write: proactively handle resizes
`add_block()` takes `VerifiedBlockInformation` such
that it can just take the inner blobs of data.
This is a problem when reactively resizing since we no longer
have the block struct we just gave away so we're forced to `.clone()`
each retry.
Instead of that - we will proactively resize so the resize error
will never occur in the first place.
* read: use type aliases
* docs
* fix import
* write: handle resizes reactively
* service: panic if response can't be sent back
* write: add loop unreachable asserts
* service: print and drop error instead of panic
* write: fix retry loop off-by-1
* write: fix docs
* review changes
* update readme
* remove `BlockBatchInRange` request/response
* Update database/README.md
Co-authored-by: Boog900 <boog900@tutanota.com>
---------
Co-authored-by: Boog900 <boog900@tutanota.com>
* use tokio's delay queue for bans
* document handles
* remove peers from address book when retrieving
* ping inbound peers during handshakes
* support receiving pings during handshakes
* add peer to anchor before reducing whit list
* clippy
* comment handshakes
* typos
* sort `use`
* use `rand::prelude::*`
* review comments
* update macro
* levin: fix fragmented messages & use bitflags
* levin: add a method to fragment a message
* levin: add tests for fragmented messages and fix issues
* fix docs
* tests: don't include bytes length
* levin: add support for sending fragmented
/ dummy messages
* fmt
* add fragmented handshake tests.
* fix handshake detection when fragmented
and alt (non-monero) protocol info
* add tracing logs
* remove `already_built`, this was an old way I was thinking of sending raw buckets
* clippy
* clippy 2
* Update net/levin/src/message.rs
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
* review comments
* add timeout to tests
* Update net/levin/src/header.rs
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
---------
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
* monerod: set a random DB and remove dir when done
* add a test for an inbound monerod handshake
* don't fail if can't remove dir on windows
* Update test-utils/src/monerod.rs
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
* use `temp_dir`
* use `tempfile`
---------
Co-authored-by: hinto-janai <hinto.janai@protonmail.com>
* error: add `NeedsResize`
accidently removed, was `MapFull` before.
We need this because we as the writer thread must
react to this error in order to resize.
The writer thread doesn't have access to `heed::Error`, but
`Runtime::Error`, so this variant must exist
* env/backend: add `MANUAL_RESIZE` and `resize()`
* heed: respect `ReadersFull`, comment errors
* free: add `resize_memory_map()`
* env: add `Env::current_map_size`
* service: add `resize_map()`
* database: make `Env` itself cheaply clonable + threadsafe
`heed::Env` already uses `Arc` internally, but `sanakirja` does
not, so abstract this at the `Env` level instead of wrapping
`ConcreteEnv` in `Arc` ourselves.
* service: `Arc<ConcreteEnv>` -> `ConcreteEnv: Clone`
* env: add `SYNCS_PER_TX`, `path()`, `shutdown()`
* database: add `src/service/state.rs`
* service: add to `state.rs`, add `DatabaseState` to readers/writer
* add `parking_lot`
Okay, turns out we need to take locks in
`database/src/service/state.rs`...
`std`'s lock fairness policies are not well defined and
depend on the OS implementation, `parking_lot` on the other hand
has a fairness policy which is important when the writer needs
the lock but readers keep pouring in, essentially never letting
the writer do stuff.
* state: use `crossbeam::atomic::AtomicCell`
We have crossbeam as a dep anyway.
* heed: `heed::Env` -> `Arc<RwLock<heed::Env>>`
* service: add reader shutdown handle, use `Select` for msgs
* service: remove `state.rs`
We don't need this, we will represent shutdowns with channel
messages and `Select`, and mutual exclusion with a `RwLock`.
* service: fix misc reader/writer stuff
* database: add `config.rs`
* service: use `ReaderThreads` when spawning readers
* service: impl `shutdown()` for readers/writer
* service: return `DatabaseReaderReceivers` on shutdown via `JoinHandle`
Solves the issue of unfortunately timed `Request`s
that come in _right_ as we are shutting down.
If we (Cuprate) drop the database channels too early the requesting
thread will probably panic as they probably use `.unwrap()`,
never expecting a channel failure.
Returning this structure containing all the channels allows the
final shutdown code to carry these channels until the very end
of the program, at which point, all threads exit - no panics.
* remove `parking_lot`
Could be used as the database mutual exclusion lock.
Needs to be tested against `std`.
* config: add `path`
* env: `path()` -> `config()`, backend: impl `Drop`
* `Arc<ConcreteEnv>`, shutdown `service` on channel disconnect
* backend: add `Config` to `ConcreteEnv`
* service: use `std:🧵:Builder` for readers/writer
* config: `PathBuf` -> `Cow<'static, Path>`
* misc docs, remove `RuntimeError::ShuttingDown`
* service: init & shutdown docs
* heed: impl `Env::resize_map()`
* heed: impl `Env::current_map_size()`
* lib.rs: add example crate usage test
* heed: `RwLock` comment
* helper: add `cuprate_database_dir()`
* config: use `cuprate_database_dir()`
* lib.rs: TODO example test
* database: add `page_size`
The database memory map size must be a multiple of
the OS page size. Why doesn't `heed` re-expose this?
It calls it when checking anyway...
https://docs.rs/heed/0.20.0-alpha.9/src/heed/env.rs.html#772
* free: impl `resize_memory_map()`
* free: docs
* env: add `disk_size_bytes()`
* config: impl `From<$num>` for `ReaderThreads`
* move `fs`-related constants to `cuprate_helper::fs`
* docs
* add `resize.rs`, `ResizeAlgorithm`
* env: use `ResizeAlgorithm` in `resize_map()`
* TODO: account for LMDB reader limit
* resize: docs, add `page_size()`, impl `fixed_bytes()`
* resize: impl `percent()`
* resize: docs
* env: take `ResizeAlgorithm` by value (it impls `Copy`)
* heed: TODO for `MDB_MAP_FULL` & `MDB_MAP_RESIZED`
* config: `From<Into<usize>>` for `ReaderThreads`
Co-authored-by: Boog900 <boog900@tutanota.com>
* env: move mutual exclusion doc to backend
* free: update invariant doc
Co-authored-by: Boog900 <boog900@tutanota.com>
* Update database/src/service/mod.rs
Co-authored-by: Boog900 <boog900@tutanota.com>
* fix `[allow(unused_imports)] // docs`
* move DB filename from `helper/` -> `database/`
* config: use `DATABASE_FILENAME`
* config: add `db_file_path()`
* lib: add non-`service` usage invariant docs
* table: seal `Table` trait, impl for all `crate::tables`
* fix docs
* fix docs pt.2
---------
Co-authored-by: Boog900 <boog900@tutanota.com>
* output monerod logs when a thread panics
* always spawn a monerod don't attempt to re-use
* set zmq port and `non-interactive`
* check monerod has started before test
* remove test panic
* review changes
* consensus: fix batch handling when we don't have a full list of outputs.
* change `scan_chain` to new API
* clippy
* add a test for calculating multiple difficulties
* fmt
* rx_seed -> rx_vms
* consensus-rules: ring members younger than current block.
* only create rx vms when required.
* fix rx initiation when syncing
* add single block verification (no batch)
* update serai
* ci: install boost, include macos + windows
* cryptonight: fix `MSVC`
* cryptonight: use `flag_if_supported()`
* fix cryptonight builds
* update randomX
* fix rx builds
* add memwipe
* include memwipe.c in build
* spawn monerod in msys2 for windows
* fix last commit
* install dependencies before spawning monerod
* remove --detach
* try another way of spawning monerod
* add /I
* download and spawn monerod as a part of tests
* add download.rs
* extend time for monerod spawn
* move sleep and show monerod output
* fix clippy
* change stdin to pipped
* #[cfg(unix)] on bytes::Buf
* fix macos capitalisation
* remove tar.bz2 on macos expected dir
* remove zip on windows expected dir
* fix todo
* add docs
* fix a couple typos
---------
Co-authored-by: Boog900 <54e72d8a-345f-4599-bd90-c6b9bc7d0ec5@aleeas.com>
Co-authored-by: Boog900 <boog900@tutanota.com>
Pins versions, preventing publish updates from being automatically applied.
Implications for #39, which at one point mentions the ability to use
`cargo vendor` for some reason?
