books: architecture skeleton (#224)

* upstream https://github.com/hinto-janai/cuprate-architecture

* tab -> 4 spaces

books/architecture/README.md

@@ -1,4 +1,4 @@
-## Cuprate's architecture (implementation) book
+## Cuprate's architecture book
 This book documents Cuprate's architecture and implementation.
 
 See:

books/architecture/book.toml

@@ -1,19 +1,17 @@
 [book]
-authors = ["hinto-janai"]
+authors = ["Cuprate Contributors"]
 language = "en"
 multilingual = false
 src = "src"
 title = "Cuprate Architecture"
 git-repository-url = "https://github.com/Cuprate/architecture-book"
 
-# TODO: fix after importing real files.
-#
-# [preprocessor.last-changed]
-# command = "mdbook-last-changed"
-# renderer = ["html"]
-#
-# [output.html]
-# default-theme = "ayu"
-# preferred-dark-theme = "ayu"
-# git-repository-url = "https://github.com/hinto-janai/cuprate-architecture"
-# additional-css = ["last-changed.css"]
+[preprocessor.last-changed]
+command = "mdbook-last-changed"
+renderer = ["html"]
+
+[output.html]
+default-theme = "ayu"
+preferred-dark-theme = "ayu"
+git-repository-url = "https://github.com/Cuprate/architecture-book"
+additional-css = ["last-changed.css"]

books/architecture/last-changed.css

@@ -0,0 +1,7 @@
+footer {
+  font-size: 0.8em;
+  text-align: center;
+  border-top: 1px solid;
+  margin-top: 4%;
+  padding: 5px 0;
+}

books/architecture/src/SUMMARY.md

@@ -1,3 +1,124 @@
 # Summary
 
-- [TODO](todo.md)
+[Cuprate Architecture](cuprate-architecture.md)
+[🟡 Foreword](foreword.md)
+
+---
+
+- [🟠 Intro](intro/intro.md)
+    - [🟡 Who this book is for](intro/who-this-book-is-for.md)
+    - [🔴 Required knowledge](intro/required-knowledge.md)
+    - [🔴 How to use this book](intro/how-to-use-this-book.md)
+
+---
+
+- [⚪️ Bird's eye view](birds-eye-view/intro.md)
+    - [⚪️ Map](birds-eye-view/map.md)
+    - [⚪️ Components](birds-eye-view/components.md)
+
+---
+
+- [⚪️ Formats, protocols, types](formats-protocols-types/intro.md)
+    - [⚪️ monero_serai](formats-protocols-types/monero-serai.md)
+    - [⚪️ cuprate_types](formats-protocols-types/cuprate-types.md)
+    - [⚪️ cuprate_helper](formats-protocols-types/cuprate-helper.md)
+    - [⚪️ Epee](formats-protocols-types/epee.md)
+    - [⚪️ Levin](formats-protocols-types/levin.md)
+
+---
+
+- [⚪️ Storage](storage/intro.md)
+    - [⚪️ Database abstraction](storage/database-abstraction.md)
+    - [⚪️ Blockchain](storage/blockchain.md)
+    - [⚪️ Transaction pool](storage/transaction-pool.md)
+    - [⚪️ Pruning](storage/pruning.md)
+
+---
+
+- [🔴 RPC](rpc/intro.md)
+    - [⚪️ Types](rpc/types/intro.md)
+        - [⚪️ JSON](rpc/types/json.md)
+        - [⚪️ Binary](rpc/types/binary.md)
+        - [⚪️ Other](rpc/types/other.md)
+    - [⚪️ Interface](rpc/interface.md)
+    - [⚪️ Router](rpc/router.md)
+    - [⚪️ Handler](rpc/handler.md)
+    - [⚪️ Methods](rpc/methods/intro.md)
+
+---
+
+- [⚪️ ZMQ](zmq/intro.md)
+    - [⚪️ TODO](zmq/todo.md)
+
+---
+
+- [⚪️ Consensus](consensus/intro.md)
+    - [⚪️ Verifier](consensus/verifier.md)
+    - [⚪️ TODO](consensus/todo.md)
+
+---
+
+- [⚪️ Networking](networking/intro.md)
+    - [⚪️ P2P](networking/p2p.md)
+    - [⚪️ Dandelion++](networking/dandelion.md)
+    - [⚪️ Proxy](networking/proxy.md)
+    - [⚪️ Tor](networking/tor.md)
+    - [⚪️ i2p](networking/i2p.md)
+    - [⚪️ IPv4/IPv6](networking/ipv4-ipv6.md)
+
+---
+
+- [🔴 Instrumentation](instrumentation/intro.md)
+    - [⚪️ Logging](instrumentation/logging.md)
+    - [⚪️ Data collection](instrumentation/data-collection.md)
+
+---
+
+- [⚪️ Binary](binary/intro.md)
+    - [⚪️ CLI](binary/cli.md)
+    - [⚪️ Config](binary/config.md)
+    - [⚪️ Logging](binary/logging.md)
+
+---
+
+- [⚪️ Resource model](resource-model/intro.md)
+    - [⚪️ File system](resource-model/file-system.md)
+    - [⚪️ Sockets](resource-model/sockets.md)
+    - [⚪️ Memory](resource-model/memory.md)
+    - [🟡 Concurrency and parallelism](resource-model/concurrency-and-parallelism/intro.md)
+        - [⚪️ Map](resource-model/concurrency-and-parallelism/map.md)
+        - [⚪️ The RPC server](resource-model/concurrency-and-parallelism/the-rpc-server.md)
+        - [⚪️ The database](resource-model/concurrency-and-parallelism/the-database.md)
+        - [⚪️ The block downloader](resource-model/concurrency-and-parallelism/the-block-downloader.md)
+        - [⚪️ The verifier](resource-model/concurrency-and-parallelism/the-verifier.md)
+        - [⚪️ Thread exit](resource-model/concurrency-and-parallelism/thread-exit.md)
+
+---
+
+- [⚪️ External Monero libraries](external-monero-libraries/intro.md)
+    - [⚪️ Cryptonight](external-monero-libraries/cryptonight.md)
+    - [🔴 RandomX](external-monero-libraries/randomx.md)
+    - [🔴 monero_serai](external-monero-libraries/monero_serai.md)
+
+---
+
+- [⚪️ Benchmarking](benchmarking/intro.md)
+    - [⚪️ Criterion](benchmarking/criterion.md)
+    - [⚪️ Harness](benchmarking/harness.md)
+- [⚪️ Testing](testing/intro.md)
+    - [⚪️ Monero data](testing/monero-data.md)
+    - [⚪️ RPC client](testing/rpc-client.md)
+    - [⚪️ Spawning `monerod`](testing/spawning-monerod.md)
+- [⚪️ Known issues and tradeoffs](known-issues-and-tradeoffs/intro.md)
+    - [⚪️ Networking](known-issues-and-tradeoffs/networking.md)
+    - [⚪️ RPC](known-issues-and-tradeoffs/rpc.md)
+    - [⚪️ Storage](known-issues-and-tradeoffs/storage.md)
+
+---
+
+- [⚪️ Appendix](appendix/intro.md)
+    - [🔴 Contributing](appendix/contributing.md)
+    - [🔴 Crate documentation](appendix/crate-documentation.md)
+    - [🔴 Build targets](appendix/build-targets.md)
+    - [🔴 Protocol book](appendix/protocol-book.md)
+    - [⚪️ User book](appendix/user-book.md)

books/architecture/src/appendix/build-targets.md

@@ -0,0 +1,7 @@
+# Build targets
+- x86
+- ARM64
+- Windows
+- Linux
+- macOS
+- FreeBSD(?)

books/architecture/src/appendix/contributing.md

@@ -0,0 +1,2 @@
+# Contributing
+<https://github.com/Cuprate/cuprate/blob/main/CONTRIBUTING.md>

books/architecture/src/appendix/crate-documentation.md

@@ -0,0 +1,4 @@
+# Crate documentation
+```bash
+cargo doc --package $CUPRATE_CRATE
+```

books/architecture/src/appendix/intro.md

@@ -0,0 +1 @@
+# Appendix

books/architecture/src/appendix/protocol-book.md

@@ -0,0 +1,2 @@
+# Protocol book
+<https://monero-book.cuprate.org>

books/architecture/src/appendix/user-book.md

@@ -0,0 +1 @@
+# ⚪️ User book

books/architecture/src/benchmarking/criterion.md

@@ -0,0 +1 @@
+# ⚪️ Criterion

books/architecture/src/benchmarking/harness.md

@@ -0,0 +1 @@
+# ⚪️ Harness

books/architecture/src/benchmarking/intro.md

@@ -0,0 +1 @@
+# ⚪️ Benchmarking

books/architecture/src/binary/cli.md

@@ -0,0 +1 @@
+# ⚪️ CLI

books/architecture/src/binary/config.md

@@ -0,0 +1 @@
+# ⚪️ Config

books/architecture/src/binary/intro.md

@@ -0,0 +1 @@
+# ⚪️ Binary

books/architecture/src/binary/logging.md

@@ -0,0 +1 @@
+# ⚪️ Logging

books/architecture/src/birds-eye-view/components.md

@@ -0,0 +1 @@
+# ⚪️ Components

books/architecture/src/birds-eye-view/intro.md

@@ -0,0 +1 @@
+# ⚪️ Bird's eye view

books/architecture/src/birds-eye-view/map.md

@@ -0,0 +1 @@
+# ⚪️ Map

books/architecture/src/consensus/intro.md

@@ -0,0 +1 @@
+# ⚪️ Consensus

books/architecture/src/consensus/todo.md

@@ -0,0 +1 @@
+# ⚪️ TODO

books/architecture/src/consensus/verifier.md

@@ -0,0 +1 @@
+# ⚪️ Verifier

books/architecture/src/cuprate-architecture.md

@@ -0,0 +1,22 @@
+# Cuprate Architecture
+WIP
+
+[Cuprate](https://github.com/Cuprate/cuprate)'s architecture book.
+
+Sections are notated with colors indicating how complete they are:
+
+| Color | Meaning |
+|-------|---------|
+| ⚪️    | Empty
+| 🔴    | Severely lacking information
+| 🟠    | Lacking some information
+| 🟡    | Almost ready
+| 🟢    | OK
+
+---
+
+Continue to the next chapter by clicking the right `>` button, or by selecting it on the left side.
+
+All chapters are viewable by clicking the top-left `☰` button.
+
+The entire book can searched by clicking the top-left 🔍 button.

books/architecture/src/external-monero-libraries/cryptonight.md

@@ -0,0 +1 @@
+# ⚪️ Cryptonight

books/architecture/src/external-monero-libraries/intro.md

@@ -0,0 +1 @@
+# ⚪️ External Monero libraries

books/architecture/src/external-monero-libraries/monero_serai.md

@@ -0,0 +1,2 @@
+# monero_serai
+<https://github.com/serai-dex/serai/tree/develop/coins/monero>

books/architecture/src/external-monero-libraries/randomx.md

@@ -0,0 +1,2 @@
+# RandomX
+<https://github.com/tari-project/randomx-rs>

books/architecture/src/foreword.md

@@ -0,0 +1,36 @@
+# Foreword
+Monero[^1] is a large software project, coming in at 329k lines of C++, C, headers, and make files.[^2] It is directly responsible for 2.6 billion dollars worth of value.[^3] It has had over 400 contributors, more if counting unnamed contributions.[^4] It has over 10,000 node operators and a large active userbase.[^5]
+
+The project wasn't always this big, but somewhere in the midst of contributors coming and going, various features being added, bugs being fixed, and celebrated cryptography being implemented - there was an aspect that was lost by the project that it could not easily gain again: **maintainability**.
+
+Within large and complicated software projects, there is an important transfer of knowledge that must occur for long-term survival. Much like an organism that must eventually pass the torch onto the next generation, projects must do the same for future contributors.
+
+However, newcomers often lack experience, past contributors might not be around, and current maintainers may be too busy. For whatever reason, this transfer of knowledge is not always smooth.
+
+There is a solution to this problem: **documentation**.
+
+The activity of writing the what, where, why, and how of the solutions to technical problems can be done in an author's lonesome.
+
+The activity of reading these ideas can be done by future readers at any time without permission.
+
+These readers may be new prospective contributors, it may be the current maintainers, it may be researchers, it may be users of various scale.  Whoever it may be, documentation acts as the link between the past and present; a bottle of wisdom thrown into the river of time for future participants to open.
+
+This book is the manifestation of this will, for Cuprate[^6], an alternative Monero node. It documents Cuprate's implementation from head-to-toe such that in the case of a contributor's untimely disappearance, the project can continue.
+
+People come and go, documentation is forever.
+
+— hinto-janai
+
+---
+
+[^1]: [`monero-project/monero`](https://github.com/monero-project/monero)
+
+[^2]: `git ls-files | grep "\.cpp$\|\.h$\|\.c$\|CMake" | xargs cat | wc -l` on [`cc73fe7`](https://github.com/monero-project/monero/tree/cc73fe71162d564ffda8e549b79a350bca53c454)
+
+[^3]: 2024-05-24: $143.55 USD * 18,151,608 XMR = $2,605,663,258
+
+[^4]: `git log --all --pretty="%an" | sort -u | wc -l` on [`cc73fe7`](https://github.com/monero-project/monero/tree/cc73fe71162d564ffda8e549b79a350bca53c454)
+
+[^5]: <https://monero.fail/map>
+
+[^6]: <https://github.com/Cuprate/cuprate>

books/architecture/src/formats-protocols-types/cuprate-helper.md

@@ -0,0 +1 @@
+# ⚪️ cuprate_helper

books/architecture/src/formats-protocols-types/cuprate-types.md

@@ -0,0 +1 @@
+# ⚪️ cuprate_types

books/architecture/src/formats-protocols-types/epee.md

@@ -0,0 +1 @@
+# ⚪️ Epee

books/architecture/src/formats-protocols-types/intro.md

@@ -0,0 +1 @@
+# ⚪️ Formats, protocols, types

books/architecture/src/formats-protocols-types/levin.md

@@ -0,0 +1 @@
+# ⚪️ Levin

books/architecture/src/formats-protocols-types/monero-serai.md

@@ -0,0 +1 @@
+# ⚪️ monero_serai

books/architecture/src/instrumentation/data-collection.md

@@ -0,0 +1 @@
+# ⚪️ Data collection

books/architecture/src/instrumentation/intro.md

@@ -0,0 +1,2 @@
+# Instrumentation
+Cuprate is built with [instrumentation](https://en.wikipedia.org/wiki/Instrumentation) in mind.

books/architecture/src/instrumentation/logging.md

@@ -0,0 +1 @@
+# ⚪️ Logging

books/architecture/src/intro.md

@@ -0,0 +1,4 @@
+# Systems
+Cuprate is made up of multiple distinct internal systems that work together.
+
+This section provides informal specifications and implementation details about each.

books/architecture/src/intro/how-to-use-this-book.md

@@ -0,0 +1,5 @@
+# How to use this book
+
+## Maintainers
+## Contributors
+## Researchers

books/architecture/src/intro/intro.md

@@ -0,0 +1,15 @@
+# Intro
+[Cuprate](https://github.com/Cuprate/cuprate) is an alternative [Monero](https://getmonero.org) node implementation.
+
+This book describes Cuprate's architecture, ranging from small things like database pruning to larger meta-components like the networking stack.
+
+A brief overview of some aspects covered within this book:
+- Component designs
+- Implementation details
+- File location and purpose
+- Design decisions and tradeoffs
+- Things in relation to `monerod`
+- Dependency usage
+
+## Source code
+The source files for this book can be found on at: <https://github.com/Cuprate/architecture-book>.

books/architecture/src/intro/required-knowledge.md

@@ -0,0 +1,28 @@
+# Required knowledge
+
+## General
+- Rust
+- Monero
+- System design
+
+## Components
+### Storage
+- Embedded databases
+- LMDB
+- redb
+
+### RPC
+- `axum`
+- `tower`
+- `async`
+- JSON-RPC 2.0
+- Epee
+
+### Networking
+- `tower`
+- `tokio`
+- `async`
+- Levin
+
+### Instrumentation
+- `tracing`

books/architecture/src/intro/who-this-book-is-for.md

@@ -0,0 +1,31 @@
+# Who this book is for
+
+## Maintainers
+As mentioned in [`Foreword`](../foreword.md), the group of people that benefit from this book's value the most by far are the current and future Cuprate maintainers.
+
+Cuprate's system design is documented in this book such that if you were ever to build it again from scratch, you would have an excellent guide on how to do such, and also where improvements could be made.
+
+Practically, what that means for maintainers is that it acts as _the_ reference. During maintenance, it is quite valuable to have a book that contains condensed knowledge on the behavior of components, or how certain code works, or why it was built a certain way.
+
+## Contributors
+Contributors also have access to the inner-workings of Cuprate via this book, which helps when making larger contributions.
+
+Design decisions and implementation details notated in this book helps answer questions such as:
+- Why is it done this way?
+- Why can it _not_ be done this way?
+- Were other methods attempted?
+
+Cuprate's testing and benchmarking suites, unknown to new contributors, are also documented within this book.
+
+## Researchers
+This book contains the why, where, and how of the _implementation_ of formal research.
+
+Although it is an informal specification, this book still acts as a more accessible overview of Cuprate compared to examining the codebase itself.
+
+## Operators & users
+This book is not a practical guide for using Cuprate itself.
+
+For configuration, data collection (also important for researchers), and other practical usage, see [Cuprate's user book](https://user.cuprate.org).
+
+## Observers
+Anyone curious enough is free to learn the inner-workings of Cuprate via this book, and maybe even contribute someday.

books/architecture/src/known-issues-and-tradeoffs/intro.md

@@ -0,0 +1 @@
+# ⚪️ Known issues and tradeoffs

books/architecture/src/known-issues-and-tradeoffs/networking.md

@@ -0,0 +1 @@
+# ⚪️ Networking

books/architecture/src/known-issues-and-tradeoffs/rpc.md

@@ -0,0 +1 @@
+# ⚪️ RPC

books/architecture/src/known-issues-and-tradeoffs/storage.md

@@ -0,0 +1 @@
+# ⚪️ Storage

books/architecture/src/networking/dandelion.md

@@ -0,0 +1 @@
+# ⚪️ Dandelion++

books/architecture/src/networking/i2p.md

@@ -0,0 +1 @@
+# ⚪️ i2p

books/architecture/src/networking/intro.md

@@ -0,0 +1 @@
+# ⚪️ Networking

books/architecture/src/networking/ipv4-ipv6.md

@@ -0,0 +1 @@
+# ⚪️ IPv4/IPv6

books/architecture/src/networking/p2p.md

@@ -0,0 +1 @@
+# ⚪️ P2P

books/architecture/src/networking/proxy.md

@@ -0,0 +1 @@
+# ⚪️ Proxy

books/architecture/src/networking/tor.md

@@ -0,0 +1 @@
+# ⚪️ Tor

books/architecture/src/resource-model/concurrency-and-parallelism/intro.md

@@ -0,0 +1,32 @@
+# Concurrency and parallelism
+It is incumbent upon software like Cuprate to take advantage of today's highly parallel hardware as much as practically possible.
+
+With that said, programs must setup guardrails when operating in a concurrent and parallel manner, [for correctness and safety](https://en.wikipedia.org/wiki/Concurrency_(computer_science)).
+
+There are "synchronization primitives" that help with this, common ones being:
+- [Locks](https://en.wikipedia.org/wiki/Lock_(computer_science))
+- [Channels](https://en.wikipedia.org/wiki/Channel_(programming))
+- [Atomics](https://en.wikipedia.org/wiki/Linearizability#Primitive_atomic_instructions)
+
+These tools are relatively easy to use in isolation, but trickier to do so when considering the entire system. It is not uncommon for _the_ bottleneck to be the [poor orchastration](https://en.wikipedia.org/wiki/Starvation_(computer_science)) of these primitives.
+
+## Analogy
+A common analogy for a parallel system is an intersection.
+
+Like a parallel computer system, an intersection contains:
+1. **Parallelism:** multiple individual units that want to move around (cars, pedestrians, etc)
+1. **Synchronization primitives:** traffic lights, car lights, walk signals
+
+In theory, the amount of "work" the units can do is only limited by the speed of the units themselves, but in practice, the slow cascading reaction speeds between all units, the frequent hiccups that can occur, and the synchronization primitives themselves become bottlenecks far before the maximum speed of any unit is reached.
+
+A car that hogs the middle of the intersection on the wrong light is akin to a system thread holding onto a lock longer than it should be - it degrades total system output.
+
+Unlike humans however, computer systems at least have the potential to move at lightning speeds, but only if the above synchronization primitives are used correctly.
+
+## Goal
+To aid the long-term maintenance of highly concurrent and parallel code, this section documents:
+1. All system threads spawned and maintained
+1. All major sections where synchronization primitives are used
+1. The asynchronous behavior of some components
+
+and how these compose together efficiently in Cuprate.

books/architecture/src/resource-model/concurrency-and-parallelism/map.md

@@ -0,0 +1 @@
+# ⚪️ Map

books/architecture/src/resource-model/concurrency-and-parallelism/the-block-downloader.md

@@ -0,0 +1 @@
+# ⚪️ The block downloader

books/architecture/src/resource-model/concurrency-and-parallelism/the-database.md

@@ -0,0 +1 @@
+# ⚪️ The database

books/architecture/src/resource-model/concurrency-and-parallelism/the-rpc-server.md

@@ -0,0 +1 @@
+# ⚪️ The RPC server

books/architecture/src/resource-model/concurrency-and-parallelism/the-verifier.md

@@ -0,0 +1 @@
+# ⚪️ The verifier

books/architecture/src/resource-model/concurrency-and-parallelism/thread-exit.md

@@ -0,0 +1 @@
+# ⚪️ Thread exit

books/architecture/src/resource-model/file-system.md

@@ -0,0 +1 @@
+# ⚪️ File system

books/architecture/src/resource-model/intro.md

@@ -0,0 +1 @@
+# ⚪️ Resource model

books/architecture/src/resource-model/memory.md

@@ -0,0 +1 @@
+# ⚪️ Memory

books/architecture/src/resource-model/sockets.md

@@ -0,0 +1 @@
+# ⚪️ Sockets

books/architecture/src/rpc/handler.md

@@ -0,0 +1 @@
+# ⚪️ Handler

books/architecture/src/rpc/interface.md

@@ -0,0 +1 @@
+# ⚪️ Interface

books/architecture/src/rpc/intro.md

@@ -0,0 +1,3 @@
+# RPC
+- <https://www.getmonero.org/resources/developer-guides/daemon-rpc.html#other-daemon-rpc-calls>
+- <https://github.com/monero-project/monero/tree/master/src/rpc>

books/architecture/src/rpc/methods/intro.md

@@ -0,0 +1 @@
+# ⚪️ Methods

books/architecture/src/rpc/router.md

@@ -0,0 +1 @@
+# ⚪️ Router

books/architecture/src/rpc/types/binary.md

@@ -0,0 +1 @@
+# ⚪️ Binary

books/architecture/src/rpc/types/intro.md

@@ -0,0 +1 @@
+# ⚪️ Types

books/architecture/src/rpc/types/json.md

@@ -0,0 +1 @@
+# ⚪️ JSON

books/architecture/src/rpc/types/other.md

@@ -0,0 +1 @@
+# ⚪️ Other

books/architecture/src/storage/blockchain.md

@@ -0,0 +1 @@
+# ⚪️ Blockchain

books/architecture/src/storage/database-abstraction.md

@@ -0,0 +1 @@
+# ⚪️ Database abstraction

books/architecture/src/storage/intro.md

@@ -0,0 +1 @@
+# ⚪️ Storage

books/architecture/src/storage/pruning.md

@@ -0,0 +1 @@
+# ⚪️ Pruning

books/architecture/src/storage/transaction-pool.md

@@ -0,0 +1 @@
+# ⚪️ Transaction pool

books/architecture/src/testing/intro.md

@@ -0,0 +1 @@
+# ⚪️ Testing

books/architecture/src/testing/monero-data.md

@@ -0,0 +1 @@
+# ⚪️ Monero data

books/architecture/src/testing/rpc-client.md

@@ -0,0 +1 @@
+# ⚪️ RPC client

books/architecture/src/testing/spawning-monerod.md

@@ -0,0 +1 @@
+# ⚪️ Spawning monerod

books/architecture/src/zmq/intro.md

@@ -0,0 +1 @@
+# ⚪️ ZMQ