SyntheticBird/cuprate-hinto-janai
1
0
Fork 0
mirror of https://github.com/hinto-janai/cuprate.git synced 2024-11-16 15:58:14 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Merge branch 'main' into rpc-json-test
Some checks failed
Audit / audit (push) Has been cancelled
Details
Deny / audit (push) Has been cancelled
Details

Browse source
This commit is contained in:
hinto.janai 2024-07-19 17:53:41 -04:00
commit ed0997645f
No known key found for this signature in database
GPG key ID: D47CE05FA175A499
100 changed files with 960 additions and 61 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
books/architecture/README.md
View file

@ -1,4 +1,4 @@
## Cuprate's architecture (implementation) book
## Cuprate's architecture book
This book documents Cuprate's architecture and implementation.
See:

22
books/architecture/book.toml
View file

@ -1,19 +1,17 @@
[book]
authors = ["hinto-janai"]
authors = ["Cuprate Contributors"]
language = "en"
multilingual = false
src = "src"
title = "Cuprate Architecture"
git-repository-url = "https://github.com/Cuprate/architecture-book"
# TODO: fix after importing real files.
#
# [preprocessor.last-changed]
# command = "mdbook-last-changed"
# renderer = ["html"]
#
# [output.html]
# default-theme = "ayu"
# preferred-dark-theme = "ayu"
# git-repository-url = "https://github.com/hinto-janai/cuprate-architecture"
# additional-css = ["last-changed.css"]
[preprocessor.last-changed]
command = "mdbook-last-changed"
renderer = ["html"]
[output.html]
default-theme = "ayu"
preferred-dark-theme = "ayu"
git-repository-url = "https://github.com/Cuprate/architecture-book"
additional-css = ["last-changed.css"]

7
books/architecture/last-changed.css Normal file
View file

@ -0,0 +1,7 @@
footer {
font-size: 0.8em;
text-align: center;
border-top: 1px solid;
margin-top: 4%;
padding: 5px 0;
}

123
books/architecture/src/SUMMARY.md
View file

@ -1,3 +1,124 @@
# Summary
- [TODO](todo.md)
[Cuprate Architecture](cuprate-architecture.md)
[🟡 Foreword](foreword.md)
---
- [🟠 Intro](intro/intro.md)
- [🟡 Who this book is for](intro/who-this-book-is-for.md)
- [🔴 Required knowledge](intro/required-knowledge.md)
- [🔴 How to use this book](intro/how-to-use-this-book.md)
---
- [⚪️ Bird's eye view](birds-eye-view/intro.md)
- [⚪️ Map](birds-eye-view/map.md)
- [⚪️ Components](birds-eye-view/components.md)
---
- [⚪️ Formats, protocols, types](formats-protocols-types/intro.md)
- [⚪️ monero_serai](formats-protocols-types/monero-serai.md)
- [⚪️ cuprate_types](formats-protocols-types/cuprate-types.md)
- [⚪️ cuprate_helper](formats-protocols-types/cuprate-helper.md)
- [⚪️ Epee](formats-protocols-types/epee.md)
- [⚪️ Levin](formats-protocols-types/levin.md)
---
- [⚪️ Storage](storage/intro.md)
- [⚪️ Database abstraction](storage/database-abstraction.md)
- [⚪️ Blockchain](storage/blockchain.md)
- [⚪️ Transaction pool](storage/transaction-pool.md)
- [⚪️ Pruning](storage/pruning.md)
---
- [🔴 RPC](rpc/intro.md)
- [⚪️ Types](rpc/types/intro.md)
- [⚪️ JSON](rpc/types/json.md)
- [⚪️ Binary](rpc/types/binary.md)
- [⚪️ Other](rpc/types/other.md)
- [⚪️ Interface](rpc/interface.md)
- [⚪️ Router](rpc/router.md)
- [⚪️ Handler](rpc/handler.md)
- [⚪️ Methods](rpc/methods/intro.md)
---
- [⚪️ ZMQ](zmq/intro.md)
- [⚪️ TODO](zmq/todo.md)
---
- [⚪️ Consensus](consensus/intro.md)
- [⚪️ Verifier](consensus/verifier.md)
- [⚪️ TODO](consensus/todo.md)
---
- [⚪️ Networking](networking/intro.md)
- [⚪️ P2P](networking/p2p.md)
- [⚪️ Dandelion++](networking/dandelion.md)
- [⚪️ Proxy](networking/proxy.md)
- [⚪️ Tor](networking/tor.md)
- [⚪️ i2p](networking/i2p.md)
- [⚪️ IPv4/IPv6](networking/ipv4-ipv6.md)
---
- [🔴 Instrumentation](instrumentation/intro.md)
- [⚪️ Logging](instrumentation/logging.md)
- [⚪️ Data collection](instrumentation/data-collection.md)
---
- [⚪️ Binary](binary/intro.md)
- [⚪️ CLI](binary/cli.md)
- [⚪️ Config](binary/config.md)
- [⚪️ Logging](binary/logging.md)
---
- [⚪️ Resource model](resource-model/intro.md)
- [⚪️ File system](resource-model/file-system.md)
- [⚪️ Sockets](resource-model/sockets.md)
- [⚪️ Memory](resource-model/memory.md)
- [🟡 Concurrency and parallelism](resource-model/concurrency-and-parallelism/intro.md)
- [⚪️ Map](resource-model/concurrency-and-parallelism/map.md)
- [⚪️ The RPC server](resource-model/concurrency-and-parallelism/the-rpc-server.md)
- [⚪️ The database](resource-model/concurrency-and-parallelism/the-database.md)
- [⚪️ The block downloader](resource-model/concurrency-and-parallelism/the-block-downloader.md)
- [⚪️ The verifier](resource-model/concurrency-and-parallelism/the-verifier.md)
- [⚪️ Thread exit](resource-model/concurrency-and-parallelism/thread-exit.md)
---
- [⚪️ External Monero libraries](external-monero-libraries/intro.md)
- [⚪️ Cryptonight](external-monero-libraries/cryptonight.md)
- [🔴 RandomX](external-monero-libraries/randomx.md)
- [🔴 monero_serai](external-monero-libraries/monero_serai.md)
---
- [⚪️ Benchmarking](benchmarking/intro.md)
- [⚪️ Criterion](benchmarking/criterion.md)
- [⚪️ Harness](benchmarking/harness.md)
- [⚪️ Testing](testing/intro.md)
- [⚪️ Monero data](testing/monero-data.md)
- [⚪️ RPC client](testing/rpc-client.md)
- [⚪️ Spawning `monerod`](testing/spawning-monerod.md)
- [⚪️ Known issues and tradeoffs](known-issues-and-tradeoffs/intro.md)
- [⚪️ Networking](known-issues-and-tradeoffs/networking.md)
- [⚪️ RPC](known-issues-and-tradeoffs/rpc.md)
- [⚪️ Storage](known-issues-and-tradeoffs/storage.md)
---
- [⚪️ Appendix](appendix/intro.md)
- [🔴 Contributing](appendix/contributing.md)
- [🔴 Crate documentation](appendix/crate-documentation.md)
- [🔴 Build targets](appendix/build-targets.md)
- [🔴 Protocol book](appendix/protocol-book.md)
- [⚪️ User book](appendix/user-book.md)

7
books/architecture/src/appendix/build-targets.md Normal file
View file

@ -0,0 +1,7 @@
# Build targets
- x86
- ARM64
- Windows
- Linux
- macOS
- FreeBSD(?)

2
books/architecture/src/appendix/contributing.md Normal file
View file

@ -0,0 +1,2 @@
# Contributing
<https://github.com/Cuprate/cuprate/blob/main/CONTRIBUTING.md>

4
books/architecture/src/appendix/crate-documentation.md Normal file
View file

@ -0,0 +1,4 @@
# Crate documentation
```bash
cargo doc --package $CUPRATE_CRATE
```

1
books/architecture/src/appendix/intro.md Normal file
View file

@ -0,0 +1 @@
# Appendix

2
books/architecture/src/appendix/protocol-book.md Normal file
View file

@ -0,0 +1,2 @@
# Protocol book
<https://monero-book.cuprate.org>

1
books/architecture/src/appendix/user-book.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ User book

1
books/architecture/src/benchmarking/criterion.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Criterion

1
books/architecture/src/benchmarking/harness.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Harness

1
books/architecture/src/benchmarking/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Benchmarking

1
books/architecture/src/binary/cli.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ CLI

1
books/architecture/src/binary/config.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Config

1
books/architecture/src/binary/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Binary

1
books/architecture/src/binary/logging.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Logging

1
books/architecture/src/birds-eye-view/components.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Components

1
books/architecture/src/birds-eye-view/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Bird's eye view

1
books/architecture/src/birds-eye-view/map.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Map

1
books/architecture/src/consensus/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Consensus

1
books/architecture/src/consensus/todo.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ TODO

1
books/architecture/src/consensus/verifier.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Verifier

22
books/architecture/src/cuprate-architecture.md Normal file
View file

@ -0,0 +1,22 @@
# Cuprate Architecture
WIP
[Cuprate](https://github.com/Cuprate/cuprate)'s architecture book.
Sections are notated with colors indicating how complete they are:
| Color | Meaning |
|-------|---------|
| ⚪️ | Empty
| 🔴 | Severely lacking information
| 🟠 | Lacking some information
| 🟡 | Almost ready
| 🟢 | OK
---
Continue to the next chapter by clicking the right `>` button, or by selecting it on the left side.
All chapters are viewable by clicking the top-left `☰` button.
The entire book can searched by clicking the top-left 🔍 button.

1
books/architecture/src/external-monero-libraries/cryptonight.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Cryptonight

1
books/architecture/src/external-monero-libraries/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ External Monero libraries

2
books/architecture/src/external-monero-libraries/monero_serai.md Normal file
View file

@ -0,0 +1,2 @@
# monero_serai
<https://github.com/serai-dex/serai/tree/develop/coins/monero>

2
books/architecture/src/external-monero-libraries/randomx.md Normal file
View file

@ -0,0 +1,2 @@
# RandomX
<https://github.com/tari-project/randomx-rs>

36
books/architecture/src/foreword.md Normal file
View file

@ -0,0 +1,36 @@
# Foreword
Monero[^1] is a large software project, coming in at 329k lines of C++, C, headers, and make files.[^2] It is directly responsible for 2.6 billion dollars worth of value.[^3] It has had over 400 contributors, more if counting unnamed contributions.[^4] It has over 10,000 node operators and a large active userbase.[^5]
The project wasn't always this big, but somewhere in the midst of contributors coming and going, various features being added, bugs being fixed, and celebrated cryptography being implemented - there was an aspect that was lost by the project that it could not easily gain again: **maintainability**.
Within large and complicated software projects, there is an important transfer of knowledge that must occur for long-term survival. Much like an organism that must eventually pass the torch onto the next generation, projects must do the same for future contributors.
However, newcomers often lack experience, past contributors might not be around, and current maintainers may be too busy. For whatever reason, this transfer of knowledge is not always smooth.
There is a solution to this problem: **documentation**.
The activity of writing the what, where, why, and how of the solutions to technical problems can be done in an author's lonesome.
The activity of reading these ideas can be done by future readers at any time without permission.
These readers may be new prospective contributors, it may be the current maintainers, it may be researchers, it may be users of various scale. Whoever it may be, documentation acts as the link between the past and present; a bottle of wisdom thrown into the river of time for future participants to open.
This book is the manifestation of this will, for Cuprate[^6], an alternative Monero node. It documents Cuprate's implementation from head-to-toe such that in the case of a contributor's untimely disappearance, the project can continue.
People come and go, documentation is forever.
— hinto-janai
---
[^1]: [`monero-project/monero`](https://github.com/monero-project/monero)
[^2]: `git ls-files | grep "\.cpp$\|\.h$\|\.c$\|CMake" | xargs cat | wc -l` on [`cc73fe7`](https://github.com/monero-project/monero/tree/cc73fe71162d564ffda8e549b79a350bca53c454)
[^3]: 2024-05-24: $143.55 USD * 18,151,608 XMR = $2,605,663,258
[^4]: `git log --all --pretty="%an" | sort -u | wc -l` on [`cc73fe7`](https://github.com/monero-project/monero/tree/cc73fe71162d564ffda8e549b79a350bca53c454)
[^5]: <https://monero.fail/map>
[^6]: <https://github.com/Cuprate/cuprate>

1
books/architecture/src/formats-protocols-types/cuprate-helper.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ cuprate_helper

1
books/architecture/src/formats-protocols-types/cuprate-types.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ cuprate_types

1
books/architecture/src/formats-protocols-types/epee.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Epee

1
books/architecture/src/formats-protocols-types/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Formats, protocols, types

1
books/architecture/src/formats-protocols-types/levin.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Levin

1
books/architecture/src/formats-protocols-types/monero-serai.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ monero_serai

1
books/architecture/src/instrumentation/data-collection.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Data collection

2
books/architecture/src/instrumentation/intro.md Normal file
View file

@ -0,0 +1,2 @@
# Instrumentation
Cuprate is built with [instrumentation](https://en.wikipedia.org/wiki/Instrumentation) in mind.

1
books/architecture/src/instrumentation/logging.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Logging

4
books/architecture/src/intro.md Normal file
View file

@ -0,0 +1,4 @@
# Systems
Cuprate is made up of multiple distinct internal systems that work together.
This section provides informal specifications and implementation details about each.

5
books/architecture/src/intro/how-to-use-this-book.md Normal file
View file

@ -0,0 +1,5 @@
# How to use this book
## Maintainers
## Contributors
## Researchers

15
books/architecture/src/intro/intro.md Normal file
View file

@ -0,0 +1,15 @@
# Intro
[Cuprate](https://github.com/Cuprate/cuprate) is an alternative [Monero](https://getmonero.org) node implementation.
This book describes Cuprate's architecture, ranging from small things like database pruning to larger meta-components like the networking stack.
A brief overview of some aspects covered within this book:
- Component designs
- Implementation details
- File location and purpose
- Design decisions and tradeoffs
- Things in relation to `monerod`
- Dependency usage
## Source code
The source files for this book can be found on at: <https://github.com/Cuprate/architecture-book>.

28
books/architecture/src/intro/required-knowledge.md Normal file
View file

@ -0,0 +1,28 @@
# Required knowledge
## General
- Rust
- Monero
- System design
## Components
### Storage
- Embedded databases
- LMDB
- redb
### RPC
- `axum`
- `tower`
- `async`
- JSON-RPC 2.0
- Epee
### Networking
- `tower`
- `tokio`
- `async`
- Levin
### Instrumentation
- `tracing`

31
books/architecture/src/intro/who-this-book-is-for.md Normal file
View file

@ -0,0 +1,31 @@
# Who this book is for
## Maintainers
As mentioned in [`Foreword`](../foreword.md), the group of people that benefit from this book's value the most by far are the current and future Cuprate maintainers.
Cuprate's system design is documented in this book such that if you were ever to build it again from scratch, you would have an excellent guide on how to do such, and also where improvements could be made.
Practically, what that means for maintainers is that it acts as _the_ reference. During maintenance, it is quite valuable to have a book that contains condensed knowledge on the behavior of components, or how certain code works, or why it was built a certain way.
## Contributors
Contributors also have access to the inner-workings of Cuprate via this book, which helps when making larger contributions.
Design decisions and implementation details notated in this book helps answer questions such as:
- Why is it done this way?
- Why can it _not_ be done this way?
- Were other methods attempted?
Cuprate's testing and benchmarking suites, unknown to new contributors, are also documented within this book.
## Researchers
This book contains the why, where, and how of the _implementation_ of formal research.
Although it is an informal specification, this book still acts as a more accessible overview of Cuprate compared to examining the codebase itself.
## Operators & users
This book is not a practical guide for using Cuprate itself.
For configuration, data collection (also important for researchers), and other practical usage, see [Cuprate's user book](https://user.cuprate.org).
## Observers
Anyone curious enough is free to learn the inner-workings of Cuprate via this book, and maybe even contribute someday.

1
books/architecture/src/known-issues-and-tradeoffs/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Known issues and tradeoffs

1
books/architecture/src/known-issues-and-tradeoffs/networking.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Networking

1
books/architecture/src/known-issues-and-tradeoffs/rpc.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ RPC

1
books/architecture/src/known-issues-and-tradeoffs/storage.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Storage

1
books/architecture/src/networking/dandelion.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Dandelion++

1
books/architecture/src/networking/i2p.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ i2p

1
books/architecture/src/networking/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Networking

1
books/architecture/src/networking/ipv4-ipv6.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ IPv4/IPv6

1
books/architecture/src/networking/p2p.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ P2P

1
books/architecture/src/networking/proxy.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Proxy

1
books/architecture/src/networking/tor.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Tor

32
books/architecture/src/resource-model/concurrency-and-parallelism/intro.md Normal file
View file

@ -0,0 +1,32 @@
# Concurrency and parallelism
It is incumbent upon software like Cuprate to take advantage of today's highly parallel hardware as much as practically possible.
With that said, programs must setup guardrails when operating in a concurrent and parallel manner, [for correctness and safety](https://en.wikipedia.org/wiki/Concurrency_(computer_science)).
There are "synchronization primitives" that help with this, common ones being:
- [Locks](https://en.wikipedia.org/wiki/Lock_(computer_science))
- [Channels](https://en.wikipedia.org/wiki/Channel_(programming))
- [Atomics](https://en.wikipedia.org/wiki/Linearizability#Primitive_atomic_instructions)
These tools are relatively easy to use in isolation, but trickier to do so when considering the entire system. It is not uncommon for _the_ bottleneck to be the [poor orchastration](https://en.wikipedia.org/wiki/Starvation_(computer_science)) of these primitives.
## Analogy
A common analogy for a parallel system is an intersection.
Like a parallel computer system, an intersection contains:
1. **Parallelism:** multiple individual units that want to move around (cars, pedestrians, etc)
1. **Synchronization primitives:** traffic lights, car lights, walk signals
In theory, the amount of "work" the units can do is only limited by the speed of the units themselves, but in practice, the slow cascading reaction speeds between all units, the frequent hiccups that can occur, and the synchronization primitives themselves become bottlenecks far before the maximum speed of any unit is reached.
A car that hogs the middle of the intersection on the wrong light is akin to a system thread holding onto a lock longer than it should be - it degrades total system output.
Unlike humans however, computer systems at least have the potential to move at lightning speeds, but only if the above synchronization primitives are used correctly.
## Goal
To aid the long-term maintenance of highly concurrent and parallel code, this section documents:
1. All system threads spawned and maintained
1. All major sections where synchronization primitives are used
1. The asynchronous behavior of some components
and how these compose together efficiently in Cuprate.

1
books/architecture/src/resource-model/concurrency-and-parallelism/map.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Map

1
books/architecture/src/resource-model/concurrency-and-parallelism/the-block-downloader.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ The block downloader

1
books/architecture/src/resource-model/concurrency-and-parallelism/the-database.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ The database

1
books/architecture/src/resource-model/concurrency-and-parallelism/the-rpc-server.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ The RPC server

1
books/architecture/src/resource-model/concurrency-and-parallelism/the-verifier.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ The verifier

1
books/architecture/src/resource-model/concurrency-and-parallelism/thread-exit.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Thread exit

1
books/architecture/src/resource-model/file-system.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ File system

1
books/architecture/src/resource-model/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Resource model

1
books/architecture/src/resource-model/memory.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Memory

1
books/architecture/src/resource-model/sockets.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Sockets

1
books/architecture/src/rpc/handler.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Handler

1
books/architecture/src/rpc/interface.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Interface

3
books/architecture/src/rpc/intro.md Normal file
View file

@ -0,0 +1,3 @@
# RPC
- <https://www.getmonero.org/resources/developer-guides/daemon-rpc.html#other-daemon-rpc-calls>
- <https://github.com/monero-project/monero/tree/master/src/rpc>

1
books/architecture/src/rpc/methods/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Methods

1
books/architecture/src/rpc/router.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Router

1
books/architecture/src/rpc/types/binary.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Binary

1
books/architecture/src/rpc/types/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Types

1
books/architecture/src/rpc/types/json.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ JSON

1
books/architecture/src/rpc/types/other.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Other

1
books/architecture/src/storage/blockchain.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Blockchain

1
books/architecture/src/storage/database-abstraction.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Database abstraction

1
books/architecture/src/storage/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Storage

1
books/architecture/src/storage/pruning.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Pruning

1
books/architecture/src/storage/transaction-pool.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Transaction pool

1
books/architecture/src/testing/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Testing

1
books/architecture/src/testing/monero-data.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Monero data

1
books/architecture/src/testing/rpc-client.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ RPC client

1
books/architecture/src/testing/spawning-monerod.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ Spawning monerod

1
books/architecture/src/zmq/intro.md Normal file
View file

@ -0,0 +1 @@
# ⚪️ ZMQ

0
books/architecture/src/todo.md → books/architecture/src/zmq/todo.md
View file

11
books/protocol/src/SUMMARY.md
View file

@ -23,5 +23,14 @@
- [Bulletproofs+](./consensus_rules/transactions/ring_ct/bulletproofs+.md)
- [P2P Network](./p2p_network.md)
- [Levin Protocol](./p2p_network/levin.md)
- [P2P Messages](./p2p_network/messages.md)
- [Admin Messages](./p2p_network/levin/admin.md)
- [Protocol Messages](./p2p_network/levin/protocol.md)
- [Common Types](./p2p_network/common_types.md)
- [Message Flows](./p2p_network/message_flows.md)
- [Handshake](./p2p_network/message_flows/handshake.md)
- [Timed Sync](./p2p_network/message_flows/timed_sync.md)
- [New Block](./p2p_network/message_flows/new_block.md)
- [New Transactions](./p2p_network/message_flows/new_transactions.md)
- [Chain Sync](./p2p_network/message_flows/chain_sync.md)
- [Get Blocks](./p2p_network/message_flows/get_blocks.md)
- [Pruning](./pruning.md)

2
books/protocol/src/p2p_network.md
View file

@ -1,3 +1,3 @@
# P2P Network
This chapter contains descriptions of Monero's peer to peer network, including messages, flows, expected responses, etc.
This chapter contains descriptions of Monero's peer to peer network, including messages, flows, etc.

116
books/protocol/src/p2p_network/common_types.md Normal file
View file

@ -0,0 +1,116 @@
# Common P2P Types
This chapter contains definitions of types used in multiple P2P messages.
### Support Flags
Support flags specify any protocol extensions the peer supports, currently only the first bit is used:
`FLUFFY_BLOCKS = 1` - for if the peer supports receiving fluffy blocks.
### Basic Node Data [^b-n-d] { #basic-node-data }
| Fields | Type | Description |
|------------------------|---------------------------------------|-------------------------------------------------------------------------------------------|
| `network_id` | A UUID (epee string) | A fixed constant value for a specific network (mainnet,testnet,stagenet) |
| `my_port` | u32 | The peer's inbound port, if the peer does not want inbound connections this should be `0` |
| `rpc_port` | u16 | The peer's RPC port, if the peer does not want inbound connections this should be `0` |
| `rpc_credits_per_hash` | u32 | States how much it costs to use this node in credits per hashes, `0` being free |
| `peer_id` | u64 | A fixed ID for the node, set to 1 for anonymity networks |
| `support_flags` | [support flags](#support-flags) (u32) | Specifies any protocol extensions the peer supports |
### Core Sync Data [^c-s-d] { #core-sync-data }
| Fields | Type | Description |
|-------------------------------|------------------------|---------------------------------------------------------------|
| `current_height` | u64 | The current chain height |
| `cumulative_difficulty` | u64 | The low 64 bits of the cumulative difficulty |
| `cumulative_difficulty_top64` | u64 | The high 64 bits of the cumulative difficulty |
| `top_id` | [u8; 32] (epee string) | The hash of the top block |
| `top_version` | u8 | The hardfork version of the top block |
| `pruning_seed` | u32 | THe pruning seed of the node, `0` if the node does no pruning |
### Network Address [^network-addr] { #network-address }
Network addresses are serialized differently than other types, the fields needed depend on the `type` field:
| Fields | Type | Description |
| ------ | --------------------------------------- | ---------------- |
| `type` | u8 | The address type |
| `addr` | An object whose fields depend on `type` | The address |
#### IPv4
`type = 1`
| Fields | Type | Description |
| -------- | ---- | ---------------- |
| `m_ip` | u32 | The IPv4 address |
| `m_port` | u16 | The port |
#### IPv6
`type = 2`
| Fields | Type | Description |
| -------- | ---------------------- | ---------------- |
| `addr` | [u8; 16] (epee string) | The IPv6 address |
| `m_port` | u16 | The port |
#### Tor
TODO:
#### I2p
TODO:
### Peer List Entry Base [^pl-entry-base] { #peer-list-entry-base }
| Fields | Type | Description |
|------------------------|-------------------------------------|-------------------------------------------------------------------------------------------------------|
| `adr` | [Network Address](#network-address) | The address of the peer |
| `id` | u64 | The random, self assigned, ID of this node |
| `last_seen` | i64 | A field marking when this peer was last seen, although this is zeroed before sending over the network |
| `pruning_seed` | u32 | This peer's pruning seed, `0` if the peer does no pruning |
| `rpc_port` | u16 | This node's RPC port, `0` if this peer has no public RPC port. |
| `rpc_credits_per_hash` | u32 | States how much it costs to use this node in credits per hashes, `0` being free |
### Tx Blob Entry [^tb-entry] { #tx-blob-entry }
| Fields | Type | Description |
| --------------- | ---------------------- | --------------------------------------- |
| `blob` | bytes (epee string) | The pruned tx blob |
| `prunable_hash` | [u8; 32] (epee string) | The hash of the prunable part of the tx |
### Block Complete Entry [^bc-entry] { #block-complete-entry }
| Fields | Type | Description |
|----------------|---------------------|-----------------------------------------------------------|
| `pruned` | bool | True if the block is pruned, false otherwise |
| `block` | bytes (epee string) | The block blob |
| `block_weight` | u64 | The block's weight |
| `txs` | depends on `pruned` | The transaction blobs, the exact type depends on `pruned` |
If `pruned` is true:
`txs` is a vector of [Tx Blob Entry](#tx-blob-entry)
If `pruned` is false:
`txs` is a vector of bytes.
---
[^b-n-d]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L185>
[^c-s-d]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_defs.h#L250>
[^network-addr]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/contrib/epee/include/net/net_utils_base.h#L320>
[^pl-entry-base]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L72>
[^tb-entry]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_defs.h#L121>
[^bc-entry]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_defs.h#L132>

3
books/protocol/src/p2p_network/epee.md
View file

@ -1,3 +0,0 @@
# Epee Binary Format
The epee binary format is described here: TODO

24
books/protocol/src/p2p_network/levin.md
View file

@ -10,16 +10,16 @@ of buckets that will be combined into a single message.
### Bucket Format
| Field | Type | Size (bytes) |
| ------ | ----------------------------- | ------------ |
|--------|-------------------------------|--------------|
| Header | [BucketHeader](#bucketheader) | 33 |
| Body | bytes | dynamic |
### BucketHeader
Format:
Format[^header-format]:
| Field | Type | Size (bytes) |
| ---------------- | ------ | ------------ |
|------------------|--------|--------------|
| Signature | LE u64 | 8 |
| Size | LE u64 | 8 |
| Expect Response | bool | 1 |
@ -32,7 +32,7 @@ Format:
The signature field is fixed for every bucket and is used to tell apart peers running different protocols.
Its value should be `0x0101010101012101`
Its value should be `0x0101010101012101` [^signature]
#### Size
@ -53,7 +53,7 @@ responses should be `1`.
#### Flags
This is a bit-flag field that determines what type of bucket this is:
This is a bit-flag field that determines what type of bucket this is[^flags]:
| Type | Bits set |
| -------------- | ----------- |
@ -66,3 +66,17 @@ This is a bit-flag field that determines what type of bucket this is:
#### Protocol Version
This is a fixed value of 1.
## Bucket Body
All bucket bodies are serialized in the epee binary format which is described here: https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/docs/PORTABLE_STORAGE.md
Exact message types are described in the next chapters.
---
[^header-format]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/contrib/epee/include/net/levin_base.h#L62>
[^signature]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/contrib/epee/include/net/levin_base.h#L38>
[^flags]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/contrib/epee/include/net/levin_base.h#L79-L82>

102
books/protocol/src/p2p_network/levin/admin.md Normal file
View file

@ -0,0 +1,102 @@
# Admin Messages
This chapter describes admin messages, and documents the current admin messages. Admin messages are a subset of messages that handle connection
creation, making sure connections are still alive, and sharing peer lists.
## Levin
All admin messages are in the request/response levin format. This means requests will set the [expect response bit](./levin.md#expect-response) and
responses will set the return code to [`1`](./levin.md#return-code).
## Messages
### Handshake
ID: `1001`[^handshake-id]
#### Request [^handshake-req] { #handshake-request }
| Fields | Type | Description |
|----------------|-------------------------------------------------------|--------------------------------------|
| `node_data` | [basic node data](../common_types.md#basic-node-data) | Static information about our node |
| `payload_data` | [core sync data](../common_types.md#core-sync-data) | Information on the node's sync state |
#### Response [^handshake-res] { #handshake-response }
| Fields | Type | Description |
|----------------------|--------------------------------------------------------------------------|-----------------------------------------|
| `node_data` | [basic node data](../common_types.md#basic-node-data) | Static information about our node |
| `payload_data` | [core sync data](../common_types.md#core-sync-data) | Information on the node's sync state |
| `local_peerlist_new` | A Vec of [peer list entry base](../common_types.md#peer-list-entry-base) | A list of peers in the node's peer list |
### Timed Sync
ID: `1002`[^timed-sync-id]
#### Request [^timed-sync-req] { #timed-sync-request }
| Fields | Type | Description |
| -------------- | --------------------------------------------------- | ------------------------------------ |
| `payload_data` | [core sync data](../common_types.md#core-sync-data) | Information on the node's sync state |
#### Response [^timed-sync-res] { #timed-sync-response }
| Fields | Type | Description |
|----------------------|--------------------------------------------------------------------------|-----------------------------------------|
| `payload_data` | [core sync data](../common_types.md#core-sync-data) | Information on the node's sync state |
| `local_peerlist_new` | A Vec of [peer list entry base](../common_types.md#peer-list-entry-base) | A list of peers in the node's peer list |
### Ping
ID: `1003`[^ping-id]
#### Request [^ping-req] { #ping-request }
No data is serialized for a ping request.
#### Response [^ping-res] { #ping-response }
| Fields | Type | Description |
| --------- | ------ | --------------------------------- |
| `status` | string | Will be `OK` for successful pings |
| `peer_id` | u64 | The self assigned id of the peer |
### Request Support Flags
ID: `1007`[^support-flags]
#### Request [^sf-req] { #support-flags-request }
No data is serialized for a ping request.
#### Response [^sf-res] { #support-flags-response }
| Fields | Type | Description |
| --------------- | ---- | ------------------------------------------------------------ |
| `support_flags` | u32 | The peer's [support flags](../common_types.md#support-flags) |
---
[^handshake-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L213>
[^handshake-req]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L215>
[^handshake-res]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L227>
[^timed-sync-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L249>
[^timed-sync-req]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L251>
[^timed-sync-res]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L260>
[^ping-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L284>
[^ping-req]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L288>
[^ping-res]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L297>
[^support-flags]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L316>
[^sf-req]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L318>
[^sf-res]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/p2p_protocol_defs.h#L325>

121
books/protocol/src/p2p_network/levin/protocol.md Normal file
View file

@ -0,0 +1,121 @@
# Protocol Messages
This chapter describes protocol messages, and documents the current protocol messages. Protocol messages are used to share protocol data
like blocks and transactions.
## Levin
All protocol messages are in the notification levin format. Although there are some messages that fall under requests/responses, levin will treat them as notifications.
All admin messages are in the request/response levin format. This means requests will set the [expect response bit](../levin.md#expect-response) and
responses will set the return code to [`1`](../levin.md#return-code).
## Messages
### Notify New Block
ID: `2001`[^notify-new-block-id]
| Fields | Type | Description |
| --------------------------- | --------------------------------------------------------------- | ------------------------ |
| `b` | [Block Complete Entry](../common_types.md#block-complete-entry) | The full block |
| `current_blockchain_height` | u64 | The current chain height |
### Notify New Transactions
ID: `2002`[^notify-new-transactions-id]
| Fields | Type | Description |
| ------------------- | ----------------- | ------------------------------------------------------ |
| `txs` | A vector of bytes | The txs |
| `_` | Bytes | Padding to prevent traffic volume analysis |
| `dandelionpp_fluff` | bool | True if this message contains fluff txs, false if stem |
### Notify Request Get Objects
ID: `2003`[^notify-request-get-objects-id]
| Fields | Type | Description |
|----------|----------------------------------------------------|------------------------------------------------------------|
| `blocks` | A vector of [u8; 32] serialized as a single string | The block IDs requested |
| `prune` | bool | True if we want the blocks in pruned form, false otherwise |
### Notify Response Get Objects
ID: `2004`[^notify-response-get-objects-id]
| Fields | Type | Description |
| --------------------------- | --------------------------------------------------------------------------- | ------------------------------ |
| `blocks` | A vector of [Block Complete Entry](../common_types.md#block-complete-entry) | The blocks that were requested |
| `missed_ids` | A vector of [u8; 32] serialized as a single string | IDs of any missed blocks |
| `current_blockchain_height` | u64 | The current blockchain height |
### Notify Request Chain
ID: `2006`[^notify-request-chain-id]
| Fields | Type | Description |
|-------------|----------------------------------------------------|-------------------------------------------------------------------------------------------------------|
| `block_ids` | A vector of [u8; 32] serialized as a single string | A list of block IDs in reverse chronological order, the top and genesis block will always be included |
| `prune` | bool | True if we want the response to contain pruned blocks, false otherwise |
### Notify Response Chain Entry
ID: `2007`[^notify-response-chain-entry-id]
| Fields | Type | Description |
|-------------------------------|----------------------------------------------------|------------------------------------------------|
| `start_height` | u64 | The start height of the entry |
| `total_height` | u64 | The height of the peer's blockchain |
| `cumulative_difficulty` | u64 | The low 64 bits of the cumulative difficulty |
| `cumulative_difficulty_top64` | u64 | The high 64 bits of the cumulative difficulty |
| `m_block_ids` | A vector of [u8; 32] serialized as a single string | The block IDs in this entry |
| `m_block_weights` | A vector of u64 serialized as a single string | The block weights |
| `first_block` | bytes (epee string) | The header of the first block in `m_block_ids` |
### Notify New Fluffy Block
ID: `2008`[^notify-new-fluffy-block-id]
| Fields | Type | Description |
| --------------------------- | --------------------------------------------------------------- | ------------------------------------- |
| `b` | [Block Complete Entry](../common_types.md#block-complete-entry) | The block, may or may not contain txs |
| `current_blockchain_height` | u64 | The current chain height |
### Notify Request Fluffy Missing Tx
ID: `2009`[^notify-request-fluffy-missing-tx-id]
| Fields | Type | Description |
|-----------------------------|-----------------------------------------------|--------------------------------------------|
| `block_hash` | [u8; 32] serialized as a string | The block hash txs are needed from |
| `current_blockchain_height` | u64 | The current chain height |
| `missing_tx_indices` | A vector of u64 serialized as a single string | The indices of the needed txs in the block |
### Notify Get Txpool Compliment
ID: `2010`[^notify-get-txpool-compliment-id]
| Fields | Type | Description |
| -------- | ------------------------------------------- | ---------------------- |
| `hashes` | A vector of [u8; 32] serialized as a string | The current txpool txs |
---
[^notify-new-block-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_defs.h#L174>
[^notify-new-transactions-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_defs.h#L194>
[^notify-request-get-objects-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_defs.h#L215>
[^notify-response-get-objects-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_defs.h#L232>
[^notify-request-chain-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_defs.h#L274>
[^notify-response-chain-entry-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_defs.h#L291>
[^notify-new-fluffy-block-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_defs.h#L324>
[^notify-request-fluffy-missing-tx-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_defs.h#L344>
[^notify-get-txpool-compliment-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_defs.h#L366>

19
books/protocol/src/p2p_network/message_flows.md Normal file
View file

@ -0,0 +1,19 @@
# Message Flows
Message flows are sets of messages sent between peers, that achieve an identifiable goal, like a handshake.
Some message flows are complex, involving many message types, whereas others are simple, requiring only 1.
The message flows here are not every possible request/response.
When documenting checks on the messages, not all checks are documented, only the ones notable. This should help
to reduce the maintenance burden.
## Different Flows
- [Handshakes](./message_flows/handshake.md)
- [Timed Sync](./message_flows/timed_sync.md)
- [New Block](./message_flows/new_block.md)
- [New Transactions](./message_flows/new_transactions.md)
- [Chain Sync](./message_flows/chain_sync.md)
- [Get Blocks](./message_flows/get_blocks.md)

28
books/protocol/src/p2p_network/message_flows/chain_sync.md Normal file
View file

@ -0,0 +1,28 @@
# Chain Sync
Chain sync is the first step in syncing a peer's blockchain, it allows a peers to find the split point in their chains and for the peer
to learn about the missing block IDs.
## Flow
The first step is for the initiating peer is to get its compact chain history. The compact chain history must be in reverse chronological
order, with the first block being the top block and the last the genesis, if the only block is the genesis then that only needs to be included
once. The blocks in the middle are not enforced to be at certain locations, however `monerod` will use the top 11 blocks and will then go power
of 2 offsets from then on, i.e. `{13, 17, 25, ...}`
Then, with the compact history, the initiating peer will send a [request chain](../levin/protocol.md#notify-request-chain) message, the receiving
peer will then find the split point and return a [response chain entry](../levin/protocol.md#notify-response-chain-entry) message.
The `response chain entry` will contain a list of block IDs with the first being a common ancestor and the rest being the next blocks that come after
that block in the peer's chain.
### Response Checks
- There must be an overlapping block.[^res-overlapping-block]
- The amount of returned block IDs must be less than `25,000`.[^res-max-blocks]
---
[^res-overlapping-block]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_handler.inl#L2568>
[^res-max-blocks]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_handler.inl#L2599>

19
books/protocol/src/p2p_network/message_flows/get_blocks.md Normal file
View file

@ -0,0 +1,19 @@
# Get Blocks
The get block flow is used to download batches of blocks from a peer.
## Flow
The initiating peer needs a list of block IDs that the receiving peer has, this can be done with
the [chain sync flow](./chain_sync.md).
With a list a block IDs the initiating peer will send a [get objects request](../levin/protocol.md#notify-request-get-objects) message, the receiving
peer will then respond with [get objects response](../levin/protocol.md#notify-response-get-objects).
### Request Checks
- The amount of blocks must be less than `100`.[^max-block-requests]
---
[^max-block-requests]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_handler.inl#L1089>

51
books/protocol/src/p2p_network/message_flows/handshake.md Normal file
View file

@ -0,0 +1,51 @@
# Handshakes
Handshakes are used to establish connections to peers.
## Flow
The default handshake flow is made up of the connecting peer sending a [handshake request](../levin/admin.md#handshake-request) and the
receiving peer responding with a [handshake response](../levin/admin.md#handshake-response).
It should be noted that not all other messages are banned during handshakes, for example, support flag requests and even some protocol
requests can be sent.
### Handshake Request Checks
The receiving peer will check:
- The `network_id` is network ID expected.[^network-id]
- The connection is an incoming connection.[^req-incoming-only]
- The peer hasn't already completed a handshake.[^double-handshake]
- If the network zone is public, then the `peer_id` must not be the same as ours.[^same-peer-id]
- The core sync data is not malformed.[^core-sync-data-checks]
### Handshake Response Checks
The initiating peer will check:
- The `network_id` is network ID expected.[^res-network-id]
- The number of peers in the peer list is less than `250`.[^max-peer-list-res]
- All peers in the peer list are in the same zone.[^peers-all-in-same-zone]
- The core sync data is not malformed.[^core-sync-data-checks]
- If the network zone is public, then the `peer_id` must not be the same as ours.[^same-peer-id-res]
---
[^network-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/net_node.inl#L2510>
[^req-incoming-only]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/net_node.inl#L2519>
[^double-handshake]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/net_node.inl#L2527>
[^same-peer-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/net_node.inl#L2539>
[^core-sync-data-checks]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_handler.inl#L341>
[^res-network-id]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/net_node.inl#L1164>
[^max-peer-list-res]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/net_node.inl#L2170>
[^peers-all-in-same-zone]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/net_node.inl#L2182>
[^same-peer-id-res]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/net_node.inl#L1195>

29
books/protocol/src/p2p_network/message_flows/new_block.md Normal file
View file

@ -0,0 +1,29 @@
# New Block
This is used whenever a new block is to be sent to peers. Only the fluffy block flow is described here, as the other method is deprecated.
## Flow
First the peer with the new block will send a [new fluffy block](../levin/protocol.md#notify-new-fluffy-block) notification, if the receiving
peer has all the txs in the block then the flow is complete. Otherwise the peer sends a [fluffy missing transactions request](../levin/protocol.md#notify-request-fluffy-missing-tx)
to the first peer, the first peer will then respond with again a [new fluffy block](../levin/protocol.md#notify-new-fluffy-block) notification but
with the transactions requested.
```bob
,-----------. ,----------.
| Initiator | | Receiver |
`-----+-----' `-----+----'
| New Fluffy Block |
|-------------------->|
| |
| Missing Txs Request |
|<- - - - - - - - - - |
| |
| New Fluffy Block |
| - - - - - - - - - ->|
| |
| |
V v
```

16
books/protocol/src/p2p_network/message_flows/new_transactions.md Normal file
View file

@ -0,0 +1,16 @@
# New Transactions
Monero uses the dandelion++ protocol to pass transactions around the network, this flow just describes the actual tx passing between nodes part.
## Flow
This flow is pretty simple, the txs are put into a [new transactions](../levin/protocol.md#notify-new-transactions) notification and sent to
peers.
Hopefully in the future [this is changed](https://github.com/monero-project/monero/issues/9334).
There must be no duplicate txs in the notification.[^duplicate-txs]
---
[^duplicate-txs]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_protocol/cryptonote_protocol_handler.inl#L991>

28
books/protocol/src/p2p_network/message_flows/timed_sync.md Normal file
View file

@ -0,0 +1,28 @@
# Timed Syncs
A timed sync request is sent every 60 seconds to make sure the connection is still live.
## Flow
First the timed sync initiator will send a [timed sync request](../levin/admin.md#timed-sync-request), the receiver will then
respond with a [timed sync response](../levin/admin.md#timed-sync-response)
### Timed Sync Request Checks
- The core sync data is not malformed.[^core-sync-data-checks]
### Timed Sync Response Checks
- The core sync data is not malformed.[^core-sync-data-checks]
- The number of peers in the peer list is less than `250`.[^max-peer-list-res]
- All peers in the peer list are in the same zone.[^peers-all-in-same-zone]
---
[^core-sync-data-checks]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/net_node.inl#L2464>
[^max-peer-list-res]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/net_node.inl#L2170>
[^peers-all-in-same-zone]: <https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/p2p/net_node.inl#L2182>

37
books/protocol/src/p2p_network/messages.md
View file

@ -1,37 +0,0 @@
# P2P Messages
This chapter contains every P2P message.
## Index
## Types
Types used in multiple P2P messages.
### Support Flags
Support flags specify any protocol extensions the peer supports, currently only the first bit is used:
`FLUFFY_BLOCKS = 1` - for if the peer supports receiving fluffy blocks.
### Basic Node Data
| Fields | Type (Epee Type) | Description |
| ---------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------- |
| `network_id` | A UUID (String) | A fixed constant value for a specific network (mainnet,testnet,stagenet) |
| `my_port` | u32 (u32) | The peer's inbound port, if the peer does not want inbound connections this should be `0` |
| `rpc_port` | u16 (u16) | The peer's RPC port, if the peer does not want inbound connections this should be `0` |
| `rpc_credits_per_hash` | u32 (u32) | TODO |
| `peer_id` | u64 (u64) | A fixed ID for the node, set to 1 for anonymity networks |
| `support_flags` | [support flags](#support-flags) (u32) | Specifies any protocol extensions the peer supports |
## Messages
### Handshake Requests
levin command: 1001
| Fields | Type (Epee Type) | Description |
| ----------- | -------------------------------------------- | ----------- |
| `node_data` | [basic node data](#basic-node-data) (Object) | |
| | | |