Commit graph

43 commits

Author SHA1 Message Date
NoodleDoodleNoodleDoodleNoodleDoodleNoo
2e293a563e Fixed binary size issue due to embedded checkpoint data.
Fixed OSX compilation issues due to random lmdb resize points.
Fixed infinite loop bug when calling core::get_block_template(..).
2015-07-15 23:20:20 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
e5d2680094 ** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY)
Bockchain:
1. Optim: Multi-thread long-hash computation when encountering groups of blocks.
2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible.
3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible.
4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks.
5. Optim: Multi-thread signature computation whenever possible.
6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD)
7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???).
8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads).

Berkeley-DB:
1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc).
2. Fix: Unable to pop blocks on reorganize due to transaction errors.
3. Patch: Large number of transaction aborts when running multi-threaded bulk queries.
4. Patch: Insufficient locks error when running full sync.
5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation.
6. Optim: Add bulk queries to get output global indices.
7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
9. Optim: Added thread-safe buffers used when multi-threading bulk queries.
10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details)
11. Mod: Added checkpoint thread and auto-remove-logs option.
12. *Now usable on 32-bit systems like RPI2.

LMDB:
1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect)
2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details)
5. Mod: Auto resize to +1GB instead of multiplier x1.5

ETC:
1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete.
2. Fix: 32-bit saturation bug when computing next difficulty on large blocks.

[PENDING ISSUES]
1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization.
   This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD.
2. Berkeley db, possible bug "unable to allocate memory". TBD.

[NEW OPTIONS] (*Currently all enabled for testing purposes)
1. --fast-block-sync arg=[0:1] (default: 1)
	a. 0 = Compute long hash per block (may take a while depending on CPU)
	b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence)
2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000)
	a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions.
	b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache.
	Fast    - Write meta-data but defer data flush.
	Fastest - Defer meta-data and data flush.
	Sync    - Flush data after nblocks_per_sync and wait.
	Async   - Flush data after nblocks_per_sync but do not wait for the operation to finish.
3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower)
        Max number of threads to use when computing long-hash in groups.
4. --show-time-stats arg=[0:1] (default: 1)
	Show benchmark related time stats.
5. --db-auto-remove-logs arg=[0:1] (default: 1)
	For berkeley-db only. Auto remove logs if enabled.

**Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version.
	At the moment, you need a full resync to use this optimized version.

[PERFORMANCE COMPARISON]
**Some figures are approximations only.
Using a baseline machine of an i7-2600K+SSD+(with full pow computation):
1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain.
2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain.
3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain.

Averate procesing times (with full pow computation):
lmdb-optimized:
1. tx_ave = 2.5 ms / tx
2. block_ave = 5.87 ms / block
memory-official-repo:
1. tx_ave = 8.85 ms / tx
2. block_ave = 19.68 ms / block
lmdb-official-repo (0f4a036437)
1. tx_ave = 47.8 ms / tx
2. block_ave = 64.2 ms / block

**Note: The following data denotes processing times only (does not include p2p download time)
lmdb-optimized processing times (with full pow computation):
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000).
2. Laptop,   Dual-core / 4-threads U4200  (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000).
3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000).

lmdb-optimized processing times (with per-block-checkpoint)
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with full pow computation)
1. Desktop, Quad-core / 8-threads 2600k  (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000).
2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with per-block-checkpoint)
1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-15 23:20:16 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
5d304cabfd Fix loop bug when calling core::get_block_template, causing calling thread to lock up. 2015-07-10 22:09:21 -07:00
Thomas Winget
b1d92bcc37
Fixes changes to sort tx by fee per kb 2015-05-13 20:27:06 -04:00
Riccardo Spagni
012164fff8
resolved merge conflict in tx_pool.cpp 2015-05-13 11:18:22 +02:00
Thomas Winget
385d7c0495
Sort txs by per-kb-fee for miners 2015-04-30 01:02:12 -04:00
Thomas Winget
1b2614ba83
When removing 'stuck' transactions, don't ignore the first tx in the pool 2015-04-30 00:23:00 -04:00
warptangent
641d824f37
Keep memory pool consistent when stuck tx removed
When a stuck tx is removed from memory pool, first remove the associated
spent key images.
2015-04-23 07:04:36 -07:00
warptangent
b76857f9d9
Add mempool output to daemon via command and RPC
This is for the "print_pool" command and "get_transaction_pool" RPC
method.

Add mempool's spent key images to the results.
2015-04-23 07:04:36 -07:00
warptangent
84fe5fbd65
Add compile-time support for both db implementations: in-memory and LMDB
Usage:

default is lmdb for blockchain branch:
$ make release

same as:
$ DATABASE=lmdb make release

for original in-memory implementation:
$ DATABASE=memory make release
2015-02-02 11:53:09 -08:00
Thomas Winget
74a1a89e27
Integrate BlockchainDB into cryptonote_core
Probably needs more looking at -- lot of things were done...in a rushed
sort of way.  That said, it all builds and *should* be at least
testable.

update for rebase (warptangent 2015-01-04)
  fix conflicts with upstream CMakeLists.txt files

  src/CMakeLists.txt (remove edits from original commit)
  tests/CMakeLists.txt (remove edits from original commit)
  src/cryptonote_core/CMakeLists.txt (edit)
  - use blockchain db .cpp and .h files
  - add LMDB_LIBRARIES
2015-01-04 19:29:51 -08:00
Riccardo Spagni
f4b69d553a
year updated in license 2015-01-02 18:52:46 +02:00
iamsmooth
dde7897fd4 Disable legacy fees for now 2014-11-11 02:03:00 +00:00
iamsmooth
cc74b43651 Remove DEFAULT_FEE, add temporary acceptance of too-small per-kb fee >= 0.1, denominations based on DEFAULT_DUST_THRESHOLD, document fee arg to create_transactions as unused, se DEFAULT_DUST_THRESHOLD for wallet dust collection instead of calcualted tx fee 2014-11-09 10:53:29 +00:00
Thomas Winget
557e27fd21 per kb fees 2014-11-06 22:09:07 -07:00
Riccardo Spagni
59a8366bb1 Revert "low risk, potentially varint overflow bug patched thanks to BBR"
This reverts commit 4e2b2b942d.
2014-09-25 08:24:42 +02:00
Riccardo Spagni
4e2b2b942d low risk, potentially varint overflow bug patched thanks to BBR 2014-09-24 22:17:33 -04:00
fluffypony
4ba680f294 a few more error messages moved to log level 1 2014-09-09 12:28:16 +02:00
fluffypony
cf5a8b1d6c moved non-critical warnings and errors to log level 1 2014-09-09 11:32:00 +02:00
Riccardo Spagni
d1b0ee7ecd Merge pull request #64 from mikezackles/bytecoin_tx_pool_tmp
tx pool fixes, courtesy of Bytecoin
2014-08-01 17:31:15 +02:00
fluffypony
6fc995fe5d License updated to BSD 3-clause 2014-07-23 15:03:52 +02:00
Zachary Michaels
61e447f7c8 Remove dead code 2014-07-17 19:18:24 -04:00
Zachary Michaels
e48cf2a3a9 Remove second tx size check 2014-07-17 18:47:18 -04:00
Zachary Michaels
23c914ccfe Don't check max tx size when adding to block
This is now done when adding transactions to the pool.
2014-07-17 17:11:21 -04:00
Zachary Michaels
e3b8c58496 Don't check for min fee when adding tx to block
This is now done when transactions enter the pool.
2014-07-17 17:10:35 -04:00
Zachary Michaels
232e23e90f Decrease max block size from 200% median to 130% 2014-07-17 16:56:31 -04:00
Zachary Michaels
c017bb0656 Use print_money in log 2014-07-17 16:56:28 -04:00
Zachary Michaels
d9f3421ce7 Minimum tx fee for entering pool 2014-07-17 16:55:40 -04:00
Zachary Michaels
8e99cee062 Start fresh if tx_pool deserialize fails 2014-07-17 16:53:48 -04:00
Zachary Michaels
a3f5764869 Add transaction size limit
Transactions larger than the limit will not enter the pool
2014-07-17 16:53:48 -04:00
Zachary Michaels
4d25350a82 Use const where appropriate in tx_pool 2014-07-17 16:31:31 -04:00
mydesktop
eba62452c0 set max block size for gbt 2014-06-26 16:42:24 -04:00
monero-project
2a2b36d425 fix function typo 2014-06-24 13:30:41 -04:00
monero-project
c050ff43bf Fix copy/paste bug, thx Boolberry 2014-06-24 13:20:15 -04:00
fluffypony
3bc16dc0e6 proper tx_pool handling from CryptoZoidberg / BBR 2014-06-15 09:48:13 +02:00
monero-project
2475ec877f typo in tx_pool.cpp 2014-05-28 13:19:24 -04:00
mydesktop
9761529941 fix typing mismatch 2014-05-27 16:45:50 -04:00
mydesktop
328a52a543 dynamic tx size scaling for tx mempool 2014-05-26 20:57:50 -04:00
mydesktop
f545fd8ff0 maximum block size 130% of median 2014-05-26 19:51:22 -04:00
mydesktop
31a59785b0 temporary fix for block reward dos 2014-05-25 20:25:37 -04:00
thankful_for_today
0fd82c910b mining bug fixed 2014-04-20 00:53:40 +04:00
Antonio Juarez
29c2859a3e json rpc for wallet and bugfix 2014-04-02 17:00:17 +01:00
Antonio Juarez
296ae46ed8 moved all stuff to github 2014-03-03 22:07:58 +00:00