diff --git a/publications/standards/bulletproofs.bib b/publications/standards/bulletproofs.bib new file mode 100644 index 0000000..bb700f3 --- /dev/null +++ b/publications/standards/bulletproofs.bib @@ -0,0 +1,7 @@ +@misc{bp, + author = {Benedikt B\"unz and Jonathan Bootle and Dan Boneh and Andrew Poelstra and Pieter Wuille and Greg Maxwell}, + title = {Bulletproofs: Efficient Range Proofs for Confidential Transactions}, + howpublished = {Cryptology ePrint Archive, Report 2017/1066}, + year = {2017}, + note = {\url{https://eprint.iacr.org/2017/1066}}, +} diff --git a/publications/standards/bulletproofs.tex b/publications/standards/bulletproofs.tex new file mode 100644 index 0000000..399575f --- /dev/null +++ b/publications/standards/bulletproofs.tex @@ -0,0 +1,105 @@ +\documentclass{mrl} + +\title{Application of Bulletproofs in Monero Transactions} +\authors{Sarang Noether\footnote{\texttt{sarang.noether@protonmail.com}}} +\affiliations{Monero Research Lab} +\date{\today} + +\type{TECHNICAL NOTE} +\ident{MRL-XXXX} + +\begin{document} + +\begin{abstract} +This technical note briefly describes the proposed application of Bulletproofs \cite{bp} in Monero. The proofs are used as a drop-in replacement of the existing Borromean bitwise non-interactive zero-knowledge range proofs used to show that a committed amount is in a specified range. Bulletproofs reduce both proof size and verification time, as well as provide a straightforward method for batch verification of proofs from multiple transactions. We describe our implementation, noting specific areas of optimization from the original paper. +\end{abstract} + +\section{Introduction} +The implementation of confidential transaction amounts in Monero is accomplished using homomorphic commitments. Each input and output amount, including fees, is represented by a commitment of the form $vG + \mu H$, where $G$ and $H$ are elliptic curve generators, $v$ is the amount, and $\mu$ is a mask. Without knowledge of the commitment opening, a third party cannot determine the amount; however, it is trivial for the third party to convince itself that a transaction balances (that is, that the difference between inputs and output amounts is zero). The homomorphic property of the commitments is such that the difference in commitments must itself be a commitment to zero. + +However, this is not sufficient to ensure a correct and safe transaction model. An adversary could easily construct a combination of positive and negative outputs such that the transaction amounts balance. A third party would still verify that the transaction balances, though the adversary has effectively printed free money in an undetected fashion. To combat this, we require that each amount commitment come equipped with a \textit{range proof} that convinces a verifier that the corresponding output is both positive and does not risk an overflow by being too large. The range proof scheme must be non-interactive and zero-knowledge; that is, the verifier does not need to communicate with the prover once the proof is generated, and the proof itself reveals no information about the amount except that it is within the stated range. + +The current range proof style used in Monero confidential transactions is a \textit{Borromean bitwise} range proof. To generate a proof that a commitment $C \equiv vG + \mu H$ represents an amount $v \in [0,2^n-1]$ for some bit length $n > 0$ (in Monero $n = 64$), the prover generates separate commitments for each bit. The prover then generates a Borromean ring signature showing that each commitment is to either $0$ or $2^i$ for appropriate $i$. Any third-party verifier can then convince itself that the bit commitments reconstruct the committed amount, that each commitment is to either $0$ or $2^i$, and therefore that the committed amount lies in the correct range. + +However, this comes at a cost. Borromean bitwise proofs scale linearly in size with the number of bits in the range. Further, if multiple outputs are used in a transaction, a separate proof is required for each. Each proof is large, taking up $6.2$ kB of space. + +\section{Bulletproofs} +Bulletproofs are a recent general non-interactive zero-knowledge proof construction \cite{bp}. Using a novel inner product argument, they can be used in a variety of applications ranging from range proofs (pun intended) to verifiable shuffles and even proofs of general arithmetic circuit evaluation. For our purposes, they can accomplish the same goal as Borromean bitwise range proofs: convincing a verifier that a committed amount is within a claimed range. + +The details of Bulletproof construction, both for prover and verifier, are discussed in the paper \cite{bp}, so we will not duplicate them here. However, several definitions are useful when discussing the scaling. A standard Bulletproof that shows an amount is within the $n$-bit range $[0,2^n-1]$ is called a \textit{single-output proof} or a \textit{1-proof}. However, it is possible for a prover to construct a single proof showing that $m$ separate amounts (with separate random masks) each lie within the range $[0,2^n-1]$, where $m$ is a power of two. Such a proof is called an \textit{aggregate proof} or, more precisely, an $m$\textit{-proof}. The scheme is constructed in such a way that a single-output proof is trivially an $m$-proof with $m=1$ (which simplifies the code). It is important to note that the construction of an aggregate proof requires that the prover know each amount and mask; this means that while it is useful for all outputs in a transaction to be contained within a single aggregate proof for space savings, it is not possible for a third party to take existing proofs and construct an aggregate proof, either within a single transaction or between different transactions. + +The size scaling benefits of Bulletproofs occur at two levels: +\begin{enumerate} +\item \textbf{Bit length of range}. The size of a Bulletproof increases logarithmically with the number of bits in the range. In bitwise range proofs, the proof size increased linearly with the number of bits. +\item \textbf{Number of amounts in aggregate proof}. The size of a Bulletproof increases logarithmically with the number of amounts included in a single aggregate proof. In bitwise range proofs, the proof size increased linearly with the number of bits (since a separate proof was needed for each amount). +\end{enumerate} +We discuss efficiency in more detail below. + +There is a separate scaling argument that is useful. A new node that comes online will receive many $m$-proofs, at least one per post-Bulletproof transaction in the blockchain. Instead of verifying each of the proofs separately, the node can perform a \textit{batch verification} of as many proofs at a time as it wishes. As described below, this process requires that certain portions of each proof be verified separately, but allows for the remaining parts of the proofs to be batched and verified together. The resulting verification time is linear in the number of proofs, but with a significantly lower time per proof. An existing node that has already verified the transactions in the blockchain can still use batch verification on new transactions it receives, but the benefits are not as great due to the lower number of transactions that must be verified in a short time. + +\section{Optimizations} +For the most part, the proposed implementation of Bulletproofs in Monero follows the Bulletproofs paper in scope and notation wherever possible. However, we include several optimizations that have also been discussed for other projects. These optimizations are algebraically equivalent to those in the paper, but reduce the time required for verification. The author understands that some or all of the optimizations may be included in an update to the Bulletproofs paper sometime in the future. However, we document them here for completeness and ease of code review. The reader is encouraged to refer to the paper for the complete context of our changes. + +\subsection{Curve group notation} +The paper is written with a general group structure in mind, so scalar-group operations are written multiplicatively (\textit{e.g.} $x = a^bc^d$). In the case of elliptic curve groups, we use additive notation instead (\textit{e.g.} $X = bA + dC$) and use case to differentiate between curve points and scalars for clarity. This is purely a notational convenience. + +\subsection{Basepoint notation} +Throughout the paper, amount commitments are expressed as $V \equiv vG + \mu H$, where $G$ and $H$ are distinct (but arbitrary) fixed elliptic curve group generators. We interchange the roles of $G$ and $H$ throughout our implementation to match the use of existing base points used in commitments elsewhere in the Monero codebase. Note that the indexed $\{G_i\}$ and $\{H_i\}$ curve points are not modified in this way. + +\subsection{Fiat-Shamir challenges} +To make the Bulletproof scheme non-interactive, we follow the paper by introducing Fiat-Shamir challenges computed by hashing the proof transcript up to the point that a new challenge is needed. This is done by introducing a rolling hash that uses as input the previous challenge and any new proof elements introduced. The prover and verifier compute these challenges identically. + +\subsection{Inner product argment} +The inner product argument in Protocol 1 of the Bulletproofs paper uses recursion to shrink the size of its input vectors down to single elements. These inputs include distinct curve group generators $\{G_i\}$ and $\{H_i\}$, which we compute using an indexed hash function. We make several optimizations to this protocol for the verifier. + +First, we observe that the curve points in Equation (10) are in fact linear combinations of $\{G_i\}$ and $\{H_i\}$ that use the scalar challenges in Equations (24)-(25). Next, we note that the point $P$ in Equation (62) is passed into Protocol 1 as described in Section 4.2 of the paper. Since this curve point contains a linear combination of the same group generators as Protocol 1, we can take advantage of this and compute a single linear combination, rather than separately compute Equations (62) and (10). + +In practice, we replace Equations (62) and (10) with the following check, where $M \equiv |\{L_j\}| = |\{R_j\}|$: +$$A + xS - \mu G + \sum_{j=0}^{M-1}(w_j^2 L_j + w_j^{-2} R_j) + (t - ab)xH - \sum_{i=0}^{mn-1}(g_iG_i + h_iH_i) = 0$$ +The symbols are mostly those used in the paper. However, we use $w_j$ to represent the round challenges in Lines (21)-(22), and $x$ to represent the challenge in Lines (32)-(33) to avoid reuse of symbols. The scalars $g_i$ and $h_i$ are computed in the following way. Express the index $i = b_0b_1 \cdots b_{M-1}$ bitwise, where $b_{M-1}$ is the least-significant bit. Then +$$g_i = a\prod_{j=0}^{M-1} w_j^{2b_j-1} + z$$ +and +$$h_i = \left(by^{-i}\prod_{j=0}^{M-1} w_j^{-2b_j+1} - zy^i + z^{2+\lfloor i/N \rfloor}2^{i\operatorname{mod}N}\right)y^{-i}$$ +This optimization is applied only to the verifier. + +\subsection{Batch verification} +Our implementation permits the verifier to take many aggregate proofs and verify them together as a batch. We do not assume that the proofs each have the same number of outputs, nor make any restrictions on the maximum size of a batch. The batch verification we describe will only succeed if each proof is valid, and will fail if one or more proofs are invalid. + +Batch verification is split into two checks, performed after iterating over each proof in the batch. During the iteration, the verifier keeps ongoing sums of components from each proof and then performs the first-stage check for Equation (61): +\begin{equation} +\sum_l (\beta_l\tau_{xl}) G + \sum_l \beta_l\left[ t_l - (k_l + z_l \langle \overline{1}^{mn},\overline{y_l}^{mn} \rangle) \right] H - \sum_l \beta_l \left( \sum_j z_l^{j+2} V_{lj} - x_lT_{1l} - x_l^2T_{2l} \right) = 0 \nonumber +\end{equation} +The second-phase check proceeds similarly: +\begin{multline} +\sum_l \beta_l(A_l + x_lS_l) - \sum_l(\beta_l\mu_l) G + \sum_l\left[\beta_l \sum_j(w_{lj}^2 L_{lj} + w_{lj}^{-2} R_{lj})\right] + \sum_l \beta_l x_l(t_l - a_lb_l) H \\ +- \sum_i \left[\sum_l(\beta_l g_{li})G_i + \sum_l(\beta_l h_{li})H_i\right] = 0 \nonumber +\end{multline} +Here each $l$-indexed sum is over each proof in the batch, and $\beta_l$ is a weighting factor chosen at random (not deterministically) by the verifier. This ensures that, except with negligible probability, the checks will only succeed if each proof is separately valid; an adversary cannot selectively provide a batch containing invalid proofs in an attempt to fool the verifier. The benefit to this approach is that the sums can be computed as large multi-exponentiation operations after the scalars from all proofs have been assembled. + +If the batch fails either check, at least one proof in the batch is invalid. To identify which proofs are at fault, the verifier can either iterate through each proof and perform the checks separately (in linear time), or perform a binary search by successively performing the checks on half-batches until it identifies all faulty proofs (in logarithmic time). + +\section{Proof size} +Including the amount commitment $V$, a single Borromean bitwise range proof occupies $6.2$ kB of space; a transaction with $m$ outputs therefore requires $6.2m$ kB of space. An $m$-proof (with a $64$-bit range) requires $2\lg m + 17$ group elements and $5$ scalars, each of which takes up $32$ bytes. Table \ref{table:size} shows the space savings from Bulletproofs for several values of $m$. + +\begin{table}[h] +\begin{center} +\begin{tabular}{r|rr|c} +$m$ & Bulletproof & Borromean & Relative size \\ +\hline +$1$ & $704$ & $6200$ & $0.114$ \\ +$2$ & $768$ & $12400$ & $0.062$ \\ +$8$ & $896$ & $49600$ & $0.018$ \\ +$16$ & $960$ & $99200$ & $0.010$ \\ +$128$ & $1152$ & $793600$ & $0.001$ +\end{tabular} +\caption{Size (bytes) of $m$ Borromean proofs versus $m$-proof} +\label{table:size} +\end{center} +\end{table} + +Using data from the Monero blockchain\footnote{Data was taken from blocks 1400000 through 1500000} on the distribution of the number of outputs in transactions, the use of Bulletproofs would reduce the total size of range proofs by $94\%$. + +\bibliographystyle{plain} +\bibliography{bulletproofs} + +\end{document} \ No newline at end of file diff --git a/publications/standards/logo.png b/publications/standards/logo.png new file mode 100644 index 0000000..7f647a2 Binary files /dev/null and b/publications/standards/logo.png differ diff --git a/publications/standards/mrl.cls b/publications/standards/mrl.cls new file mode 100644 index 0000000..0bf153f --- /dev/null +++ b/publications/standards/mrl.cls @@ -0,0 +1,43 @@ +\ProvidesClass{mrl} +\LoadClass{article} +\usepackage{amsmath,amssymb,amsthm} +\usepackage[top=1in,bottom=1in,right=1in,left=1in]{geometry} +\usepackage{color} +\usepackage{graphicx} +\usepackage{hyperref} + +\definecolor{bloo}{rgb}{0,0.2,0.4} +\renewcommand*{\thefootnote}{\fnsymbol{footnote}} +\linespread{1.2} + +\renewcommand\title[1]{\def\@title{#1}} +\let\@title=\@empty + +\newcommand\authors[1]{\def\@authors{#1}} +\let\@authors=\@empty + +\newcommand\affiliations[1]{\def\@affiliations{#1}} +\let\@affiliations=\@empty + +\renewcommand\date[1]{\def\@date{#1}} +\let\@date=\@empty + +\newcommand\ident[1]{\def\@ident{#1}} +\let\@ident=\@empty + +\newcommand\type[1]{\def\@type{#1}} +\let\@type=\@empty + +\AtBeginDocument{ +\hfill\includegraphics[width=100px]{logo.png} +\newline +\noindent\colorbox{bloo}{\parbox{\textwidth}{{\sffamily\color{white}\@type \hfill \@ident}}} +\vskip 10pt +\noindent{\Large\@title} +\vskip 5pt +\noindent{\@authors} +\newline +\noindent{\@affiliations} +\newline +\noindent{\@date} +} \ No newline at end of file diff --git a/source-code/Poisson-Graphs/Block.py b/source-code/Poisson-Graphs/Block.py new file mode 100644 index 0000000..dece697 --- /dev/null +++ b/source-code/Poisson-Graphs/Block.py @@ -0,0 +1,46 @@ +import unittest, random, time + +def newIdent(params): + nonce = params + # Generate new random identity. + return hash(str(nonce) + str(random.random())) + #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### #### +class Block(object): + ''' + Each block has: an identity, a timestamp of discovery (possibly false), + has a timestamp of arrival at the local node (possibly unnecessary), a + parent block's identity, and a difficulty score. + ''' + def __init__(self, params={}): + self.ident = None + self.discoTimestamp = None + self.arrivTimestamp = None + self.parent = None + self.diff = None + try: + assert len(params)==5 + except AssertionError: + print("Error in Block(): Tried to add a malformed block. We received params = " + str(params) + ", but should have had something of the form {\"ident\":ident, \"disco\":disco, \"arriv\":arriv, \"parent\":parent, \"diff\":diff}.") + self.ident = params["ident"] + self.discoTimestamp = params["disco"] + self.arrivTimestamp = params["arriv"] + self.parent = params["parent"] + self.diff = params["diff"] + +class Test_Block(unittest.TestCase): + def test_b(self): + #bill = Block() + name = newIdent(0) + t = time.time() + s = t+1 + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":None, "diff":diff} + bill = Block(params) + self.assertEqual(bill.ident,name) + self.assertEqual(bill.discoTimestamp,t) + self.assertEqual(bill.arrivTimestamp,t+1) + self.assertTrue(bill.parent is None) + self.assertEqual(bill.diff,diff) + +suite = unittest.TestLoader().loadTestsFromTestCase(Test_Block) +unittest.TextTestRunner(verbosity=1).run(suite) diff --git a/source-code/Poisson-Graphs/Blockchain.py b/source-code/Poisson-Graphs/Blockchain.py new file mode 100644 index 0000000..17808e3 --- /dev/null +++ b/source-code/Poisson-Graphs/Blockchain.py @@ -0,0 +1,1143 @@ +from Block import * +import math +from scipy.stats import * +from numpy import * +from copy import deepcopy + +class Blockchain(object): + ''' + Not a true blockchain, of course, but tracks block objects (timestamps) as above. + Each node should be responsible for finding the chain with most cumulative work. + Right now we assume Nakamoto consensus (konsensnakamoto). + ''' + def __init__(self, params=[], verbosity=True): + self.blocks = {} + self.leaves = {} + self.miningIdents = None + self.mIdent = None + self.verbose = verbosity + self.diff = None + self.targetRate = None + self.mode = None + + def addBlock(self, blockToAdd): + # In our model we assume difficulty scores of blocks are correct (otherwise they would + # be rejected in the real life network, and we aren't trying to model spam attacks). + assert blockToAdd.ident not in self.blocks + if len(self.blocks)==0: + # In this case, blockToAdd is a genesis block, so we set difficulty + self.diff = deepcopy(blockToAdd.diff) + + self.blocks.update({blockToAdd.ident:blockToAdd}) + self.leaves.update({blockToAdd.ident:blockToAdd}) + if blockToAdd.parent in self.leaves: + del self.leaves[blockToAdd.parent] + self.whichLeaf() + return self.computeDifficulty() + + def whichLeaf(self): + # Determine which leaf shall be the parent leaf. + # If the chain has forked *ever* this will not be the case. + maxCumDiff = 0.0 + self.miningIdents = [] + for ident in self.leaves: + tempCumDiff = 0.0 + thisBlockIdent = ident + tempCumDiff += self.blocks[thisBlockIdent].diff + while self.blocks[thisBlockIdent].parent is not None: + thisBlockIdent = self.blocks[thisBlockIdent].parent + tempCumDiff += self.blocks[thisBlockIdent].diff + if tempCumDiff > maxCumDiff: + # If more than one leaf ties for maxCumDiff, each node in the + # network should pick one of these two arbitrarily. Since we + # are storing each blockchain in a hash table (unordered!), for + # each node in the network that observes a tie, each possible leaf + # is equally likely to have been the first one found! So + # we don't need to do anything for the node to select which chain + # to work off of. + self.miningIdents = [ident] + maxCumDiff = tempCumDiff + elif tempCumDiff == maxCumDiff: + self.miningIdents.append(ident) + #print("leaf ident = ", str(ident), ", and tempCumDiff = ", str(tempCumDiff), " and maxCumDiff = ", str(maxCumDiff)) + assert len(self.miningIdents) > 0 + self.mIdent = random.choice(self.miningIdents) + + + # 1 block in 6*10^5 milliseconds=10min + def computeDifficulty(self): + result = None + if self.mode=="Nakamoto": + # Use MLE estimate of poisson process, compare to targetRate, update by multiplying by resulting ratio. + #if self.verbose: + # print("Beginning update of difficulty with Nakamoto method") + count = 2016 + #if self.verbose: + # print("Checking that blockchain is 2016*n blocks long and some mining identity has been set") + if len(self.blocks) % 2016 == 0 and len(self.miningIdents) > 0: + ident = self.mIdent + topTime = deepcopy(self.blocks[ident].discoTimestamp) + parent = self.blocks[ident].parent + count = count - 1 + touched = False + while count > 0 and parent is not None: + ident = deepcopy(parent) + parent = self.blocks[ident].parent + count = count - 1 + touched = True + if not touched: + mleDiscoRate = deepcopy(self.targetRate) + else: + botTime = deepcopy(self.blocks[ident].discoTimestamp) + + # Algebra is okay: + assert topTime != botTime + + # MLE estimate of arrivals per second: + mleDiscoRate = float(2015)/float(topTime - botTime) + + # Rates can't be negative, but this estimate could be (although it's highly unlikely given Bitcoin's standard choices + # of difficulty update rate, etc. + mleDiscoRate = abs(mleDiscoRate) + + if self.verbose: + print("MLE disco rate = " + str(mleDiscoRate) + " and targetRate = " + str(self.targetRate)) + # Rate must be positive... so the MLE for block arrival rate + # assuming a Poisson process _is not even well-defined_ as + # an estimate for block arrival rate assuming timestamps are + # inaccurately reported! + + # We use it nonetheless. + + if self.verbose: + print("MLE discovery rate = " + str(mleDiscoRate)) + print("Difficulty before adjustment = " + str(self.diff)) + + # Update difficulty multiplicatively + self.diff = self.diff*mleDiscoRate/self.targetRate + + if self.verbose: + print("Difficulty after adjustment = ", str(self.diff)) + + elif self.mode=="vanSaberhagen": + # Similar to above, except use 1200 blocks, discard top 120 and bottom 120 after sorting. + # 4 minute blocks in the original cryptonote, I believe... targetRate = 1.0/ + # 4 minutes/period, 60 seconds/minute ~ 240 seconds/period + # assert targetRate==1.0/240.0 + count = 1200 + #print(self.diff) + assert self.diff != 0.0 + if len(self.blocks) > 120 and len(self.miningIdents) > 0: + ident = self.mIdent + bl = [] + bl.append(deepcopy(self.blocks[ident].discoTimestamp)) + parent = self.blocks[ident].parent + count = count - 1 + while count > 0 and parent is not None: + ident = deepcopy(parent) + bl.append(deepcopy(self.blocks[ident].discoTimestamp)) + parent = self.blocks[ident].parent + count = count-1 + # sort + bl = sorted(bl) + assert len(bl)<=1200 + + #print("Sample size = " + str(len(bl))) + # remove 10 and 90 %-iles + numOutliers = math.ceil(float(len(bl))/float(10)) + assert numOutliers <= 120 + #print("Number of outliers = " + str(numOutliers)) + oldBL = deepcopy(bl) + if numOutliers > 0: + bl = bl[numOutliers:-numOutliers] + #if numOutliers == 120: + # print("\n\nSORTED TS LIST = " + str(oldBL) + "\nModified list = " + str(bl)) + + + # get topTime and botTime + #if self.verbose: + # print("bl[0] = " + str(bl[0]) + ",\tbl[-1] = " + str(bl[-1])) + topTime = bl[-1] + botTime = bl[0] + result = [float(topTime - botTime)] + #print(topTime - botTime) + #if self.verbose: + # print("list of timestamps = " + str(bl)) + # print("topTime = " + str(bl[-1])) + # print("botTime = " + str(bl[0])) + + # Assert algebra will work + # 1200 - 2*120 = 1200 - 240 = 960 + assert 0 < len(bl) and len(bl) < 961 + assert topTime - botTime >= 0.0 + result.append(len(bl)-1) + # Sort of the MLE: # blocks/difference in reported times + # But not the MLE, since the reported times may not be + # the actual times, the "difference in reported times" != + # "ground truth difference in block discoery times" in general + if len(bl)==0: + print("WOOP WOOP NO TIMESTAMPS WTF? We have " + str(len(self.blocks)) + " blocks available, and we are counting " + str(2*numOutliers) + " as outliers. bl = " + str(bl)) + naiveDiscoRate = float(len(bl)-1)/float(topTime - botTime) + + # How much should difficulty change? + assert naiveDiscoRate != 0.0 + assert self.targetRate != 0.0 + assert self.diff != 0.0 + self.diff = self.diff*naiveDiscoRate/self.targetRate + + elif self.mode=="MOM:expModGauss": + # Similar to "vanSaberhagen" except with 2-minute blocks and + # we attempt to take into account that "difference in timestamps" + # can be negative by: + # 1) insisting that the ordering induced by the blockchain and + # 2) modeling timestamps as exponentially modified gaussian. + # If timestamps are T = X + Z where X is exponentially dist- + # ributed with parameter lambda and Z is some Gaussian + # noise with average mu and variance sigma2, then we can est- + # imate sigma2, mu, and lambda: + # mu ~ mean - stdev*(skewness/2)**(1.0/3.0) + # sigma2 ~ variance*(1-(skewness/2)**(2.0/3.0)) + # lambda ~ (1.0/(stdev))*(2/skewness)**(1.0/3.0) + #assert targetRate==1.0/120.0 + + # Really a trash metric unless sample sizes are huge. + count = 1200 + ident = self.mIdent + bl = [] + bl.append(deepcopy(self.blocks[ident].discoTimestamp)) + parent = self.blocks[ident].parent + count = count - 1 + while count > 0 and parent is not None: + ident = deepcopy(parent) + bl.append(deepcopy(self.blocks[ident].discoTimestamp)) + parent = self.blocks[ident].parent + count = count-1 + if len(bl) > 120: + sk = abs(skew(bl)) + va = var(bl) + stdv = sqrt(va) + lam = (1.0/stdv)*(2.0/sk)**(1.0/3.0) + else: + lam = self.targetRate # we will not change difficulty unless we have at least 120 blocks of data (arbitrarily selected) + self.diff = self.diff*(lam/self.targetRate) + elif self.mode=="reciprocalOfMedian": + # In this mode we use a bitcoin-style metric except instead of 1/average inter-arrival time + # we use 1/median magnitude of inter-arrival time. + # And updated each block like with monero instead of every 2016 blocks like bitcoin. + # We assume a sample size of only 600 blocks for now + count = 600 + interArrivals = [] + if len(self.blocks) < count: + estDiscoRate = self.targetRate + elif len(self.miningIdents) > 0: + ident = self.mIdent + parent = self.blocks[ident].parent + if parent is not None: + dT = abs(self.blocks[ident].discoTimestamp - self.blocks[parent].discoTimestamp) + interArrivals.append(dT) + count = count - 1 + touched = False + while count > 0 and parent is not None: + ident = deepcopy(parent) + parent = self.blocks[ident].parent + if parent is not None: + dT = abs(self.blocks[ident].discoTimestamp - self.blocks[parent].discoTimestamp) + interArrivals.append(dT) + count = count - 1 + touched = True + if not touched: + estDiscoRate = self.targetRate + else: + estDiscoRate = 1.0/median(interArrivals) + if self.verbose: + print("Est disco rate = " + str(estDiscoRate) + " and targetRate = " + str(self.targetRate)) + + + if self.verbose: + print("MLE discovery rate = " + str(estDiscoRate)) + print("Difficulty before adjustment = " + str(self.diff)) + + # Update difficulty multiplicatively + self.diff = self.diff*estDiscoRate/self.targetRate + + if self.verbose: + print("Difficulty after adjustment = ", str(self.diff)) + else: + print("Error, invalid difficulty mode entered.") + return result + +class Test_Blockchain(unittest.TestCase): + def test_addBlock(self): + bill = Blockchain([], verbosity=True) + bill.mode="Nakamoto" + tr = 1.0/100.0 + bill.targetRate = tr + + name = newIdent(0) + t = time.time() + s = t+random.random() + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":None, "diff":diff} + genesis = Block(params) + + self.assertEqual(genesis.ident,name) + self.assertEqual(genesis.discoTimestamp,t) + self.assertEqual(genesis.arrivTimestamp,s) + self.assertTrue(genesis.parent is None) + self.assertEqual(genesis.diff,diff) + + bill.addBlock(genesis) + + self.assertTrue(genesis.ident in bill.blocks) + self.assertTrue(genesis.ident in bill.leaves) + self.assertEqual(len(bill.miningIdents),1) + self.assertEqual(genesis.ident, bill.miningIdents[0]) + self.assertEqual(len(bill.blocks),1) + + name = newIdent(1) + t = time.time() + s = t+random.random() + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":genesis.ident, "diff":diff} + blockA = Block(params) + bill.addBlock(blockA) + + self.assertTrue(blockA.ident in bill.blocks) + self.assertTrue(blockA.ident in bill.leaves) + self.assertTrue(genesis.ident not in bill.leaves) + self.assertEqual(len(bill.miningIdents),1) + self.assertEqual(blockA.ident, bill.miningIdents[0]) + self.assertEqual(len(bill.blocks),2) + + + + + + bill = Blockchain([], verbosity=True) + mode="vanSaberhagen" + tr = 1.0/100.0 + bill.targetRate = tr + + name = newIdent(0) + t = time.time() + s = t+random.random() + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":None, "diff":diff} + genesis = Block(params) + + self.assertEqual(genesis.ident,name) + self.assertEqual(genesis.discoTimestamp,t) + self.assertEqual(genesis.arrivTimestamp,s) + self.assertTrue(genesis.parent is None) + self.assertEqual(genesis.diff,diff) + + bill.addBlock(genesis, mode, tr) + + self.assertTrue(genesis.ident in bill.blocks) + self.assertTrue(genesis.ident in bill.leaves) + self.assertEqual(len(bill.miningIdents),1) + self.assertEqual(genesis.ident, bill.miningIdents[0]) + self.assertEqual(len(bill.blocks),1) + + name = newIdent(1) + t = time.time() + s = t+random.random() + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":genesis.ident, "diff":diff} + blockA = Block(params) + bill.addBlock(blockA, mode, tr) + + self.assertTrue(blockA.ident in bill.blocks) + self.assertTrue(blockA.ident in bill.leaves) + self.assertTrue(genesis.ident not in bill.leaves) + self.assertEqual(len(bill.miningIdents),1) + self.assertEqual(blockA.ident, bill.miningIdents[0]) + self.assertEqual(len(bill.blocks),2) + + + + + + bill = Blockchain([], verbosity=True) + mode="MOM:expModGauss" + tr = 1.0/100.0 + bill.targetRate = tr + + name = newIdent(0) + t = time.time() + s = t+random.random() + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":None, "diff":diff} + genesis = Block(params) + + self.assertEqual(genesis.ident,name) + self.assertEqual(genesis.discoTimestamp,t) + self.assertEqual(genesis.arrivTimestamp,s) + self.assertTrue(genesis.parent is None) + self.assertEqual(genesis.diff,diff) + + bill.addBlock(genesis, mode, tr) + + self.assertTrue(genesis.ident in bill.blocks) + self.assertTrue(genesis.ident in bill.leaves) + self.assertEqual(len(bill.miningIdents),1) + self.assertEqual(genesis.ident, bill.miningIdents[0]) + self.assertEqual(len(bill.blocks),1) + + name = newIdent(1) + t = time.time() + s = t+random.random() + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":genesis.ident, "diff":diff} + blockA = Block(params) + bill.addBlock(blockA, mode, tr) + + self.assertTrue(blockA.ident in bill.blocks) + self.assertTrue(blockA.ident in bill.leaves) + self.assertTrue(genesis.ident not in bill.leaves) + self.assertEqual(len(bill.miningIdents),1) + self.assertEqual(blockA.ident, bill.miningIdents[0]) + self.assertEqual(len(bill.blocks),2) + + + def test_bc(self): + bill = Blockchain([], verbosity=True) + mode="Nakamoto" + tr = 1.0/100.0 + bill.targetRate = tr + + name = newIdent(0) + t = time.time() + s = t+1 + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":None, "diff":diff} + genesis = Block(params) + + self.assertEqual(genesis.ident,name) + self.assertEqual(genesis.discoTimestamp,t) + self.assertEqual(genesis.arrivTimestamp,t+1) + self.assertTrue(genesis.parent is None) + self.assertEqual(genesis.diff,diff) + + bill.addBlock(genesis, mode, tr) + + self.assertTrue(genesis.ident in bill.blocks) + self.assertTrue(genesis.ident in bill.leaves) + self.assertEqual(len(bill.miningIdents),1) + self.assertEqual(genesis.ident, bill.miningIdents[0]) + + name = newIdent(1) + t = time.time() + s = t+1 + diff = 2.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":genesis.ident, "diff":diff} + blockA = Block(params) + bill.addBlock(blockA, mode, tr) + + #bill.whichLeaf() + + self.assertTrue(blockA.ident in bill.blocks) + self.assertTrue(blockA.ident in bill.leaves) + self.assertFalse(genesis.ident in bill.leaves) + self.assertTrue(genesis.ident in bill.blocks) + self.assertEqual(len(bill.miningIdents),1) + self.assertEqual(blockA.ident, bill.miningIdents[0]) + + name = newIdent(1) + t = time.time() + s = t+1 + diff = 2.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":genesis.ident, "diff":diff} + blockB = Block(params) + bill.addBlock(blockB, mode, tr) + + self.assertTrue(blockB.ident in bill.blocks) + self.assertTrue(blockB.ident in bill.leaves) + self.assertEqual(bill.blocks[blockB.ident].parent, genesis.ident) + + self.assertTrue(blockA.ident in bill.blocks) + self.assertTrue(blockA.ident in bill.leaves) + self.assertEqual(bill.blocks[blockA.ident].parent, genesis.ident) + + self.assertTrue(genesis.ident in bill.blocks) + self.assertFalse(genesis.ident in bill.leaves) + self.assertTrue(bill.blocks[genesis.ident].parent is None) + + #bill.whichLeaf() + #print(bill.miningIdents) + + self.assertEqual(type(bill.miningIdents), type([])) + self.assertTrue(len(bill.miningIdents), 2) + + name = newIdent(2) + t = time.time() + diff = 3.14159 + params = {"ident":name, "disco":t, "arriv":s, "parent":blockB.ident, "diff":diff} + blockC = Block(params) + bill.addBlock(blockC, mode, tr) + + self.assertTrue(blockC.ident in bill.blocks) + self.assertTrue(blockC.ident in bill.leaves) + + self.assertTrue(blockB.ident in bill.blocks) + self.assertFalse(blockB.ident in bill.leaves) + + self.assertTrue(blockA.ident in bill.blocks) + self.assertTrue(blockA.ident in bill.leaves) + + self.assertTrue(genesis.ident in bill.blocks) + self.assertFalse(genesis.ident in bill.leaves) + + #bill.whichLeaf() + + #for blockIdent in bill.blocks: + # ident = bill.blocks[blockIdent].ident + # disco = bill.blocks[blockIdent].discoTimestamp + # arriv = bill.blocks[blockIdent].arrivTimestamp + # parent = bill.blocks[blockIdent].parent + # diff = bill.blocks[blockIdent].diff + # print(str(ident) + ", " + str(disco) + ", " + str(arriv) + ", " + str(parent) + ", " + str(diff) + ", " + str() + "\n") + #print(bill.miningIdents) + self.assertEqual(len(bill.miningIdents), 1) + self.assertEqual(bill.miningIdents[0], blockC.ident) + + def test_median(self): + # TODO: everything. + mode = "reciprocalOfMedian" + tr = 1.0 # one block per millisecond why not + deltaT = 1.0 # let's just make this easy + bill = Blockchain([], verbosity=True) + bill.targetRate = tr + + with open("outputM.txt", "w") as writeFile: + # We will send (t, a, diff) to writeFile. + writeFile.write("time,rateConstant,difficulty\n") + name = newIdent(0) + t = 0.0 + s = 0.0 + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":None, "diff":diff} + genesis = Block(params) + + self.assertEqual(genesis.ident,name) + self.assertEqual(genesis.discoTimestamp,t) + self.assertEqual(genesis.arrivTimestamp,s) + self.assertTrue(genesis.parent is None) + self.assertEqual(genesis.diff,diff) + + bill.addBlock(genesis, mode, tr) + a = 1.0 + b = 1.0/a + + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "\n") + + self.assertTrue(genesis.ident in bill.blocks) + self.assertTrue(genesis.ident in bill.leaves) + self.assertEqual(len(bill.miningIdents),1) + self.assertEqual(genesis.ident, bill.miningIdents[0]) + + parent = genesis.ident + oldDiff = bill.diff + + while len(bill.blocks)<601: + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + self.assertEqual(bill.diff, oldDiff) + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "\n") + parent = newName + oldDiff = bill.diff + + a = 1.01 # slightly slower blocks, median won't change until half the data is corrupted! + b = 1.0/a + while len(bill.blocks)<899: + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + self.assertEqual(bill.diff, oldDiff) + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + self.assertEqual(bill.diff, oldDiff) + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "\n") + parent = newName + oldDiff = bill.diff + + # One more block and our median inter-arrival time is deltaT*(1.0+a)/2.0 + # and so estRate = 1/median = (2.0/(1.0+a))/deltaT, whereas before it was just + # 1/deltaT. So estRate/targetRate = 2.0/(1.0+a) + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + self.assertEqual(bill.diff, oldDiff) + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + err = bill.diff - oldDiff*2.0/(1.0+a) + self.assertTrue(err*err < 10**-15) + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "\n") + parent = newName + oldDiff = bill.diff + + # One more block and our median inter-arrival time is deltaT*a + # and so estRate = 1/median = (1.0/a)/deltaT, whereas before it was just + # 1/deltaT. So estRate/targetRate = 1.0/a = b + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + self.assertEqual(bill.diff, oldDiff) + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + err = bill.diff - oldDiff*b + self.assertTrue(err*err < 10**-15) + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "\n") + parent = newName + oldDiff = bill.diff + + # Note that until the median changes again, this estimated block arrival rate + # does not change. This may be true even if a lot of new data has come in. + # It is possible that the same pair of blocks remain the median inter-arrival + # magnitude for the entire time both blocks are in the sample size. + # During this period of time, difficulty will update multiplicatively, so + # will either exponentially grow or shrink. + # In other words, this model can be looked at as: exponential change over + # time with a rate proportional to the deviation between the median and + # the target inter-arrival rates. + + + + + + def test_mine(self): + # TODO: everything. + mode = "MOM:expModGauss" + tr = 1.0/120000.0 # one block per two minutes + deltaT = 120000.0 + bill = Blockchain([], verbosity=True) + bill.targetRate = tr + + with open("outputM.txt", "w") as writeFile: + # We will send (t, a, diff, ratio, awayFromOne) to writeFile. + writeFile.write("time,rateConstant,difficulty\n") + name = newIdent(0) + t = 0.0 + s = 0.0 + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":None, "diff":diff} + genesis = Block(params) + + self.assertEqual(genesis.ident,name) + self.assertEqual(genesis.discoTimestamp,t) + self.assertEqual(genesis.arrivTimestamp,s) + self.assertTrue(genesis.parent is None) + self.assertEqual(genesis.diff,diff) + + bill.addBlock(genesis, mode, tr) + a = 1.0 + b = 1.0/a + + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "\n") + + self.assertTrue(genesis.ident in bill.blocks) + self.assertTrue(genesis.ident in bill.leaves) + self.assertEqual(len(bill.miningIdents),1) + self.assertEqual(genesis.ident, bill.miningIdents[0]) + + parent = genesis.ident + oldDiff = bill.diff + + while len(bill.blocks)<120: + # Our metric divides by skewness. In reality, this is zero with + # probability zero. But for our tests, it's assured. So we + # will perturb each arrival by a small, up-to-half-percent + # variation to ensure a nonzero skewness without altering things + # too much. + newName = newIdent(len(bill.blocks)) + t += deltaT*a*(1.0 + (2.0*random.random() - 1.0)/100.0) + s += deltaT*a*(1.0 + (2.0*random.random() - 1.0)/100.0) + self.assertEqual(bill.diff, oldDiff) + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "\n") + parent = newName + oldDiff = bill.diff + + # Just one more block and difficulty should be computed for the first time. + print("Just one more block and difficulty should be computed for the first time.") + self.assertEqual(bill.diff, oldDiff) + newName = newIdent(len(bill.blocks)) + t += deltaT*a*(1.0 + (2.0*random.random() - 1.0)/100.0) + s += deltaT*a*(1.0 + (2.0*random.random() - 1.0)/100.0) + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "\n") + parent = newName + #self.assertEqual(bill.diff, oldDiff) + + oldDiff = bill.diff + + # what if we add a bunch of blocks this way? + # In the case of a static hash rate, I suppose we hope to not + # vary too far from a multiplicative factor of 1.0, or rather + # a constant difficulty. + + while len(bill.blocks)<200: + # Our metric divides by skewness. In reality, this is zero with + # probability zero. But for our tests, it's assured. So we + # will perturb each arrival by a small, up-to-half-percent + # variation to ensure a nonzero skewness without altering things + # too much. + newName = newIdent(len(bill.blocks)) + t += deltaT*a*(1.0 + (2.0*random.random() - 1.0)/100.0) + s += deltaT*a*(1.0 + (2.0*random.random() - 1.0)/100.0) + self.assertEqual(bill.diff, oldDiff) + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "\n") + parent = newName + oldDiff = bill.diff + + + + def test_vs(self): + # TODO: Still must test that outliers are being removed "appropriately" according to specifications + # TODO: Test that scrambled lists of timestamps produce the same difficulty estimate. + # TODO: Show that in the case of homogeneous poisson processes, unusual estimates are a little + # more common than in the Nakamoto difficulty (which must be the case because Nakamoto uses + # the UMVUE). + mode = "vanSaberhagen" + tr = 1.0/60000.0 # one block per minute + deltaT = 60000.0 + bill = Blockchain([], verbosity=True) + bill.targetRate = tr + + with open("output.txt", "w") as writeFile: + # We will send (t, a, diff, ratio, awayFromOne) to writeFile. + writeFile.write("time,rateConstant,difficulty,ratio\n") + name = newIdent(0) + t = 0.0 + s = 0.0 + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":None, "diff":diff} + genesis = Block(params) + + self.assertEqual(genesis.ident,name) + self.assertEqual(genesis.discoTimestamp,t) + self.assertEqual(genesis.arrivTimestamp,s) + self.assertTrue(genesis.parent is None) + self.assertEqual(genesis.diff,diff) + + bill.addBlock(genesis, mode, tr) + writeFile.write(str(t) + ",1.0," + str(bill.diff) + ",1.0\n") + + self.assertTrue(genesis.ident in bill.blocks) + self.assertTrue(genesis.ident in bill.leaves) + self.assertEqual(len(bill.miningIdents),1) + self.assertEqual(genesis.ident, bill.miningIdents[0]) + self.assertEqual(bill.diff, 1.0) + + parent = genesis.ident + oldDiff = bill.diff + a = 1.0 + b = 1.0/a + + while len(bill.blocks)<120: + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + self.assertEqual(bill.diff, oldDiff) + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + + writeFile.write(str(t) + ",1.0," + str(bill.diff) + ",1.0\n") + parent = newName + oldDiff = bill.diff + + # Just one more block and difficulty should be computed for the first time. + print("Just one more block and difficulty should be computed for the first time.") + self.assertEqual(bill.diff, oldDiff) + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + writeFile.write(str(t) + ",1.0," + str(bill.diff) + ",1.0\n") + parent = newName + self.assertEqual(bill.diff, oldDiff) + + oldDiff = bill.diff + + print("Let's add more blocks at the same rate.") + a = 1.0 + b = 1.0/a + + while len(bill.blocks)<1200: + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + + bill.addBlock(newBlock, mode, tr) + writeFile.write(str(t) + ",1.0," + str(bill.diff) + ",1.0\n") + parent = newName + self.assertEqual(bill.diff, oldDiff) + oldDiff = bill.diff + + print("Let's add more blocks at a slower rate.") + a = 1.1 + b = 1.0/a + + # If blocks arrive slightly further apart, difficulty should drop. + # However, since vanSaberhagen discards top 10% and bottom 10% of + # timestamps, it will take 120 blocks for this change to register + # in difficulty. + print("If blocks arrive slightly further apart, difficulty should drop. However, since vanSaberhagen discards top 10% and bottom 10% of timestamps, it will take 120 blocks for this change to register in difficulty.") + while len(bill.blocks)<1320: + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + + bill.addBlock(newBlock, mode, tr) + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "," + str(bill.diff/oldDiff) + "\n") + parent = newName + self.assertEqual(bill.diff, oldDiff) + oldDiff = bill.diff + + print("One more block and difficulty should register a change.") + self.assertEqual(bill.diff, oldDiff) + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "," + str(bill.diff/oldDiff) + "\n") + parent = newName + self.assertTrue(bill.diff < oldDiff) + oldDiff = bill.diff + + # Let's add another fifty blocks at this same rate and verify that difficulty continually + # drops. + print("Let's add another fifty blocks at this same rate and verify that difficulty continually drops.") + a = 1.1 + b = 1.0/a + + while len(bill.blocks)<1370: + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + + bill.addBlock(newBlock, mode, tr) + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "," + str(bill.diff/oldDiff) + "\n") + parent = newName + self.assertTrue(bill.diff < oldDiff) + oldDiff = bill.diff + + # Now we go back to the target rate. We have 170 slow blocks in the queue and 50 in the sample size. Difficulty will continue to drop for another 120 blocks... + print("Now we go back to the target rate. We have 170 slow blocks in the queue and 50 in the sample size. Difficulty will continue to drop for another 120 blocks...") + a = 1.0 + b = 1.0/a + + while len(bill.blocks)<1490: + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + + bill.addBlock(newBlock, mode, tr) + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "," + str(bill.diff/oldDiff) + "\n") + parent = newName + self.assertTrue(bill.diff < oldDiff) + oldRatio = bill.diff/oldDiff + oldDiff = bill.diff + #print(str(result) + ", " + str(bill.diff) + ", " + str(oldDiff)) + + # Now all 170 slow blocks are not only in the queue but in our sample. The *multiplicative factor* between timesteps should be identical for the next 790 blocks.. leading to AN EXPONENTIAL DECAY OF DIFFICULTY. + print("Now all 170 slow blocks are not only in the queue but in our sample. The *multiplicative factor* between timesteps should be identical for the next 790 blocks.. leading to AN EXPONENTIAL DECAY OF DIFFICULTY.") + a = 1.0 + b = 1.0/a + while len(bill.blocks)<2279: + newName = newIdent(len(bill.blocks)) + t += deltaT + s += deltaT + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + + bill.addBlock(newBlock, mode, tr) + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "," + str(bill.diff/oldDiff) + "\n") + ratio = bill.diff/oldDiff + parent = newName + err = ratio - oldRatio + #print("Difference between last ratio and next ratio:" + str(err)) + self.assertTrue(err*err < 10**-15) + oldDiff = bill.diff + oldRatio = ratio + + print("Now adding a single new block will cause our 170 slow blocks to start dropping out of our sample, so the ratio should start returning to 1.0.") + oldAwayFromOne = abs(oldRatio - 1.0) # Ratio should be returning to 1.0 so this difference should go to zero + oldAwayFromOne = oldAwayFromOne*oldAwayFromOne + + # For the next 170 blocks as our perturbed blocks drop out of our sample, our + # estimated block arrival rate will return to "normal" so the multiplicative + # difference in difficulty should return to 1.0. + print("For the next 170 blocks as our perturbed blocks drop out of our sample, ourestimated block arrival rate will return to normal so the multiplicative difference in difficulty should return to 1.0.") + a = 1.0 + b = 1.0/a + while len(bill.blocks)<2449: + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + + bill.addBlock(newBlock, mode, tr) + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "," + str(bill.diff/oldDiff) + "\n") + ratio = bill.diff/oldDiff + #print("New ratio = " + str(ratio) + " and oldRatio = " + str(oldRatio)) + self.assertTrue(ratio > oldRatio) + awayFromOne = abs(ratio - 1.0) # Ratio should be returning to 1.0 so this difference should go to zero + awayFromOne = awayFromOne*awayFromOne + self.assertTrue(awayFromOne < oldAwayFromOne) # This return will be monotonic in our manufactured example. + parent = newName + oldDiff = bill.diff + oldRatio = ratio + oldAwayFromOne = awayFromOne + + + # Now difficulty should remain frozen for as long as we like. + + a = 1.0 + b = 1.0/a + while len(bill.blocks)<2500: + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + + bill.addBlock(newBlock, mode, tr) + writeFile.write(str(t) + "," + str(a) + "," + str(bill.diff) + "," + str(bill.diff/oldDiff) + "\n") + parent = newName + self.assertEqual(bill.diff, oldDiff) + oldDiff = bill.diff + + + + + def test_nak(self): + # Since Nakamoto difficulty is derived from the MLE of the block arrival rate, + # we already know how it "should" behave in a poisson process, etc. + # TODO: Generate N samples of MLEs of Poisson rates compared to known homog. + # poisson rate, show that the resulting code does not result in unusual measurements + # more often than expected. + mode = "Nakamoto" + tr = 1.0/600000.0 + deltaT = 600000.0 + bill = Blockchain([], verbosity=True) + bill.targetRate = tr + # Bitcoin updating at 1 block per 10 minutes + + name = newIdent(0) + t = 0.0 + s = 0.0 + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":None, "diff":diff} + genesis = Block(params) + + self.assertEqual(genesis.ident,name) + self.assertEqual(genesis.discoTimestamp,t) + self.assertEqual(genesis.arrivTimestamp,s) + self.assertTrue(genesis.parent is None) + self.assertEqual(genesis.diff,diff) + + bill.addBlock(genesis, mode, tr) + + self.assertTrue(genesis.ident in bill.blocks) + self.assertTrue(genesis.ident in bill.leaves) + self.assertEqual(len(bill.miningIdents),1) + self.assertEqual(genesis.ident, bill.miningIdents[0]) + self.assertEqual(bill.diff, 1.0) + + parent = genesis.ident + oldDiff = bill.diff + i = 1 + + while len(bill.blocks)<2016*i-1: + newName = newIdent(len(bill.blocks)) + t += deltaT + s += deltaT + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + parent = newName + + # Just one more block and difficulty should recompute. + print("Just one more block and difficulty should recompute.") + self.assertEqual(bill.diff, oldDiff) + newName = newIdent(len(bill.blocks)) + t += deltaT + s += deltaT + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + parent = newName + self.assertEqual(bill.diff, oldDiff) + + oldDiff = bill.diff + i += 1 + + while len(bill.blocks)<2016*i-1: + newName = newIdent(len(bill.blocks)) + t += deltaT + s += deltaT + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + + bill.addBlock(newBlock, mode, tr) + parent = newName + + # Just one more block and difficulty should recompute. + print("Just one more block and difficulty should again recompute.") + self.assertEqual(bill.diff, oldDiff) + newName = newIdent(len(bill.blocks)) + t += deltaT + s += deltaT + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + parent = newName + self.assertEqual(bill.diff, oldDiff) + + oldDiff = bill.diff + i += 1 + a = 1.1 + b = 1.0/a + + # If blocks arrive slightly further apart, difficulty should drop. + while len(bill.blocks)<2016*i-1: + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + + bill.addBlock(newBlock, mode, tr) + parent = newName + + print("Just one more block and difficulty will go down.") + + self.assertEqual(bill.diff, oldDiff) + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + parent = newName + err = abs(bill.diff - oldDiff*b) + self.assertTrue(err*err < 10**-15) + oldDiff = bill.diff + i += 1 + + + # If blocks then arrive on target, difficulty should freeze. + a = 1.0 + b = 1.0/a + while len(bill.blocks)<2016*i-1: + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + + bill.addBlock(newBlock, mode, tr) + parent = newName + + self.assertEqual(bill.diff, oldDiff) + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + parent = newName + self.assertEqual(bill.diff, oldDiff) + oldDiff = bill.diff + i += 1 + + # If blocks arrive too close together, difficulty should increase. + a = 0.9 + b = 1.0/a + while len(bill.blocks)<2016*i-1: + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + + bill.addBlock(newBlock, mode, tr) + parent = newName + + print("Just one more block and difficulty should go up.") + self.assertEqual(bill.diff, oldDiff) + newName = newIdent(len(bill.blocks)) + t += deltaT*a + s += deltaT*a + diff = bill.diff + params = {"ident":newName, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + bill.addBlock(newBlock, mode, tr) + parent = newName + err = abs(bill.diff - oldDiff*b) + self.assertTrue(err*err < 10**-15) + + + + + +#suite = unittest.TestLoader().loadTestsFromTestCase(Test_Blockchain) +#unittest.TextTestRunner(verbosity=1).run(suite) diff --git a/source-code/Poisson-Graphs/Edge.py b/source-code/Poisson-Graphs/Edge.py new file mode 100644 index 0000000..b3634f2 --- /dev/null +++ b/source-code/Poisson-Graphs/Edge.py @@ -0,0 +1,40 @@ +from Node import * + +class Edge(object): + ''' + Edge object. Has an identity, some data, and a dict of nodes. + ''' + def __init__(self, params): + try: + assert len(params)==3 + except AssertionError: + print("Error, tried to create mal-formed edge.") + else: + self.ident = params[0] + self.data = params[1] + self.verbose = params[2] + self.nodes = {} + + def getNeighbor(self, nodeIdent): + # Given one node identity, check that the node + # identity is in the edge's node list and + # return the identity of the other adjacent node. + result = (nodeIdent in self.nodes) + if result: + for otherIdent in self.nodes: + if otherIdent != nodeIdent: + result = otherIdent + assert result in self.nodes + return result + +class Test_Edge(unittest.TestCase): + def test_e(self): + nelly = Node(params) + milly = Node(params) + ed = Edge(params) + ed.nodes.update({nelly.ident:nelly, milly.ident:milly}) + self.assertEqual(len(self.nodes),2) + + +suite = unittest.TestLoader().loadTestsFromTestCase(Test_Edge) +unittest.TextTestRunner(verbosity=1).run(suite) diff --git a/source-code/Poisson-Graphs/Graph.py b/source-code/Poisson-Graphs/Graph.py new file mode 100644 index 0000000..54e5645 --- /dev/null +++ b/source-code/Poisson-Graphs/Graph.py @@ -0,0 +1,187 @@ +from Blockchain import * +from Node import * +from Edge import * +from copy import * + +def newIntensity(params): + x = random.random() + return x + +def newOffset(params): + x = 2.0*random.random() - 1.0 + return x + +class Graph(object): + ''' + Explanation + ''' + def __init__(self, params): + self.nodes = {} + self.edges = {} + self.mode = params[0] + self.targetRate = params[1] + self.numInitNodes = params[2] + self.maxNeighbors = params[3] + self.probEdge = params[4] + self.verbosity = params[5] + self.startTime = deepcopy(time.time()) + self.runTime = params[6] + self.globalTime = deepcopy(self.startTime) + self.birthRate = params[7] + self.deathRate = params[8] + self.data = params[9] + + self.blankBlockchain = Blockchain() + self.blankBlockchain.targetRate = self.data["targetRate"] + self.blankBlockchain.mode = self.data["mode"] + + self._createInit() + + + def _createInit(self): + # For simplicity, all nodes will have a genesis block with t=0.0 and no offset + for i in range(self.numInitNodes): + offset = newOffset() + intens = newIntensity() + name = newIdent(len(self.nodes)) + dat = {"offset":offset, "intensity":intens, "blockchain":deepcopy(self.blankBlockchain)} + params = {"ident":name, "data":dat, "verbose":self.verbosity, "mode":self.mode, "targetRate":self.targetRate} + nelly = Node(params) + self.nodes.update({nelly.ident:nelly}) + t = self.startTime + self.nodes[nelly.ident].generateBlock(t, i) + + touched = {} + for xNode in self.nodes: + for yNode in self.nodes: + notSameNode = (xNode != yNode) + xNodeHasRoom = (len(self.nodes[xNode].edges) < self.maxNeighbors) + yNodeHasRoom = (len(self.ndoes[yNode].edges) < self.maxNeighbors) + xyNotTouched = ((xNode, yNode) not in touched) + yxNotTouched = ((yNode, xNode) not in touched) + if notSameNode and xNodeHasRoom and yNodeHasRoom and xyNotTouched and yxNotTouched: + touched.update({(xNode,yNode):True, (yNode,xNode):True}) + if random.random() < self.probEdge: + params = [newIdent(len(self.edges)), {}, self.verbosity] + ed = Edge(params) + ed.nodes.update({xNode:self.nodes[xNode], yNode:self.nodes[yNode]}) + self.edges.update({ed.ident:ed}) + self.nodes[xNode].edges.update({ed.ident:ed}) + self.nodes[yNode].edges.update({ed.ident:ed}) + + def eventNodeJoins(self, t): + # timestamp,nodeJoins,numberNeighbors,neighbor1.ident,edge1.ident,neighbor2.ident,edge2.ident,..., + out = "" + neighbors = [] + for xNode in self.nodes: + xNodeHasRoom = (len(self.nodes[xNode].edges) < self.maxNeighbors) + iStillHasRoom = (len(neighbors) < self.maxNeighbors) + if xNodeHasRoom and and iStillHasRoom and random.random() < self.probEdge: + neighbors.append(xNode) + + + newNodeName = newIdent(len(self.nodes)) + offset = newOffset() + intens = newIntensity() + dat = {"offset":offset, "intensity":intens, "blockchain":deepcopy(self.blankBlockchain)} + params = {"ident":newNodeName, "data":dat, "verbose":self.verbosity, "mode":self.mode, "targetRate":self.targetRate} + newNode = Node(params) + self.nodes.update({newNode.ident:newNode}) + self.nodes[newNode.ident].generateBlock(self.startTime, 0) + + out = str(t) + ",nodeJoins," + str(newNode.ident) + "," + str(len(neighbors)) + "," + for neighbor in neighbors: + out += neighbor + "," + params = [newIdent(len(self.edges)), {}, self.verbosity] + ed = Edge(params) + ed.nodes.update({neighbor:self.nodes[neighbor], newNode.ident:self.nodes[newNode.ident]}) + out += ed.ident + "," + self.edges.update({ed.ident:ed}) + self.nodes[neighbor].edges.update({ed.ident:ed}) + self.nodes[newNode.ident].edges.update({ed.ident:ed}) + return out + + def eventNodeLeaves(self, t): + out = str(t) + ",nodeLeaves," + leaverIdent = random.choice(list(self.nodes.keys())) + out += str(leaverIdent) + "," + leaver = self.nodes[leaverIdent] + neighbors = [] + for ed in leaver.edges: + neighbors.append((ed.Ident, ed.getNeighbor(leaverIdent))) + for neighbor in neighbors: + edIdent = neighbor[0] + neiIdent = neighbor[1] + del self.nodes[neiIdent].edges[edIdent] + del self.edges[edIdent] + del self.nodes[leaverIdent] + return out + + + def eventBlockDiscovery(self, discoIdent, t): + out = str(t) + ",blockDisco," + str(discoIdent) + "," + blockIdent = self.nodes[discoIdent].generateBlock(t) + out += str(blockIdent) + self.nodes[discoIdent].propagate(t, blockIdent) + return out + + def eventBlockArrival(self, destNodeIdent, edgeIdent, blockIdent, t): + out = str(t) + ",blockArriv," + str(destNodeIdent) + "," + str(edgeIdent) + "," + str(blockIdent) + "," + destNode = self.nodes[destNodeIdent] + edge = self.edges[edgeIdent] + block = deepcopy(edge.data["pendingBlocks"][blockIdent]) + block.arrivTimestamp = t + self.nodes[destNodeIdent].data["offset"] + self.nodes[destNodeIdent].updateBlockchain({blockIdent:block}) + return out + + def go(self): + with open(self.filename,"w") as writeFile: + writeFile.write("timestamp,eventId,eventData\n") + + while self.globalTime - self.startTime< self.runTime: + u = -1.0*math.log(1.0-random.random())/self.birthRate + eventType = ("nodeJoins", None) + + v = -1.0*math.log(1.0-random.random())/self.deathRate + if v < u: + eventType = ("nodeLeaves", None) + u = v + + for nodeIdent in self.nodes: + localBlockDiscoRate = self.nodes[nodeIdent].data["intensity"]/self.nodes[nodeIdent].data["blockchain"].diff + v = -1.0*math.log(1.0-random.random())/localBlockDiscoRate + if v < u: + eventType = ("blockDisco", nodeIdent) + u = v + + for edgeIdent in self.edges: + edge = self.edges[edgeIdent] + pB = edge.data["pendingBlocks"] + for pendingIdent in pB: + pendingData = pB[pendingIdent] # pendingDat = {"timeOfArrival":timeOfArrival, "destIdent":otherIdent, "block":blockToProp} + if pendingData["timeOfArrival"] - self.globalTime < u: + eventTime = ("blockArriv", (pendingData["destIdent"], edgeIdent, pendingData["block"])) + u = v + + self.globalTime += u + out = "" + if eventTime[0] == "nodeJoins": + out = self.eventNodeJoins(self.globalTime) + elif eventTime[0] == "nodeLeaves": + out = self.eventNodeLeaves(self.globalTime) + elif eventTime[0] == "blockDisco": + out = self.eventBlockDiscovery(eventTime[1], self.globalTime) + elif eventTime[0] == "blockArriv": + out = self.eventBlockArrival(eventTime[1], eventTime[2], eventTime[3], self.globalTime) + else: + print("WHAAAA") + + with open(self.filename, "a") as writeFile: + writeFile.write(out + "\n") + + + + + + + diff --git a/source-code/Poisson-Graphs/Node.py b/source-code/Poisson-Graphs/Node.py new file mode 100644 index 0000000..be9e9d9 --- /dev/null +++ b/source-code/Poisson-Graphs/Node.py @@ -0,0 +1,123 @@ +from Blockchain import * +from copy import * + +class Node(object): + ''' + Node object. params [identity, blockchain (data), verbosity, difficulty] + ''' + def __init__(self, params={}): + self.ident = None + self.data = {} + self.verbose = None + self.edges = {} + self.mode = None + self.targetRate = None + try: + assert len(params)==5 + except AssertionError: + print("Error, Tried to create malformed node.") + else: + self.ident = params["ident"] + self.data = params["data"] + self.verbose = params["verbose"] + self.edges = {} + self.mode = params["mode"] + self.targetRate = params["targetRate"] + + def generateBlock(self, discoTime): + newName = newIdent(len(self.data["blockchain"].blocks)) + t = discoTime + s = t+self.data["offset"] + diff = self.data["blockchain"].diff + params = {"ident":name, "disco":t, "arriv":s, "parent":None, "diff":diff} + newBlock = Block(params) + self.data["blockchain"].addBlock(newBlock, mode, tr) + return newName + + def updateBlockchain(self, incBlocks): + # incBlocks shall be a dictionary of block identities (as keys) and their associated blocks (as values) + # to be added to the local data. We assume difficulty scores have been reported honestly for now. + + tempData = deepcopy(incBlocks) + for key in incBlocks: + if key in self.data["blockchain"].blocks: + del tempData[key] + elif incBlocks[key].parent in self.data["blockchain"].blocks or incBlocks[key].parent is None: + self.data["blockchain"].addBlock(incBlocks[key], self.mode, self.targetRate) + del tempData[key] + incBlocks = deepcopy(tempData) + while len(incBlocks)>0: + for key in incBlocks: + if key in self.data["blockchain"].blocks: + del tempData[key] + elif incBlocks[key].parent in self.data["blockchain"].blocks: + self.data["blockchain"].addBlock(incBlocks[key], self.mode, self.targetRate) + del tempData[key] + incBlocks = deepcopy(tempData) + + def propagate(self, timeOfProp, blockIdent): + for edgeIdent in self.edges: + edge = self.edges[edgeIdent] + length = e.data["length"] + timeOfArrival = timeOfProp + length + otherIdent = e.getNeighbor(self.ident) + other = e.nodes[otherIdent] + bc = other.data["blockchain"] + if blockIdent not in bc.blocks: + pB = e.data["pendingBlocks"] + pendingIdent = newIdent(len(pB)) + mybc = self.data["blockchain"] + blockToProp = mybc.blocks[blockIdent] + pendingDat = {"timeOfArrival":timeOfArrival, "destIdent":otherIdent, "block":blockToProp} + pB.update({pendingIdent:pendingDat}) + + +class Test_Node(unittest.TestCase): + # TODO test each method separately + def test_all(self): + bill = Blockchain([], verbosity=True) + mode="Nakamoto" + tr = 1.0/600000.0 + deltaT = 600000.0 + bill.targetRate = tr + + name = newIdent(0) + t = 0.0 + s = t + diff = 1.0 + params = {"ident":name, "disco":t, "arriv":s, "parent":None, "diff":diff} + genesis = Block(params) + bill.addBlock(genesis, mode, tr) + + parent = genesis.ident + + nellyname = newIdent(time.time()) + mode = "Nakamoto" + targetRate = 1.0/600000.0 + params = {"ident":nellyname, "data":{"offset":0.0, "intensity":1.0, "blockchain":bill}, "verbose":True, "mode":mode, "targetRate":targetRate} + nelly = Node(params) + + while len(nelly.data["blockchain"].blocks) < 2015: + name = newIdent(len(nelly.data["blockchain"].blocks)) + diff = nelly.data["blockchain"].diff + t += deltaT*diff*(2.0*random.random()-1.0) + s = t + params = {"ident":name, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + nelly.updateBlockchain({newBlock.ident:newBlock}) + parent = name + + + while len(nelly.data["blockchain"].blocks) < 5000: + name = newIdent(len(nelly.data["blockchain"].blocks)) + diff = nelly.data["blockchain"].diff + t += deltaT*diff + s = t + params = {"ident":name, "disco":t, "arriv":s, "parent":parent, "diff":diff} + newBlock = Block(params) + nelly.updateBlockchain({newBlock.ident:newBlock}) + parent = name + +#suite = unittest.TestLoader().loadTestsFromTestCase(Test_Node) +#unittest.TextTestRunner(verbosity=1).run(suite) + diff --git a/source-code/StringCT-java/src/how/monero/hodl/bulletproof/Readme.md b/source-code/StringCT-java/src/how/monero/hodl/bulletproof/Readme.md index b15d86d..5fdb5d0 100644 --- a/source-code/StringCT-java/src/how/monero/hodl/bulletproof/Readme.md +++ b/source-code/StringCT-java/src/how/monero/hodl/bulletproof/Readme.md @@ -1,3 +1,2 @@ -Sarang and Moneromooo implement bulletproofs. -Usually 90+% space complexity savings, up to 25% verification time complexity savings over traditional range proofs. +Sarang and Moneromoo implement bulletproofs.