One of the important issues that has been brought up over the course of the Olympic stress-net release is the large amount of data that clients are required to store; over little more than three months of operation, and particularly during the last month, the amount of data in each Ethereum client’s blockchain folder has ballooned to an impressive 10-40 gigabytes, depending on which client you are using and whether or not compression is enabled. Although it is important to note that this is indeed a stress test scenario where users are incentivized to dump transactions on the blockchain paying only the free test-ether as a transaction fee, and transaction throughput levels are thus several times higher than Bitcoin, it is nevertheless a legitimate concern for users, who in many cases do not have hundreds of gigabytes to spare on storing other people’s transaction histories.<\/p>\n
First of all, let us begin by exploring why the current Ethereum client database is so large. Ethereum, unlike Bitcoin, has the property that every block contains something called the “state root”: the root hash of a specialized kind of Merkle tree<\/a> which stores the entire state of the system: all account balances, contract storage, contract code and account nonces are inside.<\/p>\n The purpose of this is simple: it allows a node given only the last block, together with some assurance that the last block actually is the most recent block, to “synchronize” with the blockchain extremely quickly without processing any historical transactions, by simply downloading the rest of the tree from nodes in the network (the proposed HashLookup<\/span> wire protocol message<\/a> will faciliate this), verifying that the tree is correct by checking that all of the hashes match up, and then proceeding from there. In a fully decentralized context, this will likely be done through an advanced version of Bitcoin’s headers-first-verification strategy, which will look roughly as follows:<\/p>\n For light clients, the state root is even more advantageous: they can immediately determine the exact balance and status of any account by simply asking the network for a particular branch<\/em> of the tree, without needing to follow Bitcoin’s multi-step 1-of-N “ask for all transaction outputs, then ask for all transactions spending those outputs, and take the remainder” light-client model.<\/p>\n However, this state tree mechanism has an important disadvantage if implemented naively: the intermediate nodes in the tree greatly increase the amount of disk space required to store all the data. To see why, consider this diagram here:<\/p>\n The change in the tree during each individual block is fairly small, and the magic of the tree as a data structure is that most of the data can simply be referenced twice without being copied. However, even still, for every change to the state that is made, a logarithmically large number of nodes (ie. ~5 at 1000 nodes, ~10 at 1000000 nodes, ~15 at 1000000000 nodes) need to be stored twice, one version for the old tree and one version for the new trie. Eventually, as a node processes every block, we can thus expect the total disk space utilization to be, in computer science terms, roughly O(n*log(n))<\/span>, where n<\/span> is the transaction load. In practical terms, the Ethereum blockchain is only 1.3 gigabytes, but the size of the database including all these extra nodes is 10-40 gigabytes.<\/p>\n So, what can we do? One backward-looking fix is to simply go ahead and implement headers-first syncing, essentially resetting new users’ hard disk consumption to zero, and allowing users to keep their hard disk consumption low by re-syncing every one or two months, but that is a somewhat ugly solution. The alternative approach is to implement state tree pruning<\/em>: essentially, use reference counting<\/a> to track when nodes in the tree (here using “node” in the computer-science term meaning “piece of data that is somewhere in a graph or tree structure”, not “computer on the network”) drop out of the tree, and at that point put them on “death row”: unless the node somehow becomes used again within the next X<\/span> blocks (eg. X = 5000<\/span>), after that number of blocks pass the node should be permanently deleted from the database. Essentially, we store the tree nodes that are part of the current state, and we even store recent history, but we do not store history older than 5000 blocks.<\/p>\n X<\/span> should be set as low as possible to conserve space, but setting X<\/span> too low compromises robustness: once this technique is implemented, a node cannot revert back more than X<\/span> blocks without essentially completely restarting synchronization. Now, let’s see how this approach can be implemented fully, taking into account all of the corner cases:<\/p>\n Once this is done, the database should only be storing state nodes associated with the last X<\/span> blocks, so you will still have all the information you need from those blocks but nothing more. On top of this, there are further optimizations. Particularly, after X<\/span> blocks, transaction and receipt trees should be deleted entirely, and even blocks may arguably be deleted as well – although there is an important argument for keeping some subset of “archive nodes” that store absolutely everything so as to help the rest of the network acquire the data that it needs.<\/p>\n Now, how much savings can this give us? As it turns out, quite a lot! Particularly, if we were to take the ultimate daredevil route and go X = 0<\/span> (ie. lose absolutely all ability to handle even single-block forks, storing no history whatsoever), then the size of the database would essentially be the size of the state: a value which, even now (this data was grabbed at block 670000) stands at roughly 40 megabytes – the majority of which is made up of accounts like this one<\/a> with storage slots filled to deliberately spam the network. At X = 100000<\/span>, we would get essentially the current size of 10-40 gigabytes, as most of the growth happened in the last hundred thousand blocks, and the extra space required for storing journals and death row lists would make up the rest of the difference. At every value in between, we can expect the disk space growth to be linear (ie. X = 10000<\/span> would take us about ninety percent of the way there to near-zero).<\/p>\n Note that we may want to pursue a hybrid strategy: keeping every block<\/em> but not every state tree node<\/em>; in this case, we would need to add roughly 1.4 gigabytes to store the block data. It’s important to note that the cause of the blockchain size is NOT fast block times; currently, the block headers of the last three months make up roughly 300 megabytes, and the rest is transactions of the last one month, so at high levels of usage we can expect to continue to see transactions dominate. That said, light clients will also need to prune block headers if they are to survive in low-memory circumstances.<\/p>\n The strategy described above has been implemented in a very early alpha form in pyeth<\/a>; it will be implemented properly in all clients in due time after Frontier launches, as such storage bloat is only a medium-term and not a short-term scalability concern.<\/p>\n<\/div>\n
\n![](\"https:\/\/blog.ethereum.org\/images\/posts\/2015\/06\/ethblockchain_oneblock.png\")
\n<\/center><\/p>\n\n
\n![](\"https:\/\/blog.ethereum.org\/images\/posts\/2015\/06\/ethblockchain.png\")
\n<\/center><\/p>\n\n