Intro #
If you have ever needed the full history of Ethereum call traces, you know the problem: they are not stored anywhere.
Logs are easy, they sit in the receipts. Internal calls do not. A value transfer that never shows up as its own transaction, a delegatecall three frames deep, a storage slot a contract read before it decided what to do. None of that is in any database. The only way to see it is to execute the block.
Lately I had a task that needed exactly that, for every block since genesis.
The standard answer is to run an archive node.
Why I Did Not Want an Archive Node #
An archive node keeps every intermediate state the chain has ever been in. You can go back to any block, reconstruct the world as it was at that moment, and re-execute against it.
That is a great capability. It is also a lot of disk, a sync measured in weeks, and a machine you have to keep running afterwards. Renting one is not cheap either.
But the part that bothered me was different. What an archive node really sells you is the ability to ask a question you have not thought of yet. That is why it keeps everything, it does not know what you will want.
I knew what I wanted. Call traces, every block, once, written somewhere else. The generality was the expensive part and I was going to throw it away.
The Observation #
Then something fairly obvious hit me.
A full sync from genesis executes every single block. It has to. That is what execution is: take the state after block N, apply the transactions of block N+1, get the state after block N+1. There is no shortcut and no sampling.
So every intermediate state gets built anyway, in order, on every syncing node. An archive node does not see more history than a full node. It just refuses to throw the state away as it goes past.
Which means I did not need to store any state at all. I only needed to be there at the right moment, while the block was being executed.
Patching Geth #
Geth is already doing the execution. I only needed somewhere to put my code.
That place is writeHeadBlock in core/blockchain.go, which runs when a block becomes the new head. Everything I needed goes at the end of it:
func (bc *BlockChain) writeHeadBlock(block *types.Block) {
// ... existing head-writing logic ...
bc.currentBlock.Store(block.Header())
headBlockGauge.Update(int64(block.NumberU64()))
// add your block processing here!
}
What makes that spot useful is not the block, you can get a block anywhere. It is that the state the block was executed against is still sitting in the trie database. Nothing has to be reconstructed. It is just there, for as long as you are standing in that function.
So you can run whatever tracers you like over the block right there and store the results somewhere else. I used mostly the ones geth already ships, callTracer with logs enabled and prestateTracer in diff mode, plus two I wrote for this job. One of them, the keccak256 preimage tracer, is in go-ethereum upstream now.
Why It Blocks #
Whatever you put there runs synchronously. The sync does not continue until it is done.
That looks like a mistake, and it is the most important part of the whole thing.
Hand the block to a worker and return immediately, and the sync runs ahead. Tracing falls behind. Then the state your tracer needs gets pruned while the tracer is still using it. The obvious fix is to hold that state until the tracer catches up, and congratulations, you have just written an archive node by accident.
Blocking removes the race by removing the concurrency. The sync runs exactly as fast as tracing allows. That is slower, and that is the bill. In exchange the tracer always looks at state that is guaranteed to still be there.
You also get backpressure for free. If whatever you write the traces to gets slow, your code takes longer to return, and the sync slows down with it. Nothing piles up in memory and nothing quietly drops a block.
Reorgs #
Backfilling old blocks and following the head look like two different problems. Old blocks are settled history. The head reorgs, so a block you just traced may turn out to have never happened.
It turns out you do not have to do anything about it.
writeHeadBlock runs whenever a block becomes the head, and on a reorg geth rewinds the head to the common ancestor and then writes the new canonical chain forward. So your code simply runs again with a block number lower than the one it saw a moment ago, and walks forward from there.
Seen from inside that function, a reorg is just the block number jumping backwards and some blocks being processed a second time.
That does move the problem downstream. Whatever consumes the traces has to key on block number and hash, not on the number alone, and be ready to see the same number again with a different hash. In exchange the node side stays boring and there is only one code path for backfill and head following.
What It Cost #
Two weeks, on 8 cores and 32 GB of RAM.
The two weeks is not the interesting number. A plain full sync is faster, and the tracing is exactly why this one was not.
The machine is the interesting number. 8 cores and 32 GB is nothing special. Try running a full archive node on that.
When it finished I had the traces and no node left to keep alive.
Doing It Again, In Parallel #
Two weeks is only painful because it is two weeks of one machine doing one thing. That is the part worth fixing, and it does not take much.
So the harvest takes checkpoints. Every few million blocks the node stops and keeps a copy of the datadir. That is an ordinary full node state at that height, not an archive, and there are only a handful of them.
With those checkpoints a re-run does not have to start at genesis. Start one node from each checkpoint, let it process forward only until the next checkpoint, and stop it there. The segments do not depend on each other, because the state each one needs is already in the snapshot it started from. Run them side by side and the chain gets re-traced all at once instead of in order.
One thing has to be right for this to pay off: block ranges are not equal work. Early blocks are almost free, recent ones are full of gas. Checkpointing every 5M blocks would give segments with very different runtimes, and a parallel run only takes as long as its slowest segment. So the checkpoints go in by cumulative gas rather than by block number, which makes the segments come out roughly even.
The price is disk for the snapshots, and taking them during the first harvest, when you are not yet thinking about the second one.
Epilogue #
I went into this thinking the catch was that you only get one pass, and that needing another tracer six months later would cost you two weeks all over again.
With checkpoints that is not really true any more. A new tracer costs a re-run, and a re-run is a handful of machines working at the same time instead of one machine working for two weeks.
What is left is a difference in latency, not in what you can find out. An archive node answers a question you just thought of in about a second. This one makes you wait for a re-run.
So if you are still exploring and the questions keep changing, get the archive node. Fast answers to unplanned questions is the thing you are actually buying.
But if you do know, and the answer is “all of it, once”, then an archive node is a very expensive way to avoid adding a few lines to geth.