Use hash writer for aristo computeKey

[chirag-parmar]

closed duplicate: #3226
fixes: #3164

Uses the rlp hash writer for Aristo DB's computeKey implementation

[arnetheduck]

doesn't this call to wrapEnconding depend on leafData being of a particular type? if yes, why use auto?

[bhartnett]

I'm not sure about wrapEncoding but the reason I changed this line was that we are passing a type account or slot and don't need to pass in a block of statements. Perhaps it would be better to make it even more strict and use Account | UInt256 as the type.

[arnetheduck]

ok - it feels odd that it should be needed cc @chirag-parmar - ie append should be able to figure this out ideally - but it seems to work, so ..

[chirag-parmar]

Apologies for the late reply. the call to wrapEncoding doesn't depend on the leafData being of a particular type. Only leaf nodes have nested serializations i.e. Serializing an object/list/tuple that already has a serialized rlp value in it.

[chirag-parmar]

append should be able to figure this out ideally

I didn't understand what it must be able to figure out.

[arnetheduck]

I think what I would have expected maybe is a call like appendEncoded(leafData) which does the extra round of encoding of the passed-in item - wrapEncoding looks like an internal low-level operation, specially with that number being passed in, because presumably it must be followed up with actual data - that gives the caller an unnecessary opportunity to use it the wrong way

[chirag-parmar]

makes sense actually. status-im/nim-eth#813

[arnetheduck]

so .. the final thing to do with this branch is to run a benchmark: using an unhashed snapshot like https://eth1-db.nimbus.team/mainnet-20250831-nohash.tar.gz, run the node pre/post to measure how long it takes to create the full hash tree (it's printed when starting the node) - afair, 25% of cpu time was spent allocating rlp encodings, so we should see a good boost here.

[bhartnett]

so .. the final thing to do with this branch is to run a benchmark: using an unhashed snapshot like https://eth1-db.nimbus.team/mainnet-20250831-nohash.tar.gz, run the node pre/post to measure how long it takes to create the full hash tree (it's printed when starting the node) - afair, 25% of cpu time was spent allocating rlp encodings, so we should see a good boost here.

I ran a test using an unhashed snapshot I have from just after the merge at around block 15000000. Here are the results.

When running the baseline build from master:

NTC 2025-09-15 16:20:38.534+08:00 Importing era archive blockNumber=15537394 dataDir=/home/user/.local/state/nimbus/mainnet eraDir=/mnt/5aa6b1af-8122-4eec-b52f-bdfb121e74de/era/mainnet INF 2025-09-15 16:21:44.447+08:00 Writing computeKey cache keys=2139900 accounts=0.22% INF 2025-09-15 16:22:52.479+08:00 Writing computeKey cache keys=4279800 accounts=0.91% INF 2025-09-15 16:23:57.324+08:00 Writing computeKey cache keys=6419700 accounts=1.85% INF 2025-09-15 16:24:58.980+08:00 Writing computeKey cache keys=8559600 accounts=2.67% INF 2025-09-15 16:26:02.630+08:00 Writing computeKey cache keys=10699500 accounts=3.66% INF 2025-09-15 16:26:55.133+08:00 Writing computeKey cache keys=12839400 accounts=4.09% INF 2025-09-15 16:27:52.876+08:00 Writing computeKey cache keys=14979300 accounts=4.53% INF 2025-09-15 16:28:55.905+08:00 Writing computeKey cache keys=17119200 accounts=5.27% INF 2025-09-15 16:29:58.874+08:00 Writing computeKey cache keys=19259100 accounts=5.47% INF 2025-09-15 16:30:50.617+08:00 Writing computeKey cache keys=21399000 accounts=5.52% INF 2025-09-15 16:31:44.304+08:00 Writing computeKey cache keys=23538900 accounts=6.13% INF 2025-09-15 16:32:42.459+08:00 Writing computeKey cache keys=25678800 accounts=6.75% INF 2025-09-15 16:33:39.740+08:00 Writing computeKey cache keys=27818700 accounts=7.43% INF 2025-09-15 16:34:49.518+08:00 Writing computeKey cache keys=29958600 accounts=7.77% INF 2025-09-15 16:35:42.001+08:00 Writing computeKey cache keys=32098500 accounts=7.77% INF 2025-09-15 16:36:33.841+08:00 Writing computeKey cache keys=34238400 accounts=8.32% INF 2025-09-15 16:37:35.051+08:00 Writing computeKey cache keys=36378300 accounts=9.39% INF 2025-09-15 16:38:40.428+08:00 Writing computeKey cache keys=38518200 accounts=10.28% INF 2025-09-15 16:39:32.161+08:00 Writing computeKey cache keys=40658100 accounts=10.58% INF 2025-09-15 16:40:25.481+08:00 Writing computeKey cache keys=42798000 accounts=11.07% INF 2025-09-15 16:41:25.258+08:00 Writing computeKey cache keys=44937900 accounts=11.55% INF 2025-09-15 16:42:24.596+08:00 Writing computeKey cache keys=47077800 accounts=12.50% INF 2025-09-15 16:43:28.296+08:00 Writing computeKey cache keys=49217700 accounts=13.41% INF 2025-09-15 16:44:31.012+08:00 Writing computeKey cache keys=51357600 accounts=14.25% INF 2025-09-15 16:45:32.914+08:00 Writing computeKey cache keys=53497500 accounts=15.06% INF 2025-09-15 16:46:27.690+08:00 Writing computeKey cache keys=55637400 accounts=15.50% INF 2025-09-15 16:47:20.456+08:00 Writing computeKey cache keys=57777300 accounts=16.22% INF 2025-09-15 16:48:18.473+08:00 Writing computeKey cache keys=59917200 accounts=16.95% INF 2025-09-15 16:49:20.112+08:00 Writing computeKey cache keys=62057100 accounts=17.72% INF 2025-09-15 16:50:20.016+08:00 Writing computeKey cache keys=64197000 accounts=18.66% INF 2025-09-15 16:51:20.044+08:00 Writing computeKey cache keys=66336900 accounts=19.43% INF 2025-09-15 16:52:26.452+08:00 Writing computeKey cache keys=68476800 accounts=20.11% INF 2025-09-15 16:53:29.258+08:00 Writing computeKey cache keys=70616700 accounts=20.99% INF 2025-09-15 16:54:26.445+08:00 Writing computeKey cache keys=72756600 accounts=21.73% INF 2025-09-15 16:55:21.410+08:00 Writing computeKey cache keys=74896500 accounts=22.67% INF 2025-09-15 16:56:18.622+08:00 Writing computeKey cache keys=77036400 accounts=23.21% INF 2025-09-15 16:57:22.420+08:00 Writing computeKey cache keys=79176300 accounts=24.22% INF 2025-09-15 16:58:21.585+08:00 Writing computeKey cache keys=81316200 accounts=24.66% INF 2025-09-15 16:59:18.477+08:00 Writing computeKey cache keys=83456100 accounts=25.38% INF 2025-09-15 17:00:16.475+08:00 Writing computeKey cache keys=85596000 accounts=26.30% INF 2025-09-15 17:01:19.806+08:00 Writing computeKey cache keys=87735900 accounts=27.26% INF 2025-09-15 17:02:20.428+08:00 Writing computeKey cache keys=89875800 accounts=27.95% INF 2025-09-15 17:03:11.878+08:00 Writing computeKey cache keys=92015700 accounts=28.40% INF 2025-09-15 17:04:12.591+08:00 Writing computeKey cache keys=94155600 accounts=29.06% INF 2025-09-15 17:05:15.458+08:00 Writing computeKey cache keys=96295500 accounts=30.06% INF 2025-09-15 17:06:28.405+08:00 Writing computeKey cache keys=98435400 accounts=30.64% INF 2025-09-15 17:07:17.784+08:00 Writing computeKey cache keys=100575300 accounts=30.64% INF 2025-09-15 17:08:07.956+08:00 Writing computeKey cache keys=102715200 accounts=30.64% INF 2025-09-15 17:09:07.167+08:00 Writing computeKey cache keys=104855100 accounts=31.06% INF 2025-09-15 17:10:03.867+08:00 Writing computeKey cache keys=106995000 accounts=31.81% INF 2025-09-15 17:11:02.362+08:00 Writing computeKey cache keys=109134900 accounts=32.48% INF 2025-09-15 17:12:08.893+08:00 Writing computeKey cache keys=111274800 accounts=33.47% INF 2025-09-15 17:13:13.160+08:00 Writing computeKey cache keys=113414700 accounts=34.07% INF 2025-09-15 17:14:12.160+08:00 Writing computeKey cache keys=115554600 accounts=35.00% INF 2025-09-15 17:15:17.992+08:00 Writing computeKey cache keys=117694500 accounts=35.90% INF 2025-09-15 17:16:21.763+08:00 Writing computeKey cache keys=119834400 accounts=36.93% INF 2025-09-15 17:17:21.050+08:00 Writing computeKey cache keys=121974300 accounts=37.50% INF 2025-09-15 17:18:25.373+08:00 Writing computeKey cache keys=124114200 accounts=38.28% INF 2025-09-15 17:19:29.537+08:00 Writing computeKey cache keys=126254100 accounts=39.00% INF 2025-09-15 17:20:23.648+08:00 Writing computeKey cache keys=128394000 accounts=39.90% INF 2025-09-15 17:21:28.039+08:00 Writing computeKey cache keys=130533900 accounts=40.72% INF 2025-09-15 17:22:29.183+08:00 Writing computeKey cache keys=132673800 accounts=41.50% INF 2025-09-15 17:23:24.773+08:00 Writing computeKey cache keys=134813700 accounts=41.84% INF 2025-09-15 17:24:26.209+08:00 Writing computeKey cache keys=136953600 accounts=42.66% INF 2025-09-15 17:25:29.146+08:00 Writing computeKey cache keys=139093500 accounts=43.14% INF 2025-09-15 17:26:29.551+08:00 Writing computeKey cache keys=141233400 accounts=43.76% INF 2025-09-15 17:27:19.604+08:00 Writing computeKey cache keys=143373300 accounts=44.07% INF 2025-09-15 17:28:15.755+08:00 Writing computeKey cache keys=145513200 accounts=44.63% INF 2025-09-15 17:29:13.403+08:00 Writing computeKey cache keys=147653100 accounts=45.36% INF 2025-09-15 17:30:11.022+08:00 Writing computeKey cache keys=149793000 accounts=45.69% INF 2025-09-15 17:31:15.566+08:00 Writing computeKey cache keys=151932900 accounts=46.78% INF 2025-09-15 17:32:16.083+08:00 Writing computeKey cache keys=154072800 accounts=47.48% INF 2025-09-15 17:33:22.357+08:00 Writing computeKey cache keys=156212700 accounts=48.18% INF 2025-09-15 17:34:19.229+08:00 Writing computeKey cache keys=158352600 accounts=48.63% INF 2025-09-15 17:35:27.419+08:00 Writing computeKey cache keys=160492500 accounts=49.55% INF 2025-09-15 17:36:29.622+08:00 Writing computeKey cache keys=162632400 accounts=50.41% INF 2025-09-15 17:37:33.649+08:00 Writing computeKey cache keys=164772300 accounts=51.12% INF 2025-09-15 17:38:29.502+08:00 Writing computeKey cache keys=166912200 accounts=51.85% INF 2025-09-15 17:39:33.398+08:00 Writing computeKey cache keys=169052100 accounts=52.53% INF 2025-09-15 17:40:32.744+08:00 Writing computeKey cache keys=171192000 accounts=53.15% INF 2025-09-15 17:41:30.841+08:00 Writing computeKey cache keys=173331900 accounts=53.63% INF 2025-09-15 17:42:29.521+08:00 Writing computeKey cache keys=175471800 accounts=54.44% INF 2025-09-15 17:43:28.183+08:00 Writing computeKey cache keys=177611700 accounts=55.01% INF 2025-09-15 17:44:26.181+08:00 Writing computeKey cache keys=179751600 accounts=55.70% INF 2025-09-15 17:45:28.796+08:00 Writing computeKey cache keys=181891500 accounts=56.61% INF 2025-09-15 17:46:30.381+08:00 Writing computeKey cache keys=184031400 accounts=57.40% INF 2025-09-15 17:47:30.382+08:00 Writing computeKey cache keys=186171300 accounts=58.21% INF 2025-09-15 17:48:33.944+08:00 Writing computeKey cache keys=188311200 accounts=59.08% INF 2025-09-15 17:49:32.179+08:00 Writing computeKey cache keys=190451100 accounts=60.00% INF 2025-09-15 17:50:45.305+08:00 Writing computeKey cache keys=192591000 accounts=60.80% INF 2025-09-15 17:51:34.913+08:00 Writing computeKey cache keys=194730900 accounts=60.85% INF 2025-09-15 17:52:37.564+08:00 Writing computeKey cache keys=196870800 accounts=61.65% INF 2025-09-15 17:53:45.932+08:00 Writing computeKey cache keys=199010700 accounts=62.14% INF 2025-09-15 17:54:36.760+08:00 Writing computeKey cache keys=201150600 accounts=62.14% INF 2025-09-15 17:55:33.372+08:00 Writing computeKey cache keys=203290500 accounts=62.50% INF 2025-09-15 17:56:40.304+08:00 Writing computeKey cache keys=205430400 accounts=63.33% INF 2025-09-15 17:57:42.183+08:00 Writing computeKey cache keys=207570300 accounts=63.93% INF 2025-09-15 17:58:48.725+08:00 Writing computeKey cache keys=209710200 accounts=64.54% INF 2025-09-15 17:59:40.988+08:00 Writing computeKey cache keys=211850100 accounts=64.63% INF 2025-09-15 18:00:39.082+08:00 Writing computeKey cache keys=213990000 accounts=65.42% INF 2025-09-15 18:01:47.511+08:00 Writing computeKey cache keys=216129900 accounts=66.34% INF 2025-09-15 18:02:53.625+08:00 Writing computeKey cache keys=218269800 accounts=66.83% INF 2025-09-15 18:03:45.499+08:00 Writing computeKey cache keys=220409700 accounts=67.15% INF 2025-09-15 18:04:46.975+08:00 Writing computeKey cache keys=222549600 accounts=67.67% INF 2025-09-15 18:05:39.531+08:00 Writing computeKey cache keys=224689500 accounts=68.28% INF 2025-09-15 18:06:41.164+08:00 Writing computeKey cache keys=226829400 accounts=69.28% INF 2025-09-15 18:07:36.271+08:00 Writing computeKey cache keys=228969300 accounts=69.45% INF 2025-09-15 18:08:37.445+08:00 Writing computeKey cache keys=231109200 accounts=70.27% INF 2025-09-15 18:09:42.609+08:00 Writing computeKey cache keys=233249100 accounts=71.29% INF 2025-09-15 18:10:45.241+08:00 Writing computeKey cache keys=235389000 accounts=72.03% INF 2025-09-15 18:11:43.244+08:00 Writing computeKey cache keys=237528900 accounts=72.72% INF 2025-09-15 18:12:47.213+08:00 Writing computeKey cache keys=239668800 accounts=73.52% INF 2025-09-15 18:13:53.375+08:00 Writing computeKey cache keys=241808700 accounts=74.11% INF 2025-09-15 18:14:52.007+08:00 Writing computeKey cache keys=243948600 accounts=74.88% INF 2025-09-15 18:15:52.173+08:00 Writing computeKey cache keys=246088500 accounts=75.82% INF 2025-09-15 18:16:50.155+08:00 Writing computeKey cache keys=248228400 accounts=76.05% INF 2025-09-15 18:17:43.067+08:00 Writing computeKey cache keys=250368300 accounts=76.89% INF 2025-09-15 18:18:46.327+08:00 Writing computeKey cache keys=252508200 accounts=77.57% INF 2025-09-15 18:19:48.340+08:00 Writing computeKey cache keys=254648100 accounts=78.24% INF 2025-09-15 18:20:50.735+08:00 Writing computeKey cache keys=256788000 accounts=79.16% INF 2025-09-15 18:21:56.493+08:00 Writing computeKey cache keys=258927900 accounts=80.04% INF 2025-09-15 18:23:02.936+08:00 Writing computeKey cache keys=261067800 accounts=80.91% INF 2025-09-15 18:24:07.940+08:00 Writing computeKey cache keys=263207700 accounts=81.48% INF 2025-09-15 18:25:00.869+08:00 Writing computeKey cache keys=265347600 accounts=81.86% INF 2025-09-15 18:25:56.763+08:00 Writing computeKey cache keys=267487500 accounts=82.74% INF 2025-09-15 18:26:59.555+08:00 Writing computeKey cache keys=269627400 accounts=83.19% INF 2025-09-15 18:27:54.809+08:00 Writing computeKey cache keys=271767300 accounts=83.95% INF 2025-09-15 18:28:54.713+08:00 Writing computeKey cache keys=273907200 accounts=84.66% INF 2025-09-15 18:30:01.365+08:00 Writing computeKey cache keys=276047100 accounts=85.47% INF 2025-09-15 18:30:59.768+08:00 Writing computeKey cache keys=278187000 accounts=86.30% INF 2025-09-15 18:32:00.589+08:00 Writing computeKey cache keys=280326900 accounts=86.99% INF 2025-09-15 18:33:03.228+08:00 Writing computeKey cache keys=282466800 accounts=87.86% INF 2025-09-15 18:33:58.248+08:00 Writing computeKey cache keys=284606700 accounts=88.48% INF 2025-09-15 18:35:06.039+08:00 Writing computeKey cache keys=286746600 accounts=89.29% INF 2025-09-15 18:36:00.328+08:00 Writing computeKey cache keys=288886500 accounts=89.51% INF 2025-09-15 18:37:01.847+08:00 Writing computeKey cache keys=291026400 accounts=90.32% INF 2025-09-15 18:38:06.135+08:00 Writing computeKey cache keys=293166300 accounts=91.19% INF 2025-09-15 18:39:07.796+08:00 Writing computeKey cache keys=295306200 accounts=91.63% INF 2025-09-15 18:39:58.331+08:00 Writing computeKey cache keys=297446100 accounts=91.96% INF 2025-09-15 18:40:55.568+08:00 Writing computeKey cache keys=299586000 accounts=92.54% INF 2025-09-15 18:41:55.021+08:00 Writing computeKey cache keys=301725900 accounts=93.15% INF 2025-09-15 18:43:00.918+08:00 Writing computeKey cache keys=303865800 accounts=94.15% INF 2025-09-15 18:44:03.171+08:00 Writing computeKey cache keys=306005700 accounts=95.01% INF 2025-09-15 18:45:03.879+08:00 Writing computeKey cache keys=308145600 accounts=95.87% INF 2025-09-15 18:45:59.104+08:00 Writing computeKey cache keys=310285500 accounts=96.87% INF 2025-09-15 18:47:02.770+08:00 Writing computeKey cache keys=312425400 accounts=97.66% INF 2025-09-15 18:48:06.319+08:00 Writing computeKey cache keys=314565300 accounts=98.37% INF 2025-09-15 18:48:57.134+08:00 Writing computeKey cache keys=316705200 accounts=98.57% INF 2025-09-15 18:50:10.355+08:00 Writing computeKey cache keys=318845100 accounts=99.26% INF 2025-09-15 18:50:59.066+08:00 Writing computeKey cache keys=320985000 accounts=99.26% INF 2025-09-15 18:51:47.675+08:00 Writing computeKey cache keys=323124900 accounts=99.26% INF 2025-09-15 18:52:36.229+08:00 Writing computeKey cache keys=325264800 accounts=99.26% INF 2025-09-15 18:53:35.329+08:00 Writing computeKey cache keys=327404700 accounts=99.85% INF 2025-09-15 18:53:50.773+08:00 Wrote computeKey cache keys=328002861 accounts=100.00% INF 2025-09-15 18:53:57.772+08:00 Imported blocks blockNumber=15537404 slot=4700023 blocks=10 txs=1250 mgas=249.2 bps=0.001 tps=0.136 mgps=0.027 avgBps=0.001 avgTps=0.136 avgMGps=0.027 elapsed=2h33m19s NTC 2025-09-15 18:53:57.772+08:00 Import complete blockNumber=15537404 slot=4700023 blocks=10 txs=1250 mgas=249.2 

When running the hash writer build from the branch:

NTC 2025-09-15 19:26:20.563+08:00 Importing era archive blockNumber=15537394 dataDir=/home/user/.local/state/nimbus/mainnet eraDir=/mnt/5aa6b1af-8122-4eec-b52f-bdfb121e74de/era/mainnet INF 2025-09-15 19:27:37.729+08:00 Writing computeKey cache keys=2139900 accounts=0.22% INF 2025-09-15 19:28:49.592+08:00 Writing computeKey cache keys=4279800 accounts=0.91% INF 2025-09-15 19:30:08.960+08:00 Writing computeKey cache keys=6419700 accounts=1.85% INF 2025-09-15 19:31:14.136+08:00 Writing computeKey cache keys=8559600 accounts=2.67% INF 2025-09-15 19:32:22.474+08:00 Writing computeKey cache keys=10699500 accounts=3.66% INF 2025-09-15 19:33:26.324+08:00 Writing computeKey cache keys=12839400 accounts=4.09% INF 2025-09-15 19:34:33.946+08:00 Writing computeKey cache keys=14979300 accounts=4.53% INF 2025-09-15 19:35:42.955+08:00 Writing computeKey cache keys=17119200 accounts=5.27% INF 2025-09-15 19:36:51.182+08:00 Writing computeKey cache keys=19259100 accounts=5.47% INF 2025-09-15 19:37:52.208+08:00 Writing computeKey cache keys=21399000 accounts=5.52% INF 2025-09-15 19:38:56.242+08:00 Writing computeKey cache keys=23538900 accounts=6.13% INF 2025-09-15 19:40:01.120+08:00 Writing computeKey cache keys=25678800 accounts=6.75% INF 2025-09-15 19:41:12.641+08:00 Writing computeKey cache keys=27818700 accounts=7.43% INF 2025-09-15 19:42:29.520+08:00 Writing computeKey cache keys=29958600 accounts=7.77% INF 2025-09-15 19:43:28.824+08:00 Writing computeKey cache keys=32098500 accounts=7.77% INF 2025-09-15 19:44:35.461+08:00 Writing computeKey cache keys=34238400 accounts=8.32% INF 2025-09-15 19:45:47.644+08:00 Writing computeKey cache keys=36378300 accounts=9.39% INF 2025-09-15 19:47:04.209+08:00 Writing computeKey cache keys=38518200 accounts=10.28% INF 2025-09-15 19:48:04.754+08:00 Writing computeKey cache keys=40658100 accounts=10.58% INF 2025-09-15 19:49:09.813+08:00 Writing computeKey cache keys=42798000 accounts=11.07% INF 2025-09-15 19:50:20.992+08:00 Writing computeKey cache keys=44937900 accounts=11.55% INF 2025-09-15 19:51:25.415+08:00 Writing computeKey cache keys=47077800 accounts=12.50% INF 2025-09-15 19:52:36.620+08:00 Writing computeKey cache keys=49217700 accounts=13.41% INF 2025-09-15 19:53:46.802+08:00 Writing computeKey cache keys=51357600 accounts=14.25% INF 2025-09-15 19:54:50.010+08:00 Writing computeKey cache keys=53497500 accounts=15.06% INF 2025-09-15 19:55:54.526+08:00 Writing computeKey cache keys=55637400 accounts=15.50% INF 2025-09-15 19:57:02.787+08:00 Writing computeKey cache keys=57777300 accounts=16.22% INF 2025-09-15 19:58:09.101+08:00 Writing computeKey cache keys=59917200 accounts=16.95% INF 2025-09-15 19:59:25.804+08:00 Writing computeKey cache keys=62057100 accounts=17.72% INF 2025-09-15 20:00:36.530+08:00 Writing computeKey cache keys=64197000 accounts=18.66% INF 2025-09-15 20:01:42.061+08:00 Writing computeKey cache keys=66336900 accounts=19.43% INF 2025-09-15 20:02:50.430+08:00 Writing computeKey cache keys=68476800 accounts=20.11% INF 2025-09-15 20:04:04.884+08:00 Writing computeKey cache keys=70616700 accounts=20.99% INF 2025-09-15 20:05:21.246+08:00 Writing computeKey cache keys=72756600 accounts=21.73% INF 2025-09-15 20:06:34.855+08:00 Writing computeKey cache keys=74896500 accounts=22.67% INF 2025-09-15 20:07:37.878+08:00 Writing computeKey cache keys=77036400 accounts=23.21% INF 2025-09-15 20:08:52.893+08:00 Writing computeKey cache keys=79176300 accounts=24.22% INF 2025-09-15 20:10:00.510+08:00 Writing computeKey cache keys=81316200 accounts=24.66% INF 2025-09-15 20:11:10.486+08:00 Writing computeKey cache keys=83456100 accounts=25.38% INF 2025-09-15 20:12:28.824+08:00 Writing computeKey cache keys=85596000 accounts=26.30% INF 2025-09-15 20:13:43.746+08:00 Writing computeKey cache keys=87735900 accounts=27.26% INF 2025-09-15 20:14:55.222+08:00 Writing computeKey cache keys=89875800 accounts=27.95% INF 2025-09-15 20:16:00.719+08:00 Writing computeKey cache keys=92015700 accounts=28.40% INF 2025-09-15 20:17:09.686+08:00 Writing computeKey cache keys=94155600 accounts=29.06% INF 2025-09-15 20:18:29.257+08:00 Writing computeKey cache keys=96295500 accounts=30.06% INF 2025-09-15 20:19:41.840+08:00 Writing computeKey cache keys=98435400 accounts=30.64% INF 2025-09-15 20:20:41.238+08:00 Writing computeKey cache keys=100575300 accounts=30.64% INF 2025-09-15 20:21:38.118+08:00 Writing computeKey cache keys=102715200 accounts=30.64% INF 2025-09-15 20:22:42.010+08:00 Writing computeKey cache keys=104855100 accounts=31.06% INF 2025-09-15 20:23:55.764+08:00 Writing computeKey cache keys=106995000 accounts=31.81% INF 2025-09-15 20:25:09.835+08:00 Writing computeKey cache keys=109134900 accounts=32.48% INF 2025-09-15 20:26:13.771+08:00 Writing computeKey cache keys=111274800 accounts=33.47% INF 2025-09-15 20:27:26.306+08:00 Writing computeKey cache keys=113414700 accounts=34.07% INF 2025-09-15 20:28:32.137+08:00 Writing computeKey cache keys=115554600 accounts=35.00% INF 2025-09-15 20:29:50.038+08:00 Writing computeKey cache keys=117694500 accounts=35.90% INF 2025-09-15 20:31:06.685+08:00 Writing computeKey cache keys=119834400 accounts=36.93% INF 2025-09-15 20:32:10.922+08:00 Writing computeKey cache keys=121974300 accounts=37.50% INF 2025-09-15 20:33:21.148+08:00 Writing computeKey cache keys=124114200 accounts=38.28% INF 2025-09-15 20:34:31.695+08:00 Writing computeKey cache keys=126254100 accounts=39.00% INF 2025-09-15 20:35:52.627+08:00 Writing computeKey cache keys=128394000 accounts=39.90% INF 2025-09-15 20:37:03.749+08:00 Writing computeKey cache keys=130533900 accounts=40.72% INF 2025-09-15 20:38:07.366+08:00 Writing computeKey cache keys=132673800 accounts=41.50% INF 2025-09-15 20:39:12.531+08:00 Writing computeKey cache keys=134813700 accounts=41.84% INF 2025-09-15 20:40:29.954+08:00 Writing computeKey cache keys=136953600 accounts=42.66% INF 2025-09-15 20:41:39.445+08:00 Writing computeKey cache keys=139093500 accounts=43.14% INF 2025-09-15 20:42:47.342+08:00 Writing computeKey cache keys=141233400 accounts=43.76% INF 2025-09-15 20:43:46.907+08:00 Writing computeKey cache keys=143373300 accounts=44.07% INF 2025-09-15 20:44:58.159+08:00 Writing computeKey cache keys=145513200 accounts=44.63% INF 2025-09-15 20:46:07.439+08:00 Writing computeKey cache keys=147653100 accounts=45.36% INF 2025-09-15 20:47:07.117+08:00 Writing computeKey cache keys=149793000 accounts=45.69% INF 2025-09-15 20:48:22.385+08:00 Writing computeKey cache keys=151932900 accounts=46.78% INF 2025-09-15 20:49:28.255+08:00 Writing computeKey cache keys=154072800 accounts=47.48% INF 2025-09-15 20:50:46.481+08:00 Writing computeKey cache keys=156212700 accounts=48.18% INF 2025-09-15 20:51:50.617+08:00 Writing computeKey cache keys=158352600 accounts=48.63% INF 2025-09-15 20:52:59.097+08:00 Writing computeKey cache keys=160492500 accounts=49.55% INF 2025-09-15 20:54:13.562+08:00 Writing computeKey cache keys=162632400 accounts=50.41% INF 2025-09-15 20:55:30.251+08:00 Writing computeKey cache keys=164772300 accounts=51.12% INF 2025-09-15 20:56:36.451+08:00 Writing computeKey cache keys=166912200 accounts=51.85% INF 2025-09-15 20:57:44.287+08:00 Writing computeKey cache keys=169052100 accounts=52.53% INF 2025-09-15 20:58:53.454+08:00 Writing computeKey cache keys=171192000 accounts=53.15% INF 2025-09-15 21:00:04.885+08:00 Writing computeKey cache keys=173331900 accounts=53.63% INF 2025-09-15 21:01:12.859+08:00 Writing computeKey cache keys=175471800 accounts=54.44% INF 2025-09-15 21:02:19.798+08:00 Writing computeKey cache keys=177611700 accounts=55.01% INF 2025-09-15 21:03:34.174+08:00 Writing computeKey cache keys=179751600 accounts=55.70% INF 2025-09-15 21:04:50.550+08:00 Writing computeKey cache keys=181891500 accounts=56.61% INF 2025-09-15 21:05:58.464+08:00 Writing computeKey cache keys=184031400 accounts=57.40% INF 2025-09-15 21:07:14.686+08:00 Writing computeKey cache keys=186171300 accounts=58.21% INF 2025-09-15 21:08:24.643+08:00 Writing computeKey cache keys=188311200 accounts=59.08% INF 2025-09-15 21:09:41.384+08:00 Writing computeKey cache keys=190451100 accounts=60.00% INF 2025-09-15 21:10:57.796+08:00 Writing computeKey cache keys=192591000 accounts=60.80% INF 2025-09-15 21:11:57.208+08:00 Writing computeKey cache keys=194730900 accounts=60.85% INF 2025-09-15 21:13:06.843+08:00 Writing computeKey cache keys=196870800 accounts=61.65% INF 2025-09-15 21:14:24.297+08:00 Writing computeKey cache keys=199010700 accounts=62.14% INF 2025-09-15 21:15:25.246+08:00 Writing computeKey cache keys=201150600 accounts=62.14% INF 2025-09-15 21:16:32.517+08:00 Writing computeKey cache keys=203290500 accounts=62.50% INF 2025-09-15 21:17:48.147+08:00 Writing computeKey cache keys=205430400 accounts=63.33% INF 2025-09-15 21:18:53.169+08:00 Writing computeKey cache keys=207570300 accounts=63.93% INF 2025-09-15 21:20:07.121+08:00 Writing computeKey cache keys=209710200 accounts=64.54% INF 2025-09-15 21:21:08.788+08:00 Writing computeKey cache keys=211850100 accounts=64.63% INF 2025-09-15 21:22:14.199+08:00 Writing computeKey cache keys=213990000 accounts=65.42% INF 2025-09-15 21:23:31.253+08:00 Writing computeKey cache keys=216129900 accounts=66.34% INF 2025-09-15 21:24:42.057+08:00 Writing computeKey cache keys=218269800 accounts=66.83% INF 2025-09-15 21:25:53.255+08:00 Writing computeKey cache keys=220409700 accounts=67.15% INF 2025-09-15 21:27:05.889+08:00 Writing computeKey cache keys=222549600 accounts=67.67% INF 2025-09-15 21:28:19.280+08:00 Writing computeKey cache keys=224689500 accounts=68.28% INF 2025-09-15 21:29:37.937+08:00 Writing computeKey cache keys=226829400 accounts=69.28% INF 2025-09-15 21:30:47.619+08:00 Writing computeKey cache keys=228969300 accounts=69.45% INF 2025-09-15 21:31:56.963+08:00 Writing computeKey cache keys=231109200 accounts=70.27% INF 2025-09-15 21:33:15.470+08:00 Writing computeKey cache keys=233249100 accounts=71.29% INF 2025-09-15 21:34:38.249+08:00 Writing computeKey cache keys=235389000 accounts=72.03% INF 2025-09-15 21:35:49.408+08:00 Writing computeKey cache keys=237528900 accounts=72.72% INF 2025-09-15 21:37:02.131+08:00 Writing computeKey cache keys=239668800 accounts=73.52% INF 2025-09-15 21:38:15.693+08:00 Writing computeKey cache keys=241808700 accounts=74.11% INF 2025-09-15 21:39:35.177+08:00 Writing computeKey cache keys=243948600 accounts=74.88% INF 2025-09-15 21:40:55.293+08:00 Writing computeKey cache keys=246088500 accounts=75.82% INF 2025-09-15 21:42:02.884+08:00 Writing computeKey cache keys=248228400 accounts=76.05% INF 2025-09-15 21:43:22.149+08:00 Writing computeKey cache keys=250368300 accounts=76.89% INF 2025-09-15 21:44:34.093+08:00 Writing computeKey cache keys=252508200 accounts=77.57% INF 2025-09-15 21:45:56.280+08:00 Writing computeKey cache keys=254648100 accounts=78.24% INF 2025-09-15 21:47:05.465+08:00 Writing computeKey cache keys=256788000 accounts=79.16% INF 2025-09-15 21:48:29.412+08:00 Writing computeKey cache keys=258927900 accounts=80.04% INF 2025-09-15 21:49:40.380+08:00 Writing computeKey cache keys=261067800 accounts=80.91% INF 2025-09-15 21:50:53.547+08:00 Writing computeKey cache keys=263207700 accounts=81.48% INF 2025-09-15 21:52:05.654+08:00 Writing computeKey cache keys=265347600 accounts=81.86% INF 2025-09-15 21:53:18.433+08:00 Writing computeKey cache keys=267487500 accounts=82.74% INF 2025-09-15 21:54:32.773+08:00 Writing computeKey cache keys=269627400 accounts=83.19% INF 2025-09-15 21:55:47.556+08:00 Writing computeKey cache keys=271767300 accounts=83.95% INF 2025-09-15 21:56:57.886+08:00 Writing computeKey cache keys=273907200 accounts=84.66% INF 2025-09-15 21:58:16.321+08:00 Writing computeKey cache keys=276047100 accounts=85.47% INF 2025-09-15 21:59:39.387+08:00 Writing computeKey cache keys=278187000 accounts=86.30% INF 2025-09-15 22:00:52.418+08:00 Writing computeKey cache keys=280326900 accounts=86.99% INF 2025-09-15 22:02:05.400+08:00 Writing computeKey cache keys=282466800 accounts=87.86% INF 2025-09-15 22:03:19.116+08:00 Writing computeKey cache keys=284606700 accounts=88.48% INF 2025-09-15 22:04:32.416+08:00 Writing computeKey cache keys=286746600 accounts=89.29% INF 2025-09-15 22:05:35.806+08:00 Writing computeKey cache keys=288886500 accounts=89.51% INF 2025-09-15 22:06:52.993+08:00 Writing computeKey cache keys=291026400 accounts=90.32% INF 2025-09-15 22:08:11.544+08:00 Writing computeKey cache keys=293166300 accounts=91.19% INF 2025-09-15 22:09:22.948+08:00 Writing computeKey cache keys=295306200 accounts=91.63% INF 2025-09-15 22:10:31.507+08:00 Writing computeKey cache keys=297446100 accounts=91.96% INF 2025-09-15 22:11:52.808+08:00 Writing computeKey cache keys=299586000 accounts=92.54% INF 2025-09-15 22:13:01.552+08:00 Writing computeKey cache keys=301725900 accounts=93.15% INF 2025-09-15 22:14:19.488+08:00 Writing computeKey cache keys=303865800 accounts=94.15% INF 2025-09-15 22:15:46.068+08:00 Writing computeKey cache keys=306005700 accounts=95.01% INF 2025-09-15 22:16:59.816+08:00 Writing computeKey cache keys=308145600 accounts=95.87% INF 2025-09-15 22:18:09.411+08:00 Writing computeKey cache keys=310285500 accounts=96.87% INF 2025-09-15 22:19:26.134+08:00 Writing computeKey cache keys=312425400 accounts=97.66% INF 2025-09-15 22:20:43.071+08:00 Writing computeKey cache keys=314565300 accounts=98.37% INF 2025-09-15 22:21:46.918+08:00 Writing computeKey cache keys=316705200 accounts=98.57% INF 2025-09-15 22:23:08.712+08:00 Writing computeKey cache keys=318845100 accounts=99.26% INF 2025-09-15 22:24:07.597+08:00 Writing computeKey cache keys=320985000 accounts=99.26% INF 2025-09-15 22:25:07.338+08:00 Writing computeKey cache keys=323124900 accounts=99.26% INF 2025-09-15 22:26:06.227+08:00 Writing computeKey cache keys=325264800 accounts=99.26% INF 2025-09-15 22:27:15.670+08:00 Writing computeKey cache keys=327404700 accounts=99.85% INF 2025-09-15 22:27:37.590+08:00 Wrote computeKey cache keys=328002861 accounts=100.00% INF 2025-09-15 22:27:43.170+08:00 Imported blocks blockNumber=15537404 slot=4700023 blocks=10 txs=1250 mgas=249.2 bps=0.001 tps=0.115 mgps=0.023 avgBps=0.001 avgTps=0.115 avgMGps=0.023 elapsed=3h1m22s NTC 2025-09-15 22:27:43.171+08:00 Import complete blockNumber=15537404 slot=4700023 blocks=10 txs=1250 mgas=249.2 

Total run time for the master build was 2h33m19s while the total run time for the hash writer build was 3h1m22s.

Unfortunately with these changes the stateroot computation appears to be slower than the master branch. @chirag-parmar Perhaps you could do a benchmark as well to confirm these results?

[chirag-parmar]

Unfortunately with these changes the stateroot computation appears to be slower than the master branch.

I'm not sure why we are measuring the compute time. The purpose of the hash writer was to save on memory and not time. It still does improve on time a little as seen here but only for large data objects like entire blocks.

@chirag-parmar Perhaps you could do a benchmark as well to confirm these results?

I'll try and run the benchmarks and measure the memory usage too. But I wouldn't expect results too far off from @bhartnett 's benchmarks.

[arnetheduck]

hash writer was to save on memory

not really - it's both - memory allocations take time and the idea is to avoid creating work for the garbage collector, which ultimately saves both time and memory, when done right. 25% of time is spent allocating and collecting, and by creating the hashes directly we should be able to avoid that work entirely.

[chirag-parmar]

memory allocations take time and the idea is to avoid creating work for the garbage collector

This is true. But we use the saved time to do two passes of the data to be encoded. And we also do allocate some memory for prefixes. Maybe I can try and use the StaticLengthTracker since the types are pretty much fixed for aristo db (not the case for orderedTrieRoot)

when done right.

I agree. I will look into how I can customize the optimization for aristo db so that we save on time as well.