Flate: Fix/tweak huffman bit estimates

[klauspost]

New blocks where given an advantage since estimates for new blocks did not include extra bits.

Optimize log2 calculation with faster estimate:

Benchmark_tokens_EstimatedBits-32 299931 4062 ns/op 0.25 MB/s Benchmark_tokens_EstimatedBits-32 2201341 543 ns/op 1.84 MB/s 

Before/After

file	out	level	insize	outsize	millis	mb/s gob-stream	gzkp	-2	1911399616	1410346540	6927	263.13 gob-stream	gzkp	1	1911399616	360099905	5708	319.33 gob-stream	gzkp	2	1911399616	346448934	6334	287.77 gob-stream	gzkp	3	1911399616	339896850	8308	219.39 gob-stream	gzkp	4	1911399616	319850581	9092	200.49 gob-stream	gzkp	5	1911399616	310556560	10675	170.75 gob-stream	gzkp	6	1911399616	309710534	12074	150.96 gob-stream	gzkp	7	1911399616	305960448	14837	122.86 gob-stream	gzkp	8	1911399616	300291259	17893	101.88 gob-stream	gzkp	9	1911399616	280306976	31682	57.54 file	out	level	insize	outsize	millis	mb/s gob-stream	gzkp	-2	1911399616	1410346519	6767	269.35 gob-stream	gzkp	1	1911399616	361916183	5635	323.47 gob-stream	gzkp	2	1911399616	349857310	6256	291.36 gob-stream	gzkp	3	1911399616	341885261	6885	264.74 gob-stream	gzkp	4	1911399616	321792143	7890	231.01 gob-stream	gzkp	5	1911399616	312085491	8911	204.54 gob-stream	gzkp	6	1911399616	311292135	11114	164.01 gob-stream	gzkp	7	1911399616	305960448	14712	123.90 gob-stream	gzkp	8	1911399616	300291259	17781	102.51 gob-stream	gzkp	9	1911399616	280306976	31161	58.50 file	out	level	insize	outsize	millis	mb/s github-june-2days-2019.json	gzkp	-2	6273951764	4191425185	22107	270.64 github-june-2days-2019.json	gzkp	1	6273951764	1150816630	18647	320.87 github-june-2days-2019.json	gzkp	2	6273951764	1112997952	21263	281.39 github-june-2days-2019.json	gzkp	3	6273951764	1103339187	25701	232.80 github-june-2days-2019.json	gzkp	4	6273951764	997880134	28780	207.89 github-june-2days-2019.json	gzkp	5	6273951764	963159098	34158	175.16 github-june-2days-2019.json	gzkp	6	6273951764	949542385	39269	152.36 github-june-2days-2019.json	gzkp	7	6273951764	934872037	49069	121.94 github-june-2days-2019.json	gzkp	8	6273951764	921116719	57011	104.95 github-june-2days-2019.json	gzkp	9	6273951764	899793019	84347	70.94 file	out	level	insize	outsize	millis	mb/s github-june-2days-2019.json	gzkp	-2	6273951764	4196148824	21661	276.22 github-june-2days-2019.json	gzkp	1	6273951764	1125991387	18940	315.90 github-june-2days-2019.json	gzkp	2	6273951764	1107583034	21865	273.64 github-june-2days-2019.json	gzkp	3	6273951764	1103676413	23369	256.03 github-june-2days-2019.json	gzkp	4	6273951764	999153827	25717	232.65 github-june-2days-2019.json	gzkp	5	6273951764	964959169	29933	199.89 github-june-2days-2019.json	gzkp	6	6273951764	951460252	35396	169.03 github-june-2days-2019.json	gzkp	7	6273951764	934872037	48143	124.28 github-june-2days-2019.json	gzkp	8	6273951764	921116719	56076	106.70 github-june-2days-2019.json	gzkp	9	6273951764	899793019	83476	71.68 file	out	level	insize	outsize	millis	mb/s enwik9	gzkp	-2	1000000000	651299040	3615	263.75 enwik9	gzkp	1	1000000000	384456678	6512	146.44 enwik9	gzkp	2	1000000000	372262266	6973	136.76 enwik9	gzkp	3	1000000000	364294282	8140	117.15 enwik9	gzkp	4	1000000000	341904112	9049	105.39 enwik9	gzkp	5	1000000000	338375880	10652	89.52 enwik9	gzkp	6	1000000000	336071599	11201	85.14 enwik9	gzkp	7	1000000000	329248670	18506	51.53 enwik9	gzkp	8	1000000000	326623117	23616	40.38 enwik9	gzkp	9	1000000000	324205510	29803	32.00 file	out	level	insize	outsize	millis	mb/s enwik9	gzkp	-2	1000000000	652568269	3534	269.85 enwik9	gzkp	1	1000000000	383140821	6523	146.19 enwik9	gzkp	2	1000000000	372365357	7062	135.04 enwik9	gzkp	3	1000000000	364568604	7674	124.26 enwik9	gzkp	4	1000000000	342411924	8418	113.28 enwik9	gzkp	5	1000000000	338866672	9879	96.53 enwik9	gzkp	6	1000000000	336511427	10474	91.05 enwik9	gzkp	7	1000000000	329248670	18497	51.56 enwik9	gzkp	8	1000000000	326623117	24016	39.71 enwik9	gzkp	9	1000000000	324205510	29511	32.32 file	out	level	insize	outsize	millis	mb/s 10gb.tar	gzkp	-2	10065157632	6540870519	30341	316.36 10gb.tar	gzkp	1	10065157632	5280659104	45780	209.67 10gb.tar	gzkp	2	10065157632	5147972222	48354	198.51 10gb.tar	gzkp	3	10065157632	5059091941	54003	177.75 10gb.tar	gzkp	4	10065157632	4947136306	59389	161.63 10gb.tar	gzkp	5	10065157632	4933679787	71788	133.71 10gb.tar	gzkp	6	10065157632	4895606626	76606	125.30 file	out	level	insize	outsize	millis	mb/s 10gb.tar	gzkp	-2	10065157632	6548123344	29288	327.73 10gb.tar	gzkp	1	10065157632	5214106110	45982	208.75 10gb.tar	gzkp	2	10065157632	5120120494	48900	196.29 10gb.tar	gzkp	3	10065157632	5047291855	52904	181.44 10gb.tar	gzkp	4	10065157632	4943352328	55613	172.60 10gb.tar	gzkp	5	10065157632	4931789557	67581	142.03 10gb.tar	gzkp	6	10065157632	4894858258	71080	135.04 10gb.tar	gzkp	7	10065157632	4849130270	204530	46.93 

name old speed new speed delta EncodeDigitsConstant1e4-12 349MB/s ± 1% 355MB/s ± 0% ~ (p=0.107 n=3+3) EncodeDigitsConstant1e5-12 326MB/s ± 0% 333MB/s ± 1% +2.24% (p=0.008 n=3+3) EncodeDigitsConstant1e6-12 368MB/s ± 1% 381MB/s ± 0% +3.58% (p=0.008 n=3+3) EncodeDigitsSpeed1e4-12 132MB/s ± 1% 135MB/s ± 1% +2.35% (p=0.024 n=3+3) EncodeDigitsSpeed1e5-12 103MB/s ± 0% 102MB/s ± 0% ~ (p=0.276 n=3+3) EncodeDigitsSpeed1e6-12 103MB/s ± 0% 107MB/s ± 0% +4.10% (p=0.001 n=3+3) EncodeDigitsDefault1e4-12 65.6MB/s ± 0% 66.7MB/s ± 1% ~ (p=0.060 n=3+3) EncodeDigitsDefault1e5-12 60.4MB/s ± 1% 61.7MB/s ± 1% +2.02% (p=0.018 n=3+3) EncodeDigitsDefault1e6-12 59.2MB/s ± 0% 60.1MB/s ± 1% ~ (p=0.103 n=3+3) EncodeDigitsCompress1e4-12 45.8MB/s ± 0% 45.8MB/s ± 0% ~ (p=0.552 n=3+3) EncodeDigitsCompress1e5-12 26.5MB/s ± 1% 26.7MB/s ± 0% ~ (p=0.354 n=3+3) EncodeDigitsCompress1e6-12 24.6MB/s ± 1% 24.9MB/s ± 0% +1.23% (p=0.041 n=3+3) EncodeDigitsSL1e4-12 76.4MB/s ± 0% 78.6MB/s ± 0% +2.83% (p=0.000 n=3+3) EncodeDigitsSL1e5-12 112MB/s ± 0% 117MB/s ± 0% +5.21% (p=0.001 n=3+3) EncodeDigitsSL1e6-12 120MB/s ± 0% 126MB/s ± 1% +5.07% (p=0.000 n=3+3) EncodeTwainConstant1e4-12 255MB/s ± 1% 278MB/s ± 0% +9.29% (p=0.002 n=3+3) EncodeTwainConstant1e5-12 289MB/s ± 1% 301MB/s ± 0% +4.24% (p=0.014 n=3+3) EncodeTwainConstant1e6-12 339MB/s ± 1% 347MB/s ± 0% ~ (p=0.055 n=3+3) EncodeTwainSpeed1e4-12 98.5MB/s ± 0% 98.8MB/s ± 0% ~ (p=0.108 n=3+3) EncodeTwainSpeed1e5-12 104MB/s ± 1% 105MB/s ± 0% ~ (p=0.241 n=3+3) EncodeTwainSpeed1e6-12 112MB/s ± 0% 113MB/s ± 0% +1.03% (p=0.019 n=3+3) EncodeTwainDefault1e4-12 65.9MB/s ± 1% 65.3MB/s ± 1% ~ (p=0.092 n=3+3) EncodeTwainDefault1e5-12 68.5MB/s ± 0% 68.7MB/s ± 0% ~ (p=0.114 n=3+3) EncodeTwainDefault1e6-12 67.1MB/s ± 2% 70.8MB/s ± 3% +5.46% (p=0.049 n=3+3) EncodeTwainCompress1e4-12 37.3MB/s ± 0% 37.5MB/s ± 1% ~ (p=0.297 n=3+3) EncodeTwainCompress1e5-12 20.9MB/s ± 1% 20.6MB/s ± 2% ~ (p=0.197 n=3+3) EncodeTwainCompress1e6-12 19.0MB/s ± 0% 18.6MB/s ± 1% -1.90% (p=0.018 n=3+3) EncodeTwainSL1e4-12 70.8MB/s ± 0% 69.9MB/s ± 0% -1.14% (p=0.012 n=3+3) EncodeTwainSL1e5-12 111MB/s ± 0% 111MB/s ± 2% ~ (p=0.635 n=3+3) EncodeTwainSL1e6-12 120MB/s ± 1% 123MB/s ± 1% +2.26% (p=0.037 n=3+3) 

Compression size changes pending...