Improve memory usage for weakHeap sorting #68
Conversation
MaxGraey commented Apr 7, 2018
MaxGraey commented - Now we use full 32-bits for bitset array which store one extra bit per element.
- Now we use full 32-bits for bitset array which store one extra bit per element
| Fwiw, at this point it appears that the maximum size of an array will eventually be limited by its backing ArrayBuffer. Effective maximum size might then become |
| This changes not concern ArrayBuffer or maximum allocation size. It just reduce memory usage for helper array, which now use more compressed view. Just example After: |
| const intShift = alignof<i32>(); | ||
| | ||
| var blen = (len + 7) >> 3; | ||
| var blen = (len + 31) >> 5; |
There was a problem hiding this comment.
Trying to understand this change. Before, there was one byte per 8 elements (1 bit per element), if I understood this correctly:
8 elements -> blen=1
9 elements -> blen=2
Afterwards, with the change, this becomes:
8 elements -> blen=1
9 elements -> blen=1 (is this correct?)
There was a problem hiding this comment.
Right!
Before:
8 elements -> blen = 1
9 elements -> blen = 2
After:
8 elements -> blen = 1
9 elements -> blen = 1
32 elements -> blen = 1
33 elements -> blen = 2
64 elements -> blen = 2
So for 32 elements now used full space of 32-bit word
There was a problem hiding this comment.
I might be missing something, but my impression was that the bitset needs one bit per element? Or does it calculate the number of 32-bit ints now and blen isn't synonymous for byteLength anymore?
There was a problem hiding this comment.
Yes, bitset just add one extra bit per element.
There was a problem hiding this comment.
So, if there are 3 elements, wouldn't it need 3 distinct bits still (one per element)? Otherwise, let's say there are three elements, the bitset would be 0011b, unable to distinguish this from element 1 and 2 being set?
Or, let me rephrase: If there are 32 elements, the bitset would need 32 bits. With 8 bits per byte, that are 4 bytes, not 1?
There was a problem hiding this comment.
Because I hope Uint32Array is more performant. In JS version it's true
There was a problem hiding this comment.
Than an Uint8Array? Even though an Uint32Array involves additional (obviously hard to understand ^^) shifts?
There was a problem hiding this comment.
If you mean Array<bool> or Array<u8> which store only one bit of 8-bits current version slower of course but we loose 7-bits and our memory usage is worst. So this is balance between memory usage/speed. I mean current version faster previous when I used 8-bit vector inside 32-bit array =)
There was a problem hiding this comment.
Sure Array<bool> doesn't make sense because it wastes 7/8 bits, but why wouldn't u8 work, storing 8 element bits per u8, over an u32 storing 32 element bits per u32?
There was a problem hiding this comment.
Yes. That's what I wrote and older version works perfectly, but we loose 24-bits and use the same xor-shifts techniques for access to bits. Now I rewrite this to use full 32-bits range
| Ok, going to merge this, maybe I should just try it out a bit :) |
| A touched a few things here to get a handle on it. Nothing special, just a bit of local caching, operation elimination, AST crunching etc. Let me know if you spot a mistake :) |
2 participants
