You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<p>Bitsets, also called bitmaps, are commonly used as fast data structures. Unfortunately, they can use too much memory. To compensate, we often use compressed bitmaps.</p>
91
-
88
+
<p><p>Bitsets, also called bitmaps, are commonly used as fast data structures. Unfortunately, they can use too much memory. To compensate, we often use compressed bitmaps.</p>
92
89
<p>Roaring bitmaps are compressed bitmaps which tend to outperform conventional compressed bitmaps such as WAH, EWAH or Concise. In some instances, they can be hundreds of times faster and they often offer significantly better compression.</p>
93
-
94
90
<blockquote>
95
91
<p>Use Roaring for bitmap compression whenever possible. Do not use other bitmap compression methods (<ahref="http://db.ucsd.edu/wp-content/uploads/2017/03/sidm338-wangA.pdf">Wang et al., SIGMOD 2017</a>)</p>
96
92
</blockquote>
97
-
98
93
<p>Roaring bitmaps are used by several important systems:</p>
<li><ahref="http://lucene.apache.org/core/">Apache Lucene</a> and derivative systems such as Solr and <ahref="https://www.elastic.co/">Elastic</a>,</li>
<p>There are freely available software libraries providing Roaring bitmaps in almost all the popular programming languages (Java, C, C++, Go, C#, Rust, Python…).</p>
132
-
133
126
<p>There is a serialized <ahref="https://github.com/RoaringBitmap/RoaringFormatSpec/">format specification</a> for interoperability between implementations.</p>
134
-
135
127
<p>Roaring bitmaps are used in many proprietary systems. For example, it is used at SEEK:</p>
136
-
137
128
<blockquote>
138
129
<p>The SEEK Group is a broad array of companies that specialise in employment and education. We maintain a presence in 18 countries, reach approximately 3 billion people and employ over 10,000 individuals around the world. We invest heavily in AI and technology to deliver on our purpose of helping people live more fulfilling and productive working lives and enjoy a leading position in online employment marketplaces. A core product in online employment is Job Search. We recently released a new version of this search engine, called <ahref="https://www.seek.com.au/about/news/seek-smarter-search-with-ai">Smarter Search</a>. It blends AI with advanced search technology and has delivered significant lifts in relevance and performance for both candidates and hirers. The engine indexes a massive volume of data and must answer large volumes of diverse queries with extremely low latency. We invest heavily in our own proprietary technology, but also make use of best-in-class data structures where available. We selected the open source <ahref="https://roaringbitmap.org/">Roaring Bitmap</a> library as the compressed bitset at the core of the matching engine, and it has delivered fantastic speed and stability. Research into fields such as advanced data structures and high performance computing remain extremely important to SEEK and many other technology companies. We are excited to continue our own investments in these fields, along with supporting the open source ecosystem and research community through our graduate programs and sponsorships.</p>
139
-
140
130
<p>Mark Pritchard,
141
131
Director of Search and Technology,
142
132
AI Platform Services,
143
133
SEEK Limited</p>
144
134
</blockquote>
145
-
146
135
<h2id="when-should-you-use-a-bitmap">When should you use a bitmap?</h2>
147
-
148
136
<p>Sets are a fundamental abstraction in software. They can be implemented in various ways, as hash sets, as trees, and so forth. In databases and search engines, sets are often an integral part of indexes. For example, we may need to maintain a set of all documents or rows (represented by numerical identifier) that satisfy some property. Besides adding or removing elements from the set, we need fast functions to compute the intersection, the union, the difference between sets, and so on.</p>
149
-
150
137
<p>To implement a set of integers, a particularly appealing strategy is the bitmap (also called bitset or bit vector). Using n bits, we can represent any set made of the integers from the range [0,n): it suffices to set the ith bit is set to one if integer i is present in the set. Commodity processors use words of W=32 or W=64 bits. By combining many such words, we can support large values of n. Intersections, unions and differences can then be implemented as bitwise AND, OR and ANDNOT operations. More complicated set functions can also be implemented as bitwise operations.</p>
151
-
152
138
<p>When the bitset approach is applicable, it can be orders of magnitude faster than other possible implementation of a set (e.g., as a hash set) while using several times less memory.</p>
153
-
154
-
<p>However, a bitset, even a compressed one is not always applicable. For example, if the you have 1000 random-looking integers, then a simple array might be the best representation. We refer to this case as the “sparse” scenario.</p>
155
-
139
+
<p>However, a bitset, even a compressed one is not always applicable. For example, if you have 1000 random-looking integers, then a simple array might be the best representation. We refer to this case as the “sparse” scenario.</p>
156
140
<h2id="when-should-you-use-compressed-bitmaps">When should you use compressed bitmaps?</h2>
157
-
158
-
<p>An uncompressed BitSet can use a lot of memory. For example, if you take a BitSet and set the bit at position 1,000,000 to true and you have just over 100kB. That’s over 100kB to store the position of one bit. This is wasteful even if you do not care about memory: suppose that you need to compute the intersection between this BitSet and another one that has a bit at position 1,000,001 to true, then you need to go through all these zeroes, whether you like it or not. That can become very wasteful.</p>
159
-
160
-
<p>This being said, there are definitively cases where attempting to use compressed bitmaps is wasteful. For example, if you have a small universe size. E.g., your bitmaps represent sets of integers from [0,n) where n is small (e.g., n=64 or n=128). If you can use an uncompressed BitSet and it does not blow up your memory usage, then compressed bitmaps are probably not useful to you. In fact, if you do not need compression, then a BitSet offers remarkable speed.</p>
161
-
141
+
<p>An uncompressed BitSet can use a lot of memory. For example, if you take a BitSet and set the bit at position 1,000,000 to true and you have just over 100kB. That is over 100kB to store the position of one bit. This is wasteful even if you do not care about memory: suppose that you need to compute the intersection between this BitSet and another one that has a bit at position 1,000,001 to true, then you need to go through all these zeroes, whether you like it or not. That can become very wasteful.</p>
142
+
<p>This being said, there are definitively cases where attempting to use compressed bitmaps is wasteful. For example, if you have a small universe size. E.g., your bitmaps represent sets of integers from [0,n) where n is small (e.g., n=64 or n=128). If you are able to use uncompressed BitSet and it does not blow up your memory usage, then compressed bitmaps are probably not useful to you. In fact, if you do not need compression, then a BitSet offers remarkable speed.</p>
162
143
<p>The sparse scenario is another use case where compressed bitmaps should not be used.
163
144
Keep in mind that random-looking data is usually not compressible. E.g., if you have a small set of
164
145
32-bit random integers, it is not mathematically possible to use far less than 32 bits per integer,
165
146
and attempts at compression can be counterproductive.</p>
166
-
167
147
<h2id="how-does-roaring-compares-with-the-alternatives">How does Roaring compares with the alternatives?</h2>
168
-
169
148
<p>Most alternatives to Roaring are part of a larger family of compressed bitmaps that are run-length-encoded bitmaps. They identify long runs of 1s or 0s and they represent them with a marker word. If you have a local mix of 1s and 0, you use an uncompressed word.</p>
170
-
171
149
<p>There are many formats in this family:</p>
172
-
173
150
<ul>
174
151
<li>Oracle’s BBC is an obsolete format at this point: though it may provide good compression, it is likely much slower than more recent alternatives due to excessive branching.</li>
175
152
<li>WAH is a patented variation on BBC that provides better performance.</li>
176
153
<li>Concise is a variation on the patented WAH. It some specific instances, it can compress much better than WAH (up to 2x better), but it is generally slower.</li>
177
154
<li>EWAH is both free of patent, and it is faster than all the above. On the downside, it does not compress quite as well. It is faster because it allows some form of “skipping” over uncompressed words. So though none of these formats are great at random access, EWAH is better than the alternatives.</li>
178
155
</ul>
179
-
180
156
<p>There is a big problem with these formats however that can hurt you badly in some cases: there is no random access. If you want to check whether a given value is present in the set, you have to start from the beginning and “uncompress” the whole thing. This means that if you want to intersect a big set with a large set, you still have to uncompress the whole big set in the worst case…</p>
181
-
182
-
<p>Roaring solves this problem. It works in the following manner. It divides the data into chunks of 2<sup>16</sup> integers (e.g., [0, 2<sup>16</sup>), [2<sup>16</sup>, 2 x 2<sup>16</sup>), …). Within a chunk, it can use an uncompressed bitmap, a simple list of integers, or a list of runs. Whatever format it uses, they all allow you to check for the present of any one value quickly (e.g., with a binary search). The net result is that Roaring can compute many operations much faster than run-length-encoded formats like WAH, EWAH, Concise… Maybe surprisingly, Roaring also generally offers better compression ratios.</p>
183
-
157
+
<p>Roaring solves this problem. It works in the following manner. It divides the data into chunks of 2<!-- raw HTML omitted -->16<!-- raw HTML omitted --> integers (e.g., [0, 2<!-- raw HTML omitted -->16<!-- raw HTML omitted -->), [2<!-- raw HTML omitted -->16<!-- raw HTML omitted -->, 2 x 2<!-- raw HTML omitted -->16<!-- raw HTML omitted -->), …). Within a chunk, it can use an uncompressed bitmap, a simple list of integers, or a list of runs. Whatever format it uses, they all allow you to check for the presence of any one value quickly (e.g., with a binary search). The net result is that Roaring can compute many operations much faster than run-length-encoded formats like WAH, EWAH, Concise… Maybe surprisingly, Roaring also generally offers better compression ratios.</p>
184
158
<h2id="funding">Funding</h2>
185
-
186
159
<p>This work was supported by NSERC grant number 26143.</p>
<li><ahref="http://lucene.apache.org/core/">Apache Lucene</a> and derivative systems such as Solr and <ahref="https://www.elastic.co">Elasticsearch</a>,</li>
<p>There is a serialized <ahref="https://github.com/RoaringBitmap/RoaringFormatSpec/">format specification</a> for interoperability between implementations.</p>
0 commit comments