This Javascript library provides the most performant tiny polyfill for window.TextEncoder
, TextEncoder.prototype.encodeInto
, and window.TextDecoder
for use in the browser, in NodeJS, in RequireJS, in web Workers, in SharedWorkers, and in ServiceWorkers.
Add the following HTML Code inside the <head>
:
<script src="https://dl.dropboxusercontent.com/s/r55397ld512etib/EncoderDecoderTogether.min.js?dl=0" nomodule="" type="text/javascript"></script>
If no script on the page requires this library until the DOMContentLoaded event, then use the the much less blocking version below:
<script defer="" src="https://dl.dropboxusercontent.com/s/r55397ld512etib/EncoderDecoderTogether.min.js?dl=0" nomodule="" type="text/javascript"></script>
Alternatively, either use https://dl.dropboxusercontent.com/s/47481btie8pb95h/FastestTextEncoderPolyfill.min.js?dl=0
to polyfill window.TextEncoder
for converting a String
into a Uint8Array
or use https://dl.dropboxusercontent.com/s/qmoknmp86sytc74/FastestTextDecoderPolyfill.min.js?dl=0
to only polyfill window.TextDecoder
for converting a Uint8Array
/ArrayBuffer
/[typedarray]/global.Buffer
into a String
.
The nomodule
attribute prevents the script from being needlessly downloaded and executed on browsers which already support TextEncoder
and TextDecoder
. nomodule
does not test for the presence of TextEncoder
or TextDecoder
, but it is very safe to assume that browsers advanced enough to support modules also support TextEncoder
and TextDecoder
.
See the MDN here for documentation. For the TextEncoder.prototype.encodeInto polyfill, please use https://dl.dropboxusercontent.com/s/i2e2rho1ohtbhfg/EncoderDecoderTogether.min.js?dl=0
for the full package, https://dl.dropboxusercontent.com/s/nlcgzbr0ayd5pjs/FastestTextEncoderPolyfill.min.js?dl=0
for only TextEncoder and TextEncoder.prototype.encodeInto, and npm i fastestsmallesttextencoderdecoder-encodeinto
for NodeJS, es6 modules, RequireJS, AngularJS, or whatever it is that floats your boat. The encodeInto folder of this repository contains the auto-generated encodeInto build of the main project.
For dropping into either RequireJS or NodeJS, please use the fastestsmallesttextencoderdecoder
npm repository, this minified file, or the corresponding source code file. To install via npm, use the following code.
npm install fastestsmallesttextencoderdecoder
Alternatively, if one do not know how to use the command line, save the script corresponding to one's operating system to the directory where the nodejs script will run and use the file manager to run the script (on Windows, it's a double-click).
- fastestsmallesttextencoderdecoder
- Microsoft Windows batch: install-FastestSmallestTextEncoderDecoder-windows.bat
- Bash for Apple MacOS and Linux (e.x. Ubuntu): install-FastestSmallestTextEncoderDecoder-unix.sh
- fastestsmallesttextencoderdecoder-encodeinto
- Microsoft Windows batch: install-FastestSmallestTextEncoderDecoder-encodeInto.bat
- Bash for Apple MacOS and Linux (e.x. Ubuntu): install-FastestSmallestTextEncoderDecoder-encodeInto.sh
After installing via npm, one can use require("fastestsmallesttextencoderdecoder")
. Alternatively, one can drop the EncoderAndDecoderNodeJS.min.js file into the same directory as their NodeJS script and do require("./EncoderAndDecoderNodeJS.min.js")
. Both methods are functionally equivalent.
Open a terminal in the project's directory, and install fastestsmallesttextencoderdecoder via npm.
npm install fastestsmallesttextencoderdecoder
Then, add import 'fastestsmallesttextencoderdecoder';
to the polyfills.ts file.
Don't take my word that FastestSmallestTextEncoderDecoder is the fastest. Instead, check out the benchmarks below. You can run your own benchmarks by cloning this repo and running npm run benchmark
, but beware that you need a beefy computer with 20GB of free RAM, as the NodeJS garbage collector is practically disabled via --noconcurrent_sweeping
so that it does not interfer with the timing of the tests.
The tests below were performed on an ascii file.
Library | Decode 32 bytes | Decode 32768 | Decode 16777216 | Encode 32 bytes | Encode 32768 | Encode 16777216 |
---|---|---|---|---|---|---|
fast-text-encoding | 47792 Kb/sec | 65870 Kb/sec | 89018 Kb/sec | 11197 Kb/sec | 20105 Kb/sec | 29248 Kb/sec |
text-encoding) | 5456 Kb/sec | 6290 Kb/sec | 7569 Kb/sec | 5731 Kb/sec | 4842 Kb/sec | 7361 Kb/sec |
TextEncoderTextDecoder.js | 5577 Kb/sec | 5205 Kb/sec | 6232 Kb/sec | 8557 Kb/sec | 7469 Kb/sec | 12518 Kb/sec |
TextEncoderLite | 20563 Kb/sec | 24377 Kb/sec | 13708 Kb/sec | 10599 Kb/sec | 8225 Kb/sec | 12718 Kb/sec |
text-encoding-shim | 14791 Kb/sec | 17272 Kb/sec | 27324 Kb/sec | 7203 Kb/sec | 9243 Kb/sec | 15112 Kb/sec |
FastestSmallestTextEncoderDecoder | 48714 Kb/sec | 83601 Kb/sec | 149612 Kb/sec | 14060 Kb/sec | 21391 Kb/sec | 31692 Kb/sec |
Native | 22584 Kb/sec | 841778 Kb/sec | 862736 Kb/sec | 18169 Kb/sec | 26682 Kb/sec | 36591 Kb/sec |
The tests below were performed on a mixed ascii-utf8 file.
Library | Decode 32 bytes | Decode 32768 | Decode 16777216 | Encode 32 bytes | Encode 32768 | Encode 16777216 |
---|---|---|---|---|---|---|
fast-text-encoding | 34694 Kb/sec | 62305 Kb/sec | 90462 Kb/sec | 10175 Kb/sec | 18690 Kb/sec | 29335 Kb/sec |
text-encoding) | 4860 Kb/sec | 5941 Kb/sec | 7878 Kb/sec | 6222 Kb/sec | 3958 Kb/sec | 7666 Kb/sec |
TextEncoderTextDecoder.js | 4639 Kb/sec | 3678 Kb/sec | 5418 Kb/sec | 8741 Kb/sec | 6811 Kb/sec | 12253 Kb/sec |
TextEncoderLite | 8999 Kb/sec | 10227 Kb/sec | 10112 Kb/sec | 10868 Kb/sec | 7457 Kb/sec | 12790 Kb/sec |
text-encoding-shim | 14081 Kb/sec | 14852 Kb/sec | 23365 Kb/sec | 6699 Kb/sec | 8255 Kb/sec | 15838 Kb/sec |
FastestSmallestTextEncoderDecoder | 47975 Kb/sec | 71299 Kb/sec | 131499 Kb/sec | 14427 Kb/sec | 21483 Kb/sec | 31780 Kb/sec |
Native | 24853 Kb/sec | 353511 Kb/sec | 521529 Kb/sec | 18637 Kb/sec | 26570 Kb/sec | 36828 Kb/sec |
For a more accurate real world test, we'll examine the 1876 The Russian Synodal Bible.txt. It's a whoping 4.4MB rat's-nest of complex Russian UTF-8, sure to give any encoder/decoder a bad day. Let's see how they perform at their worst.
Library | Decode Russian Bible | Encode Russian Bible |
---|---|---|
fast-text-encoding | 93371 Kb/sec | 33357 Kb/sec |
text-encoding) | 7396 Kb/sec | 8386 Kb/sec |
TextEncoderTextDecoder.js | 4368 Kb/sec | 13050 Kb/sec |
TextEncoderLite | 15828 Kb/sec | 12953 Kb/sec |
text-encoding-shim | 29189 Kb/sec | 17155 Kb/sec |
FastestSmallestTextEncoderDecoder | 148384 Kb/sec | 35537 Kb/sec |
Native | 893880 Kb/sec | 40933 Kb/sec |
This polyfill will bring support for TextEncoder/TextDecoder to the following browsers.
Feature | Chrome | Firefox | Opera | Edge | Internet Explorer | Safari | Android | Samsung Internet | Node.js |
---|---|---|---|---|---|---|---|---|---|
Full Polyfill | 7.0 | 4.0 | 11.6 | 12.0** | 10 | 5.1 (Desktop) / 4.2 (iOS) | 4.0 | 1.0 | 3.0 |
Partial Polyfill* | 1.0** | 0.6 | 7.0 (Desktop) / 9.5** (Mobile) | 12.0** | 4.0 | 2.0 | 1.0** | 1.0** | 0.10 |
Also note that while this polyfill may work in these old browsers, it is very likely that the rest of one's website will not unless if one make a concious effort to have their code work in these old browsers.
* Partial polyfill means that Array
(or Buffer
in NodeJS) will be used instead of Uint8Array
/[typedarray].
** This is the first public release of the browser
Please review the MDN at window.TextEncoder
and window.TextDecoder
for information on how to use TextEncoder and TextDecoder.
As for NodeJS, calling require("EncoderAndDecoderNodeJS.min.js")
yields the following object. Note that this polyfill checks for global.TextEncoder
and global.TextDecoder
and returns the native implementation if available.
module.exports = { TextEncoder: function TextEncoder(){/*...*/}, TextDecoder: function TextDecoder(){/*...*/}, encode: TextEncoder.prototype.encode, decode: TextDecoder.prototype.decode }
In NodeJS, one does not ever have to use new
just to get the encoder/decoder (although one still can do so if they want to). All of the code snippets below function identically (aside from unused local variables introduced into the scope).
// Variation 1 const {TextEncoder, TextDecoder} = require("fastestsmallesttextencoderdecoder"); const encode = (new TextEncoder).encode; const decode = (new TextDecoder).decode;
// Variation 2 const {encode, decode} = require("fastestsmallesttextencoderdecoder");
// Variation 3 (a rewording of Variation 2) const encodeAndDecodeModule = require("fastestsmallesttextencoderdecoder"); const encode = encodeAndDecodeModule.encode; const decode = encodeAndDecodeModule.decode;
Or, one can use the new and shiny ES6 module importation statements.
// Variation 1 import {TextEncoder, TextDecoder} from "fastestsmallesttextencoderdecoder"; const encode = (new TextEncoder).encode; const decode = (new TextDecoder).decode;
// Variation 2 import {encode, decode} from "fastestsmallesttextencoderdecoder";
// Variation 3 (a rewording of Variation 2) import * as encodeAndDecodeModule from "fastestsmallesttextencoderdecoder"; const encode = encodeAndDecodeModule.encode; const decode = encodeAndDecodeModule.decode;
Visit the GithubPage to see a demonstation. As seen in the Web Worker hexWorker.js, the Github Pages demonstration uses a special encoderAndDecoderForced.src.js version of this library to forcefully install the TextEncoder and TextDecoder even when there is native support. That way, this demonstraton should serve to truthfully demonstrate this polyfill.
This project can be found on npm here at this link.
On Linux, the project can be developed by cloning it with the following command line. The development scripts are designed to be interpeted by Dash, and whether they work on Mac OS is unknown, but they certainly won't work on Windows.
git clone https://github.com/anonyco/FastestSmallestTextEncoderDecoder.git; cd FastestSmallestTextEncoderDecoder; npm run install-dev
Emphasize the npm run install-dev
, which downloads closure-compiler.jar
into the repository for minifying the files.
Now that the repository is cloned, edit the files as one see fit. Do not edit the files in the encodeInto
folder. Those are all auto-generated. Also, do not run npm run build
in the encodeInto
. That's done automatically when npm run build
is runned in the topmost folder. Now that the files have been edited, run the following in the terminal in the root folder of the repository in order to minify the NodeJS JavaScript files.
npm run build
To edit tests, edit test/node.js
. These tests are compared against the native implementation to ensure validity. To run tests, do the following.
npm run test
I try my best to be a realist, and what's more realistic than death? I am going to die someday and it may be tomorrow in a car crash. You never know. As I have no coder freinds to look out for my projects, I'm looking for anyone who wants to be a collaborator on this project in the event of the unforseen. Reach out to me at wowzeryest@gmail.com. If issues/pulls start piling up over the course of months, assume the worst. As I am trying my best to do my part to help the community, I encourage every developer to share their projects with other people to ensure continuity.