Skip to content

davidliu/FastestSmallestTextEncoderDecoder

Repository files navigation

npm version GitHub stars GitHub file size in bytes GitHub file size in bytes npm bundle size (version) CC0 license npm downloads

This Javascript library provides the most performant tiny polyfill for window.TextEncoder, TextEncoder.prototype.encodeInto, and window.TextDecoder for use in the browser, in NodeJS, in RequireJS, in web Workers, in SharedWorkers, and in ServiceWorkers.

Quick Start

Add the following HTML Code inside the <head>:

<script src="https://dl.dropboxusercontent.com/s/r55397ld512etib/EncoderDecoderTogether.min.js?dl=0" nomodule="" type="text/javascript"></script>

If no script on the page requires this library until the DOMContentLoaded event, then use the the much less blocking version below:

<script defer="" src="https://dl.dropboxusercontent.com/s/r55397ld512etib/EncoderDecoderTogether.min.js?dl=0" nomodule="" type="text/javascript"></script>

Alternatively, either use https://dl.dropboxusercontent.com/s/47481btie8pb95h/FastestTextEncoderPolyfill.min.js?dl=0 to polyfill window.TextEncoder for converting a String into a Uint8Array or use https://dl.dropboxusercontent.com/s/qmoknmp86sytc74/FastestTextDecoderPolyfill.min.js?dl=0 to only polyfill window.TextDecoder for converting a Uint8Array/ArrayBuffer/[typedarray]/global.Buffer into a String.

The nomodule attribute prevents the script from being needlessly downloaded and executed on browsers which already support TextEncoder and TextDecoder. nomodule does not test for the presence of TextEncoder or TextDecoder, but it is very safe to assume that browsers advanced enough to support modules also support TextEncoder and TextDecoder.

EncodeInto

See the MDN here for documentation. For the TextEncoder.prototype.encodeInto polyfill, please use https://dl.dropboxusercontent.com/s/i2e2rho1ohtbhfg/EncoderDecoderTogether.min.js?dl=0 for the full package, https://dl.dropboxusercontent.com/s/nlcgzbr0ayd5pjs/FastestTextEncoderPolyfill.min.js?dl=0 for only TextEncoder and TextEncoder.prototype.encodeInto, and npm i fastestsmallesttextencoderdecoder-encodeinto for NodeJS, es6 modules, RequireJS, AngularJS, or whatever it is that floats your boat. The encodeInto folder of this repository contains the auto-generated encodeInto build of the main project.

RequireJS and NodeJS

For dropping into either RequireJS or NodeJS, please use the fastestsmallesttextencoderdecoder npm repository, this minified file, or the corresponding source code file. To install via npm, use the following code.

npm install fastestsmallesttextencoderdecoder

Alternatively, if one do not know how to use the command line, save the script corresponding to one's operating system to the directory where the nodejs script will run and use the file manager to run the script (on Windows, it's a double-click).

After installing via npm, one can use require("fastestsmallesttextencoderdecoder"). Alternatively, one can drop the EncoderAndDecoderNodeJS.min.js file into the same directory as their NodeJS script and do require("./EncoderAndDecoderNodeJS.min.js"). Both methods are functionally equivalent.

AngularJS

Open a terminal in the project's directory, and install fastestsmallesttextencoderdecoder via npm.

npm install fastestsmallesttextencoderdecoder

Then, add import 'fastestsmallesttextencoderdecoder'; to the polyfills.ts file.

Benchmarks

Don't take my word that FastestSmallestTextEncoderDecoder is the fastest. Instead, check out the benchmarks below. You can run your own benchmarks by cloning this repo and running npm run benchmark, but beware that you need a beefy computer with 20GB of free RAM, as the NodeJS garbage collector is practically disabled via --noconcurrent_sweeping so that it does not interfer with the timing of the tests.

The tests below were performed on an ascii file.

Library Decode 32 bytes Decode 32768 Decode 16777216 Encode 32 bytes Encode 32768 Encode 16777216
fast-text-encoding 47792 Kb/sec 65870 Kb/sec 89018 Kb/sec 11197 Kb/sec 20105 Kb/sec 29248 Kb/sec
text-encoding) 5456 Kb/sec 6290 Kb/sec 7569 Kb/sec 5731 Kb/sec 4842 Kb/sec 7361 Kb/sec
TextEncoderTextDecoder.js 5577 Kb/sec 5205 Kb/sec 6232 Kb/sec 8557 Kb/sec 7469 Kb/sec 12518 Kb/sec
TextEncoderLite 20563 Kb/sec 24377 Kb/sec 13708 Kb/sec 10599 Kb/sec 8225 Kb/sec 12718 Kb/sec
text-encoding-shim 14791 Kb/sec 17272 Kb/sec 27324 Kb/sec 7203 Kb/sec 9243 Kb/sec 15112 Kb/sec
FastestSmallestTextEncoderDecoder 48714 Kb/sec 83601 Kb/sec 149612 Kb/sec 14060 Kb/sec 21391 Kb/sec 31692 Kb/sec
Native 22584 Kb/sec 841778 Kb/sec 862736 Kb/sec 18169 Kb/sec 26682 Kb/sec 36591 Kb/sec

The tests below were performed on a mixed ascii-utf8 file.

Library Decode 32 bytes Decode 32768 Decode 16777216 Encode 32 bytes Encode 32768 Encode 16777216
fast-text-encoding 34694 Kb/sec 62305 Kb/sec 90462 Kb/sec 10175 Kb/sec 18690 Kb/sec 29335 Kb/sec
text-encoding) 4860 Kb/sec 5941 Kb/sec 7878 Kb/sec 6222 Kb/sec 3958 Kb/sec 7666 Kb/sec
TextEncoderTextDecoder.js 4639 Kb/sec 3678 Kb/sec 5418 Kb/sec 8741 Kb/sec 6811 Kb/sec 12253 Kb/sec
TextEncoderLite 8999 Kb/sec 10227 Kb/sec 10112 Kb/sec 10868 Kb/sec 7457 Kb/sec 12790 Kb/sec
text-encoding-shim 14081 Kb/sec 14852 Kb/sec 23365 Kb/sec 6699 Kb/sec 8255 Kb/sec 15838 Kb/sec
FastestSmallestTextEncoderDecoder 47975 Kb/sec 71299 Kb/sec 131499 Kb/sec 14427 Kb/sec 21483 Kb/sec 31780 Kb/sec
Native 24853 Kb/sec 353511 Kb/sec 521529 Kb/sec 18637 Kb/sec 26570 Kb/sec 36828 Kb/sec

For a more accurate real world test, we'll examine the 1876 The Russian Synodal Bible.txt. It's a whoping 4.4MB rat's-nest of complex Russian UTF-8, sure to give any encoder/decoder a bad day. Let's see how they perform at their worst.

Library Decode Russian Bible Encode Russian Bible
fast-text-encoding 93371 Kb/sec 33357 Kb/sec
text-encoding) 7396 Kb/sec 8386 Kb/sec
TextEncoderTextDecoder.js 4368 Kb/sec 13050 Kb/sec
TextEncoderLite 15828 Kb/sec 12953 Kb/sec
text-encoding-shim 29189 Kb/sec 17155 Kb/sec
FastestSmallestTextEncoderDecoder 148384 Kb/sec 35537 Kb/sec
Native 893880 Kb/sec 40933 Kb/sec

Browser Support

This polyfill will bring support for TextEncoder/TextDecoder to the following browsers.

Feature Chrome Firefox Opera Edge Internet Explorer Safari Android Samsung Internet Node.js
Full Polyfill 7.0 4.0 11.6 12.0** 10 5.1 (Desktop) / 4.2 (iOS) 4.0 1.0 3.0
Partial Polyfill* 1.0** 0.6 7.0 (Desktop) / 9.5** (Mobile) 12.0** 4.0 2.0 1.0** 1.0** 0.10

Also note that while this polyfill may work in these old browsers, it is very likely that the rest of one's website will not unless if one make a concious effort to have their code work in these old browsers.

* Partial polyfill means that Array (or Buffer in NodeJS) will be used instead of Uint8Array/[typedarray].

** This is the first public release of the browser

API Documentation

Please review the MDN at window.TextEncoder and window.TextDecoder for information on how to use TextEncoder and TextDecoder.

As for NodeJS, calling require("EncoderAndDecoderNodeJS.min.js") yields the following object. Note that this polyfill checks for global.TextEncoder and global.TextDecoder and returns the native implementation if available.

module.exports = { TextEncoder: function TextEncoder(){/*...*/}, TextDecoder: function TextDecoder(){/*...*/}, encode: TextEncoder.prototype.encode, decode: TextDecoder.prototype.decode }

In NodeJS, one does not ever have to use new just to get the encoder/decoder (although one still can do so if they want to). All of the code snippets below function identically (aside from unused local variables introduced into the scope).

 // Variation 1 const {TextEncoder, TextDecoder} = require("fastestsmallesttextencoderdecoder"); const encode = (new TextEncoder).encode; const decode = (new TextDecoder).decode;
 // Variation 2 const {encode, decode} = require("fastestsmallesttextencoderdecoder");
 // Variation 3 (a rewording of Variation 2) const encodeAndDecodeModule = require("fastestsmallesttextencoderdecoder"); const encode = encodeAndDecodeModule.encode; const decode = encodeAndDecodeModule.decode;

Or, one can use the new and shiny ES6 module importation statements.

 // Variation 1 import {TextEncoder, TextDecoder} from "fastestsmallesttextencoderdecoder"; const encode = (new TextEncoder).encode; const decode = (new TextDecoder).decode;
 // Variation 2 import {encode, decode} from "fastestsmallesttextencoderdecoder";
 // Variation 3 (a rewording of Variation 2) import * as encodeAndDecodeModule from "fastestsmallesttextencoderdecoder"; const encode = encodeAndDecodeModule.encode; const decode = encodeAndDecodeModule.decode;

Demonstration

Visit the GithubPage to see a demonstation. As seen in the Web Worker hexWorker.js, the Github Pages demonstration uses a special encoderAndDecoderForced.src.js version of this library to forcefully install the TextEncoder and TextDecoder even when there is native support. That way, this demonstraton should serve to truthfully demonstrate this polyfill.

npm Project

This project can be found on npm here at this link.

Development

On Linux, the project can be developed by cloning it with the following command line. The development scripts are designed to be interpeted by Dash, and whether they work on Mac OS is unknown, but they certainly won't work on Windows.

git clone https://github.com/anonyco/FastestSmallestTextEncoderDecoder.git; cd FastestSmallestTextEncoderDecoder; npm run install-dev

Emphasize the npm run install-dev, which downloads closure-compiler.jar into the repository for minifying the files.

Now that the repository is cloned, edit the files as one see fit. Do not edit the files in the encodeInto folder. Those are all auto-generated. Also, do not run npm run build in the encodeInto. That's done automatically when npm run build is runned in the topmost folder. Now that the files have been edited, run the following in the terminal in the root folder of the repository in order to minify the NodeJS JavaScript files.

npm run build

To edit tests, edit test/node.js. These tests are compared against the native implementation to ensure validity. To run tests, do the following.

npm run test

Continuity

I try my best to be a realist, and what's more realistic than death? I am going to die someday and it may be tomorrow in a car crash. You never know. As I have no coder freinds to look out for my projects, I'm looking for anyone who wants to be a collaborator on this project in the event of the unforseen. Reach out to me at wowzeryest@gmail.com. If issues/pulls start piling up over the course of months, assume the worst. As I am trying my best to do my part to help the community, I encourage every developer to share their projects with other people to ensure continuity.

About

The fastest smallest Javascript polyfill for encodeInto of TextEncoder, encode of TextEncoder, and decode of TextDecoder for UTF-8 only.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • JavaScript 94.4%
  • HTML 5.4%
  • Other 0.2%