A few years ago I got introduced to reactive programming and Bacon.js, a FRP (functional reactive programming) library for Javascript. I got fond of it very fast, as it allowed you to create highly reactive UI’s with ease. While it works great in propagating changes to your views, it was still rather cumbersome to attach all the data flows to individual DOM nodes and properties.

React solved that problem completely. React makes the DOM reactive to your applications state, making your views predictable and declarative. Effectively, it allows you to create immutable views which get re-rendered each time the state of your application changes. However, recreating the entire DOM of your application for each state change would be far too expensive to be usable. React solves this problem by implementing a virtual DOM, which it uses to diff the changes that occur to the DOM, so that it’ll only perform the minimal changes it needs to make the DOM up-to date.

Flux is an application architecture commonly used with React. The architecture is based around unidirectional data flow. The Flux architecture fits well with Bacon.js and allows you to easily keep the application logic separate from individual views reducing coupling in your application. Facebook has a great introduction to the Flux architecture, so I won’t go into more detail regarding the basics of it and instead focus on how it can be applied with Bacon.js.

Mixin’ Bacon with React

My goal was to make my React components update their state automatically when one of their underlying Bacon streams would update. Similarly to the Flux architecture, I also wanted to keep my EventStreams (action creators) and stores away from my components, so that the application logic would reside with its logical domain for better cohesion. To keep the logic as simple and functional as possible, I opted to use the Immutable.js library for storing the data. Immutable.js provides persistent immutable data collections for JavaScript.

In your typical Flux setup, you would have a single central dispatcher which would transport all actions from your action creators to your stores. While this pattern could easily be followed with a Bacon.js implementation, it felt redundant as you can make each action an individual EventStream that can be subscribed and merged by stores as necessary. As a result, I attached each action creator method to form a separate EventStream.

The stores could then subscribe to the EventStreams (actions) that were relevant to them and perform the necessary updates to the store. As the changes to the store in itself results in another EventStream, it can easily be filtered, sorted, and mapped as necessary to the views, and exposed out for the views to subscribe to. Below is the full store logic for the todomvc application (which was based on Facebooks flux todomvc example):

var Immutable = require('immutable'); var Bacon = require('baconjs'); var Constants = require('../constants/AppConstants'); var create = text => Immutable.Map({ id: (+new Date() + Math.floor(Math.random() * 999999)).toString(36), complete: false, text: text }); var isComplete = todo => todo.get('complete'); var toJS = todos => todos.toList().toJS(); var applyModification = (todos, modification) => modification(todos); module.exports = function(actions) { var createTodo = actions.todo.create.map(create).map(todo => todos => todos.set(todo.get('id'), todo)); var destroyTodo = actions.todo.destroy.map(id => todos => todos.delete(id)); var updateText = actions.todo.updateText.map(todo => todos => todos.setIn([todo.id, "text"], todo.text)); var updateComplete = actions.todo.toggleComplete.map(todo => todos => todos.setIn([todo.id, "complete"], todo.complete)); var updateAllComplete = actions.todo.toggleCompleteAll.map(() => todos => todos.map(todo => todo.set('complete', true))); var destroyCompleted = actions.todo.destroyCompleted.map(() => todos => todos.filterNot(isComplete)); var todos = Bacon.mergeAll(createTodo, destroyTodo, updateText, updateComplete, updateAllComplete, destroyCompleted) .scan(Immutable.OrderedMap(), applyModification); var currentFilter = actions.todo.setFilter.toProperty(Constants.FILTER_ALL); var todoFilter = currentFilter.map(filter => { switch(filter) { case Constants.FILTER_ACTIVE: return todos => todos.filterNot(isComplete); case Constants.FILTER_COMPLETE: return todos => todos.filter(isComplete); default: return todos => todos } }); return { getAll: todos.map(toJS), getFiltered: todos.combine(todoFilter, applyModification).map(toJS), getFilter: currentFilter, allComplete: todos.map(todos => todos.every(todo => todo.get('complete'))) }; }; 

To simplify the binding of EventStreams to React components, I created a small mixin that updates the state of the component on updates to the stores EventStreams. The mixin takes care of applying the initial state (if synchronous), applying subscribers that update the state when the stores update, and cleaning up the subscribers when the component unmounts. Making the state of the components easily bound to the stores:

stateFromStores: stores => ({ allTodos: stores.todo.getAll, filteredTodos: stores.todo.getFiltered, areAllComplete: stores.todo.allComplete, filter: stores.todo.getFilter }) 

Isomorphic applications

Another goal I had was to simplify the creation of isomorphic applications with React. One of the common problems with doing this with React is that the library doesn’t provide any asynchronous methods for loading state. Unless you want to use fibers with something like react-async, the state of the application must be ready before you start to render the React application into markup. This means that stores must be populated with the required data before you can proceed to render it.

With having all your stores and component states as EventStreams (or any deferred computation for that matter), it greatly simplifies the process for loading the necessary data prior to proceeding with the render.

Conclusion

Replacing the EventEmitter with EventStreams from Bacon.js allowed me to make the application stores far simpler and purely functional. With the store mixin, keeping the components state up to date with the stores was easy as well. Creating generic EventStreams (effectively the same as Bacon.Bus) as action creators is a bit of an anti-pattern. However, in contrast to having the EventStreams created directly inside the components, by having them detached in the action creators it results in less coupling and makes testing easier.

The stack I used has three distinctive libraries in use; Bacon.js for transporting data and events, React for rendering the data into DOM, and Immutable.js for maintaining the data. With the relative low coupling that the Flux architecture provides, any one of those libraries that I used could be switched with relative ease if necessary.

I recreated the two Facebook Flux examples using Bacon.js:

• TodoMVC source (demo)
• Flux-Chat source (demo)

When I originally started working on the html2canvas project, I was trying to create a 3D representation of the webpage using WebGL. While I did end up getting a very elementary version created, it occurred to me that the real value of the project (if there was any) was in fact with a 2D representation of the webpage, or as some would call it, a “screenshot”. That’s where the project originally got started and the 3D rendering target eventually died off.

If you are familiar with browser render trees, and how they are formed, quite a lot of similarities can be found with html2canvas. The html2canvas library creates the “screenshot” of the page by evaluating the DOM. It iterates through every node on the page, evaluates the computed styles such as position, dimensions, borders, border-radius, colors, backgrounds, z-index, floats, overflows, opacity etc. Using this information, it forms a render queue which simply consists of calls to the canvas element, such as “draw shape with n-coordinates and fill it with x-color”. The canvas doesn’t eat CSS, so each CSS property that applies a style you want to render, needs to be implemented separately and as such, the projects scope is really never ending.

The fact that the text rendering methods available with canvas are slightly limited makes things just a bit more complicated. You can’t for example set letter-spacing when rendering text, so to apply the effect, you’ll need to calculate the position of each letter on a page manually and render them separately. Considering calculating the position of the text isn’t trivial to begin with, and requires different approaches depending on the browser, it does have performance implications.

Another problem area lies with rendering images. You can render most images to a canvas without a problem, but if the images aren’t from the same-origin as the page, they will end up tainting the canvas. In other words, the canvas won’t be readable anymore (getImageData/toDataURL). This in fact isn’t even limited to cross-origin images, but for example with Chrome, SVG images taint the canvas as well. To work around this problem, the library can attempt to sniff whether the image taints the canvas and ignore it if it does, or use a proxy to load the image so it can be safely drawn. The images can also be attempted to be loaded with CORS enabled, but unfortunately images are rarely served with the required headers.

With the render queue formed, its just a matter of applying the queue onto a canvas element and you’ll get your 2D “screenshot” of the webpage. However, the purpose of this blog post was to illustrate how to create a 3D representation of the webpage. It is in fact very easy to form the 3D representation using the screenshot as a diffuse map and just creating a heightmap using the same rendering queue we formed earlier. However, instead of using the calculated colors, you swap them to a color based on the DOM tree-depth of the element that formed the render item.

Using the heightmap, we can then form the appropriates vertices and triangles based on where the depth changes. I decided to pre-calculate the vertex positions instead of doing it within the vertex shader to avoid forming unnecessarily many vertices. While forming the vertices, each vertex will be assigned a color based on the pixel color from the diffuse map. For the vertices forming the vertical faces (i.e. depth differences), I applied a shadow depending on the direction of the face, to slightly give it a feel of some lighting.

To form a better view for distance and to add artificial detail to the page, I formed the fragment shader to draw squares on the faces, to give the illusion of them being constructed of small cubes (1 cube is approximately 1x1 pixel). Additionally, a small fog is applied in the fragment shader as well to give the depth even further clarity, as it may not be as evident with single colored faces.

For the physics and collision detection I set up a very simple detection based on the heightmap map data which just checks whether the camera is on the floor and whether it crosses a boundary that has a higher heightmap level than where it currently resides.

When considering what the library goes through to form 3D representation of a webpage, I personally find it quite amazing how fast some of the browsers are able to process it. Keeping in mind that the library could be evaluating thousands of nodes, tens of thousands of letters, for each, processing lots of different CSS properties that can result up to hundred thousand calls to the canvas drawing context. From there, it forms the geometry which can easily end up of consisting over a million vertices, with half a million triangles, which it dumps into the GPU and a full 3D representation of the webpage pops up within a second or two.

Of course the type of page and the amount of content impacts a lot on the performance, as well as the type of computer/browser you use and how many images needs to be pre-loaded prior to parsing the DOM. If you want to give 3D browsing a go, I’ve added it to my own homepage, and its accessible with the hashtag #3d. If you want to try it on your own page, you can get the built script from here or view the sources at GitHub.

Please keep in mind the image cross-origin limitations. If you wish to try the script without hassle on any page, you can use it through this chrome extension, which works around the cross-origin image issue.

If injecting unknown scripts into your browsers console is your thing, here is a snippet ready for you:

(function(d) { var s = d.createElement("script"); s.src = "http://hertzen.com/js/domfps.min.js?v2"; d.body.appendChild(s); })(document);

Or you can execute it on this page by clicking here.

About a year ago I setup click event tracking up for *hertzen.com with the intention of doing some click heatmaps with the data. Obviously, I completely forgot about it within a few weeks and had already moved onto other things. Now, roughly a year later, well over a million click events had been recorded, with about 6000 new click events each weekday on average.

For recording the clicks, I setup custom event tracking to record the coordinates of mouse clicks. As client resolution, page path, host etc. are all already recorded by default, no other data was necessary. I labelled these events as “Click heatmap” to make them filterable from other custom events. However, a major mistake I did here was not to take into account window.devicePixelRatio, which resulted in a lot of the higher resolution click events recording unusable or invalid data (which is quite evident from the heatmaps created).

For retrieving the data from the Google Analytics API, I setup a nodejs application that uses Google service accounts to authenticate to the API, so that no user credentials need to be stored in the application as it authenticates through signed JWTs. The application loads the clicks for n number of most used resolutions and concatenates the data together. It can optionally combine multiple resolutions’ data together into a single data set, by computing the center offsets of the clicks based on their resolution.

Once the data was downloaded from the API, I got to the more interesting part of creating the heatmaps using canvas. The heatmap implementation I created draws grayscale radial gradients with a low opacity for each click. Multiple clicks around an area increase the alpha of the pixels as there are multiple low opacity gradients overlapping.

After all the clicks have been drawn, the canvas pixel data is extracted with getImageData so that the contents of the canvas can be read through pixel by pixel and a heat color can be assigned depending on the alpha of the pixel. A higher alpha means a “warmer” color. Each alpha value correspond to a color on a gradient from [0-255]. Once each pixel has been processed, the image data is returned to the canvas and the frame is rendered.

Finally, I added some date filters to the data so that heatmap wouldn’t be bloated with too much content. The heatmaps are more difficult to use if the content or the position of the content on the page changes frequently, as it will require you to be previewing the correct state of the page while viewing the heatmap. As far as my two index page examples below, the html2canvas page had some minor changes about half a year ago, which is evident from the button positions changing.

Examples:

• hertzen.com
• html2canvas.hertzen.com

Source code:

• github

This has taken far longer than I had hoped for it to do. Partly because of work on some other projects and partly due to lack of time. As far as the update is concerned though, it is the biggest one so far for html2canvas.

Perhaps the biggest improvement that comes with 0.4.0 are the automated rendering tests that allow lot easier updating of the code while still having assurance that the changes do not affect other parts of the rendering process. As I discussed in the previous blog post, the tests are ran through a grunt task and they use webdriver to launch real instances of Chrome, Firefox and IE to capture what the page should look like, and do a pixel comparison with the output generated by html2canvas. Current rendering results can be found here.

In addition, most of the code in the library has been refactored and in the process support for IE<9 was dropped as well. This means that even with the use of Flashcanvas or similar canvas emulation, html2canvas won’t work for IE8 or older.

As for new features, background rendering was improved, and support for background-clipping, sizing, and multiple backgrounds was added. The method used for border rendering was recoded, and along with it support for border-radius and box-sizing was added. In addition, support for pseudo elements (:before and :after) was added. For a full list of changes, check out the commit history.

As for the build process, html2canvas now uses grunt for linting, testing (qunit and webdriver) and building the files. The readme has been updated with instructions on how to build the library using grunt.

Finally, the documentation for the library got some long needed updates, and although there is still more that could be done there, it should at least be up to date now.

Automated testing for html2canvas has been an issue for a long time already. The library has had a number of qunit tests that check that different parsed DOM values are correctly calculated across all browsers, but they only touch the surface of the testing requirements of the library.

Problem

The purpose of html2canvas is to generate as close of a representation of the DOM, as it is rendered by the browser the script is ran on. In other words, there is no one clear result it should render for any given page, but instead it should attempt to represent the page as it is rendered with that particular browser on that particular screen with any browser specific issues that may be present. If the browser doesn’t support some particular CSS properties, the result shouldn’t render them either.

This meant that there couldn’t really be any premade screen renders that could be used to compare the results generated by html2canvas, as the results would and should vary between browsers and systems.

Approach

Version 0.4 will introduce testing capabilities with webdriver which will allow automating the testing on a number of different browsers, while still taking into account the expected differences in results.

The tests capture a screenshot of the actual browser, after which html2canvas runs and renders its representation of the page. The base64 encoded png images are then sent back to node where they are first converted into an arraybuffer and then into pixel arrays. It then calculates the percentage difference between the two images by comparing them pixel by pixel. The results are then compared to previous baseline values which allows us to analyze whether there has been any changes in the results. Sample results from the tests can be previewed here.

The reason the html2canvas render is sent as a base64 encoded string and converted into a pixel array in node instead of simply sending a CanvasPixelArray is because it is a lot faster.

Analyzing the results

The results vary a lot depending on the browser, with Chrome generating the most accurate results. The slightly lower results in Firefox are mainly caused by slight displacement of some texts and some aliasing issues. For IE9, the results with text are lot worse, which is in part to be expected considering there is no support for getBoundingClientRect for text ranges and a slightly less accurate method of finding text positions have to be used.

With the automated tests in place, updating and improving the library is significantly easier and less time consuming, and will hopefully result in more frequent updates from me as well. Overall, it was satisfying to see that there were a number of tests that are 100% accurate to what the corresponding browser renders.