Offline caching

[AVGP]

After seeing the question about offline appear a few times in the A-Frame slack & Stack Overflow, I had the epiphany on the airplane, that we can enable people using Service Workers to have a much easier time adding data3d + textures to their cache.

It turns out the Cache API is available from the mainthread as well and we can detect if it's available or not (you can test that with the stable Safari that is lacking this feature).

I'll add an example as well, but basically this is how it works:

• The developer can present the user with a UI to make io3d-data3d available offline by suppying the URL and a cache name (because people can name their custom cache for later usage in the service worker). The new method io3d.utils.data3d.storeInCache parses the materials and gets the additional resources (i.e. textures) and puts them in the specified cache.

When the developer adds a service worker that reads from the cache, the model will work just fine even when the user is offline and it loads nearly instantaneously when the developer decides for a cache-first-update-in-the-background workflow. The choice is with the developer, who are used to this when using Service Workers.

[stavros-papadopoulos]

Adding my two satoshis here:

This is hugely important IMHO. as it's not just about being able to work offline. There are various levels of 'being online' as well: there's 'being online at the Zürich office' (1 Gbps) and then there's the 'working at a Starbucks in Athens' kind of online (8 Mbps ADSL 1, actual 5-7 Mbps, shared by 20 people, 3 of which are watching video). I'm sure that Athens is not the worst of it either (the US is actually behind many parts of Europe in network connectivity and bandwidth).

Being able to somehow store and use your model locally would be a huge win. I've had to face this problem since the very beginning while working on the editor in Athens (which has to load other resources as well on top of the 3D model, such as the furniture thumbnails).

The viewer solves this by embedding drag & drop support directly in the ModelIO subsystem, so that resources that are dropped on the canvas take the exact same code path as if they've been XHR'ed in very, very quickly, before reaching 'viewer proper'.

Having direct 3d.io support for local storage/access either via Martin's proposed API or in any other form that anybody could use would be great and a priority IMHO.

[AVGP]

Thanks Stavros, I completely failed to mention: 1.) This makes the online experience faster, too as models are "instantly" loaded. 2.) Yes, there's "Lie-Fi" (i.e. at airports - you're connected, but there's a capturing portal in the middle, so no "real" internet connection) 3.) Markets such as Asia (especially China) will require this. I've seen the question "Does this work in <country>?" three times in the last couple of months.. For what it's worth: I should mention that for development, you can also export a model as data3d and then load it from your local webserver: <a-entity io3d-data3d="model/export.data3d.buffer"></a-entity> works - but those are two different (but somewhat connected) concerns :) So, yes, thanks Stavros for fixing the huge hole I left in the description (I'll buy the first round on Tuesday!) and I'd love to see this land ^^ 2017-10-14 15:55 GMT+03:00 stavros-papadopoulos <notifications@github.com>:

…

Adding my two satoshis here: This is hugely important IMHO. as it's not just about being able to work offline. There are various levels of 'being online' as well: there's 'being online at the Zurich office' (1 Gbps) and then there's the 'working at a Starbucks in Athens' kind of online (8 Mbps ADSL 1, actual 5-7 Mbps, shared by 20 people, 3 of which are watching video). I'm sure that Athens is not the worst of it, either (the US is actually behind many parts of Europe in network connectivity and bandwidth). Being able to somehow store and use your model locally would be a huge win. I've had to face this problem since the very beginning while working on the editor in Athens (which has to load other resources as well on top of the 3D model, such as the furniture thumbnails). The viewer solves this by embedding drag & drop support directly in the ModelIO subsystem, so that resources that are dropped on the canvas take the exact same code path as if they've been XHR'ed in when they reach 'viewer proper'. Having direct 3d.io support for local storage/access that anybody could use would be great! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#86 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAWmRqItH7jiiU6lNdw647YrjC-3zp4Kks5ssK9OgaJpZM4P5DiK> .

[stavros-papadopoulos]

Keeping the 3d.io API's surface area as small as possible (especially at this early point, when not all things are as clear as they will be) should be IMHO a primary design consideration, so that would be my only reservation at this point ('reservation' is not quite the right term - 'factor I would weigh this against' is better).

As I'm not really familiar with the Cache API (example: is it here to stay?) or at the heart of 3d.io's side of things at the moment, I'll leave this to your good judgement my good Sir!

[AVGP]

The Cache API finally gives developers a mechanism to create an HTTP cache under their own control, which is pretty neat.

It's relatively straight forward - yet not the simplest thing.
Have a look at

3dio-js/src/utils/data3d/store-in-cache.js

Lines 5 to 14 in 7416cbd

	export default function storeInCache (url, cacheName) {
	if (!isCacheAvailable()) return Promise.reject()
	return caches.open(cacheName || '3dio-data3d').then(function onCacheReady(cache) {
	return loadData3d(url).then(function onData3dReady(data3d) {
	var cacheUrls = new Array(url).concat(getTextureUrls(data3d))
	return cache.addAll(cacheUrls)
	})
	})
	}

1. We create a cache (with the name given in the parameter or a default name)
2. Once we have the cache, we load the data3d file
3. We extract all related assets
4. We call cache.putAll(allTheUrls) to put the data3d and the related assets into our cache

These caches are usually accessed by the service worker, which can implement a bunch of different patterns to decide how it deals with network requests before these requests are actually performed.

For instance, in

3dio-js/examples-browser/offline-data/sw.js

Lines 16 to 29 in 7416cbd

	self.addEventListener('fetch', function (evt) {
	// skip POST requests so the browser default kicks in
	if(evt.request.method === 'POST') {
	return
	}
	console.log('The service worker is serving the asset.');
	// You can use `respondWith()` to answer immediately, without waiting for the
	// network response to reach the service worker...
	evt.respondWith(fromCache(evt.request));
	// ...and `waitUntil()` to prevent the worker from being killed until the
	// cache is updated.
	evt.waitUntil(update(evt.request));
	});

we do the following:

1. if it's a POST request, just let the browser do it as if there was no Service Worker
2. We create a response with whatever fetchFromCache returns* - so we always give you what we've got in cache immediately and then
3. We go to the network to update our cache, if the network

*) When there's nothing in the cache, we don't respond yet. In that case, only the updateCache function will return something the browser will use to "respond" to whatever made that network request.

It's also good to mention that the service worker gets to decide for every network request. HTML, CSS, JS, images - from within CSS, HTML and JS. It's not just AJAX requests (like XHR or fetch(...)).

Our API surface only expands by two pretty low-level methods but the gain for the developers is huge as they don't need to recreate this building block and they can plug it into the ready-to-use service worker examples without having to find out how they could populate the cache with the entire model.

The example illustrates that by using a service worker from the service worker cookbook & the new methods

[stavros-papadopoulos]

Looks perfectly fine to me, although I would be using it more like a simple local storage (i.e. don't update the cache, don't even touch the network, just fetch me the resource or give me an error if it's not there). I like to be explicit about things and I want to know when something is being retrieved locally or from the network.

I assume this can be programmed, right?

[AVGP]

That's entirely up to you, yes. You can write a different service worker and make other do whatever you prefer! It could even create the model from scratch and use neither the cache nor the network 😂 So, yes, it's programmable and the way it interacts is up to the service worker, like the one provided in the example (there given strategy is the most popular one in web applications).

…

On 16 Oct 2017 7:23 pm, "stavros-papadopoulos" ***@***.***> wrote: Looks perfectly fine to me, although I would be using it more like a simple local storage (i.e. don't update the cache, don't even touch the network, just fetch me the resource or give me an error if it's not there). I like to be explicit about things and I want to know when something is being retrieved locally or from the network. I assume this can be programmed, right? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#86 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAWmRh1ZyIPG9OQBZP0OdFHQDAZKai5mks5ss5EdgaJpZM4P5DiK> .