0

I have multiple client e-commerce web applications running on a single VM, each web application is a node.js express application.

The express web application communicates with the back end via API's to retrieve the content for the pages and the list of products etc. However, to make the product searches quicker we're currently using an in-memory database (lokijs) for all search to happen against, this works pretty well for us at the moment as the average product catalogue for each client is only around 80 items, we have a simple cron running within each clients web app to refresh the in-memory database with the latest product list via an API call.

Some of the downsides to this are since we're using PM2 to run each application in a 2 node cluster, the in-memory database has to be duplicated as you cannot share memory between the nodes, to ensure that in-memory databases stay in sync we use IPC's to pass messages to all processes within the cluster.

As we're bringing larger clients on-board with larger product catalogues we don't really want to be keeping duplicate in-memory databases for each child process.

The way our offering works is although the product catalogues aren't large per client, the search volume is quite large, so a client with 80 items may still get 1000's of searches a day.

I had a couple of options in mind, however unsure what would be the best:

Option 1 - Single global elasticsearch cluster

I have spoken to and thought about using elasticsearch and then point every single client web application to a single global elasticsearch cluster to perform product searches against, this way we can use events to keep the elasticsearch database up to date in real time.

However, i don't know how this will perform as we scale up, i don't want elasticsearch to become a bottleneck

Option 2 - Local nosql database

The second option was to simply replace the use of lokijs as the in-memory database and have a shared nosql database (such as mongo) for each VM, all the web apps then use the local database for queries, each web app is still responsible for updating the product catalogue for their own store. Then as we add more VM's each VM will have it's own local database for any apps running on there to use

We are heavy users of faceted search, and aggregations on facets to get counts for the facets

Improve this question
12
  • Wait, what exactly is being duplicated in memory? Do different clients have identical product catalogues? Commented Dec 4, 2020 at 15:39
  • No the clients will have different product catalogues, but if i create a new in-memory datastore in node, that in-memory datastore can only be accessed by that one particular process, when spinning up 2 instances of the application they are effectively 2 individual processes and don't share memory, they each hold a copy of the clients product catalogue. They way to overcome that would be to move the datastore outside of node to a shared one such as mongodb and then both instances can connect to the same instance of mongodb Commented Dec 4, 2020 at 16:11
  • Seems more like something you would use redis for, (or Amazon's ElastiCache branded version) rather than mongodb. Commented Dec 4, 2020 at 16:43
  • We store our products as individual JSON documents, then need the ability to perform a search against the documents for things like name, price, colours etc - Don't think you can perform searches against JSON documents in redis as it just key-value? Commented Dec 4, 2020 at 16:54
  • No, you certainly need a document store. But your question was about caching, not your document storage. Commented Dec 4, 2020 at 16:56

0

Reset to default

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.