0

We want to change how we deploy MongoDB on AWS because of three issues:

  1. the shared database is way too big (some clients are shared)
  2. even the separate databases are getting noisy-neighbour issues because the server itself is holding too much data (hitting bad IOPS levels)
  3. we want better disaster recovery

So I want to split things up into sets of replica sets where each set has three EC2 instances and handles between 1 and N databases. Some clients have the budget to get their own set, others need to still be shared somehow to avoid a min cost of too many EC2 instances. I would like to avoid sharding and just have different server setups that we administer separately with our internal tools. My question is: how to determine how many and how large of databases to put on each set of 3 server replicas? Are there good heuristics for determining this? Or particularly good resources for learning this dark art? I realize this is rather general, but I don't think it's opinion based, so hope it's ok.

Improve this question
1
  • I think you're going to need to do some benchmarking. Commented Nov 1, 2017 at 18:34

1 Answer 1

Reset to default
0

There are many variables in this situation (type of storage used, configuration of instances, average database size, average database complexity, average query/insert/update complexity....). So I can only provide general guidelines, but these may point you in the right direction:

In the current configuration: estimate the current average dataset size per customer, number of concurrent users per customer. That will give you a rough view of your average customer.

In the current configuration: if you have analysis tools available obtain some benchmark numbers on query response times, data load/update times, errors occurring on shared resources (cache, buffers, etc.).

A. Now you have some details on where you are.

  • Can you determine if/which of your customers are using more resources than others? (Candidates for separate databases or instances.)

  • Which of your customers have small datasets or need few resources -- candidates for sharing.

B. Estimate the needed configuration (computing options for EC2) and storage sizes for your new setup. You may have to do an average and just multiply by the number of instances needed.

C. Determine the total number of AWS EC2 instances needed, the total storage, users (both total and concurrent on each set).

D. Use calculators to help determine costs and further details of your new setup.

Some to use: cost of ownership: https://cloud.netapp.com/tco-calculator
AWS services cost (this is beta): https://calculator.s3.amazonaws.com/index.html

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.