Migration tool documentation

_data/3.10-manual.yml

@@ -467,7 +467,7 @@
  - text: Monitoring
  href: monitoring-dc2dc.html
  - text: Troubleshooting
- href: troubleshooting-dc2dc.html
+ href: troubleshooting-dc2dc.html 
 - text: Foxx Microservices
  href: foxx.html
  children:

_data/3.11-manual.yml

@@ -467,7 +467,7 @@
  - text: Monitoring
  href: monitoring-dc2dc.html
  - text: Troubleshooting
- href: troubleshooting-dc2dc.html
+ href: troubleshooting-dc2dc.html 
 - text: Foxx Microservices
  href: foxx.html
  children:

_data/oasis.yml

@@ -13,6 +13,8 @@
  href: projects.html
 - text: Deployments
  href: deployments.html
+- text: Migrate to the cloud
+ href: cloud-migration-tool.html 
 - text: My Account
  href: my-account.html
 - text: Monitoring & Metrics

_plugins/HintBlock.rb

@@ -9,12 +9,15 @@ def render(context)
  converter = site.find_converter_instance(::Jekyll::Converters::Markdown)
 
  # Remove indention based on first actual line (but only spaces)
- content = super.sub(/^[\r\n]+/, '')
+ content = super.sub(/^\R+/, '')
+ #content.rstrip!
  indent = content.index(/[^ ]/) || 0
  content = content.lines.map{ |line| line.slice([indent, line.index(/[^ ]/) || 0].min, line.length) }.join ''
- # Parse Markdown and strip trailing whitespace (especially line breaks).
- # Otherwise below <div>s will be wrapped in <p> for some reason!
- content = converter.convert(content).sub(/\s+$/, '')
+ # Parse Markdown and collapse line breaks to spaces.
+ # Line breaks can cause pre-mature closing of elements it seems,
+ # observed when placing a hint block in a Markdown list,
+ # leading to unwanted literal "</div>" (HTML-encoded) in the output.
+ content = converter.convert(content).gsub(/\R+/, ' ')
 
  case @type
  when "tip"
@@ -35,9 +38,9 @@ def render(context)
  end
 
  return "<div class=\"alert alert-#{className}\" style=\"display: flex\">" +
- "<i class=\"fa fa-#{icon}\" style=\"margin-right: 10px; margin-top: 4px;\"></i>" +
- "<div>#{content}</div>" +
- "</div>"
+  "<i class=\"fa fa-#{icon}\" style=\"margin-right: 10px; margin-top: 4px;\"></i>" +
+  "<div>#{content}</div>" +
+  "</div>"
  end
  end
  Liquid::Template.register_tag('hint', HintTag)

oasis/cloud-migration-tool.md

@@ -0,0 +1,228 @@
+---
+layout: default
+description: >-
+ Migrating data from bare metal servers to the cloud with minimal downtime
+---
+# Cloud Migration Tool
+
+{{ page.description }}
+{:class="lead"}
+
+The `arangosync-migration` tool allows you to easily move from on-premises to 
+the cloud while ensuring a smooth transition with minimal downtime.
+Start the cloud migration, let the tool do the job and, at the same time,
+keep your local cluster up and running. 
+
+Some of the key benefits of the cloud migration tool include:
+- Safety comes first - pre-checks and potential failures are carefully handled.
+- Your data is secure and fully encrypted.
+- Ease-of-use with a live migration while your local cluster is still in use.
+- Get access to what a cloud-based fully managed service has to offer: 
+high availability and reliability, elastic scalability, and much more.
+
+## Prerequisites 
+
+Before getting started, make sure the following prerequisites are in place:
+
+- Go to [Oasis Cloud](https://cloud.arangodb.com/home){:target="_blank"}
+and sign in. If you don’t have an account yet, sign-up to create one.
+
+- Generate an Oasis API key and API secret. See a detailed guide on 
+[how to create an API key](api-getting-started.html#creating-an-api-key).
+
+### Setting up the target deployment in Oasis
+
+Continue by [creating a new Oasis deployment](deployments.html#how-to-create-a-new-deployment)
+or choose an existing one.
+
+The target deployment in Oasis requires specific configuration rules to be
+set up before the migration can start:
+
+- **Configuration settings**: The target deployment must be compatible with the
+source data cluster. This includes the ArangoDB version that is being used,
+the DB-Servers count, and disk space.
+- **Deployment region and cloud provider**: Choose the closest region to your
+data cluster. This factor can speed up your migration to the cloud.
+
+After setting up your Oasis deployment, wait for a few minutes to become 
+fully operational.
+
+## Running the migration tool
+
+The `arangosync-migration` tool provides a set of commands that allow you to:
+- start the migration process
+- check whether your source and target clusters are fully compatible
+- get the current status of the migration process
+- stop or abort the migration process
+- switch the local cluster to read-only mode
+
+### Starting the migration process
+
+To start the migration process, run the following command:
+
+```bash
+arangosync-migration start
+```
+The `start` command runs some pre-checks. Among other things, it measures
+the disk space which is occupied by your ArangoDB cluster. If you are using the
+same data volume for ArangoDB servers and other data as well, the measurements
+can be incorrect. Provide the `--source.ignore-metrics` option to overcome this.
+
+You also have the option of doing a `--check-only` without starting the actual
+migration. If specified, this checks if your local cluster and target deployment
+are compatible without sending any data to Oasis.
+
+Once the migration starts, the local cluster enters into monitoring mode and the
+synchronization status is displayed in real-time. If you don't want to see the
+status you can also terminate this process, as the underlying agent process
+continues to work. If something goes wrong, restarting the same command restores
+the replication state.
+
+To restart the migration, first `stop` or `stop --abort` the migration. Then,
+start it again using the `start` command.
+Note that restarting works only if you are not using
+auto-generated certificates.
+
+{% hint 'warning' %}
+Starting the migration creates a full copy of all data from the source cluster
+to the target deployment in Oasis. All data that has previously existed in the
+target deployment will be lost.
+{% endhint %}
+
+### During the migration...
+
+What happens during active migration:
+- The source data cluster remains usable. 
+- The target deployment in Oasis is switched to read-only mode.
+- Your root user password is not copied to the target deployment in Oasis.
+To get your root password, select the target deployment from the Oasis
+Dashboard and go to the **Overview** tab. All other users are fully synchronized.
+
+{% hint 'warning' %}
+The migration tool increases the CPU and memory usage of the server you are
+running on. Depending on your ArangoDB usage pattern, it may take a lot of CPU
+to handle the replication. You can stop the migration process anytime
+if you see any problems.
+{% endhint %}
+
+During active migration, the agent HTTPS server is executed on the `$MG_HOST:8629` 
+endpoint. Make sure the host and port are available for the Oasis platform.
+To change the default port, use the `--agent.master-port` option.
+
+```bash
+export MG_HOST=<your IP or publicly-available hostname here>
+./arangosync-migration start \
+ --source.cacert=tls-ca.crt \
+ --source.keyfile=client-auth.keyfile \
+ --source.endpoint=$COORDINATOR_ENDPOINT \
+ --source.jwtSecret=clusterSecret \
+ --oasis.api-key=$OASIS_API_KEY \
+ --oasis.api-secret=$OASIS_API_SECRET \
+ --oasis.deployment-id=$OASIS_DEPLOYMENT_ID \
+ --agent.listen-host=$MG_HOST
+```
+
+### TLS server certificates
+
+The migration agent HTTPS server uses TLS certificate pairs to ensure a secure
+connection between your local cluster and the Oasis platform.
+If you do not provide them, the migration tool creates self-signed certificates.
+If you wish to provide TLS certificates, use the `arangodb` tool to convert them in
+a suitable format for the migration tool.
+See a detailed guide on how to [create a new certificate/keyfile pair](../programs-starter-security.html).
+Make sure to specify your publicly available host name, `$MG_HOST` when creating
+the keyfile. 
+
+When starting the migration, specify the generated files on the command line:
+
+```bash
+--agent.cacert=tls.crt --agent.keyfile=tls.keyfile --agent.client-auth-cacert=client-auth-ca.crt --agent.client-auth-keyfile=client-auth.keyfile
+```
+
+### How long does it take?
+
+The total time required to complete the migration depends on how much data you
+have and how often write operations are executed during the process.
+
+You can also track the progress by checking the **Migration status** section of
+your target deployment in Oasis dashboard.
+
+![Oasis Cloud Migration Progress](images/oasis-migration-agent.png)
+
+### Getting the current status
+
+To print the current status of the migration, run the following command:
+
+```bash
+./arangosync-migration status --watch \
+ --oasis.api-key=$OASIS_API_KEY \
+ --oasis.api-secret=$OASIS_API_SECRET \
+ --oasis.deployment-id=$OASIS_DEPLOYMENT_ID
+```
+
+You can also add the `--watch` option to start monitoring the status in real-time.
+
+### Stopping the migration process
+
+The `arangosync-migration stop` command stops the migration and terminates
+the migration agent process.
+
+If replication is running normally, the command waits until all shards are
+in sync. The local cluster is then switched into read-only mode.
+After all shards are in-sync and the migration stopped, the target deployment
+is switched into normal mode (read/write) while the source cluster stays in
+read-only mode. 
+
+```bash
+./arangosync-migration stop \
+ --oasis.api-key=$OASIS_API_KEY \
+ --oasis.api-secret=$OASIS_API_SECRET \
+ --oasis.deployment-id=$OASIS_DEPLOYMENT_ID
+```
+
+An optional `--abort` option is supported. If specified, the `stop` command 
+will not check anymore if both deployments are in-sync and stops all
+migration-related processes as soon as possible.
+
+### Switching the local cluster to read-only mode
+
+The `arangosync-migration set-server-mode` command allows switching [read-only mode](../http/administration-and-monitoring.html#update-whether-or-not-a-server-is-in-read-only-mode)
+for your local cluster on and off.
+
+In a read-only mode, all write operations are going to fail with an error code
+of `1004` (ERROR_READ_ONLY).
+Creating or dropping databases and collections are also going to fail with 
+error code `11` (ERROR_FORBIDDEN).
+
+```bash
+./arangosync-migration set-server-mode \
+ --source.cacert=tls-ca.crt \
+ --source.keyfile=client-auth.keyfile \
+ --source.endpoint=$COORDINATOR_ENDPOINT \
+ --source.jwtSecret=clusterSecret \
+ --source.server-mode=readonly
+``` 
+The `--source.server-mode` option allows you to specify the desired server mode.
+Allowed values are `readonly` or `default`.
+
+## Cloud migration workflow for minimal downtime
+
+1. Download and start the `arangosync-migration` tool. The target deployment
+ is switched into read-only mode automatically.
+2. Wait until all shards are in sync. You can use the `status` or the `start`
+ command with the same parameters to track that.
+3. Optional: when all shards are in-sync, you can switch your applications
+ to use the Oasis deployment, but note that it stays into read-only mode
+ until the migration process is fully completed.
+4. Stop the migration using the `stop` subcommand. What happens:
+ - The source data cluster is switched into read-only mode.
+ - It waits until all shards are synchronized.
+ - The target deployment is switched into default read/write mode.
+
+ {% hint 'info' %}
+ After finishing the migration, the source data cluster will remain read-only. 
+ You can use the `set-server-mode` 
+ subcommand to switch it back to default, if needed.
+ In case something goes wrong during the migration, the `stop` command is not
+ switching the source data cluster into read-only mode. 
+ {% endhint %}

oasis/deployments.md

@@ -37,7 +37,7 @@ Also see the video
  The configuration options depend on the tier your organization belongs to.
  For more details about available resources and usage limits, refer to the 
  [Oasis tiers](organizations.html#oasis-tiers) section.
- {% endhint %} 
+ {% endhint %}
 
 ![Oasis New Deployment](images/oasis-new-deployment1.png)
 
@@ -228,22 +228,22 @@ For more details about available resources and usage limits, refer to the
 5. In the **Configuration** section, you can do the following:
  - Upgrade the memory size per node. 
  - Modify the CPU per node from General to Low or vice-versa, if made available
- by the cloud provider.
+  by the cloud provider.
  - Select a different disk size per node. The available ranges for the disk size
- depend on the selected memory size. To enable automatic disk size scaling, move
- the slider to a value higher than the current disk size.
+  depend on the selected memory size. To enable automatic disk size scaling, move
+  the slider to a value higher than the current disk size.
  - Change **OneShard** deployments into **Sharded** deployments. To do so,
- click **Sharded**. In addition to the other configuration options, you can
- select the number of nodes for your deployment. This can also be modified later on.
+  click **Sharded**. In addition to the other configuration options, you can
+  select the number of nodes for your deployment. This can also be modified later on.
 
  {% hint 'warning' %}
  Notice that you cannot switch from **Sharded** back to **OneShard**.
  {% endhint %}
 
 - AWS deployments have an additional option that allows you to select the
- **Disk Performance** either with general settings, or optimised for large
- and very large data sets. This option is dependent on the selected memory
- size. For example, larger deployments have optimised settings by default.
+  **Disk Performance** either with general settings, or optimised for large
+  and very large data sets. This option is dependent on the selected memory
+  size. For example, larger deployments have optimised settings by default.
 
  {% hint 'warning' %}
  When upgrading the memory size, disk size, and/or disk performance in AWS deployments,