DOC-771 | Review and update production checklist (#784)

* review and update production checklist * review * add note on enabling RocksDB statistics and other minor changes * added more best practices for the production checklist via mamoona

Author: nerpaula

site/content/3.12/deploy/production-checklist.md

@@ -10,34 +10,89 @@ have been performed on your production system before you go live.
 
 ## Operating System
 
-- Executed the OS optimization scripts if you run ArangoDB on Linux.
+- Executed the operating system (OS) optimization scripts if you run ArangoDB on Linux.
  See [Installing ArangoDB on Linux](../operations/installation/linux/_index.md) and its sub pages
  [Linux Operating System Configuration](../operations/installation/linux/operating-system-configuration.md) and
  [Linux OS Tuning Script Examples](../operations/installation/linux/linux-os-tuning-script-examples.md) for details.
 
-- OS monitoring is in place
- (most common metrics, e.g. disk, CPU, RAM utilization).
+- Ensure your OS is compatible with your ArangoDB version
+ and keep it up to date at all times for security and stability.
+
+- OS monitoring is in place with specific alerting thresholds:
+ - **Disk usage**: Alert when reaching 60% (red line threshold).
+ - **CPU usage**: Alert when reaching 90% (red line threshold).
+ - **Memory usage**: Alert when reaching 85% (red line threshold).
 
 - Disk space monitoring is in place. Consider setting up alerting to avoid out-of-disk situations.
 
 ## ArangoDB
 
-- The user _root_ is not used to run any ArangoDB processes
+- **Use the latest versions**: Deploy the latest version series
+ of ArangoDB to benefit from performance improvements and security fixes.
+
+- **Testing environments**: Use QA environments and UAT (User Acceptance Testing)
+ to test all changes, in particular queries, before going live with production deployments.
+
+### Security
+
+- Create a dedicated system user and group (e.g., "arango")
+ to run ArangoDB processes. Never use the _root_ user to run any ArangoDB processes
  (if you run ArangoDB on Linux).
 
+- **Access control**: Restrict access to the deployment to authorized personnel only.
+ Implement proper authentication and authorization mechanisms.
+
+- **JWT authentication**: Enable JWT authentication
+ for production deployments. See [JWT authentication](../develop/http-api/authentication.md#jwt-user-tokens) for more details.
+
+- **Encryption**: Enable [Encryption at Rest](../operations/security/encryption-at-rest.md)
+ for sensitive data. Make sure to safely store any secret keys you create for this.
+
+### Logging and Monitoring
+
 - The _arangod_ (server) process and the _arangodb_ (_Starter_) process
  (if in use) have some form of logging enabled and logs can easily be
  located and inspected.
- 
-- *Memory considerations*
- - If you run multiple processes (e.g. DB-Server and Coordinator) on a single
- machine, adjust the [`ARANGODB_OVERRIDE_DETECTED_TOTAL_MEMORY`](../components/arangodb-server/environment-variables.md)
- environment variable accordingly.
- - For versions prior to 3.8, make sure to change the
- [`--query.memory-limit`](../components/arangodb-server/options.md#--querymemory-limit)
- query option according to the node size and workload.
- - Disable swap space to avoid slowdown which can result in servers being incorrectly 
- detected as failed.
+
+- **Third-party monitoring**: Configure third-party metrics monitoring tools like
+ Grafana with Prometheus to monitor ArangoDB metrics comprehensively.
+
+- **Configure metrics collection**: Enable the ArangoDB metrics API for production monitoring:
+ - Set [`--server.export-metrics-api`](../components/arangodb-server/options.md#--serverexport-metrics-api) to `true` to enable the metrics endpoints
+ - Enable [`--server.export-read-write-metrics`](../components/arangodb-server/options.md#--serverexport-read-write-metrics) for additional document read/write metrics
+ - Consider enabling [`--server.export-shard-usage-metrics`](../components/arangodb-server/options.md#--serverexport-shard-usage-metrics) for detailed shard usage tracking
+ - Configure your monitoring system (Prometheus/Grafana) to scrape the `/_admin/metrics/v2` endpoint
+ - See [HTTP interface for server metrics](../develop/http-api/monitoring/metrics.md) for detailed information
+
+- **Enable RocksDB statistics**: Consider enabling [`--rocksdb.enable-statistics`](../components/arangodb-server/options.md#--rocksdbenable-statistics) to `true` for detailed RocksDB performance metrics.
+
+- Monitor the ArangoDB provided metrics with alerting based on the threshold guidelines:
+ - Disk usage: 60% (red line)
+ - CPU usage: 90% (red line)
+ - Memory usage: 85% (red line)
+
+### Memory
+
+- For DB-Servers and Coordinators, override the
+ [`ARANGODB_OVERRIDE_DETECTED_TOTAL_MEMORY`](../components/arangodb-server/environment-variables.md)
+ environment variable using this rule of thumb:
+ - Multiply available memory by 0.9 to leave headspace for OS/Kubernetes, client connections, etc.
+ - Use 3/4 of that value for DB-Servers.
+ - Use 1/4 of that value for Coordinators.
+ - Agents typically don't need much memory and can use the remaining 10% headspace.
+
+- Note that if ArangoDB "sees" x GB of memory in a pod,
+ it will try to use those x GB. Memory accounting has been vastly improved in 3.12,
+ but overshooting in certain cases may still occur.
+
+- Disable swap space to avoid slowdown which can result in servers being incorrectly 
+ detected as failed.
+
+- **Query memory limits**: Configure appropriate memory limits for AQL queries:
+ - Set [`--query.max-memory-per-query`](../components/arangodb-server/options.md#--querymax-memory-per-query) to limit memory usage per individual query.
+ - Consider setting [`--query.global-memory-limit`](../components/arangodb-server/options.md#--queryglobal-memory-limit) to limit total memory used by all concurrent queries.
+
+### Service Management
 
 - Ensure ArangoDB will be automatically restarted (e.g. by using a systemd service file). Typically
  you would use the Kubernetes operator or use systemd to launch the _Starter_.
@@ -50,36 +105,56 @@ have been performed on your production system before you go live.
  update-rc.d -f arangodb3 remove
  ```
 
-- If you have deployed a Cluster, the _replication factor_ and 
- _minimal_replication_factor_ of your collections
- are set to a value equal or higher than 2, otherwise you run the risk of
- losing data in case of a node failure. See
- [cluster startup options](../components/arangodb-server/options.md#cluster).
-
-- *Disk Performance considerations*
- - Verify that your **storage performance** is at least 100 IOPS for each
- volume in production mode. This is the bare minimum and it's recommended to
- provide more for performance. It is probably only a concern if you use a
- cloud infrastructure. Note that IOPS might be allotted based on a volume size,
- so make sure to check your storage provider for details. Furthermore, you should
- be careful with burst mode guarantees as ArangoDB requires a sustainable
- high IOPS rate. 
-
- - The considerations should be given to an IO bandwidth (especially considering 
- RocksDB write-amplification which can easily be 10x or more).
-
-- Whenever possible use **block storage**. Database data is based on append
- operations, so filesystem which support this should be used for best
- performance. We would not recommend to use NFS for performance reasons,
+### Cluster Configuration
+
+- **Replication configuration**: For production clusters, configure collections with:
+ - _replication factor_ of 3 for optimal data availability and fault tolerance.
+ - _minimal_replication_factor_ of a value equal or higher than 2.
+ - _writeConcern_ of 2.
+ See [cluster startup options](../components/arangodb-server/options.md#cluster).
+
+- **Shard limits**: Keep the total number of shards below 10,000 across your cluster
+ to maintain optimal performance and avoid resource exhaustion.
+
+### Disk Performance
+
+- **Storage performance**: Verify that your storage performance is at least 100 IOPS for each
+ volume in production mode. This is the bare minimum and it's recommended to
+ provide more for performance. It is probably only a concern if you use a
+ cloud infrastructure. Note that IOPS might be allotted based on a volume size,
+ so make sure to check your storage provider for details. Furthermore, you should
+ be careful with burst mode guarantees as ArangoDB requires a sustainable
+ high IOPS rate.
+
+- **DB-Server storage limit**: Keep individual DB-Server storage below 2TB per server to maintain optimal performance.
+
+- **I/O bandwidth**: Give considerations to I/O bandwidth, especially considering 
+ RocksDB write-amplification which can easily be 10x or more.
+
+- **Block storage**: Whenever possible use block storage. Database data is based on append
+ operations, so filesystems which support this should be used for best
+ performance. ArangoDB does not recommend using NFS for performance reasons,
  furthermore we experienced some issues with hard links required for
  Hot Backup.
 
-- Verify your **Backup** and restore procedures are working.
+### Backup and Recovery
+
+- **Test restore procedures**: Verify your backup and restore procedures are working.
+ **TEST YOUR RESTORE PROCEDURE** regularly to ensure you can recover from failures.
+
+- **Hot Backup frequency**: Take Hot Backups with a frequency that matches your
+ RTO (Recovery Time Objective) and RPO (Recovery Point Objective) requirements.
+
+- **arangodump backups**: Take backups with arangodump from time to time as an
+ additional backup strategy alongside Hot Backups.
 
-- Consider enabling [Encryption at Rest](../operations/security/encryption-at-rest.md).
- Make sure to safely store any secret keys you create for this.
+- **Secure backup storage**: Store backups in a secure, separate location from your
+ production systems. Use encrypted storage and ensure backups are geographically
+ distributed to protect against regional disasters. Implement proper access controls
+ for backup storage locations.
 
-- Monitor the ArangoDB provided metrics (e.g. by using Prometheus/Grafana).
+- **Retry mechanisms**: Implement exponential retry with jitter in your applications
+ when connecting to ArangoDB to handle temporary network issues and failovers gracefully.
 
 ## Kubernetes Operator (kube-arangodb)
 
@@ -89,4 +164,4 @@ have been performed on your production system before you go live.
 - The [**ReclaimPolicy**](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming)
  of your persistent volumes should be set to `Retain` to prevent volumes from premature deletion.
 
-- Use native networking whenever possible to reduce delays.
+- Use native networking whenever possible to reduce delays.