Fix loophole for creating duplicate keys (#1087)

* documentation PR for arangodb/arangodb#16764 * applied to 3.11 * Update release-notes-known-issues38.md * Update release-notes-known-issues39.md Co-authored-by: ansoboleva <93702078+ansoboleva@users.noreply.github.com>

Author: jsteemann

3.10/administration-cluster.md

@@ -118,9 +118,12 @@ UPDATE "123" WITH { … } IN sharded_collection
 ```
 
 If custom shard keys are used, you can no longer specify the primary key value
-for a new document, but must let the server generated one automatically. This
+for a new document, but must let the server generate one automatically. This
 restriction comes from the fact that ensuring uniqueness of the primary key
 would be very inefficient if the user could specify the document key.
+If custom shard keys are used, trying to store documents with the primary key value
+(`_key` attribute) set will result in a runtime error ("must not specify _key
+for this collection").
 
 Unique indexes on sharded collections are only allowed if the fields used to 
 determine the shard key are also included in the list of attribute paths for the index:

3.11/administration-cluster.md

@@ -118,9 +118,12 @@ UPDATE "123" WITH { … } IN sharded_collection
 ```
 
 If custom shard keys are used, you can no longer specify the primary key value
-for a new document, but must let the server generated one automatically. This
+for a new document, but must let the server generate one automatically. This
 restriction comes from the fact that ensuring uniqueness of the primary key
 would be very inefficient if the user could specify the document key.
+If custom shard keys are used, trying to store documents with the primary key value
+(`_key` attribute) set will result in a runtime error ("must not specify _key
+for this collection").
 
 Unique indexes on sharded collections are only allowed if the fields used to 
 determine the shard key are also included in the list of attribute paths for the index:

3.8/administration-cluster.md

@@ -118,9 +118,12 @@ UPDATE "123" WITH { … } IN sharded_collection
 ```
 
 If custom shard keys are used, you can no longer specify the primary key value
-for a new document, but must let the server generated one automatically. This
+for a new document, but must let the server generate one automatically. This
 restriction comes from the fact that ensuring uniqueness of the primary key
 would be very inefficient if the user could specify the document key.
+If custom shard keys are used, trying to store documents with the primary key value
+(`_key` attribute) set will result in a runtime error ("must not specify _key
+for this collection").
 
 Unique indexes (hash, skiplist, persistent) on sharded collections are
 only allowed if the fields used to determine the shard key are also

3.8/release-notes-known-issues38.md

@@ -32,6 +32,7 @@ AQL
 | **Date Added:** 2018-09-05 <br> **Component:** AQL <br> **Deployment Mode:** Cluster <br> **Description:** In a very uncommon edge case there is an issue with an optimization rule in the cluster. If you are running a cluster and use a custom shard key on a collection (default is `_key`) **and** you provide a wrong shard key in a modifying query (`UPDATE`, `REPLACE`, `DELETE`) **and** the wrong shard key is on a different shard than the correct one, a `DOCUMENT NOT FOUND` error is returned instead of a modification (example query: `UPDATE { _key: "123", shardKey: "wrongKey"} WITH { foo: "bar" } IN mycollection`). Note that the modification always happens if the rule is switched off, so the suggested workaround is to [deactivate the optimizing rule](aql/execution-and-performance-optimizer.html#turning-specific-optimizer-rules-off) `restrict-to-single-shard`. <br> **Affected Versions:** 3.4.x, 3.5.x, 3.6.x, 3.7.x, 3.8.x <br> **Fixed in Versions:** - <br> **Reference:** [arangodb/arangodb#6399](https://github.com/arangodb/arangodb/issues/6399){:target="_blank"} |
 | **Date Added:** 2021-04-19 <br> **Component:** AQL <br> **Deployment Mode:** All <br> **Description:** User-defined variable names in AQL cannot start with an underscore, in contrast to what the AQL documentation claimed. <br> **Affected Versions:** 3.5.x, 3.6.x, 3.7.x, 3.8.x <br> **Fixed in Versions:** 3.9.0 <br> **Reference:** [arangodb/arangodb#13513](https://github.com/arangodb/arangodb/issues/13513){:target="_blank"} |
 | **Date Added:** 2021-05-25 <br> **Component:** AQL <br> **Deployment Mode:** All <br> **Description:** Global and per-query memory limits are not enforced for `SHORTEST_PATH` and `K_SHORTEST_PATHS` AQL graph traversals. Such queries may thus cause out-of-memory issues as they will not be killed despite exceeding the configured `--query.memory-limit` or Cursor API `memoryLimit`. <br> **Affected Versions:** 3.5.x, 3.6.x, 3.7.x, 3.8.x <br> **Fixed in Versions:** 3.8.1 <br> **Reference:** [BTS-411](https://arangodb.atlassian.net/browse/BTS-411){:target="_blank"} (internal) |
+| **Date Added:** 2022-08-12 <br> **Component:** AQL <br> **Deployment Mode:** Cluster <br> **Description:** Since ArangoDB 3.8, there was a loophole for creating duplicate keys in the same collection. This was only possible in cluster and needed an AQL query that copied over data from a collection (source) into another (target). The target collection must have more than one shard and must use a custom shard key. Inserting documents into the target collection must have happened via an AQL query like `FOR doc IN source INSERT doc INTO target`. In this particular situation, the document keys (`_key` attribute) from the source collection were used as-is for insertion into the target collection. However, as the target collection is not sharded by `_key` and uses a custom shard key, it is actually not allowed to specify user-defined values for `_key`. Versions before 3.8 failed with the error "must not specifiy _key for this collection" when running such an AQL query, but versions between 3.8.0 and 3.8.7 did not. <br> **Affected Versions:** 3.8.0 - 3.8.7 <br> **Fixed in Versions:** 3.8.8 <br> **Reference:** [arangodb/arangodb#16764](https://github.com/arangodb/arangodb/issues/16764){:target="_blank"} |
 
 Upgrading
 ---------

3.9/administration-cluster.md

@@ -118,9 +118,12 @@ UPDATE "123" WITH { … } IN sharded_collection
 ```
 
 If custom shard keys are used, you can no longer specify the primary key value
-for a new document, but must let the server generated one automatically. This
+for a new document, but must let the server generate one automatically. This
 restriction comes from the fact that ensuring uniqueness of the primary key
 would be very inefficient if the user could specify the document key.
+If custom shard keys are used, trying to store documents with the primary key value
+(`_key` attribute) set will result in a runtime error ("must not specify _key
+for this collection").
 
 Unique indexes on sharded collections are only allowed if the fields used to 
 determine the shard key are also included in the list of attribute paths for the index:

3.9/release-notes-known-issues39.md

@@ -29,6 +29,7 @@ AQL
 | Issue |
 |------------|
 | **Date Added:** 2018-09-05 <br> **Component:** AQL <br> **Deployment Mode:** Cluster <br> **Description:** In a very uncommon edge case there is an issue with an optimization rule in the cluster. If you are running a cluster and use a custom shard key on a collection (default is `_key`) **and** you provide a wrong shard key in a modifying query (`UPDATE`, `REPLACE`, `DELETE`) **and** the wrong shard key is on a different shard than the correct one, a `DOCUMENT NOT FOUND` error is returned instead of a modification (example query: `UPDATE { _key: "123", shardKey: "wrongKey"} WITH { foo: "bar" } IN mycollection`). Note that the modification always happens if the rule is switched off, so the suggested workaround is to [deactivate the optimizing rule](aql/execution-and-performance-optimizer.html#turning-specific-optimizer-rules-off) `restrict-to-single-shard`. <br> **Affected Versions:** 3.4.x, 3.5.x, 3.6.x, 3.7.x, 3.8.x, 3.9.x <br> **Fixed in Versions:** - <br> **Reference:** [arangodb/arangodb#6399](https://github.com/arangodb/arangodb/issues/6399){:target="_blank"} |
+| **Date Added:** 2022-08-12 <br> **Component:** AQL <br> **Deployment Mode:** Cluster <br> **Description:** Since ArangoDB 3.8 there was a loophole for creating duplicate keys in the same collection. This was only possible in cluster and needed an AQL query that copied over data from a collection (source) into another (target). The target collection must have more than one shard and must use a custom shard key. Inserting documents into the target collection must have happened via an AQL query like `FOR doc IN source INSERT doc INTO target`. In this particular situation, the document keys (`_key` attribute) from the source collection were used as-is for insertion into the target collection. However, as the target collection is not sharded by `_key` and uses a custom shard key, it is actually not allowed to specify user-defined values for `_key`. Versions before 3.8 failed with the error "must not specifiy _key for this collection" when running such AQL query, but versions between 3.8.0 and 3.9.2 did not. <br> **Affected Versions:** 3.8.0 - 3.8.7, 3.9.0 - 3.9.2 <br> **Fixed in Versions:** 3.8.8, 3.9.3 <br> **Reference:** [arangodb/arangodb#16764](https://github.com/arangodb/arangodb/issues/16764){:target="_blank"} |
 
 Upgrading
 ---------