ArangoDB3.7.3

Added the following metrics for synchronous replication in the cluster:

• `arangodb_refused_followers_count`: Number of times a shard leader received a refusal answer from a follower during synchronous replication.
• `arangodb_sync_wrong_checksum`: Number of times a mismatching shard checksum was detected when syncing shards. In case this happens, a resync will be triggered for the shard.
• Fixed handling of failedLeaderJob. In case of a plan modification, that removes a server from the plan, e.g. reduce replication factor. Directly followed by a failure of the current shard leader, would reinsert the just removed server in the plan, which is undesired, we first need to have a full "desync" cycle on this server to be reusable in the plan again.
• Make sure the optimizer doesn't pick another index than the TTL index itself while fulfilling the expiry of TTL.
• Added optional verbose logging for agency write operations. This logging is configurable by using the new log topic "agencystore".

The following log levels can be used for for the "agencystore" log topic to log writes to the agency:

• DEBUG: will log all writes on the leader
• TRACE: will log all writes on both leaders and followers
• The default log level for the "agencystore" log topic is WARN, meaning no agency writes will be logged.
• Turning on this logging can be used for auditing and debugging, but it is not recommended in the general case, as it can lead to large amounts of data being logged, which can have a performance impact and will lead to higher disk space usage.
• Print image base address and CPU context (if available) in crash handler messages.
• Added configuration option `--query.tracking-slow-queries` to decide whether slow queries are tracked extra.
• Added configuration option `--query.tracking-with-querystring` to decide whether the query string is shown in the slow query log and the list of currently running queries. The option is true by default.
• When turned off, querystrings in the slow query log and the list of currently running queries are just shown as "".
• Added configuration option `--query.tracking-with-datasources` to toggle whether the names of data sources used by queries are shown in the slow query log and the list of currently running queries. The option is false by default.
• When turned on, the names of data sources used by the query will be shown in the slow query log and the list of currently running queries.
• Fixed handling of failoverCandidates. Sometimes, a server can still be a failoverCandidate even though it has been taken out of the Plan. With this fix, such a server is quickly taken out of failoverCandidates and it can never be re-added to the Plan before this has happened.
• Fix #12693: SORT inside a subquery could sometimes swallow part of its input when it crossed boundaries of internal row batches.
• Fixed issue BTS-212: Web UI doesn't let to make partial view update and partial view update should be audited (also reported as ES-700).
• Fixed link definition comparison logic: equality wasn`t properly detected and led to link recreation.
• Added configuration option `--rocksdb.sync-delay-threshold`.
• This option can be used to track if any RocksDB WAL sync operations is delayed by more than the configured value (in milliseconds). The intention is to get aware of severely delayed WAL sync operations.
• Add database, shard name and error information to several shard-related log messages.
• Display shard names of a collection in the web interface when in the details view of the collection.

Added HTTP requests metrics for tracking the number of superuser and normal user requests separately:

• `arangodb_http_request_statistics_superuser_requests`: Total number of HTTP requests executed by superuser/JWT
• `arangodb_http_request_statistics_user_requests`: Total number of HTTP requests executed by clients
• Fixed a bug in handling of followers which refuse to replicate operations.
• In the case that the follower has simply been dropped in the meantime, we now avoid an error reported by the shard leader.
• Fix a performance regression when a LIMIT is combined with a COLLECT WITH COUNT INTO. Reported in ES-692.
• Fix REST handler GET /_admin/status when called with URL parameter value `overview=true`. For generating the `hash` attribute in the response, the current Plan was retrieved and analyzed. Due to a change in the internal Plan format the REST handler code failed to pick up the number of servers, which resulted in the REST handler returning HTTP 500 in cluster mode.
• Use rclone built from v1.51.0 source with go1.15.2 instead of prebuilt v1.51.0 release.
• Fixed a bug in AQL COLLECT with OPTIONS { "hash" } that lead to a quadratic runtime in the number of output rows.
• Added startup option `--database.old-system-collections` to toggle automatic creation of system collections `_modules` and `_fishbowl`, along with their internal usage. These collections are useful only in very few cases, so it is normally not worth to create them in all databases.
• The `_modules` collection is only used to register custom JavaScript modules, for which there exists no API, and `_fishbowl` is used to store the temporary list of Foxx apps retrieved from the Github Foxx store.
• If the option value is `false` (which is the default from v3.8 onwards, but for v3.7 the default value is `true` for downwards-compatibility), the two collections will not be created for any new database. The `_fishbowl`collection will still be created dynamically when needed. If the option value is `true` (the default value in v3.7), the collections will be created regularly as before.
• The default value for the option is going to change to `false` in v3.8, meaning the collections will not be created anymore there by default.
• Any functionality related to the `_modules` system collection is deprecate and will be removed in ArangoDB v3.9.

Two side effects of turning this option off are:

• there will no be iteration over all databases at server startup just to check the contents of all `_modules` collections.
• less collections/shards will be around for deployments that create a large number of databases.
• Already existing `_modules` and `_fishbowl` system collections will not be modified by this PR, even though they will likely be empty and unused.
• Don't iterate over all databases as server startup in order to initialize the routing information. This is not necessary, as the routing information is global and not tied to a specific database.
• Fixed a possible crash during instantiation of an AQL graph traversal. Reported in #12597.
• Added safeguards against using V8 internally in environments that have JavaScript turned off via the `--javascript.enabled false` option.
• Make scheduler propertly count down the number of working threads in case an exception happens in a worker thread.
• Turn off upgrade checks in arangod in alpha/beta/preview Enterprise builds, too.
• Previously it was already turned off in arangod for Enterprise builds already, but only for stable releases and not preview releases.
• Fixed and extended LDAP log messages.
• Added LDAP_OFF if referrals and restart are false.
• If LDAP search fails, also retry (update to given number of retries).
• Fixed inifinite reload of the login window after logout of an LDAP user.
• Make the reboot tracker catch failed coordinators, too. Previously the reboot tracker was invoked only when a DB server failed or was restarted, and when a coordinator was restarted. Now it will also act if a coordinator just fails (without restart).

Added scheduler thread creation/destruction metrics:

• `arangodb_scheduler_threads_started`: Number of scheduler threads started
• `arangodb_scheduler_threads_stopped`: Number of scheduler threads stopped
• Added startup option `--query.max-runtime` to limit the maximum runtime of all
• AQL queries to a specified threshold value (in seconds). By default, the threshold is 0, meaning that the runtime of AQL queries is not limited.
• Setting it to any positive value will restrict the runtime of all AQL queries unless it is overwritten in the per-query "maxRuntime" query option.
• Please note that setting this option will affect *all* queries in all databases, and also queries issues for administration and database-internal purposes.
• If a query exceeds the configured runtime, it will be killed on the next occasion when the query checks its own status. Killing is best effort, so it is not guaranteed that a query will no longer than exactly the configured amount of time.
• Ensure that the argument to an AQL OPTIONS clause is always an object which
• does not contain any dynamic (run-time) values. Previously, this was only enforced for traversal options and options for data-modification queries. This change extends the check to all occurrences of OPTIONS.

Added `details` option to figures command of a collection:

• `collection.figures(details)`
• Setting `details` to `true` will return extended storage engine-specific details to the figures. The details are intended for debugging ArangoDB itself and their format is subject to change. There is not much use in using the details from a client application.
• By default, `details` is set to `false`, so no details are returned and the behavior is identical to previous versions of ArangoDB.
• Implement RebootTracker usage for AQL queries in case of coordinator restarts or failures. This will clean up the rest of an AQL query on dbservers more quickly and in particular release locks faster.
• Serialize maintenance actions for each shard. This addresses lost document problems found in chaos testing.
• Enforce a maximum result register usage limit in AQL queries. In an AQL query, every user-defined or internal (unnamed) variable will need a register to store results in.
• AQL queries that use more result registers than allowed (currently 1000) will now abort deterministically during the planning stage with error 32 (`resource limit exceeded`) and the error message "too many registers (1000) needed for AQL query".
• Before this fix, an AQL query that used more than 1000 result registers crashed the server when assertions were turned on, and the behavior was undefined when assertions were turned off.
• Fixed some cases where subqueries in PRUNE did not result in a parse error, but either in an incomprehensible error (in 3.7), or undefined behaviour during execution (pre 3.7).
• Fixed an issue with audit logging misreporting some document requests as internal instead of logging the proper request information.
• Add attributes `database` and `user` when tracking current and slow AQL queries.
• `database` contains the name of the database the query is/was running in, `user` contains the name of the user that started the query.

These attributes will be returned in addition when calling the APIs for current and slow query inspection:

• GET `/_api/query/current` and `require("arangodb/aql/queries").current()`
• GET `/_api/query/slow` and `require("arangodb/aql/queries").slow()`
• The "slow query" log message has also been augmented to contain the database name and the user name.
• The `user` attribute is now also displayed in the web interface in the "Running queries" and "Slow queries" views.

Added metrics for V8 contexts usage:

• `arangodb_v8_context_alive`: number of V8 contexts currently alive.
• `arangodb_v8_context_busy`: number of V8 contexts currently busy.
• `arangodb_v8_context_dirty`: number of V8 contexts currently dirty.
• `arangodb_v8_context_free`: number of V8 contexts currently free.
• `arangodb_v8_context_max`: maximum number of concurrent V8 contexts.
• `arangodb_v8_context_min`: minimum number of concurrent V8 contexts.
• Updated arangosync to 0.7.11.
• Make followers in active failover run a compaction after they process a truncate operation and the truncate removed more than 4k documents. This can help to reclaim disk space on the follower earlier than without running the truncate.
• The REST API PUT `/_api/collection//truncate` will now also run a compaction if the truncation affected more than 4k documents. This may add extra latency to the truncate operation, but can help to reclaim disk space earlier.
• Added REST API PUT `/_admin/compact` for compacting the entire database data.
• This endpoint can be used to reclaim disk space after substantial data deletions have taken place. The command is also exposed via the JavaScript API as `db._compact();`.
• This command can cause a full rewrite of all data in all databases, which may take very long for large databases. It should thus only be used with care and only when additional I/O load can be tolerated for a prolonged time.
• This command requires superuser access and is only available for the RocksDB storage engine.
• Don't allow creation of smart satellite graphs or collections (i.e. using `"isSmart":true` together with `"replicationFactor":"satellite"` when creating graphs or collections. This combination of parameters makes no sense, so that the server will now respond with "bad parameter" and an HTTP status code of HTTP 400 ("Bad request").
• Add exit code for ICU database loading startup errors.
• Fixed issue #12507: SegFault when using an AQL for loop through edges.
• Make the `IS_IPV4` AQL function behave identical on macOS as on other platforms. It previously allowed leading zeros in octets on macOS, whereas on other platforms they were disallowed. Now this is disallowed on macOS as well.
• Added new metric "arangodb_aql_slow_query" for slow AQL queries, so this can be monitored more easily.
• Added new metric "arangodb_scheduler_queue_length" for the scheduler's internal queue length.
• Added new metric "arangodb_scheduler_queue_full_failures" for tracking cases of a full scheduler queue and dropping requests.
• Added new metrics for the number of V8 contexts dynamically created and destroyed ("arangodb_v8_context_created" and "arangodb_v8_context_destroyed")
• and for the number of times a V8 context was entered and left ("arangodb_v8_context_entered" and "arangodb_v8_context_exited"). There is
• also a new metric for tracking the cases when a V8 context cannot be successfully acquired and an operation is not performed ("arangodb_v8_context_enter_failures").
• Added extra info to "queue full" and "giving up waiting for unused v8 context" log messages.
• Request to the `/_admin/statistics` API now processed via the CLIENT_FAST lane.
• Previously they were handled in the CLIENT_SLOW lane, meaning that monitoring requests using that API didn't get through when the queue was rather full.
• Introduce an internal high-water mark for the maximum row number that was written to in an AqlItemBlock. Using this number several operations on the whole block, such as cleaning up or copying can be made more efficient when run on only partially filled blocks.
• Fixed issue BTS-169: cost estimation for LIMIT nodes showed wrong number of estimated items.