|
| 1 | +--- |
| 2 | +# We use sentence case and present imperative tone |
| 3 | +title: "Bot signatures" |
| 4 | +# Weights are assigned in increments of 100: determines sorting order |
| 5 | +weight: 550 |
| 6 | +# Creates a table of contents and sidebar, useful for large documents |
| 7 | +toc: true |
| 8 | +# Types have a 1:1 relationship with Hugo archetypes, so you shouldn't need to change this |
| 9 | +nd-content-type: reference |
| 10 | +# Intended for internal catalogue and search, case sensitive: |
| 11 | +# Agent, N4Azure, NIC, NIM, NGF, NAP-DOS, NAP-WAF, NGINX One, NGINX+, Solutions, Unit |
| 12 | +nd-product: NAP-WAF |
| 13 | +--- |
| 14 | + |
| 15 | +Bot signatures are a feature that protects applications by detecting signatures and clients that falsely claim to be browsers or search engines. |
| 16 | + |
| 17 | +This feature is enabled by default with the `bot-defense` parameter, and includes both bot signatures and header anomalies, which can be disabled separately. |
| 18 | + |
| 19 | +## Bot signatures |
| 20 | + |
| 21 | +Bot signature detection works by inspecting the the User-Agent header and URI of a request. |
| 22 | + |
| 23 | +Each detected bot signature belongs to a bot class: search engine signatures such as `googlebot` are under the trusted_bots class, but F5 WAF for NGINX performs additional checks to authenticate a trusted bot. |
| 24 | + |
| 25 | +If these checks fail, it means the detected bot signature impersonated a search engine, and it will be given the class `malicous_bot`, anomaly `Search engine verification failed`. |
| 26 | + |
| 27 | +The request will be blocked, regardless of the class's mitigation actions configuration. |
| 28 | + |
| 29 | +This is a list of trusted bots, all of which are search engines. |
| 30 | + |
| 31 | +| Name | Description | |
| 32 | +| ------------------ | ----------- | |
| 33 | +| Ask | [Ask.com engine](https://www.ask.com) | |
| 34 | +| Baidu | [Baidu search engine](https://www.baidu.com/) | |
| 35 | +| Baidu Image Spider | [Baidu search engine for images](https://image.baidu.com/) | |
| 36 | +| Bing | [Microsoft Bing search engine](https://www.bing.com/) | |
| 37 | +| BingPreview | [Microsoft Bing page snapshot generation engine](https://www.bing.com/) | |
| 38 | +| Daum | [Daum search engine](https://www.daum.net/) | |
| 39 | +| DuckDuckGo Bot | [DuckDuckGo search engine](https://duckduckgo.com/) | |
| 40 | +| fastbot | [fastbot search engine](https://www.fastbot.de/) | |
| 41 | +| Google | [Google search engine](https://www.google.com/) | |
| 42 | +| MojeekBot | [Mojeek search engine](https://www.mojeek.com/) | |
| 43 | +| Yahoo! Slurp | [Yahoo search engine](https://www.yahoo.com/) | |
| 44 | +| Yandex | [Yandex search engine](https://yandex.com/) | |
| 45 | +| YioopBot | Yioop search engine | |
| 46 | + |
| 47 | +An action can be configured for each bot class, or configured for each bot signature individually: |
| 48 | + |
| 49 | +* `ignore` - Bot signature is ignored (disabled) |
| 50 | +* `detect` - Only report without raising the violation - `VIOL_BOT_CLIENT`. The request is considered `legal` unless another violation is triggered. |
| 51 | +* `alarm` - Report, raise the violation, but pass the request. The request is marked as `illegal`. |
| 52 | +* `block` - Report, raise the violation, and block the request |
| 53 | + |
| 54 | +This example enables bot signatures using the default bot configuration: |
| 55 | + |
| 56 | +```json |
| 57 | +{ |
| 58 | + "policy": { |
| 59 | + "name": "bot_defense_policy", |
| 60 | + "template": { |
| 61 | + "name": "POLICY_TEMPLATE_NGINX_BASE" |
| 62 | + }, |
| 63 | + "applicationLanguage": "utf-8", |
| 64 | + "enforcementMode": "blocking", |
| 65 | + "bot-defense": { |
| 66 | + "settings": { |
| 67 | + "isEnabled": true |
| 68 | + } |
| 69 | + } |
| 70 | + } |
| 71 | +} |
| 72 | +``` |
| 73 | + |
| 74 | +The default actions for classes are: `detect` for `trusted-bot`, `alarm` for `untrusted-bot`, and `block` for `malicious-bot`. |
| 75 | + |
| 76 | +The next example enables bot defense, configuring a violation for `trusted-bot`, and block for `untrusted-bot`. |
| 77 | + |
| 78 | +```json |
| 79 | +{ |
| 80 | + "policy": { |
| 81 | + "name": "bot_defense_policy", |
| 82 | + "template": { |
| 83 | + "name": "POLICY_TEMPLATE_NGINX_BASE" |
| 84 | + }, |
| 85 | + "applicationLanguage": "utf-8", |
| 86 | + "enforcementMode": "blocking", |
| 87 | + "bot-defense": { |
| 88 | + "settings": { |
| 89 | + "isEnabled": true |
| 90 | + }, |
| 91 | + "mitigations": { |
| 92 | + "classes": [ |
| 93 | + { |
| 94 | + "name": "trusted-bot", |
| 95 | + "action": "alarm" |
| 96 | + }, |
| 97 | + { |
| 98 | + "name": "untrusted-bot", |
| 99 | + "action": "block" |
| 100 | + }, |
| 101 | + { |
| 102 | + "name": "malicious-bot", |
| 103 | + "action": "block" |
| 104 | + } |
| 105 | + ] |
| 106 | + } |
| 107 | + } |
| 108 | + } |
| 109 | +} |
| 110 | +``` |
| 111 | + |
| 112 | +The next example overrides the action for a specific signature (python-requests): |
| 113 | + |
| 114 | +```json |
| 115 | +{ |
| 116 | + "policy": { |
| 117 | + "name": "bot_defense_policy", |
| 118 | + "template": { |
| 119 | + "name": "POLICY_TEMPLATE_NGINX_BASE" |
| 120 | + }, |
| 121 | + "applicationLanguage": "utf-8", |
| 122 | + "enforcementMode": "blocking", |
| 123 | + "bot-defense": { |
| 124 | + "settings": { |
| 125 | + "isEnabled": true |
| 126 | + }, |
| 127 | + "mitigations": { |
| 128 | + "signatures": [ |
| 129 | + { |
| 130 | + "action": "ignore", |
| 131 | + "name": "python-requests" |
| 132 | + } |
| 133 | + ] |
| 134 | + } |
| 135 | + } |
| 136 | + } |
| 137 | +} |
| 138 | +``` |
| 139 | + |
| 140 | +The bot signature file, `included_bot_signatures`, is located at the following path: `/opt/app-protect/var/update_files/bot_signatures/included_bot_signatures`. |
| 141 | + |
| 142 | +This file is an up-to-date list of all bot signatures, following a format similar to the README-style text file found for [attack signatures]({{< ref "/waf/policies/attack-signatures.md" >}}). |
| 143 | + |
| 144 | +It contains information such as: |
| 145 | + |
| 146 | +- Bot name |
| 147 | +- Bot type |
| 148 | +- Bot classification/category |
| 149 | + |
| 150 | +It is part of the _app-protect-bot-signatures_ package: for more information, see the [Update F5 WAF for NGINX signatures]({{< ref "/waf/install/update-signatures.md" >}}) topic. |
| 151 | + |
| 152 | +## Header anomalies |
| 153 | + |
| 154 | +In addition to detecting bot signatures, F5 WAF for NGINX verifies that a client is the browser it claims to be by inspecting the HTTP headers. |
| 155 | + |
| 156 | +Each request receives a score and anomaly category, and is enforced according to the default anomaly action: |
| 157 | + |
| 158 | +| Range | Anomaly | Action | Class | |
| 159 | +|--------------- | ----------------------------------------- | ------ | ------------------ | |
| 160 | +| 0-49 | None | None | Browser | |
| 161 | +| 50-99 | Suspicious HTTP Headers Presence or Order | Alarm | Suspicious Browser | |
| 162 | +| 100 and above | Invalid HTTP Headers Presence or Order | Block | Malicious Bot | |
| 163 | +| Non Applicable | SEARCH_ENGINE_VERIFICATION_FAILED | Block | Malicious Bot | |
| 164 | + |
| 165 | +The default scores for each anomaly can be changed. |
| 166 | + |
| 167 | +In this example, the score and action of the default bot configuration has been overrided: |
| 168 | + |
| 169 | +```json |
| 170 | +{ |
| 171 | + "policy": { |
| 172 | + "name": "bot_anomalies_and_signatures", |
| 173 | + "template": { |
| 174 | + "name": "POLICY_TEMPLATE_NGINX_BASE" |
| 175 | + }, |
| 176 | + "applicationLanguage": "utf-8", |
| 177 | + "enforcementMode": "blocking", |
| 178 | + "bot-defense": { |
| 179 | + "mitigations": { |
| 180 | + "anomalies": [ |
| 181 | + { |
| 182 | + "name": "Suspicious HTTP Headers", |
| 183 | + "action": "alarm", |
| 184 | + "scoreThreshold": 50 |
| 185 | + }, |
| 186 | + { |
| 187 | + "name": "Invalid HTTP Headers", |
| 188 | + "action": "block", |
| 189 | + "scoreThreshold": 99 |
| 190 | + } |
| 191 | + ] |
| 192 | + } |
| 193 | + } |
| 194 | + } |
| 195 | +} |
| 196 | + |
| 197 | +``` |
0 commit comments