v1.20.0: Thancred Waters The big ticket items are as follows: - Implement a no-JS challenge method: [`metarefresh`](./admin/configuration/challenges/metarefresh.mdx) ([TecharoHQ#95](TecharoHQ#95)) - Implement request "weight", allowing administrators to customize the behaviour of Anubis based on specific criteria - Implement GeoIP and ASN based checks via [Thoth](https://anubis.techaro.lol/docs/admin/thoth) ([TecharoHQ#206](TecharoHQ#206)) - Add [custom weight thresholds](./admin/configuration/thresholds.mdx) via CEL ([TecharoHQ#688](TecharoHQ#688)) - Move Open Graph configuration [to the policy file](./admin/configuration/open-graph.mdx) - Enable support for Open Graph metadata to be returned by default instead of doing lookups against the target - Add `robots2policy` CLI utility to convert robots.txt files to Anubis challenge policies using CEL expressions ([TecharoHQ#409](TecharoHQ#409)) - Refactor challenge presentation logic to use a challenge registry - Allow challenge implementations to register HTTP routes - [Imprint/Impressum support](./admin/configuration/impressum.mdx) ([TecharoHQ#362](TecharoHQ#362)) - Fix "invalid response" after "Success!" in Chromium ([TecharoHQ#564](TecharoHQ#564)) A lot of performance improvements have been made: - Replace internal SHA256 hashing with xxhash for 4-6x performance improvement in policy evaluation and cache operations - Optimized the OGTags subsystem with reduced allocations and runtime per request by up to 66% - Replace cidranger with bart for IP range checking, improving IP matching performance by 3-20x with zero heap allocations And some cleanups/refactors were added: - Fix OpenGraph passthrough ([TecharoHQ#717](TecharoHQ#717)) - Remove the unused `/test-error` endpoint and update the testing endpoint `/make-challenge` to only be enabled in development - Add `--xff-strip-private` flag/envvar to toggle skipping X-Forwarded-For private addresses or not - Bump AI-robots.txt to version 1.37 - Make progress bar styling more compatible (UXP, etc) - Add `--strip-base-prefix` flag/envvar to strip the base prefix from request paths when forwarding to target servers - Fix an off-by-one in the default threshold config - Add functionality for HS512 JWT algorithm - Add support for dynamic cookie domains with the `--cookie-dynamic-domain`/`COOKIE_DYNAMIC_DOMAIN` flag/envvar Request weight is one of the biggest ticket features in Anubis. This enables Anubis to be much closer to a Web Application Firewall and when combined with custom thresholds allows administrators to have Anubis take advanced reactions. For more information about request weight, see [the request weight section](./admin/policies.mdx#request-weight) of the policy file documentation. TL;DR when you have one or more WEIGHT rules like this: ```yaml bots: - name: gitea-session-token action: WEIGH expression: all: - '"Cookie" in headers' - headers["Cookie"].contains("i_love_gitea=") # Remove 5 weight points weight: adjust: -5 ``` You can configure custom thresholds like this: ```yaml thresholds: - name: minimal-suspicion # This client is likely fine, its soul is lighter than a feather expression: weight < 0 # a feather weighs zero units action: ALLOW # Allow the traffic through # For clients that had some weight reduced through custom rules, give them a # lightweight challenge. - name: mild-suspicion expression: all: - weight >= 0 - weight < 10 action: CHALLENGE challenge: # https://anubis.techaro.lol/docs/admin/configuration/challenges/metarefresh algorithm: metarefresh difficulty: 1 report_as: 1 # For clients that are browser-like but have either gained points from custom # rules or report as a standard browser. - name: moderate-suspicion expression: all: - weight >= 10 - weight < 20 action: CHALLENGE challenge: # https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work algorithm: fast difficulty: 2 # two leading zeros, very fast for most clients report_as: 2 # For clients that are browser like and have gained many points from custom # rules - name: extreme-suspicion expression: weight >= 20 action: CHALLENGE challenge: # https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work algorithm: fast difficulty: 4 report_as: 4 ``` These thresholds apply when no other `ALLOW`, `DENY`, or `CHALLENGE` rule matches the request. `WEIGHT` rules add and remove request weight as needed: ```yaml bots: - name: gitea-session-token action: WEIGH expression: all: - '"Cookie" in headers' - headers["Cookie"].contains("i_love_gitea=") # Remove 5 weight points weight: adjust: -5 - name: bot-like-user-agent action: WEIGH expression: '"Bot" in userAgent' # Add 5 weight points weight: adjust: 5 ``` Of note: the default "generic browser" rule assigns 10 weight points: ```yaml # Generic catchall rule - name: generic-browser user_agent_regex: >- Mozilla|Opera action: WEIGH weight: adjust: 10 ``` Adjust this as you see fit.

v1.20.0-pre2: Thancred Waters prerelease 2 * Makefile: Build robots2policy by @heftig in TecharoHQ#699 * fix(default-config): off-by-one error in the default thresholds by @Xe in TecharoHQ#701 * feat: implement imprint/impressum support by @Xe in TecharoHQ#706 * fix(web/js): broken progress bar with slow algo by @yut23 in TecharoHQ#673 * build(deps): bump the github-actions group with 3 updates by @dependabot in TecharoHQ#708 * fix(lib): fix invalid response after success in Chrome by @Xe in TecharoHQ#711

v1.20.0-pre1: Thancred Waters Anubis gains its biggest feature in this version: the weight system. The big ticket items are as follows: - Implement a no-JS challenge method: [`metarefresh`](./admin/configuration/challenges/metarefresh.mdx) ([TecharoHQ#95](TecharoHQ#95)) - Implement request "weight", allowing administrators to customize the behaviour of Anubis based on specific criteria - Implement GeoIP and ASN based checks via [Thoth](https://anubis.techaro.lol/docs/admin/thoth) ([TecharoHQ#206](TecharoHQ#206)) - Add [custom weight thresholds](./admin/configuration/thresholds.mdx) via CEL ([TecharoHQ#688](TecharoHQ#688)) - Move Open Graph configuration [to the policy file](./admin/configuration/open-graph.mdx) - Enable support for Open Graph metadata to be returned by default instead of doing lookups against the target - Add `robots2policy` CLI utility to convert robots.txt files to Anubis challenge policies using CEL expressions ([TecharoHQ#409](TecharoHQ#409)) - Refactor challenge presentation logic to use a challenge registry - Allow challenge implementations to register HTTP routes A lot of performance improvements have been made: - Replace internal SHA256 hashing with xxhash for 4-6x performance improvement in policy evaluation and cache operations - Optimized the OGTags subsystem with reduced allocations and runtime per request by up to 66% - Replace cidranger with bart for IP range checking, improving IP matching performance by 3-20x with zero heap allocations And some cleanups/refactors were added: - Remove the unused `/test-error` endpoint and update the testing endpoint `/make-challenge` to only be enabled in development - Add `--xff-strip-private` flag/envvar to toggle skipping X-Forwarded-For private addresses or not - Bump AI-robots.txt to version 1.37 - Make progress bar styling more compatible (UXP, etc) - Add `--strip-base-prefix` flag/envvar to strip the base prefix from request paths when forwarding to target servers Request weight is one of the biggest ticket features in Anubis. This enables Anubis to be much closer to a Web Application Firewall and when combined with custom thresholds allows administrators to have Anubis take advanced reactions. For more information about request weight, see [the request weight section](./admin/policies.mdx#request-weight) of the policy file documentation. TL;DR when you have one or more WEIGHT rules like this: ```yaml bots: - name: gitea-session-token action: WEIGH expression: all: - '"Cookie" in headers' - headers["Cookie"].contains("i_love_gitea=") # Remove 5 weight points weight: adjust: -5 ``` You can configure custom thresholds like this: ```yaml thresholds: - name: minimal-suspicion # This client is likely fine, its soul is lighter than a feather expression: weight < 0 # a feather weighs zero units action: ALLOW # Allow the traffic through # For clients that had some weight reduced through custom rules, give them a # lightweight challenge. - name: mild-suspicion expression: all: - weight >= 0 - weight < 10 action: CHALLENGE challenge: # https://anubis.techaro.lol/docs/admin/configuration/challenges/metarefresh algorithm: metarefresh difficulty: 1 report_as: 1 # For clients that are browser-like but have either gained points from custom # rules or report as a standard browser. - name: moderate-suspicion expression: all: - weight >= 10 - weight < 20 action: CHALLENGE challenge: # https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work algorithm: fast difficulty: 2 # two leading zeros, very fast for most clients report_as: 2 # For clients that are browser like and have gained many points from custom # rules - name: extreme-suspicion expression: weight >= 20 action: CHALLENGE challenge: # https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work algorithm: fast difficulty: 4 report_as: 4 ``` These thresholds apply when no other `ALLOW`, `DENY`, or `CHALLENGE` rule matches the request. `WEIGHT` rules add and remove request weight as needed: ```yaml bots: - name: gitea-session-token action: WEIGH expression: all: - '"Cookie" in headers' - headers["Cookie"].contains("i_love_gitea=") # Remove 5 weight points weight: adjust: -5 - name: bot-like-user-agent action: WEIGH expression: '"Bot" in userAgent' # Add 5 weight points weight: adjust: 5 ``` Of note: the default "generic browser" rule assigns 10 weight points: ```yaml - name: generic-browser user_agent_regex: >- Mozilla|Opera action: WEIGH weight: adjust: 10 ``` Adjust this as you see fit.

v1.19.1: Jenomis cen Lexentale - Echo 1 Return `data/bots/ai-robots-txt.yaml` to avoid breaking configs [TecharoHQ#599](TecharoHQ#599)

v1.19.0: Jenomis cen Lexentale Mostly a bunch of small features, no big ticket things this time. - Record if challenges were issued via the API or via embedded JSON in the challenge page HTML ([TecharoHQ#531](TecharoHQ#531)) - Ensure that clients that are shown a challenge support storing cookies - Imprint the version number into challenge pages - Encode challenge pages with gzip level 1 - Add PowerPC 64 bit little-endian builds (`GOARCH=ppc64le`) - Add `check-spelling` for spell checking - Add `--target-insecure-skip-verify` flag/envvar to allow Anubis to hit a self-signed HTTPS backend - Minor adjustments to FreeBSD rc.d script to allow for more flexible configuration. - Added Podman and Docker support for running Playwright tests - Add a default rule to throw challenges when a request with the `X-Firefox-Ai` header is set. - Updated the nonce value in the challenge JWT cookie to be a string instead of a number - Rename cookies in response to user feedback - Ensure cookie renaming is consistent across configuration options - Add Bookstack app in data - Truncate everything but the first five characters of Accept-Language headers when making challenges - Ensure client JavaScript is served with Content-Type text/javascript. - Add `--target-host` flag/envvar to allow changing the value of the Host header in requests forwarded to the target service. - Bump AI-robots.txt to version 1.31 - Add `RuntimeDirectory` to systemd unit settings so native packages can listen over unix sockets - Added SearXNG instance tracker whitelist policy - Added Qualys SSL Labs whitelist policy - Fixed cookie deletion logic ([TecharoHQ#520](TecharoHQ#520), [TecharoHQ#522](TecharoHQ#522)) - Add `--target-sni` flag/envvar to allow changing the value of the TLS handshake hostname in requests forwarded to the target service. - Fixed CEL expression matching validator to now properly error out when it receives empty expressions - Added OpenRC init.d script. - Added `--version` flag. - Added `anubis_proxied_requests_total` metric to count proxied requests. - Add `Applebot` as "good" web crawler - Reorganize AI/LLM crawler blocking into three separate stances, maintaining existing status quo as default. - Split out AI/LLM user agent blocking policies, adding documentation for each.

v1.19.0-pre1: Jenomis cen Lexentale Minor upgrades, refactoring, test fixes, etc.

v1.18.0: Varis zos Galvus The big ticket feature in this release is [CEL expression matching support](https://anubis.techaro.lol/docs/admin/configuration/expressions). This allows you to tailor your approach for the individual services you are protecting. These can be as simple as: ```yaml - name: allow-api-requests action: ALLOW expression: all: - '"Accept" in headers' - 'headers["Accept"] == "application/json"' - 'path.startsWith("/api/")' ``` Or as complicated as: ```yaml - name: allow-git-clients action: ALLOW expression: all: - >- ( userAgent.startsWith("git/") || userAgent.contains("libgit") || userAgent.startsWith("go-git") || userAgent.startsWith("JGit/") || userAgent.startsWith("JGit-") ) - '"Git-Protocol" in headers' - headers["Git-Protocol"] == "version=2" ``` The docs have more information, but here's a tl;dr of the variables you have access to in expressions: | Name | Type | Explanation | Example | | :-------------- | :-------------------- | :---------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------- | | `headers` | `map[string, string]` | The [headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers) of the request being processed. | `{"User-Agent": "Mozilla/5.0 Gecko/20100101 Firefox/137.0"}` | | `host` | `string` | The [HTTP hostname](https://web.dev/articles/url-parts#host) the request is targeted to. | `anubis.techaro.lol` | | `method` | `string` | The [HTTP method](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Methods) in the request being processed. | `GET`, `POST`, `DELETE`, etc. | | `path` | `string` | The [path](https://web.dev/articles/url-parts#pathname) of the request being processed. | `/`, `/api/memes/create` | | `query` | `map[string, string]` | The [query parameters](https://web.dev/articles/url-parts#query) of the request being processed. | `?foo=bar` -> `{"foo": "bar"}` | | `remoteAddress` | `string` | The IP address of the client. | `1.1.1.1` | | `userAgent` | `string` | The [`User-Agent`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent) string in the request being processed. | `Mozilla/5.0 Gecko/20100101 Firefox/137.0` | This will be made more elaborate in the future. Give me time. This is a [simple, lovable, and complete](https://longform.asmartbear.com/slc/) implementation of this feature so that administrators can get hacking ASAP. Other changes: - Use CSS variables to deduplicate styles - Fixed native packages not containing the stdlib and botPolicies.yaml - Change import syntax to allow multi-level imports - Changed the startup logging to use JSON formatting as all the other logs do. - Added the ability to do [expression matching with CEL](./admin/configuration/expressions.mdx) - Add a warning for clients that don't store cookies - Disable Open Graph passthrough by default ([TecharoHQ#435](TecharoHQ#435)) - Clarify the license of the mascot images ([TecharoHQ#442](TecharoHQ#442)) - Started Suppressing 'Context canceled' errors from http in the logs ([TecharoHQ#446](TecharoHQ#446))

Prelease: v1.18.0-pre1: Varis zos Galvus - Use CSS variables to deduplicate styles - Fixed native packages not containing the stdlib and botPolicies.yaml - Change import syntax to allow multi-level imports - Changed the startup logging to use JSON formatting as all the other logs do. - Added the ability to do [expression matching with CEL](./admin/configuration/expressions.mdx) - Add a warning for clients that don't store cookies - Disable Open Graph passthrough by default ([TecharoHQ#435](TecharoHQ#435)) - Clarify the license of the mascot images ([TecharoHQ#442](TecharoHQ#442))

Version 1.17.1: Asahi sas Brutus: Echo 1 - Added customization of authorization cookie expiration time with `--cookie-expiration-time` flag or envvar - Updated the `OG_PASSTHROUGH` to be true by default, thereby allowing OpenGraph tags to be passed through by default - Added the ability to [customize Anubis' HTTP status codes](./admin/configuration/custom-status-codes.mdx) ([TecharoHQ#355](TecharoHQ#355))

v1.17.0: Asahi sas Brutus Signed-off-by: Xe Iaso <me@xeiaso.net>