Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
180e6ac
fix whitespace
Feb 2, 2021
1c4b0e3
docs: add logger
Feb 2, 2021
b4dce5e
docs: add middleware
Feb 2, 2021
01147a6
docs: parameters
Feb 2, 2021
7f93bbf
docs: batch
Feb 2, 2021
2a5f50c
docs: typing
Feb 2, 2021
0c492e1
docs: validation
Feb 2, 2021
e174d15
docs: dataclasses
Feb 2, 2021
98b3428
docs: parser
Feb 2, 2021
f9da787
docs: use mkdocs for website
Feb 2, 2021
7636f61
docs: add docs branch to have live preview for the PR
Feb 2, 2021
707b79b
docs: change color and adjust logo
Feb 2, 2021
cb9e881
docs: merged async and generator sections
Feb 2, 2021
1123a31
docs: merged cloudwatch logs to tabs
Feb 2, 2021
1cfdce2
docs: merged tabs
Feb 2, 2021
add4f4f
docs: remove highlight line markers
Feb 2, 2021
f4ea9b7
docs: add meta tags
Feb 2, 2021
2d05c72
docs: remove old docs folder
Feb 2, 2021
264a028
chore: typo in GetCfnTemplate
heitorlessa Feb 3, 2021
1fa47d9
Update docs/core/logger.md
heitorlessa Feb 3, 2021
8f418cb
Update docs/core/logger.md
heitorlessa Feb 3, 2021
7ddb544
Update docs/utilities/parameters.md
heitorlessa Feb 3, 2021
e425d6d
Update docs/utilities/validation.md
heitorlessa Feb 3, 2021
36cba90
Update docs/utilities/validation.md
heitorlessa Feb 3, 2021
4b9ca27
Update docs/utilities/batch.md
heitorlessa Feb 3, 2021
84c8e35
Update docs/core/tracer.md
heitorlessa Feb 3, 2021
f96abe7
Merge branch 'develop' of github.com:awslabs/aws-lambda-powertools-py…
Feb 3, 2021
da46dff
Merge branch 'docs/mkdocs' of github.com:am29d/aws-lambda-powertools-…
Feb 3, 2021
2556afa
remove code formatting from table headers
Feb 3, 2021
a6eb2b4
docs: increse max-width with extra css for readability
Feb 3, 2021
421145a
docs: add changes from review
Feb 4, 2021
1a483c9
docs: refactor doc targets, add docker build
Feb 4, 2021
8e548f5
docs: tabs naming for cloudwatch logs
Feb 4, 2021
32fdd23
docs: fix build-docs-website target
Feb 4, 2021
a815b40
docs: enhance the visual aid for accessibility purposes
Feb 4, 2021
0511fa5
fix: syntax previously broken in last set of commits
heitorlessa Feb 4, 2021
c21faea
fix: async ops snippet completeness
heitorlessa Feb 4, 2021
cca91d3
improv: screen real estate in home page.
heitorlessa Feb 4, 2021
8f0860f
feat: enable strict mode for broken links
heitorlessa Feb 5, 2021
4c07709
improv: merge Layer example to be more pragmatic
heitorlessa Feb 5, 2021
18e7a6f
improv: key features, getting started, clean non-essential
heitorlessa Feb 8, 2021
7bcb477
improv: expand escape hatch, async, reuse examples
heitorlessa Feb 8, 2021
cb2a570
improv: add getting started
heitorlessa Feb 8, 2021
d895006
improv: add getting started, trim content, expand examples
heitorlessa Feb 9, 2021
9d46917
improv: wording on logger reuse
heitorlessa Feb 9, 2021
6e8e108
improv: additional examples on formatting, order, etc. plus advanced …
heitorlessa Feb 9, 2021
383cc19
fix: typo on logging keys in FAQ
heitorlessa Feb 9, 2021
361634f
feat: enable snippets extension; include changelog in the docs
heitorlessa Feb 9, 2021
b78d2c3
fix: docker no longer works with strict due to an issue upstream
heitorlessa Feb 9, 2021
12853eb
feat: add favicon
heitorlessa Feb 9, 2021
e06d584
chore: consistency with metrics
heitorlessa Feb 9, 2021
cdf13f4
feat: docs revision date, smaller TOC, and footer copyright
heitorlessa Feb 9, 2021
aaaff91
improv: enhanced navigation, trimmed content, added more examples
heitorlessa Feb 9, 2021
ac29164
feat: add openGraph and Twitter card, attempt 1
heitorlessa Feb 9, 2021
d5365e0
Update docs/core/tracer.md
heitorlessa Feb 9, 2021
7d1be8c
Update docs/core/tracer.md
heitorlessa Feb 9, 2021
13a4070
docs: set tabs for yaml/json to 2 spaces
Feb 10, 2021
2422d3b
fix: yaml indent for cfn
Feb 10, 2021
fd548e1
fix: formatting yaml cfn
Feb 10, 2021
96524a3
fix: typo in features table
heitorlessa Feb 10, 2021
f2fab62
fix: typo in features table, match sidebar
heitorlessa Feb 10, 2021
7f5b7ed
Merge remote-tracking branch 'am29d/docs/mkdocs' into docs/mkdocs
heitorlessa Feb 10, 2021
12a6827
feat: add built-in search
heitorlessa Feb 10, 2021
9ab143c
fix: broken links
heitorlessa Feb 10, 2021
4ad97f9
improv: bring tenets up, clarify core vs general
heitorlessa Feb 10, 2021
9e38495
fix: broken links
heitorlessa Feb 10, 2021
b7191f0
fix: contrast for layer IAM snippet
heitorlessa Feb 10, 2021
e295ca2
fix: change extra dep to warning; wording
heitorlessa Feb 10, 2021
ad63957
remove ... from cfn docs
Feb 10, 2021
8c71f6b
Merge branch 'docs/mkdocs' of github.com:am29d/aws-lambda-powertools-…
Feb 10, 2021
6ca1d15
fix: width; remove old comments
heitorlessa Feb 10, 2021
5d27c18
improv: add explicit section for LOG_EVENT
heitorlessa Feb 10, 2021
3ce7c94
fix: highlight inject_lambda_context
heitorlessa Feb 10, 2021
7d907b3
feat: add docker-based docs for dev
heitorlessa Feb 10, 2021
2607f1d
chore: copyright to 2021
heitorlessa Feb 10, 2021
51a0ae6
revert: search width
heitorlessa Feb 10, 2021
3495899
chore: Add higher res metrics image
Feb 10, 2021
892dc5f
Merge branch 'docs/mkdocs' of https://github.com/am29d/aws-lambda-pow…
heitorlessa Feb 10, 2021
d340128
chore: remove custom font; correct edit uri
heitorlessa Feb 10, 2021
d63e231
feat: record page view and search pattern
heitorlessa Feb 12, 2021
e03e7f0
feat: add API reference
heitorlessa Feb 12, 2021
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
docs: validation
  • Loading branch information
Alex Melnyk committed Feb 2, 2021
commit 0c492e15a9970c5e7ca6d07259aa9deaf64edb61
235 changes: 122 additions & 113 deletions docs/utilities/validation.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,5 @@
---
title: Validation
description: Utility
---


import Note from "../../src/components/Note"
# Validation
## Utility

This utility provides JSON Schema validation for events and responses, including JMESPath support to unwrap events before validation.

Expand All @@ -22,9 +17,8 @@ You can also use the standalone `validate` function, if you want more control ov

We support any JSONSchema draft supported by [fastjsonschema](https://horejsek.github.io/python-fastjsonschema/) library.

<Note type="warning">
Both <code>validator</code> decorator and <code>validate</code> standalone function expects your JSON Schema to be
a <strong>dictionary</strong>, not a filename.
!!! warning
Both `validator` decorator and `validate` standalone function expects your JSON Schema to be a **dictionary**, not a filename.
</Note>


Expand All @@ -34,16 +28,18 @@ We support any JSONSchema draft supported by [fastjsonschema](https://horejsek.g

It will fail fast with `SchemaValidationError` exception if event or response doesn't conform with given JSON Schema.

```python:title=validator_decorator.py
from aws_lambda_powertools.utilities.validation import validator
=== "validator_decorator.py"

json_schema_dict = {..}
response_json_schema_dict = {..}
```python
from aws_lambda_powertools.utilities.validation import validator

@validator(inbound_schema=json_schema_dict, outbound_schema=response_json_schema_dict)
def handler(event, context):
return event
```
json_schema_dict = {..}
response_json_schema_dict = {..}

@validator(inbound_schema=json_schema_dict, outbound_schema=response_json_schema_dict)
def handler(event, context):
return event
```

**NOTE**: It's not a requirement to validate both inbound and outbound schemas - You can either use one, or both.

Expand All @@ -53,27 +49,28 @@ def handler(event, context):

You can also gracefully handle schema validation errors by catching `SchemaValidationError` exception.

```python:title=validator_decorator.py
from aws_lambda_powertools.utilities.validation import validate
from aws_lambda_powertools.utilities.validation.exceptions import SchemaValidationError
=== "validator_decorator.py"

json_schema_dict = {..}
```python
from aws_lambda_powertools.utilities.validation import validate
from aws_lambda_powertools.utilities.validation.exceptions import SchemaValidationError

def handler(event, context):
try:
validate(event=event, schema=json_schema_dict)
except SchemaValidationError as e:
# do something before re-raising
raise
json_schema_dict = {..}

return event
```
def handler(event, context):
try:
validate(event=event, schema=json_schema_dict)
except SchemaValidationError as e:
# do something before re-raising
raise

return event
```

### Validating custom formats

> New in 1.10.0
>
> **NOTE**: JSON Schema DRAFT 7 [has many new built-in formats](https://json-schema.org/understanding-json-schema/reference/string.html#format) such as date, time, and specifically a regex format which might be a better replacement for a custom format, if you do have control over the schema.
!!! note "New in 1.10.0"
JSON Schema DRAFT 7 [has many new built-in formats](https://json-schema.org/understanding-json-schema/reference/string.html#format) such as date, time, and specifically a regex format which might be a better replacement for a custom format, if you do have control over the schema.

If you have JSON Schemas with custom formats, for example having a `int64` for high precision integers, you can pass an optional validation to handle each type using `formats` parameter - Otherwise it'll fail validation:

Expand Down Expand Up @@ -112,48 +109,54 @@ Envelopes are [JMESPath expressions](https://jmespath.org/tutorial.html) to extr

Here is a sample custom EventBridge event, where we only validate what's inside the `detail` key:

```json:title=sample_wrapped_event.json
{
"id": "cdc73f9d-aea9-11e3-9d5a-835b769c0d9c",
"detail-type": "Scheduled Event",
"source": "aws.events",
"account": "123456789012",
"time": "1970-01-01T00:00:00Z",
"region": "us-east-1",
"resources": ["arn:aws:events:us-east-1:123456789012:rule/ExampleRule"],
"detail": {"message": "hello hello", "username": "blah blah"}, // highlight-line
}
```
=== "sample_wrapped_event.json"

```json hl_lines="9"
{
"id": "cdc73f9d-aea9-11e3-9d5a-835b769c0d9c",
"detail-type": "Scheduled Event",
"source": "aws.events",
"account": "123456789012",
"time": "1970-01-01T00:00:00Z",
"region": "us-east-1",
"resources": ["arn:aws:events:us-east-1:123456789012:rule/ExampleRule"],
"detail": {"message": "hello hello", "username": "blah blah"}
}
```

Here is how you'd use the `envelope` parameter to extract the payload inside the `detail` key before validating:

```python:title=unwrapping_events.py
from aws_lambda_powertools.utilities.validation import validator, validate
=== "unwrapping_events.py"

json_schema_dict = {..}
```python hl_lines="5 7"
from aws_lambda_powertools.utilities.validation import validator, validate

@validator(inbound_schema=json_schema_dict, envelope="detail") # highlight-line
def handler(event, context):
validate(event=event, schema=json_schema_dict, envelope="detail") # highlight-line
return event
```
json_schema_dict = {..}

@validator(inbound_schema=json_schema_dict, envelope="detail") # highlight-line
def handler(event, context):
validate(event=event, schema=json_schema_dict, envelope="detail") # highlight-line
return event
```

This is quite powerful because you can use JMESPath Query language to extract records from [arrays, slice and dice](https://jmespath.org/tutorial.html#list-and-slice-projections), to [pipe expressions](https://jmespath.org/tutorial.html#pipe-expressions) and [function expressions](https://jmespath.org/tutorial.html#functions), where you'd extract what you need before validating the actual payload.

## Built-in envelopes

This utility comes with built-in envelopes to easily extract the payload from popular event sources.

```python:title=unwrapping_popular_event_sources.py
from aws_lambda_powertools.utilities.validation import envelopes, validate, validator
=== "unwrapping_popular_event_sources.py"

json_schema_dict = {..}
```python hl_lines="5 7"
from aws_lambda_powertools.utilities.validation import envelopes, validate, validator

@validator(inbound_schema=json_schema_dict, envelope=envelopes.EVENTBRIDGE) # highlight-line
def handler(event, context):
validate(event=event, schema=json_schema_dict, envelope=envelopes.EVENTBRIDGE) # highlight-line
return event
```
json_schema_dict = {..}

@validator(inbound_schema=json_schema_dict, envelope=envelopes.EVENTBRIDGE)
def handler(event, context):
validate(event=event, schema=json_schema_dict, envelope=envelopes.EVENTBRIDGE)
return event
```

Here is a handy table with built-in envelopes along with their JMESPath expressions in case you want to build your own.

Expand All @@ -174,98 +177,104 @@ You might have events or responses that contain non-encoded JSON, where you need

You can use our built-in JMESPath functions within your expressions to do exactly that to decode JSON Strings, base64, and uncompress gzip data.

<Note type="info">
!!! info
We use these for built-in envelopes to easily to decode and unwrap events from sources like Kinesis, CloudWatch Logs, etc.
</Note>

### powertools_json function

Use `powertools_json` function to decode any JSON String.

This sample will decode the value within the `data` key into a valid JSON before we can validate it.

```python:title=powertools_json_jmespath_function.py
from aws_lambda_powertools.utilities.validation import validate
=== "powertools_json_jmespath_function.py"

json_schema_dict = {..}
sample_event = {
'data': '{"payload": {"message": "hello hello", "username": "blah blah"}}'
}
```python hl_lines="9"
from aws_lambda_powertools.utilities.validation import validate

def handler(event, context):
validate(event=event, schema=json_schema_dict, envelope="powertools_json(data)") # highlight-line
return event
json_schema_dict = {..}
sample_event = {
'data': '{"payload": {"message": "hello hello", "username": "blah blah"}}'
}

handler(event=sample_event, context={})
```
def handler(event, context):
validate(event=event, schema=json_schema_dict, envelope="powertools_json(data)")
return event

handler(event=sample_event, context={})
```

### powertools_base64 function

Use `powertools_base64` function to decode any base64 data.

This sample will decode the base64 value within the `data` key, and decode the JSON string into a valid JSON before we can validate it.

```python:title=powertools_json_jmespath_function.py
from aws_lambda_powertools.utilities.validation import validate
=== "powertools_json_jmespath_function.py"

json_schema_dict = {..}
sample_event = {
"data": "eyJtZXNzYWdlIjogImhlbGxvIGhlbGxvIiwgInVzZXJuYW1lIjogImJsYWggYmxhaCJ9="
}
```python hl_lines="9"
from aws_lambda_powertools.utilities.validation import validate

def handler(event, context):
validate(event=event, schema=json_schema_dict, envelope="powertools_json(powertools_base64(data))") # highlight-line
return event
json_schema_dict = {..}
sample_event = {
"data": "eyJtZXNzYWdlIjogImhlbGxvIGhlbGxvIiwgInVzZXJuYW1lIjogImJsYWggYmxhaCJ9="
}

handler(event=sample_event, context={})
```
def handler(event, context):
validate(event=event, schema=json_schema_dict, envelope="powertools_json(powertools_base64(data))")
return event

handler(event=sample_event, context={})
```

### powertools_base64_gzip function

Use `powertools_base64_gzip` function to decompress and decode base64 data.

This sample will decompress and decode base64 data, then use JMESPath pipeline expression to pass the result for decoding its JSON string.

```python:title=powertools_json_jmespath_function.py
from aws_lambda_powertools.utilities.validation import validate
=== "powertools_json_jmespath_function.py"

json_schema_dict = {..}
sample_event = {
"data": "H4sIACZAXl8C/52PzUrEMBhFX2UILpX8tPbHXWHqIOiq3Q1F0ubrWEiakqTWofTdTYYB0YWL2d5zvnuTFellBIOedoiyKH5M0iwnlKH7HZL6dDB6ngLDfLFYctUKjie9gHFaS/sAX1xNEq525QxwFXRGGMEkx4Th491rUZdV3YiIZ6Ljfd+lfSyAtZloacQgAkqSJCGhxM6t7cwwuUGPz4N0YKyvO6I9WDeMPMSo8Z4Ca/kJ6vMEYW5f1MX7W1lVxaG8vqX8hNFdjlc0iCBBSF4ERT/3Pl7RbMGMXF2KZMh/C+gDpNS7RRsp0OaRGzx0/t8e0jgmcczyLCWEePhni/23JWalzjdu0a3ZvgEaNLXeugEAAA=="
}
```python hl_lines="9"
from aws_lambda_powertools.utilities.validation import validate

def handler(event, context):
validate(event=event, schema=json_schema_dict, envelope="powertools_base64_gzip(data) | powertools_json(@)") # highlight-line
return event
json_schema_dict = {..}
sample_event = {
"data": "H4sIACZAXl8C/52PzUrEMBhFX2UILpX8tPbHXWHqIOiq3Q1F0ubrWEiakqTWofTdTYYB0YWL2d5zvnuTFellBIOedoiyKH5M0iwnlKH7HZL6dDB6ngLDfLFYctUKjie9gHFaS/sAX1xNEq525QxwFXRGGMEkx4Th491rUZdV3YiIZ6Ljfd+lfSyAtZloacQgAkqSJCGhxM6t7cwwuUGPz4N0YKyvO6I9WDeMPMSo8Z4Ca/kJ6vMEYW5f1MX7W1lVxaG8vqX8hNFdjlc0iCBBSF4ERT/3Pl7RbMGMXF2KZMh/C+gDpNS7RRsp0OaRGzx0/t8e0jgmcczyLCWEePhni/23JWalzjdu0a3ZvgEaNLXeugEAAA=="
}

handler(event=sample_event, context={})
```
def handler(event, context):
validate(event=event, schema=json_schema_dict, envelope="powertools_base64_gzip(data) | powertools_json(@)")
return event

handler(event=sample_event, context={})
```

## Bring your own JMESPath function

<Note type="warning">
!!! warning
This should only be used for advanced use cases where you have special formats not covered by the built-in functions.
<br/><br/>
This will <strong>replace all provided built-in functions such as `powertools_json`, so you will no longer be able to use them</strong>.
</Note>

This will **replace all provided built-in functions such as `powertools_json`, so you will no longer be able to use them**.

For special binary formats that you want to decode before applying JSON Schema validation, you can bring your own [JMESPath function](https://github.com/jmespath/jmespath.py#custom-functions) and any additional option via `jmespath_options` param.

```python:title=custom_jmespath_function
from aws_lambda_powertools.utilities.validation import validate
from jmespath import functions
=== "custom_jmespath_function.py"

json_schema_dict = {..}
```python hl_lines="15"
from aws_lambda_powertools.utilities.validation import validate
from jmespath import functions

class CustomFunctions(functions.Functions):
json_schema_dict = {..}

@functions.signature({'types': ['string']})
def _func_special_decoder(self, s):
return my_custom_decoder_logic(s)
class CustomFunctions(functions.Functions):

custom_jmespath_options = {"custom_functions": CustomFunctions()}
@functions.signature({'types': ['string']})
def _func_special_decoder(self, s):
return my_custom_decoder_logic(s)

def handler(event, context):
validate(event=event, schema=json_schema_dict, envelope="", jmespath_options=**custom_jmespath_options) # highlight-line
return event
```
custom_jmespath_options = {"custom_functions": CustomFunctions()}

def handler(event, context):
validate(event=event, schema=json_schema_dict, envelope="", jmespath_options=**custom_jmespath_options)
return event
```