generated from amazon-archives/__template_MIT-0
- Notifications
You must be signed in to change notification settings - Fork 458
feat: SQS Partial failure #100
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
heitorlessa merged 26 commits into aws-powertools:develop from gmcrocetti:partial-sqs-batch Sep 1, 2020
Merged
Changes from 1 commit
Commits
Show all changes
26 commits Select commit Hold shift + click to select a range
2e00554 feat: add batch module
gmcrocetti 9015d43 feat: include base processors
gmcrocetti 48a7687 feat: add sqs failure processors
gmcrocetti 9c7324b fix: include proposed suggestions
gmcrocetti e7e36a9 feat(sqs): improve validation for queue_url
gmcrocetti 8dbaf67 feat: add package level import for batch utility
gmcrocetti 524d27f refactor: change return for failure/success handlers
gmcrocetti 7d32e8c test: add unit tests for partial sqs processor
gmcrocetti f9c53e0 test: add unit tests for partial sqs processor
gmcrocetti 680ee69 test: functional tests for partial sqs processor and its middleware
gmcrocetti 6cddf76 feat(sqs): add optional config parameter
gmcrocetti 13e3132 refactor(tests): processor using default config
gmcrocetti 4275fb2 refactor(sqs): change methods to protected
gmcrocetti 54a76df fix(base-partial): append record instead of entry
gmcrocetti 438ecd5 docs(sqs): docstrings for PartialSQS
gmcrocetti c7582f3 docs(sqs-base): docstring for base class
gmcrocetti f419ef6 refactor(sqs): add module middlewares
gmcrocetti c999f96 refactor: changes partial_sqs middleware in favor of a generic interf…
gmcrocetti 5cb6b89 refactor(tests): update tests to new batch processor middleware
gmcrocetti be06149 refactor: batch middleware
gmcrocetti 8ba81b4 docs(partial-processor): add simple docstrings to success/failure han…
gmcrocetti edcc14a fix: typo in example
gmcrocetti ded3d75 docs: user specific documentation
gmcrocetti 14cc383 docs: fix suggestions made by @heitorlessa
gmcrocetti 25d67ca refactor: remove references to BaseProcessor. Left BasePartialProcessor
gmcrocetti 9caf3d1 docs: refactor example; improve docs about creating your own processor
gmcrocetti File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
docs: user specific documentation
- Loading branch information
commit ded3d755bd1b11fd62c73dd11ca59a983bef62a1
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| --- | ||
| title: Batch | ||
| description: Utility | ||
| --- | ||
| | ||
| import Note from "../../src/components/Note" | ||
| | ||
| The batch utility provides an abstraction to process a batch event. Useful for lambda integrations with [AWS SQS](https://aws.amazon.com/sqs/), [AWS Kinesis](https://aws.amazon.com/kinesis/) and [AWS DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html). | ||
| It also provides base classes (`BaseProcessor`, `BasePartialProcessor`) allowing you to create your **own** batch processor. | ||
| | ||
| **Key Features** | ||
| | ||
| * Run batch processing logic with a clean interface; | ||
gmcrocetti marked this conversation as resolved. Outdated Show resolved Hide resolved | ||
| * Middleware and context to handle a batch event; | ||
gmcrocetti marked this conversation as resolved. Outdated Show resolved Hide resolved | ||
| * Removal of successful messages for [AWS SQS](https://aws.amazon.com/sqs/) batch - in case of partial failure. | ||
gmcrocetti marked this conversation as resolved. Outdated Show resolved Hide resolved | ||
| | ||
| **IAM Permissions** | ||
| | ||
| This utility requires additional permissions to work as expected. See the following table: | ||
| | ||
| Processor | Function/Method | IAM Permission | ||
| |---------|-----------------|---------------| | ||
| PartialSQSProcessor | `_clean` | `sqs:DeleteMessageBatch` | ||
| | ||
| ### PartialSQSProcessor | ||
| | ||
gmcrocetti marked this conversation as resolved. Show resolved Hide resolved | ||
| A special batch processor which aims to `clean` your SQS:Queue if one or more (not all) records of the batch fails. | ||
| A batch's partial failure sends back all the records to the queue, reprocessing this batch until all records succed. | ||
| This processor exists to improve performance in such cases, deleting successful messages of a batch with partial failure. | ||
| | ||
| ### Middleware | ||
| | ||
| ```python:title=app.py | ||
| from aws_lambda_powertools.utilities.batch import batch_processor, PartialSQSProcessor | ||
| | ||
| def record_handler(record): | ||
| return record["body"] | ||
| | ||
| # highlight-start | ||
| @batch_processor(record_handler=record_handler, processor=PartialSQSProcessor()) | ||
| # highlight-end | ||
| def lambda_handler(event, context): | ||
| return {"statusCode": 200} | ||
| ``` | ||
| | ||
| ## Create your own processor | ||
| | ||
| You can create your own batch processor by inheriting the `BaseProcessor` class, and implementing `_prepare()`, `_clean` and `_process_record()`. | ||
gmcrocetti marked this conversation as resolved. Outdated Show resolved Hide resolved | ||
| It's also possible to inherit the `BasePartialProcessor` which contains additional logic to handle a partial failure and keep track of record status. | ||
gmcrocetti marked this conversation as resolved. Outdated Show resolved Hide resolved | ||
| | ||
| Here is an example implementation of a DynamoDBStream custom processor: | ||
| | ||
| ```python:title=custom_processor.py | ||
| import json | ||
| | ||
| from aws_lambda_powertools.utilities.batch import BaseProcessor, batch_processor | ||
| import boto3 | ||
| | ||
| class DynamoDBProcessor(BaseProcessor): | ||
gmcrocetti marked this conversation as resolved. Outdated Show resolved Hide resolved | ||
| | ||
| def __init__(self, queue_url: str): | ||
| self.queue_url = queue_url | ||
| self.client = boto3.client("sqs") | ||
heitorlessa marked this conversation as resolved. Outdated Show resolved Hide resolved | ||
| | ||
| def _prepare(self): | ||
| pass | ||
| | ||
| def _clean(self): | ||
| pass | ||
| | ||
| def _process_record(self, record): | ||
| """ | ||
| Process record and send result to sqs | ||
| """ | ||
| result = self.handler(record) | ||
| body = json.dumps(result) | ||
| self.client.send_message(QueueUrl=self.queue_url, MessageBody=body) | ||
| return result | ||
| | ||
| def record_handler(record): | ||
| return record["Keys"] | ||
| | ||
| # As context | ||
| | ||
| processor = DynamoDBProcessor("dummy") | ||
gmcrocetti marked this conversation as resolved. Outdated Show resolved Hide resolved gmcrocetti marked this conversation as resolved. Outdated Show resolved Hide resolved | ||
| records = {"Records": []} | ||
| | ||
| with processor(records=records, handler=record_handler) as ctx: | ||
| result = ctx.process() | ||
| | ||
| # As middleware | ||
| @batch.batch_processor(record_handler=record_handler, processor=DynamoDBProcessor("dummy")) | ||
gmcrocetti marked this conversation as resolved. Outdated Show resolved Hide resolved | ||
| def lambda_handler(event, context): | ||
| return {"statusCode": 200} | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.