Tailpipe is an open-source CLI tool that allows you to collect logs and query them with SQL.

AWS provides on-demand cloud computing platforms and APIs to authenticated customers on a metered pay-as-you-go basis.

The AWS Plugin for Tailpipe allows you to collect and query AWS logs using SQL to track activity, monitor trends, detect anomalies, and more!

• Get started →
• Documentation: Table definitions & examples
• Community: Join #tailpipe on Slack →
• Get involved: Issues

Collect and query logs:

Install Tailpipe from the downloads page:

# MacOS brew install turbot/tap/tailpipe

# Linux or Windows (WSL) sudo /bin/sh -c "$(curl -fsSL https://tailpipe.io/install/tailpipe.sh)"

Install the plugin:

tailpipe plugin install aws

Configure your connection credentials, table partition, and data source (examples):

vi ~/.tailpipe/config/aws.tpc

connection "aws" "logging_account" { profile = "my-logging-account" } partition "aws_cloudtrail_log" "my_logs" { source "aws_s3_bucket" { connection = connection.aws.logging_account bucket = "aws-cloudtrail-logs-bucket" } }

Download, enrich, and save logs from your source (examples):

tailpipe collect aws_cloudtrail_log

Enter interactive query mode:

tailpipe query

Run a query:

select event_source, event_name, count(*) as event_count from aws_cloudtrail_log where not read_only group by event_source, event_name order by event_count desc;

+----------------------+-----------------------+-------------+ | event_source | event_name | event_count | +----------------------+-----------------------+-------------+ | logs.amazonaws.com | CreateLogStream | 793845 | | ecs.amazonaws.com | RunTask | 350836 | | ecs.amazonaws.com | SubmitTaskStateChange | 190185 | | s3.amazonaws.com | PutObject | 60842 | | sns.amazonaws.com | TagResource | 25499 | | lambda.amazonaws.com | TagResource | 20673 | +----------------------+-----------------------+-------------+

Pre-built dashboards and detections for the AWS plugin are available in Powerpipe mods, helping you monitor and analyze activity across your AWS accounts.

For example, the AWS CloudTrail Logs Detections mod scans your CloudTrail logs for anomalies, such as an S3 bucket being made public or a change in your VPC network infrastructure.

Dashboards and detections are open source, allowing easy customization and collaboration.

To get started, choose a mod from the Powerpipe Hub.

Prerequisites:

• Tailpipe
• Golang

Clone:

git clone https://github.com/turbot/tailpipe-plugin-aws.git cd tailpipe-plugin-aws

After making your local changes, build the plugin, which automatically installs the new version to your ~/.tailpipe/plugins directory:

make

Re-collect your data:

tailpipe collect aws_cloudtrail_log

Try it!

tailpipe query > .inspect aws_cloudtrail_log

This repository is published under the Apache 2.0 (source code) and CC BY-NC-ND (docs) licenses. Please see our code of conduct. We look forward to collaborating with you!

Tailpipe is a product produced from this open source software, exclusively by Turbot HQ, Inc. It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our Open Source FAQ.

Join #tailpipe on Slack →

Want to help but don't know where to start? Pick up one of the help wanted issues:

• Tailpipe
• AWS Plugin