Skip to content

Amazon Bedrock - Add Guardrails metrics #12250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 0 commits into from
Jan 10, 2025
Merged

Amazon Bedrock - Add Guardrails metrics #12250

merged 0 commits into from
Jan 10, 2025

Conversation

@agithomas
Copy link
Contributor

@agithomas agithomas commented Jan 8, 2025

  • Enhancement

Proposed commit message

Add guardrails dataset for guardrail metrics

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • Integration testing with Guardrail API
  • Integration testing without using Guardrail API
  • Agent invocation testing
  • Input data guardrail testing
  • Output data guardrail testing

How to test this PR locally

  • elastic-package build
  • elastic-package stack up -v -d --services package-registry

Screenshots

image
@andrewkroh andrewkroh added enhancement New feature or request Integration:aws_bedrock Amazon Bedrock labels Jan 8, 2025
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Jan 8, 2025

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport
@agithomas agithomas marked this pull request as ready for review January 8, 2025 07:06
@agithomas agithomas requested review from a team as code owners January 8, 2025 07:06
@andrewkroh andrewkroh added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Jan 8, 2025
@elasticmachine
Copy link

elasticmachine commented Jan 8, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
muthu-mps
muthu-mps reviewed Jan 8, 2025
View reviewed changes
packages/aws_bedrock/data_stream/guardrails/elasticsearch/ingest_pipeline/default.yml Outdated
Copy link
Contributor

@muthu-mps muthu-mps Jan 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we exclude the guradrail_ from guardrails.guardrail_content_source for all dimension fields? This would make the field name consistent with the rest of the fields. WDYT?
Copy link
Contributor Author

@agithomas agithomas Jan 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had the below reasoning to follow this naming convension

  1. When i looked at the AWS documentation, i could see the dimension names Operation and GuardrailContentSource. If we format the name of the field as ContentSource or 'arn`, in the context of LLM application ,where content source can also mean the source of knowledge, arn can be the arn of the model or arn of agent and so on. So, i think, it is best to keep the name as guardrail_content_source or guardrail_arn to make it less ambiguous for the user.

  2. We don't have a description of the dimension fields.
@elasticmachine
Copy link

elasticmachine commented Jan 8, 2025

💚 Build Succeeded

History

  • 💚 Build #20111 succeeded 8a666d5433d8eb1c8427daf6cb708874f82072c1
  • 💔 Build #20108 failed 8bffc6f004789102797c09db624128aea5e20411
@elastic-sonarqube
Copy link

elastic-sonarqube bot commented Jan 8, 2025

Quality Gate failed Quality Gate failed

Failed conditions
26.9% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube
muthu-mps
muthu-mps approved these changes Jan 9, 2025
View reviewed changes
Copy link
Contributor

@muthu-mps muthu-mps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
@agithomas agithomas merged commit 5468ed6 into elastic:main Jan 10, 2025
4 of 5 checks passed
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Jan 10, 2025

Package aws_bedrock - 0.19.0 containing this change is available at https://epr.elastic.co/package/aws_bedrock/0.19.0/
This was referenced Jan 13, 2025
Merged
Merged
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@agithomas
Amazon Bedrock - Add Guardrails metrics (elastic#12250)
f811a42 
* Amazon Bedrock Add Guardrails metrics
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@agithomas
Amazon Bedrock - Add Guardrails metrics (elastic#12250)
a88c410 
* Amazon Bedrock Add Guardrails metrics
@agithomas agithomas deleted the issue-1434-amazon-bedrock-guardrails branch February 6, 2025 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:aws_bedrock Amazon Bedrock Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

5 participants

@agithomas @elasticmachine @shmsr @muthu-mps @andrewkroh