Add Amazon Bedrock Guardrails Improvements

[agithomas]

• Enhancement

Proposed commit message

Add improvements to the Bedrock overview dashboard and Guardrails dashboard

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Integration testing
• Screenshot rendering testing

How to test this PR locally

• elastic-package build
• elastic-package stack up -v -d --services package-registry
• Perform package upgrade

Related issues

• Closes [Amazon Bedrock] Guardrails dashboard improvements #12420

Screenshots

Amazon Bedrock Overview dashboard

---

Guardrails Overview Dashboard

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[muthu-mps]

LGTM!

[ishleenk17]

Are we using LLM or models in the charts ?
Eg: "Overall latency by LLM" or "Overall Latency by model"
Since we plan to support non LLM models as well ?

cc: @daniela-elastic

[agithomas]

Eg: "Overall latency by LLM" or "Overall Latency by model"

@daniela-elastic , the alternate option is to use "by foundation models". Reference. What do you suggest?

[shmsr]

Yes, dashboard looks LGTM. Yes, let's make the titles FM or models instead of LLM. Rest, looks good.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 9831ce1a72b4fda9db3db83317f5609598a67c75

History

• 💔 Build #20827 failed 68b9a8cc1422b3bc22e5309c106d8d2c19a16310
• 💚 Build #20771 succeeded 4c78a478216918337f5b71a9ca11a09b2d88c0f5

cc @agithomas

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elastic-vault-github-plugin-prod]

Package aws_bedrock - 0.22.0 containing this change is available at https://epr.elastic.co/package/aws_bedrock/0.22.0/

[ishleenk17]

@agithomas We were reviewing the GCP vertex AI dashboard. Some feedback from AWS as part of those discussions.

1. For the Number of invocations that are intervened panel, the Y axis should be Invocations, instead of Interventions.
2. The model usage can be a donut instead of a pie chart, like in Vertex AI.

cc: @daniela-elastic

[daniela-elastic]

For no1 yes it makes sense as long as we don't change the name of the widget - "umber of invocations where the guardrails intervened"
And yes, we agreed let's use donuts instead of pie charts

[daniela-elastic]

I'm fine to call it "- by foundation model" (singular, not plural). Or even just "by model". Is there any reason why we need to specify that it's "foundation" or it can just be assumed?

[ishleenk17]

I'm fine to call it "- by foundation model" (singular, not plural). Or even just "by model". Is there any reason why we need to specify that it's "foundation" or it can just be assumed?

I think model should be a generic term that we can use across LLM Integrations.
@muthu-mps : In case Azure OpenAI needs such changes as well.

[shmsr]

I'm fine to call it "- by foundation model" (singular, not plural). Or even just "by model". Is there any reason why we need to specify that it's "foundation" or it can just be assumed?

The term "Foundation Models" originated in a 2021 paper (https://arxiv.org/pdf/2108.07258). While almost all models we are plotting here qualify as Foundation Models (FM), except most embedding models, because:

• Limited scope: They focus on data representations in vectors and act as components of FM (like a building block).
• No emergent behavior: While embedding models can capture complex relationships, they lack the emergent properties (performing arithmetic, reasoning, answering questions, summarizing passages seen in language model, and more).
• Limited adaptability: They serve as building blocks for FMs but aren't typically fine-tuned for various downstream tasks in the same way that a language model is.

But yes in case an embedding model starts showing these properties, then they come under FM.

Since "models" is a superset of "foundation models", using "model" as also suggested by Daniela seems a better choice for the dashboard - it looks cleaner while future-proofing our terminology for any new model types that may not exhibit foundation model properties.

[muthu-mps]

@agithomas - The i icon to add description in the metrics chart is updated in the recent version. This doesn't include the icon by default. You have to hover over the chart to see the icon. This is with the 8.17.1 release version.

Without hover

Hover over the total invocations chart.

[agithomas]

Presently, the amazon bedrock integration has 8.16.2 dependency. I think it may not be good to increment the version to 8.17.1 for a small visual improvement. I assume the users who upgrade to 8.17.1 stack will be able to take the benefit of the above feature