[8.19][Backport] Implement SAML custom attributes support for Identity Provider (#128796)

* Implement SAML custom attributes support for Identity Provider (#128176) * Implement SAML custom attributes support for Identity Provider This commit adds support for custom attributes in SAML single sign-on requests in the Elasticsearch X-Pack Identity Provider plugin. This feature allows passage of custom key-value attributes in SAML requests and responses. Key components: - Added SamlInitiateSingleSignOnAttributes class for holding attributes - Added validation for null and empty attribute keys - Updated request and response objects to handle attributes - Modified authentication flow to process attributes - Added test coverage to validate attributes functionality The implementation follows Elasticsearch patterns with robust validation and serialization mechanisms, while maintaining backward compatibility. * Add test for SAML custom attributes in authentication response This commit adds a comprehensive test that verifies SAML custom attributes are correctly handled in the authentication response builder. The test ensures: 1. Custom attributes with single and multiple values are properly included 2. The response with custom attributes is still correctly signed 3. The XML schema validation still passes with custom attributes 4. We can locate and verify individual attribute values in the response This provides critical test coverage for the SAML custom attributes feature implementation. * Add backward compatibility overload for SuccessfulAuthenticationResponseMessageBuilder.build This commit adds an overloaded build method that accepts only two parameters (user and authenticationState) and forwards the call to the three-parameter version with null for the customAttributes parameter. This maintains backward compatibility with existing code that doesn't use custom attributes. This fixes a compilation error in ServerlessSsoIT.java which was still using the two-parameter method signature. Signed-off-by: lloydmeta <lloydmeta@gmail.com> * Add validation for duplicate SAML attribute keys This commit enhances the SAML attributes implementation by adding validation for duplicate attribute keys. When the same attribute key appears multiple times in a request, the validation will now fail with a clear error message. Signed-off-by: lloydmeta <lloydmeta@gmail.com> * Refactor SAML attributes validation to follow standard patterns This commit improves the SAML attributes validation by: 1. Adding a dedicated validate() method to SamlInitiateSingleSignOnAttributes that centralizes validation logic in one place 2. Moving validation from constructor to dedicated method for better error reporting 3. Checking both for null/empty keys and duplicate keys in the validate() method 4. Updating SamlInitiateSingleSignOnRequest to use the new validation method 5. Adding comprehensive tests for the new validation approach These changes follow standard Elasticsearch validation patterns, making the code more maintainable and consistent with the rest of the codebase. * Update docs/changelog/128176.yaml * Improve SAML response validation in identity provider tests Enhanced the testCustomAttributesInIdpInitiatedSso test to properly validate both SAML response structure and custom attributes using DOM parsing and XPath. Key improvements: - Validate SAML Response/Assertion elements exist - Precisely validate custom attributes (department, region) and their values - Use namespace-aware XML parsing for resilience to format changes Signed-off-by: lloydmeta <lloydmeta@gmail.com> * Simplify SAML attributes representation using JSON object/Map structure Also, replace internal Attribute class list with a simpler Map<String, List<String>> structure This change: - Removes the redundant Attribute class and replaces it with a direct Map implementation for storing attribute key-value pairs - Eliminates the duplicate "attributes" nesting in the JSON structure - Simplifies attribute validation without needing duplicate key checking - Updates all related tests and integration points to work with the new structure Before: ```js { // others "attributes": { "attributes": [ { "key": "department", "values": ["engineering", "product"] } ] } } After: ```js { // other "attributes": { "department": ["engineering", "product"] } } ``` (Verified by spitting out JSON entity in IdentityProviderAuthenticationIT.generateSamlResponseWithAttributes ... saw `{"entity_id":"ec:123456:abcdefg","acs":"https://sp1.test.es.elasticsearch.org/saml/acs","attributes":{"department":["engineering","product"],"region":["APJ"]}}`) Signed-off-by: lloydmeta <lloydmeta@gmail.com> * * Fix up toString dangling quote. Signed-off-by: lloydmeta <lloydmeta@gmail.com> * * Remove attributes from Response object. Signed-off-by: lloydmeta <lloydmeta@gmail.com> * * Remove friendly name. * Make attributes map final in SamlInitiateSingleSignOnAttributes Signed-off-by: lloydmeta <lloydmeta@gmail.com> * * Cleanup serdes by using existing utils in the ES codebase Signed-off-by: lloydmeta <lloydmeta@gmail.com> * Touchup comment Signed-off-by: lloydmeta <lloydmeta@gmail.com> * Update x-pack/plugin/identity-provider/src/test/java/org/elasticsearch/xpack/idp/action/SamlInitiateSingleSignOnRequestTests.java Co-authored-by: Tim Vernum <tim@adjective.org> * Add transport-version checks --------- Signed-off-by: lloydmeta <lloydmeta@gmail.com> Co-authored-by: Tim Vernum <tim@adjective.org> * * TV fixups Signed-off-by: lloydmeta <lloydmeta@gmail.com> --------- Signed-off-by: lloydmeta <lloydmeta@gmail.com> Co-authored-by: Tim Vernum <tim@adjective.org>

Author: lloydmeta

docs/changelog/128176.yaml

@@ -0,0 +1,5 @@
+pr: 128176
+summary: Implement SAML custom attributes support for Identity Provider
+area: Authentication
+type: enhancement
+issues: []

server/src/main/java/org/elasticsearch/TransportVersions.java

@@ -229,7 +229,7 @@ static TransportVersion def(int id) {
  public static final TransportVersion ML_INFERENCE_SAGEMAKER_CHAT_COMPLETION_8_19 = def(8_841_0_37);
  public static final TransportVersion ML_INFERENCE_VERTEXAI_CHATCOMPLETION_ADDED_8_19 = def(8_841_0_38);
  public static final TransportVersion INFERENCE_CUSTOM_SERVICE_ADDED_8_19 = def(8_841_0_39);
-
+ public static final TransportVersion IDP_CUSTOM_SAML_ATTRIBUTES_ADDED_8_19 = def(8_841_0_40);
  /*
  * STOP! READ THIS FIRST! No, really,
  * ____ _____ ___ ____ _ ____ _____ _ ____ _____ _ _ ___ ____ _____ ___ ____ ____ _____ _

x-pack/plugin/identity-provider/qa/idp-rest-tests/src/javaRestTest/java/org/elasticsearch/xpack/idp/IdentityProviderAuthenticationIT.java

@@ -18,23 +18,38 @@
 import org.elasticsearch.common.util.concurrent.ThreadContext;
 import org.elasticsearch.core.Nullable;
 import org.elasticsearch.xcontent.ObjectPath;
+import org.elasticsearch.xcontent.XContentBuilder;
 import org.elasticsearch.xcontent.json.JsonXContent;
 import org.elasticsearch.xpack.core.security.action.saml.SamlPrepareAuthenticationResponse;
 import org.elasticsearch.xpack.idp.saml.sp.SamlServiceProviderIndex;
 import org.junit.Before;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+import org.w3c.dom.NodeList;
+import org.xml.sax.InputSource;
 
 import java.io.IOException;
+import java.io.StringReader;
 import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
 import java.util.Base64;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
 
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.xpath.XPath;
+import javax.xml.xpath.XPathConstants;
+import javax.xml.xpath.XPathFactory;
+
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.hasSize;
 import static org.hamcrest.Matchers.instanceOf;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.notNullValue;
 
 public class IdentityProviderAuthenticationIT extends IdpRestTestCase {
 
@@ -74,6 +89,81 @@ public void testRegistrationAndIdpInitiatedSso() throws Exception {
  authenticateWithSamlResponse(samlResponse, null);
  }
 
+ public void testCustomAttributesInIdpInitiatedSso() throws Exception {
+ final Map<String, Object> request = Map.ofEntries(
+ Map.entry("name", "Test SP With Custom Attributes"),
+ Map.entry("acs", SP_ACS),
+ Map.entry("privileges", Map.ofEntries(Map.entry("resource", SP_ENTITY_ID), Map.entry("roles", List.of("sso:(\\w+)")))),
+ Map.entry(
+ "attributes",
+ Map.ofEntries(
+ Map.entry("principal", "https://idp.test.es.elasticsearch.org/attribute/principal"),
+ Map.entry("name", "https://idp.test.es.elasticsearch.org/attribute/name"),
+ Map.entry("email", "https://idp.test.es.elasticsearch.org/attribute/email"),
+ Map.entry("roles", "https://idp.test.es.elasticsearch.org/attribute/roles")
+ )
+ )
+ );
+ final SamlServiceProviderIndex.DocumentVersion docVersion = createServiceProvider(SP_ENTITY_ID, request);
+ checkIndexDoc(docVersion);
+ ensureGreen(SamlServiceProviderIndex.INDEX_NAME);
+
+ // Create custom attributes
+ Map<String, List<String>> attributesMap = Map.of("department", List.of("engineering", "product"), "region", List.of("APJ"));
+
+ // Generate SAML response with custom attributes
+ final String samlResponse = generateSamlResponseWithAttributes(SP_ENTITY_ID, SP_ACS, null, attributesMap);
+
+ // Parse XML directly from samlResponse (it's not base64 encoded at this point)
+ DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
+ factory.setNamespaceAware(true); // Required for XPath
+ DocumentBuilder builder = factory.newDocumentBuilder();
+ Document document = builder.parse(new InputSource(new StringReader(samlResponse)));
+
+ // Create XPath evaluator
+ XPathFactory xPathFactory = XPathFactory.newInstance();
+ XPath xpath = xPathFactory.newXPath();
+
+ // Validate SAML Response structure
+ Element responseElement = (Element) xpath.evaluate("//*[local-name()='Response']", document, XPathConstants.NODE);
+ assertThat("SAML Response element should exist", responseElement, notNullValue());
+
+ Element assertionElement = (Element) xpath.evaluate("//*[local-name()='Assertion']", document, XPathConstants.NODE);
+ assertThat("SAML Assertion element should exist", assertionElement, notNullValue());
+
+ // Validate department attribute
+ NodeList departmentAttributes = (NodeList) xpath.evaluate(
+ "//*[local-name()='Attribute' and @Name='department']/*[local-name()='AttributeValue']",
+ document,
+ XPathConstants.NODESET
+ );
+
+ assertThat("Should have two values for department attribute", departmentAttributes.getLength(), is(2));
+
+ // Verify department values
+ List<String> departmentValues = new ArrayList<>();
+ for (int i = 0; i < departmentAttributes.getLength(); i++) {
+ departmentValues.add(departmentAttributes.item(i).getTextContent());
+ }
+ assertThat(
+ "Department attribute should contain 'engineering' and 'product'",
+ departmentValues,
+ containsInAnyOrder("engineering", "product")
+ );
+
+ // Validate region attribute
+ NodeList regionAttributes = (NodeList) xpath.evaluate(
+ "//*[local-name()='Attribute' and @Name='region']/*[local-name()='AttributeValue']",
+ document,
+ XPathConstants.NODESET
+ );
+
+ assertThat("Should have one value for region attribute", regionAttributes.getLength(), is(1));
+ assertThat("Region attribute should contain 'APJ'", regionAttributes.item(0).getTextContent(), equalTo("APJ"));
+
+ authenticateWithSamlResponse(samlResponse, null);
+ }
+
  public void testRegistrationAndSpInitiatedSso() throws Exception {
  final Map<String, Object> request = Map.ofEntries(
  Map.entry("name", "Test SP"),
@@ -125,17 +215,37 @@ private SamlPrepareAuthenticationResponse generateSamlAuthnRequest(String realmN
  }
  }
 
- private String generateSamlResponse(String entityId, String acs, @Nullable Map<String, Object> authnState) throws Exception {
+ private String generateSamlResponse(String entityId, String acs, @Nullable Map<String, Object> authnState) throws IOException {
+ return generateSamlResponseWithAttributes(entityId, acs, authnState, null);
+ }
+
+ private String generateSamlResponseWithAttributes(
+ String entityId,
+ String acs,
+ @Nullable Map<String, Object> authnState,
+ @Nullable Map<String, List<String>> attributes
+ ) throws IOException {
  final Request request = new Request("POST", "/_idp/saml/init");
- if (authnState != null && authnState.isEmpty() == false) {
- request.setJsonEntity(Strings.format("""
- {"entity_id":"%s", "acs":"%s","authn_state":%s}
- """, entityId, acs, Strings.toString(JsonXContent.contentBuilder().map(authnState))));
- } else {
- request.setJsonEntity(Strings.format("""
- {"entity_id":"%s", "acs":"%s"}
- """, entityId, acs));
+
+ XContentBuilder builder = JsonXContent.contentBuilder();
+ builder.startObject();
+ builder.field("entity_id", entityId);
+ builder.field("acs", acs);
+
+ if (authnState != null) {
+ builder.field("authn_state");
+ builder.map(authnState);
+ }
+
+ if (attributes != null) {
+ builder.field("attributes");
+ builder.map(attributes);
  }
+
+ builder.endObject();
+ String jsonEntity = Strings.toString(builder);
+
+ request.setJsonEntity(jsonEntity);
  request.setOptions(
  RequestOptions.DEFAULT.toBuilder()
  .addHeader("es-secondary-authorization", basicAuthHeaderValue("idp_user", new SecureString("idp-password".toCharArray())))

x-pack/plugin/identity-provider/src/main/java/org/elasticsearch/xpack/idp/action/SamlInitiateSingleSignOnRequest.java

@@ -6,12 +6,14 @@
  */
 package org.elasticsearch.xpack.idp.action;
 
+import org.elasticsearch.TransportVersions;
 import org.elasticsearch.action.ActionRequestValidationException;
 import org.elasticsearch.action.LegacyActionRequest;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.common.io.stream.StreamOutput;
 import org.elasticsearch.xpack.idp.saml.support.SamlAuthenticationState;
+import org.elasticsearch.xpack.idp.saml.support.SamlInitiateSingleSignOnAttributes;
 
 import java.io.IOException;
 
@@ -22,12 +24,16 @@ public class SamlInitiateSingleSignOnRequest extends LegacyActionRequest {
  private String spEntityId;
  private String assertionConsumerService;
  private SamlAuthenticationState samlAuthenticationState;
+ private SamlInitiateSingleSignOnAttributes attributes;
 
  public SamlInitiateSingleSignOnRequest(StreamInput in) throws IOException {
  super(in);
  spEntityId = in.readString();
  assertionConsumerService = in.readString();
  samlAuthenticationState = in.readOptionalWriteable(SamlAuthenticationState::new);
+ if (in.getTransportVersion().onOrAfter(TransportVersions.IDP_CUSTOM_SAML_ATTRIBUTES_ADDED_8_19)) {
+ attributes = in.readOptionalWriteable(SamlInitiateSingleSignOnAttributes::new);
+ }
  }
 
  public SamlInitiateSingleSignOnRequest() {}
@@ -41,6 +47,17 @@ public ActionRequestValidationException validate() {
  if (Strings.isNullOrEmpty(assertionConsumerService)) {
  validationException = addValidationError("acs is missing", validationException);
  }
+
+ // Validate attributes if present
+ if (attributes != null) {
+ ActionRequestValidationException attributesValidationException = attributes.validate();
+ if (attributesValidationException != null) {
+ for (String error : attributesValidationException.validationErrors()) {
+ validationException = addValidationError(error, validationException);
+ }
+ }
+ }
+
  return validationException;
  }
 
@@ -68,17 +85,38 @@ public void setSamlAuthenticationState(SamlAuthenticationState samlAuthenticatio
  this.samlAuthenticationState = samlAuthenticationState;
  }
 
+ public SamlInitiateSingleSignOnAttributes getAttributes() {
+ return attributes;
+ }
+
+ public void setAttributes(SamlInitiateSingleSignOnAttributes attributes) {
+ this.attributes = attributes;
+ }
+
  @Override
  public void writeTo(StreamOutput out) throws IOException {
  super.writeTo(out);
  out.writeString(spEntityId);
  out.writeString(assertionConsumerService);
  out.writeOptionalWriteable(samlAuthenticationState);
+ if (out.getTransportVersion().onOrAfter(TransportVersions.IDP_CUSTOM_SAML_ATTRIBUTES_ADDED_8_19)) {
+ out.writeOptionalWriteable(attributes);
+ }
  }
 
  @Override
  public String toString() {
- return getClass().getSimpleName() + "{spEntityId='" + spEntityId + "', acs='" + assertionConsumerService + "'}";
+ return getClass().getSimpleName()
+ + "{"
+ + "spEntityId='"
+ + spEntityId
+ + "', "
+ + "acs='"
+ + assertionConsumerService
+ + "', "
+ + "attributes='"
+ + attributes
+ + "'}";
  }
 
 }

x-pack/plugin/identity-provider/src/main/java/org/elasticsearch/xpack/idp/action/TransportSamlInitiateSingleSignOnAction.java

@@ -139,7 +139,7 @@ protected void doExecute(
  identityProvider
  );
  try {
- final Response response = builder.build(user, authenticationState);
+ final Response response = builder.build(user, authenticationState, request.getAttributes());
  listener.onResponse(
  new SamlInitiateSingleSignOnResponse(
  user.getServiceProvider().getEntityId(),