© Hitachi, Ltd. 2021. All rights reserved. Lightweight Zero-trust Network Implementation and Transition with Keycloak and NGINX Open Source Summit Japan 2021 Hitachi, Ltd. OSS Solution Center Yoshiyuki Tabata
1 © Hitachi, Ltd. 2021. All rights reserved. About the speaker • Consultant of API system  For example, building a high-security banking API system. • Contributor of 3scale (OSS of API Management)  Developed features around security and access control.  “Edge Limiting”, “RBAC”, “OAuth MTLS”, … • Contributor of Keycloak (OSS of Identity Management)  Developed features required for API use cases based on OAuth/OIDC.  “Token Revocation”, “Refresh token setting per client”, … Yoshiyuki Tabata  Software Engineer  OSS Solution Center, Hitachi, Ltd.  GitHub: @y-tabata
2 © Hitachi, Ltd. 2021. All rights reserved. Session Overview (1/3) API Gateway Authorization Server Client App Resource Server Resource Server - In the traditional API system, the security boundary between the public network and the private network is clear, so we only needed to focus on it. Security Boundary Expose API! Call API Delegate Authentication and Authorization Build API system by following OAuth 2.0 the de facto standard of API security private network public network
3 © Hitachi, Ltd. 2021. All rights reserved. - In the world of microservice, various services expose their APIs, it's much difficult to define a security boundary between the public network and the private network. Session Overview (2/3) Authorization Server Client App Call API Delegate Authentication and Authorization Expose API! Expose API! Expose API! Expose API! Expose API! Expose API! Expose API! Call API Call API Call API Call API Call API Call API Call API Build API system by following OAuth 2.0 the de facto standard of API security
4 © Hitachi, Ltd. 2021. All rights reserved. zero-trust network - So we need to consider introducing "zero-trust network" to secure each service independently (per service/pod). Session Overview (3/3) Client App Call API Call API Call API Call API Call API Call API security boundary security boundary security boundary security boundary security boundary security boundary security boundary security boundary We propose Lightweight zero-trust network implementation typical way to achieve zero-trust network is with service mesh, but  it needs rich resources for its rich features  it affects the architecture a lot
© Hitachi, Ltd. 2021. All rights reserved. Contents 5 1. What is zero-trust network? 2. How to achieve the underlying technology behind zero-trust network 3. Transition from traditional security boundary definition to per service/pod definition 4. Additional topic 1: How to achieve east-west traffic 5. Additional topic 2: How to resolve chokepoint issue of Policy Decision Point
© Hitachi, Ltd. 2021. All rights reserved. Contents 6 1. What is zero-trust network? 2. How to achieve the underlying technology behind zero-trust network 3. Transition from traditional security boundary definition to per service/pod definition 4. Additional topic 1: How to achieve east-west traffic 5. Additional topic 2: How to resolve chokepoint issue of Policy Decision Point
7 © Hitachi, Ltd. 2021. All rights reserved. - To treat the system internal network the same as the public network. - The underlying technology behind zero-trust network is  JWT validation  mutual TLS What is zero-trust network? - Underlying technology Source: Istio (https://istio.io/latest/docs/concepts/security/) Istio only requires mTLS in the system internal network, but JWT validation is said to be also needed among API security professionals. Require mTLS and JWT validation in the public network.
8 © Hitachi, Ltd. 2021. All rights reserved. - In the context of OAuth 2.0, what the JWT (presented during an API call) means is OAuth 2.0 Access Token. - Access token is issued by an authorization server as representing an authorization. Underlying technology 1 - JWT validation API Gateway Authorization Server Client App 3. Call API w/ Access Token (JWT) 2. Issue Access Token after Authentication and Authorization Typical OAuth 2.0 authorization code flow 1. Delegate Authentication and Authorization - Generally, validate JWT with authorization server, i.e. following Token Introspection: RFC 7662. - What validate:  Signature: check the JWT is not tampered  Expiry: check the JWT is not expired  Scope: check the client was authorized to call the API  Audience: check the resource server (API GW) is intended to return the response Token Introspection
9 © Hitachi, Ltd. 2021. All rights reserved. during TLS handshake - As the same as authenticating the server using a server certificate, authenticate the client using a client certificate. Underlying technology 2 - mutual TLS API Gateway Client App presenting the client certificate presenting the server certificate verify client certificate using trusted CA certificates
10 © Hitachi, Ltd. 2021. All rights reserved. - In the context of OAuth 2.0, OAuth 2.0 Mutual-TLS Client Certificate-Bound Access Tokens (hereinafter OAuth MTLS) is defined in RFC 8705. - OAuth MTLS prevents access token theft. Underlying technology 2’ - OAuth MTLS (RFC 8705) API Gateway Client App - verify client certificate using trusted CA certificates - verify the hash of client certificate equals one in the Access Token Authorization Server 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake)
11 © Hitachi, Ltd. 2021. All rights reserved. - Extending token introspection, resource server can delegate client certificate check to authorization server. Underlying technology 2’ - OAuth MTLS (RFC 8705) API Gateway Client App delegate client certificate check to authorization server Authorization Server 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake) mTLS OAuth MTLS manage trusted CA certificates Both authorization server and resource server (API GW) Authorization server only prevents access token theft No Yes If you publish your API to a large number of unspecified clients on the public network, OAuth MTLS may suitable.
© Hitachi, Ltd. 2021. All rights reserved. Contents 12 1. What is zero-trust network? 2. How to achieve the underlying technology behind zero-trust network 3. Transition from traditional security boundary definition to per service/pod definition 4. Additional topic 1: How to achieve east-west traffic 5. Additional topic 2: How to resolve chokepoint issue of Policy Decision Point
13 © Hitachi, Ltd. 2021. All rights reserved. - We use Keycloak as Authorization Server, NGINX as API Gateway. What OSS do we use? NGINX (API Gateway) Keycloak (Authorization Server) Client App 3. Call API w/ Access Token (JWT) 2. Issue Access Token after Authentication and Authorization 1. Delegate Authentication and Authorization Token Introspection
14 © Hitachi, Ltd. 2021. All rights reserved. - Keycloak is an identity management OSS. - Keycloak provides OAuth 2.0 authorization server features. What is Keycloak? Major features  Supports OAuth 2.0 / OpenID Connect and SAML  Can be linked with LDAP and Active Directory  Also supports social login with such as GitHub ID management and authentication social login (Identity Brokering) ID federation corresponding to major standards (Including OAuth 2.0 authorization server) Keycloak LDAP Active Directory RDB OpenID Connect SAML GitHub Twitter Facebook
15 © Hitachi, Ltd. 2021. All rights reserved. - Keycloak supports Token Introspection as a standard feature. - NGINX supports ngx_http_auth_request_module module which implements client authorization based on the result of a subrequest. If the subrequest returns a 2xx response code, access is allowed. If it returns 401 or 403, access is denied with the corresponding error code. How to achieve JWT validation NGINX Keycloak Client App 3. Call API w/ Access Token (JWT) 2. Issue Access Token after Authentication and Authorization 1. Delegate Authentication and Authorization NGINX delegates JWT validation to Keycloak  send introspection request to Keycloak's token introspection endpoint  proxy to resource server if access token is active  deny access if access token is not active Token Introspection
16 © Hitachi, Ltd. 2021. All rights reserved. - NGINX supports ssl_verify_client syntax which enables verification of client certificates. Trusted CA certificates are specified in ssl_client_certificate syntax. How to achieve mTLS during TLS handshake NGINX Client App presenting the client certificate presenting the server certificate verify client certificate using trusted CA certificates
17 © Hitachi, Ltd. 2021. All rights reserved. - NGINX supports optional_no_ca parameter in ssl_verify_client syntax, which requests client certificate but does not require it to be signed by a trusted CA certificate. This is intended for use in cases when a service that is external to NGINX performs the actual certificate verification. - $ssl_client_escaped_cert variable returns client certificate in the PEM format for an established SSL connection. How to achieve OAuth MTLS NGINX Client App send access token and client certificate to token introspection endpoint Keycloak 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake) Token Introspection by using SPI, we can extend token introspection to check client certificate at Keycloak
© Hitachi, Ltd. 2021. All rights reserved. Contents 18 1. What is zero-trust network? 2. How to achieve the underlying technology behind zero-trust network 3. Transition from traditional security boundary definition to per service/pod definition 4. Additional topic 1: How to achieve east-west traffic 5. Additional topic 2: How to resolve chokepoint issue of Policy Decision Point
19 © Hitachi, Ltd. 2021. All rights reserved. - So far, we explained how to use Keycloak and NGINX to achieve the underlying technology behind zero-trust network. - From here, we make the grain size of security boundary finer. Security boundary transition scenario NGINX Keycloak Client App Resource Server Resource Server Security Boundary Expose API! Call API Delegate Authentication and Authorization private network public network
20 © Hitachi, Ltd. 2021. All rights reserved. - First of all, lifting the existing system to the world of containers. - Compared to cloud services, in NGINX, this API Gateway lifting is very easy because we can reuse the NGINX conf file, for example, using server-snippet and location-snippet annotations, or using custom resources named VirtualServer. Step 1: Change API Gateway to NGINX Ingress Controller NGINX Ingress Controller Keycloak Client App Resource Server Resource Server Security Boundary Call API Delegate Authentication and Authorization private network public network
21 © Hitachi, Ltd. 2021. All rights reserved. Security Boundary Security Boundary - This step is useful if the resource server's service takes a little time to be modified. - Again we can reuse the NGINX conf file. - To pass through client certificates at NGINX Ingress Controller, we use custom resources named TransportServer to configure stream context. Step 2: Shift security boundary to per service NGINX Ingress Controller Keycloak Client App Proxy Service Proxy Service Call API Delegate Authentication and Authorization Resource Server Resource Server
22 © Hitachi, Ltd. 2021. All rights reserved. Security Boundary Security Boundary - Achieve so-called sidecar. - Again we can reuse the NGINX conf file and we can reuse TransportServer setting too. - API call is first sent to proxy container, and if all checks are passed, proxy to resource server container. Step 3: Shift security boundary to per pod NGINX Ingress Controller Keycloak Client App Call API Delegate Authentication and Authorization Resource Server Resource Server container Proxy container
© Hitachi, Ltd. 2021. All rights reserved. Contents 23 1. What is zero-trust network? 2. How to achieve the underlying technology behind zero-trust network 3. Transition from traditional security boundary definition to per service/pod definition 4. Additional topic 1: How to achieve east-west traffic 5. Additional topic 2: How to resolve chokepoint issue of Policy Decision Point
24 © Hitachi, Ltd. 2021. All rights reserved. - So far, we targeted north-south traffic, i.e. traffic from client to resource server, from external to internal. How to achieve east-west traffic NGINX Ingress Controller Keycloak Client App Call API Delegate Authentication and Authorization Resource Server Resource Server container Proxy container
25 © Hitachi, Ltd. 2021. All rights reserved. - So far, we targeted north-south traffic, i.e. traffic from client to resource server, from external to internal. - How to achieve east-west traffic, i.e. traffic from internal to internal? - For example, when the resource server wants to get resources from other services. How to achieve east-west traffic NGINX Ingress Controller Keycloak Client App Call API Delegate Authentication and Authorization Another Service Resource Server container Proxy container ? Resource Server
26 © Hitachi, Ltd. 2021. All rights reserved. - So far, we targeted north-south traffic, i.e. traffic from client to resource server, from external to internal. - How to achieve east-west traffic, i.e. traffic from internal to internal? - For example, when the resource server wants to get resources from other services. - In this case, resource server container should send a request using again the proxy container. How to achieve east-west traffic NGINX Ingress Controller Keycloak Client App Call API Delegate Authentication and Authorization Another Service Resource Server container Proxy container Resource Server
27 © Hitachi, Ltd. 2021. All rights reserved. - To validate JWT, the proxy container must send JWT.  Option A: send the same access token which client sent  Option B: send the different access token getting from authorization server by following OAuth 2.0 Token Exchange: RFC 8693 - Both options, need to get user's consent. How to achieve JWT validation (east-west traffic) Proxy container Another Service Resource Server container Keycloak Option A: same Access Token Option B: different Access Token Token Exchange AT AT AT AT AT’ AT’
28 © Hitachi, Ltd. 2021. All rights reserved. - Audience check is very important for this case. - Without Audience check, user's resources may be provided to malicious service. How to achieve JWT validation (east-west traffic) Another Service Malicious Service Client App may redirect access token to other services without user's consent AT AT must validate Audience to check this service is intended to return the response
29 © Hitachi, Ltd. 2021. All rights reserved. - As same in the case of north-south traffic, there are 2 methods, mTLS and OAuth MTLS. - Different in the case of north-south traffic, the requesting parties are limited, so mTLS may be enough. (OAuth MTLS may be over-engineered.) - We can add client certificate and key with proxy_ssl_certificate syntax and proxy_ssl_certificate_key syntax. How to achieve mTLS (east-west traffic) during TLS handshake presenting the client certificate presenting the server certificate verify client certificate using trusted CA certificates Proxy container Another Service
© Hitachi, Ltd. 2021. All rights reserved. Contents 30 1. What is zero-trust network? 2. How to achieve the underlying technology behind zero-trust network 3. Transition from traditional security boundary definition to per service/pod definition 4. Additional topic 1: How to achieve east-west traffic 5. Additional topic 2: How to resolve chokepoint issue of Policy Decision Point
31 © Hitachi, Ltd. 2021. All rights reserved. - As the number of API calls increases, Keycloak may become a chokepoint of this architecture, because Keycloak is accessed every time API calls. - In Zero Trust Architecture, NIST SP 800-207, Keycloak is PE (Policy Engine), and NGINX is PEP (Policy Enforcement Point) + PA (Policy Administrator). - From here, we consider how to reduce the load of PE. Where is the chokepoint? Client App PEP + PA Keycloak 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake) Token Introspection PE NGINX
32 © Hitachi, Ltd. 2021. All rights reserved. - Considering using Open Policy Agent (OPA), the defacto standard OSS of PE (Policy Engine).  Option A: cache token introspection responses  Option B: make OPA act as PE and Keycloak act as PIP (Policy Information Point) How to reduce the load of PE (Keycloak) Client App Keycloak 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake) Token Introspection NGINX OPA +
33 © Hitachi, Ltd. 2021. All rights reserved. - Caching token introspection responses, we can reduce the access frequency to Keycloak. - Even if the access token is revoked at Keycloak, it's not revoked at resource server immediately, so the access token lifespan should be proper short value. Option A: Cache token introspection responses Client App Keycloak 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake) NGINX OPA 4. Delegate JWT validation MongoDB 5. Check cache 7. Save result 6. Token Introspection (if cache miss)
34 © Hitachi, Ltd. 2021. All rights reserved. - OPA container becomes PE completely, and Keycloak becomes PIP, only provides information for decisions to grant access. Option B: Make OPA act as PE and Keycloak act as PIP Client App Keycloak 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake) I. Notify resource (User, Client, Key, …) and operation (Create, Update, Delete, …) NGINX OPA 4. Delegate JWT validation MongoDB 5. Check data II. Save data Converter OPA by using SPI, we can extend event listener feature to notify operation to converter
35 © Hitachi, Ltd. 2021. All rights reserved. - Option B is an ideal implementation, but it is expensive to implement. - Option A, on the other hand, has the security concern that access token revocation cannot be synchronized immediately, although the implementation cost is minimal. - A hybrid proposal may be the most acceptable implementation.  caches the results of token introspection  synchronizes only token revocation notifications How to reduce the load of PE (Keycloak) Option A: cache token introspection responses Option B: make OPA act as PE and Keycloak act as PIP need to prepare Converter No Yes implement SPI No Yes synchronize access token revocation immediately No Yes
36 © Hitachi, Ltd. 2021. All rights reserved. Summary • The underlying technology behind zero-trust network is JWT validation and mTLS.  OAuth MTLS (RFC 8705) is forwards-compatible with mTLS. • By using Keycloak and NGINX, we can achieve the underlying technology, and transit from traditional security boundary definition to per service/pod definition smoothly.  Not only north-south traffic but also east-west traffic can be covered. • By using OPA, we can reduce the load of Keycloak (PE).
37 © Hitachi, Ltd. 2021. All rights reserved. Trademarks • OpenID is a trademark or registered trademark of OpenID Foundation in the United States and other countries. • GitHub is a trademark or registered trademark of GitHub, Inc. in the United States and other countries. • Facebook is a registered trademark of Facebook, Inc. • Twitter is a registered trademark of Twitter, Inc. • The Kubernetes logo is a registered trademark of The Linux Foundation. • Red Hat, and 3scale are registered trademarks of Red Hat, Inc. in the United States and other countries. • NGINX is a trademark of F5 Networks, Inc. in the U.S. and in certain other countries. • Istio is a trademark or registered trademark of Google LLC. • Other brand names and product names used in this material are trademarks, registered trademarks, or trade names of their respective holders.
Lightweight Zero-trust Network Implementation and Transition with Keycloak and NGINX

Lightweight Zero-trust Network Implementation and Transition with Keycloak and NGINX

  • 1.
    © Hitachi, Ltd.2021. All rights reserved. Lightweight Zero-trust Network Implementation and Transition with Keycloak and NGINX Open Source Summit Japan 2021 Hitachi, Ltd. OSS Solution Center Yoshiyuki Tabata
  • 2.
    1 © Hitachi, Ltd.2021. All rights reserved. About the speaker • Consultant of API system  For example, building a high-security banking API system. • Contributor of 3scale (OSS of API Management)  Developed features around security and access control.  “Edge Limiting”, “RBAC”, “OAuth MTLS”, … • Contributor of Keycloak (OSS of Identity Management)  Developed features required for API use cases based on OAuth/OIDC.  “Token Revocation”, “Refresh token setting per client”, … Yoshiyuki Tabata  Software Engineer  OSS Solution Center, Hitachi, Ltd.  GitHub: @y-tabata
  • 3.
    2 © Hitachi, Ltd.2021. All rights reserved. Session Overview (1/3) API Gateway Authorization Server Client App Resource Server Resource Server - In the traditional API system, the security boundary between the public network and the private network is clear, so we only needed to focus on it. Security Boundary Expose API! Call API Delegate Authentication and Authorization Build API system by following OAuth 2.0 the de facto standard of API security private network public network
  • 4.
    3 © Hitachi, Ltd.2021. All rights reserved. - In the world of microservice, various services expose their APIs, it's much difficult to define a security boundary between the public network and the private network. Session Overview (2/3) Authorization Server Client App Call API Delegate Authentication and Authorization Expose API! Expose API! Expose API! Expose API! Expose API! Expose API! Expose API! Call API Call API Call API Call API Call API Call API Call API Build API system by following OAuth 2.0 the de facto standard of API security
  • 5.
    4 © Hitachi, Ltd.2021. All rights reserved. zero-trust network - So we need to consider introducing "zero-trust network" to secure each service independently (per service/pod). Session Overview (3/3) Client App Call API Call API Call API Call API Call API Call API security boundary security boundary security boundary security boundary security boundary security boundary security boundary security boundary We propose Lightweight zero-trust network implementation typical way to achieve zero-trust network is with service mesh, but  it needs rich resources for its rich features  it affects the architecture a lot
  • 6.
    © Hitachi, Ltd.2021. All rights reserved. Contents 5 1. What is zero-trust network? 2. How to achieve the underlying technology behind zero-trust network 3. Transition from traditional security boundary definition to per service/pod definition 4. Additional topic 1: How to achieve east-west traffic 5. Additional topic 2: How to resolve chokepoint issue of Policy Decision Point
  • 7.
    © Hitachi, Ltd.2021. All rights reserved. Contents 6 1. What is zero-trust network? 2. How to achieve the underlying technology behind zero-trust network 3. Transition from traditional security boundary definition to per service/pod definition 4. Additional topic 1: How to achieve east-west traffic 5. Additional topic 2: How to resolve chokepoint issue of Policy Decision Point
  • 8.
    7 © Hitachi, Ltd.2021. All rights reserved. - To treat the system internal network the same as the public network. - The underlying technology behind zero-trust network is  JWT validation  mutual TLS What is zero-trust network? - Underlying technology Source: Istio (https://istio.io/latest/docs/concepts/security/) Istio only requires mTLS in the system internal network, but JWT validation is said to be also needed among API security professionals. Require mTLS and JWT validation in the public network.
  • 9.
    8 © Hitachi, Ltd.2021. All rights reserved. - In the context of OAuth 2.0, what the JWT (presented during an API call) means is OAuth 2.0 Access Token. - Access token is issued by an authorization server as representing an authorization. Underlying technology 1 - JWT validation API Gateway Authorization Server Client App 3. Call API w/ Access Token (JWT) 2. Issue Access Token after Authentication and Authorization Typical OAuth 2.0 authorization code flow 1. Delegate Authentication and Authorization - Generally, validate JWT with authorization server, i.e. following Token Introspection: RFC 7662. - What validate:  Signature: check the JWT is not tampered  Expiry: check the JWT is not expired  Scope: check the client was authorized to call the API  Audience: check the resource server (API GW) is intended to return the response Token Introspection
  • 10.
    9 © Hitachi, Ltd.2021. All rights reserved. during TLS handshake - As the same as authenticating the server using a server certificate, authenticate the client using a client certificate. Underlying technology 2 - mutual TLS API Gateway Client App presenting the client certificate presenting the server certificate verify client certificate using trusted CA certificates
  • 11.
    10 © Hitachi, Ltd.2021. All rights reserved. - In the context of OAuth 2.0, OAuth 2.0 Mutual-TLS Client Certificate-Bound Access Tokens (hereinafter OAuth MTLS) is defined in RFC 8705. - OAuth MTLS prevents access token theft. Underlying technology 2’ - OAuth MTLS (RFC 8705) API Gateway Client App - verify client certificate using trusted CA certificates - verify the hash of client certificate equals one in the Access Token Authorization Server 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake)
  • 12.
    11 © Hitachi, Ltd.2021. All rights reserved. - Extending token introspection, resource server can delegate client certificate check to authorization server. Underlying technology 2’ - OAuth MTLS (RFC 8705) API Gateway Client App delegate client certificate check to authorization server Authorization Server 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake) mTLS OAuth MTLS manage trusted CA certificates Both authorization server and resource server (API GW) Authorization server only prevents access token theft No Yes If you publish your API to a large number of unspecified clients on the public network, OAuth MTLS may suitable.
  • 13.
    © Hitachi, Ltd.2021. All rights reserved. Contents 12 1. What is zero-trust network? 2. How to achieve the underlying technology behind zero-trust network 3. Transition from traditional security boundary definition to per service/pod definition 4. Additional topic 1: How to achieve east-west traffic 5. Additional topic 2: How to resolve chokepoint issue of Policy Decision Point
  • 14.
    13 © Hitachi, Ltd.2021. All rights reserved. - We use Keycloak as Authorization Server, NGINX as API Gateway. What OSS do we use? NGINX (API Gateway) Keycloak (Authorization Server) Client App 3. Call API w/ Access Token (JWT) 2. Issue Access Token after Authentication and Authorization 1. Delegate Authentication and Authorization Token Introspection
  • 15.
    14 © Hitachi, Ltd.2021. All rights reserved. - Keycloak is an identity management OSS. - Keycloak provides OAuth 2.0 authorization server features. What is Keycloak? Major features  Supports OAuth 2.0 / OpenID Connect and SAML  Can be linked with LDAP and Active Directory  Also supports social login with such as GitHub ID management and authentication social login (Identity Brokering) ID federation corresponding to major standards (Including OAuth 2.0 authorization server) Keycloak LDAP Active Directory RDB OpenID Connect SAML GitHub Twitter Facebook
  • 16.
    15 © Hitachi, Ltd.2021. All rights reserved. - Keycloak supports Token Introspection as a standard feature. - NGINX supports ngx_http_auth_request_module module which implements client authorization based on the result of a subrequest. If the subrequest returns a 2xx response code, access is allowed. If it returns 401 or 403, access is denied with the corresponding error code. How to achieve JWT validation NGINX Keycloak Client App 3. Call API w/ Access Token (JWT) 2. Issue Access Token after Authentication and Authorization 1. Delegate Authentication and Authorization NGINX delegates JWT validation to Keycloak  send introspection request to Keycloak's token introspection endpoint  proxy to resource server if access token is active  deny access if access token is not active Token Introspection
  • 17.
    16 © Hitachi, Ltd.2021. All rights reserved. - NGINX supports ssl_verify_client syntax which enables verification of client certificates. Trusted CA certificates are specified in ssl_client_certificate syntax. How to achieve mTLS during TLS handshake NGINX Client App presenting the client certificate presenting the server certificate verify client certificate using trusted CA certificates
  • 18.
    17 © Hitachi, Ltd.2021. All rights reserved. - NGINX supports optional_no_ca parameter in ssl_verify_client syntax, which requests client certificate but does not require it to be signed by a trusted CA certificate. This is intended for use in cases when a service that is external to NGINX performs the actual certificate verification. - $ssl_client_escaped_cert variable returns client certificate in the PEM format for an established SSL connection. How to achieve OAuth MTLS NGINX Client App send access token and client certificate to token introspection endpoint Keycloak 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake) Token Introspection by using SPI, we can extend token introspection to check client certificate at Keycloak
  • 19.
    © Hitachi, Ltd.2021. All rights reserved. Contents 18 1. What is zero-trust network? 2. How to achieve the underlying technology behind zero-trust network 3. Transition from traditional security boundary definition to per service/pod definition 4. Additional topic 1: How to achieve east-west traffic 5. Additional topic 2: How to resolve chokepoint issue of Policy Decision Point
  • 20.
    19 © Hitachi, Ltd.2021. All rights reserved. - So far, we explained how to use Keycloak and NGINX to achieve the underlying technology behind zero-trust network. - From here, we make the grain size of security boundary finer. Security boundary transition scenario NGINX Keycloak Client App Resource Server Resource Server Security Boundary Expose API! Call API Delegate Authentication and Authorization private network public network
  • 21.
    20 © Hitachi, Ltd.2021. All rights reserved. - First of all, lifting the existing system to the world of containers. - Compared to cloud services, in NGINX, this API Gateway lifting is very easy because we can reuse the NGINX conf file, for example, using server-snippet and location-snippet annotations, or using custom resources named VirtualServer. Step 1: Change API Gateway to NGINX Ingress Controller NGINX Ingress Controller Keycloak Client App Resource Server Resource Server Security Boundary Call API Delegate Authentication and Authorization private network public network
  • 22.
    21 © Hitachi, Ltd.2021. All rights reserved. Security Boundary Security Boundary - This step is useful if the resource server's service takes a little time to be modified. - Again we can reuse the NGINX conf file. - To pass through client certificates at NGINX Ingress Controller, we use custom resources named TransportServer to configure stream context. Step 2: Shift security boundary to per service NGINX Ingress Controller Keycloak Client App Proxy Service Proxy Service Call API Delegate Authentication and Authorization Resource Server Resource Server
  • 23.
    22 © Hitachi, Ltd.2021. All rights reserved. Security Boundary Security Boundary - Achieve so-called sidecar. - Again we can reuse the NGINX conf file and we can reuse TransportServer setting too. - API call is first sent to proxy container, and if all checks are passed, proxy to resource server container. Step 3: Shift security boundary to per pod NGINX Ingress Controller Keycloak Client App Call API Delegate Authentication and Authorization Resource Server Resource Server container Proxy container
  • 24.
    © Hitachi, Ltd.2021. All rights reserved. Contents 23 1. What is zero-trust network? 2. How to achieve the underlying technology behind zero-trust network 3. Transition from traditional security boundary definition to per service/pod definition 4. Additional topic 1: How to achieve east-west traffic 5. Additional topic 2: How to resolve chokepoint issue of Policy Decision Point
  • 25.
    24 © Hitachi, Ltd.2021. All rights reserved. - So far, we targeted north-south traffic, i.e. traffic from client to resource server, from external to internal. How to achieve east-west traffic NGINX Ingress Controller Keycloak Client App Call API Delegate Authentication and Authorization Resource Server Resource Server container Proxy container
  • 26.
    25 © Hitachi, Ltd.2021. All rights reserved. - So far, we targeted north-south traffic, i.e. traffic from client to resource server, from external to internal. - How to achieve east-west traffic, i.e. traffic from internal to internal? - For example, when the resource server wants to get resources from other services. How to achieve east-west traffic NGINX Ingress Controller Keycloak Client App Call API Delegate Authentication and Authorization Another Service Resource Server container Proxy container ? Resource Server
  • 27.
    26 © Hitachi, Ltd.2021. All rights reserved. - So far, we targeted north-south traffic, i.e. traffic from client to resource server, from external to internal. - How to achieve east-west traffic, i.e. traffic from internal to internal? - For example, when the resource server wants to get resources from other services. - In this case, resource server container should send a request using again the proxy container. How to achieve east-west traffic NGINX Ingress Controller Keycloak Client App Call API Delegate Authentication and Authorization Another Service Resource Server container Proxy container Resource Server
  • 28.
    27 © Hitachi, Ltd.2021. All rights reserved. - To validate JWT, the proxy container must send JWT.  Option A: send the same access token which client sent  Option B: send the different access token getting from authorization server by following OAuth 2.0 Token Exchange: RFC 8693 - Both options, need to get user's consent. How to achieve JWT validation (east-west traffic) Proxy container Another Service Resource Server container Keycloak Option A: same Access Token Option B: different Access Token Token Exchange AT AT AT AT AT’ AT’
  • 29.
    28 © Hitachi, Ltd.2021. All rights reserved. - Audience check is very important for this case. - Without Audience check, user's resources may be provided to malicious service. How to achieve JWT validation (east-west traffic) Another Service Malicious Service Client App may redirect access token to other services without user's consent AT AT must validate Audience to check this service is intended to return the response
  • 30.
    29 © Hitachi, Ltd.2021. All rights reserved. - As same in the case of north-south traffic, there are 2 methods, mTLS and OAuth MTLS. - Different in the case of north-south traffic, the requesting parties are limited, so mTLS may be enough. (OAuth MTLS may be over-engineered.) - We can add client certificate and key with proxy_ssl_certificate syntax and proxy_ssl_certificate_key syntax. How to achieve mTLS (east-west traffic) during TLS handshake presenting the client certificate presenting the server certificate verify client certificate using trusted CA certificates Proxy container Another Service
  • 31.
    © Hitachi, Ltd.2021. All rights reserved. Contents 30 1. What is zero-trust network? 2. How to achieve the underlying technology behind zero-trust network 3. Transition from traditional security boundary definition to per service/pod definition 4. Additional topic 1: How to achieve east-west traffic 5. Additional topic 2: How to resolve chokepoint issue of Policy Decision Point
  • 32.
    31 © Hitachi, Ltd.2021. All rights reserved. - As the number of API calls increases, Keycloak may become a chokepoint of this architecture, because Keycloak is accessed every time API calls. - In Zero Trust Architecture, NIST SP 800-207, Keycloak is PE (Policy Engine), and NGINX is PEP (Policy Enforcement Point) + PA (Policy Administrator). - From here, we consider how to reduce the load of PE. Where is the chokepoint? Client App PEP + PA Keycloak 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake) Token Introspection PE NGINX
  • 33.
    32 © Hitachi, Ltd.2021. All rights reserved. - Considering using Open Policy Agent (OPA), the defacto standard OSS of PE (Policy Engine).  Option A: cache token introspection responses  Option B: make OPA act as PE and Keycloak act as PIP (Policy Information Point) How to reduce the load of PE (Keycloak) Client App Keycloak 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake) Token Introspection NGINX OPA +
  • 34.
    33 © Hitachi, Ltd.2021. All rights reserved. - Caching token introspection responses, we can reduce the access frequency to Keycloak. - Even if the access token is revoked at Keycloak, it's not revoked at resource server immediately, so the access token lifespan should be proper short value. Option A: Cache token introspection responses Client App Keycloak 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake) NGINX OPA 4. Delegate JWT validation MongoDB 5. Check cache 7. Save result 6. Token Introspection (if cache miss)
  • 35.
    34 © Hitachi, Ltd.2021. All rights reserved. - OPA container becomes PE completely, and Keycloak becomes PIP, only provides information for decisions to grant access. Option B: Make OPA act as PE and Keycloak act as PIP Client App Keycloak 3. Call API w/ Access Token (JWT) (present client certificate during TLS handshake) 2. Issue Access Token (Access Token includes hash of client certificate) 1. Delegate AuthN/AuthZ (present client certificate during TLS handshake) I. Notify resource (User, Client, Key, …) and operation (Create, Update, Delete, …) NGINX OPA 4. Delegate JWT validation MongoDB 5. Check data II. Save data Converter OPA by using SPI, we can extend event listener feature to notify operation to converter
  • 36.
    35 © Hitachi, Ltd.2021. All rights reserved. - Option B is an ideal implementation, but it is expensive to implement. - Option A, on the other hand, has the security concern that access token revocation cannot be synchronized immediately, although the implementation cost is minimal. - A hybrid proposal may be the most acceptable implementation.  caches the results of token introspection  synchronizes only token revocation notifications How to reduce the load of PE (Keycloak) Option A: cache token introspection responses Option B: make OPA act as PE and Keycloak act as PIP need to prepare Converter No Yes implement SPI No Yes synchronize access token revocation immediately No Yes
  • 37.
    36 © Hitachi, Ltd.2021. All rights reserved. Summary • The underlying technology behind zero-trust network is JWT validation and mTLS.  OAuth MTLS (RFC 8705) is forwards-compatible with mTLS. • By using Keycloak and NGINX, we can achieve the underlying technology, and transit from traditional security boundary definition to per service/pod definition smoothly.  Not only north-south traffic but also east-west traffic can be covered. • By using OPA, we can reduce the load of Keycloak (PE).
  • 38.
    37 © Hitachi, Ltd.2021. All rights reserved. Trademarks • OpenID is a trademark or registered trademark of OpenID Foundation in the United States and other countries. • GitHub is a trademark or registered trademark of GitHub, Inc. in the United States and other countries. • Facebook is a registered trademark of Facebook, Inc. • Twitter is a registered trademark of Twitter, Inc. • The Kubernetes logo is a registered trademark of The Linux Foundation. • Red Hat, and 3scale are registered trademarks of Red Hat, Inc. in the United States and other countries. • NGINX is a trademark of F5 Networks, Inc. in the U.S. and in certain other countries. • Istio is a trademark or registered trademark of Google LLC. • Other brand names and product names used in this material are trademarks, registered trademarks, or trade names of their respective holders.