Download as PDF, PPTX
![HASHICORP Consul Server 2/3 "dns_config": { "allow_stale": true, "max_stale": "10080m", "node_ttl": "60s", "service_ttl": { "*": "5s", "stable-service": "86400s" } },
"encrypt": "[ random mime encoded data ]", "log_level": "debug", "ports": { "https": -1 }, "server": true, "unix_sockets": { "mode": "0700" } }](https://image.slidesharecdn.com/geo-pg-160822085214/75/PostgreSQL-High-Availability-and-Geographic-Locality-using-consul-26-2048.jpg)
![HASHICORP Consul Cluster % consul info agent: check_monitors = 0 check_ttls = 0 checks = 0 services = 1 build: prerelease = dev revision = 'fa26d5f version = 0.7.0 consul: bootstrap = false known_datacenters = 2 leader = false leader_addr = 172.16.139.139:8300 server = true [snip]](https://image.slidesharecdn.com/geo-pg-160822085214/75/PostgreSQL-High-Availability-and-Geographic-Locality-using-consul-29-2048.jpg)
![HASHICORP Consul Cluster % consul info [snip] raft: applied_index = 103339 commit_index = 103339 fsm_pending = 0 last_contact = 82.95803ms last_log_index = 103339 last_log_term = 50663 last_snapshot_index = 98437 last_snapshot_term = 2228 num_peers = 2 raft_peers = 172.16.139.139:8300,172.16.139.138:8300,172.16.139.140:8300 state = Follower term = 50663 [snip]](https://image.slidesharecdn.com/geo-pg-160822085214/75/PostgreSQL-High-Availability-and-Geographic-Locality-using-consul-30-2048.jpg)
![HASHICORP PostgreSQL Service % hostname pg002 % cat config.d/pg-db.json { "service": { "name": "pg-db", "tags": ["follower"], "port": 5432, "checks": [{ "id": "pg-alive", "notes": "Make sure connect and queries work", "script": "/usr/local/bin/check_postgresql", "interval": "10s" }] } }](https://image.slidesharecdn.com/geo-pg-160822085214/75/PostgreSQL-High-Availability-and-Geographic-Locality-using-consul-34-2048.jpg)
![Terminal HASHICORP $ curl -X PUT -d 'bar' http://localhost:8500/v1/kv/foo true $ curl http://localhost:8500/v1/kv/foo [ { "CreateIndex": 100, "ModifyIndex": 200, "Key": "foo", "Flags": 0, "Value": "YmFy" } ] % echo -n 'bar' | base64 YmFy % echo -n 'YmFy' | base64 -d ; echo bar](https://image.slidesharecdn.com/geo-pg-160822085214/75/PostgreSQL-High-Availability-and-Geographic-Locality-using-consul-57-2048.jpg)
![pg-db with Failover $ curl -X POST -d '{ "Name": "geo-pg-db—follower", "Service": { "Service": "pg-db", "Failover": { "NearestN": 3 }, "Tags": ["follower"] } }’ localhost:8500/v1/query geo-pgdb—follower.query.consul](https://image.slidesharecdn.com/geo-pg-160822085214/75/PostgreSQL-High-Availability-and-Geographic-Locality-using-consul-63-2048.jpg)
![PostgreSQL Template $ curl -X POST -d '{ "Name": "geo-db", "Template": { "Type": "name_prefix_match", "Regexp": "^geo-db-(.*?)-([^-]+?)$" }, "Service": { "Service": "pg—${match(1)}", "Failover": { "NearestN": 3, "Datacenters": ["dc1", "dc2"] }, "OnlyPassing": true, "Tags": ["${match(2)}"] } }' localhost:8500/v1/query geo-db-customer-leader.query.consul geo-db-customer-follower.query.consul geo-db-billing-follower.query.consul leader.pg-customer.service.consul follower.pg-customer.service.consul follower.pg-billing.service.consul](https://image.slidesharecdn.com/geo-pg-160822085214/75/PostgreSQL-High-Availability-and-Geographic-Locality-using-consul-64-2048.jpg)
Uploaded bySean Chittenden
PostgreSQL High-Availability and Geographic Locality using consul
This document provides an overview of HashiCorp's Consul, focusing on its role in enabling high availability and geographic locality for PostgreSQL databases. It covers core functionalities such as service discovery, health checks, and distributed key/value storage, along with detailed examples of configuration and health checking mechanisms. Furthermore, it addresses the architecture of Consul across single and multiple datacenters, highlighting its capabilities in efficient workload management and failover strategies.
PostgreSQL High-Availability and Geographic Locality using consul
- 1. PostgreSQL High-Availability and GeographicLocality using consul
- 2.
- 3.
- 4.
- 5.
- 6. HASHICORP Key Value Store HTTPAPI Host & Service Level Health Checks Datacenter Aware Consul solves four central challenges with SOA Service Discovery HTTP + DNS
- 7.
- 8. HASHICORP Overview 1. Introduction toConsul 2. Review of Consul a. Architecture b. Agent Functionality c. Agent Configuration d. Features 3. Further Reading
- 9.
- 10. HASHICORP Consul powers runtimeorchestration
- 11. CONSUL HASHICORP 1. Service discovery 2.Service registry 3. Key/value store 4. Health checks
- 12. HASHICORP Glossary Agent - Long-runningdaemon on every member of the Consul cluster. The agent is able to run in either client or server mode. Client - Agent that forwards all RPCs to a server and participates in the LAN gossip pool. Server - Agent that maintains cluster state, responds to RPC queries, exchanges WAN gossip with other datacenters, and forwards queries to leaders of remote datacenters. Consensus - Agreement upon the elected leader
- 13. HASHICORP Glossary Gossip - Randomnode-to-node communication primarily over UDP that provides membership, failure detection, and event broadcast information to the cluster. Built on Serf. Consul has both LAN and WAN Gossip. Datacenter - Networking environment that is private, low latency, and high bandwidth. A Consul cluster is run per datacenter, so its important to have low latency for the gossip protocol.
- 14. HASHICORP Consul vs. OtherSoftware - Opinionated framework for service discovery using DNS or HTTP - Scalable gossip system that links server nodes and clients - Distributed health checking with edge triggered updates - Globally aware with multi-datacenter support - Operationally simple - Incorporation into the HashiCorp ecosystem
- 15.
- 16. HASHICORP Single Datacenter CLIENT CLIENTCLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION RPC RPC LAN GOSSIP
- 17. HASHICORP Multi-Datacenter CLIENT CLIENT CLIENTCLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION RPC RPC LAN GOSSIP SERVERSERVER SERVER REPLICATION REPLICATION WAN GOSSIP
- 18.
- 19. HASHICORP TCP and UDPPorts Client HTTP RPC TCP/8500 DNS TCP/8600 UDP/8600 LAN Gossip TCP/8301 UDP/8301 LAN Gossip TCP/8301 UDP/8301 RPC TCP/8400 RPC TCP/8400 WAN Gossip TCP/8301 UDP/8301 Clients consul1.dc1 Client RPC (HTTP) DNS TCP/8600 UDP/8600 Server RPC TCP/8300 consulN.dc2 consul2.dc1
- 20. HASHICORP Agent functionality (clientor server) - RPC, HTTP, DNS APIs - Health Checks - Event Execution - Gossip Participation - Membership - Failure detection
- 21. HASHICORP Agent functionality (server) -State replication - Query Handling - Leader election - WAN Gossip
- 22.
- 23. HASHICORP DNS Failover • Worksacross L3 boundaries in LAN environments • Works across L3 boundaries in WAN environments • Small TTLs • Workload Distribution • Clients cache DNS data • Not subject to spanning-tree • Requires TCP connections be reset on failover • Clients can cache stale DNS data Pro Con
- 24.
- 25. HASHICORP consul Server 1/3 %cat config.json { "acl_datacenter": "lab1", "acl_default_policy": "deny", "acl_master_token": "rootToken", "addresses": { "dns": "0.0.0.0", "http": "unix:///tmp/.consul.http.sock", "https": "0.0.0.0", "rpc": "unix:///tmp/.consul.rpc.sock" }, "bootstrap_expect": 3, "datacenter": "lab1", "data_dir": "./svc/data", "disable_remote_exec": true,
- 26. HASHICORP Consul Server 2/3 "dns_config":{ "allow_stale": true, "max_stale": "10080m", "node_ttl": "60s", "service_ttl": { "*": "5s", "stable-service": "86400s" } }, "encrypt": "[ random mime encoded data ]", "log_level": "debug", "ports": { "https": -1 }, "server": true, "unix_sockets": { "mode": "0700" } }
- 27. HASHICORP Consul Server 3/3 %cat svc/run #!/bin/sh -- set -e exec 2>&1 exec /usr/bin/env -i ./bin/consul agent -config-file=./config.json -config-dir=./conf.d/ % cat svc/log/run #!/bin/sh — set -e set 2>&1 exec chpst -u _log:_log svlogd ./main
- 28. HASHICORP Consul Cluster % consulmembers Node Address Status Type Build Protocol DC vm1 172.16.139.140:8301 alive server 0.7.0dev 2 lab1 % consul join 172.16.139.139 172.16.139.138 Successfully joined cluster by contacting 2 nodes. % consul members Node Address Status Type Build Protocol DC vm1 172.16.139.140:8301 alive server 0.7.0dev 2 lab1 vm2 172.16.139.138:8301 alive server 0.7.0dev 2 lab1 vm3 172.16.139.139:8301 alive server 0.7.0dev 2 lab1
- 29. HASHICORP Consul Cluster % consulinfo agent: check_monitors = 0 check_ttls = 0 checks = 0 services = 1 build: prerelease = dev revision = 'fa26d5f version = 0.7.0 consul: bootstrap = false known_datacenters = 2 leader = false leader_addr = 172.16.139.139:8300 server = true [snip]
- 30. HASHICORP Consul Cluster % consulinfo [snip] raft: applied_index = 103339 commit_index = 103339 fsm_pending = 0 last_contact = 82.95803ms last_log_index = 103339 last_log_term = 50663 last_snapshot_index = 98437 last_snapshot_term = 2228 num_peers = 2 raft_peers = 172.16.139.139:8300,172.16.139.138:8300,172.16.139.140:8300 state = Follower term = 50663 [snip]
- 31. HASHICORP dnsmasq Config % cat/usr/local/etc/dnsmasq.conf local-service port=53 server=/consul/127.0.0.1#8600 rev-server=172.16.0.0/12,127.0.0.1#8600 server=208.67.222.222 server=208.67.220.220 cache-size=65536 % cat /etc/resov.conf search localdomain nameserver 127.0.0.1
- 32.
- 33. HASHICORP - Nodes, Services,Checks - Simple registration (JSON) - DNS Interface - HTTP API Service Discovery
- 34. HASHICORP PostgreSQL Service % hostname pg002 %cat config.d/pg-db.json { "service": { "name": "pg-db", "tags": ["follower"], "port": 5432, "checks": [{ "id": "pg-alive", "notes": "Make sure connect and queries work", "script": "/usr/local/bin/check_postgresql", "interval": "10s" }] } }
- 35.
- 36. Terminal HASHICORP $ dig follower.pg-db.service.consul ;<<>> DiG 9.8.3-P1 <<>> follower.pg-db.service.consul ; (3 servers found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 946 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;follower.pg-db.service.consul. IN A ;; ANSWER SECTION: follower.pg-db.service.consul. 0 IN A 172.16.139.141
- 37. Terminal HASHICORP $ dig follower.pg-db.service.consulSRV ; <<>> DiG 9.8.3-P1 <<>> follower.pg-db.service.consul SRV ; (3 servers found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 480 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;follower.pg-db.service.consul. IN SRV ;; ANSWER SECTION: follower.pg-db.service.consul. 0 IN SRV 1 1 5432
- 38. HASHICORP DNS Interface - ZeroTouch - Randomized Round-Robin DNS - Filters on Health Checks
- 39. HASHICORP HTTP API - HTTPAPI - Custom Integrations
- 40. HASHICORP Host & ServiceLevel Health Checks
- 41. HASHICORP What is ahealth check? 0PASSING 1WARNING _ _FAILING Any command that returns an exit code
- 42. HASHICORP Health Checks &Monitoring - Nagios-compatible - Scalable - Actionable - Edge Triggered
- 43. Text Editor HASHICORP % catconf.d/mem-check.json { "check": { "id": "mem-util", "name": "Memory utilization", "script": "/usr/local/bin/mem_check.sh", "interval": "10s" } } Creating a check Use a custom script
- 44. Text Editor HASHICORP % catconf.d/http-check.json { "check": { "id": "api", "name": "HTTP API on port 4455", "http": "http://localhost:4455/_health", "interval": "10s", "timeout": "1s" } } Creating a check Use a built-in check type
- 45. HASHICORP Traditional Health Checking(pull) HEALTH CHECKING SERVICE DB 1 DB 2 DB N "Are you healthy?"
- 46. HASHICORP Traditional Health Checking(pull) HEALTH CHECKING SERVICE DB 1 DB 2 DB N "Are you healthy?" "Yessir!"
- 47. HASHICORP Traditional Health Checking(pull) HEALTH CHECKING SERVICE DB 1 DB 2 DB N "Are you healthy?" "What about you?" "Yessir!"
- 48. HASHICORP Traditional Health Checking(pull) HEALTH CHECKING SERVICE DB 1 DB 2 DB N "Are you healthy?" "What about you?" "Yessir!" "Nah"
- 49. HASHICORP Traditional Health Checking(pull) DB 1 DB 2 DB N HEALTH CHECKING SERVICE "Are you healthy?" "What about you?" "Yessir!" "Nah"
- 50. HASHICORP Traditional Health Checking(pull) DB 1 DB 2 DB N HEALTH CHECKING SERVICE 1,000'S OF REQUESTS
- 51. HASHICORP Consul Health Checking(push) CONSUL DB 1 DB 2 DB N My status has changed
- 52. HASHICORP Consul Health Checking(push) CONSUL DB 1 DB 2 DB N 10'S OF REQUESTS
- 53. HASHICORP Liveness - No Heartbeats -Gossip-based Failure Detector built on Serf - Constant Load
- 54.
- 55.
- 56. Terminal HASHICORP $ curl -XPUT -d 'bar' http://localhost:8500/v1/kv/foo true
- 57. Terminal HASHICORP $ curl -XPUT -d 'bar' http://localhost:8500/v1/kv/foo true $ curl http://localhost:8500/v1/kv/foo [ { "CreateIndex": 100, "ModifyIndex": 200, "Key": "foo", "Flags": 0, "Value": "YmFy" } ] % echo -n 'bar' | base64 YmFy % echo -n 'YmFy' | base64 -d ; echo bar
- 58. Terminal HASHICORP % cat <<EOF> acl.anonymous.json { "ID": "anonymous", "Name": "Anonymous Token", "Type": "client", "Rules": "# Default all keys to read-only key "" { policy = "read" } # Default all services to read-only service "" { policy = "read" } # Allow hearing any user event by default. event "" { policy = "read" }
- 59. Terminal HASHICORP # Default preparedqueries to read-only. query "" { policy = "read" } # Read-only mode for the encryption keyring by default (list only) keyring = "read"" } EOF % curl -v -X PUT -d @acl.anonymous.json --unix-socket / tmp/.consul.http.sock 'http://consul/v1/acl/update? token=rootToken'
- 60.
- 61. Use Case • Multipleinstances of a given service exist in multiple datacenters • Clients can talk to any of them, and always prefer the instances with lowest latency • Policies can change, desire to not have the clients know the details of how to locate a healthy service
- 62. Prepared Queries • Newquery namespace, similar to services • Register queries to answer for parts of this namespace • Clients use APIs, or “.query.consul” DNS lookups to run queries • Magic happens :-)
- 63. pg-db with Failover $curl -X POST -d '{ "Name": "geo-pg-db—follower", "Service": { "Service": "pg-db", "Failover": { "NearestN": 3 }, "Tags": ["follower"] } }’ localhost:8500/v1/query geo-pgdb—follower.query.consul
- 64. PostgreSQL Template $ curl-X POST -d '{ "Name": "geo-db", "Template": { "Type": "name_prefix_match", "Regexp": "^geo-db-(.*?)-([^-]+?)$" }, "Service": { "Service": "pg—${match(1)}", "Failover": { "NearestN": 3, "Datacenters": ["dc1", "dc2"] }, "OnlyPassing": true, "Tags": ["${match(2)}"] } }' localhost:8500/v1/query geo-db-customer-leader.query.consul geo-db-customer-follower.query.consul geo-db-billing-follower.query.consul leader.pg-customer.service.consul follower.pg-customer.service.consul follower.pg-billing.service.consul
- 65. Catch All Template $curl -X POST -d '{ "Name": "", "Template": { "Type": "name_prefix_match" }, "Service": { "Service": "${name.full}", "Failover": { "NearestN": 3 } } }' localhost:8500/v1/query *.query.consul With a single query template, all services can fail over to the nearest healthy service in a different datacenter!
- 66. Under the Hood:Network Tomography • Rides on pings that are part of LAN and WAN gossip • Models networking round trip time using simple physics simulation with masses and springs • Develops a set of “network coordinates” for round trip time estimation with a simple calculation
- 67. Under the Hood:Network Tomography
- 68.
- 69. HASHICORP Key Value Store HTTPAPI Host & Service Level Health Checks Datacenter Aware Consul solves four central challenges with SOA Service Discovery HTTP + DNS
- 70. HASHICORP Further reading - Consulvs. Other Software: consul.io/intro/vs/index.html - Consul Agent: consul.io/docs/agent/basics.html - Consul Commands: consul.io/docs/commands/index.html - Consul Internals: consul.io/docs/internals/index.html
- 71.
- 72.