Downloaded 18 times
![JSON Data Type Example { "firstName": "John", -- String Type "lastName": "Smith", -- String Type "isAlive": true, -- Boolean Type "age": 25, -- Number Type "height_cm": 167.6, -- Number Type "address": { -- Object Type "streetAddress": "21 2nd Street”, "city": "New York”, "state": "NY”, "postalCode": "10021-3100” } "phoneNumbers": [ -- Object Array { -- Object "type": "home”, "number": "212 555-1234” }, { "type": "office”, "number": "646 555-4567” } ], "children": [], "spouse": null -- Null }](https://image.slidesharecdn.com/pgconfnewjersey20180419-v01-180503195210/75/PgConf-2018-Postgres-in-a-World-of-DevOps-13-2048.jpg)
Uploaded byEDB
PgConf 2018 - Postgres in a World of DevOps
This document discusses how Postgres fits into a DevOps world. It notes that as companies become more software-focused, DevOps practices like continuous integration/delivery, microservices, and containers are on the rise. This means databases need to be developer-friendly, support versatile data models like JSON, integrate with other technologies, and allow for rapid deployment including on databases-as-a-service platforms. Postgres is well-suited to this new environment as an open-source, multi-platform database that can scale easily and works well with other data systems through foreign data wrappers. The role of the database administrator is also changing to focus more on strategic tasks like performance, security, and data management rather than system administration.
PgConf 2018 - Postgres in a World of DevOps
- 1. Postgres in aworld of DevOps MOVE FAST, OR … MARC LINSTER SVP, PRODUCT DEVELOPMENT AND SUPPORT
- 2. As every companybecomes a software company, DevOps is the new lean manufacturing. PaaS CICD Microservices Containers
- 3. What does it mean for the database? DEVELOPER FRIENDLY VERSATILE DATAMODEL POLYGLOT FRIENDLY RAPID DEPLOYMENT
- 4. What does it mean for the database? DEVELOPER FRIENDLY VERSATILE DATAMODEL POLYGLOT FRIENDLY RAPID DEPLOYMENT
- 5. Popularity trend drivenby cost and flexibility
- 6.
- 7. Industry recognition ofskill proficiency 7 Postgres Certification
- 8. Major DB DeveloperTooling 8
- 9. DATABASE WORLD ISCHANGING 9 PAST “the corporate standard” Company data centers Waterfall IT provisioning Develop –then- operate Manual tasks Routine tasks Database expert PRESENT Many new databases The cloud Agile Self-service provisioning DevOps Automation Hard problems Data Expert DBA and Developer are no longer living in separate worlds
- 10. What does it mean for the database? DEVELOPER FRIENDLY VERSATILE DATAMODEL POLYGLOT FRIENDLY RAPID DEPLOYMENT
- 11. Postgres: The most versatile DBMS Runs onall platforms Speaks every language Not only SQL Scales in both directions
- 12. JSON Data Types 1.Number: • Signed decimal number that may contain a fractional part and may use exponential notation. • No distinction between integer and floating-point 2. String • A sequence of zero or more Unicode characters. • Strings are delimited with double-quotation mark • Supports a backslash escaping syntax. 3. Boolean • Either of the values true or false. 4. Array • An ordered list of zero or more values, • Each values may be of any type. • Arrays use square bracket notation with elements being comma-separated. 5. Object • An unordered associative array (name/value pairs). • Objects are delimited with curly brackets • Commas to separate each pair • Each pair the colon ':' character separates the key or name from its value. • All keys must be strings and should be distinct from each other within that object. 6. null • An empty value, using the word null JSON is defined per RFC – 7159 For more detail please refer http://tools.ietf.org/html/rfc7159
- 13. JSON Data TypeExample { "firstName": "John", -- String Type "lastName": "Smith", -- String Type "isAlive": true, -- Boolean Type "age": 25, -- Number Type "height_cm": 167.6, -- Number Type "address": { -- Object Type "streetAddress": "21 2nd Street”, "city": "New York”, "state": "NY”, "postalCode": "10021-3100” } "phoneNumbers": [ -- Object Array { -- Object "type": "home”, "number": "212 555-1234” }, { "type": "office”, "number": "646 555-4567” } ], "children": [], "spouse": null -- Null }
- 14. No need forprogrammatic logic to combine SQL and NoSQL in the application. Postgres does it all. Why choose between NoSQL and Relational? SELECT DISTINCT product_type, data->>'brand' as Brand, data->>'available' as Availability FROM json_data JOIN products ON (products.product_type=json_data.data->>'name') WHERE json_data.data->>'available'=true; product_type | brand | availability ---------------------------+-----------+-------------- AC3 Phone | ACME | true ANSI SQL JSON
- 15. Why not justSQL? START SCHEMALESS • Leverage structure as it emerges • Support agile, iterative development • Create data models where they provide value LEVERAGE JSONB • Leverage JSONB for rapidly changing data models • Example: address data records • Conventional columns: First Name, Last Name • JSONB: Contact Information • Phone numbers (home, cell, car, weekend, boyfriend… ) • Email (work, private, spam …) AVOID PENALTY • Avoid the DDL penalty for adding columns
- 16. What does it mean for the database? DEVELOPER FRIENDLY VERSATILE DATAMODEL POLYGLOT FRIENDLY RAPID DEPLOYMENT
- 17. Postgres in thePolyglot Ecosystem Event Capture Legacy Systems Data Warehouse Big Data Open Source ODBMS
- 18. Foreign Data Wrappers– Making Polyglot Happen FDW IMPLEMENTS SQL/MED ("SQL MANAGEMENT OF EXTERNAL DATA") PostgreSQL 9.1 - read-only support PostgreSQL 9.3 – read/write support PostgreSQL 9.6 – pushdown joins, sorts, UPDATE, DELETE PostgreSQL 10 - aggregate pushdown FDW: Makes data on other servers (or services) look like tables in Postgres. Available for many data sources (MongoDB, MySQL, HDFS, Spark, …)
- 19. What does it mean for the database? DEVELOPER FRIENDLY VERSATILE DATAMODEL POLYGLOT FRIENDLY RAPID DEPLOYMENT
- 20. Rapid Deployment CLOUD • Public(AWS, Google, Azure, Alibaba Cloud, …) • Private (OpenStack, VCloud, Puppet/Chef on Virtualization, Pivotal Cloud Foundry, Container/Kubernetes/ OpenShift) DBaaS • Stop deploying databases • Deploy clusters w. HA, DR, self healing, scaling, etc. • provide services, not databases Micro Services • Large monolithic databases refactored into transaction sets • Applications get refactored into micro-services • Deployment models move from VM/Bare Metal to DBaaS and Containers
- 21. SUPPORTING DevOps WITH DBaaS Addressthe tension between developers and operations DEVELOPERS WANT | Agility | Speed to deploy | Flow through tool chain OPERATIONS WANTS | Visibility | Control – budgeted resources – cost –data models | Efficient use of resources
- 22. Micro Services andContainers 22 Customer Transactions Inventory Transactions Shipping Transactions Sales TransactionsMonolithic DB System of Record Refactored by transactions sets Employee Transactions Compensation Transactions Salary Transactions Payroll Transactions Key to DevOps MAJOR TRENDS • Large monolithic databases refactored into transaction sets • Applications get refactored into micro-services • Deployment models move from VM/Bare Metal to DBaaS and Containers
- 23. THE CHANGING ROLEOF THE DBA From Ops Supporter to DevOps Collaborator 23
- 24. FROM DATABASE ADMIN TODATA STRATEGIST 24 Automation ends mundane and repetitive tasks Adding new servers by clicking a few buttons Frees up time for strategic efforts More focus on the data and applications Understanding data sources and value to organization Develop new skills in areas such as data science and machine learning
- 25. FROM HARDWARE TWEAKER TOMASTER OF CLOUD TECHNOLOGIES CONTAINERSCLOUDS
- 26. FROM SCHEMA INDEXER TOMASTER MODELER 26 DBAs historically specialized by RDBMS Now need to know/recommend different platforms Also still need deep DB knowledge And understand DB as part of a full stack
- 27. FROM SERVER JOCKEY TOUTILITY PROVIDER 27 DBs now viewed as a utility Flip a switch and get it now Burstable capacity Less time to plan
- 28. FROM LICENSE GATEKEEPER TOBUDGET OWNER 28 Most cloud services are provided by a third-party vendor Need to understand services How to optimize cost vs. performance including infrastructure Architecting to meet requirements and budget
- 29. PUBLIC FROM SITE ADMINTO PLATFORM BROKER 29 DBs becoming platform agnostic Support on premises or public cloud Many deployment and service level options Collaborate with teams supporting the cloud/full application stack Managed Cloud Infrastructure On Premises DBaaS platform Self-managed DBMS Licenses Managed On Premises DB Self-Managed Cloud Compute Fully Managed DBaaS PRIVATE
- 30. FROM QUERY OPTIMIZER TOPERFORMANCE PRO 30 Less time updating/troubleshooting servers More time optimizing data architecture Monitoring of infrastructure and database performance holistically Advanced data modeling, query tuning and Indexing strategies to improve performance
- 31. FROM TO ACCOUNTADMIN TO SECURITY EXPERT 31 Cloud vendors provide secure infrastructure But must ensure that all systems are using that infrastructure properly, especially the database Protecting sensitive information across highly-distributed infrastructures can be a challenge DBA needs to understand possible threat sources Compliance with corporate, industry, and governmental regulations add new responsibilities on DBA
- 32. FROM OPS SUPPORTTO DevOps COLLABORATOR 32 Increasing focus on continuous deployment and delivery DBAs take on tasks once reserved for developers and work closely with DevOps teams More moving out of centralized IT and into the line-of- business/applications teams While continuing as data steward and trusted adviser
- 33. AND STILL NEED TOKEEP IT RUNNING 3333 NOW HAVE TO WORRY ABOUT THE CLOUD FOR: Service providers apply fix packs and upgrade DBMS versions Backup and recovery in the cloud is fully automated DBA still needs awareness and understanding of the potential impact Scalability Fault tolerance Replication
- 34. DATABASE AS ASERVICE DBaaS CAN HELP 34 Self-service provisioning Automation of routine tasks “Infinite” capacity Multi-cloud/hybrid deployment Tradeoff control and options for agility
- 35. What does it mean for the database? •Open co-existence with HDFS, Mongo, Kafka, ... • Rich FDW library • Cloud friendly • Micro/Mini services database refactoring • DBaaS focus • Multi model database • Rich set of data types and extensions (JSONB, Hstore, PostGIS, ...) • Easy to use • High adoption rate • Readily available skill set • Training and certification DEVELOPER FRIENDLY VERSATILE DATA MODEL POLYGLOT FRIENDLY RAPID DEPLOYMENT
- 36.
Editor's Notes
- #10 Add arrows from Past to Present – one on one…
- #12 Patterns Larger databases 2-15 TB, 10’s of k users, 1000’s concurrent users Mixed workloads and polyglot More developers Enterprise readiness
- #13 Note: 1. JSON generally ignores any whitespace around or between syntactic elements (values and punctuation, but not within a string value). 2. JSON only recognizes four specific whitespace characters: i. the space ii. horizontal tab, iii. line feed, and carriage return. 3. JSON does not provide or allow any sort of comment syntax.
- #18 Introduce EDB This is how EDB helps you address complexity The data management space for new digital applications spans a wide range from the edge where customer interaction events and streaming feed from the Internet-of-Things are captured through operational databases that capture transactions and to the world of Big Data, the Hadoop ecosystem. Legacy systems play a role in sales execution, asset management, and core financials and have to be linked where data has to be pulled from them. Data Warehouses still have a place for ad-hoc investigation and reporting. EDB Postgres supports this through replication from legacy systems, links to Big Data through Spark, and connections to the edge capture systems through microservices and data adapters.