Download as PDF, PPTX
Uploaded byAlluxio, Inc.
Building an open data platform with apache iceberg
The document outlines the development of an open data platform using Apache Iceberg, which serves as a table format for analytic data, providing transactional guarantees and performance enhancements. It highlights the need for a multi-engine architecture that leverages various tools like Spark, Trino, and Flink, while addressing usability and productivity issues within data management. Key goals of Iceberg include improving transactions, performance, and usability through features like schema evolution and reliable updates.
Softwareβ¦
Building an open data platform with apache iceberg
- 1. Building an Open DataPlatform with Apache Iceberg Ryan Blue Alluxio Day 8, December 2021
- 2.
- 3.
- 4. Where are wegoing?
- 5. Current data architecture βMulti-engine β Spark for ETL, ML β Trino for ad-hoc, ETL β Flink for streaming β Druid for aggregates β In the cloud (or moving) β Hive Metastore β No metastore? β Investing in data β In people β In tools β In infrastructure
- 6.
- 7.
- 8. What is Iceberg? βA table format β Akin to columnar file formats β Transactional guarantees β Performance enhancements β A standard for analytic tables β Open source spec and library β Integrated into query engines
- 9. And how doesthat help?
- 10. Object storage The gap Data& metadata Compute Apache Spark Catalog ???
- 11. Shared storage requirements Technical: βMust handle concurrent writes β Must be scalable, performant β Must be cloud native Practical: β Must be open source β Must be neutral β Must address productivity
- 12. Icebergβs goals β Add reliabletransactions β Unlock performance β Fix usability
- 13. Object storage Open dataplatform Data & metadata Compute Apache Spark Catalog Vertical solutions Open data stack Data Services
- 14.
- 15. Iceberg exists toο¬x productivity
- 16. Lessons learned β Avoidunpleasant surprises β Principle of least surprise β DonΚΌt steal attention β Reduce context switching
- 17. We try tomake Iceberg invisible
- 18. Usability improvements β Schemaevolution β Instantaneous β no rewrites β Safe β no undead columns π§ β Saves days of headache ALTER TABLE db.tab RENAME COLUMN id TO customer_id β Layout evolution β Lazy β only rewrite if needed β Partitioning mistakes are okay β Changes with your data β Saves a month of headache ALTER TABLE db.tab ADD PARTITION FIELD bucket(256, id)
- 19. Practical improvements β Hiddenpartitioning β No silent correctness bugs β No conversion mistakes β Query without understanding a tableΚΌs physical layout β Reliable updates β Stop manual cleanup β Use any query engine β Automate maintenance
- 20. Performance improvements β Indexedmetadata β Fast job planning β Fast query execution β Faster iteration β Table configuration β Tune tables, not jobs β Automate table tuning β Cluster and sort from config