Download as PDF, PPTX
Uploaded byLuciano Resende
Data access layer and schema definitions
Luciano Resende is a staff software engineer at Shutterfly and committer to Apache projects. He presented on using MongoDB and developing a data access layer and schema definition. The schema definition uses a textual domain-specific language to define models for code generation including persistence, DTOs, mapping, validation and more across platforms like Java, RDBMS and MongoDB.
Data access layer and schema definitions
- 1. Data Access Layer andSchema Definitions Luciano Resende lresende@shutterfly.com lresende@apache.org http://lresende.blogspot.com/
- 2. About Luciano Resende • Staff Software Engineer – Shutterfly, Inc. • Member – Apache Software Foundation • Committer: Apache Tuscany, Wink, Nuvem, PhotoArk December 8, 2011 MongoSV 2011 2
- 3. MongoDB Characteristics • Scalability • Simplicity • Flexible Schema • Low TCO December 8, 2011 MongoSV 2011 3
- 4. MongoDB initial POCs .NET Component firstName first_name MongoDB firstname Java Java Component Component Floor plan image from : h3p://benchmarkoffices.com/floorplans.htm December 8, 2011 MongoSV 2011 4
- 5. The Platform • Applications will interact with MongoDB persistence via services • Enforce a consistent schema across multiple applications and cross-platform Services • Data Access Layer • Data Access Layer that in Data Access Layer conjunction with UDDL abstracts the persistence layer allowing for more flexible and UDDL clean code • A cross platform data schema definition (text DSL) • Simple human (& machine) readable language to describe our domain model entities December 8, 2011 MongoSV 2011 5
- 6.
- 7. Schema Definition –Why using Text DSL • Platform independent data definition. • (Annotated) POJO (JAXB, JPA, Morphia) has too much noises and is abuse-prone. We want to enforce the patterns. • XSD is too complicated and it doesn’t fit all for web 2.0 • We need a simple human (& machine) readable language to describe our domain model to provide/promote: • Service-friendly DTO, Persistence (RDB and NoSQL), Governance, Reuse, Validation, MVC, Best Practice, Documentation • We call the DSL as Universal Data Definition Language (UDDL). • Unify Data Definition across different platform application domains December 8, 2011 MongoSV 2011 7
- 8. Our decision: TextualDSL • Derived from an Apache licensed open source project Sculptor • http://fornax.itemis.de/confluence/display/fornax/Sculptor+%28CSC%29 • Easy to learn, intuitive syntax of the textual DSL, based on the concepts from DDD. • Textual DSL has a lot of productivity benefits over graphical tools • Quick development round trip, short feedback loop • Generation of complete application from a single model, not only fragments that are hard to fit in to the overall design • Supports JPA (oracle/mysql/postgresql)/MongoDB/JAXB/Spring/SpringMVC • Great extensibility and customization • Based on Eclipse Xtext/Xtend/Xpand code generation framework • Can be used with text editor or any IDE, but DSL editor with error highlight, code completion, and outline is provided for Eclipse users • Documentation w/ diagrams for domain model • No runtime magic is built into the tools (generated code can be used without the tooling) December 8, 2011 MongoSV 2011 8
- 9. Schema Definition -The idea 4 metadata POJO XSD Protocol Domain 1. Persistence Buffer/ThriT 2. DTO Model JSON Textual DSL 3. Mapping Schema (such as DDL 4. IntrospecFon Sculptor) 5. ValidaFon 6. MVC/UI 2 1 CRUD DTO DTO DTO REST Root Locate 3 RDB REST EnFty Resource Sub-‐ CRUD EnFty Resource EnFty REST Sub-‐ CRUD MongoDB Resource Generated arFfacts Java code Code uddl generaFon ConfiguraFon files templates Documents (UML diagrams, HTML December 8, 2011 docs) MongoSV 2011 9
- 10. Schema Definition -Sample Application AddressBook { basePackage=com.shutterfly.model Persistence in Module addressbook { hint="store=mongodb” use… Entity Contact { dto mappedName="contact" String id key; String userId ! nullable indexed; EnFty String prefix; String firstName; String middleName; String lastName; Required String company; Indexes - @ContactType type; - @Gender gender; List<String> groups; //group names - List<@ContactEmail> emails; Embedded and - List<@ContactAddress> addresses; References @'javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class)' Timestamp created mappedName="updtime"; @'javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class)' Timestamp lastUpdated mappedName="instime"; } BasicType ContactEmail { String label; Boolean isPrimary; String emailAddress ! nullable indexed; Embedded and } } References December 8, 2011 MongoSV 2011 10
- 11. Schema Definition –Generated Java Code @Generated("uddl:AddressBook/addressbook/Contact") @XmlRootElement @XmlAccessorType(javax.xml.bind.annotation.XmlAccessType.FIELD) @Entity(value = "contact", noClassnameStored = true) public class Contact implements Serializable, Cloneable { @Id private String id; @XmlElement() private String userId; @XmlElement() private String prefix; @XmlElement() private String firstName; @XmlElement() private String middleName; @XmlElement() private String lastName; @XmlElement() private String company; @XmlElement() private List<String> groups = new ArrayList<String>(); @XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class) @XmlElement(name = "updtime") @Property("updtime") private Date created; @XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class) @XmlElement(name = "instime") @Property("instime") private Date lastUpdated; @XmlElement() private ContactType type; @XmlElement() private Gender gender; @XmlElementWrapper(name = "emails") @XmlElement(name = "email") @Embedded private List<ContactEmail> emails = new ArrayList<ContactEmail>(); @XmlElementWrapper(name = "addresses") @XmlElement(name = "address") @Embedded private List<ContactAddress> addresses = new ArrayList<ContactAddress>(); … } December 8, 2011 MongoSV 2011 11
- 12. Schema Definition -Documentation December 8, 2011 MongoSV 2011 12
- 13. DSL language development SemanFc model Generated arFfacts Grammar Language Eclipse Model Load/save (DSL) Workflow Engine Java code (MWE) Parse/validate Eclipse Xtext UI ConfiguraFon files Xpand Documents Parse tree Grammar model model (UML diagrams, HTML docs) EMF Ecore model Xtend/ Check EMF Ecore model Other DSLs (such as XSD, GPB, ThriT or Extend/ Avro) transform/ validate December 8, 2011 MongoSV 2011 13
- 14. Schema Definition -Open source tools Sculptor: • http://fornax.itemis.de/confluence/display/fornax/Sculptor+%28CSC%29 Eclipse Xtext: • http://www.eclipse.org/Xtext/ December 8, 2011 MongoSV 2011 14
- 15.
- 16. Data Access Layer • Provides an Abstraction layer to Persistence API and storage • Also facilitating migration from one Persistence storage to another > Or even Y adaptor pattern to write to multiple Persistence storages • Promote rapidly adoption of new technologies • Provides different concrete APIs • Query based APIs • Entity based APIs • Binary APIs (GridFS) • Cache • Query Abstraction • Ability to plug key generators • Promotes best practices across all applications December 8, 2011 MongoSV 2011 16
- 17. Data Access Layer Services Data Access Layer Query Persistence ORM Cache Builder Driver UDDL December 8, 2011 MongoSV 2011 17
- 18. Data Access Layer– Query Builder • Abstract how application developers define queries • Agnostic of Persistence Storage • Support the following Operators • EQUAL, NOTEQUAL, HASTHISONE, HASALLOF, HASANYOF, HASNONEOF, HASTHISELEMENT,GREATERTHAN,GREATERHANOREQ,LESSTHAN,LESST HANOREQ,WHERE,EXISTS,SIZEEQ,MOD,PATTERN • Support AND and OR compound operators • Syntax originally based on the Morphia ‘Query’ QueryBuilder query = dal.createQueryBuilder().filter(Group.USERID, QueryOperator.EQUAL, userId) .filter(Group.ID, QueryOperator.HASANYOF, groupNames); QueryBuilder query = dal.createQueryBuilder().orFilter("_id", QueryOperator.EQUAL, contactId) .orFilter(Contact.LEGACYID, QueryOperator.EQUAL, contactId); December 8, 2011 MongoSV 2011 18
- 19. Data Access Layer- Cache • Provides out-of-the-box support for cache annotations for Queries and entities @Cacheable(cacheName = ”contacts") public final List<Contact> getContacts(QueryBuilder query) throws ExcepFon December 8, 2011 MongoSV 2011 19
- 20. Data Access Layer- ORM • Morphia is a lightweight type-safe library for mapping Java objects to/from MongoDB • Morphia provides easy support for a strongly-typed programming model December 8, 2011 MongoSV 2011 20
- 21. Data Access Layer– ORM & Partial Updates • Scenario : Allow client applications to send partial documents and merge these changes on current document. • Regular behavior : update({"_id":ObjectId(”…")}, {lastName:"Updated"}, false) MongoDB { "_id" : ObjectId ("4ee1b6ac5c31a483c62d2b15"), "firstName" : "Luciano", "lastName" : "Resende", "email" : { "isPrimary" : true, "emailAddress" : “lresende@...” } } See : git://gist.github.com/1450628.git December 8, 2011 MongoSV 2011 21
- 22. Data Access Layer– ORM & Partial Updates • Scenario : Allow client applications to send partial documents and merge these changes on current document. • Regular behavior : update({"_id":ObjectId(”…")}, {lastName:"Updated"}, false) MongoDB { "_id" : ObjectId ("4ee1b6ac5c31a483c62d2b15"), "firstName" : "Luciano", } See : git://gist.github.com/1450628.git December 8, 2011 MongoSV 2011 22
- 23. Data Access Layer– ORM & Partial Updates • Supporting Partial Updates • Provides specific operations for explicitly applying partial updates • Internally, flatten the update structure into various $set operations MongoDB { update({"_id":ObjectId(”…")}, "_id" : ObjectId {"$set": {"lastName": "Updated","email.isPrimary": ("4ee1b6ac5c31a483c62d2b15"), "firstName" : "Luciano", false}}, false) "lastName" : ”Resende", "email" : { "isPrimary" : false, "emailAddress" : “lresende@...” } } See : git://gist.github.com/1450628.git December 8, 2011 MongoSV 2011 23
- 24. Data Access Layer– ORM & Partial Updates • Supporting Partial Updates • Provides specific operations for explicitly applying partial updates • Internally, flatten the update structure into various $set operations MongoDB { update({"_id":ObjectId(”…")}, "_id" : ObjectId {"$set": {"lastName": "Updated","email.isPrimary": ("4ee1b6ac5c31a483c62d2b15"), "firstName" : "Luciano", false}}, false) "lastName" : ”Updated", "email" : { "isPrimary" : false, "emailAddress" : “lresende@...” } } See : git://gist.github.com/1450628.git December 9, 2011 MongoSV 2011 24
- 25. Data Access Layer- Open source tools • Spring Data - MongoDB: • http://www.springsource.org/spring-data/mongodb • Morphia • http://code.google.com/p/morphia/ December 9, 2011 MongoSV 2011 25
- 26.
- 27. Summary • SchemaDefinition • Cross Platform, Simple human (& machine) readable language to describe our domain model entities • Data Access Layer • Data Access Layer that in conjunction with UDDL abstracts the persistence layer allowing for more flexible and clean code • Promote rapidly adoption of new technologies December 8, 2011 MongoSV 2011 27
- 28.