Downloaded 151 times
![Rapid, Scalable Web Development with MongoDB, Ming, and Python Rick Copeland @rick446 [email_address]](https://image.slidesharecdn.com/mongodc2011-110627214336-phpapp02/75/Rapid-Scalable-Web-Development-with-MongoDB-Ming-and-Python-1-2048.jpg)
![Repository Cache Lessons Learned Using MongoDB to represent graph structures (commit graph, commit trees) requires careful query planning. Pointer-chasing is no fun! Sometimes Ming validation and ORM overhead can be prohibitively expensive – time to drop down a layer. Benchmarking and profiling are your friends, as are queries like {‘_id’: {‘$in’:[…]}} for returning multiple objects](https://image.slidesharecdn.com/mongodc2011-110627214336-phpapp02/75/Rapid-Scalable-Web-Development-with-MongoDB-Ming-and-Python-21-2048.jpg)
![Rick Copeland @rick446 [email_address]](https://image.slidesharecdn.com/mongodc2011-110627214336-phpapp02/75/Rapid-Scalable-Web-Development-with-MongoDB-Ming-and-Python-30-2048.jpg)
Uploaded byRick Copeland
Rapid, Scalable Web Development with MongoDB, Ming, and Python
The document discusses the use of MongoDB, Ming, and Python for scalable web development, particularly in the context of SourceForge.net's project management and analytics systems. It outlines architecture changes, performance metrics, and the advantages of NoSQL databases for supporting complex data and real-time analytics. Key components include the implementation of systems for document storage, asynchronous event logging, and the benefits of flexible data schemas.
Rapid, Scalable Web Development with MongoDB, Ming, and Python
- 1. Rapid, Scalable WebDevelopment with MongoDB, Ming, and Python Rick Copeland @rick446 [email_address]
- 2.
- 3. NoSQL at SourceForgeRewriting Consume Introducing Ming Allura – Open-Sourcing Open Source Zarkov – MongoDB-based (near) real-time analytics
- 4. SF.net “BlackOps”: FossFor.usUser Editable! Web 2.0! (ish) Not Ugly!
- 5. FossFor.us used CouchDB(NoSQL) “ Just adding new fields was trivial, and was happening all the time” – Mark Ramm Scaling up to the level of SF.net needs research CouchDB MongoDB Tokyo Cabinet/Tyrant Cassandra... and others Moving to NoSQL
- 6. What we werelooking for Performance – how does a single node perform? Scalability – needs to support simple replication Ability to handle complex data and queries Ease of development
- 7. NoSQL at SourceForgeRewriting Consume Introducing Ming Allura – Open-Sourcing Open Source Zarkov – MongoDB-based (near) real-time analytics
- 8. Rewriting “Consume” Mosttraffic on SF.net hits 3 types of pages: Project Summary File Browser Download Pages are read-mostly, with infrequent updates from the “Develop” side of sf.net Original goal is 1 MongoDB document per project Later split release data because some projects have lots of releases Periodic updates via RSS and AMQP from “Develop”
- 9. Deployment Architecture LoadBalancer / Proxy Master DB Server MongoDB Master Apache mod_wsgi / TG 2.0 MongoDB Slave Apache mod_wsgi / TG 2.0 MongoDB Slave Apache mod_wsgi / TG 2.0 MongoDB Slave Gobble Server Develop Apache mod_wsgi / TG 2.0 MongoDB Slave
- 10. Deployment Architecture (revised)Load Balancer / Proxy Master DB Server MongoDB Master Apache mod_wsgi / TG 2.0 Gobble Server Develop Apache mod_wsgi / TG 2.0 Apache mod_wsgi / TG 2.0 Apache mod_wsgi / TG 2.0 Scalability is good Single-node performance is good, too
- 11. NoSQL at SourceForgeRewriting Consume Introducing Ming Allura – Open-Sourcing Open Source Zarkov – MongoDB-based (near) real-time analytics
- 12. Ming – an “Object-Document Mapper?” Your data has a schema Your database can define and enforce it It can live in your application (as with MongoDB) Nice to have the schema defined in one place in the code Sometimes you need a “migration” Changing the structure/meaning of fields Adding indexes, particularly unique indexes Sometimes lazy, sometimes eager “ Unit of work:” Queuing up all your updates can be handy Python dicts are nice; objects are nicer
- 13. Ming Concepts Inspiredby SQLAlchemy Group of collection objects with schemas defined Group of classes to which you map your collections Use collection-level operations for performance Use class-level operations for abstraction Convenience methods for loading/saving objects and ensuring indexes are created Migrations Unit of Work – great for web applications MIM – “Mongo in Memory” nice for unit tests
- 14. Ming Example from ming import schema, Field from ming.orm import (mapper, Mapper, RelationProperty, ForeignIdProperty) WikiDoc = collection(‘ wiki_page' , session, Field( '_id' , schema . ObjectId()), Field( 'title' , str , index = True ), Field( 'text' , str )) CommentDoc = collection(‘ comment' , session, Field( '_id' , schema . ObjectId()), Field( 'page_id' , schema . ObjectId(), index = True ), Field( 'text' , str )) class WikiPage ( object ): pass class Comment ( object ): pass ormsession . mapper(WikiPage, WikiDoc, properties = dict ( comments = RelationProperty( 'WikiComment' ))) ormsession . mapper(Comment, CommentDoc, properties = dict ( page_id = ForeignIdProperty( 'WikiPage' ), page = RelationProperty( 'WikiPage' ))) Mapper . compile_all()
- 15. NoSQL at SourceForgeRewriting Consume Introducing Ming Allura – Open-Sourcing Open Source Zarkov – MongoDB-based (near) real-time analytics
- 16. Python / MongoDBTaking Over…. Allura Load Balancer / Proxy Master DB Server MongoDB Master Apache mod_wsgi / TG 2.0 Gobble Server Develop Apache mod_wsgi / TG 2.0 Apache mod_wsgi / TG 2.0 Apache mod_wsgi / TG 2.0 Scalability is good Single-node performance is good, too
- 17. Allura Architecture Web-facingApp Server Task Daemon SMTP Server FUSE Filesystem (repository hosting)
- 18. Allura Threaded DiscussionsMessageDoc = collection( 'message' , project_doc_session, Field( '_id' , str , if_missing = h . gen_message_id), Field( 'slug' , str , if_missing = h . nonce), Field( 'full_slug' , str ), Field( 'parent_id' , str ),…) _id – use an email Message-ID compatible key slug – threaded path of random 4-digit hex numbers prefixed by parent (e.g. dead/beef/f00d dead/beef dead) full_slug – slug interspersed with ISO-formatted message datetime (20110627…dead/20110627…beef….) Easy queries for hierarchical data Find all descendants of a message – slug prefix search “dead/.*” Sort messages by thread, then by date – full_slug sort
- 19. MonQ: Async Queueingin MongoDB states = ( 'ready' , 'busy' , 'error' , 'complete' ) result_types = ( 'keep' , 'forget' ) MonQTaskDoc = collection( 'monq_task' , main_doc_session, Field( '_id' , schema . ObjectId()), Field( 'state' , schema . OneOf( * states)), Field( 'result_type' , Schema . OneOf( * result_types)), Field( 'time_queue' , datetime), Field( 'time_start' , datetime), Field( 'time_stop' , datetime), # dotted path to function Field( 'task_name' , str ), Field( 'process' , str ), # worker process name: “locks” the task Field( 'context' , dict ( project_id = schema . ObjectId(), app_config_id = schema . ObjectId(), user_id = schema . ObjectId())), Field( 'args' , list ), Field( 'kwargs' , { None : None }), Field( 'result' , None , if_missing = None ))
- 20. Repository Cache ObjectsOn commit to a repo (Hg, SVN, or Git) Build commit graph in MongoDB for new commits Build auxiliary structures tree structure, including all trees in a commit & last commit to modify linear commit runs (useful for generating history) commit difference summary (must be computed in Hg and Git) Note references to other artifacts and commits Repo browser uses cached structure to serve pages DiffInfo Tree Trees CommitRun LastCommit Commit
- 21. Repository Cache LessonsLearned Using MongoDB to represent graph structures (commit graph, commit trees) requires careful query planning. Pointer-chasing is no fun! Sometimes Ming validation and ORM overhead can be prohibitively expensive – time to drop down a layer. Benchmarking and profiling are your friends, as are queries like {‘_id’: {‘$in’:[…]}} for returning multiple objects
- 22. NoSQL at SourceForgeRewriting Consume Introducing Ming Allura – Open-Sourcing Open Source Zarkov – MongoDB-based (near) real-time analytics
- 23. And now, forsomething completely different… Business: we need more visibility into what users are doing Low overhead Near real-time Unified view of lots of systems Python PHP Perl
- 24. Introducing Zarkov AsynchronousTCP server for event logging with gevent Turn OFF “safe” writes, turn OFF Ming validation (or do it in the client) Incrementally calculate aggregate stats based on event log using mapreduce with {‘out’:’reduce’}
- 25.
- 26. What We LikedPerformance, performance, performance – Easily handle 90% of SF.net traffic from 1 DB server, 4 web servers Dynamic schema allows fast schema evolution in development, making many migrations unnecessary Replication is easy , making scalability and backups easy Query Language You mean I can have performance without map-reduce? GridFS
- 27. Pitfalls Too-large documentsStore less per document Return only a few fields Ignoring indexing Watch your server log; bad queries show up there Too much denormalization Try to use an index if all you need is a backref Stale data is a tricky problem Using many databases when one will do Using too many queries
- 28. Open Source Minghttp://sf.net/projects/merciless/ MIT License Allura http://sf.net/p/allura/ Apache License Zarkov http://sf.net/p/zarkov/ Apache License
- 29. Future Work mongosNew Allura Tools Migrating legacy SF.net projects to Allura Continue to optimize stats & analytics (Zarkov and others) Better APIs to access your project data
- 30. Rick Copeland @rick446[email_address]
Editor's Notes
- #3 Used to be PHP + MySQL + Postgres + ….. How did we get started down the NoSQL path?
- #5 New team hired, which had some new ideas Crazy short timeline
- #6 FossFor.us was a technical success, if a market failure Can we bring the same technical success to sf.net?
- #10 Content begins life in “Develop” world – old PHP/Relational codebase Gobble runs off existing APIs and new AMQP feeds to populate master mongodb server Webheads run off local MongoDB slaves
- #11 Questionable architectural decision, but it made sense at the time. All the while, our Data model evolved Factored out a lot of our custom mongodb code into…
- #15 Index=True (also has a unique=True arg) ForeignIdProperty like foreign keys – hints to the mapper how to actually infer RelationProperties Empty classes can have properties added by their mappers
- #18 Several interacting apps, with MongoDB and SOLR at the center Used to have RabbitMQ but we wanted more visibility into our queues Now I’ll show some of the model classes we use
- #19 Efficient, threaded messages Short “slugs” for linking to a particular message (and for URLs) Easy sorting Note the ‘if missing’ – basically default values that are created by Ming (in this case, python functions to generate the values)
- #20 Async Queuing Some perf instrumentation Can use findandmodify to grab and lock tasks for processing Uses both slow polling and amqp notification (but only to say POLL NOW)
- #21 UI-centered design Make sure it doesn’t take a lot of queries to build a page
- #24 Still alpha-quality (OSS because that’s the way we roll)
- #25 Can record many more than 4k events per second 345M events per day (single-thread, VM on a laptop) – we get a lot of traffic, but not that much MR makes this much lower if calculated continuously, still hundreds of events even with MR locking Probably use a slave for MR processing, may end up using Hadoop or Disco if we need to