Back to Basics 2017 : Webinar 2 Your First MongoDB Application Joe Drumgoole Director of Developer Advocacy, EMEA MongoDB @jdrumgoole V1.1
3 Summary of Part 1 • Why NoSQL exists • The types of NoSQL database • The key features of MongoDB
4 Agenda • Database concepts • Installing MongoDB • Building a basic blogging application • Adding an index • Query optimization with explain
5 Concepts Relational MongoDB Database Database Table Collection Row Document Index Index Join Lookup Foreign Key Reference Multi-table transaction Single document transaction
6 Document Store { name : “Joe Drumgoole”, title : “Director of Developer Advocacy”, Address : { address1 : “Latin Hall”, address2 : “Golden Lane”, eircode : “D09 N623”, } expertise: [ “MongoDB”, “Python”, “Javascript” ], employee_number : 320, location : [ 53.34, -6.26 ] }
7 MongoDB Documents are Typed { name : “Joe Drumgoole”, title : “Director of Developer Advocacy”, Address : { address1 : “Latin Hall”, address2 : “Golden Lane”, eircode : “D09 N623”, } expertise: [ “MongoDB”, “Python”, “Javascript” ], employee_number : 320, location : [ 53.34, -6.26 ] } Strings Nested Document Array Integer Geo-spatial Coordinates
8 MongoDB Drivers http://bsonspec.org/spec.html
9 Installing MongoDB $ curl -O https://fastdl.mongodb.org/osx/mongodb-osx-ssl-x86_64-3.4.1.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 60.9M 100 60.9M 0 0 2730k 0 0:00:22 0:00:22 --:--:-- 1589k $ tar xzvf mongodb-osx-x86_64-3.2.6.tgz x mongodb-osx-x86_64-3.2.6/README x mongodb-osx-x86_64-3.2.6/THIRD-PARTY-NOTICES x mongodb-osx-x86_64-3.2.6/MPL-2 x mongodb-osx-x86_64-3.2.6/GNU-AGPL-3.0 x mongodb-osx-x86_64-3.2.6/bin/mongodump x mongodb-osx-x86_64-3.2.6/bin/mongorestore x mongodb-osx-x86_64-3.2.6/bin/mongoexport x mongodb-osx-x86_64-3.2.6/bin/mongoimport x mongodb-osx-x86_64-3.2.6/bin/mongostat x mongodb-osx-x86_64-3.2.6/bin/mongotop x mongodb-osx-x86_64-3.2.6/bin/bsondump x mongodb-osx-x86_64-3.2.6/bin/mongofiles x mongodb-osx-x86_64-3.2.6/bin/mongooplog x mongodb-osx-x86_64-3.2.6/bin/mongoperf x mongodb-osx-x86_64-3.2.6/bin/mongosniff x mongodb-osx-x86_64-3.2.6/bin/mongod x mongodb-osx-x86_64-3.2.6/bin/mongos x mongodb-osx-x86_64-3.2.6/bin/mongo $ ln -s mongodb-osx-x86_64-3.2.6 mongodb
10 Running mongod JD10Gen:mongodb jdrumgoole$ ./bin/mongod --dbpath /data/b2b 2016-05-23T19:21:07.767+0100 I CONTROL [initandlisten] MongoDB starting : pid=49209 port=27017 dbpath=/data/b2b 64- bit host=JD10Gen.local 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] db version v3.2.6 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] allocator: system 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] modules: none 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] build environment: 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] distarch: x86_64 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] target_arch: x86_64 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] options: { storage: { dbPath: "/data/b2b" } } 2016-05-23T19:21:07.769+0100 I - [initandlisten] Detected data files in /data/b2b created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2016-05-23T19:21:07.769+0100 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true ,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB) ,statistics_log=(wait=0), 2016-05-23T19:21:08.837+0100 I CONTROL [initandlisten] 2016-05-23T19:21:08.838+0100 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000 2016-05-23T19:21:08.840+0100 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker 2016-05-23T19:21:08.840+0100 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/b2b/diagnostic.data' 2016-05-23T19:21:08.841+0100 I NETWORK [initandlisten] waiting for connections on port 27017 2016-05-23T19:21:09.148+0100 I NETWORK [initandlisten] connection accepted from 127.0.0.1:59213 #1 (1 connection now open)
11 Connecting Via The Shell $ ./bin/mongo MongoDB shell version: 3.2.6 connecting to: test Server has startup warnings: 2016-05-17T11:46:03.516+0100 I CONTROL [initandlisten] 2016-05-17T11:46:03.516+0100 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000 >
12 Inserting your first record > show databases local 0.000GB > use test switched to db test > show databases local 0.000GB > db.demo.insert( { "key" : "value" } ) WriteResult({ "nInserted" : 1 }) > show databases local 0.000GB test 0.000GB > show collections demo > db.demo.findOne() { "_id" : ObjectId("573af7085ee4be80385332a6"), "key" : "value" } >
13 Object ID 573af7085ee4be80385332a6 TS------ID----PID-Count-
14 Using Compass
15 A Simple Blog Application • Lets create a blogging application with: – Articles – Users – Comments
16 Typical Entity Relation Diagram
17 In MongoDB we can build organically > use blog switched to db blog > db.users.insert( { "username" : "jdrumgoole", "password" : "top secret", "lang" : "EN" } ) WriteResult({ "nInserted" : 1 }) > db.users.findOne() { "_id" : ObjectId("573afff65ee4be80385332a7"), "username" : "jdrumgoole", "password" : "top secret", "lang" : "EN" }
18 How do we do this in a program? ''' Created on 17 May 2016 @author: jdrumgoole ''' import pymongo # # client defaults to localhost and port 27017. eg MongoClient('localhost', 27017) client = pymongo.MongoClient() blogDatabase = client[ "blog" ] usersCollection = blogDatabase[ "users" ] usersCollection.insert_one( { "username" : "jdrumgoole", "password" : "top secret", "lang" : "EN" }) user = usersCollection.find_one() print( user )
19 Next up Articles … articlesCollection = blogDatabase[ "articles" ] author = "jdrumgoole" article = { "title" : "This is my first post", "body" : "The is the longer body text for my blog post. We can add lots of text here.", "author" : author, "tags" : [ "joe", "general", "Ireland", "admin" ] } # # Lets check if our author exists # if usersCollection.find_one( { "username" : author }) : articlesCollection.insert_one( article ) else: raise ValueError( "Author %s does not exist" % author )
20 Create a new type of article # # Lets add a new type of article with a posting date and a section # author = "jdrumgoole" title = "This is a post on MongoDB" newPost = { "title" : title, "body" : "MongoDB is the worlds most popular NoSQL database. It is a document database", "author" : author, "tags" : [ "joe", "mongodb", "Ireland" ], "section" : "technology", "postDate" : datetime.datetime.now(), } # # Lets check if our author exists # if usersCollection.find_one( { "username" : author }) : articlesCollection.insert_one( newPost )
21 Make a lot of articles 1 import pymongo import string import datetime import random def randomString( size, letters = string.letters ): return "".join( [random.choice( letters ) for _ in xrange( size )] ) client = pymongo.MongoClient() def makeArticle( count, author, timestamp ): return { "_id" : count, "title" : randomString( 20 ), "body" : randomString( 80 ), "author" : author, "postdate" : timestamp } def makeUser( username ): return { "username" : username, "password" : randomString( 10 ) , "karma" : random.randint( 0, 500 ), "lang" : "EN" }
22 Make a lot of articles 2 blogDatabase = client[ "blog" ] usersCollection = blogDatabase[ "users" ] articlesCollection = blogDatabase[ "articles" ] bulkUsers = usersCollection.initialize_ordered_bulk_op() bulkArticles = articlesCollection.initialize_ordered_bulk_op() ts = datetime.datetime.now() for i in range( 1000000 ) : #username = randomString( 10, string.ascii_uppercase ) + "_" + str( i ) username = "USER_" + str( i ) bulkUsers.insert( makeUser( username ) ) ts = ts + datetime.timedelta( seconds = 1 ) bulkArticles.insert( makeArticle( i, username, ts )) if ( i % 500 == 0 ) : bulkUsers.execute() bulkArticles.execute() bulkUsers = usersCollection.initialize_ordered_bulk_op() bulkArticles = articlesCollection.initialize_ordered_bulk_op() bulkUsers.execute() bulkArticles.execute()
23 Find a User > db.users.findOne() { "_id" : ObjectId("5742da5bb26a88bc00e941ac"), "username" : "FLFZQLSRWZ_0", "lang" : "EN", "password" : "vTlILbGWLt", "karma" : 448 } > db.users.find( { "username" : "VHXDAUUFJW_45" } ).pretty() { "_id" : ObjectId("5742da5bb26a88bc00e94206"), "username" : "VHXDAUUFJW_45", "lang" : "EN", "password" : "GmRLnCeKVp", "karma" : 284 }
24 Find Users with high Karma > db.users.find( { "karma" : { $gte : 450 }} ).pretty() { "_id" : ObjectId("5742da5bb26a88bc00e941ae"), "username" : "JALLFRKBWD_1", "lang" : "EN", "password" : "bCSKSKvUeb", "karma" : 487 } { "_id" : ObjectId("5742da5bb26a88bc00e941e4"), "username" : "OTKWJJBNBU_28", "lang" : "EN", "password" : "HAWpiATCBN", "karma" : 473 } { …
25 Using projection > db.users.find( { "karma" : { "$gte" : 450 }}, { "_id" : 0, "username" : 1 , "karma" : 1 }) { "username" : "USER_1", "karma" : 461 } { "username" : "USER_3", "karma" : 494 } { "username" : "USER_20", "karma" : 464 } { "username" : "USER_34", "karma" : 475 } { "username" : "USER_46", "karma" : 462 } { "username" : "USER_47", "karma" : 486 } { "username" : "USER_48", "karma" : 488 } { "username" : "USER_49", "karma" : 452 } { "username" : "USER_61", "karma" : 483 } { "username" : "USER_73", "karma" : 452 } { "username" : "USER_80", "karma" : 494 } { "username" : "USER_87", "karma" : 497 } …
26 Update an article to add Comments 1 > db.articles.find( { "_id" : 19 } ).pretty() { "_id" : 19, "body" : "nTzOofOcnHKkJxpjKAyqTTnKZMFzzkWFeXtBRuEKsctuGBgWIrEBrYdvFIVHJWaXLUTVUXblOZZgUq Wu", "postdate" : ISODate("2016-05-23T12:02:46.830Z"), "author" : "ASWTOMMABN_19", "title" : "CPMaqHtAdRwLXhlUvsej" } > db.articles.update( { _id : 18 }, { $set : { comments : [] }} ) WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
27 Update an article to add comments 2 > db.articles.find( { _id :18 } ).pretty() { "_id" : 18, "body" : "KmwFSIMQGcIsRNTDBFPuclwcVJkoMcrIPwTiSZDYyatoKzeQiKvJkiVSrndXqrALVIYZxGpaMjucgX UV", "postdate" : ISODate("2016-05-23T16:04:39.497Z"), "author" : "USER_18", "title" : "wTLreIEyPfovEkBhJZZe", "comments" : [ ] } >
28 Update an article to add comments 3 > db.articles.update( { _id : 18 }, { $push : { comments : { username : "joe", comment : "hey first post" }}} ) WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 }) > db.articles.find( { _id :18 } ).pretty() { "_id" : 18, "body" : "KmwFSIMQGcIsRNTDBFPuclwcVJkoMcrIPwTiSZDYyatoKzeQiKvJkiVSrndXqrALVIYZxGpaMjucgXUV" , "postdate" : ISODate("2016-05-23T16:04:39.497Z"), "author" : "USER_18", "title" : "wTLreIEyPfovEkBhJZZe", "comments" : [ { "username" : "joe", "comment" : "hey first post" } ] } >
29 Delete an article > db.articles.remove( { "_id" : 25 } ) WriteResult({ "nRemoved" : 1 }) > db.articles.remove( { "_id" : 25 } ) WriteResult({ "nRemoved" : 0 }) > db.articles.remove( { "_id" : { $lte : 5 }} ) WriteResult({ "nRemoved" : 6 }) • Deletion leaves holes • Dropping a collection is cheaper than deleting a large collection element by element
30 A quick look at users and articles again > db.users.findOne() { "_id" : ObjectId("57431c07b26a88bf060e10cb"), "username" : "USER_0", "lang" : "EN", "password" : "kGIxPxqKGJ", "karma" : 266 } > db.articles.findOne() { "_id" : 0, "body" : "hvJLnrrfZQurmtjPfUWbMhaQWbNjXLzjpuGLZjsxHXbUycmJVZTeOZesTnZtojThrebRcUoiYwivjpwG" , "postdate" : ISODate("2016-05-23T16:04:39.246Z"), "author" : "USER_0", "title" : "gpNIoPxpfTAxWjzAVoTJ" } >
31 Find a user > db.users.find( { "username" : "USER_99" } ).explain() { "queryPlanner" : { "plannerVersion" : 1, "namespace" : "blog.users", "indexFilterSet" : false, "parsedQuery" : { "username" : { "$eq" : "USER_99" } }, "winningPlan" : { "stage" : "COLLSCAN", "filter" : { "username" : { "$eq" : "USER_99" } }, "direction" : "forward" }, }, }
32 Find a user – execution stats > db.users.find( { "username" : "USER_99" } ).explain( "executionStats" ).executionStats { "executionSuccess" : true, "nReturned" : 1, "executionTimeMillis" : 412, "totalKeysExamined" : 0, "totalDocsExamined" : 1000000, "executionStages" : { "stage" : "COLLSCAN", "filter" : { "username" : { "$eq" : "USER_99" } }, "nReturned" : 1, "executionTimeMillisEstimate" : 302, "works" : 1000002, "advanced" : 1, "needTime" : 1000000, "needYield" : 0, "saveState" : 7823, "restoreState" : 7823, "isEOF" : 1, "invalidates" : 0, "direction" : "forward", "docsExamined" : 1000000 } }
33 We need an index > db.users.createIndex( { username : 1 } ) { "createdCollectionAutomatically" : false, "numIndexesBefore" : 1, "numIndexesAfter" : 2, "ok" : 1 } >
34 Indexes Overview • Parameters – Background : Create an index in the background as opposed to locking the database – Unique : All keys in the collection must be unique. Duplicate key insertions will be rejected with an error. – Name : Explicitly name an index. Otherwise the index name is autogenerated from the index field. • Deleting an index – db.users.dropIndex({ “username” : 1 }) • List indexes – db.users.getIndexes()
35 Query Plan Execution Stages • COLLSCAN : for a collection scan • IXSCAN : for scanning index keys • FETCH : for retrieving documents • SHARD_MERGE : for merging results from shards
36 Add an index > db.users.find( {"username" : "USER_999999”} ).explain("executionStats”).executionStats { "executionSuccess" : true, "nReturned" : 1, "executionTimeMillis" : 0, "totalKeysExamined" : 1, "totalDocsExamined" : 1, …
37 Execution stage "executionStages" : { "stage" : "FETCH", "nReturned" : 1, "executionTimeMillisEstimate" : 0, "docsExamined" : 1,, "inputStage" : { "stage" : "IXSCAN", "nReturned" : 1, "executionTimeMillisEstimate" : 0, "keyPattern" : { "username" : 1 }, "indexName" : "username_1", "isMultiKey" : false, "isUnique" : false, "isSparse" : false, "isPartial" : false, "indexVersion" : 1, "direction" : "forward", "indexBounds" : { "username" : [ "["USER_999999", "USER_999999"]" ] }, "keysExamined" : 1, "seenInvalidated" : 0 } } }
38 What we have learned • How to create a database and a collection • How to insert content into that collection • How to query the collection • How to update a document in place • How to delete a document • How to check the efficiency of an operation • How to add an index • How to check an index is being used in an operation
39 Next Webinar : Introduction to Replica Sets • How to ensure your data is durable • How to recover from failures automatically • How to write safe client code Thursday, 2-Feb-2016, 11:00 am GMT.
Q&A

Back to Basics: My First MongoDB Application

  • 2.
    Back to Basics2017 : Webinar 2 Your First MongoDB Application Joe Drumgoole Director of Developer Advocacy, EMEA MongoDB @jdrumgoole V1.1
  • 3.
    3 Summary of Part1 • Why NoSQL exists • The types of NoSQL database • The key features of MongoDB
  • 4.
    4 Agenda • Database concepts •Installing MongoDB • Building a basic blogging application • Adding an index • Query optimization with explain
  • 5.
    5 Concepts Relational MongoDB Database Database TableCollection Row Document Index Index Join Lookup Foreign Key Reference Multi-table transaction Single document transaction
  • 6.
    6 Document Store { name :“Joe Drumgoole”, title : “Director of Developer Advocacy”, Address : { address1 : “Latin Hall”, address2 : “Golden Lane”, eircode : “D09 N623”, } expertise: [ “MongoDB”, “Python”, “Javascript” ], employee_number : 320, location : [ 53.34, -6.26 ] }
  • 7.
    7 MongoDB Documents areTyped { name : “Joe Drumgoole”, title : “Director of Developer Advocacy”, Address : { address1 : “Latin Hall”, address2 : “Golden Lane”, eircode : “D09 N623”, } expertise: [ “MongoDB”, “Python”, “Javascript” ], employee_number : 320, location : [ 53.34, -6.26 ] } Strings Nested Document Array Integer Geo-spatial Coordinates
  • 8.
  • 9.
    9 Installing MongoDB $ curl-O https://fastdl.mongodb.org/osx/mongodb-osx-ssl-x86_64-3.4.1.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 60.9M 100 60.9M 0 0 2730k 0 0:00:22 0:00:22 --:--:-- 1589k $ tar xzvf mongodb-osx-x86_64-3.2.6.tgz x mongodb-osx-x86_64-3.2.6/README x mongodb-osx-x86_64-3.2.6/THIRD-PARTY-NOTICES x mongodb-osx-x86_64-3.2.6/MPL-2 x mongodb-osx-x86_64-3.2.6/GNU-AGPL-3.0 x mongodb-osx-x86_64-3.2.6/bin/mongodump x mongodb-osx-x86_64-3.2.6/bin/mongorestore x mongodb-osx-x86_64-3.2.6/bin/mongoexport x mongodb-osx-x86_64-3.2.6/bin/mongoimport x mongodb-osx-x86_64-3.2.6/bin/mongostat x mongodb-osx-x86_64-3.2.6/bin/mongotop x mongodb-osx-x86_64-3.2.6/bin/bsondump x mongodb-osx-x86_64-3.2.6/bin/mongofiles x mongodb-osx-x86_64-3.2.6/bin/mongooplog x mongodb-osx-x86_64-3.2.6/bin/mongoperf x mongodb-osx-x86_64-3.2.6/bin/mongosniff x mongodb-osx-x86_64-3.2.6/bin/mongod x mongodb-osx-x86_64-3.2.6/bin/mongos x mongodb-osx-x86_64-3.2.6/bin/mongo $ ln -s mongodb-osx-x86_64-3.2.6 mongodb
  • 10.
    10 Running mongod JD10Gen:mongodb jdrumgoole$./bin/mongod --dbpath /data/b2b 2016-05-23T19:21:07.767+0100 I CONTROL [initandlisten] MongoDB starting : pid=49209 port=27017 dbpath=/data/b2b 64- bit host=JD10Gen.local 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] db version v3.2.6 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] allocator: system 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] modules: none 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] build environment: 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] distarch: x86_64 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] target_arch: x86_64 2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] options: { storage: { dbPath: "/data/b2b" } } 2016-05-23T19:21:07.769+0100 I - [initandlisten] Detected data files in /data/b2b created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2016-05-23T19:21:07.769+0100 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true ,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB) ,statistics_log=(wait=0), 2016-05-23T19:21:08.837+0100 I CONTROL [initandlisten] 2016-05-23T19:21:08.838+0100 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000 2016-05-23T19:21:08.840+0100 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker 2016-05-23T19:21:08.840+0100 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/b2b/diagnostic.data' 2016-05-23T19:21:08.841+0100 I NETWORK [initandlisten] waiting for connections on port 27017 2016-05-23T19:21:09.148+0100 I NETWORK [initandlisten] connection accepted from 127.0.0.1:59213 #1 (1 connection now open)
  • 11.
    11 Connecting Via TheShell $ ./bin/mongo MongoDB shell version: 3.2.6 connecting to: test Server has startup warnings: 2016-05-17T11:46:03.516+0100 I CONTROL [initandlisten] 2016-05-17T11:46:03.516+0100 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000 >
  • 12.
    12 Inserting your firstrecord > show databases local 0.000GB > use test switched to db test > show databases local 0.000GB > db.demo.insert( { "key" : "value" } ) WriteResult({ "nInserted" : 1 }) > show databases local 0.000GB test 0.000GB > show collections demo > db.demo.findOne() { "_id" : ObjectId("573af7085ee4be80385332a6"), "key" : "value" } >
  • 13.
  • 14.
  • 15.
    15 A Simple BlogApplication • Lets create a blogging application with: – Articles – Users – Comments
  • 16.
  • 17.
    17 In MongoDB wecan build organically > use blog switched to db blog > db.users.insert( { "username" : "jdrumgoole", "password" : "top secret", "lang" : "EN" } ) WriteResult({ "nInserted" : 1 }) > db.users.findOne() { "_id" : ObjectId("573afff65ee4be80385332a7"), "username" : "jdrumgoole", "password" : "top secret", "lang" : "EN" }
  • 18.
    18 How do wedo this in a program? ''' Created on 17 May 2016 @author: jdrumgoole ''' import pymongo # # client defaults to localhost and port 27017. eg MongoClient('localhost', 27017) client = pymongo.MongoClient() blogDatabase = client[ "blog" ] usersCollection = blogDatabase[ "users" ] usersCollection.insert_one( { "username" : "jdrumgoole", "password" : "top secret", "lang" : "EN" }) user = usersCollection.find_one() print( user )
  • 19.
    19 Next up Articles … articlesCollection= blogDatabase[ "articles" ] author = "jdrumgoole" article = { "title" : "This is my first post", "body" : "The is the longer body text for my blog post. We can add lots of text here.", "author" : author, "tags" : [ "joe", "general", "Ireland", "admin" ] } # # Lets check if our author exists # if usersCollection.find_one( { "username" : author }) : articlesCollection.insert_one( article ) else: raise ValueError( "Author %s does not exist" % author )
  • 20.
    20 Create a newtype of article # # Lets add a new type of article with a posting date and a section # author = "jdrumgoole" title = "This is a post on MongoDB" newPost = { "title" : title, "body" : "MongoDB is the worlds most popular NoSQL database. It is a document database", "author" : author, "tags" : [ "joe", "mongodb", "Ireland" ], "section" : "technology", "postDate" : datetime.datetime.now(), } # # Lets check if our author exists # if usersCollection.find_one( { "username" : author }) : articlesCollection.insert_one( newPost )
  • 21.
    21 Make a lotof articles 1 import pymongo import string import datetime import random def randomString( size, letters = string.letters ): return "".join( [random.choice( letters ) for _ in xrange( size )] ) client = pymongo.MongoClient() def makeArticle( count, author, timestamp ): return { "_id" : count, "title" : randomString( 20 ), "body" : randomString( 80 ), "author" : author, "postdate" : timestamp } def makeUser( username ): return { "username" : username, "password" : randomString( 10 ) , "karma" : random.randint( 0, 500 ), "lang" : "EN" }
  • 22.
    22 Make a lotof articles 2 blogDatabase = client[ "blog" ] usersCollection = blogDatabase[ "users" ] articlesCollection = blogDatabase[ "articles" ] bulkUsers = usersCollection.initialize_ordered_bulk_op() bulkArticles = articlesCollection.initialize_ordered_bulk_op() ts = datetime.datetime.now() for i in range( 1000000 ) : #username = randomString( 10, string.ascii_uppercase ) + "_" + str( i ) username = "USER_" + str( i ) bulkUsers.insert( makeUser( username ) ) ts = ts + datetime.timedelta( seconds = 1 ) bulkArticles.insert( makeArticle( i, username, ts )) if ( i % 500 == 0 ) : bulkUsers.execute() bulkArticles.execute() bulkUsers = usersCollection.initialize_ordered_bulk_op() bulkArticles = articlesCollection.initialize_ordered_bulk_op() bulkUsers.execute() bulkArticles.execute()
  • 23.
    23 Find a User >db.users.findOne() { "_id" : ObjectId("5742da5bb26a88bc00e941ac"), "username" : "FLFZQLSRWZ_0", "lang" : "EN", "password" : "vTlILbGWLt", "karma" : 448 } > db.users.find( { "username" : "VHXDAUUFJW_45" } ).pretty() { "_id" : ObjectId("5742da5bb26a88bc00e94206"), "username" : "VHXDAUUFJW_45", "lang" : "EN", "password" : "GmRLnCeKVp", "karma" : 284 }
  • 24.
    24 Find Users withhigh Karma > db.users.find( { "karma" : { $gte : 450 }} ).pretty() { "_id" : ObjectId("5742da5bb26a88bc00e941ae"), "username" : "JALLFRKBWD_1", "lang" : "EN", "password" : "bCSKSKvUeb", "karma" : 487 } { "_id" : ObjectId("5742da5bb26a88bc00e941e4"), "username" : "OTKWJJBNBU_28", "lang" : "EN", "password" : "HAWpiATCBN", "karma" : 473 } { …
  • 25.
    25 Using projection > db.users.find({ "karma" : { "$gte" : 450 }}, { "_id" : 0, "username" : 1 , "karma" : 1 }) { "username" : "USER_1", "karma" : 461 } { "username" : "USER_3", "karma" : 494 } { "username" : "USER_20", "karma" : 464 } { "username" : "USER_34", "karma" : 475 } { "username" : "USER_46", "karma" : 462 } { "username" : "USER_47", "karma" : 486 } { "username" : "USER_48", "karma" : 488 } { "username" : "USER_49", "karma" : 452 } { "username" : "USER_61", "karma" : 483 } { "username" : "USER_73", "karma" : 452 } { "username" : "USER_80", "karma" : 494 } { "username" : "USER_87", "karma" : 497 } …
  • 26.
    26 Update an articleto add Comments 1 > db.articles.find( { "_id" : 19 } ).pretty() { "_id" : 19, "body" : "nTzOofOcnHKkJxpjKAyqTTnKZMFzzkWFeXtBRuEKsctuGBgWIrEBrYdvFIVHJWaXLUTVUXblOZZgUq Wu", "postdate" : ISODate("2016-05-23T12:02:46.830Z"), "author" : "ASWTOMMABN_19", "title" : "CPMaqHtAdRwLXhlUvsej" } > db.articles.update( { _id : 18 }, { $set : { comments : [] }} ) WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
  • 27.
    27 Update an articleto add comments 2 > db.articles.find( { _id :18 } ).pretty() { "_id" : 18, "body" : "KmwFSIMQGcIsRNTDBFPuclwcVJkoMcrIPwTiSZDYyatoKzeQiKvJkiVSrndXqrALVIYZxGpaMjucgX UV", "postdate" : ISODate("2016-05-23T16:04:39.497Z"), "author" : "USER_18", "title" : "wTLreIEyPfovEkBhJZZe", "comments" : [ ] } >
  • 28.
    28 Update an articleto add comments 3 > db.articles.update( { _id : 18 }, { $push : { comments : { username : "joe", comment : "hey first post" }}} ) WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 }) > db.articles.find( { _id :18 } ).pretty() { "_id" : 18, "body" : "KmwFSIMQGcIsRNTDBFPuclwcVJkoMcrIPwTiSZDYyatoKzeQiKvJkiVSrndXqrALVIYZxGpaMjucgXUV" , "postdate" : ISODate("2016-05-23T16:04:39.497Z"), "author" : "USER_18", "title" : "wTLreIEyPfovEkBhJZZe", "comments" : [ { "username" : "joe", "comment" : "hey first post" } ] } >
  • 29.
    29 Delete an article >db.articles.remove( { "_id" : 25 } ) WriteResult({ "nRemoved" : 1 }) > db.articles.remove( { "_id" : 25 } ) WriteResult({ "nRemoved" : 0 }) > db.articles.remove( { "_id" : { $lte : 5 }} ) WriteResult({ "nRemoved" : 6 }) • Deletion leaves holes • Dropping a collection is cheaper than deleting a large collection element by element
  • 30.
    30 A quick lookat users and articles again > db.users.findOne() { "_id" : ObjectId("57431c07b26a88bf060e10cb"), "username" : "USER_0", "lang" : "EN", "password" : "kGIxPxqKGJ", "karma" : 266 } > db.articles.findOne() { "_id" : 0, "body" : "hvJLnrrfZQurmtjPfUWbMhaQWbNjXLzjpuGLZjsxHXbUycmJVZTeOZesTnZtojThrebRcUoiYwivjpwG" , "postdate" : ISODate("2016-05-23T16:04:39.246Z"), "author" : "USER_0", "title" : "gpNIoPxpfTAxWjzAVoTJ" } >
  • 31.
    31 Find a user >db.users.find( { "username" : "USER_99" } ).explain() { "queryPlanner" : { "plannerVersion" : 1, "namespace" : "blog.users", "indexFilterSet" : false, "parsedQuery" : { "username" : { "$eq" : "USER_99" } }, "winningPlan" : { "stage" : "COLLSCAN", "filter" : { "username" : { "$eq" : "USER_99" } }, "direction" : "forward" }, }, }
  • 32.
    32 Find a user– execution stats > db.users.find( { "username" : "USER_99" } ).explain( "executionStats" ).executionStats { "executionSuccess" : true, "nReturned" : 1, "executionTimeMillis" : 412, "totalKeysExamined" : 0, "totalDocsExamined" : 1000000, "executionStages" : { "stage" : "COLLSCAN", "filter" : { "username" : { "$eq" : "USER_99" } }, "nReturned" : 1, "executionTimeMillisEstimate" : 302, "works" : 1000002, "advanced" : 1, "needTime" : 1000000, "needYield" : 0, "saveState" : 7823, "restoreState" : 7823, "isEOF" : 1, "invalidates" : 0, "direction" : "forward", "docsExamined" : 1000000 } }
  • 33.
    33 We need anindex > db.users.createIndex( { username : 1 } ) { "createdCollectionAutomatically" : false, "numIndexesBefore" : 1, "numIndexesAfter" : 2, "ok" : 1 } >
  • 34.
    34 Indexes Overview • Parameters –Background : Create an index in the background as opposed to locking the database – Unique : All keys in the collection must be unique. Duplicate key insertions will be rejected with an error. – Name : Explicitly name an index. Otherwise the index name is autogenerated from the index field. • Deleting an index – db.users.dropIndex({ “username” : 1 }) • List indexes – db.users.getIndexes()
  • 35.
    35 Query Plan ExecutionStages • COLLSCAN : for a collection scan • IXSCAN : for scanning index keys • FETCH : for retrieving documents • SHARD_MERGE : for merging results from shards
  • 36.
    36 Add an index >db.users.find( {"username" : "USER_999999”} ).explain("executionStats”).executionStats { "executionSuccess" : true, "nReturned" : 1, "executionTimeMillis" : 0, "totalKeysExamined" : 1, "totalDocsExamined" : 1, …
  • 37.
    37 Execution stage "executionStages" :{ "stage" : "FETCH", "nReturned" : 1, "executionTimeMillisEstimate" : 0, "docsExamined" : 1,, "inputStage" : { "stage" : "IXSCAN", "nReturned" : 1, "executionTimeMillisEstimate" : 0, "keyPattern" : { "username" : 1 }, "indexName" : "username_1", "isMultiKey" : false, "isUnique" : false, "isSparse" : false, "isPartial" : false, "indexVersion" : 1, "direction" : "forward", "indexBounds" : { "username" : [ "["USER_999999", "USER_999999"]" ] }, "keysExamined" : 1, "seenInvalidated" : 0 } } }
  • 38.
    38 What we havelearned • How to create a database and a collection • How to insert content into that collection • How to query the collection • How to update a document in place • How to delete a document • How to check the efficiency of an operation • How to add an index • How to check an index is being used in an operation
  • 39.
    39 Next Webinar :Introduction to Replica Sets • How to ensure your data is durable • How to recover from failures automatically • How to write safe client code Thursday, 2-Feb-2016, 11:00 am GMT.
  • 40.

Editor's Notes

  • #3 Who I am, how long have I been at MongoDB.
  • #13 This is javascript. Lazy evaluation. Databases and collections spring to life as needed.
  • #14 12 byte value.