Download as PDF, PPTX
![@MargrietGr Cloudant is "schemaless" { "_id": "1", "firstname": "John", "lastname": "Smith", "dob": "1970-01-01", "email": "john.smith@gmail.com", "confirmed": true, "tags": ["tall", "glasses"] }](https://image.slidesharecdn.com/170227watsonopendata-170228171041/75/Introduction-to-the-IBM-Watson-Data-Platform-24-2048.jpg)
![@MargrietGr Cloudant is "schemaless" { "_id": "1", "firstname": "John", "lastname": "Smith", "dob": "1970-01-01", "email": "john.smith@gmail.com", "confirmed": true, "tags": ["tall", "glasses"], "address" : { "number": 14, "street": "Front Street", "town": "Luton", "postcode": "LU1 1AB" } }](https://image.slidesharecdn.com/170227watsonopendata-170228171041/75/Introduction-to-the-IBM-Watson-Data-Platform-25-2048.jpg)
Uploaded byMargriet Groenendijk
Introduction to the IBM Watson Data Platform
The document discusses IBM Watson Data Platform and its features, focusing on the Cloudant database and its schemaless design for storing JSON documents. It emphasizes the platform's fault tolerance, scalability, and ability to integrate with various data sources and services like Apache Spark and DashDB. Additionally, it provides examples of utilizing open data and APIs, along with steps for creating projects within the IBM Data Science Experience.
Introduction to the IBM Watson Data Platform
- 1. IBM Watson DataPlatform and Open Data 27 February 2017 Margriet Groenendijk | Developer Advocate | IBM Watson Data Platform @MargrietGr https://medium.com/ibm-watson-data-lab
- 2. @MargrietGr About me Developer Advocate, Datascientist Previous Research Fellow at University of Exeter, UK PhD at VU University Amsterdam, the Netherlands
- 3. @MargrietGr IBM Watson DataPlatform Connect Discover Accelerate
- 4. @MargrietGr IBM Watson DataPlatform
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 17.
- 18. @MargrietGr Example : WatsonTone Analyser
- 19. @MargrietGr Emotion Language style Social propensities Analyze how youare coming across to others
- 20.
- 21. @MargrietGr Cloudant is adatabase id firstname lastname dob 1 John Smith 1970-01-01 2 Kate Jones 1971-12-25 { "_id": "1", "firstname": "John", "lastname": "Smith", "dob": "1970-01-01" }
- 22. @MargrietGr Cloudant is "schemaless" { "_id":"1", "firstname": "John", "lastname": "Smith", "dob": "1970-01-01", "email": "john.smith@gmail.com" }
- 23. @MargrietGr Cloudant is "schemaless" { "_id":"1", "firstname": "John", "lastname": "Smith", "dob": "1970-01-01", "email": "john.smith@gmail.com", "confirmed": true }
- 24. @MargrietGr Cloudant is "schemaless" { "_id":"1", "firstname": "John", "lastname": "Smith", "dob": "1970-01-01", "email": "john.smith@gmail.com", "confirmed": true, "tags": ["tall", "glasses"] }
- 25. @MargrietGr Cloudant is "schemaless" { "_id":"1", "firstname": "John", "lastname": "Smith", "dob": "1970-01-01", "email": "john.smith@gmail.com", "confirmed": true, "tags": ["tall", "glasses"], "address" : { "number": 14, "street": "Front Street", "town": "Luton", "postcode": "LU1 1AB" } }
- 26. @MargrietGr Cloudant is builtfor the web ▪Store JSON Documents ▪Speaks an HTTP API ▪Lives on the web
- 27. @MargrietGr Cloudant is faulttolerant
- 28. @MargrietGr Cloudant is faulttolerant
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38. @MargrietGr Open Street MapData IBM Cloudant Use from anywhere! Daily updates VM daily cron Python script Always up to date! Currently 12,467,460 POIs
- 39. @MargrietGr wget -c http://download.geofabrik.de/europe/netherlands- latest.osm.pbf Severaldata sources - world, continent, country, city or a user defined box Several data formats for which free to use conversion tools exist - pbf, osm, json, shp Example:
- 40. @MargrietGr Extract the POIswith osmosis osmosis --read-pbf netherlands-latest.osm.pbf --tf accept-nodes aerialway=station aeroway=aerodrome,helipad,heliport amenity=* craft=* emergency=* highway=bus_stop,rest_area,services historic=* leisure=* office=* public_transport=stop_position,stop_area shop=* tourism=* --tf reject-ways --tf reject-relations --write-xml netherlands.nodes.osm (easy to install with brew on Mac)
- 41. @MargrietGr Some cleaning upwith osmconvert Convert from osm to json format with ogr2ogr osmconvert $netherlands.nodes.osm --drop-ways --drop-author --drop-relations --drop-versions >$netherlands.poi.osm ogr2ogr -f GeoJSON $netherlands.poi.json $netherlands.poi.osm points
- 42. @MargrietGr Upload to Cloudantwith couchimport export COUCH_URL="https:// username:password@username.cloudant.com" cat $netherlands.poi.json | couchimport --db poi-$netherlands --type json --jsonpath "features.*" https://github.com/glynnbird/couchimport IBM Cloudant
- 43.
- 44.
- 45. @MargrietGr UK Crime Datafrom https://data.police.uk/data/
- 46.
- 47.
- 48.
- 49. @MargrietGr Add the dashDBservice in Bluemix Add a service Search for dashDB
- 50.
- 51. @MargrietGr 3 1 2 posted:2016-08-01,2016-10-01 followers_count:3000 friends_count: 3000 (weatherOR sun OR sunny OR rain OR hail OR storm OR rainy OR drought OR flood OR hurricane OR tornado OR cold OR snow OR drizzle OR cloudy OR thunder OR lightning OR wind OR windy OR heatwave) REST API docs: https://new-console.ng.bluemix.net/docs/ services/Twitter/ twitter_rest_apis.html#rest_apis Search for tweets 4 Select table Use an existing service
- 52.
- 53.
- 54.
- 55.
- 56. @MargrietGr RDDs : ResilientDistributed Datasets Data does not have to fit on a single machine Data is separated into partitions Creation of RDDs Load an external dataset Distribute a collection of objects Transformations construct a new RDD from a previous one (lazy!) Actions compute a result based on an RDD
- 57. @MargrietGr Load tweets fromdashDB with Spark SQL
- 58. @MargrietGr Clean data, summariseand load into pandas DataFrame
- 59.
- 60.
- 61.
- 62. Getting started ▪ Goto datascience.ibm.com and sign in with your Bluemix account when you have one, else sign up for one at the top right of the screen
- 63. Create a project ▪Create New project, click on the link in top of the screen ▪ Or go to the My Projects in the menu on the left of the screen and click Create New Project here
- 64. Create a project ▪Name the Project ▪ Choose a Spark Service ▪ Choose an Object Storage ▪ Click Create
- 65. Add collaborators ▪ Clickadd collaborator ▪ Search for your project members ▪ Select Permission
- 66. Add a notebook ▪Click add notebooks
- 67. Add a notebook ▪Click add notebooks ▪ Pick your favourite: ▪ Python 2 ▪ Scala ▪ R ▪ Choose Spark 1.6 or 2.0 ▪ Click Create Notebook
- 68. Let’s write somecode ▪ Click the pen icon to start adding code (edit mode) ▪ When collaborating only one person can edit, others can add comments to the notebook when in view mode
- 69. @MargrietGr Example : Bristolopen data
- 70.
- 71.
- 72.
- 73. @MargrietGr IBM Watson DataPlatform Bluemix Data storage Apps Watson APIs Weather Data Science Experience Watson Machine Learning Watson Analytics
- 74.