Downloaded 71 times
Graph Processing with Apache TinkerPop
- 1. Graph Processing with ApacheTinkerPop (incubating) Jason Plurad Software Engineer, IBM | Committer, Apache TinkerPop
- 2. • Project Update • Graph Landscape •A Graph Problem • Hands-On Graph http://tinkerpop.apache.org
- 3. About Me • Twitter @pluradj • GitHub @pluradj •Open channels – TinkerPop mailing lists – Titan mailing list – Stack Overflow
- 4. (Apache) TinkerPop (incubating) • 2009: Inception •2012: TinkerPop 2 • 2015: Apache Incubator • 2016: Top Level Project? – TLP VOTE passed! – Waiting on board meeting to establish TLP
- 5. Podling Releases • 3.0 –Major refactor, Java 8 lambda expressions, Gremlin Server, OLAP graph computers • 3.1 – Hadoop 2 support, persisted RDDs • 3.2 – OLAP job chaining, OLAP graph filters, performance improvements
- 6. Common graph data domains • Social Network Analysis • Configuration Management Database •Master Data Management • Recommendation Engines • Knowledge Graphs • Internet of Things
- 7. Property Graph and Gremlin • Structure – Vertex –Edge – Properties • Gremlin – Domain specific language (DSL) for graph – Data flow: forward and backward – Traversal Steps – Bindings for non-JVM languages
- 8.
- 9. Graph Landscape • Graph database vs Graph processor – OLTP vs OLAP –Neighborhood vs whole graph • Multi-model: not the only store in your app
- 10. IBM Graph (Beta) •Managed Graph-as-a-Service (OLTP) • Focus on your data, not install and operations • #sleepMore http://ibm.biz/IBMGraph
- 11. What is this? module.exports = xxxxxxx; functionxxxxxxx (str, len, ch) { str = String(str); var i = -1; if (!ch && ch !== 0) ch = ' '; len = len - str.length; while (++i < len) { str = ch + str; } return str; }
- 12. A Graph Problem: Dependency Management • On March 22, 2016 npm broke the Internet •Left-pad was unpublished – 11 lines of code – WTFPL license – Hundreds of breaking builds per minute – http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm • Are we safe with Apache?
- 13. Questions for the graph • Which dependencies are at risk? • Which ones should be refactored to avoid? •Risk factors – Unsuitable license – Single developer – Too little code / Too much code – Changes too frequently / Code is stagnant – Nobody else is using it
- 14.
- 15. Titan (Aurelius) • Pick a graph database for OLTP… – Apache license but not in ASF •Code has stagnated in the open – DataStax Enterprise (DSE) Graph – Wide open opportunities • Genesis Graph is up next! • Apache S2Graph (incubating) • Apache Flink (Gelly) • Apache Solr (GraphQuery)
- 16. Apache Spark or Apache Giraph • Pick a graph processor for OLAP… – Spark is the new hotness –Giraph is better suited for gigantic graphs • By using Apache TinkerPop and Gremlin, we can use either one seamlessly
- 17. Vagrant and Virtualbox • Developers don’t always get keys to the cloud • Virtual machines to the rescue –Host: 16 GB RAM or more – 3-4 VMs with 3 GB RAM • Prove out your graph algorithms on a small data set before wasting time on a big data set
- 18. Apache Ambari • Simple install for Apache Hadoop and related Apache big data packages – HDFS, YARN, MapReduce, HBase, Spark, etc •Management and monitoring dashboard • Enables integration of other software
- 19. Getting the data • NPM registry runs on Apache CouchDB • Replication in Apache CouchDBis awesome – https://skimdb.npmjs.com/registry
- 20. Transform the data • Apache CouchDB is a document store •Dependencies are graph data • Other things can be too – Users – Keywords – License • Graph model depends on the questions you want to ask of the graph
- 21.
- 22.
- 23.
- 24.
- 25. Out of the box Vertex Programs • Traversal • BulkLoader •BulkDumper • PageRank • PeerPressure
- 26.
- 27. OLAP Traversal Sources >graph = GraphFactory.open('conf/npmgraph- olap.properties') > g = graph.traversal().withComputer(SparkGraphComputer) > g = graph.traversal().withComputer(GiraphGraphComputer) Graph Statistics via TraversalVertexProgram > g.V().count() // vertex count > g.E().count() // edge count > g.V().label().groupCount() // vertex label distribution > g.E().label().groupCount() // edge label distribution > g.V().properties().key().groupCount() // vertex property distribution
- 28. Next stop? More data! • Graphs are for connecting data! • Consume data from GitHub –User data – Static code analysis – Code usage analysis • Consume data from Twitter – Trending news – Security alerts
- 29. Summary • Apache TinkerPop is for graph computing •OLTP vs OLAP is an important distinction – Gremlin allows you to seamless bridge the two • Graph thinking is different than relational – Is the future multi-model? • Many opportunities to innovate in this space
- 30. Acknowledgements • Marko Rodriguez – Gremlin language, Gremlin OLAP •Ketrina Yim – Illustrator, creator of Gremlin and friends • Stephen Mallette – TinkerPop release manager, Gremlin applications • Daniel Kuppitz – Gremlin language guru • David Robinson – Big data, multi-model architect/developer
- 31.
- 32.