Downloaded 46 times
![Discovery •Nodes discover each other using multicast. -Unicast is an option •Each cluster has an elected master node -Beware of split-brain discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["host1", "host2:port", "host3"] 34](https://image.slidesharecdn.com/devnexus2015-150314190910-conversion-gate01/75/Elasticsearch-DevNexus-2015-34-2048.jpg)
![Cluster State •Cluster State -Node Membership -Indices Settings -Shard Allocation Table -Shard State 39 cURL -XGET http://localhost:9200/_cluster/state?pretty=1' { "cluster_name" : "elasticsearch_royrusso", "version" : 27, "master_node" : "s3fpXfPKSFeUqo1MYZxSng", "blocks" : { }, "nodes" : { "s3fpXfPKSFeUqo1MYZxSng" : { "name" : "Bulldozer", "transport_address" : "inet[localhost/127.0.0.1:9300]", "attributes" : { } } }, "metadata" : { "templates" : { "logging_index_all" : { "template" : "logstash-09-*", "order" : 1, "settings" : { "index" : { "number_of_shards" : "2", "number_of_replicas" : "1" } }, "mappings" : { "date" : { "store" : false } } }, "logging_index" : { "template" : "logstash-*", "order" : 0, "settings" : { "index" : { "number_of_shards" : "2", "“number_of_replicas”" : "1" } }, "mappings" : { "“date”" : { "“store”" : true](https://image.slidesharecdn.com/devnexus2015-150314190910-conversion-gate01/75/Elasticsearch-DevNexus-2015-39-2048.jpg)
![REST API - Search •Free Text Search -URL Request •Complex Query 50 http://localhost:9200/imdb/movie/_search?q=scar* http://localhost:9200/imdb/movie/_search?q=scarface+OR+star http://localhost:9200/imdb/movie/_search?q=(scarface+OR+star)+AND+year:[1981+TO+1984]](https://image.slidesharecdn.com/devnexus2015-150314190910-conversion-gate01/75/Elasticsearch-DevNexus-2015-50-2048.jpg)
![REST API – Query DSL curl -XPOST 'localhost:9200/_search?pretty' -d '{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "scarface or star" } }, { "range" : { "year" : { "gte" : 1931 } } } ] } } }' 51](https://image.slidesharecdn.com/devnexus2015-150314190910-conversion-gate01/75/Elasticsearch-DevNexus-2015-51-2048.jpg)
![REST API – Query DSL •Boolean Query "bool":{ "must":[ { "match":{ "color":"blue" } }, { "match":{ "title":"shirt" } } ], "must_not":[ { "match":{ "size":"xxl" } } ], "should":[ { "match":{ "textile":"cotton" } 52](https://image.slidesharecdn.com/devnexus2015-150314190910-conversion-gate01/75/Elasticsearch-DevNexus-2015-52-2048.jpg)
![Filters •Filters recommended over Queries -Better cache support 54 curl -XGET 'http://localhost:9200/my_index/events/_search?pretty=1' -d ' { "from" : 0, "size" : 0, "query" : { "terms" : { "message" : [ "apples"], "minimum_should_match" : "1" } }, "post_filter" : { "terms" : { "userId" : [ "25476c6788ce", "g20d5470d7b4" ], "execution" : "or" } }, "sort": { "eventDate": { "order": "desc" }}, "explain" : false }';](https://image.slidesharecdn.com/devnexus2015-150314190910-conversion-gate01/75/Elasticsearch-DevNexus-2015-54-2048.jpg)
![Analyzers / Tokenizers 55 curl -XPUT ‘http://localhost:9200/my_index/' -d ' { "settings" : { "analysis" : { "analyzer" : { "str_search_analyzer" : { "tokenizer" : "keyword", "filter": ["lowercase"] }, "str_index_analyzer" : { "tokenizer" : "substring", "filter" : ["lowercase", "stop"] } }, "tokenizer" : { "substring" : { "type" : "edgeNgram", "min_gram" : "3", "max_gram" : "42", "token_chars" : ["letter", "digit"] } } curl -XPUT 'http://localhost:9200/my_index/events/_mapping' -d ' { "events" : { "properties" : { "eventId" : {"type" : "string", "store" : true, "index" : "not_analyzed" }, "userId" : {"type" : "string", "store" : false, "index" : "not_analyzed" }, "message" : { "type" : "string", "store" : false, "search_analyzer" : "str_search_analyzer", "index_analyzer" : "str_index_analyzer" } } } }';](https://image.slidesharecdn.com/devnexus2015-150314190910-conversion-gate01/75/Elasticsearch-DevNexus-2015-55-2048.jpg)
Uploaded byRoy Russo
Elasticsearch - DevNexus 2015
This document provides an introduction to Elasticsearch. It begins by introducing the speaker and their background. It then discusses what search is and how search engines work by using an inverted index to map tokens to documents. Elasticsearch is introduced as a search and analytics engine that is document-oriented, distributed, schema-free, and uses HTTP and JSON. It can be used for real-time search and analytics. The document discusses how Elasticsearch is based on Apache Lucene and can be run on multiple nodes in a cluster for high availability. It provides examples of using Elasticsearch for centralized logging, and discusses indexing, querying, and interacting with Elasticsearch via its RESTful API.
Elasticsearch - DevNexus 2015
- 1. Introduction To ElasticSearch Real-TimeSearch and Analytics Roy Russo, DevNexus, 2015
- 2. Who Am I •RoyRusso •VP Engineering, Predikto •Co-Author - Elasticsearch in Action -Due ~April 2015. •ElasticHQ.org •Other (*) 2 * Silverpop, JBoss, AltiSource Labs
- 3. Why Am IHere? •What is Search •What is Elasticsearch •Real-World Use •Scale Out •Interacting with Elasticsearch 3
- 4. Search is about filteringinformation and determining relevance. 4
- 5. How does aSearch Engine Work? 5 Select * FROM make WHERE name LIKE ‘%Tesla%’
- 6. Search Engines useMagic 6 Where Magic == Inverted Index It’s FM!
- 7. Inverted Index •Take somedocuments •Tokenize them •Find unique tokens •Map tokens to documents 7 apple oranges peach Document 1 Document 2 Document 3 Document 4 Document 5 Document 6
- 8. Inverted Index 8 apples orangespeach Document 1 Document 2 Document 3 Document 4 Document 5 Document 6 Search for “apple peach”
- 9. Relevance •How many tokensper document? •How many tokens relative to the number of total tokens in the document? •What is the frequency of token across all documents? 9
- 10. Relevance in Elasticsearch •AtSearch Time •At Index Time •Term Frequency -Term / Document •Inverse Document Frequency (IDF) -Term / All Documents in the collection •Field-Length Norm 10
- 11.
- 12. Elasticsearch is… •Search andAnalytics engine •Document Store -Every field is indexed/searchable •Distributed 12
- 13. What Elasticsearch isnot •Key-Value Store -Redis, Riak •Column Family Store -C*, HBase •Graph Database -Neo4J 13
- 14. ElasticSearch in aNutshell •Based on Apache Lucene •Distributed •Document-Oriented •Schema free •HTTP + JSON •(Near) Real-time search •Ecosystem -Hosting, Monitoring Apps, Clients (SDK) 14
- 15. Where can Iget it? •Free and Open Source •https://www.elastic.co/ •https://github.com/elastic/elasticsearch •Backed by a Company, Elastic -Training -Support -Auth/AuthZ -Marvel for Monitoring 15
- 16. How do Irun it? •Download it -https://www.elastic.co/downloads •bin/elasticsearch •http://localhost:9200 16 { status: 200, name: "Tesla", cluster_name: "elasticsearch_royrusso", version: { number: "1.4.2", build_hash: "927caff6f05403e936c20bf4529f144f0c89fd8c", build_timestamp: "2014-12-16T14:11:12Z", build_snapshot: false, lucene_version: "4.10.2" }, tagline: "You Know, for Search" }
- 17. Elasticsearch requires Java.* 17 * You have 5 seconds to whine about it and then shutup.
- 18.
- 19. ElasticSearch for Centralized Logs •Logstash+ ElasticSearch + Kibana (ELK) •Well… and then there’s Loggly 19 “Netflix is a Log generating company that happens to stream movies”
- 20.
- 21. Elasticsearch at Predikto 21 •WriteFrom Spark to Elasticsearch •Query from Spark to Elasticsearch •Visualize
- 22.
- 23. Based on ApacheLucene •Free and Open Source •Started in 1999 •Created by Doug Cutting •What’s it do? -Tokenizing -Locations -Relevance scoring -Filtering -Text search -Date Parsing 23
- 24. Elasticsearch is aDocument Store 24
- 25. Document Store •Like MongoDBand CouchDB •Document DBs: -JSON documents -Collections of key-value collections -Nesting -Versioned 25
- 26. What is adocument? 26 { "genre": "Crime", “language": "English", "country": "USA", "runtime": 170, "title": "Scarface", "year": 1983 }
- 27. Modeled in JSON 27 { "genre":"Crime", "language": "English", "country": "USA", "runtime": 170, "title": "Scarface", "year": 1983 } { "_index": "imdb", "_type": "movie", "_id": "u17o8zy9RcKg6SjQZqQ4Ow", "_version": 1, "_source": { "genre": "Crime", "language": "English", "country": "USA", "runtime": 170, "title": "Scarface", "year": 1983 } }
- 28. Schema-Free •Dynamic Mapping -Elasticsearch guessesthe data-types (string, int, float…) 28 "imdb": { "movie": { "properties": { "country": { "type": "string“, “store”:true, “index”:false }, "genre": { "type": "string“, "null_value" : "na“, “store”:false, “index:true }, "year": { "type": "long" } } } }
- 29.
- 30. Terminology 30 MySQL Elasticsearch Database Index TableType Row Document Column Field Schema Mapping Index (Everything is indexed) SQL Query DSL •Cluster: 1..N Nodes w/ same Cluster Name •Node: One ElasticSearch instance (1 java proc) •Shard = One Lucene instance -0 or more replicas
- 31. High Availability •No needfor load balancer •Different Node Types •Indices are Sharded •Replica shards on different Nodes •Automatic Master election & failover 31
- 32. About Indices /Shards 32 $ curl -XPUT 'http://localhost:9200/twitter/' -d '{ "settings" : { "index" : { "number_of_shards" : 3, "number_of_replicas" : 2 } } }'
- 33. Cluster Topology 33 A1 A2B2B2 B1 B3 B1 A1 A2 B3 4 Node Cluster Index A: 2 Shards & 1 Replica Index B: 3 Shards & 1 Replica
- 34. Discovery •Nodes discover eachother using multicast. -Unicast is an option •Each cluster has an elected master node -Beware of split-brain discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["host1", "host2:port", "host3"] 34
- 35. Nodes •Master node handlescluster-wide (Meta-API) events: -Node participation -New indices create/delete -Re-Allocation of shards •Data Nodes -Indexing / Searching operations •Client Nodes -REST calls -Light-weight load balancers 35 node.data | node.master
- 36. The Basics -Shards •Primary Shard: -First time Indexing -Index has 1..N primary shards (default: 5) -# Not changeable once index created •Replica Shard: -Copy of the primary shard -# Can be changed later -Each primary has 0..N replicas - HA: • Promoted to primary if primary fails • Get/Search handled by primary||replica36
- 37. Shard Auto-Allocation •Shard Phases: -Unassigned -Initializing -Started -Relocating 37 Node1 0P 1R Node 2 1P 0R Node 2 0R Add a Node: Shards Relocate
- 38. Allocation Awareness •Shard AllocationAwareness -cluster.routing.allocation.awareness.attributes: rack -Shards RELOCATE to even distribution -Primary & Replica will NOT be on the same rack value. 38
- 39. Cluster State •Cluster State -NodeMembership -Indices Settings -Shard Allocation Table -Shard State 39 cURL -XGET http://localhost:9200/_cluster/state?pretty=1' { "cluster_name" : "elasticsearch_royrusso", "version" : 27, "master_node" : "s3fpXfPKSFeUqo1MYZxSng", "blocks" : { }, "nodes" : { "s3fpXfPKSFeUqo1MYZxSng" : { "name" : "Bulldozer", "transport_address" : "inet[localhost/127.0.0.1:9300]", "attributes" : { } } }, "metadata" : { "templates" : { "logging_index_all" : { "template" : "logstash-09-*", "order" : 1, "settings" : { "index" : { "number_of_shards" : "2", "number_of_replicas" : "1" } }, "mappings" : { "date" : { "store" : false } } }, "logging_index" : { "template" : "logstash-*", "order" : 0, "settings" : { "index" : { "number_of_shards" : "2", "“number_of_replicas”" : "1" } }, "mappings" : { "“date”" : { "“store”" : true
- 40.
- 41. REST •HTTP Verbs: GET,POST, PUT, DELETE •JSON •_cat API 41 % curl '192.168.56.10:9200/_cat/health?v&ts=0' cluster status nodeTotal nodeData shards pri relo init unassign foo green 3 3 3 3 0 0 0
- 42.
- 43. Create a Document 43 curl-XPOST 'http://127.0.0.1:9200/imdb/movie' -d ' { "genre": "Crime", "language": "English", "country": "USA", "runtime": 170, "title": "Scarface", "year": 1983 }'; { "_index":"imdb", "_type":"movie", "_id":"AUwGeWib1u4mCngDYT7y", "_version":1, "created":true }
- 44. Of note… 44 curl -XPOST'http://127.0.0.1:9200/imdb/movie' -d ' { "genre": "Crime", "language": "English", "country": "USA", "runtime": 170, "title": "Scarface", "year": 1983 }'; { "_index":"imdb", "_type":"movie", "_id":"AUwGeWib1u4mCngDYT7y", "_version":1, "created":true } Auto-creates Index & Type Auto-Gen ID Auto-Version
- 45. Get a Document 45 curl-XGET 'http://127.0.0.1:9200/imdb/movie/AUwGeWib1u4mCngDYT7y'; { "_index":"imdb", "_type":"movie", "_id":"AUwGeWib1u4mCngDYT7y", "_version":1, "found":true, "_source": { "genre": "Crime", "language": "English", "country": "USA", "runtime": 170, "title": "Scarface", "year": 1983 } }
- 46. Update a Document 46 curl-XPUT 'http://127.0.0.1:9200/imdb/movie/AUwGeWib1u4mCngDYT7y' -d ' { "genre": "Crime", "language": "English", "country": "USA", "runtime": 180, "title": "Scarface", "year": 1983 }'; { "_index":"imdb", "_type":"movie", "_id":"AUwGeWib1u4mCngDYT7y", "_version":2, "created":false } * More like an Upsert.
- 47. Delete a Document 47 curl-XDELETE 'http://127.0.0.1:9200/imdb/movie/AUwGeWib1u4mCngDYT7y' { “found":true, “_index":"imdb", “_type":"movie", “_id":"AUwGeWib1u4mCngDYT7y", “_version":2 }
- 48. You can also… •Partialdocument updating •Specify Version •Specify ID •Multi-Get API •Exists API •Bulk API 48
- 49. How Searching Works •Howit works: -Search request hits a node -Node broadcasts to every shard in the index -Each shard performs query -Each shard returns metadata about results -Node merges results and scores them -Node requests documents from shards -Results merged, sorted, and returned to client. 49
- 50. REST API -Search •Free Text Search -URL Request •Complex Query 50 http://localhost:9200/imdb/movie/_search?q=scar* http://localhost:9200/imdb/movie/_search?q=scarface+OR+star http://localhost:9200/imdb/movie/_search?q=(scarface+OR+star)+AND+year:[1981+TO+1984]
- 51. REST API –Query DSL curl -XPOST 'localhost:9200/_search?pretty' -d '{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "scarface or star" } }, { "range" : { "year" : { "gte" : 1931 } } } ] } } }' 51
- 52. REST API –Query DSL •Boolean Query "bool":{ "must":[ { "match":{ "color":"blue" } }, { "match":{ "title":"shirt" } } ], "must_not":[ { "match":{ "size":"xxl" } } ], "should":[ { "match":{ "textile":"cotton" } 52
- 53. REST API –Query DSL •Range Query -Numeric / Date Types •Prefix/Wildcard Query -Match on partial terms •RegExp Query •Geo_bbox -Bounding box filter •Geo_distance -Geo_distance_range { "range":{ "founded_year":{ "gte":1990, "lt":2000 } } } 53
- 54. Filters •Filters recommended overQueries -Better cache support 54 curl -XGET 'http://localhost:9200/my_index/events/_search?pretty=1' -d ' { "from" : 0, "size" : 0, "query" : { "terms" : { "message" : [ "apples"], "minimum_should_match" : "1" } }, "post_filter" : { "terms" : { "userId" : [ "25476c6788ce", "g20d5470d7b4" ], "execution" : "or" } }, "sort": { "eventDate": { "order": "desc" }}, "explain" : false }';
- 55. Analyzers / Tokenizers 55 curl-XPUT ‘http://localhost:9200/my_index/' -d ' { "settings" : { "analysis" : { "analyzer" : { "str_search_analyzer" : { "tokenizer" : "keyword", "filter": ["lowercase"] }, "str_index_analyzer" : { "tokenizer" : "substring", "filter" : ["lowercase", "stop"] } }, "tokenizer" : { "substring" : { "type" : "edgeNgram", "min_gram" : "3", "max_gram" : "42", "token_chars" : ["letter", "digit"] } } curl -XPUT 'http://localhost:9200/my_index/events/_mapping' -d ' { "events" : { "properties" : { "eventId" : {"type" : "string", "store" : true, "index" : "not_analyzed" }, "userId" : {"type" : "string", "store" : false, "index" : "not_analyzed" }, "message" : { "type" : "string", "store" : false, "search_analyzer" : "str_search_analyzer", "index_analyzer" : "str_index_analyzer" } } } }';
- 56.
- 57. Clients •Client list: http://www.elasticsearch.org/guide/clients/ -Java (Node)Client, JS, PHP, Perl, Python, Ruby •Spring Data: -Uses TransportClient -Implementation of ElasticsearchRepository aligns with generic Repository interfaces 57
- 58.
- 59.