An introduction to Elasticsearch's advanced relevance ranking toolbox

An Introduction to Elasticsearch’s Advanced Relevance Ranking Toolbox The hallmark of a great search experience is always delivering the most relevant results, quickly, to every user. The difficulty lies behind the scenes in making that happen elegantly and at a scale. From App Search’s intuitive drag and drop interface to the advanced relevance capabilities built into the core of Elasticsearch — Elastic offers a range of tools for developers to tune relevance ranking and create incredible search experiences. In this session, we’ll explore some of Elasticsearch’s advanced relevance ranking features, such as dense vector fields, BM25f, ranking evaluation, and more. Plus we’ll give you some ideas for how these features are being used by other Elastic users to create world-class, category defining search experiences. Opening suggestion: Show poor search experience (1). Show amazing search experience (2). Show how to achieve amazing search experience. First with App Search (quick) (3). Then, pivot, for more complex situations or for more control / power over whatever - this is the list of features on the next slide (4). Abstract

Key topics • Performance - separating docs from aggs • Ranking evaluation API • BM25f • Synonyms administration via API • Distance feature query • Script score query • Rank_feature • Pinned query • Vector similarity for reranking • Tuning params with ML LTR I will likely need to cancel some of the slides since there’s too much here to fit in 25 min.

3 Advanced Relevance Ranking Gilad Gal Product Manager, Elasticsearch ElasticON

Agenda Slide Use color to highlight Performance when ranking Relevance ranking evaluation Specific query types and their usefulness Vector similarity in relevance ranking Relevance ranking research and roadmap 1 2 3 4 5

Docs first & stats separately - better performance Separate the query that returns the results from the stats/agg query A Z A X A Y A X B Y B Z B X B Y B X A B Z Top Ranking A Z A X A Y A X B Y B Z B X B Y B X A B Z Result count Skipping uncompetitive results Full index scan Review the results while facets and total count load Faster UI response where it matters!

Docs first & stats separately - better performance Separate the query that returns the results from the stats/agg query If no aggregation or hit count (beyond configurable k results) Performance impact: Term queries: 3x to 7x faster Conjunction (and) queries: 3% to 7x faster Disjunction (or) queries: 8% to 15x faster Running Lucene's benchmark suite

• Set of representative queries – Representative ! Frequent – Larger is better, e.g.: – 10^3 is probably reasonable – 10^1 is probably too little • Set of correct results per query – Set size per query that covers most users’ interest, e.g. 510 – Examine more results; potentially expand the query • Some metrics use score and “not in set”, others require a set ! ! ! ? Tagged Data Curation ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ! ! ! ? ! ! !

Supported Metrics • Precision at K • Mean Reciprocal Rank • Discounted Cumulative Gain

Precision at K • Number of correct results within the first k results divided by k • For one query: • For a set of |Q|queries: Supported Metrics

Mean Reciprocal Rank • |Q|  number of queries in test set • Ranki := position of the first correct result for query i Supported Metrics

• Cumulative Gain – Sum of ranking scores till position P – Problem: different sorting of records 1..P would yield the same CGP – Relating to – Position within k (i indicates the position) – Ranking score (reli:= ranking score of record in position i) • Higher positions are more influential • Average DCG across multiple queries – Ideally keep the same number of results in the result set – Issue: different queries have different impact due to average higher/lower score Discounted Cumulative Gain

Supported Metrics • Precision at K - Simple to understand • Mean Reciprocal Rank - First correct answer • Discounted Cumulative Gain - Relevance ranking order

BM25F The oversimplified history of term frequency algorithm TF/IDF Frequent terms are less indicative than rare terms TF Rank by frequency of the query term in the document BM25 Adjust for lengthy text being more probable to contain the query term and limits TF impact for common words BM25F Adjust for multiple fields with varying field importance

Combined Fields Query • Benefit: – Improved relevance ranking when ranking on multiple fields • How – Treat as if unified into a single field – Support boosting per field – Use BM25F Using BM25F

The importance of the record’s intrinsic value • Records differ in their value, regardless of query • Typical examples: – URL length – User reviews score – Popularity – Number of links to / citations of – Algorithms: • Hubs and authorities (HITS • PageRank Static but crucial Pre-calculated numbers

Rank Feature Query • New datatypes: – rank_feature – rank_features (vector of rank_feature variables) • Add to the relevance ranking score • Allow normalization prior to adding: – Saturation – Logarithm – Sigmoid

Script Score Query • Based on all fields • Including _score • Normalize – Saturation – Logarithm – Sigmoid • Write your own painless or use efficient ready-made functions Define the function for relevance ranking score

Distance Feature Query • Benefits: – Newer data is frequently more relevant – Geographical proximity as a relevance indicator in non-geo searches • Easy to use, sophisticated under the hood: – Designed to leverage top-k faster queries for improved performance – Supports normalization through Saturation Geo and Time Proximity: closer is frequently more relevant

Result Pinning • Pinning the highest ranking results per query • Facilitates tailoring SERPs for frequent queries What is it good for?

Vector Similarity • Vector distance as similarity indicator – Embedding for natural language processing – Feature vectors for image analysis (using convolutional neural networks) • Different types of distances/similarities used for different needs – Dot Product – Cosine – Manhattan – Euclidean What is it good for? x1 x2 x2048 ... rather than this This

Vector Manhattan & Euclidean Distances • Euclidean: • Manhattan: p = [p1 , p2 , p3 , …, pn ] q = [q1 , q2 , q3 , …, qn ] What is it?

Vector Similarity Functions Use deep learning models for relevance ranking Natural Language Processing Model Text documents Queries Documents with vectors Text documents Documents Queries Queries a1 a2 … an a1 a2 … an a1 a2 … an Text Convolutional Neural Network Image Embeddings Feature vectors a1 a2 … an a1 a2 … an For Example: Search by vector similarity & text/keywords/num Ingest documents

Agenda Performance when ranking Relevance ranking evaluation Specific query types and their usefulness Vector similarity in relevance ranking Relevance ranking research and roadmap 1 2 3 4 5

Vector Similarity Usage with Elasticsearch Use deep learning models for relevance ranking • Image search – CNN (convolutional neural networks) • Natural language processing – BERT, Word2vec, GloVe https://image-search.eden.elstc.co/home

Approximate Nearest Neighbours Vector similarity performance and scalability • Vector similarity as an addition to term frequency ranking works great – Very typical situation – Scales well – Limit the set using term frequency – Re-rank based on vector similarity • ANN for performance and scalability – Ranking only by vector similarity – Extending the term frequency based set with vector similarity results • Based on HNSW support in Lucene Tentative Roadmap Layer 2 Layer 1 Layer 0

Embeddings Creation Translating text into vectors • Pre-trained deep learning model support, e.g. for BERT • Out of the box translation of text into embeddings – Docs at ingest time – Queries at query time • Useful for NLP – Search by meaning – Venture beyond bag of keywords – E.g. for helpdesk and documentation search Tentative Roadmap Source: Siobhán Grayson in Wikimedia.org - https://commons.wikimedia.org/wiki/File:TSNE_visualisation_of_word_e mbeddings_generated_using_19th_century_literature.pdf

Tuning Ranking With Machine Learning Configuring ranking parameters using ranking evaluation • Tuning relevance ranking using machine learning – Field boosting (see “Improving search relevance with data-driven query optimization” by Josh Devins) – BM25 parameters – Normalization parameters • Possible today using ranking evaluation API, but run outside Elasticsearch • Considered for the roadmap including a queries curation mechanism to obtain a tagged set for training/testing/validation Tentative Roadmap

Summary • Lots of mechanism • Designed to work together • To facilitate the relevance ranking that best suits your need BM25F & combined_fields query rank_feature query Normalization: saturation, logarithm, or sigmoid distance_feature query pinned query Ranking by vector similarity Precision at K Mean Reciprocal Rank Discounted Cumulative Gain Embedding creation using pre-trained models Tuning ranking with learning to rank Approximate Nearest Neighbours using HNSW

34 ElasticON Solution Series Presenter Name Title, Team Name Event Name | Region

35 Quote slide ipsum do sed Lorem ipsum dolar sit amet, consectur adipiscing elit sed do eiumod ipsum epsum dolar sit  First, Lastname | Title, Company Name

36 Place a quote from someone really, really important and it will shrink to ﬁt this space… Author Name Here “

38 Transition Slide Title Short and Sweet

44 Transition Slide Title Goes Here and Can Be a Few Lines Long Subtitle goes here in sentence case

45 Transition Slide Title Goes Here and Can Be a Few Lines Long Subtitle goes here in sentence case

46 Safe Harbor Statement This presentation includes forward-looking statements that are subject to risks and uncertainties. Actual results may diﬀer materially as a result of various risk factors included in the reports on the Forms 10-K, 10-Q, and 8-K, and in other ﬁlings we make with the SEC from time to time. Elastic undertakes no obligation to update any of these forward-looking statements.

47 Safe Harbor Statement This presentation includes forward-looking statements that are subject to risks and uncertainties. Actual results may diﬀer materially as a result of various risk factors included in the reports on the Forms 10-K, 10-Q, and 8-K, and in other ﬁlings we make with the SEC from time to time. Elastic undertakes no obligation to update any of these forward-looking statements.

48 Closing slide This presentation includes forward-looking statements that are subject to risks and uncertainties. Actual results may diﬀer materially as a result of various risk factors included in the reports on the Forms 10-K, 10-Q, and 8-K, and in other ﬁlings we make with the SEC from time to time. Elastic undertakes no obligation to update any of these forward-looking statements.

49 Closing slide This presentation includes forward-looking statements that are subject to risks and uncertainties. Actual results may diﬀer materially as a result of various risk factors included in the reports on the Forms 10-K, 10-Q, and 8-K, and in other ﬁlings we make with the SEC from time to time. Elastic undertakes no obligation to update any of these forward-looking statements.

Bullet title (Inter 24 pt) • Try to keep your use of bullet slides to a minimum • Be creative and think visually • If you need to source something copy and paste the text box at the bottom left onto your page Subtitle sentence case (Inter 18pt)

Bullet slide title treatment can be up to two lines in length (Inter bold 24 pt) Subtitle sentence case (Inter 18pt) Bullet slide title treatment can be up to two lines in length (Inter bold 24 pt) • Bullets are sentence case (Inter 18pt) – Second-line bullets are Inter 14pt • Third-line bullets are Inter 12pt • Limit the number of bullets on a slide • Text highlights are orange, but not underlined • Try not to go below the recommended font sizes

Bullet slide title treatment can be up to two lines in length (Inter bold 24 pt) • Bullets are sentence case (Inter 18pt) ‒ Second-line bullets are Inter 14pt ‒ Third-line bullets are Inter 12pt • Limit the number of bullets on a slide • Text highlights are orange, but not underlined • Try not to go below the recommended font sizes Subtitle sentence case (Inter 18pt)

Place a quote from someone really, really important and it will shrink to fit this space… Author Name Here

Author Name Here Place a quote from someone really, really important and it will shrink to fit this space…

Chart Slide With Multiple Colors Sub-title or chart title here in sentence case

Pie Chart Slide With Multiple Colors Sub-title or chart title here in sentence case 62% Supporting text goes here under the number 62% Supporting text goes here under the number

Pie Chart Slide With Multiple Colors Sub-title or chart title here in sentence case

Transition Slide Title Goes Here and Can Be a Few Lines Long Subtitle goes here in sentence case

Transition Slide Title Short and Sweet

1M 1M 1M HEADER HERE Supporting text goes here under the number HEADER HERE Supporting text goes here under the number HEADER HERE Supporting text goes here under the number Big Number Treatment

1M 1M 1M HEADER HERE Supporting text goes here under the number HEADER HERE Supporting text goes here under the number HEADER HERE Supporting text goes here under the number Big Number Treatment (Dark Mode)

Table Layout Treatment Subtitle text placeholder sentence case HEADER HEADER HEADER HEADER Information Information Information Information Information Information Information Information Information Information Information Information Information Information Information Information Information Information Information Information Information Information Information Information Option 1

Please use this area for content, screen shot, or quote; the next few slide show examples

We mine and analyze 4 billion events every day to detect security hacks and threats.

77 With organic logging growing 50% year over year, and monitoring infrastructure spend at nearly 10%, one rogue log can ruin the platform. The checks and balances necessary to make sure we don’t hit that roadblock are built with the Elastic Stack and Beats. TEXT GOES HERE IN ALL CAPS Additional text goes here to support the content and can be a couple lines in length and sits bottom left aligned

78 With organic logging growing 50% year over year, and monitoring infrastructure spend at nearly 10%, one rogue log can ruin the platform. The checks and balances necessary to make sure we don’t hit that roadblock are built with the Elastic Stack and Beats. TEXT GOES HERE IN ALL CAPS Additional text goes here to support the content and can be a couple lines in length and sits bottom left aligned

” The Elastic Stack is critical to us. Every day millions of users and customers worldwide trust Box to execute mission-critical business functions. “

Some text can go here Some text can go here

You can use this area for a text treatment that supports your chosen imagery

Slide Title Here With a Few Bullets Subtitle goes here • Bullet one goes here in sentence case and no period • Bullets should be kept short and sweet; stay focused • Use bullets to help break up content that you need to have on the screen

Slide Title Here With a Few Bullets Subtitle goes here ● Bullet one goes here in sentence case and no period ● Bullets should be kept short and sweet; stay focused ● Use bullets to help break up content that you need to have on the screen

Slide Title Here With Key Points Subtitle goes here Header Here Body copy goes here and just increase the indent level to get to the proper formatting Header Here Body copy goes here and just increase the indent level to get to the proper formatting Header Here Body copy goes here and just increase the indent level to get to the proper formatting Header Here Body copy goes here and just increase the indent level to get to the proper formatting LOGGING METRICS APM ADVANCED SEARCH SECURITY ANALYTICS DATA SCIENCE FOUNDATION SPECIALIZATIONS

Slide Title Here With Key Points Subtitle goes here Header Here Body copy goes here and just increase the indent level to get to the proper formatting Header Here Body copy goes here and just increase the indent level to get to the proper formatting Header Here Body copy goes here and just increase the indent level to get to the proper formatting Header Here Body copy goes here and just increase the indent level to get to the proper formatting

Image Treatment With Caption Layout How to add your own photos and crop properly… Your image will populate the container but you will likely need to adjust the crop. Double click on the image to adjust. Use the blue dots to adjust the size. Click on the grayed out portion of the image and drag to the left or right until you are happy with the crop. 1 2 3 Right click on the image and go to replace image. Select a new image from your machine.

Agenda Slide Use color to highlight Enter title for section one here and use sentence case 1 Enter title for section three here and use sentence case 3 Enter title for section four here and use sentence case 4 Enter title for section five here and use sentence case 5 Enter title for section two here and use sentence case 2 Option 1A NOTE THIS SLIDE IS NOT IN THE LAYOUT OPTIONS. ALWAYS START A NEW PRESENTATION USING THE CORPORATE TEMPLATE AND ADD YOUR CONTENT TO THIS SLIDE.

Bullet slide title treatment can be up to two lines in length (Inter bold 24 pt) • Bullets are sentence case (Inter 18pt) ○ Second-line bullets are Inter 14pt ■ Third-line bullets are Inter 12pt • Limit the number of bullets on a slide • Text highlights are orange, but not underlined • Try not to go below the recommended font sizes Subtitle sentence case (Inter 18pt) Agenda Slide Use color to highlight Enter title for section one here and use sentence case 1 Enter title for section three here and use sentence case Enter title for section four here and use sentence case Enter title for section five here and use sentence case Enter title for section two here and use sentence case 2 Option 1B NOTE THIS SLIDE IS NOT IN THE LAYOUT OPTIONS. ALWAYS START A NEW PRESENTATION USING THE CORPORATE TEMPLATE AND ADD YOUR CONTENT TO THIS SLIDE. 3 4 5

Agenda Slide Use color to highlight Enter title for section one here and use sentence case Enter title for section two here and use sentence case Enter title for section three here and use sentence case Enter title for section four here and use sentence case Enter title for section five here and use sentence case 1 2 3 4 5 Option 2 NOTE THIS SLIDE IS NOT IN THE LAYOUT OPTIONS. ALWAYS START A NEW PRESENTATION USING THE CORPORATE TEMPLATE AND ADD YOUR CONTENT TO THIS SLIDE.

Process Diagram Treatment, 5 Ideas See style page for more color options 1 2 3 4 5 Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number

Process Diagram Treatment, 5 Ideas + Highlight See style page for more color options 1 2 3 4 5 Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number

Process Diagram Treatment, 4 Ideas See style page for more color options Supporting text goes here under the number 1 2 3 4 Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number

Process Diagram Treatment, 3 Ideas See style page for more color options Supporting text goes here under the number 1 2 3 Supporting text goes here under the number Supporting text goes here under the number

Process Diagram Treatment, 5 Ideas See style page for more color options 1 2 3 4 Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number 5 Supporting text goes here under the number

Process Diagram Treatment, 5 Ideas + Highlight See style page for more color options Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number 1 2 3 4 5

Process Diagram Treatment, 4 Ideas See style page for more color options 1 2 3 4 Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number

Process Diagram Treatment, 3 Ideas See style page for more color options 1 2 3 Supporting text goes here under the number Supporting text goes here under the number Supporting text goes here under the number

Title Here Title Here Title Here • One bullet here • Two bullet here • Three bullet here • One bullet here • Two bullet here • Three bullet here • One bullet here • Two bullet here • Three bullet here Box With Bullet Treatment

Title Here Title Here Title Here • One bullet here • Two bullet here • Three bullet here • One bullet here • Two bullet here • Three bullet here • One bullet here • Two bullet here • Three bullet here Box With Bullet Treatment with Color Choice

• One bullet here • Two bullet here • Three bullet here Title Here • One bullet here • Two bullet here • Three bullet here Title Here • One bullet here • Two bullet here • Three bullet here Title Here Box Bullet Treatment

• One bullet here • Two bullet here • Three bullet here • One bullet here • Two bullet here • Three bullet here • One bullet here • Two bullet here • Three bullet here Title Here Title Here Title Here Box Bullet Treatment with Color Scheme

Thank You Search. Observe. Protect.

Screenshot Treatment With Browser Window How to drop in your screen shot… The browser window is like a frame so anything you drop behind it will show through. Drop in your screen shot, go to the format menu and crop it to show only what you want. 2 Last, be sure to right click on your screen shot, go to order and send to back. 3 1

Screenshot Treatment With Title and Browser Window

NOTE USE THIS LAYOUT FOR PLACING ONE FULL BLEED SCREENSHOT

Use This Slide for Code, Light Version Use template colors to highlight curl –XPUT localhost:9200/ _template/twitter –d ‘ { “template” : “twitter_*”, “setting” : { “number_of_shards” : 4, “number_of_replicas” : 1 } }’

Use This Slide for Code, Dark Version Use template colors to highlight curl –XPUT localhost:9200/ _template/twitter –d ‘ { “template” : “twitter_*”, “setting” : { “number_of_shards” : 4, “number_of_replicas” : 1 } }’

Color Palette 254 197 20 47 67 145 250 115 79 240 78 152 151 156 171 67 71 83 0 119 204 0 191 179 PRIMARY

Styles and Treatments SHAPES LOGOS Please use logos according to brand guidelines. These logos can be sized up and down without losing quality. Please press shift before sizing to keep proper proportions. Various template colors can be used for shapes. Shapes should have a 3pt line stroke.

Video or Large Image Treatment Sub header goes here

3 solutions Elastic Enterprise Search Elastic Security Elastic Observability

Elastic Enterprise Search Workplace Search App Search Site Search

Elastic Observability Logs Metrics APM Uptime

Endpoint SIEM Elastic Security

3 solutions powered by 1 stack Kibana Elasticsearch Beats Logstash Elastic Stack Elastic Enterprise Search Elastic Security Elastic Observability

The Elastic Stack Reliably and securely take data from any source, in any format, then search, analyze, and visualize it in real time.

Deploy anywhere. SaaS Orchestration Elastic Cloud on Kubernetes Elastic Cloud Elastic Cloud Enterprise Elastic Enterprise Search Elastic Security Elastic Observability Kibana Elasticsearch Beats Logstash Powered by the stack 3 solutions Deployed anywhere

Deploy anywhere. SaaS Orchestration Elastic Cloud Elastic Cloud on Kubernetes Elastic Cloud Enterprise

Subscription Options ELASTIC CLOUD FREE PAID Open Source Features Free Proprietary Features Paid Proprietary Features + Elastic Support PAID OPEN SOURCE BASIC GOLD PLATINUM ENTERPRISE SELFMANAGED SaaS

Resource-based Pricing Endpoint Security No endpoint-based pricing SIEM No seat/ingest-based pricing APM No agent-based pricing Metrics No host-based pricing Logs No ingest-based pricing App Search No docs-based pricing Site Search No query-based pricing Workplace Search No user-based pricing Elastic Enterprise Search Elastic Security Elastic Observability

31 Solution Logos ENTERPRISE SEARCH OBSERVABILITY SECURITY Elastic Logo + Tagline FULL COLOR REVERSE

Product Logos ELASTIC CLOUD ON KUBERNETES ECK KIBANA LOGSTASH ELASTICSEARCH BEATS ELASTIC CLOUD ELASTIC CLOUD ENTERPRISE

Product Logos APM APP SEARCH WORKPLACE SEARCH METRICS SIEM LOGS SITE SEARCH ENDPOINT

Iconography Usage Product Feature Icons Do not use these icons for anything other than what they are created for. Product Feature Icons are created to correlate with a specific feature within the product and are not flexible in use. Please see labels as a guide. Generic Icons These icons are made to fit across multiple concepts within reason. See labels as a general guide. Please use discretion. Training Icons Do not use these icons for anything other than what they are created for. Training Icons are created to correlate with a specific feature within the training relm and are not flexible in use. Please see labels as a guide.

Feature Icons winlogbeat heartbeat packetbeat metricbeat functionbeat filebeat auditbeat index patterns Index management Life cycle management create single job create advanced job create multi metric job create population job machine learning advanced settings apm sql visualize dashboards canvas upgrade assistant management security analytics add data search profiler users and roles saved objects reporting security settings grok debugger language clients infra console discover dev tools watcher rollups cross cluster replication data visualizer metrics monitoring notebook logging spaces logstash pipeline gis application timelion graph --

Training Icons apm metrics Security analytics logging specialization Engineering 1 Engineering 2 certification Advanced search Data science subscription on-demand Instructor led stack

Generic Icons training support subscription customers structured schema schemaless rapid query execution sql No sql Horizontal scale flexible data model downloads custom consulting community community members Sophisticated query language node idea chart news user reliable extensible upgrade IoT plugin scale real-time high-five location distributed visibility plan E commerce family vacation presentation education guide book benefits certificate video contribution target Health monitor overlap conversation speaker government

Generic Icons To do Source code Color outside of the lines blog Send message docs mobile browser Love letter connection

Generic Icons training support subscription customers structured schema schemaless rapid query execution sql No sql Horizontal scale flexible data model downloads custom consulting community community members Sophisticated query language node idea Light bulb news user reliable extensible upgrade IoT plugin scale real-time high-five location distributed visibility plan E commerce family vacation presentation education guide book benefits certificate video contribution target Health monitor overlap conversation speaker government

An introduction to Elasticsearch's advanced relevance ranking toolbox

More Related Content

What's hot

Similar to An introduction to Elasticsearch's advanced relevance ranking toolbox

More from Elasticsearch

Recently uploaded

An introduction to Elasticsearch's advanced relevance ranking toolbox