Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -232,9 +232,9 @@
"\n",
"A search for \"tax-free investment\" might have given the following results, with similarity scores:\n",
"\n",
"1. Document: Opening a NISA account (Score: 0.45)\n",
"2. Document: Opening a Regular Saving Account (Score: 0.48) <- *Similar score, potentially confusing*\n",
"3. Document: Home Loan Application Guide (Score: 0.42)\n",
"1. Document: Opening a NISA account (Score: 0.51)\n",
"2. Document: Opening a Regular Saving Account (Score: 0.50) <- *Similar score, potentially confusing*\n",
"3. Document: Home Loan Application Guide (Score: 0.44)\n",
"\n",
"> Note: To generate optimal embeddings with EmbeddingGemma, you should add an \"instructional prompt\" or \"task\" to the beginning of your input text. You will use `STS` for sentence similarity. For details on all available EmbeddingGemma prompts, see the [model card](http://ai.google.dev/gemma/docs/embeddinggemma/model_card#prompt_instructions)."
]
Expand All @@ -250,9 +250,9 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Document: Opening a NISA Account -> 🤖 Score: 0.45698774\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.48092696\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.42127067\n"
"Document: Opening a NISA Account -> 🤖 Score: 0.51571906\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.5035889\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.4406476\n"
]
}
],
Expand All @@ -261,8 +261,8 @@
"\n",
"def get_scores(query, documents):\n",
" # Calculate embeddings by calling model.encode()\n",
" query_embeddings = model.encode(query, prompt=task_name)\n",
" doc_embeddings = model.encode(documents, prompt=task_name)\n",
" query_embeddings = model.encode(query, prompt_name=task_name)\n",
" doc_embeddings = model.encode(documents, prompt_name=task_name)\n",
"\n",
" # Calculate the embedding similarities\n",
" similarities = model.similarity(query_embeddings, doc_embeddings)\n",
Expand Down Expand Up @@ -341,35 +341,35 @@
"output_type": "stream",
"text": [
"Step 3 finished. Running evaluation:\n",
"Document: Opening a NISA Account -> 🤖 Score: 0.6449194\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.44123\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.46752414\n",
"Document: Opening a NISA Account -> 🤖 Score: 0.6459116\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.42690125\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.40419024\n",
"Step 6 finished. Running evaluation:\n",
"Document: Opening a NISA Account -> 🤖 Score: 0.68873787\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.34069622\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.50065553\n",
"Document: Opening a NISA Account -> 🤖 Score: 0.68530923\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.3611964\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.40812016\n",
"Step 9 finished. Running evaluation:\n",
"Document: Opening a NISA Account -> 🤖 Score: 0.7148906\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.30480516\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.52454984\n",
"Document: Opening a NISA Account -> 🤖 Score: 0.7168733\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.3449782\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.44477722\n",
"Step 12 finished. Running evaluation:\n",
"Document: Opening a NISA Account -> 🤖 Score: 0.72614634\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.29255486\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.5370023\n",
"Document: Opening a NISA Account -> 🤖 Score: 0.73008573\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.34124148\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.4676212\n",
"Step 15 finished. Running evaluation:\n",
"Document: Opening a NISA Account -> 🤖 Score: 0.7294032\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.2893038\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.54087913\n",
"Document: Opening a NISA Account -> 🤖 Score: 0.73378766\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.34055778\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.47503752\n",
"Step 15 finished. Running evaluation:\n",
"Document: Opening a NISA Account -> 🤖 Score: 0.7294032\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.2893038\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.54087913\n"
"Document: Opening a NISA Account -> 🤖 Score: 0.73378766\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.34055778\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.47503752\n"
]
},
{
"data": {
"text/plain": [
"TrainOutput(global_step=15, training_loss=0.009651281436261646, metrics={'train_runtime': 63.2486, 'train_samples_per_second': 0.237, 'train_steps_per_second': 0.237, 'total_flos': 0.0, 'train_loss': 0.009651281436261646, 'epoch': 5.0})"
"TrainOutput(global_step=15, training_loss=0.009651267528511198, metrics={'train_runtime': 195.3004, 'train_samples_per_second': 0.077, 'train_steps_per_second': 0.077, 'total_flos': 0.0, 'train_loss': 0.009651267528511198, 'epoch': 5.0})"
]
},
"execution_count": 8,
Expand Down Expand Up @@ -431,9 +431,9 @@
"\n",
"The same search now yields much clearer results:\n",
"\n",
"1. Document: Opening a NISA account (Score: 0.72) <- *Much more confident*\n",
"2. Document: Opening a Regular Saving Account (Score: 0.28) <- *Clearly less relevant*\n",
"3. Document: Home Loan Application Guide (Score: 0.54)"
"1. Document: Opening a NISA account (Score: 0.73) <- *Much more confident*\n",
"2. Document: Opening a Regular Saving Account (Score: 0.34) <- *Clearly less relevant*\n",
"3. Document: Home Loan Application Guide (Score: 0.47)"
]
},
{
Expand All @@ -447,9 +447,9 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Document: Opening a NISA Account -> 🤖 Score: 0.7294032\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.2893038\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.54087913\n"
"Document: Opening a NISA Account -> 🤖 Score: 0.73378766\n",
"Document: Opening a Regular Savings Account -> 🤖 Score: 0.34055778\n",
"Document: Home Loan Application Guide -> 🤖 Score: 0.47503752\n"
]
}
],
Expand Down
Loading