intermediate_steps do not show the query generated by gql_generation_chain when verify_gql is True #173
Description
Environment details
- OS type and version: Linux
- Python version: 3.10
langchain-google-spannerversion: 0.82
Proposed solution
See PR: #174
Problem
To help clients of the library to debug problems, please consider returning in intermediate_steps the query returned by gql_generation_chain when verify_gql is True (the default). By default if the gql_verify_chain modifies the query you cannot see the original query returned by gql_generation_chain.
Why is this important:
- If you are trying to improve the generation prompt you cannot see the impact since you only see the vertification prompt
- Makes difficult to collect insights when queries fail and you are developing automated evaluation
With verify_gql is True:
***verified_gql - GRAPH CNE_GRAPH MATCH (company:Company)-[:hasIncomeStatement]->(incomeStatement:IncomeStatement) WHERE LOWER(company.name) LIKE "%apple%" AND incomeStatement.fiscal_year = 2023 RETURN incomeStatement.net_income AS apple_net_income_2023 ***query_failed_0 - GRAPH CNE_GRAPH MATCH (company:Company)-[:hasIncomeStatement]->(incomeStatement:IncomeStatement) WHERE LOWER(company.name) LIKE "%apple%" AND incomeStatement.fiscal_year = 2023 RETURN incomeStatement.net_income AS apple_net_income_2023 ***generated_query - GRAPH CNE_GRAPH MATCH (company:Company)-[:hasFiscalPeriodKeyFigures]->(fiscalPeriodKeyFigures:FiscalPeriodKeyFigures) WHERE LOWER(company.name) LIKE "%apple%" AND fiscalPeriodKeyFigures.fiscal_year = 2023 RETURN fiscalPeriodK With verify_gql is False:
***query_failed_0 - gql -- Retrieving Apple's net income in 2023 GRAPH CNE_GRAPH MATCH pathData = (company:Company)-[:hasIncomeStatement]->(incomeStatement:IncomeStatement) WHERE LOWER(company.name) LIKE "%apple%" AND incomeStatement.fiscal_year = 2023 RETURN TO_JSON(pathData) AS apple_net_income_2023_pathData ***generated_query - GRAPH CNE_GRAPH MATCH (company:Company)-[:hasFiscalPeriodKeyFigures]->(fiscalPeriodKeyFigures:FiscalPeriodKeyFigures) WHERE LOWER(company.name) LIKE "%apple%" AND fiscalPeriodKeyFigures.fiscal_year = 2023 RETURN fiscalPeriodKeyFigures.net_income AS apple_net_income_2023 Notice how in this last example you can see how "RETURN TO_JSON(pathData) AS apple_net_income_2023_pathData" has been changed by " RETURN fiscalPeriodKeyFigures.net_income AS apple_net_income_2023" by the verification prompt, something you cannot see when when verify_gql is True