Supercharge BigQuery with BigFunctions

This repository is a customized fork of Unytics BigFunctions, designed to deploy specific, custom functions within our Venom Engineering projects. For detailed information about the underlying BigFunctions framework, please refer to the original README.

Purpose: This repository enables us to create and deploy custom functions directly into our BigQuery environment, streamlining data processing and integration.

Current Custom Functions:

• load_pennylane: Imports data from Pennylane's List API endpoints into a designated BigQuery table.
• load_qonto_transactions: Imports transaction data from all Qonto bank accounts within our organization into a designated BigQuery table.

Our custom load functions are implemented in Python and deployed as Cloud Run services. These services are then made accessible within BigQuery via remote connections, enabling direct invocation from BigQuery queries.

Deployment Steps:

1. Deploy using bigfun deploy:

   bigfun deploy <function_name> \ --project <project_id> \ --dataset <dataset_name>

   • Replace <function_name>, <project_id>, and <dataset_name> with the appropriate values for your deployment.

2. Invoke from BigQuery:

   Once deployed, the function can be called directly from BigQuery using the following syntax:

   project_id.dataset_name.function_name

   • BigFunctions manages the underlying infrastructure and connection complexities, providing a streamlined user experience.

• Functionality: Imports data from Pennylane's List API endpoints into a designated BigQuery table. This function is designed to be versatile, handling various data types available through Pennylane's list endpoints.
• Dependencies: Requires a secret stored in Secret Manager named api_bearer_token for authentication.
• Service Account Permissions: The Cloud Run service account requires the following IAM roles:
  • BigQuery Editor
  • Secret Manager Secret Accessor

• Functionality: Retrieves and inserts transaction data from all Qonto bank accounts within the organization into a specified BigQuery table.
• Dependencies: Requires two secrets stored in Google Cloud Secret Manager for API authentication: api_login and api_secret_key.
• Service Account Permissions: The Cloud Run service account associated with this function requires the following IAM roles:
  • BigQuery Editor: Allows the function to write data to BigQuery tables.
  • Secret Manager Secret Accessor: Enables the function to retrieve API credentials from Secret Manager.