This project is a python service for data ingestion inside a Postgres database and inside an Elastic.
- Postgres : It takes all
csv filesfromdata/postgresand insert data inside database, - Elastic : It takes all documents from
data/elasticand insert them inside elastic.
-
Setup environment :
mv .env.example .env
-
Fill
.envwith your values -
Launch databases and Python service :
make start
-
Put your csv file(s) inside
data/postgresfolder and then run :python utils/ingestPg.py
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) ---------- Start Data Ingestion ---------- We found 3 csv files to process ------------------------- 2 lines from .../data-ingestor-service/data/test.csv inserted into test .../data-ingestor-service/fastapi_service/postgresIngestor.py:57: UserWarning: We can't add data from .../data-ingestor-service/data/test2.csv, columns doesn't match tables from public schema... warnings.warn(f"We can't add data from {path}, columns doesn't match tables from {SCHEMA} schema...") 2 lines from .../data-ingestor-service/data/test_copy.csv inserted into test ------------------------- Total : 4 inserted lines ---------- End Data Ingestion ---------- INFO: 127.0.0.1:64419 - "POST /ingest HTTP/1.1" 200 OK-
Put your documents inside
data/elasticfolder and then run :python utils/ingestElastic.py
data-ingestor-service-backend-1 | ---------- Start Data Ingestion ---------- data-ingestor-service-backend-1 | data-ingestor-service-backend-1 | We found 2 documents to process data-ingestor-service-backend-1 | data-ingestor-service-backend-1 | ------------------------- data-ingestor-service-backend-1 | data-ingestor-service-backend-1 | data-ingestor-service-backend-1 | data-ingestor-service-backend-1 | data-ingestor-service-backend-1 | ------------------------- data-ingestor-service-backend-1 | data-ingestor-service-backend-1 | data-ingestor-service-backend-1 | Total : 2 documents inserted data-ingestor-service-backend-1 | data-ingestor-service-backend-1 | ---------- End Data Ingestion ---------- data-ingestor-service-backend-1 | INFO: 192.168.0.1:61372 - "GET /ingestElastic HTTP/1.1" 200 OK