- Notifications
You must be signed in to change notification settings - Fork 1.4k
Open
Description
Brief summary
In my use case I need to load entries from a CSV file which contains requests to do. The CSV file size is around 12 GB with over 70k lines. When using fs.open and csv.Parser recommended by the docs for handling such cases the memory usages increases during the init phase until the process is killed when OOM. Tested this in a 64GB RAM machine and all gets used. Also, tried the same script with a smaller file of 20 MB and it completes with the expected results.
k6 version
k6 v1.2.2 (commit/e0215db54a, go1.24.6, linux/amd64)
OS
Ubuntu 24.04
Docker version and image (if applicable)
Tested also with grafana/k6:1.1.0
Steps to reproduce the problem
Isolated the issue to:
- Generate a big CSV file(12 GB)
- Create a
k6-csv-issue.js
file with:import fs from 'k6/experimental/fs'; import csv from 'k6/experimental/csv'; const csvFilePath = "data.csv"; const csvFile = await fs.open(csvFilePath); const parser = new csv.Parser(csvFile, { skipFirstLine: true, }); export const options = { scenarios: { default: { executor: "shared-iterations", vus: 1, iterations: 100, } } }; export default async function () { const {done, value} = await parser.next(); if (done) { throw new Error("No more lines found"); } console.log(value[0]); }
- Execute
k6 run k6-csv-issue.js
Expected behaviour
K6 completes the test execution printing the first 100
values from the CSV file.
Actual behaviour
K6 process gets killed due to the machine getting OOM.