Skip to content

CSV Parser causes OOM when loading big files #5080

@cbermudez97

Description

@cbermudez97

Brief summary

In my use case I need to load entries from a CSV file which contains requests to do. The CSV file size is around 12 GB with over 70k lines. When using fs.open and csv.Parser recommended by the docs for handling such cases the memory usages increases during the init phase until the process is killed when OOM. Tested this in a 64GB RAM machine and all gets used. Also, tried the same script with a smaller file of 20 MB and it completes with the expected results.

k6 version

k6 v1.2.2 (commit/e0215db54a, go1.24.6, linux/amd64)

OS

Ubuntu 24.04

Docker version and image (if applicable)

Tested also with grafana/k6:1.1.0

Steps to reproduce the problem

Isolated the issue to:

  1. Generate a big CSV file(12 GB)
  2. Create a k6-csv-issue.js file with:
     import fs from 'k6/experimental/fs'; import csv from 'k6/experimental/csv'; const csvFilePath = "data.csv"; const csvFile = await fs.open(csvFilePath); const parser = new csv.Parser(csvFile, { skipFirstLine: true, }); export const options = { scenarios: { default: { executor: "shared-iterations", vus: 1, iterations: 100, } } }; export default async function () { const {done, value} = await parser.next(); if (done) { throw new Error("No more lines found"); } console.log(value[0]); }
  3. Execute k6 run k6-csv-issue.js

Expected behaviour

K6 completes the test execution printing the first 100 values from the CSV file.

Actual behaviour

K6 process gets killed due to the machine getting OOM.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions