DEV Community

Mario
Mario

Posted on • Originally published at mariokandut.com on

How to stream to an HTTP response in Node.js

#node #javascript

A Node.js stream can help process large files, larger than the free memory of your computer, since it processes the data in small chunks.

Streams are a built-in feature in Node.js and represent asynchronous flow of data. This article explains how to stream a large file to an HTTP response in Node.js.

Streams in Node.js

Send a stream back to the client as a response to an HTTP request

This article is using Node v16.14.0 and ExpressJS 4.17.2.

In previous articles I have covered a lot of basic stream handling and theoretical background. Now, let's look at some implementation. In this tutorial we are going to use streams to efficiently send a file as an HTTP response and download it.

The big advantage of streams is that you can process large files (bigger than the available memory). In general, reading a large file into memory is an inefficient use of resources.

We are going to use ExpressJS with some CSV sample data from a previous article, How to use streams to ETL data.

Overview

  1. Initialize project and install dependencies
  2. Create a readable stream from a file
  3. Error handling and set headers for downloading file

1. Initialize project and install dependencies

We start with creating a folder for the project

mkdir streams-http cd streams-http
Enter fullscreen mode Exit fullscreen mode

We are going to use npm packages, hence, we have to initialize the project to get a package.jsonInitialize empty project to install dependencies, add -y flag to agree to everything.

npm init -y
Enter fullscreen mode Exit fullscreen mode

Install ExpressJS.

npm i express
Enter fullscreen mode Exit fullscreen mode

Create a folder for sample data and add the CSV data.

mkdir data cd data touch sample-data.csv
Enter fullscreen mode Exit fullscreen mode

Copy all sample data into sample-data.csv and save it. Use copy+paste or fs.writeFile in the REPL or with the -p flag in the terminal.

id,firstName,lastName,email,email2,randomized 100,Jobi,Taam,Jobi.Taam@yopmail.com,Jobi.Taam@gmail.com,Z lsmDLjL 101,Dacia,Elephus,Dacia.Elephus@yopmail.com,Dacia.Elephus@gmail.com,Za jfPaJof 102,Arlina,Bibi,Arlina.Bibi@yopmail.com,Arlina.Bibi@gmail.com,zmzlfER 103,Lindie,Torray,Lindie.Torray@yopmail.com,Lindie.Torray@gmail.com,ibVggFEh 104,Modestia,Leonard,Modestia.Leonard@yopmail.com,Modestia.Leonard@gmail.com," Tit KCrdh" 105,Karlee,Cornelia,Karlee.Cornelia@yopmail.com,Karlee.Cornelia@gmail.com,PkQCUXzq 106,Netty,Travax,Netty.Travax@yopmail.com,Netty.Travax@gmail.com,psJKWDBrXm 107,Dede,Romelda,Dede.Romelda@yopmail.com,Dede.Romelda@gmail.com,heUrfT 108,Sissy,Crudden,Sissy.Crudden@yopmail.com,Sissy.Crudden@gmail.com,cDJxC 109,Sherrie,Sekofski,Sherrie.Sekofski@yopmail.com,Sherrie.Sekofski@gmail.com,dvYHUJ 110,Sarette,Maryanne,Sarette.Maryanne@yopmail.com,Sarette.Maryanne@gmail.com,rskGIJNF 111,Selia,Waite,Selia.Waite@yopmail.com,Selia.Waite@gmail.com,DOPBe 112,Karly,Tjon,Karly.Tjon@yopmail.com,Karly.Tjon@gmail.com,zzef nCMVL 113,Sherrie,Berriman,Sherrie.Berriman@yopmail.com,Sherrie.Berriman@gmail.com,rQqmjw 114,Nadine,Greenwald,Nadine.Greenwald@yopmail.com,Nadine.Greenwald@gmail.com,JZsmKafeIf 115,Antonietta,Gino,Antonietta.Gino@yopmail.com,Antonietta.Gino@gmail.com,IyuCBqwlj 116,June,Dorothy,June.Dorothy@yopmail.com,June.Dorothy@gmail.com,vyCTyOjt 117,Belva,Merriott,Belva.Merriott@yopmail.com,Belva.Merriott@gmail.com,MwwiGEjDfR 118,Robinia,Hollingsworth,Robinia.Hollingsworth@yopmail.com,Robinia.Hollingsworth@gmail.com,wCaIu 119,Dorthy,Pozzy,Dorthy.Pozzy@yopmail.com,Dorthy.Pozzy@gmail.com,fmWOUCIM 120,Barbi,Buffum,Barbi.Buffum@yopmail.com,Barbi.Buffum@gmail.com,VOZEKSqrZa 121,Priscilla,Hourigan,Priscilla.Hourigan@yopmail.com,Priscilla.Hourigan@gmail.com,XouVGeWwJ 122,Tarra,Hunfredo,Tarra.Hunfredo@yopmail.com,Tarra.Hunfredo@gmail.com,NVzIduxd 123,Madalyn,Westphal,Madalyn.Westphal@yopmail.com,Madalyn.Westphal@gmail.com,XIDAOx 124,Ruthe,McAdams,Ruthe.McAdams@yopmail.com,Ruthe.McAdams@gmail.com,iwVelLKZH 125,Maryellen,Brotherson,Maryellen.Brotherson@yopmail.com,Maryellen.Brotherson@gmail.com,nfoiVBjjqw 126,Shirlee,Mike,Shirlee.Mike@yopmail.com,Shirlee.Mike@gmail.com,MnTkBSFDfo 127,Orsola,Giule,Orsola.Giule@yopmail.com,Orsola.Giule@gmail.com,VPrfEYJi 128,Linzy,Bennie,Linzy.Bennie@yopmail.com,Linzy.Bennie@gmail.com,ZHctp 129,Vanessa,Cohdwell,Vanessa.Cohdwell@yopmail.com,Vanessa.Cohdwell@gmail.com,RvUcbJihHf 130,Jaclyn,Salvidor,Jaclyn.Salvidor@yopmail.com,Jaclyn.Salvidor@gmail.com,gbbIxz 131,Mildrid,Pettiford,Mildrid.Pettiford@yopmail.com,Mildrid.Pettiford@gmail.com,snyeV 132,Carol-Jean,Eliathas,Carol-Jean.Eliathas@yopmail.com,Carol-Jean.Eliathas@gmail.com,EAAjYHiij 133,Susette,Ogren,Susette.Ogren@yopmail.com,Susette.Ogren@gmail.com," BhYgr" 134,Farrah,Suanne,Farrah.Suanne@yopmail.com,Farrah.Suanne@gmail.com,hYZbZIc 135,Cissiee,Idelia,Cissiee.Idelia@yopmail.com,Cissiee.Idelia@gmail.com,PNuxbvjx 136,Alleen,Clara,Alleen.Clara@yopmail.com,Alleen.Clara@gmail.com,YkonJWtV 137,Merry,Letsou,Merry.Letsou@yopmail.com,Merry.Letsou@gmail.com,sLfCumcwco 138,Fanny,Clywd,Fanny.Clywd@yopmail.com,Fanny.Clywd@gmail.com,Go kx 139,Trixi,Pascia,Trixi.Pascia@yopmail.com,Trixi.Pascia@gmail.com,lipLcqRAHr 140,Sandie,Quinn,Sandie.Quinn@yopmail.com,Sandie.Quinn@gmail.com,KrGazhI 141,Dania,Wenda,Dania.Wenda@yopmail.com,Dania.Wenda@gmail.com,CXzs kDv 142,Kellen,Vivle,Kellen.Vivle@yopmail.com,Kellen.Vivle@gmail.com,RrKPYqq 143,Jany,Whittaker,Jany.Whittaker@yopmail.com,Jany.Whittaker@gmail.com,XAIufn 144,Lusa,Fillbert,Lusa.Fillbert@yopmail.com,Lusa.Fillbert@gmail.com,FBFQnPm 145,Farrah,Edee,Farrah.Edee@yopmail.com,Farrah.Edee@gmail.com,TrCwKb 146,Felice,Peonir,Felice.Peonir@yopmail.com,Felice.Peonir@gmail.com,YtVZywf 147,Starla,Juan,Starla.Juan@yopmail.com,Starla.Juan@gmail.com,aUTvjVNyw 148,Briney,Elvyn,Briney.Elvyn@yopmail.com,Briney.Elvyn@gmail.com,tCEvgeUbwF 149,Marcelline,Ricarda,Marcelline.Ricarda@yopmail.com,Marcelline.Ricarda@gmail.com,sDwIlLckbd 150,Mureil,Rubie,Mureil.Rubie@yopmail.com,Mureil.Rubie@gmail.com,HbcfbKd 151,Nollie,Dudley,Nollie.Dudley@yopmail.com,Nollie.Dudley@gmail.com,EzjjrNwVUm 152,Yolane,Melony,Yolane.Melony@yopmail.com,Yolane.Melony@gmail.com,wfqSgpgL 153,Brena,Reidar,Brena.Reidar@yopmail.com,Brena.Reidar@gmail.com,iTlvaS 154,Glenda,Sabella,Glenda.Sabella@yopmail.com,Glenda.Sabella@gmail.com,zzaWxeI 155,Paola,Virgin,Paola.Virgin@yopmail.com,Paola.Virgin@gmail.com,gJO hXTWZl 156,Aryn,Erich,Aryn.Erich@yopmail.com,Aryn.Erich@gmail.com,qUoLwH 157,Tiffie,Borrell,Tiffie.Borrell@yopmail.com,Tiffie.Borrell@gmail.com,cIYuVMHwF 158,Anestassia,Daniele,Anestassia.Daniele@yopmail.com,Anestassia.Daniele@gmail.com,JsDbQbc 159,Ira,Glovsky,Ira.Glovsky@yopmail.com,Ira.Glovsky@gmail.com,zKITnYXyhC 160,Sara-Ann,Dannye,Sara-Ann.Dannye@yopmail.com,Sara-Ann.Dannye@gmail.com,wPClmU 161,Modestia,Zina,Modestia.Zina@yopmail.com,Modestia.Zina@gmail.com,YRwcMqPK 162,Kelly,Poll,Kelly.Poll@yopmail.com,Kelly.Poll@gmail.com,zgklmO 163,Ernesta,Swanhildas,Ernesta.Swanhildas@yopmail.com,Ernesta.Swanhildas@gmail.com,tWafP 164,Giustina,Erminia,Giustina.Erminia@yopmail.com,Giustina.Erminia@gmail.com,XgOKKAps 165,Jerry,Kravits,Jerry.Kravits@yopmail.com,Jerry.Kravits@gmail.com,olzBzS 166,Magdalena,Khorma,Magdalena.Khorma@yopmail.com,Magdalena.Khorma@gmail.com,BBKPB 167,Lory,Pacorro,Lory.Pacorro@yopmail.com,Lory.Pacorro@gmail.com,YmWQB 168,Carilyn,Ethban,Carilyn.Ethban@yopmail.com,Carilyn.Ethban@gmail.com,KUXenrJh 169,Tierney,Swigart,Tierney.Swigart@yopmail.com,Tierney.Swigart@gmail.com,iQCQJ 170,Beverley,Stacy,Beverley.Stacy@yopmail.com,Beverley.Stacy@gmail.com,NMrS Zpa f 171,Ida,Dex,Ida.Dex@yopmail.com,Ida.Dex@gmail.com,hiIgOCxNg 172,Sam,Hieronymus,Sam.Hieronymus@yopmail.com,Sam.Hieronymus@gmail.com,dLSkVe 173,Lonnie,Colyer,Lonnie.Colyer@yopmail.com,Lonnie.Colyer@gmail.com,ZeDosRy 174,Rori,Ethban,Rori.Ethban@yopmail.com,Rori.Ethban@gmail.com,SXFZQmX 175,Lelah,Niles,Lelah.Niles@yopmail.com,Lelah.Niles@gmail.com,NwxvCXeszl 176,Kathi,Hepsibah,Kathi.Hepsibah@yopmail.com,Kathi.Hepsibah@gmail.com,SOcAOSn 177,Dominga,Cyrie,Dominga.Cyrie@yopmail.com,Dominga.Cyrie@gmail.com,IkjDyuqK 178,Pearline,Bakerman,Pearline.Bakerman@yopmail.com,Pearline.Bakerman@gmail.com,vHVCkQ 179,Selma,Gillan,Selma.Gillan@yopmail.com,Selma.Gillan@gmail.com,hSZgpBNsw 180,Bernardine,Muriel,Bernardine.Muriel@yopmail.com,Bernardine.Muriel@gmail.com,AnSDTDa U 181,Ermengarde,Hollingsworth,Ermengarde.Hollingsworth@yopmail.com,Ermengarde.Hollingsworth@gmail.com,IYQZ Nmv 182,Marguerite,Newell,Marguerite.Newell@yopmail.com,Marguerite.Newell@gmail.com,kSaD uaHH 183,Albertina,Nisbet,Albertina.Nisbet@yopmail.com,Albertina.Nisbet@gmail.com,Y jHyluB 184,Chere,Torray,Chere.Torray@yopmail.com,Chere.Torray@gmail.com,loElYdo 185,Vevay,O'Neill,Vevay.O'Neill@yopmail.com,Vevay.O'Neill@gmail.com,uLZSdatVn 186,Ann-Marie,Gladstone,Ann-Marie.Gladstone@yopmail.com,Ann-Marie.Gladstone@gmail.com,fwKlEksI 187,Donnie,Lymann,Donnie.Lymann@yopmail.com,Donnie.Lymann@gmail.com,deBrqXyyjf 188,Myriam,Posner,Myriam.Posner@yopmail.com,Myriam.Posner@gmail.com,gEMZo 189,Dale,Pitt,Dale.Pitt@yopmail.com,Dale.Pitt@gmail.com,OeMdG 190,Cindelyn,Thornburg,Cindelyn.Thornburg@yopmail.com,Cindelyn.Thornburg@gmail.com,kvhFmKGoMZ 191,Maisey,Hertzfeld,Maisey.Hertzfeld@yopmail.com,Maisey.Hertzfeld@gmail.com,OajjJ 192,Corina,Heisel,Corina.Heisel@yopmail.com,Corina.Heisel@gmail.com,luoDJeHo 193,Susette,Marcellus,Susette.Marcellus@yopmail.com,Susette.Marcellus@gmail.com,AXHtR AyV 194,Lanae,Sekofski,Lanae.Sekofski@yopmail.com,Lanae.Sekofski@gmail.com,FgToedU 195,Linet,Beebe,Linet.Beebe@yopmail.com,Linet.Beebe@gmail.com,DYGfRP 196,Emilia,Screens,Emilia.Screens@yopmail.com,Emilia.Screens@gmail.com,LXUcleSs 197,Tierney,Avi,Tierney.Avi@yopmail.com,Tierney.Avi@gmail.com,VegzbHH 198,Pollyanna,Thar,Pollyanna.Thar@yopmail.com,Pollyanna.Thar@gmail.com,GjYeEGK 199,Darci,Elephus,Darci.Elephus@yopmail.com,Darci.Elephus@gmail.com,DaQNdN
Enter fullscreen mode Exit fullscreen mode

Create an index.js file (in root folder), which will be the main file for our code.

cd .. # if you are in the data folder touch index.js
Enter fullscreen mode Exit fullscreen mode

2. Create a readable stream from a file

First, we are going to create a basic express server, which listens on port 3000. Open index.js in your IDE and add the following code.

const express = require('express'); const app = express(); const PORT = 3000; app.listen(PORT, () => console.log(`Server listening on port ${PORT}`), );
Enter fullscreen mode Exit fullscreen mode

Start the node server with running node index.js in the project root folder. You should see Server listening on port 3000 in your terminal. Terminate the server with CTRL+C.

Let's make another route to download the csv file. Add a GET handler with the route /get-data to index.js.

app.get('/get-data', (req, res, next) => { // TBD });
Enter fullscreen mode Exit fullscreen mode

Now we have a route on which we are going to download the file in the end. We can proceed to create a readable stream to read the file. For creating a stream we have to import the fs module.

const fs = require('fs');
Enter fullscreen mode Exit fullscreen mode

Create a stream to read file sample-data.csv.

app.get('/dl', (req, res, next) => { const fileStream = fs.createReadStream( `${__dirname}/data/sample-data.csv`, ); });
Enter fullscreen mode Exit fullscreen mode

The constant fileStream represents the data stream from the file. This stream we directly pipe into the response.

app.get('/dl', (req, res, next) => { const fileStream = fs.createReadStream( `${__dirname}/data/sample-data.csv`, ); fileStream.pipe(res); });
Enter fullscreen mode Exit fullscreen mode

Start the server again node index.js. And open a web browser with http://localhost:3000/get-data. You should see the csv file.

3. Error handling and set headers for downloading file

Now we are sending a file as a stream, but it should download. Let's make it happen. As always, we have to think of error handling first. What could go wrong? The file could not exist.

To handle this, we have to listen to the open event on the readStream to check, if the file exists and only pipe the stream if it does.

app.get('/get-data', (req, res, next) => { const fileStream = fs.createReadStream( `${__dirname}/data/sample-data.csv`, ); fileStream.on('open', () => { fileStream.pipe(res); }); });
Enter fullscreen mode Exit fullscreen mode

And if the file doesn't exist, we return the error as response.

app.get('/get-data', (req, res, next) => { const fileStream = fs.createReadStream( `${__dirname}/data/sample-data.csv`, ); fileStream.on('open', () => { fileStream.pipe(res); }); fileStream.on('error', err => { next(err); }); });
Enter fullscreen mode Exit fullscreen mode

In most cases the pipeline method should be used, but pipeline destroys streams when an error occurs, and we would not be able to send a response back. Hence, for this use case manually error handling is acceptable.

At the moment the browser is displaying the file inline, it's loading the file in the browser. To tell the browser to download the file, we have to set:

  • a Content-Type header on the response to specify what file we are sending, and
  • the Content-Disposition header to attachment with a file name.

Express has a method for this attachment("FILENAME"). It sets the HTTP response Content-Disposition header field to “attachment”, and if a filename is given, then it sets the Content-Type based on the extension name via res.type(), and sets the Content-Disposition “filename=” parameter.

When using res.attachment('streamed-sample-data'), the content-type header will be set to text/csv and the content-disposition to the streamed-sample-data.csv.

app.get('/get-data', (req, res, next) => { const fileStream = fs.createReadStream( `${__dirname}/data/sample-data.csv`, ); fileStream.on('open', () => { res.attachment('streamed-sample-data.csv'); fileStream.pipe(res); }); fileStream.on('error', err => { next(err); }); });
Enter fullscreen mode Exit fullscreen mode

Restart your node server and go to http://localhost:3000/get-data. The file streamed-sample-data.csv should be downloaded.

TL;DR

  • Error handling has to be done always, especially when working with streams.
  • Error handling should be done in most cases with pipeline, only if the stream should not be destroyed, manually error handling is necessary.
  • The express method attachment() is used to add Content-Type and Content-Disposition headers to a response.
  • For the future, the Express framework has a method for sending files via stream sendFile().

Thanks for reading and if you have any questions , use the comment function or send me a message @mariokandut.

If you want to know more about Node, have a look at these Node Tutorials.

References (and Big thanks):

ExpressJS,HeyNode,Node.js - Streams,MDN - Streams,MDN - HTTP

Top comments (1)

Collapse
 
projektorius96 profile image
LG •

I've started recently dig into C++ . I am JavaScript (Node.js) developer ,can you tell me what are capstone cases I should touch in C(++) lang family , to get closer to Node.js nature, File System (Linux/Unix), Stream API, any other suggestions or roadmaps , Mario ? Thanks !

p.s. your tuts always smells like fresh loaf of bread, appreciated it !