DEV Community

Cover image for Building real-time chat with no external dependencies
Zenika profile image Huiren Woo
Huiren Woo for Zenika

Posted on • Originally published at blog.zenika.com

Building real-time chat with no external dependencies

#webdev #javascript #node

Introduction

There are many ways to build a real-time chat application. Before websockets was a commonplace, long-polling was a strategy deployed by various companies, including Facebook / Meta. With the rise of websockets, many developers went this route instead of using long-polling methods. However, depending on circumstances or scenarios, websockets may not be what you're looking for. In this article, we will explore how to build a simple chat application that doesn't use any third party libraries or websockets.

This application will use Server-Sent Events (SSE) to push messages from the server to the client in real time. Whilst it sends messages from the client to the server using a simple HTTP POST request.

Benefits of SSE over websockets

Minimal setup

SSE requires minimal setup on both the server and client sides whereas websockets require a more complex handshake process with an upgrade to the protocol. On the devops side, this means that websockets require additional plumbing. In a multi-instance setup you often need session-affinity or a dedicated WebSocket gateway cluster, so messages go to the same server that holds that socket. SSE is a read-only protocol, so it can be load-balanced without worrying about two way-routing.

Simplicity

SSE is simpler to implement than websockets. It uses a simple HTTP connection and does not require any additional libraries or protocols. Typically in websockets, you might need an external library like socket.io to handle it on both the client and server sides.

Downsides of SSE over websockets

No native binary support

In order to send binary data through Server Sent Events (SSE), you need to encode it as a string (e.g. base64) which increases the size of the payload. Whereas websockets can send binary frames natively without any encoding.

No built-in ping/pong heartbeat mechanism

SSE does not have a built-in ping/pong mechanism to keep the connection alive. If there are no frequent messages, some firewalls or proxies may close the connection after a period of inactivity. Whereas websockets have a built-in heartbeat mechanism that allows the server and client to keep the connection alive. You can overcome this by implementing a heartbeat mechanism on the server.

Designing the simple non-scalable chat application

Designing the message structure

Let's keep it simple and have only a few fields in our message. We will have an ID, a timestamp, an author, and the content of the message. For brevity, we will not implement any authentication or authorization, so anyone can send messages.

The message will be a simple JSON object with the following structure:

{ "id": "unique-message-id", "timestamp": "2023-10-01T12:00:00Z", "author": "author-name", "content": "message-content" }
Enter fullscreen mode Exit fullscreen mode

Initial client content

The client will consist of a simple HTML page with a form to submit messages and a div to display the messages. We will use the Fetch API to send messages to the server. The following is the HTML contents without the JavaScript code. The endpoints subsections will contain the respective client JavaScript code.

<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Chat</title> </head> <body> <div id="chat-messages"></div> <form id="chat-form" action="/send-message" method="POST"> <input type="text" name="author" placeholder="Username" required/> <input type="text" name="content" placeholder="Type your message here..." required/> <button type="submit">Send</button> </form> <script src="chat-app.js"></script> </body> </html>
Enter fullscreen mode Exit fullscreen mode

Server serving HTML content

Next, we will create a simple server that serves the HTML content to the client. It uses directory on the public folder to serve static files like HTML and JavaScript files. We will use Node.js's built-in http and fs modules to create the server.

const __dirname = path.dirname(new URL(import.meta.url).pathname); const PUBLIC_DIR = path.join(__dirname, 'public'); const MIME_TYPES = { '.html': 'text/html; charset=UTF-8', '.js': 'application/javascript; charset=UTF-8', }; // ... other code if (req.method === "GET") { const safeSuffix = path.normalize(decodeURI(req.url)).replace(/^(\.\.[\/\\])+/, ''); let filePath = path.join(PUBLIC_DIR, safeSuffix); // Check if the file exists try { const stats = fs.statSync(filePath); if (stats.isFile()) { const ext = path.extname(filePath).toLowerCase(); const contentType = MIME_TYPES[ext] || 'application/octet-stream'; res.writeHead(200, {'Content-Type': contentType}); res.end(fs.readFileSync(filePath)); return; } } catch (err) { // File does not exist or is not accessible } }
Enter fullscreen mode Exit fullscreen mode

Server message endpoints

We will create two endpoints on the server to handle messages.

GET endpoint

Allows new clients to connect and receive messages that will be broadcasted. This makes use of Server-Sent Events on the server side. Whenever a client connects to this endpoint, it will be added to a list of connected clients. This list of connected clients will be used later.

if (req.url === '/server-sent-events') { res.writeHead(200, { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache', 'Connection': 'keep-alive' }); const clientsConnectionsClone = [...clientsConnections]; clientsConnectionsClone.push(res); clientsConnections = [...clientsConnectionsClone]; res.on('close', () => { const clientsConnectionsClone = [...clientsConnections]; const index = clientsConnectionsClone.indexOf(res); if (index !== -1) { clientsConnectionsClone.splice(index, 1); clientsConnections = [...clientsConnectionsClone]; console.log('Client disconnected, removed from connections list'); } }); return; }
Enter fullscreen mode Exit fullscreen mode

Interacting with GET endpoint

On the client side, we will use the EventSource API to connect to the server and listen for messages. Whenever a message comes through, we will parse it and add it to the chat messages container in the HTML.

const source = new EventSource('/server-sent-events'); source.addEventListener('message', (e) => { console.log("Received message from server sent event:", e.data); try { const parsedData = JSON.parse(e.data); addChatMessage(parsedData); } catch (e) { console.log('Error parsing JSON:', e); } }); source.addEventListener('open', (e) => { console.log('connected'); }); source.addEventListener('error', (e) => { if (e.eventPhase === EventSource.CLOSED) { source.close(); } if (e.target.readyState === EventSource.CLOSED) { console.log('closed'); } if (e.target.readyState === EventSource.CONNECTING) { console.log('connecting'); } }); // ... other client code function addChatMessage(chatMessage) { const chatContainer = document.getElementById('chat-messages'); const messageElement = document.createElement('div'); messageElement.innerHTML = `<strong>${chatMessage.author}</strong>: ${chatMessage.content} <span class="timestamp">${new Date(chatMessage.timestamp).toLocaleTimeString()}</span>`; chatContainer.appendChild(messageElement); }
Enter fullscreen mode Exit fullscreen mode

POST endpoint

Allows connected clients to send messages to the server. Whenever the server receives a message, it will parse it and add additional information before broadcasting the message to all connected clients.

if (req.method === "POST") { if (req.url === '/send-message') { let body = ''; req.on('data', chunk => { body += chunk.toString(); }); req.on('end', () => { try { const jsonData = JSON.parse(body); // Validate required fields if (!jsonData.author || !jsonData.content) { res.writeHead(400, {'Content-Type': 'application/json'}); res.end(JSON.stringify({ status: 'error', message: 'Both author and content fields are required' })); return; } // Create message object with validated fields const message = { id: crypto.randomUUID().toString(), author: jsonData.author, content: jsonData.content, timestamp: new Date().toISOString() }; broadcastMessage(message); res.writeHead(201, {'Content-Type': 'application/json'}); res.end(JSON.stringify({status: 'Message sent'})); } catch (error) { // Handle JSON parsing errors res.writeHead(400, {'Content-Type': 'application/json'}); res.end(JSON.stringify({status: 'error', message: 'Invalid JSON format'})); } }); return; } } function broadcastMessage(messageJsonObject) { clientsConnections.forEach(client => { client.write(`data: ${JSON.stringify(messageJsonObject)}\n\n`); }); }
Enter fullscreen mode Exit fullscreen mode

How it looks like


Chatting with another person works and the message is instant!


No message history sent to newly connected clients due to design limitations.

Making it scalable

The application we have built so far works well for a single instance. However, it is not cloud native and does not scale. This is because the server stores messages in memory, and when the server restarts, all messages are lost. Additionally, if we want to run multiple instances of the server, they will not be able to share messages with each other.

The business benefit of a scalable application is that it can handle more traffic and more users easily, scaling up or down conveniently and when needed.

There are several ways to make this application horizontally scalable. For this article, we will go with the approach of using a file to store the messages. In a more complex setup, you would want to consider using a more reliable datastore like MySQL or Redis. Storing the messages allows enables the following:

  1. Messages are persistent and can be retrieved even if the server restarts
  2. Multiple instances of the server can read from the same file, allowing for horizontal scaling
  3. Messages can be served to new clients that connect after the server has started

Each server instance will store what messages has been broadcasted. We will only need to store the ID of the last message broadcasted.

Server message storage

We will write a file-based database to store the messages. There are some things we want to make sure:

  • The file is readable and writable by the server process
  • When multiple instances of the server are running, they can read without conflicts
  • When multiple instances of the server are running, the file is not written to at the same time

I decided to use a write-lock file to ensure that only one instance of the server can write to the file at a time. The messages will also be stored in a JSON format to make it easy to read and write. To prevent an infinite wait, the lockfile will also store a timestamp of the last write attempt. If the lockfile is older than a certain threshold, the server will assume that we can write to the file again. The flaw and limitation of this approach is that if there are many messages coming in, the server instances might be stuck in a queue as it uses a greedy approach and there is no order to the queue. For the sake of brevity, we will not implement a more complex queue system.

import fs from 'node:fs/promises'; import path from 'node:path'; import {fileURLToPath} from 'node:url'; const __dirname = path.dirname(fileURLToPath(import.meta.url)); const DB_DIR = path.join(__dirname, 'db'); const DB_FILE = path.join(DB_DIR, 'messages.json'); const LOCK_FILE = path.join(DB_DIR, 'messages.lock'); const LOCK_TIMEOUT = 5000; // 5 seconds before considering a lock stale // Ensure database directory exists async function ensureDbDir() { try { await fs.mkdir(DB_DIR, {recursive: true}); } catch (err) { if (err.code !== 'EEXIST') { throw err; } } } // Check if database is currently locked async function isLocked() { try { await fs.access(LOCK_FILE); const lockData = await fs.readFile(LOCK_FILE, 'utf8'); const lockTime = parseInt(lockData, 10); // Check for stale lock if (Date.now() - lockTime > LOCK_TIMEOUT) { console.log('Found stale lock, removing it'); await fs.unlink(LOCK_FILE); return false; } return true; } catch (err) { if (err.code === 'ENOENT') { // Lock file doesn't exist return false; } console.error('Lock check error:', err); return false; } } // Create a lock with current timestamp async function acquireLock() { await fs.writeFile(LOCK_FILE, Date.now().toString()); } // Release the lock async function releaseLock() { try { await fs.access(LOCK_FILE); await fs.unlink(LOCK_FILE); } catch (err) { if (err.code !== 'ENOENT') { console.error('Error releasing lock:', err); } } } // Read all messages from database async function readMessages() { try { await ensureDbDir(); try { await fs.access(DB_FILE); const data = await fs.readFile(DB_FILE, 'utf8'); return JSON.parse(data); } catch (err) { if (err.code === 'ENOENT') { return []; } } } catch (err) { console.error('Error reading messages:', err); return []; } } // Write messages to database with lock async function writeMessages(messages) { await ensureDbDir(); // Wait until lock is available let acquired = false; while (!acquired) { if (await isLocked()) { console.log('Database is locked, waiting...'); await new Promise(resolve => setTimeout(resolve, 100)); } else { acquired = true; } } try { await acquireLock(); await fs.writeFile(DB_FILE, JSON.stringify(messages, null, 2)); } finally { await releaseLock(); } } // Add new message to database async function saveMessage(message) { const messages = await readMessages(); messages.push(message); await writeMessages(messages); return message; } // Get messages newer than the specified ID async function getMessagesAfter(lastId) { const messages = await readMessages(); if (lastId == null) { return messages; } const lastIndex = messages.findIndex(msg => msg.id === lastId); if (lastIndex === -1) { return messages; } return messages.slice(lastIndex + 1); } export default { saveMessage, getMessagesAfter, readMessages, dbFilePath: DB_FILE, lockFilePath: LOCK_FILE, };
Enter fullscreen mode Exit fullscreen mode

Rewriting the server messaging

The server will now need to read information from the database, and as well as messages sent to its endpoint. We will poll the database file every 1 second for new messages and broadcast them to the connected clients. We can know where was the last broadcasted message by storing the ID of the last broadcasted message. In a more complex setup with Redis, you would use a pub/sub mechanism to notify all instances of new messages. As our file-based database is limited, we can only use polling to check for new messages and the downside would be the latency of the message updates. Increasing the polling interval would reduce the load on the server, but increase the latency of message delivery. Decreasing the polling interval would increase the load on the server, but decrease the latency of message delivery.

Modify the server message sending endpoint to save the message to the database and broadcast it to all connected clients.

// ... other code const message = { id: crypto.randomUUID().toString(), author: jsonData.author, content: jsonData.content, timestamp: new Date().toISOString() }; await db.saveMessage(message); broadcastMessage(message); res.writeHead(201, {'Content-Type': 'application/json'}); res.end(JSON.stringify({status: 'Message sent'})); // ... other code
Enter fullscreen mode Exit fullscreen mode 
// ... other code let lastBroadcastedId = null; // ... other code function broadcastMessage(messageJsonObject) { lastBroadcastedId = messageJsonObject.id; clientsConnections.forEach(client => { client.write(`data: ${JSON.stringify(messageJsonObject)}\n\n`); }); } async function pollNewMessages() { console.log('polling for new messages...'); try { const newMessages = await db.getMessagesAfter(lastBroadcastedId); if (newMessages.length > 0) { newMessages.forEach(message => { broadcastMessage(message); }); lastBroadcastedId = newMessages[newMessages.length - 1].id; } } catch (err) { console.error('Error polling for messages:', err); } // Poll again after delay setTimeout(pollNewMessages, 1000); } // ... other code server.listen(portNumber, async () => { console.log('Listening on http://localhost:' + portNumber); // Initialize lastBroadcastedId const messages = await db.readMessages(); if (messages.length > 0) { lastBroadcastedId = messages[messages.length - 1].id; } // Start polling await pollNewMessages(); });
Enter fullscreen mode Exit fullscreen mode

Upon connecting, the client will receive all messages that have been sent so far. This is done by reading all messages from the database and sending them to the client.

const clientsConnectionsClone = [...clientsConnections]; clientsConnectionsClone.push(res); clientsConnections = [...clientsConnectionsClone]; // Read all messages from the database and send them to the newly connected client db.readMessages().then(messages => { messages.forEach(message => { res.write(`data: ${JSON.stringify(message)}\n\n`); }); }).catch(err => { console.error('Error sending initial messages to client:', err); });
Enter fullscreen mode Exit fullscreen mode

How it looks like now


Now there's message history sent to the newly connected client. However, due to the 1 second polling, you will notice there's a slight delay on receiving the message on the other server instance.

Source code repository

The code repository for this article will be made available on GitHub after the last article in the series is published.

Conclusion

We have learnt how to build a simple chat application using Server-Sent Events and HTTP POST requests without any third-party dependencies. Through this process, we have learnt:

  • How to use Server-Sent Events to push messages from the server to the client in real time
  • How to create a simple server that serves HTML content and handles messages
  • Business and technical benefits of making cloud native applications
  • How to handle multiple instances of the server and ensure that messages are not lost
  • How to make our own file-based database with write-lock to store messages
  • How to poll the database for new messages and broadcast them to connected clients

Based on your experience and knowledge, are there anything that you think should be added or improved upon? Leave your suggestions or comments below! Or if you have any questions, please feel free to ask. Collaboration is how our community grows. Happy coding!

Top comments (0)