DEV Community

Cover image for Building a Real-Time Analytics Dashboard with Next.js, Tinybird, and Tremor: A Comprehensive Guide
Marc Seitz
Marc Seitz

Posted on • Edited on

Building a Real-Time Analytics Dashboard with Next.js, Tinybird, and Tremor: A Comprehensive Guide

#nextjs #opensource #analytics #tutorial

What you will find in this article?

Real-time analytics have become a vital part of today's applications, especially when it comes to understanding user behavior and improving your platform based on data-driven insights. Even more so, if your users rely on your product to deliver them with insights.

In this post, we are going to explore how we can create a real-time analytics dashboard for page views using Tinybird, Tremor.so, and Next.js.

We will be using Tinybird for data ingestion and real-time data analysis, Tremor for data visualization, and Next.js for server-side rendering.

Analytics GIF

Papermark - the open-source DocSend alternative.

Before we kick off, let me introduce you to Papermark. It's an open-source project for securely sharing documents with built-in real-time, page-by-page analytics, powered by Tinybird and visualized by Tremor.

I would be absolutely thrilled if you could give us a star! Don't forget to share your thoughts in the comments section ❤️

https://github.com/mfts/papermark

Papermark Analytics

Setup the project

Let's set up our project environment. We will be setting up a Next.js app, installing Tinybird CLI and configuring the needed services and tools.

Set up tea

It's a good idea to have a package manager handy, like tea. It'll handle your development environment and simplify your (programming) life!

sh <(curl https://tea.xyz) # --- OR --- # using brew brew install teaxyz/pkgs/tea-cli
Enter fullscreen mode Exit fullscreen mode

tea frees you to focus on your code, as it takes care of installing python and pipenv (which I'm using to run tinybird-cli), node, npm and any other packages you may need. The best part is, tea installs all packages in a dedicated directory (default: ~/.tea), keeping your system files neat and tidy.

Set up Next.js with TypeScript and Tailwindcss

We will use create-next-app to generate a new Next.js project. We will also be using TypeScript and Tailwind CSS, so make sure to select those options when prompted.

npx create-next-app # --- # you'll be asked the following prompts What is your project named? my-app Would you like to add TypeScript with this project? Y/N # select `Y` for typescript Would you like to use ESLint with this project? Y/N # select `Y` for ESLint Would you like to use Tailwind CSS with this project? Y/N # select `Y` for Tailwind CSS Would you like to use the `src/ directory` with this project? Y/N # select `N` for `src/` directory What import alias would you like configured? `@/*` # enter `@/*` for import alias
Enter fullscreen mode Exit fullscreen mode

Install Tinybird

Tinybird's command line interface (CLI) helps us to manage data sources and pipes. I'm installing tinybird-cli using pipenv, which manages a virtual environment for my pip packages, in our local environment.

# Navigate to your Next.js repo cd my-app # Create a new virtual environment and install Tinybird CLI # if you install `tea` in the previous step, then tea will take care of installing pipenv and its dependencies pipenv install tinybird-cli # Activate the virtual environment pipenv shell
Enter fullscreen mode Exit fullscreen mode

Configuring Tinybird

Head over to tinybird.co and create a free account. You need to have a Tinybird account and an associated token. You can get the token from your Tinybird's dashboard.

Image description

# Authenticate with tinybird-cli using your auth token when prompted tb auth
Enter fullscreen mode Exit fullscreen mode

A .tinyb file will be added to your repo's root. No need to modify it. However add it it to your .gitignore file to avoid exposing your token.

echo ".tinyb" >> .gitignore
Enter fullscreen mode Exit fullscreen mode

Building the application

Now that we have our setup in place, we are ready to start building our application. The main features we'll cover are:

  • Tinybird Pipes and Datasources
  • Page View Recording
  • Tremor Bar Chart

#1 Tinybird Pipes and Datasource

The ability to programmatically configure pipes and datasources for Tinybird offers a significant advantage. This flexibility enables us to treat our data infrastructure as code, meaning that the entire configuration can be committed into a version control system. For an open-source project like Papermark, this capability is highly beneficial. It fosters transparency and collaboration, as contributors can readily understand the data structure without any ambiguity.

We set up Tinybird pipes and datasource as follows:

mkdir -p lib/tinybird/{datasources,endpoints}
Enter fullscreen mode Exit fullscreen mode

1. Datasources

This is basically a versioned schema for page_views. This the only datasource we need for ingesting, storing and reading analytics about page views. Feel free to add/remove fields.

# lib/tinybird/datasources/page_views.datasource VERSION 1 DESCRIPTION > Page views are events when a user views a document SCHEMA > `id` String `json:$.id`, `linkId` String `json:$.linkId`, `documentId` String `json:$.documentId`, `viewId` String `json:$.viewId`, # Unix timestamp `time` Int64 `json:$.time`, `duration` UInt32 `json:$.duration`, # The page number `pageNumber` LowCardinality(String) `json:$.pageNumber`, `country` String `json:$.country`, `city` String `json:$.city`, `region` String `json:$.region`, `latitude` String `json:$.latitude`, `longitude` String `json:$.longitude`, `ua` String `json:$.ua`, `browser` String `json:$.browser`, `browser_version` String `json:$.browser_version`, `engine` String `json:$.engine`, `engine_version` String `json:$.engine_version`, `os` String `json:$.os`, `os_version` String `json:$.os_version`, `device` String `json:$.device`, `device_vendor` String `json:$.device_vendor`, `device_model` String `json:$.device_model`, `cpu_architecture` String `json:$.cpu_architecture`, `bot` UInt8 `json:$.bot`, `referer` String `json:$.referer`, `referer_url` String `json:$.referer_url` ENGINE "MergeTree" ENGINE_SORTING_KEY "linkId,documentId,viewId,pageNumber,time,id"
Enter fullscreen mode Exit fullscreen mode

2. Pipes

In Tinybird, a pipe is a series of transformations that are applied to your data. These transformations could include anything from simple data cleaning operations to complex aggregations and analytics.

We have one versioned pipe (also sometimes called endpoint) to retrieve page_view data. This Tinybird pipe, named endpoint, calculates the average duration users spend on each page of a specific document within a defined timeframe, and presents the results in ascending order by page number.

# lib/endpoints/get_average_page_duration.pipe VERSION 1 NODE endpoint SQL > % SELECT pageNumber, AVG(duration) AS avg_duration FROM page_views__v1 WHERE documentId = {{ String(documentId, required=True) }} AND time >= {{ Int64(since, required=True) }} GROUP BY pageNumber ORDER BY pageNumber ASC
Enter fullscreen mode Exit fullscreen mode

Now that we have Tinybird datasources and pipes set up, you need to push them to your Tinybird's account using the CLI.

# Navigate to the directory containing your datasource and pipe files cd lib/tinybird # Push your files to Tinybird tb push datasources/*.datasource pipes/*.pipe
Enter fullscreen mode Exit fullscreen mode

3. Typescript functions

Let's set up the appropriate typescript functions to actually send and retrieve data from Tinybird. We are using zod and chronark's zod-bird library

This function is for retrieving data from Tinybird:

// lib/tinybird/pipes.ts import { z } from "zod"; import { Tinybird } from "@chronark/zod-bird"; const tb = new Tinybird({ token: process.env.TINYBIRD_TOKEN! }); export const getTotalAvgPageDuration = tb.buildPipe({ pipe: "get_total_average_page_duration__v1", parameters: z.object({ documentId: z.string(), since: z.number(), }), data: z.object({ pageNumber: z.string(), avg_duration: z.number(), }), });
Enter fullscreen mode Exit fullscreen mode

This function is for sending data to Tinybird:

// lib/tinybird/publish.ts import { z } from "zod"; import { Tinybird } from "@chronark/zod-bird"; const tb = new Tinybird({ token: process.env.TINYBIRD_TOKEN! }); export const publishPageView = tb.buildIngestEndpoint({ datasource: "page_views__v1", event: z.object({ id: z.string(), linkId: z.string(), documentId: z.string(), viewId: z.string(), time: z.number().int(), duration: z.number().int(), pageNumber: z.string(), country: z.string().optional().default("Unknown"), city: z.string().optional().default("Unknown"), region: z.string().optional().default("Unknown"), latitude: z.string().optional().default("Unknown"), longitude: z.string().optional().default("Unknown"), ua: z.string().optional().default("Unknown"), browser: z.string().optional().default("Unknown"), browser_version: z.string().optional().default("Unknown"), engine: z.string().optional().default("Unknown"), engine_version: z.string().optional().default("Unknown"), os: z.string().optional().default("Unknown"), os_version: z.string().optional().default("Unknown"), device: z.string().optional().default("Desktop"), device_vendor: z.string().optional().default("Unknown"), device_model: z.string().optional().default("Unknown"), cpu_architecture: z.string().optional().default("Unknown"), bot: z.boolean().optional(), referer: z.string().optional().default("(direct)"), referer_url: z.string().optional().default("(direct)"), }), });
Enter fullscreen mode Exit fullscreen mode

4. Configure Auth Token for production

Don't forget to add TINYBIRD_TOKEN to your .env file. It's advised that you create a token that has the minimum scope for your operations:

  • Read from specific pipe
  • Append to specific datasource

Image description

Congrats! 🎉 You successfully configured Tinybird and ready to move to the next

#2 Page View Recording

We will capture the page view event in our PDFviewer component and publish it to our Tinybird's datasource.

Let's build an API function to send data to Tinybird every time a page is viewed:

// pages/api/record_view.ts import { NextApiRequest, NextApiResponse } from "next"; import { publishPageView } from "@/lib/tinybird"; import { z } from "zod"; import { v4 as uuidv4 } from 'uuid'; // Define the validation schema const bodyValidation = z.object({ id: z.string(), linkId: z.string(), documentId: z.string(), viewId: z.string(), time: z.number().int(), duration: z.number().int(), pageNumber: z.string(), ... }); export default async function handle( req: NextApiRequest, res: NextApiResponse ) { // We only allow POST requests if (req.method !== "POST") { res.status(405).json({ message: "Method Not Allowed" }); return; } const { linkId, documentId, viewId, duration, pageNumber } = req.body; const time = Date.now(); // in milliseconds const pageViewId = uuidv4(); const pageViewObject = { id: pageViewId, linkId, documentId, viewId, time, duration, pageNumber: pageNumber.toString(), ... }; const result = bodyValidation.safeParse(pageViewObject); if (!result.success) { return res.status(400).json( { error: `Invalid body: ${result.error.message}` } ); } try { await publishPageView(result.data); res.status(200).json({ message: "View recorded" }); } catch (error) { res.status(500).json({ message: (error as Error).message }); } }
Enter fullscreen mode Exit fullscreen mode

And finally the PDFViewer component that sends the request:

// components/PDFViewer.tsx import { useState, useEffect } from 'react'; const PDFViewer = () => { const [pageNumber, setPageNumber] = useState<number>(1) useEffect(() => { startTime = Date.now(); // update the start time for the new page // when component unmounts, calculate duration and track page view return () => { const endTime = Date.now(); const duration = Math.round(endTime - startTime); trackPageView(duration); }; }, [pageNumber]); // monitor pageNumber for changes async function trackPageView(duration: number = 0) { await fetch("/api/record_view", { method: "POST", body: JSON.stringify({ linkId: props.linkId, documentId: props.documentId, viewId: props.viewId, duration: duration, pageNumber: pageNumber, }), headers: { "Content-Type": "application/json", }, }); } return ( // Your PDF Viewer implementation ); } export default PDFViewer;
Enter fullscreen mode Exit fullscreen mode

#3 Tremor Bar Chart

Let's now create a bar chart to display the page views for each document. We are using tremor.so to build our beautiful dashboard.

# Install tremor with their CLI  npx @tremor/cli@latest init
Enter fullscreen mode Exit fullscreen mode 
// components/bar-chart.tsx import { BarChart } from "@tremor/react"; const timeFormatter = (number) => { const totalSeconds = Math.floor(number / 1000); const minutes = Math.floor(totalSeconds / 60); const seconds = Math.round(totalSeconds % 60); // Adding zero padding if seconds less than 10 const secondsFormatted = seconds < 10 ? `0${seconds}` : `${seconds}`; return `${minutes}:${secondsFormatted}`; }; export default function BarChartComponent({data}) { return ( <BarChart className="mt-6 rounded-tremor-small" data={data} index="pageNumber" categories={["Time spent per page"]} colors={["gray"]} valueFormatter={timeFormatter} yAxisWidth={50} showGridLines={false} /> ); }
Enter fullscreen mode Exit fullscreen mode 
// lib/swr/use-stats.ts import { useRouter } from "next/router"; import useSWR from "swr"; import { getTotalAvgPageDuration } from "@/lib/tinybird/pipes"; export function useStats() { const router = useRouter(); const { id } = router.query as { id: string }; const { data, error } = useSWR( id, () => getTotalAvgPageDuration({ documentId: id, since: 0 }), { dedupingInterval: 10000, } ); return { durationData: data, isLoading: !error && !data, error, }; }
Enter fullscreen mode Exit fullscreen mode 
// pages/document/[id].tsx import { useDocument } from "@/lib/swr/use-document"; import { useStats } from "@/lib/swr/use-stats"; import BarChartComponent from "@/components/bar-chart"; export default function DocumentPage() { const { document, error: documentError } = useDocument(); const { stats, error: statsError } = useStats(); if (documentError) { // handle document error } if (statsError) { // handle stats error } return ( <> <main> {document && ( <header> <h1>{document.name}</h1> {stats && <BarChartComponent data={stats.durationData} />} </header> )} </main> </> ); }
Enter fullscreen mode Exit fullscreen mode

Voila 🎉 The Bar Chart with accurate page-by-page time!

Image description

Conclusion

That's it! We've built a real-time analytics dashboard for page views using Tinybird, Tremor, and Next.js. While the example here is simple, the same concepts can be expanded to handle any kind of analytics your app might need to perform.

Thank you for reading. I'm Marc, an open-source advocate. I am building papermark.com - the open-source alternative to DocSend with millisecond-accurate page analytics.

Help me out!

If you found this article helpful and got to understand Tinybird and dashboarding with Tremor better, I would be eternally grateful if you could give us a star! And don't forget to share your thoughts in the comments ❤️

https://github.com/mfts/papermark

Image description

Top comments (10)

Collapse
 
tbperegrine profile image
Cameron Archer •

Amazing work Marc! And thanks for speaking so highly of Tinybird.

For those interested, Tinybird has a Web Analytics Starter Kit that includes a JavaScript tracker and some predefined SQL metrics. Can be a good kick start if you're working on a similar project.

Collapse
 
mfts profile image
Marc Seitz •

The starter kit is great 🤩

There’s still so much to explore in Tinybird. Materialized Views is next on the list.

Collapse
 
shnai0 profile image
Iuliia Shnai •

Now 10x more value for Papermark

Collapse
 
mfts profile image
Marc Seitz •

10x more value for any project that uses real-time analytics 🤗📈

Collapse
 
vishnusatheesh profile image
Vishnu Satheesh •

Very Informative article. Thank you.

Collapse
 
mfts profile image
Marc Seitz •

You’re very welcome ☺️ so happy it’s helpful to you

Collapse
 
nevodavid profile image
Nevo David •

Amazing!
I didn't know about Tinybird and Tremor.so!

I'm gonna need to utilize them for my next project 🚀

Collapse
 
mfts profile image
Marc Seitz •

Thanks Nevo 🤗

I’m really glad my article is helpful for you. My article says it all - it’s never been easier to bake in analytics for your users.

Collapse
 
mfts profile image
Marc Seitz •

Thanks! 🤩 so glad it's helpful

Collapse
 
chirpier profile image
Gavin Wong •

This is super cool!
Cool stuff! For anyone interested in real-time data visualization and alerting - in just a few lines of code - I'm interested in feedback on the SaaS service I'm launching called Chirpier (chirpier.co).
There are only two steps required - (1) emit data values to the Chirpier service, (2) configure your out of the box charts and watch as they are updated in real-time via web sockets.