DEV Community

Wachira
Wachira

Posted on • Originally published at blog.bywachira.com

Create API with Gin in Golang Part 3

#scrapper #go #mongodb #mgo

The third part of this series is here, its actually an interesting one because we dabble with a bit of web scrapping and Gin middlewares. In the last part, I mentioned we are going to create a bookmark API instead of TODO, I figured there are way too many out there.

By the end of this article you will have a number of endpoints that perform certain CRUD operations, these are:

  • Create a bookmark The link is scrapped to collect the meta tag details for Twitter and Facebook if we ever want to show Rich link previews on the client side.
  • Delete a bookmark
  • Protect the endpoints with the help of the middleware
  • Fetch all bookmarks they saved.

Introduction

Prerequisites

The basics of Golang and you read the previous articles in this series.

Getting Started

Goals

  • A logged-in user is able to create a bookmark
  • A logged-in user is able to fetch their own bookmarks
  • A logged-in user is able to delete a bookmark they created
  • Protect bookmark endpoints with middleware to confirm that

Setup

Go through the last articles for the project setup

Let's hack it

Authentication Middleware

We will need to create a middleware to check if users have logged in order to perform certain actions which are creating, fetching, and deleting bookmarks.

First, we will create a new file under middlewares folder, called authenticate.go

// Define the package package middlewares import ( // Import Gin to abort on any errors "github.com/gin-gonic/gin" // To retrieve certain methods and User model "github.com/tesh254/golang_todo_api/models" // Services to enable use to handle decoding jwt tokens "github.com/tesh254/golang_todo_api/services" ) // This will help in handling error response func responseWithError(c *gin.Context, code int, message interface{}) { c.AbortWithStatusJSON(code, gin.H{"message": message}) } // Authenticate is a middleware that fetches user details from token func Authenticate() gin.HandlerFunc { return func(c *gin.Context) { // Fetch token from the headers requiredToken := c.Request.Header["Authorization"] // Check if the token is provided if len(requiredToken) == 0 { // Abort with error responseWithError(c, 403, "Please login to your account") } // Get email from encoded token userID, _ := services.DecodeToken(requiredToken[0]) // Fetch user based on email retrieved from token result, err := new(models.UserModel).GetUserByEmail(userID) // Check if an account was not found if result.Email == "" { // Respond with a 404 when resource is not found responseWithError(c, 404, "User account not found") return } // Check if an error occured while fetching a user if err != nil { // Respond with an Internal Server Error responseWithError(c, 500, "Something went wrong giving you access") return } // Set the User variable so that we can easily retrieve from other middlewares c.Set("User", result) // Call the next middlware c.Next() } }
Enter fullscreen mode Exit fullscreen mode

Scrapping a website

While making this API, having it all plain CRUD did seem fun and felt lazy so I decided to make a scrapper to check the meta tags for the save bookmark.

I was able to achieve this with the help of goquery and net library

The meta tags that we are interested in are twitter: and og: which help Twitter and Facebook respectively show rich link previews when you share links, like the one below

An example of a Twitter meta tag is shown below

<meta property="twitter:site" content="Dev Community" />
Enter fullscreen mode Exit fullscreen mode

All Facebook meta tags have og guess it means Open Graph like their API.

Alt Text

So lets install goquery

go get github.com/PuerkitoBio/goquery
Enter fullscreen mode Exit fullscreen mode

Next lets write the scrapper

// Define the package package services import ( // will allow us to make calls to the website "net/http" // help in defining a timeout for the request "time" // Help in finding tags and attributes in the html loaded "github.com/PuerkitoBio/goquery" // To handle failed responses "github.com/gin-gonic/gin" ) // Scrapper holds methods to scrape a site type Scrapper struct{} // Meta define meta data type type Meta struct { Image, Description, URL, Title, Site string } // CallWebsite make an http request to a website func (scrapper *Scrapper) CallWebsite(websiteURL string, c *gin.Context) Meta { // Define variable to hold the attributes srapped var meta Meta = Meta{ Image: "", Description: "", URL: "", Title: "", Site: "", } // Define the client timeout client := &http.Client{ // Set timeout to abort if the request takes too long Timeout: 30 * time.Second, } // Define the request obviously it is a GET one request, err := http.NewRequest("GET", websiteURL, nil) // Check if we get an error if err != nil { // Respond with a 500 status code c.AbortWithStatusJSON(500, gin.H{"message": err}) } // Set headers request.Header.Set("pragma", "no-cache") request.Header.Set("cache-control", "no-cache") request.Header.Set("dnt", "1") request.Header.Set("upgrade-insecure-requests", "1") request.Header.Set("referer", websiteURL) // Make website request call resp, err := client.Do(request) // If we have a successful request if resp.StatusCode == 200 { doc, err := goquery.NewDocumentFromReader(resp.Body) if err != nil { c.AbortWithStatusJSON(400, gin.H{"message": err}) } // Map through all meta tags fetched doc.Find("meta").Each(func(i int, s *goquery.Selection) { // Specify the meta tags we want and assign the variables as the doc.Find // loops through all of them metaProperty, _ := s.Attr("property") metaContent, _ := s.Attr("content") // If we happen to get any of the two then assign the Site attribute for Meta if metaProperty == "og:site_name" || metaProperty == "twitter:site" { meta.Site = metaContent } // If we happen to get any of the two then assign the URL attribute for Meta if metaProperty == "og:url" { meta.URL = metaContent } // If we happen to get any of the two then assign the Image attribute for Meta if metaProperty == "og:image" || metaProperty == "twitter:image" { meta.Image = metaContent } // If we happen to get any of the two then assign the Title attribute for Meta if metaProperty == "og:title" || metaProperty == "twitter:title" { meta.Title = metaContent } // If we happen to get any of the two then assign the Description attribute for Meta if metaProperty == "og:description" || metaProperty == "twitter:description" { meta.Description = metaContent } }) } // Return the meta variable with updated fields return meta }
Enter fullscreen mode Exit fullscreen mode

Create a bookmark model

Create a file in the models folder called bookmark.go and lets jump into some code, there is nothing new about this file just defining the methods we are going to use to perform Read and Write operations to the database

// Define the package package models import ( "time" "gopkg.in/mgo.v2/bson" ) // Link defines user object structure type Link struct { ID bson.ObjectId `json:"_id,omitempty" bson:"_id,omitempty"` Name string `json:"name" bson:"name"` MetaImage string `json:"meta_image" bson:"meta_image"` MetaDescription string `json:"meta_description" bson:"meta_description"` MetaSite string `json:"meta_site" bson:"meta_site"` MetaURL string `json:"meta_url" bson:"meta_url"` MetaTitle string `json:"meta_title" bson:"meta_title"` Link string `json:"link" bson:"link"` Owner bson.ObjectId `json:"owner" bson:"owner"` CreateAt time.Time `json:"created_at" bson:"created_at"` } // BookmarkModel defines the model structure type BookmarkModel struct{} // CreateBookmark handles creating a bookmark by the user func (l *BookmarkModel) CreateBookmark(data Link) error { // Connect to the bookmark collection collection := dbConnect.Use(databaseName, "bookmark") // Assign result to error object while saving bookmark err := collection.Insert(bson.M{ "name": data.Name, "meta_image": data.MetaImage, "meta_description": data.MetaDescription, "meta_site": data.MetaSite, "meta_url": data.MetaURL, "meta_title": data.MetaTitle, "link": data.Link, "owner": data.Owner, "created_at": data.CreateAt, }) return err } // FetchBookmarks handles fetching bookmarks by a user func (l *BookmarkModel) FetchBookmarks(id bson.ObjectId) (bookmarks []Link, err error) { collection := dbConnect.Use(databaseName, "bookmark") err = collection.Find(bson.M{"owner": id}).Sort("-$natural").All(&bookmarks) return bookmarks, err } // DeleteBookmark handles deleting a bookmark func (l *BookmarkModel) DeleteBookmark(id string) error { collection := dbConnect.Use(databaseName, "bookmark") err := collection.RemoveId(id) return err }
Enter fullscreen mode Exit fullscreen mode

Create the bookmark controllers

Go ahead and create a file on the controllers folder called bookmark.go

package controllers import ( "log" "time" "github.com/gin-gonic/gin" "github.com/tesh254/golang_todo_api/forms" "github.com/tesh254/golang_todo_api/helpers" "github.com/tesh254/golang_todo_api/models" "github.com/tesh254/golang_todo_api/services" ) // Import bookmark model from the models file var bookmarkModel = new(models.BookmarkModel) // BookmarkController defines the bookmark controller type BookmarkController struct{} func responseWithError(c *gin.Context, code int, message interface{}) { c.AbortWithStatusJSON(code, gin.H{"message": message}) } // FetchBookmarks controller handles fetching all bookmarks of a specific user func (b *BookmarkController) FetchBookmarks(c *gin.Context) { // Get the user we set in the Authenticate middleware user := c.MustGet("User").(models.User) // Check if we have user and respond if not if user.Email == "" { responseWithError(c, 404, "Please login") return } // Define a variable to hold the model methods var linkModel models.BookmarkModel // Fetch own bookmarks based on ID of the user results, err := linkModel.FetchBookmarks(user.ID) // Check if we have an error performing the database operation if err != nil { responseWithError(c, 500, "Problem fetching your articles") return } // Return results if none an empty array if results != nil { c.JSON(200, gin.H{"bookmarks": results}) } else { c.JSON(200, gin.H{"bookmarks": []string{}}) } } // DeleteBookmark controller handles deleting a single bookmark func (b *BookmarkController) DeleteBookmark(c *gin.Context) { // Get the user we set in the Authenticate middleware user := c.MustGet("User").(models.User) // Check if we have user and respond if not if user.Email == "" { responseWithError(c, 404, "Please login") return } // Get query parameter value that holds the bookmakr id bookmarkID, found := c.GetQuery("bookmark_id") // Check if the query parameter `bookmark_id` is provided, respond if not if !found { responseWithError(c, 400, "Please provide bookmark id") return } // Define variable to hold the model methods var linkModel models.BookmarkModel // Delete record err := linkModel.DeleteBookmark(bookmarkID) // Check if we got an error while deleting the file if err != nil { responseWithError(c, 500, "Problem deleting bookmark") return } // Respond with a 204 No Content on successful delete c.JSON(204, gin.H{"message": "Deleted bookmark successfully"}) } // CreateBookmak controller handles creating a bookmark of a specifi user func (b *BookmarkController) CreateBookmak(c *gin.Context) { // Get the user we set in the Authenticate middleware user := c.MustGet("User").(models.User) // Check if we have user and respond if not if user.Email == "" { responseWithError(c, 404, "Please login") return } // Define variable to hold the payload structure var data forms.BookmarkPayload // Check if required fields are provided if c.BindJSON(&data) != nil { log.Fatal(c.BindJSON(&data)) responseWithError(c, 406, "Please provide link, and name") return } // Define variable to hold the model methods var linkModel models.BookmarkModel // Check if the url is valid and respond if its not if !helpers.IsValidURL(data.Link) { responseWithError(c, 400, "Link is invalid") } // Define a variable to hold out scrapper methods var scrapper services.Scrapper // Make a website request based on the link provided on the request body var meta services.Meta = scrapper.CallWebsite(data.Link, c) // Define the data to be save to the database var bookmarkPayload models.Link = models.Link{ Name: data.Name, MetaImage: meta.Image, MetaDescription: meta.Description, MetaSite: meta.Site, MetaURL: meta.URL, Link: data.Link, Owner: user.ID, CreateAt: time.Now(), } // Save the bookmakr err := linkModel.CreateBookmark(bookmarkPayload) // Check if we got an error while saving and respond if we do if err != nil { responseWithError(c, 500, "Problem saving your bookmark") log.Fatal(err) return } // Return a 201 Created on successful creation with the bookmark saved c.JSON(201, gin.H{"message": "Bookmark saved", "bookmark": bookmarkPayload}) }
Enter fullscreen mode Exit fullscreen mode

Some things that I shouldn't leave out

  1. The URL validator 
// Define the package package helpers // Import the URL import "net/url" // IsValidURL checks validity of URL func IsValidURL(link string) bool { // Parse the URL with the net library _, err := url.ParseRequestURI(link) // Check if we got an error while parsing the url if err != nil { return false } u, err := url.Parse(link) // Check if we had an error while parsing // Check if the scheme is provided https:// http:// // Check if a host exists google.com, bywachira.com if err != nil || u.Scheme == "" || u.Host == "" { return false } return true }
Enter fullscreen mode Exit fullscreen mode
  1. Bookmark request body structure 
package forms // BookmarkPayload defines the payload sent by the user type BookmarkPayload struct { Name string `json:"name" binding:"required"` Link string `json:"link" binding:"required"` }
Enter fullscreen mode Exit fullscreen mode

This will belong in the forms/bookmark.go file.

And that's it

Define the endpoints

Finally we need to define the endpoints for the bookmark endpoints for us to make calls and make use of the blocks of code we created

// v1 group... // Define a group to hold the bookmarks bookmarks := v1.Group("/bookmarks") // Use the Authenticate middleware bookmarks.Use(middlewares.Authenticate()) { // Create GET request for the bookmakrs bookmarks.GET("/all", link.FetchBookmarks) // Create POST request to create a bookmark bookmarks.POST("/create", link.CreateBookmak) // Create DELETE request to delete a bookmark bookmarks.DELETE("/delete", link.DeleteBookmark) } // ...
Enter fullscreen mode Exit fullscreen mode

Summary

  • Created endpoints to handle creating, fetching and deleting bookmark(s).
  • Create a middleware to check if user is authenticated.

Extras

  • Repo link here
  • Follow me on twitter here
  • Join Discord server for any questions here

Top comments (3)

Collapse
 
twocs profile image
Tom Anderson • • Edited

Got to this post when investigating the difference between these two:
c.AbortWithStatusJSON and c.JSON
I think when you know that the value and we don't want any further processing, we can use c.AbortWithStatusJSON

So in general, I've been replacing all c.JSON with c.AbortWithStatusJSON in the examples like above.

The problem occurs because further processing can occur after the c.JSON calls, and if there's another c.JSON, it will also add JSON to the response stream. These two are exactly equivalent except that the c.JSON stays alive, which can trip up a developer when edge cases occur.
c.JSON(201, gin.H{"message": "Bookmark saved", "bookmark": bookmarkPayload})
c.AbortWithStatusJSON(201, gin.H{"message": "Bookmark saved", "bookmark": bookmarkPayload})

I'm preferring the second because it returns a successful status and message and stops processing.

Collapse
 
r4lly99 profile image
mohd rully k •

Great post and explanation , glad to read this and try learn it

Collapse
 
duktigdev profile image
David A. •

Very good Article !!!
Helped me a lot of !!!

Thanks !!!