Cover image credit: Hunter x Hunter manga by Yoshihiro Togashi, meme-ified by yours truly. <3

In a recent technical challenge, I was asked to build a small Node.js app that first needed to retrieve some JSON from the web. Since I'm still relatively new to Node.js, I did not realize that Node.js does not natively include the fetch() API, which I was used to using in my front-end JavaScript.

(What I didn't realize is that fetch() is actually a method from the Window interface--and not having a front-end, there was no window! Awkward thing to realize at the beginning of a technical interview...)

After getting through the challenge, I spent this past weekend refactoring the code and experimenting with a few different ways to retrieve JSON from the web using Node.js. Spoiler: there's an http/https module in Node.js, but also some cool packages that mimic fetch(), or that simplify request syntax even further!

But before we get into that, I want to introduce a tool that I foolishly did not use during my technical challenge: JSONView, a super-handy web extension for Chrome and Firefox that pretty-prints JSON in your browser.

JSONView, a super-handy web extension for Chrome and Firefox that pretty-prints JSON in your browser

One issue I had during my code challenge was (due to my own error) not working on my own computer, which includes this extension. Compare the following:

Unformatted JSON from https://www.reddit.com/r/popular.json

versus

Same JSON from https://www.reddit.com/r/popular.json, pretty-printed with JSONView

On top of that, hovering your cursor over a particular field will show the path to access it:

Cursor hovering over the "ups" field, with the path shown in the bottom-left corner

Having this handy will make parsing and accessing the data you need way faster and easier.

Replicating fetch() with 'node-fetch' package

The node-fetch package does pretty much what you expect: provide you with the fetch() syntax in Node.js. To install, run npm install node-fetch, and set up your code like this:

const fetch = require('node-fetch');

let url = "https://www.reddit.com/r/popular.json";

let settings = { method: "Get" };

fetch(url, settings)
    .then(res => res.json())
    .then((json) => {
        // do something with JSON
    });

Here, we've started by importing the package via require(), and created a settings variable to define our http method as a Get request. From there, we use fetch(url, settings) just like we would on the front-end. As usual, we can parse the response res as JSON, and then do whatever we need to with it.

Note: from some VERY RUDIMENTARY benchmark testing, it appears that node-fetch is the fastest of the three options covered in this article. Here are the times clocked by each (however, this DOES include running the rest of the code from the challenge, not just the fetch/https/request itself):

fetch: 0.689 seconds
https: 2.827 seconds
request: 3.65 seconds

I'd love for someone else to do a little more testing and verify/disprove this! Feel free to comment below if you're that person. ;)

Using the http/https modules provided by Node.js

Node.js comes with a pair of http/https modules, and in this case, the https module provides a built-in method for Get requests. Here's the code we'll be looking at:

const https = require('https');

let url = "https://www.reddit.com/r/popular.json";

https.get(url,(res) => {
    let body = "";

    res.on("data", (chunk) => {
        body += chunk;
    });

    res.on("end", () => {
        try {
            let json = JSON.parse(body);
            // do something with JSON
        } catch (error) {
            console.error(error.message);
        };
    });

}).on("error", (error) => {
    console.error(error.message);
});

There's a bit more going on here! First, we import the https module with require(). We can then call https.get(url, (res) => {} ) to initiate a Get request. Then, inside the body of the callback, we start by creating an empty string body that we'll add our the text of our response (again called res) to.

From there, we have a few examples of the .on syntax, which will listen for a few different events--namely, "data", "end", and "error".

When the response encounters "data", we add each chunk as text to our body variable. Once we hit the "end" of the response, we use the try / catch syntax to try to parse our body's text as JSON, and return an error if it can't. Lastly, we chain another .on call to catch "error" for our initial https.get() request.

I find this syntax to be pretty clunky and verbose, although I do like the explicit error handling that is required by https.get(). However, this module is slower than the node-fetch package--see the benchmark results above.

Simplifying syntax with 'request' package

The third strategy I used was the request package, which aims to simplify the (often verbose) syntax of Node.js's http requests. Since this is an external package, start by installing it with npm install request.

Here's the code we'll be looking at:

const request = require('request');

let url = "https://www.reddit.com/r/popular.json";

let options = {json: true};



request(url, options, (error, res, body) => {
    if (error) {
        return  console.log(error)
    };

    if (!error && res.statusCode == 200) {
        // do something with JSON, using the 'body' variable
    };
});

Wow, that's really readable! Let's break it down. As with the other examples, we import the package with require(), and set our url variable. The request package also has a nifty options feature, where you can specify a lot of things--but here, in setting { json: true }, we tell the request to automatically parse the response's body as JSON if there's no error (and we get a 200 status code back). So, to access the JSON we want, just use the body variable!

This readability comes at the price of speed, however. Per the benchmark results above, this is the slowest option, most likely because so much is happening under-the-hood. However, the readability is top-notch, and configuring other http requests are just as simple as this Get request example!

Conclusion

This particular technical challenge was a great opportunity to dive into Node.js's http requests! Now, you should feel armed with a variety of tools to meet different situations.

As I said above, I'd love to have another person do some testing/benchmarking and verify or disprove the speed test results that I got! Since testing is still relatively new to me, I'd very much like to see how others approach benchmarking these methods. Thanks for reading, and feel free to comment below!