Chris NoringFollow me on Twitter, happy to take your suggestions on topics or improvements /Chris
TLDR; this article teaches you to implement the framework Express to a certain degree. Great for your own learning but don't use in production unless you got issues with space or bandwidth doing an NPM install. Hope it's helpful
The reason I write these kinds of articles is not that I want people to reinvent the wheel but to learn from the experience. I bet if you search npmjs you would find 100s of implementations looking more or less like one of the big known frameworks, Express, Nest, Koa, or Fastify. So what would creating one more framework do? Isn't that a waste of time? I don't think so and the reason is that you can learn a lot by trying to implement it yourself. You can acquire skills that help you in your everyday web dev life. It can also set you up nicely for OSS work as you now see the Matrix.
Implementing the Express framework
For this article, I've chosen to try to implement a part of the Express framework. What parts is that exactly?
-
Routes, Express has a way of associating specific routes and have specific code run if a route is hit. You are also able to differentiate routes based on HTTP Verb. So a GET to
/productsis different from aPOSTto/products. - Middleware, is a piece of code that can run before or after your request and even control what should happen to the request. Middleware is how you could inspect a header for an auth token and if valid return the asked for resources. If the token is not valid then the request stops there and a suitable message can be sent back.
-
Query parameters, this is the end part of the URL and is able to help further filter down what you want the response to look at. Given a URL looking like so
/products?page=1&pagesize=20, the query parameters are everything that happens after?. - Sending data with a Body, data can be sent from the client to the server application. It can be sent either over the URL or through a body. The body can contain different things, everything from JSON to simple form fields to even files.
An example express App
Let's look at a few lines of implementing an Express app. There are a lot of things happening even with a few lines:
const express = require('express')
const app = express();
app.get('/products/:id', (req, res) => {
res.send(`You sent id ${req.params.id}`)
})
app.listen(3000, () => {
console.log('Server up and running on port 3000')
})
A Vanilla HTTP app
How would we go about implementing that? Well, we have the HTTP module at our disposal. So let's look at a very small implementation to understand what's missing:
const http = require('http');
const PORT = 3000;
const server = http.createServer((req, res) => {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('hello world');
});
server.listen(PORT, () => {
console.log(`listening on port ${PORT}`)
})
The HTTP module only has a very basic sense of routing. If you navigate toward such an app with URL http://localhost:3000/products the req.url will contain /products and req.method will contain the string get. That's it, that's all you have.
Implementing routing and HTTP Verbs
We are about to implement the following
-
HTTP Verb methods, we need methods like
get(),post()etc. -
Routing and route parameters, we need to be able to match
/productsand we need to be able to break out the route parameter id from an expression looking like this/products/:id. -
Query parameters, we should be able to take a URL like so
http://localhost:3000/products?page=1&pageSize=20and parse out the parameterspageandpageSizeso that they are easy to work with.
HTTP Verb methods
Let's create a server.js and start implementing our server like so:
// server.js
const http = require('http')
function myServer() {
let routeTable = {};
http.createServer((req, res) => {
});
return {
get(path, cb) {
routeTable[path] = { 'get': cb }
}
}
}
Let's leave the code like that and continue to implement the routing.
Parsing route parameters
Implementing /products is easy, that's just string comparison with or without RegEx. Digging out an id parameter from /products/:id is slightly more tricky. We can do so with a RegEx once we realize that /product/:id can be rewritten as the RegEx /products/:(?<id>\w+). This is a so-called named group that when we run the match() method will return an object containing a groups property with content like so { id: '1' } for a route looking like so /products/1. Let's show such an implementation:
// url-to-regex.js
function parse(url) {
let str = "";
for (var i =0; i < url.length; i++) {
const c = url.charAt(i);
if (c === ":") {
// eat all characters
let param = "";
for (var j = i + 1; j < url.length; j++) {
if (/\w/.test(url.charAt(j))) {
param += url.charAt(j);
} else {
break;
}
}
str += `(?<${param}>\\w+)`;
i = j -1;
} else {
str += c;
}
}
return str;
}
module.exports = parse;
And to use it:
const parse = require('./url-to-regex');
const regex = parse("/products/:id")).toBe("/products/(?<id>\\w+)");
const match = "/products/114".match(new RegExp(regex);
// match.groups is { id: '114' }
Adding routing to the server
Let's open up our server.js file again and add the route management part.
// server.js
const http = require('http')
const parse = require('./regex-from-url')
function myServer() {
let routeTable = {};
http.createServer((req, res) => {
const routes = Object.keys(routeTable);
let match = false;
for(var i =0; i < routes.length; i++) {
const route = routes[i];
const parsedRoute = parse(route);
if (
new RegExp(parsedRoute).test(req.url) &&
routeTable[route][req.method.toLowerCase()]
) {
let cb = routeTable[route][req.method.toLowerCase()];
const m = req.url.match(new RegExp(parsedRoute));
req.params = m.groups;
cb(req, res);
match = true;
break;
}
}
if (!match) {
res.statusCode = 404;
res.end("Not found");
}
});
return {
get(path, cb) {
routeTable[path] = { 'get': cb }
}
}
}
What we are doing is looping through all the routes in our route dictionary until we find a match. The comparison looks like this:
if (
new RegExp(parsedRoute).test(req.url) &&
routeTable[route][req.method.toLowerCase()]
)
Note also how the router params are parsed and placed on the params property like so:
const m = req.url.match(new RegExp(parsedRoute));
req.params = m.groups;
Query parameters
We already know that using the HTTP module, the URL will contain our route, like so /products?page=1&pageSize. The next step is to dig out those parameters. That can be accomplished by using a RegEx like and the below code:
// query-params.js
function parse(url) {
const results = url.match(/\?(?<query>.*)/);
if (!results) {
return {};
}
const { groups: { query } } = results;
const pairs = query.match(/(?<param>\w+)=(?<value>\w+)/g);
const params = pairs.reduce((acc, curr) => {
const [key, value] = curr.split(("="));
acc[key] = value;
return acc;
}, {});
return params;
}
module.exports = parse;
Now we need to tie that into the server code. That's just a few lines, fortunately:
const queryParse = require('./query-params.js')
// the rest omitted for brevity
ress.query = queryParse(req.url);
Sending data with a Body
Reading the body can be done by realizing that the input parameter req is of type stream. It's good to know that data arrives in small pieces, so-called chunks. By listening to the event end the client is letting is now that the transmission is complete and no more data will be sent.
You can listen to incoming data by listening to the event data, like so:
req.on('data', (chunk) => {
// do something
})
req.on('end', () => {
// no more data
})
To implement listening to data being transmitted from a client we can, therefore, create the following helper method:
function readBody(req) {
return new Promise((resolve, reject) => {
let body = "";
req.on("data", (chunk) => {
body += "" + chunk;
});
req.on("end", () => {
resolve(body);
});
req.on("error", (err) => {
reject(err);
});
});
}
and then use it in our server code like so:
res.body = await readBody(req);
The full code at this point should look like this:
// server.js
const http = require('http')
const queryParse = require('./query-params.js')
const parse = require('./regex-from-url')
function readBody(req) {
return new Promise((resolve, reject) => {
let body = "";
req.on("data", (chunk) => {
body += "" + chunk;
});
req.on("end", () => {
resolve(body);
});
req.on("error", (err) => {
reject(err);
});
});
}
function myServer() {
let routeTable = {};
http.createServer(async(req, res) => {
const routes = Object.keys(routeTable);
let match = false;
for(var i =0; i < routes.length; i++) {
const route = routes[i];
const parsedRoute = parse(route);
if (
new RegExp(parsedRoute).test(req.url) &&
routeTable[route][req.method.toLowerCase()]
) {
let cb = routeTable[route][req.method.toLowerCase()];
const m = req.url.match(new RegExp(parsedRoute));
req.params = m.groups;
req.query = queryParse(req.url);
req.body = await readBody(req);
cb(req, res);
match = true;
break;
}
}
if (!match) {
res.statusCode = 404;
res.end("Not found");
}
});
return {
get(path, cb) {
routeTable[path] = { 'get': cb }
},
post(path, cb) {
routeTable[path] = { 'post': cb }
}
}
}
At this point you should be able to call your code like so:
const server = require('./server')
const app = server();
app.get('/products/:id', (req, res) => {
// for route /products/1, req.params has value { id: '1' }
})
app.get('/products/', (req, res) => {
// for route /products?page=1&pageSize=10, req.query has value { page: '1', pageSize: '10' }
})
app.post('/products/', (req, res) => {
// req.body should contain whatever you sent across as client
})
Response helpers
At this point, a lot is working. But how do you actually return data back to the client? Because you are implementing the HTTP module the res parameter can be used. By calling its end() you can send data back. Here's an example:
res.end('some data')
However, if you look at how Express does it, it has all sorts of helpers for this like send(), json(), html() and so on. You can have that too with a few lines of code:
function createResponse(res) {
res.send = (message) => res.end(message);
res.json = (message) => {
res.setHeader("Content-Type", "application/json");
res.end(JSON.stringify(message));
};
res.html = (message) => {
res.setHeader("Content-Type", "text/html");
res.end(message);
}
return res;
}
and make sure you add it in the server code:
res = createResponse(res);
Middleware
Having middleware allows us to run code before or after the request, or even control the request itself. Have a look at the following code:
server.get("/protected", (req, res, next) => {
if (req.headers["authorization"] === "abc123") {
next();
} else {
res.statusCode = 401;
res.send("Not allowed");
}
}, (req, res) => {
res.send("protected route");
});
The second argument is the middleware. It inspects req.headers for an authorization property and checks its value. If everything is ok it invokes next(). If it's not ok then the request stop here and res.send() is invoked and status code set to 401, not allowed.
The last argument is the route response you want the client to see providing they send you an ok header value.
Let's implement this. Create the following function in server.js:
function processMiddleware(middleware, req, res) {
if (!middleware) {
// resolve false
return new Promise((resolve) => resolve(true));
}
return new Promise((resolve) => {
middleware(req, res, function () {
resolve(true);
});
});
}
Above the middleware param is being called and you can see how the last argument to it is a function that resolves a Promise like so:
middleware(req, res, function () {
resolve(true);
});
For the server code to use this, there are a few steps we need to take:
- Ensure we register the middleware
- Get a hold of the middleware when we have a matching request
- Call the middleware
Register middleware
We need to change slightly how we register routes by first adding this helper method:
function registerPath(path, cb, method, middleware) {
if (!routeTable[path]) {
routeTable[path] = {};
}
routeTable[path] = { ...routeTable[path], [method]: cb, [method + "-middleware"]: middleware };
}
So trying to register a route like so:
server.get('/products', (req, res, next) => {}, (req, res) => {})
leads to the middleware callback being saved on a property get-middleware
Then when we register the route we do something like this instead:
return {
get: (path, ...rest) => {
if (rest.length === 1) {
registerPath(path, rest[0] , "get");
} else {
registerPath(path, rest[1], "get", rest[0]);
}
},
Get a reference to the middleware
To get a reference to the middleware we can use this code:
let middleware = routeTable[route][`${req.method.toLowerCase()}-middleware`];
Process middleware
Finally, to run the middleware write the below code:
const result = await processMiddleware(middleware, req, createResponse(res));
if (result) {
cb(req, res);
}
Summary
The full code is available at this repo:
and it can also be used via NPM by calling:
npm install quarkhttp
That was a lot, routing, routing parameters, query parameters, body parsing, and middleware. Hopefully, you can now understand what's going on. Remember that there are great libraries out there for you to use that are well tested. However, understanding how things are implemented can be really beneficial for your understanding.
