Create custom Google search engine for your domain(s) and fetch results as JSON

Jochem Stoel - Aug 9 '18 - - Dev Community

I decided to write this simple 5 minute tutorial before I go to sleep.

Create a Custom Google Search Engine for your website domain(s) and access it programmatically with barely any code.

Navigate to https://cse.google.com/cse/all where you can create a custom search engine. Click on the "Add" button and provide one or multiple sites to search on. In this case just add dev.to/* to the list.
If you enable the setting called 'Search the entire web', your search results will be augmented with results from the web if nothing (or not enough) is found. That means disable this to make sure it will only ever show results from your web domain(s).

s1

If you want to embed your custom search engine on a webpage with a simple copy/paste then you can choose between 7 different layouts of your search engine. I personally prefer the compact one. These layouts show advertisements.

s2

There are two options for programmatic access. The option Custom Search JSON API is free and has a limit of 10,000 queries per day. This is what you want in this case.

If your Custom Search Engine is restricted to only searching specific sites (10 or fewer), you can use the Custom Search Site Restricted JSON API. This API is similar to the JSON Custom Search API except this version has no daily query limit. Custom Search Site Restricted JSON API requests cost $5 per 1000 queries and there is no daily query limit. You may sign up for billing in the API Console.

You need

API key (get it here: https://developers.google.com/custom-search/json-api/v1/introduction)
Search Engine identifier cx (which you can find in your search engine public URL: https://cse.google.com/cse?cx=009833334622897458665:rtvizlbvdpk)

To query your search engine for 'open source', simply make a GET request to https://www.googleapis.com/customsearch/v1?key=YOUR_API_KEY&cx=SEARCH_ENGINE_ID&q=open+source

Let's say you're on top of things. Add a sort=date parameter to the querystring to sort the results by date (newest first). Google picks up on changes pretty fast as you can see in this screenshot.

s3

/* since this is a public API, it permits cross origin XMLHttpRequests from the browser */
fetch('https://www.googleapis.com/customsearch/v1?key=YOUR_API_KEY&cx=SEARCH_ENGINE_ID&q=open+source&sort=date').then(response => response.json()).then(json => {
   // json.items has the results 
}).catch(console.error)
Enter fullscreen mode Exit fullscreen mode

ZzzzZZ..

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Terabox Video Player