CrawlbaseThis article was originally posted on Crawlbase Blog.
Scraping Walmart Best Sellers is a strategic move for anyone interested in the latest market trends. By utilizing the Crawlbase Crawling API and JavaScript, you can easily extract information about the most popular products on Walmart's online store.
This approach is particularly useful for retailers who need to keep their finger on the pulse of consumer demand or for shoppers keen on discovering trending items. The combination of JavaScript and the Crawlbase API simplifies the process, allowing you to automate data retrieval and consistently stay updated with the best-selling products on Walmart.
Our step-by-step guide is designed to help you efficiently collect the data you need, ensuring that you're always informed and ready to make savvy decisions in the dynamic environment of online retail..
Table of Contents
-
Understanding Walmart Best Sellers
- What are Walmart Best Sellers?
- The Significance of Scraping This Data
- Identifying the Specific Data You Want to Extract
-
Scrape Walmart Best Sellers: A Step-by-Step Guide
- Setting Up the Environment
- Fetching HTML using the Crawling API
- Scrape meaningful data with Crawlbase Scrapers
- Scrape Walmart Best Selling Products Details
- Data Extraction Tips: Strategies For Efficiently Scraping Walmart Best Sellers
- Final Words
- Frequently Asked Questions
Understanding Walmart Best Sellers
Before you start into scraping Walmart's Best Sellers, it's essential to understand what this term means, why it's important, and what kind of data you can extract from it.
1. What are Walmart Best Sellers?
Walmart Best Sellers are products currently selling like hotcakes on Walmart's online platform. These are the top-rated items and are in demand among Walmart's customers. They can include a wide range of products, from electronics and clothing to home goods and more.
2. The Significance of Scraping This Data
- Market Insights: Scraping Walmart Best Sellers provides valuable market insights. It helps businesses and individuals understand what products are trending and in high demand, which can be crucial for making informed decisions in e-commerce and retail.
- Price Tracking: By monitoring Best Sellers, you can keep track of price changes, discounts, and promotions. This information can be used for competitive pricing strategies and finding the best deals.
- Product Research: Researchers and analysts use this data to study consumer preferences, identify emerging trends, and evaluate the performance of different product categories over time.
- Content Creation: Content creators, such as bloggers and vloggers, often use Best Sellers’ data to create engaging content, like product reviews and recommendations.
3. Identifying the Specific Data You Want to Extract
When scraping Walmart Best Sellers, the specific data you might want to extract includes:
- Product Names: The names of the top-selling products.
- Prices: The current prices of these products.
- Ratings: Customer ratings and reviews for each product.
- Descriptions: Descriptions or details about the products.
- URLs: Links to the product pages on Walmart's website.
You can choose to extract all or some of this information based on your goals and the insights you want to gain. Having a clear plan for what data you need is essential, as this will guide your scraping efforts and help you use the information effectively.
Understanding Walmart Best Sellers and their associated data is the first step in your scraping journey. With this knowledge, you can move on to using the Crawlbase Crawling API and JavaScript to collect the data you need for your specific purposes.
Scrape Walmart Best Sellers: A Step-by-Step Guide
Setting Up the Environment
To sign up for a free account on Crawlbase and get your private token, go to your Crawlbase account documentation section.
To install the Crawlbase Node.js library, follow these steps:
Make sure you have Node.js installed on your computer. You can download and install it from the official Node.js website if you don't have it.
After you've confirmed that Node.js is installed, open your terminal and enter the following command:
npm install crawlbase
This command will download and install the Crawlbase Node.js library on your system so you can use it for your web scraping project.
- To create a file named
"walmart-scraper.jsyou can use a text editor or an integrated development environment (IDE). Here's how to make the file using a standard command-line approach:
Run this command:
touch walmart-scraper.js
Executing this command will generate an empty walmart-scraper.js file in the specified directory. You can then open this file with your preferred text editor and add your JavaScript code.
Fetching HTML using the Crawling API
You have your API credentials, installed the Crawlbase Node.js library, and created a file called walmart-scraper.js. Now, pick the Walmart best sellers page you want to scrape. In this example, we've chosen the Walmart best sellers page for the electronics category.
To set up the Crawlbase Crawling API, you need to do a few simple steps:
- Ensure you've made the
walmart-scraper.jsfile, as discussed in the previous part. - Just copy and paste the script we give you below into that file.
- Run the script in your terminal using the command
node walmart-scraper.js.
// Import the Crawling API
const { CrawlingAPI } = require('crawlbase');
// Set your Crawlbase token
const api = new CrawlingAPI({ token: 'YOUR_CRAWLBASE_TOKEN' });
// URL of the Walmart page to scrape
const walmartPageURL = 'https://www.walmart.com/shop/best-sellers/electronics';
// Get request to crawl the URL
api
.get(walmartPageURL)
.then((response) => {
if (response.statusCode === 200) {
console.log(response.body);
}
})
.catch((error) => console.error);
The instructions in the script above show you how to use Crawlbase's Crawling API to get data from a Walmart best sellers page. You'll need to set up the API token, specify the Walmart page you want to fetch, and then send a GET request. When you run this code, it will display the raw HTML content of the Walmart page on your console.
Scrape meaningful data with Crawling API Parameters
In the last example, we learned how to get the basic layout of Walmart's bestselling items: the HTML code from their website. However, we might only sometimes want this basic code. What we often need is the specific details from the webpage. The great news is that the Crawlbase Crawling API has special settings that let us easily extract the key details from Walmart's pages. To do this, you must use the "autoparse" feature when working with the Crawling API. This feature simplifies the process of collecting the most critical information in a JSON format. You can do this by updating the walmart-scraper.js file. Let's look at the next example to understand how it works.
// Import the Crawling API
const { CrawlingAPI } = require('crawlbase');
// Set your Crawlbase token
const api = new CrawlingAPI({ token: 'YOUR_CRAWLBASE_TOKEN' });
// URL of the Walmart page to scrape
const walmartPageURL = 'https://www.walmart.com/shop/best-sellers/electronics';
// options for Crawling API
const options = {
autoparse: 'true',
};
// Get request to crawl the URL
api
.get(walmartPageURL, options)
.then((response) => {
if (response.statusCode === 200) {
// Parse the JSON response and print it
console.log(JSON.parse(response.body));
}
})
.catch((error) => {
console.error('API request error:', error);
});
JSON Response:
{
"original_status": 200,
"pc_status": 200,
"url": "https://www.walmart.com/shop/best-sellers/electronics",
"body": {
"alert": "A generic web scraper has been selected. Please contact support if you require a more detailed scraper for your given URL.",
"title": "Best Sellers in Electronics - Walmart.com",
"favicon": "",
"meta": {
"description": "Shop for Best Sellers in Electronics at Walmart and save.",
"keywords": ""
},
"content": "How do you want your items?How do you want your items? Screen Reader Instructions: In this dialog, you will find toggle buttons to select how you want to shop. After selecting, this option saves to customize your shopping experience throughout Walmart to show you relevant products. If you would like to change your preference, you can trigger this dialog on any page. The button to trigger the dialog appears after the \"Cart\" button in the reading and keyboard order. The button will be labeled according to the last option you chose. To exit this dialog, activate the \"Close\" button or press the \"Escape\" key. Shipping Pickup Delivery Please try again in a moment Save with Roku Ultra LT Streaming Device 4K/HDR/Dolby Vision/Dual-Band Wi-Fi with Roku Voice Remote and HDMI Cable 5074.6 out of 5 Stars. 507 reviews Save with Pickup available Shipping, arrives in 2 days Was $1,049.99 SGIN 15.6inch Laptop 4GB DDR4 128GB SSD Windows 11 with 4 Core Intel Celeron, Full HD 1920x1080 26974.5 out of 5 Stars. 2697 reviews Save with Free shipping, arrives in 2 days Was $399.99 TOPVISION 4pcs Security Wired Camera System, 8CH 3MP NVR Home Security, 1080P IP Security Surveillance Cameras with Color Night Vision, IP66 Waterproof, for Indoor Outdoor, No HDD (Wireless Wi-Fi) 23574.6 out of 5 Stars. 2357 reviews Save with Free shipping, arrives in 2 days Was $369.98 ROCONIA 5G WiFi Bluetooth Native 1080P Projector, 13000LM Full HD Movie Projector, LCD Technology 300\" Display Support 4k Home Theater,(Projector Screen Included) 23774.6 out of 5 Stars. 2377 reviews Save with Free shipping, arrives in 2 days Was $169.99 CRUA 24\" 165Hz/180Hz Curved Gaming Monitor - FHD 1080P Frameless Computer Monitor, AMD FreeSync, Low Motion Blur,DP&HDMI Port, Black 9364.7 out of 5 Stars. 936 reviews Save with Free shipping, arrives in 2 days Was $59.99 USX MOUNT Full Motion TV Wall Mount for 47-90 inch TVs Swivels Tilts Extension Leveling Hold up to 132lb Max VESA 600x400mm, 16\" Wood Stud 7894.7 out of 5 Stars. 789 reviews Save with Free shipping, arrives in 2 days Options from $29.99 – $35.99 TOPVISION Wireless Security Camera, 2K WiFi Camera with Outdoor Night Vision, IP66 Outdoor Waterproof Camera for Home Security System, Surveillance Camera with PIR Motion Sensor, 2 Way Audio 32764.5 out of 5 Stars. 3276 reviews Save with Proscan Elite 10.1\" Tablet/Portable DVD Combo, 32GB Storage, Android 11, 1280x800 Resolution, Black 1614 out of 5 Stars. 161 reviews Save with Free shipping, arrives in 2 days Was $99.99 ULTIMEA 2.2ch Sound Bar for TV, Built-in Dual Subwoofer, 2 in 1 Separable Bluetooth 5.3 Soundbar , Bassmax Adjustable TV Surround Sound Bar, HDMI-ARC/Optical/Aux Home Theater Speakers, Wall Mount 4344.5 out of 5 Stars. 434 reviews Save with Free shipping, arrives in 2 days current price $2,999.00 LG 77\" Class 4K UHD OLED Web OS Smart TV with Dolby Vision C2 Series OLED77C2PUA 30604.7 out of 5 Stars. 3060 reviews Free shipping, arrives in 3+ days Was $169.99 5G WiFi Projector with Bluetooth 5.1, 9000 Lumens HD Movie Projector, 1080P 250'' Display Supported 4024.6 out of 5 Stars. 402 reviews Save with Free shipping, arrives in 2 days Options from $449.00 – $1,199.99 Skytech Blaze 3.0 Gaming PC Desktop AMD Ryzen 5 5600G 3.9 GHz, AMD Radeon Graphics, 500GB NVME SSD, 16GB DDR4 RAM 3200, 600W GOLD PSU, 11AC Wi-Fi, Windows 11 Home 64-bit 224.1 out of 5 Stars. 22 reviews Free shipping, arrives in 3+ days Was $749.00 IPASON Gaming Desktop PC, Amd Ryzen 5 5600G 6 Core 3.9GHz, AMD Radeon Graphics Igpu, 1TB SSD, 16GB DDR4 Ram, Windows 11 Home 743.5 out of 5 Stars. 74 reviews Save with 2021 Apple 10.2-inch iPad Wi-Fi 64GB - Space Gray (9th Generation) 51024.7 out of 5 Stars. 5102 reviews Save with Add $189.00 37484.6 out of 5 Stars. 3748 reviews Save with Free shipping, arrives in 2 days Was $298.00 Hisense 58\" Class 4K UHD LED LCD Roku Smart TV HDR R6 Series 58R6E3 83984.2 out of 5 Stars. 8398 reviews Save with Free shipping, arrives tomorrow Options from $149.00 – $179.55 Beats Studio3 Wireless Noise Cancelling Headphones with Apple W1 Headphone Chip- Matte Black 12114.6 out of 5 Stars. 1211 reviews Free shipping, arrives in 3+ days current price $198.00 onn. 50 Class 4K UHD (2160P) LED Roku Smart TV HDR (100012585) 110844.3 out of 5 Stars. 11084 reviews Save with VIZIO 50\" Class V-Series 4K UHD LED Smart TV V505-J09 200994.4 out of 5 Stars. 20099 reviews Save with 7094.7 out of 5 Stars. 709 reviews Save with onn. 32 Class HD (720P) LED Roku Smart TV (100012589) 146814.4 out of 5 Stars. 14681 reviews Save with 16304.4 out of 5 Stars. 1630 reviews Save with VIZIO 65\" Class V-Series 4K UHD LED Smart TV V655-J09 200994.4 out of 5 Stars. 20099 reviews Save with Free shipping, arrives in 2 days current price $248.00 TCL 55\" Class 4-Series 4K UHD HDR Smart Roku TV - 55S451 106024.1 out of 5 Stars. 10602 reviews Save with Free shipping, arrives tomorrow $699.00/ca CyberPowerPC Gamer Master Gaming Desktop, AMD Ryzen 5 5500, 16GB, AMD Radeon RX 6700 10GB, 1TB SSD, Black, GMA6800WST 6704.7 out of 5 Stars. 670 reviews Save with 3104.6 out of 5 Stars. 310 reviews Free shipping, arrives in 3+ days LG 70\" Class 4K UHD 2160P webOS Smart TV - 70UQ7070ZUD 14634.4 out of 5 Stars. 1463 reviews Free pickup tomorrow Free shipping available Was $229.00 HP Stream 14\" Laptop, Intel Celeron N4020 Processor, 4GB RAM, 64GB eMMC, Pink, Windows 11 (S mode) with Office 365 1-yr, 14-cf2112wm 29683.8 out of 5 Stars. 2968 reviews Save with Free shipping, arrives today Was $179.00 Beats Solo3 Wireless On-Ear Headphones with Apple W1 Headphone Chip - Black 20904.5 out of 5 Stars. 2090 reviews Free shipping, arrives in 3+ days SAMSUNG 65\" Class CU7000B Crystal UHD 4K Smart Television UN65CU7000BXZA 15044.5 out of 5 Stars. 1504 reviews Save with Free shipping, arrives in 2 days current price $498.00 onn. 75 Class 4K UHD (2160P) LED Frameless Roku Smart TV (100044717) 28554.1 out of 5 Stars. 2855 reviews Free shipping available current price $248.00 onn. 55 Class 4K UHD (2160P) LED Roku Smart TV HDR (100012586) 37494.2 out of 5 Stars. 3749 reviews Save with Free shipping, arrives in 2 days Was $329.00 HP Chromebook X360 14\" HD Touchscreen 2-in-1, Intel Celeron N4020, 4GB RAM, 64GB eMMC, Teal, 14a-ca0130wm 3764.2 out of 5 Stars. 376 reviews Save with Beats Studio Buds – True Wireless Noise Cancelling Bluetooth Earbuds - Black 15114 out of 5 Stars. 1511 reviews Save with SAMSUNG Galaxy Tab A8, 10.5\" Tablet 32GB (Wi-Fi), Dark Gray 14164.4 out of 5 Stars. 1416 reviews Save with Free shipping, arrives in 2 days Was $133.00 Philips 32\" Class HD (720P) Smart Roku Borderless LED TV (32PFL6452/F7) 36714.4 out of 5 Stars. 3671 reviews Save with Free shipping, arrives in 2 days Options from $479.00 – $618.00 MSI GF63 15\" Gaming Laptop, 144Hz FHD, Intel i5-11400H, NVIDIA RTX 3050, 16GB DDR4, 512GB SSD, Win11 914.4 out of 5 Stars. 91 reviews Store availability varies Out of stock TOPVISION Sound Bar for TV, Soundbar with Subwoofer, Wired & Wireless Bluetooth 5.0 3D Surround Speakers, Optical/AUX/RCA/USB Connection, Wall Mountable, Remote Control 36874.3 out of 5 Stars. 3687 reviews Out of stock VEATOOL Bluetooth Headphones True Wireless Earbuds 60H Playback LED Power Display Earphones with Wireless Charging Case IPX7 Waterproof in-Ear Earbuds with Mic for TV Smart Phone Computer Laptop 16164.6 out of 5 Stars. 1616 reviews Out of stock",
"canonical": "https://www.walmart.com/shop/best-sellers/electronics",
"images": [
"//i5.walmartimages.com/dfw/63fd9f59-b3e1/7a569e53-f29a-4c3d-bfaf-6f7a158bfadd/v1/walmartLogo.svg",
"https://i5.walmartimages.com/dfwrs/76316474-3850/k2-_c6d4aec7-b4a7-4ea4-9223-07c8daef4fcf.v1.png",
"https://i5.walmartimages.com/dfwrs/76316474-f13c/k2-_d4e8ebb4-9d70-46b4-8f2b-ecc4ac774e07.v1.png",
"https://i5.walmartimages.com/dfwrs/76316474-8720/k2-_d747b89f-5900-404d-a101-1a3452480882.v1.png",
"https://i5.walmartimages.com/dfwrs/76316474-39c2/k2-_8deea800-0d44-4984-b1ce-5a3f12b192b7.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-70f6/k2-_b29e64c4-bea1-474c-9b4d-a28e11524b56.v1.jpg",
"https://i5.walmartimages.com/dfw/4ff9c6c9-bf3d/k2-_f17f25c7-dff7-4ac6-8806-10943a345daf.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-70e1/k2-_f03cf59c-7356-455c-b1c8-23131fe6dc36.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-4bca/k2-_a131fab0-528b-45df-87eb-fa18c8eb0c9c.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-e022/k2-_95447efb-7a97-444b-89c8-f52151e8c2ee.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-bbb5/k2-_3299d0e4-ccb1-4bb3-83eb-31173286f728.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-9008/k2-_9abdaa53-07da-4f73-a08a-cfbbb6a13fc6.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-628c/k2-_365e3c70-a606-42e8-b5a1-f02e43f8dfc2.v1.png"
],
"videos": [],
"grouped_images": {
"db": ["//i5.walmartimages.com/dfw/63fd9f59-b3e1/7a569e53-f29a-4c3d-bfaf-6f7a158bfadd/v1/walmartLogo.svg"],
"mr2 br-100 v-btm dn db-m": [
"https://i5.walmartimages.com/dfwrs/76316474-3850/k2-_c6d4aec7-b4a7-4ea4-9223-07c8daef4fcf.v1.png"
],
"v-btm": [
"https://i5.walmartimages.com/dfwrs/76316474-f13c/k2-_d4e8ebb4-9d70-46b4-8f2b-ecc4ac774e07.v1.png",
"https://i5.walmartimages.com/dfwrs/76316474-8720/k2-_d747b89f-5900-404d-a101-1a3452480882.v1.png",
"https://i5.walmartimages.com/dfwrs/76316474-39c2/k2-_8deea800-0d44-4984-b1ce-5a3f12b192b7.v1.png"
],
"mw-100 absolute left-0 bottom-0": [
"https://i5.walmartimages.com/dfw/4ff9c6c9-70f6/k2-_b29e64c4-bea1-474c-9b4d-a28e11524b56.v1.jpg",
"https://i5.walmartimages.com/dfw/4ff9c6c9-bf3d/k2-_f17f25c7-dff7-4ac6-8806-10943a345daf.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-70e1/k2-_f03cf59c-7356-455c-b1c8-23131fe6dc36.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-4bca/k2-_a131fab0-528b-45df-87eb-fa18c8eb0c9c.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-e022/k2-_95447efb-7a97-444b-89c8-f52151e8c2ee.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-bbb5/k2-_3299d0e4-ccb1-4bb3-83eb-31173286f728.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-9008/k2-_9abdaa53-07da-4f73-a08a-cfbbb6a13fc6.v1.png",
"https://i5.walmartimages.com/dfw/4ff9c6c9-628c/k2-_365e3c70-a606-42e8-b5a1-f02e43f8dfc2.v1.png"
],
"absolute top-0 left-0": [
"https://i5.walmartimages.com/asr/ac793ff4-d3da-4cbe-beba-f956e7494490_1.5bae15688eb0aafd22a99d01a072f9db.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/09219b67-9e5f-4658-994a-05ab956e6ffb.f61269f971887a13c79d2931e1c62694.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/5ef57c1c-eb19-4975-a8f5-468530ca131a.2237d30635f2d0b3ab76518bf69ccb2d.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/448940ca-3434-493d-ba05-8405f337392a.f58aea8fbf600a69f7aa8f2969cde029.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/7611030d-ded9-419e-b9df-bbd6e591f187.65129e104b761b44dd15365ec0edb600.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF"
],
"flex": ["//i5.walmartimages.com/dfw/63fd9f59-ac39/29c6759d-7f14-49fa-bd3a-b870eb4fb8fb/v1/wplus-icon-blue.svg"],
"br-100 v-btm ba b--transparent": [
"https://i5.walmartimages.com/asr/76763ed0-f926-417c-8285-a328d6e91201.37c036e847a376f92d284c922b5c6ef4.jpeg?odnHeight=30&odnWidth=30&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/9c22524e-3900-4a20-8a00-e0703a37d5a6.939ba8e3711f058c2015a770ee3be00f.jpeg?odnHeight=30&odnWidth=30&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/96085919-473b-4348-af43-dd52162d51ad.9b9c40d4e2f1916bf023024a6882a7e3.jpeg?odnBg=FFFFFF&odnHeight=30&odnWidth=30",
"https://i5.walmartimages.com/asr/8766773c-33e8-45f0-a5e2-7b8d7b5a2807.e1ebcd1be09e63793b37c59b1e4e7a43.jpeg?odnBg=FFFFFF&odnHeight=30&odnWidth=30",
"https://i5.walmartimages.com/asr/d1e7a394-40bf-4bd0-ab2f-05962b339c11.7765451801d8914aa40bf76d8e4ee44c.png?odnHeight=30&odnWidth=30&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/ee67ea77-6e64-4148-9f57-048d5c8bb2cf.4ce415aa426d2c96832aa9a9478184ee.png?odnHeight=30&odnWidth=30&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/c52b904a-d551-4a22-96a8-7b79d0427e74_1.894be2a6177ecea43d3141b0fb8de755.png?odnHeight=30&odnWidth=30&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/1a81dac4-f33b-4a5c-9087-51303de450eb_1.6ecc6132b80c59da5e27509de279c0c0.png?odnHeight=30&odnWidth=30&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/ad6b701f-a6f0-4c92-886b-078ae6934a1e_1.8c2fe49d6e265471886e256e3f9fd9b8.png?odnHeight=30&odnWidth=30&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/0ce6ba44-ad2e-445c-a9ff-6c1ab11ad29d.f2b82cc9e22fbc2f4012ea2dcdbec766.png?odnHeight=30&odnWidth=30&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/845982ab-2ef5-4500-b0e4-9aa4c89d79cf.30c60a92a589d8cf97d1f49ec04f2622.png?odnHeight=30&odnWidth=30&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/42acaa56-6085-4729-8512-6547bffcfdb7.16099e0af118f0e2237b885a629c7d16.jpeg?odnHeight=30&odnWidth=30&odnBg=FFFFFF",
"https://i5.walmartimages.com/asr/67f761e1-eefc-47d8-8dd0-5ad824debf9a.01f64ab355f971030adaca50c4b959b3.jpeg?odnBg=FFFFFF&odnHeight=30&odnWidth=30"
],
"no_class_found": [
"https://www.walmart.com/akam/13/pixel_17309a7b?a=dD01MDc2YzdiNWRiODEyYTZiYmRhNzMzYjYxNzZiNTgzZGZhYTU0M2E3JmpzPW9mZg=="
]
},
"og_images": [
"https://i5.walmartimages.com/asr/ac793ff4-d3da-4cbe-beba-f956e7494490_1.5bae15688eb0aafd22a99d01a072f9db.jpeg?odnHeight=180&odnWidth=180&odnBg=ffffff"
],
"links": [
"https://www.walmart.com/shop/best-sellers/electronics#maincontent",
"https://www.walmart.com/",
"https://www.walmart.com/my-items",
"https://www.walmart.com/lists",
"https://www.walmart.com/my-registries",
"https://www.walmart.com/account/login?vid=oaoh",
"https://www.walmart.com/orders",
"https://www.walmart.com/shop/best-sellers",
"https://www.walmart.com/shop/best-sellers/toys",
"https://www.walmart.com/shop/best-sellers/sports-and-outdoors",
"https://www.walmart.com/shop/best-sellers/electronics",
"https://www.walmart.com/shop/best-sellers/home",
"https://www.walmart.com/shop/best-sellers/fashion",
"https://www.walmart.com/shop/best-sellers/baby",
"https://www.walmart.com/shop/best-sellers/beauty",
"https://www.walmart.com/ip/Apple-AirPods-with-Charging-Case-2nd-Generation/604342441?athbdg=L1800",
"https://www.walmart.com/ip/Roku-Ultra-LT-Streaming-Device-4K-HDR-Dolby-Vision-Dual-Band-Wi-Fi-with-Roku-Voice-Remote-and-HDMI-Cable/855978264?athbdg=L1800",
"https://www.walmart.com/ip/SGIN-15-6inch-Laptop-4GB-DDR4-128GB-SSD-Windows-11-with-4-Core-Intel-Celeron-Full-HD-1920x1080/1044996074?athbdg=L1800",
"https://www.walmart.com/ip/TOPVISION-4pcs-Security-Wired-Camera-System-8CH-3MP-NVR-Home-Security-1080P-IP-Surveillance-Cameras-Color-Night-Vision-IP66-Waterproof-Indoor-Outdoor/166697493?athbdg=L1800",
"https://www.walmart.com/ip/ROCONIA-5G-WiFi-Bluetooth-Native-1080P-Projector-13000LM-Full-HD-Movie-LCD-Technology-300-Display-Support-4k-Home-Theater-Projector-Screen-Included/663038446?athbdg=L1800",
"https://www.walmart.com/ip/CRUA-24-165Hz-180Hz-Curved-Gaming-Monitor-FHD-1080P-Frameless-Computer-Monitor-AMD-FreeSync-Low-Motion-Blur-DP-HDMI-Port-Black/1277532195?athbdg=L1800",
"https://www.walmart.com/ip/Skytech-Blaze-3-0-Gaming-PC-Desktop-AMD-Ryzen-5-5600G-3-9-GHz-Radeon-Graphics-500GB-NVME-SSD-16GB-DDR4-RAM-3200-600W-GOLD-PSU-11AC-Wi-Fi-Windows-11-H/1965053079?variantFieldId=actual_color",
"https://www.walmart.com/ip/Skytech-Blaze-Gaming-PC-Desktop-INTEL-Core-i7-12700F-2-1-GHz-NVIDIA-RTX-4060-Ti-1TB-NVME-SSD-16GB-DDR4-RAM-3200-600W-GOLD-PSU-240mm-AIO-11AC-Wi-Fi-Wi/2472931979?variantFieldId=actual_color",
"https://www.walmart.com/ip/IPASON-Gaming-Desktop-PC-Amd-Ryzen-5-5600G-6-Core-3-9GHz-AMD-Radeon-Graphics-Igpu-1TB-SSD-16GB-DDR4-Ram-Windows-11-Home/375037946?athbdg=L1700"
]
}
}
Now that we have the JSON data from the Walmart best sellers page, let's focus on extracting key details such as product titles, prices, ratings, and so on. This step will allow us to understand the product's performance and customer opinions better. Let's go ahead and see what useful information we can gather!
Scrape Walmart Best Selling Products Details
In this example, we'll show you how to extract details of the best-selling products from the HTML content of a Walmart best-seller page that you initially scraped. This involves using two JavaScript libraries: cheerio, commonly used for web scraping, and fs, which is often employed for file system operations.
The below JavaScript code uses the Cheerio library to scrape product details from a Walmart best-seller page. It reads HTML from a “walmart-scraper.js” file, loads it into Cheerio, and grabs info like product name, price, rating, reviews, and image URL. The script goes through each product container, saves the data in a JSON array.
// Import the necessary libraries
const fs = require('fs');
const cheerio = require('cheerio');
// Load the HTML content from the file
const htmlContent = fs.readFileSync('walmart-scraper.html', 'utf8');
// Load HTML content into cheerio
const $ = cheerio.load(htmlContent);
// Select all product containers (assuming they have the same class)
const productContainers = $('.sans-serif.mid-gray.relative.flex.flex-column.w-100.hide-child-opacity');
// Array to store information for all products
const products = [];
// Loop through each product container
productContainers.each((index, element) => {
// Extract product information for each product
const product = {};
// Extract product name
const productNameElement = $(element).find('[data-automation-id="product-title"]');
product.name = productNameElement ? productNameElement.text().trim() : '';
// Extract product price and currency symbol
const productPriceContainer = $(element).find('[data-automation-id="product-price"]');
// Extract the entire price string
const priceString = productPriceContainer.find('.w_iUH7').text().trim();
// Use a regular expression to separate currency symbol and numeric part
const priceMatch = priceString.match(/([^\d]+)([\d,\.]+)/);
if (priceMatch) {
// Combine currency symbol and numeric part into one key: price
// Remove the specific text "[Now]" from the price value
product.price = `${priceMatch[1].trim()}${priceMatch[2]}`;
} else {
// Default value if there is no match
product.price = '';
}
// Extract product rating and reviews
const ratingContainer = $(element).find('.flex.items-center.mt2');
const ratingText = ratingContainer.find('.w_iUH7').text().trim();
// Extract only the rating without the number of reviews and reviews text
const ratingWithoutReviews = ratingText.replace(/\d+\s*reviews/i, '').trim();
product.rating = ratingWithoutReviews !== '' ? ratingWithoutReviews : 'Rating not available';
// Extract only the numeric part of the reviews
const reviewsMatch = ratingText.match(/(\d+)\s*reviews/i);
product.reviews = reviewsMatch ? parseInt(reviewsMatch[1], 10) : 0;
// Extract product image URL
const imageUrlElement = $(element).find('img[data-testid="productTileImage"]');
product.image = imageUrlElement ? imageUrlElement.attr('src') : '';
// Add the product information to the array
products.push(product);
});
// Create a JSON object with the extracted information for all products
const productsJson = JSON.stringify(products, null, 2);
// Print the JSON object to the console
console.log(productsJson);
JSON Response:
[
{
"name": "Apple AirPods with Charging Case (2nd Generation)",
"price": "current price Now $69.00",
"rating": "4.6 out of 5 Stars.",
"reviews": 23387,
"image": "https://i5.walmartimages.com/asr/ac793ff4-d3da-4cbe-beba-f956e7494490_1.5bae15688eb0aafd22a99d01a072f9db.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF"
},
{
"name": "Roku Ultra LT Streaming Device 4K/HDR/Dolby Vision/Dual-Band Wi-Fi with Roku Voice Remote and HDMI Cable",
"price": "current price Now $34.00",
"rating": "4.6 out of 5 Stars.",
"reviews": 507,
"image": "https://i5.walmartimages.com/asr/09219b67-9e5f-4658-994a-05ab956e6ffb.f61269f971887a13c79d2931e1c62694.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF"
},
{
"name": "SGIN 15.6inch Laptop 4GB DDR4 128GB SSD Windows 11 with 4 Core Intel Celeron, Full HD 1920x1080",
"price": "current price Now $259.99",
"rating": "4.5 out of 5 Stars.",
"reviews": 2697,
"image": "https://i5.walmartimages.com/asr/5ef57c1c-eb19-4975-a8f5-468530ca131a.2237d30635f2d0b3ab76518bf69ccb2d.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF"
},
{
"name": "TOPVISION 4pcs Security Wired Camera System, 8CH 3MP NVR Home Security, 1080P IP Security Surveillance Cameras with Color Night Vision, IP66 Waterproof, for Indoor Outdoor, No HDD (Wireless Wi-Fi)",
"price": "current price Now $89.99",
"rating": "4.6 out of 5 Stars.",
"reviews": 2357,
"image": "https://i5.walmartimages.com/asr/448940ca-3434-493d-ba05-8405f337392a.f58aea8fbf600a69f7aa8f2969cde029.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF"
},
{
"name": "ROCONIA 5G WiFi Bluetooth Native 1080P Projector, 13000LM Full HD Movie Projector, LCD Technology 300\" Display Support 4k Home Theater,(Projector Screen Included)",
"price": "current price Now $105.99",
"rating": "4.6 out of 5 Stars.",
"reviews": 2377,
"image": "https://i5.walmartimages.com/asr/7611030d-ded9-419e-b9df-bbd6e591f187.65129e104b761b44dd15365ec0edb600.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF"
},
{
"name": "USX MOUNT Full Motion TV Wall Mount for 47-90 inch TVs Swivels Tilts Extension Leveling Hold up to 132lb Max VESA 600x400mm, 16\" Wood Stud",
"price": "current price Now $35.99",
"rating": "4.7 out of 5 Stars.",
"reviews": 789,
"image": "https://i5.walmartimages.com/asr/e10076ea-53dc-478f-a266-d9a4125e8863.1635e558678d57ae7cb2693fd86a6b5c.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF"
},
{
"name": "CRUA 24\" 165Hz/180Hz Curved Gaming Monitor - FHD 1080P Frameless Computer Monitor, AMD FreeSync, Low Motion Blur,DP&HDMI Port, Black",
"price": "current price Now $99.99",
"rating": "4.7 out of 5 Stars.",
"reviews": 936,
"image": "https://i5.walmartimages.com/asr/1de1931f-f57b-4ab8-9a3a-88fa4da336a7.d2b8d7b8169070a8d8e42cbe69dd9c67.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF"
},
{
"name": "TOPVISION Wireless Security Camera, 2K WiFi Camera with Outdoor Night Vision, IP66 Outdoor Waterproof Camera for Home Security System, Surveillance Camera with PIR Motion Sensor, 2 Way Audio",
"price": "current price Now $29.99",
"rating": "4.5 out of 5 Stars.",
"reviews": 3276,
"image": "https://i5.walmartimages.com/seo/TOPVISION-Wireless-Security-Camera-2K-WiFi-Camera-Outdoor-Night-Vision-IP66-Waterproof-Home-System-Surveillance-PIR-Motion-Sensor-2-Way-Audio_910950fd-5ccf-4aab-b9e6-8a526a1ea8b2.40b1cb2164575b011f635749e2902a5b.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF"
},
{
"name": "Proscan Elite 10.1\" Tablet/Portable DVD Combo, 32GB Storage, Android 11, 1280x800 Resolution, Black",
"price": "current price Now $59.00",
"rating": "4 out of 5 Stars.",
"reviews": 161,
"image": "https://i5.walmartimages.com/asr/85c66610-6cef-421b-b920-e88ed58648d5.848cf4839bf5839369e741c4d2624a0b.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF"
},
{
"name": "ULTIMEA 2.2ch Sound Bar for TV, Built-in Dual Subwoofer, 2 in 1 Separable Bluetooth 5.3 Soundbar , Bassmax Adjustable TV Surround Sound Bar, HDMI-ARC/Optical/Aux Home Theater Speakers, Wall Mount",
"price": "current price Now $55.99",
"rating": "4.5 out of 5 Stars.",
"reviews": 434,
"image": "https://i5.walmartimages.com/asr/69f73701-876b-40dd-9ee5-45baa22436bf.af06bef0cf1b60cc4b9db55d27175e2c.jpeg?odnHeight=784&odnWidth=580&odnBg=FFFFFF"
}
]
Data Extraction Tips: Strategies for Efficiently Scraping Walmart Best Sellers
When scraping Walmart Best Sellers data, it's essential to use effective strategies and follow best practices to ensure a smooth data collection process without encountering issues. Here are some key tips:
- Use Crawlbase Crawling API:
Leverage the Crawlbase Crawling API for structured data extraction. It simplifies the scraping process and provides reliable access to Walmart's Best Sellers data.
- Implement Rate Limiting:
Introduce time delays between your API requests and Walmart's website. This prevents overloading their servers and reduces the risk of getting blocked.
- Rotate User-Agents:
Vary the User-Agent headers in your requests to simulate different web browsers. This makes your scraping activity appear more like human browsing.
- Handle CAPTCHAs Gracefully:
Be prepared to deal with CAPTCHAs, which Walmart may use to verify if you're a bot. Consider using CAPTCHA-solving services or automation techniques to address them.
- Keep Your Code Updated:
Regularly review and update your scraping code to accommodate any changes in Walmart's website structure. This ensures the continued accuracy of your data extraction.
- Respect Robots.txt:
Comply with Walmart's robots.txt file, which outlines guidelines for web crawling. Adhering to these rules helps you avoid legal and ethical concerns.
- Utilize Proxies:
Employ proxy servers to change your IP address, reducing the risk of IP bans and distributing your requests across multiple IPs.
- Verify Data Quality:
Consistently check the quality, accuracy, and currency of the data you scrape. Ensuring the reliability of your collected information is crucial.
- Ethical Data Handling:
Handle scraped data ethically, respecting user privacy and complying with copyright laws and terms of service.
- Test on Small Samples:
Before scaling up your scraping operations, test your code on a smaller sample to identify and address potential issues in a controlled environment.
Final Words
This tutorial has equipped you with the knowledge to efficiently scrape Walmart Best Sellers using JavaScript and the Crawlbase Crawling API. Our additional guides are at your disposal for those interested in extending their data extraction skills to other major retail platforms such as Amazon, eBay, and AliExpress.
We recognize the complexities associated with web scraping and are committed to facilitating your experience. Should you require further assistance or encounter any obstacles, Crawlbase support team is on standby to provide expert help. We look forward to assisting you in your web scraping endeavors.
Frequently Asked Questions
What are Walmart's best sellers?
Walmart's best sellers are the most popular products and are in demand among their customers. These are the items that many people are buying at Walmart stores or online. Best sellers can include various products, from electronics to clothing, toys, and household essentials. By keeping an eye on Walmart's best sellers, you can get a sense of what's currently trending and see what other shoppers are enjoying. This information can help you make informed choices when you're shopping at Walmart or looking for gift ideas.
How do I scrape data from Walmart?
To scrape data from Walmart, you can use JavaScript along with the Crawlbase Crawling API. This powerful combo allows you to automate the process of gathering information from Walmart's website. You can extract product details, prices, ratings, and more. Start by writing a script in JavaScript that interacts with the Walmart website, and then utilize the Crawlbase Crawling API to access and collect the data. It's a straightforward way to retrieve the information you need for price comparison, trend analysis, or any other purpose, making your data extraction tasks easier and more efficient.
Can I scrape data from Walmart?
Yes, you can scrape data from Walmart's website. You can gather information like product details, prices, and more using web scraping tools and techniques. However, reviewing Walmart's terms of service and robots.txt file is important to ensure you're scraping within their guidelines and policies.
What is Walmart's data strategy?
Walmart's data strategy is all about using information to make better decisions. They collect data from in-store and online purchases, analyze it to understand customer preferences and improve their operations. By harnessing data, Walmart aims to offer customers what they want and streamline their business processes for efficiency.
What tools do I need for scraping Walmart Best Sellers?
You'll need a programming environment, a web browser, Crawlbase Crawling API, and basic knowledge of JavaScript.
