When working with e-commerce website is important to ensure that the landing pages actually offer the products that customers are looking for, otherwise, the whole purpose of bringing them to the website is missed. Ideally, the content management system should be able to handle this information and furthermore, create reports for relevant departments. I said “ideally” because sometimes that is not the case.

Recently just I came across to a solution that could be easily done when crawling the website. Using the Custom filters on Screaming Frog Spider, you can find those pages that don’t have any products listed.

How to set up Filters in Screaming Frog

1. Select a keyword or phrase that will always appear when products are displayed. It is important to make sure that the phrase just appears in this case, otherwise, it could bring incorrect information. In my case, I selected “quote WEB“, which just appear in product listings.

Screaming Frog Filter

Phrase Selected to be included in the filter

2. Then, go to Screaming Frog > Configuration > Custom.

Screaming Frog Menu

Select Configuration in the Menu

Screaming Frog Custom Menu

Select Custom in the Configuration

3. In Custom, you will see 5 filters available. Go to Filter 1, select “Doesn’t contain” and add the phrase previously selected. Press “OK”. This means that the spider will crawl the site and those pages that don’t contain the phrase will be added to under Filter 1.

Filters in Screaming Frog

Up to 5 Filters available in Screaming Frog

4. Crawl the site. Add the URL and press “Start”

5. In the tabs, you will see the Custom tab, click it. Then, there will be a drop-down menu. Select Filter 1. There you will see all the URLs that don’t contain the phrase that you selected.

Select Custom Tab in Dashboard

Select Custom Tab in Dashboard

6. After the crawl is finished, export the list on Filter 1 to a .csv file.

Now you have a list of pages that don’t contain the selected phrase. You can go through the list and double check that your findings are correct. After that, it’s ready to be shared with relevant department.

The following two tabs change content below.
I’m a Senior Search Consultant with experience working in different websites across several industries. My aim is to facilitate the interaction between products/services and customers via organic and paid searches.