Skip to content

Competitor New Pages Alerts: Monitor a Sitemap or Blog Index

A sitemap lists every public page a competitor publishes, and a blog index lists only posts. Here is what each shows, a rule for both and where each one fails.

By AyeWatch Team4 min read

Short answer: Point an AyeWatch web page monitor at a competitor's sitemap or blog index and add a rule that alerts only on new addresses or posts. A sitemap lists every public page, so it catches landing and comparison pages a blog index never shows. A blog index shows posts with titles and dates.

What a sitemap is

A sitemap is an XML file that lists the addresses a site wants search engines to find. The format is defined at sitemaps.org. A file holds a urlset of url entries, each with a loc address and optional lastmod, changefreq and priority fields. One file can hold up to 50,000 addresses or 50 MB uncompressed. Larger sites split the list across several files and publish a sitemap index that points to them.

The usual place is /sitemap.xml, and a site's robots.txt file may carry a "Sitemap:" line with the address. WordPress sites have used /wp-sitemap.xml since version 5.5, and Shopify stores publish /sitemap.xml. Google says it uses lastmod only when the dates are consistently accurate and ignores priority and changefreq.

How a sitemap differs from a blog index

A blog index is a page of posts, newest first, with titles, dates and short summaries. It shows what a competitor chooses to write about. A sitemap shows what pages exist. That includes comparison pages, pricing variants, docs and customer stories that never appear on a blog index.

The two answer different questions. The index tells you what they are saying. The sitemap tells you what they built. A sitemap alert gives you an address, not the page's content, so you open it to read it. The path usually says enough.

A rule for the sitemap

Sample rule. "Alert me when a new address appears in this sitemap that was not there at the last check. Group new addresses by the first folder in the path. Flag any address that contains /vs/, /pricing, /compare, /alternatives or /customers. Ignore addresses for tag pages and page numbers."

Sample alert. The headline reads "Larkspur CRM added 3 pages." The summary lists /vs/hubnest, /vs/pipemint and /pricing/nonprofits, and says two are comparison pages against named rivals and one is a new pricing page for nonprofits. Larkspur CRM and the addresses are invented.

A rule for the blog index

Sample rule. "Alert me when a new post appears on this blog. Give the title, the date and the first line of the summary. Ignore posts about events, hiring and webinars."

That rule is quieter than the sitemap one and easier to read. It is the right choice if you care about messaging and launches and do not need to see every landing page.

Test the first check and pick the right file

The first check lists what is in the file now. Read it. For a large site, open the sitemap index and monitor the child sitemap for the section you care about, such as posts or product pages, instead of the whole list. AyeWatch reads a limited amount of text from a page, so a very long file may be read in part.

Order matters too. Some sitemaps list new pages at the end of the file. If the first check shows only the top of a long list, new addresses may fall outside what is read. Choose a smaller child sitemap or the blog index.

When AyeWatch is the wrong tool

Some sites publish no sitemap. Check robots.txt, and fall back to the blog index. A sitemap never lists private pages, pages behind a login or pages the competitor chose to leave out. A page that was edited rather than added needs its own page monitor. Monitoring competitor pricing pages covers that setup.

A monitor also does not crawl a site. If you need every page with its title, headings and links in a spreadsheet, use an SEO crawler. For delivery, sending alerts to Slack puts new pages where your team already reads.

Cost and effort

One topic per sitemap or blog. AyeWatch Pro is $9/mo with 50 topics and hourly checks; Pro+ is $19/mo with 200 topics and ASAP mode. Every plan starts with a 7-day free trial. Checking one competitor's sitemap and blog uses two topics.

SourceShowsMissesBest for
SitemapEvery public address the site listsPage content, unlisted pages and, often, reliable datesNew landing, comparison and pricing pages
Blog indexTitles, dates and summaries of postsPages that are not postsWhat a competitor writes about
Single page monitorEdits to the text of one known pageNew pagesA pricing or feature page you already know
Topic monitorNews and posts about the competitor across the internetPages nobody has written aboutLaunches and press coverage

Start a monitor in under a minute

Describe a topic or paste a URL. Cancel before the trial ends and you pay nothing.

Try it free for 7 days

The bottom line

Open your closest competitor's robots.txt, find the sitemap and run the first check on it. If the list reads cleanly, keep the rule. If it does not, use the blog index.

FAQ

What is a sitemap?

An XML file that lists the addresses a site wants search engines to find. The format is defined at sitemaps.org.

Where do I find a competitor's sitemap?

Try /sitemap.xml on their domain and check their robots.txt for a Sitemap line. WordPress sites use /wp-sitemap.xml.

How many addresses can one sitemap hold?

Up to 50,000 addresses or 50 MB uncompressed. Larger sites split the list into several files and publish a sitemap index.

Can a sitemap monitor tell me when a page was edited?

Only through the lastmod date, and Google says it uses that date only when it is consistently accurate. For edits to a known page, monitor that page directly.

Is a sitemap or a blog index better to monitor?

The sitemap catches every public page a competitor lists, and the blog index catches only posts. Use the sitemap for launches and the blog index for messaging.

competitor new pages alertsmonitor competitor sitemapcompetitor blog monitorget notified when a competitor adds a pagesitemap change alerts

Related articles