Sitemap Generator
Customizations such as Catalog Segmentation, Gift Wrap, and Add-on Items can affect the Sitemap Generator output. Therefore, you should not use the Sitemap Generator for sites that use these customizations. Note that the legacy Sitemap Generator that crawls the links on your Commerce web stores has been renamed to Sitemap Crawler. The Sitemap Crawler can work for some Commerce web stores, but your customizations can affect its output. For enhanced performance and experience, you should use the new Sitemap Generator for your Commerce web stores.
Sitemap Generator creates an XML sitemap for your Commerce web store for Search Engine Optimization. By default, the Sitemap Generator includes your Homepage URL with URLs of all Commerce Categories, Subcategories, and Items published to your Commerce web store. In addition to this, by editing the Sitemap Generator settings, you can specify up to an additional 10,000 URLs to be included in the sitemap.
Site Builder Category URLs are not included by default. However, you can manually add the Site Builder Category URLs with other user-defined URLs.
Sitemap Generator creates your sitemap files and then automatically places them in the file cabinet. In your Domain Settings, the value specified for the HTML Hosting Root field determines the exact folder in which the sitemap files are placed. In most cases, it is the Live Hosting Files folder. For more information, see Set Up Domains for Web Stores.
In accordance with the Sitemap Protocol, the sitemap generation process generates one index file referencing one or more sitemap files for each set of 50,000 URLs.
The XML index file is named sitemap_DomainName_Index.xml and the XML sitemap files use the following naming convention: sitemap_DomainName_Timestamp_SequenceNumber. When there are multiple sitemap files in the Live Hosting Files folder, the timestamp in the file name helps you differentiate between sitemaps generated at different times.
Every time a sitemap is generated, the Sitemap Generator automatically notifies search engines such as Google and Bing that your sitemap has been updated. You can also schedule automatic sitemap generation so that your sitemap is always up-to-date. For more information, see Sitemap Generator Scheduler.
If you regenerate a sitemap, the new sitemap files overwrite the existing sitemap files in the same file cabinet folder. If the sitemap regeneration is unsuccessful, an error file is created in the same folder with the same name as the index file, but with a .txt extension. Consequently, the existing sitemap files remain intact and are not replaced when the sitemap regeneration is unsuccessful.
This topic includes the following subsections:
Prerequisites
An Item URL is added to the generated sitemap only if the following conditions are fulfilled:
-
The Display in Web Store option is enabled on the item record.
-
The Exclude from Sitemap option is not enabled on the item record.
-
The item is not marked as Inactive.
A Commerce Category URL is added to the generated sitemap only if the following conditions are fulfilled:
-
The Commerce Categories feature is enabled as described in Enable Commerce Categories Feature.
-
A Commerce Catalog is created, and the Commerce Category is assigned to the Catalog.
-
A URL is specified for the Commerce Category in the URL Fragment field.
-
Display in Web Site is enabled for the Commerce Category.
-
The Commerce Category is not marked as Inactive.
A Commerce Subcategory URL is added to the generated sitemap only if the following conditions are fulfilled:
-
The Commerce Categories feature is enabled as described in Enable Commerce Categories Feature.
-
A Commerce Catalog is created, and the parent Commerce Category is assigned to the Catalog.
-
The Commerce Subcategory is specified under the Subcategories tab of the parent Commerce Category.
-
A URL is specified for the Commerce Subcategory in the URL Fragment field.
-
Display in Web Site is enabled for the Commerce Subcategory.
-
The Commerce Subcategory is not marked as Inactive.
When an item goes out of stock, the item URL is not removed from the sitemap. If your selected Out Of Stock Behavior is to remove an item when out-of-stock, web crawlers might still be able to discover the product details page through the sitemap.
Generate a Sitemap
When you generate a sitemap, your Homepage URL, Item URLs, Commerce Category URLs, and Commerce Subcategory URLs are added by default. To add any additional URLs such as Site Builder Category URLs, Blog posts, and so on, follow the steps outlined in Add User-Defined URLs to the Sitemap. To add the change frequency for each URL in the sitemap, follow the steps outlined in Advanced Sitemap Configuration.
To add URL metadata such as last modified date, change frequency, and priority, follow the steps outlined in Advanced Sitemap Configuration.
For Matrix Items, URLs of only the matrix parent items are added to the sitemap file.
To generate a sitemap:
-
Go to Commerce > Marketing > SEO > Sitemap Generator.
The Sitemap Generator – Websites page appears.
-
From the list of domains, select one or more domains for which you would like to generate a sitemap.
-
Click Generate Sitemap.
You can view the progress in the Status column.
-
When you generate the sitemap first time, you must add the sitemap index file name to the
robots.txtfile as described in Update Sitemap Details in the robots.txt File.
If the sitemap is generated without any errors, the Status column displays the status as Generated, the timestamp appears in the Last Generated column, and the number of URLs generated appears in the URL Count column. If the sitemap is generated with some errors, the Status column displays the status as Generated with errors. If the sitemap generation fails, the Status column displays the status as Failed. You can then click Generated with errors or Failed to view the error report and troubleshoot the issue.
Add Site Management Tools Landing Pages to the Sitemap
You can add SMT landing page URLs to your sitemap in addition to the default Homepage URL, Item URLs, Commerce Category and Subcategy URLs. For more information about SMT Landing Pages, see Landing Pages.
To add SMT landing page URLs to the sitemap:
-
Go to Commerce > Marketing > SEO > Sitemap Generator.
-
Click Edit in the Settings column for the desired domain.
-
Check the Add SMT Landing Pages box.
-
Click Save.
The next time you generate a sitemap for this domain, URLs of the SMT landing pages in the published state are included in the sitemap. For more information, see Generate a Sitemap.
If you have configured a landing page to be visible as of a certain date and time, the landing page URL is not added automatically to the sitemap after the landing page is published. The changes are effective in the sitemap when you manually regenerate the sitemap or during the next scheduled time interval (if you have scheduled automatic sitemap generation), subject to the landing page being visible on that date.
Also, if you have configured a landing page to expire as of a certain date and time, the landing page URL is not removed automatically from the sitemap after the landing page expires. The changes are effective in the sitemap when the sitemap is regenerated after expiry of the landing page.
Add User-Defined URLs to the Sitemap
By editing the Sitemap Generator settings, you can add up to 10,000 user-defined URLs to the sitemap in addition to the most important page URLs that are added by default.
To add user-defined page URLs to the sitemap:
-
Go to Commerce > Marketing > SEO > Sitemap Generator.
-
Click Edit in the Settings column for the desired domain.
-
Select the Include these Pages in the Sitemap Generation option.
-
Manually add all the desired page URL paths.
-
Click Save.
The next time you generate a sitemap for this domain, the user-defined page URLs you added are included in the sitemap. For more information, see Generate a Sitemap.
Sitemap Generator Scheduler
You can configure the Sitemap Generator to automatically update your XML sitemap either daily, weekly, or monthly. Scheduling ensures you always have an up-to-date sitemap available for search engines to index.
For example, if you frequently add new items or update the product details page, you can use the Sitemap Generator Scheduler to configure sitemap generation to occur weekly.
To schedule automatic sitemap generation:
-
Go to Commerce > Marketing > SEO > Sitemap Generator.
-
Click Edit in the Settings column for the desired domain.
-
Click the Scheduling subtab.
-
Check the Enable Scheduling of Sitemap Generation box.
-
To set the frequency of automatic sitemap generation, select one of the following:
-
Daily – Triggers sitemap generation every day.
-
Weekly – Triggers sitemap generation on the specified day of every week.
-
Monthly – Triggers sitemap generation on the specified day of every month.
-
-
Click Save.
Advanced Sitemap Configuration
You can use advanced sitemap configuration to add additional metadata about each URL in the generated XML sitemap. Search engines scan this information when they index your site. Adding metadata such as the last modified date, change frequency, and priority, enables search engine crawlers to index your site more effectively.
You can add the following optional metadata in your XML sitemap through advanced sitemap configuration:
-
Show Last Modified — Specifies the date when each page was last modified.
-
Change Frequency — Defines how often you change your website pages. Valid values are: Always, Hourly, Daily, Weekly, Monthly, Yearly, and Never.
-
Priority — Specifies the relative importance of this URL as compared to other URLs in your site. You can select a priority ranging from 0.0 to 1.0 on item and category records. The default priority all the URLs is 0.5 including the Homepage URL.
Note:Using the advanced sitemap configuration, you can only specify if you want to include the URL priority in the sitemap or not. The URL priority itself is defined on the item and category record pages.
To add or edit the advanced sitemap configuration:
-
Go to Commerce > Marketing > SEO > Sitemap Generator.
-
Click Edit in the Settings column for the desired domain.
-
Click the Advanced subtab.
-
(Optional) To add the last modified date for each URL in the sitemap, select Yes from the Show Last Modified dropdown.
-
(Optional) To add the relative priority for each URL in the sitemap, select Yes from the Priority dropdown.
-
(Optional) To include the change frequency for each URL, select a value from the Change Frequency dropdown.
-
Click Save.
The next time you generate a sitemap for this domain, the metadata you selected is included in the sitemap. For more information, see Generate a Sitemap.
View Generated Sitemap Files
The generated sitemap files are stored in the file cabinet.
To view the generated sitemap files:
-
Go to Commerce > Marketing > SEO > Sitemap Generator.
-
Click Sitemap Files in the View column for the desired site.
You are redirected to the folder in the file cabinet where the XML sitemap files are stored.
Update Sitemap Details in the robots.txt File
To help search engines locate your sitemap files, you must update your robots.txt file with the name and location of the sitemap index file.
This process is currently not automated because robots.txt files can be heavily customized.
To update the robots.txt file with the sitemap index file name:
-
Go to Commerce > Marketing > SEO > Sitemap Generator.
-
Click Sitemap Files in the View column for the desired site.
You are redirected to the Web Site Hosting Files folder in the file cabinet where the
robots.txtfile is stored. -
Note the sitemap index file name. For example,
sitemap_mywebstore.com_Index.xml. -
Edit the
robots.txtfile as described in Editing Files in the File Cabinet. -
Add the sitemap index file name in the following format:
sitemap: http://domain/indexfilename.For example, if your sitemap index file name is
sitemap_mywebstore.com_Index.xml, add the following entry in therobots.txtfile:sitemap: http://mywebstore.com/sitemap_mywebstore.com_Index.xml -
Save the
robots.txtfile.
Because sitemaps are domain-specific, if you have multiple domains sharing the file cabinet root directory, you need to have two robots.txt files pointing to two different sitemaps. You can achieve this by setting up URL redirects as shown in the following example:
-
www.mywebstore1.com/robots.txtredirects to/robots-mywebstore1.txt -
www.mywebstore2.com/robots.txtredirects to/robots-mywebstore2.txt
For information about URL redirects, see SEO and Redirects.
Best Practices
When you use the Sitemap Generator to generate a sitemap, it is important you consider the following best practices:
-
If you use Site Builder Categories, edit the Sitemap Generator settings and manually add the Site Builder Category URLs.
-
Use standard item names and avoid special symbols in the Web Store Display Name field or in the URL Component field. The Sitemap Generator ignores incorrectly formatted URLs. However, the Sitemap Generator generates the sitemap without any errors.
-
Use a short, descriptive name for the item in the URL Component field of the item record. Setting a name to show in the URL can result in better ranking from search engines. If you leave it blank, internal item ID is used as identifier in the URL.
-
Do not update the generated sitemap files manually. If you missed adding some URLs, edit the Sitemap Generator settings to add the missing URLs, and then regenerate the sitemap.
If you use a third-party sitemap generator, you should verify that all URLs are valid in the generated sitemap. If your sitemap contains invalid URLs such as those containing undefined strings (url:undefined), the invalid URLs may be returned in the Google search results.