Creating a site map using web crawler
Web crawler is an Oracle Eloqua tool to create a site map. A site map updates automatically based on a schedule you can manage. However, you can use the web crawler to build a site map of pages that have not yet been visited. After the site map has been built, the pages can be tagged with page tags. The next time that Oracle Eloqua crawls your site, only pages that has been visited since the last crawl are added to the site map. If a page was not visited, it is removed from the site map.
Note: Web crawler always crawls at 01:00 Eastern Time (GMT - 05:00) regardless of the update frequency.
Before you begin:
- Add the domains you want to crawl to the domain allowlist.
To add a domain to web crawler:
- Navigate to Assets > Website Setup, then click Tracking.
- Click Web Crawler Settings.
The Web Crawler page appears.
- Select Web Crawler options, then click Add New Domain.
- Enter the fully qualified domain you want to crawl (
https://www.example.com). To include pages that contain query strings, select the check box.
Note: The domain URL needs to be in the domain allowlist, otherwise Oracle Eloqua returns an error.
- Click Save to continue setup.
- Enter the fully qualified URL of the starting page in the URL field, then click Add. To remove a page, double-click the page in the list.
The crawler starts the crawl from each page you add. If Oracle Eloqua crawls the same page multiple times, it only adds the page once to the site map.
- Click Save and Close.
After you finish:
Oracle Eloqua crawls the starting pages at 01:00 Eastern Time (GMT - 05:00) according to the schedule. You can run the web crawler immediately using the Update Site Map area. After the crawl finishes, you can view all the pages that were crawled using the View Page List option, or you can view the site map.
Important: For large domains, Oracle Eloqua takes more time to create the page list when you select View Page List.
Try the following solutions to help resolve common issues:
- If you cannot add a domain:
- Verify that the domain you are adding is in the domain allowlist.
- Verify that the domain you are adding does not contain a page name. For example, do not enter
https://www.example.com/index.htmin the domain field. You can use the fully qualified page URL as the starting page instead.
- Verify that you are using the fully qualified URL for the domain and that
httpsis specified correctly.
- If you cannot add a starting page:
- Verify that you are using the fully qualified URL for the domain and the starting page.
- Verify that the starting page is a sub-page of the domain you added.
- Verify that your starting page returns a 200 or 302 status code and has a MIME type of
text/html. The web crawler only crawls these pages.
- If a page doesn't appear in your site map, review the site map troubleshooting information.