Chapter 49 Robot Application Functions - Enumeration Functions

This chapter contains the following sections

• Introduction

• enumerate-urls

• enumerate-urls-from-text

Introduction

The functions discussed in this chapter operate at the Enumerate stage. These functions control if and how a robot gathers links from a given resource in order to use as starting points for further resource discovery.

enumerate-urls

The enumerate-urls function scans the resource and enumerates all URLs found in hypertext links. The results are used to spawn further resource discovery. You can specify a content-type to restrict the kind of URLs enumerated.

Parameters

The parameters used with the enumerate-urls function and their description are:

max

The maximum number of URLs to spawn from a given resource. The default, if max is omitted, is 1024.

type

Content-type that restricts enumeration to those URLs that have the specified content-type. type is an optional parameter. If omitted, it will enumerate all URLs.

Example

The following example enumerates HTML URLs only, up to a maximum of 1024:

Enumerate fn=enumerate-urls type=text/html

enumerate-urls-from-text

The enumerate-urls-from-text function scans text resources, looking for strings matching this regular expression: URL:.*. It spawns robots to enumerate the URLs from these strings and generate further resource descriptions.

Parameters

The parameter used with the enumerate-urls-from-text function and its description is:

max

The maximum number of URLs to spawn from a given resource. The default, if max is omitted, is 1024.

Example

Enumerate fn=enumerate-urls-from-text