Running a crawl

This section provides information about running a crawl using the CAS Server.

Related links

Running a crawl
Order of execution in a crawl configuration
Full and incremental crawling modes
Crawls and archive files
About writing records to a Record Store instance
About the record output file

Copyright © Legal Notices

Content Acquisition System Developer's Guide

Documentation Home

Contents
Search

loading table of contents...

Preface
- About this guide
- Who should use this guide
- Conventions used in this guide
- Contacting Oracle Support
Introduction
- About the CAS Service
- About the CAS Server
- About the Component Instance Manager
- About the Record Store
- About the Dimension Value Id Manager
- Overview of the default CAS data sources and manipulators
- Security requirements
Creating a crawl
- About creating a crawl
- About filters
- About CAS output types and the Deployment Template
  - Characteristics of a MDEX-compatible output
  - Characteristics of a Record Store instance versus record output files
- Creating a crawl using the CAS Server Command-line Utility
- Creating a crawl using CAS Console
- Creating a crawl using the CAS Server API
- Setting document conversion options
  - Default values for the options
  - Setting the options
- Configuring document conversion filters
- Modifying a crawl using the CAS Server Command-line Utility
Configuring a Record Store instance
- About record generations
- About transactions
- About the last read generation for a client
- About deleted records
- Configuring a Record Store instance
- Configuration properties for a Record Store instance
  - Example of a configuration file for a Record Store instance
- Change properties and new Record Store instances
- Deleting stale generations of records
- Disabling automatic management of a Record Store instance
- Performance considerations when using a Record Store instance
Running a crawl
- Running a crawl
- Order of execution in a crawl configuration
- Full and incremental crawling modes
- Crawls and archive files
- About writing records to a Record Store instance
- About the record output file
  - Record output file
  - Archived output files
Running the CAS sample applications
- About the sample CAS applications
Using the Delimited File data source
- Configuration properties for the Delimited File data source
Using the Record File data source
- Configuration properties for the Record File data source
Using the File System data source
- Configuration properties for the File System data source
Using the JDBC data source
- Installing a JDBC driver into CAS
- Configuration properties for the JDBC data source
- Feature notes and known limitations of the JDBC data source
Creating a Forge pipeline to read from or write to a Record Store
- Overview of a Forge pipeline
- Creating a Forge pipeline
Creating a CAS crawl to write MDEX-compatible output
- Overview of a CAS crawl that produces MDEX-compatible output
- Loading dimensions, properties, and precedence rules
- Loading dimension value records into a Record Store instance
- Loading data records into a Record Store instance
  - Property mapping for data records
  - Dimension values and data records
- Configuring features of a CAS-based application through XML files
- Creating a crawl to write MDEX-compatible output
  - Storing Workbench Credentials in OCS
CAS Server Command-line Utility
- Overview of the CAS Server Command-line Utility
- About CAS capabilities
- Saving passwords in a crawl configuration file
- Inspecting installed modules
- Managing crawls
- Managing dimension value Ids
- Viewing crawl status and results
Component Instance Manager Command-line Utility
- Overview of the CIM Command-line Utility
- Creating a Record Store
- Deleting a Record Store
- Listing components
- Listing types
Record Store Command-line Utility
- Overview of the Record Store Command-line Utility
- Writing tasks
  - Writing records
- Reading tasks
- Utility tasks
Forge-to-CAS Conversion Utility
- Overview of the Forge-to-CAS Conversion Utility
- Transforming Forge data
  - Example
  - Output
- Generating dval ID mappings
  - Example
  - Output
- Mapping information
- About transforming data
- Log messages
- Migration considerations
Running CAS components
- About running CAS components
- Running the CAS Service from the Windows Services console
- Starting the CAS Service from a command prompt
  - Command-line flags to CAS Service
    - Specifying JVM arguments
- Stopping the CAS Service from a command prompt
Backing up and restoring CAS
- Coordinating backups and restore operations
- Online backup and restore operations
- Offline backup and restore operations
  - Backing up CAS state
  - Restoring CAS state
Configuring logging
- Configuring logging for CAS components and command-line utilities
- Setting log properties for troubleshooting CMS data source issues
- Excluding failed records from the CAS Service log file
- Enabling log timing information for crawl processing steps
- Examining the CAS Service log
Tips and troubleshooting CAS
- Fixing crawl performance issues
  - Periodic crawl performance problems caused by defragmenting the crawl history database
- Modifying the CAS Server connection information for the CAS Console
- Modifying the CAS Service temporary directory
- Responding to a "Too many open files" error
- Setting the group entry size
Sample crawl configuration files
- Common properties for crawl configurations
- Sample configuration for a file system data source
- Sample configuration for a Record Store Merger data source
- Sample configuration for a manipulator
- Sample configuration for writing output to a Record Store instance
- Sample configuration for writing output to an MDEX compatible format
- Sample configuration for writing output to a file
File Formats Supported by the CAS Document Conversion Module
- Archive formats
- Database formats
- E-mail formats
- Multimedia formats
- Other formats
- Presentation formats
- Raster image formats
- Spreadsheet formats
- Text and markup formats
- Vector image formats
  - Notes on Adobe PDF text extraction
- Word processing formats
Record properties generated by crawling
- Common record properties
- Record properties generated by file system crawls
- Document Conversion properties
- Record properties generated by CMS crawls
- How CMS crawls handle multiple pieces of content

Search Terms