Managing crawls
The following
cas-cmd
tasks manage crawls.
Copyright ©
Legal Notices
Oracle Commerce Content Acquisition System Developer's Guide
Documentation Home
Highlighting
Prev
Up
Next
Skip to main content
Contents
Search
Preface
About this guide
Who should use this guide
Conventions used in this guide
Contacting Oracle Support
Introduction
About the CAS Service
About the CAS Server
About the Component Instance Manager
About the Record Store
Provides an efficient repository for records
Retrieves baseline and incremental records
Operates asynchronously
Operates as a lookup table
Creates a separate Record Store instance for each data source
Automatically cleans stale records
Easily configured and managed with a Record Store Command-line utility
About the Dimension Value Id Manager
Using the Dimension Value Id Manager and the Discover Electronics sample application
Using the Dimension Value Id Manager with other applications
Backing up and restoring dimension value Ids
Propagating dimension value Ids across environments
Lifecycle considerations
Overview of the default CAS data sources and manipulators
Security requirements
Creating a crawl
About creating a crawl
About filters
One filter per property per file or folder (unless a wildcard)
AND'ing and OR'ing
Include and exclude filters
Filter precedence
Missing properties on a record
Unfilterable properties
Case sensitivity
About CAS output types and the Deployment Template
Characteristics of a MDEX-compatible output
Characteristics of a Record Store instance versus record output files
Creating a crawl using the CAS Server Command-line Utility
Creating a crawl using CAS Console
Creating a crawl using the CAS Server API
Setting document conversion options
Default values for the options
Setting the options
Configuring document conversion filters
Modifying a crawl using the CAS Server Command-line Utility
Configuring a Record Store instance
About record generations
About transactions
About the last read generation for a client
About deleted records
Deleting all records for a full crawl
Deleting records for an incremental crawl
Reading records marked with the DELETE property value
Configuring a Record Store instance
Configuration properties for a Record Store instance
Example of a configuration file for a Record Store instance
Change properties and new Record Store instances
Deleting stale generations of records
Disabling automatic management of a Record Store instance
Performance considerations when using a Record Store instance
Running a crawl
Running a crawl
Crawling from the CAS Console for Oracle Commerce Workbench
Crawling from the CAS Server Command-line Utility
Crawling programmatically from the CAS Server API
Order of execution in a crawl configuration
Full and incremental crawling modes
Crawling in full mode
Crawling in incremental mode
Incremental mode and MDEX compatible output
Crawls and archive files
Archive file support
Support for ZIP files
Support for Tar files
How archive files are handled
About writing records to a Record Store instance
About the record output file
Record output file
Archived output files
Running the CAS sample applications
About the sample CAS applications
Order of running the Forge applications
Writing records from Forge into the Record Store
Reading records from the Record Store into Forge
Reading records from multiple Record Stores into Forge
Running the sample Web Crawler
Using the CAS Server Java Client
CAS Server Java Client Sample Files and Directories
About the CAS Server Java Client Program
Building and Running the Java Client with Ant
Opening the cas-server-java-client project in Eclipse
Running the operations of the Java Client
Using the Record Store Java Client
Record Store Client Sample Files and Directories
About the Record Store Sample Client Applications
The writer client
The reader client
Building and Running the Sample Writer Client with Ant
Building and Running the Sample Reader Client with Ant
Opening the recordstore-java-client project in Eclipse
Running the operations of the Sample Writer Client
Running the operations of the Sample Reader Client
Using the Delimited File data source
Configuration properties for the Delimited File data source
Using the Record File data source
Configuration properties for the Record File data source
Using the File System data source
Configuration properties for the File System data source
Using the JDBC data source
Installing a JDBC driver into CAS
Configuration properties for the JDBC data source
Feature notes and known limitations of the JDBC data source
BLOBs and document conversion
Record spec and Key Column values
BLOBs
Unsupported data types
Creating a Forge pipeline to read from or write to a Record Store
Overview of a Forge pipeline
Creating a Forge pipeline
Creating a record adapter to read from one or more Record Store instances
Creating a record adapter to read from crawl output files
Creating a record adapter to write to a Record Store instance
Creating a CAS crawl to write MDEX-compatible output
Overview of a CAS crawl that produces MDEX-compatible output
Loading dimensions, properties, and precedence rules
Loading dimension value records into a Record Store instance
Scenario 1 - Crawling extracts of the source data
Scenario 2 - Crawling the source data
Scenario 3 - Programmatically writing to a Record Store instance
Record properties for all dimension values
Record properties for range dimension values
Example of price range dimension values
About automatically generating dimension values
About dimension value matching
Range dimension values
Length limitations on name and spec values
Loading data records into a Record Store instance
Property mapping for data records
Dimension values and data records
Configuring features of a CAS-based application through XML files
Creating a crawl to write MDEX-compatible output
Storing Workbench Credentials in OCS
CAS Server Command-line Utility
Overview of the CAS Server Command-line Utility
Help options
Command-line options
Host and port settings
Setting the bin directory in the PATH environment variable
About error handling
About CAS capabilities
Saving passwords in a crawl configuration file
Inspecting installed modules
Getting the specifications of all modules
Getting the specification of a module
Listing modules
Managing crawls
Creating crawls
Deleting a crawl
Getting all crawls
Example of getting all crawls
Getting a crawl
Getting the incremental support status of a crawl
Listing crawls
Starting acquisition from a data source
Stopping acquisition from a data source
Updating crawls
Managing dimension value Ids
Creating a Dimension Value Id Manager
Deleting a Dimension Value Id Manager
Exporting dimension value Id mappings
Generating a dimension value Id
Getting a dimension value Id
Getting a dimension value specification
Importing dimension value Id mappings
Viewing crawl status and results
Getting metrics for all crawls
Getting the metrics for a crawl
Getting the status of a crawl
Component Instance Manager Command-line Utility
Overview of the CIM Command-line Utility
Help options
Command-line options
Host and port settings
Setting the bin directory in the PATH environment variable
About error handling
Creating a Record Store
Deleting a Record Store
Listing components
Listing types
Record Store Command-line Utility
Overview of the Record Store Command-line Utility
Transactions
Help options
Command-line options
Host and port settings
Setting the bin directory in the PATH environment variable
About error handling
Writing tasks
Writing records
Reading tasks
Reading baselines
Reading delta records
Reading specific records
Utility tasks
Cleaning a Record Store instance
Clearing the last read generation
Committing transactions
Getting the configuration of a Record Store instance
Getting the ID of the last-committed generation
Getting the last-read generation
Getting the ID of the write generation
Listing active transactions
Listing generations
Rolling back transactions
Setting the configuration of a Record Store instance
Setting the last-read generation
Starting transactions
Forge-to-CAS Conversion Utility
Overview of the Forge-to-CAS Conversion Utility
Transforming Forge data
Example
Output
Generating dval ID mappings
Example
Output
Mapping information
Dimension value ID mapping
Data mapping
Dimensions mapping
Index configuration mapping
About transforming data
Log messages
Records contain multiple Record Spec properties inside a record
Dimension values have the same display name
Duplicate records (records that share the same Record Spec)
Dimension names/values with special characters
Migration considerations
Running CAS components
About running CAS components
Running CAS components in the CAS Service
Running CAS components using the command-line utilities
Running CAS components from the CAS Console for Workbench
Running the CAS Service on Windows
Running the CAS Service on UNIX
Restarting the CAS Service
Running the CAS Service from the Windows Services console
Starting the CAS Service from a command prompt
Command-line flags to CAS Service
Specifying JVM arguments
Stopping the CAS Service from a command prompt
Backing up and restoring CAS
Coordinating backups and restore operations
Online backup and restore operations
Backing up crawl configurations
Backing up the last generation of Endeca records
Backing up dimension value ID mappings
Restoring crawl configurations
Restoring the last generation of Endeca records
Restoring dimension value ID mappings
Offline backup and restore operations
Backing up CAS state
Restoring CAS state
Configuring logging
Configuring logging for CAS components and command-line utilities
Log location and rolling
Configuration files for CAS components
Logging options and levels
Log file encoding
Setting log properties for troubleshooting CMS data source issues
Excluding failed records from the CAS Service log file
Enabling log timing information for crawl processing steps
Examining the CAS Service log
Location of the CAS Service log
Format of log entries
Enabling crawl statistics
Tips and troubleshooting CAS
Fixing crawl performance issues
Periodic crawl performance problems caused by defragmenting the crawl history database
Modifying the CAS Server connection information for the CAS Console
Modifying the CAS Service temporary directory
Responding to a "Too many open files" error
Setting the group entry size
Sample crawl configuration files
Common properties for crawl configurations
Sample configuration for a file system data source
Sample configuration for a Record Store Merger data source
Sample configuration for a manipulator
Sample configuration for writing output to a Record Store instance
Sample configuration for writing output to an MDEX compatible format
Sample configuration for writing output to a file
File Formats Supported by the CAS Document Conversion Module
Archive formats
Database formats
E-mail formats
Multimedia formats
Other formats
Presentation formats
Raster image formats
Spreadsheet formats
Text and markup formats
Vector image formats
Notes on Adobe PDF text extraction
Word processing formats
Record properties generated by crawling
Common record properties
Record properties generated by file system crawls
Common File System properties
Record properties for file system crawls on Windows
Record properties for file system crawls on UNIX
Limitations with ACL properties
Document Conversion properties
Record properties generated by CMS crawls
How CMS crawls handle multiple pieces of content
Search Terms
