5.3 Ingest Documents

MODEL INGEST reads a file or recursively scans a directory, creates embeddings using the configured embedding model, and stores the document chunks in the specified Oracle vector table.

Syntax

MODEL INGEST -path <file-or-directory>
-table <vector-table> [-debug] [-verbose]

5.3.1 Prerequisites

Before running model ingest, ensure the following:

  • Configure an embedding model by using MODEL CONFIG EMBEDDING-MODEL.
  • Connect SQLcl to the target Oracle Database.
  • Ensure that the specified path is readable.
  • Ensure that the connected database user can create or use the target vector table and vector index.

Note:

The chat model is not required for ingestion.

5.3.2 Supported File Types for Model Ingest

model ingest supports the following file types:

File Type Parser Description
PDF Apache PDFBox Extracts text from PDF documents.
DOC, DOCX Apache POI Extracts content from Microsoft Word documents.
XLS, XLSX Apache POI Extracts content from Microsoft Excel workbooks.
PPT, PPTX Apache POI Extracts content from Microsoft PowerPoint presentations.
TXT, MD Text parser Processes the file as plain text.

When -path specifies a directory, SQLcl scans the directory and its subdirectories for supported files. Unsupported files are ignored.

5.3.3 Ingest Workflow

For each model ingest operation, SQLcl:

  1. Verifies that the specified file or directory exists.
  2. Loads the saved embedding model configuration.
  3. Creates the target vector table and HNSW (Hierarchical Navigable Small World) vector index if they do not already exist.
  4. Verifies that the target table is compatible with the configured embedding model.
  5. Processes the specified file or, if a directory is specified, the supported files in that directory and its subdirectories.
  6. Computes a hash for each document and skips documents that have already been ingested into the target table.
  7. Extracts the document content and metadata.
  8. Divides the content into smaller chunks.
  9. Generates a vector embedding for each chunk.
  10. Stores the chunks, vector embeddings, and document metadata in the target vector table.
  11. Commits the operation.

SQLcl uses the document content hash to detect documents that have already been ingested. Running the same command again skips documents whose content is already present in the target table.

If an operation fails, SQLcl rolls back the current transaction and restores the connection's original auto-commit setting.

5.3.4 Duplicate Detection and Model Compatibility Checks

SQLcl uses the document’s content hash to detect documents that have already been ingested. Duplicate detection uses the selected target table and the document content hash, it does not rely on the file name alone. If the contents of a file change, the document hash typically changes, allowing the modified file to be ingested as a new document.

The embedding-model configuration used for ingestion must match the model fingerprint recorded for the target vector table. If the embedding model or another configuration value that affects its fingerprint changes, use a separate vector table.

SQLcl creates and maintains the compatibility metadata. Users do not normally set these values manually.

5.3.5 Examples

Configure the embedding model before the first ingest:

SQL> MODEL CONFIG EMBEDDING-MODEL -model-name "cohere.embed-english-v3.0" 
-serving-type ON-DEMAND -compartment-id "ocid1.compartment.oc1..example"

SQL> model ingest -path "./knowledge-base" -table HR_DOCS -verbose

Ingest one document: 

SQL> MODEL INGEST -path "./knowledge-base/guide.pdf" -table HR_DOCS

Recursively ingest supported files: 

SQL> MODEL INGEST -path "./knowledge-base" -table HR_DOCS -verbose

Running the command again checks the stored document hashes and skips documents that have already been ingested into HR_DOCS.