57 Custom Document Loaders

For specialized use cases, users can define custom document loaders tailored to their specific needs.

To implement a custom document loader you should create a CDI Bean that implements com.oracle.coherence.rag.DocumentLoader interface, and provide the name of the URI protocol it is intended to handle through @Named class annotation.

The load method should fetch the document based on the contents of the specified uri, parse it using the injected DocumentParser, and return an instance of a LangChain4J dev.langchain4j.data.document.Document containing extracted text and metadata.

As an example, here is the actual implementation of the OciObjectStorageDocumentLoader that does all of the preceding:

@Named("oci.os")
@ApplicationScoped
public class OciObjectStorageDocumentLoader
        implements DocumentLoader
    {
    @Inject
    DocumentParser documentParser;                                              // <1>

    @Inject
    ObjectStorageClient client;                                                 // <2>
    
    public Document load(URI uri)
        {
        String ns     = uri.getHost();
        Path   path   = Path.of(uri.getPath());
        String bucket = path.getName(0).toString();
        String object = path.subpath(1, path.getNameCount()).toString();

        var request = GetObjectRequest.builder()                                // <3>
                .namespaceName(ns)
                .bucketName(bucket)
                .objectName(object)
                .build();

        var response = client.getObject(request);                               // <4>

        var source = new DocumentSource()                                       // <5>
            {
            public InputStream inputStream() throws IOException
                {
                return response.getInputStream();                               // <6>
                }

            public Metadata metadata()
                {
                var metadata = Metadata.metadata("url", uri.toString());        // <7>
                metadata.put("ns", ns);
                metadata.put("bucket", bucket);
                metadata.put("object", object);
                metadata.put("content_type", response.getContentType());
                metadata.put("content_md5", response.getContentMd5());
                metadata.put("content_length", response.getContentLength());
                return metadata;
                }
            };

        return dev.langchain4j.data.document.DocumentLoader.load(source, documentParser);  // <8>
        }
    }

1. DocumentParser to use is injected based on store configuration
2. ObjectStorageClient is created and injected based on configuration
3. Create a GetObjectRequest based on information from the specified uri
4. Run the request and get a response from the service
5. Create LangChain4J DocumentSource that is used to access document content and metadata
6. Return document content from the response
7. Create document metadata based on the response
8. Create an instance of a Document from the DocumentSource you created in step 5, using injected DocumentParser to extract text from the document content.

Security Prerequisites

A document loader can cross filesystem, network and parser trust boundaries. Restrict import locations and URI schemes, validate size and content expectations, and deploy custom loader classes and application metadata deliberately. Use the configured import policy for local files and remote documents before invoking ingestion.

For the complete configuration and policy, see Restricting RAG Document Imports in Securing Oracle Coherence.