Home / Search Engine Considerations

Search Engine and Crawler Considerations

Prerender and Compiled Sites

Prerender is a technology that allows Oracle Content Management to deliver fully-formed HTML pages from what would otherwise be client-side JavaScript rendered pages. This is beneficial for bots and crawlers that don’t execute JavaScript as part of their page evaluation processes. Prerendering the pages on the server allows these bots to process HTML instead of a sparse controller.html page.

The use of prerender technology can be enabled on the SEO for Sites administration page:

Prerender Administration.

When a site is configured for compilation, pages will be translated into HTML as part of the compilation procedure. In this case, the server will deliver the compiled pages. The Oracle Content Management server will serve the compiled HTML pages to crawlers and bots and not any corresponding prerendered pages.

Page Delivery Priority

The following chart details the delivery flow with compiled pages and the Prerender service:

Page Delivery Flow.

As noted above, compiled (static) page delivery takes priority over prerendered responses. Similarly, compiled (static) page delivery takes priority over dynamic page delivery, where pages are rendered client-side in the browser.

Specify Additional Crawler/Bot User-Agents

When crawlers and bots make requests to the Oracle Content Management server, they identify themselves by sending a “User-Agent” request header. By default, the Oracle Content Management server recognizes most major crawlers and bots using a built-in list of User-Agent identifiers. Customers can add to the list of crawler User-Agent strings using the Configure User-Agents controls in the SEO for Sites administration page. This can be helpful if custom crawlers will be used against the site.

Prerender User-Agent Administration.

When configured, the listed User-Agent strings will be added to the built-in default identifiers. (The User-Agent configured strings are substring-matched case-insensitively against the User-Agent value in the HTTP request.)

Note: If User-Agents are added, the CDN configuration associated with the site may also need to be reconfigured to avoid serving prerendered pages to browsers and sparse controller.html pages to crawlers. Also note that the default <service>-<tenant>.ocecdn.oraclecloud.com CDN configuration can’t be changed to accommodate custom User-Agent identifiers.

Search Engine Considerations

Certain crawlers, such as Google, have abilities to execute JavaScript to some degree, while others don’t. The Oracle Content Management delivery mechanism should be compatible with the expected bots that will fetch pages from the site. In particular, for a dynamically delivered site using client-side JavaScript rendering, Prerender should be used in order for most bots to understand the pages rendered by Oracle Content Management. For a compiled site, the crawler will receive the HTML generated during the compilation process.

Add <meta> Tags to Compiled Pages

Crawlers will often use targeted hints within the page HTML markup. One example of this is the use of Open Graph <meta> within web pages for the benefit of Facebook. These tags can be added to the compiled pages to help crawlers understand the content of the page. The example below demonstrates how an Open Graph tag can be added to a compiled web page using custom component compilers and a page (layout) compiler.

Example: Add an Open Graph Tag to a Compiled Page

This example demonstrates one mechanism of adding an Open Graph tag to a compiled web page using a custom content layout compiler and a page compiler. The content layout compiler extracts and formulates the relevant Open Graph tag from the content item related to the page, and the page compiler effectively postprocesses the compiled page to move all of the emitted <meta> tags to the <head> section of the HTML markup.

Manufacture the Open Graph Tag

The first step in this process involves extracting or manufacturing the data for the Open Graph tag. Although this could be done entirely in the page compiler, the data in this example is manufactured in a custom content layout component compiler based on the content item’s data. In the sample below, note that lines 11-13 add a sample Open Graph tag into the body of the page, embedded in a special comment which will be used later.

Sample content layout compile.mjs additions:

...

// Note: Lines before this construct the markup for the content item into a string named "componentHTML" ...
  
// It is up to the developer to ensure the output from the compile.mjs:compile() function remains in sync with the render.mjs:render() function.
// To see what the default output would be, switch the following to true
if (true) {
    // Based upon the Content Item data, construct an Open Graph tag and write it directly to the page.
    // Later the Page compiler will move this to the <head> of the HTML markup.
    if (this.content.fields && this.content.fields.article_headline) {
      componentHTML += '<!-- Head Markup:';
      componentHTML += `<meta name="og:title" content="${this.content.fields.article_headline}">`;
      componentHTML += '-->';
    }
  
    // return the generated HTML and use hydrate in the render.mjs file to add any event handlers at runtime
    // change hydrate to true to use hydrate in the render.mjs file to add any event handlers at runtime
    return resolve({
      hydrate: false,
      content: componentHTML
    });

  ...
Move the Tags to the <head> Section

Since Open Graph tags are normally placed in the <head> section of the document, the compiled page is postprocessed in a page compile script to move the <meta> tags in the body into the correct position. The following afterPageCompile function moves the tags from the <body> to the <head> section in a layout-compile.mjs script.

Sample layout-compile.mjs script:

export default class {
    constructor () {
    }
 
    // The "compile" method pre-processes markup in the the theme's layout associated with the page. In this example
    // no processing is done, so the original layout markup is returned.
    compile (args) {
        const SCSCompileAPI = args.SCSCompileAPI;
        const compiledPage = args.layoutMarkup;
  
        return new Promise((resolve, reject) => {
            resolve(compiledPage);
        });
    }
  
    // The "afterPageCompile" method is used to postprocess the page after its other compilation activities have completed.
    afterPageCompile (args) {
        const SCSCompileAPI = args.SCSCompileAPI;
        let compiledPage = args.layoutMarkup;
  
        return new Promise((resolve, reject) => {
            // Collect out of the page markup all of the blocks that should be moved to the <head>.
            let headMarkupBlocks = [];
            compiledPage = compiledPage.replaceAll(/<!--\s*?Head Markup:(.*?)-->/gs, function(match, markup) {
                headMarkupBlocks.push(markup);
                return '';
            });
  
            // Write the blocks to the <head>.
            compiledPage = compiledPage.replace('</head>', headMarkupBlocks.join('\n') + '</head>');
            resolve(compiledPage);
        });
    }
}

This example can be extended to emit other Open Graph tags and other <meta> tags applicable to other crawlers.