Implement Search Engine Optimization

19 Optimize Your App for Search Engines

Search Engine Optimization (SEO) covers a variety of techniques that make your application's pages more accessible to web crawlers, the scripts used by search engines to crawl the web and gather pages for indexing. SEO looks to increase the ranking of indexed pages in search results to give your content more visibility.

You can often improve the ranking of your application's pages using sitemaps. Sitemaps list the URLs of your pages, enabling web crawlers to identify your content without relying solely on their ability to crawl and parse the pages. But while sitemaps can help web crawlers find pages for indexing, the indexing process itself might be less than ideal for single-page applications. This is especially true when content is dynamically added to pages based on the results of REST calls. In such scenarios, web crawlers may not wait long enough for the REST calls to complete and the page to be fully rendered before indexing the page. Pages that are indexed before their content can be fully loaded don't feature well in search results.

To address this issue, VB Studio "prerenders" pages that web crawlers access. When a request for a page is received from Googlebot (or any other search engine), the request is routed through a prerender server, which loads the page, runs any JavaScript required to fully render that page, then strips it out before returning the page to Googlebot. This way, Googlebot always receives a fully rendered page for indexing—just one without any JavaScript in the source.

A page, once prerendered, is always cached, so page markup can be returned immediately when subsequent requests are made for the same URL. This helps reduce page load times that may adversely impact page ranking. If markup is yet to be cached, the process of rendering the page can take some time. To optimize these response times, it's recommended that you warm the prerender server's cache for URLs listed in your application's sitemap, so web crawlers always get the fastest possible response when requesting those URLs.

Here's a recommended list of actions to optimize your app for search engines:

Create a sitemap for your web application.
Add the sitemap to your web application's resources.
Once your web application is deployed (or redeployed), warm the cache for URLs listed in your app's sitemap (using the vb-prerender-cache-warm Grunt task).

Create a Sitemap for a Web App

A sitemap is a document that contains URLs to pages representative of your application. It ensures that important locations in your application are possible for web crawlers to locate and are properly ranked.

A simple sitemap may look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <loc>https://eta.myexample.org/?page=shell</loc>
      <lastmod>2023-08-21T16:12:20+03:00</lastmod>
   </url>
   <url>
      <loc>https://eta.myexample.org/?page=shell&shell=bugs&bugs=main-bug</loc>
      <lastmod>2023-08-21T16:12:20+03:00</lastmod>
   </url>
   <url>
      <loc>https://eta.myexample.org/?page=shell&shell=tags</loc>
      <lastmod>2023-08-21T16:12:20+03:00</lastmod>
   </url>
</urlset>

Each <url> tag is used to specify the URL of a page. This tag has several child tags, but only the <loc> and <lastmod> tags are used to populate the prerender server's cache:

<loc> is a required tag that specifies the actual URL of a page. The value of a <loc> tag must begin with the protocol (such as https) and end with a trailing slash, if your web server requires it. This value must be less than 2,048 characters long.
<lastmod> is a required tag that specifies the date the contents of the URL changed between application deployments.

Make sure the sitemap contains each individual URL you want indexed. It should also include up-to-date lastmod entries if the contents of the URL can change between deployments (for example, when pages contain data from other sources).

For general information on sitemaps, see https://www.sitemaps.org/protocol.html.

Add a Sitemap to a Web App's Resources

You can add a sitemap to your web application's resources to provide web crawlers information about the content in the application.

Before you add a sitemap file to your web app's resources, make sure it contains each URL you want indexed as well as up-to-date information for each URL. See Create a Sitemap for a Web App.

To add a sitemap to your web application:

Upload the sitemap file (for example, sitemap.xml or sitemap.txt) to your application in VB Studio.

You can upload the sitemap file to a web application's directory (for example, /webApps/mywebapp/sitemap.xml) or directly to the root directory of your visual application (at the same level as Readme.md or visual-application.json). Note that the root /webApps directory has higher priority and uploading a sitemap file here should work for all your web applications.

To upload a sitemap file at the root of your visual application:
1. Open your web application in the Navigator.
2. Click Source View.
3. Right-click the webApps directory and choose Import in the pop-up menu:
  
  Alternatively, drag a file from your local file system onto the webApps directory in the Navigator.
4. In the Import Resources dialog box, remove webApps in the Import location field, then click the drop target area and navigate to the file on your local system:
5. Click Import.
The sitemap is added to the root of your visual application:
Add the description/snippet for your application:
1. Click Web Applications in the Navigator and open your application.
2. Click HTML to open the application's index.html file.
3. Update the <meta name="description" content="description"> tag in the <head> section to add a description and a snippet that a web crawler will extract.
  
  Ensure that your description and snippet meet recommended guidelines, as described in Google documentation for snippets and meta-tags.
Deploy the application (see Deploy a Visual Application).

After your application is successfully deployed, you can access the sitemap, for example, at:

https://visualbuilder-dev.integration.test.ocp.oraclecloud.com/ic/builder/rt/myproject/live/webApps/mywebapp/version_324616315386523389/sitemap.xml

Note:
Each time you deploy your web app, the location of the resources changes to use a new version, and the sitemap file is moved to the newer /webApps/webAppName/version_id directory. To access the sitemap after each deployment, remember to use the new version_id in the resource path.
If you want to automatically copy the sitemap file to the visual application's root directory after each deployment, see Move Your Sitemap to a Visual Application's Root Directory.

Warm the Cache for URLs in a Sitemap

When your web app includes a sitemap that lists URLs to pages representative of your application, you can use the vb-prerender-cache-warm Grunt task to warm the prerender server's cache for these URLs, so web crawlers get the fastest possible response when requesting those URLs.

To use the vb-prerender-cache-warm Grunt task, you must set up your file system to build the visual application by installing Node.js and its package manager (npm). Once you have installed the necessary tools, you need to save the visual application's sources to your local system. See Build Your Application With Grunt.

To warm the cache for URLs in a sitemap using the vb-prerender-cache-warm Grunt task:

In the command-line interface, navigate to the folder on your local system containing the package.json and Gruntfile.js files.
Enter npm install to retrieve the node dependencies required to build the application.
Enter vb-prerender-cache-warm in the command-line interface to warm the cache. Here are some examples:
- To initially warm the cache after a redeployment:
```
grunt --id=MyApplication --ver=1.0 --sitemap="./webApps/myWebApp/sitemap.xml" --url=http://my.vbinstance.com/ic/builder --clearCache=true vb-prerender-cache-warm
```
- To update the cache after minor modifications to the sitemap:
```
grunt --id=MyApplication --ver=1.0 --sitemap="./webApps/myWebApp/sitemap.xml" --url=http://my.vbinstance.com/ic/builder vb-prerender-cache-warm 
```
The vb-prerender-cache-warm task reads the manifest.json file on startup. It will compare the timestamps of the pages in the manifest with the application's sitemap, and warm the cache with any files that have changed since the last run (or which failed on the last run) and write details of the run back to the same file on completion. If the clearCache flag is set, any existing manifest is ignored and overwritten when the task completes.
In the preceding examples, metadata about the last operation is saved to manifest.json in the current directory. To save or load from a different manifest file, you can specify the --manifest option:
```
grunt --id=MyApplication --ver=1.0 --sitemap="./webApps/myWebApp/sitemap.xml" --url=http://my.vbinstance.com/ic/builder --manifest="/tmp/manifest.json" vb-prerender-cache-warm
```
For more information on the supported command-line options, see vb-prerender-cache-warm.

Move Your Sitemap to a Visual Application's Root Directory

When you manually add a sitemap to your web application's resources, you can set up the sitemap file to be copied to the visual application's root directory after the app is deployed, so the sitemap path remains unchanged post-deployment.

Each time a web application is deployed, the location of the resources changes to use a new version, and the sitemap file is automatically moved to the new /webApps/webAppName/version_id directory. As a result, you'll need to include the newest version_id in the path to access the application's sitemap.

To automatically copy the sitemap file from the /webApps/webAppName/version_id directory to the application's root directory after each deployment, you can edit your application's Gruntfile.js file and add the vb-post-package task:

Open Gruntfile.js.

Edit the file to define the tasks performed for the vb-post-package task:

{code:java}
    /**
    Moves sitemap.xml resource back to Visual Application Root.
    */
    grunt.registerTask('vb-post-package', () => {
        const fileName = 'sitemap.txt'; // REPLACE WITH YOUR FILENAME
        const webAppName = 'mywebapp';  // REPLACE WITH YOUR WEB APP NAME
        const webAppDirectory = `build/optimized/webApps/${webAppName}`;
        // find out version dir
        const files = fs.readdirSync(webAppDirectory);
        const versionDir = files.find(file => /^version_\d+/.test(file));
        const source =
`${webAppDirectory}/${versionDir}/resources/${fileName}`;
        const target = `build/optimized/${fileName}`;
        // move the file back to assets root
        fs.renameSync(source, target);
        console.log(`${source} moved to ${target}`);
        });
{code}

Remember to update the task to use your sitemap resource and your web app.

See Customize Your Grunt Build Process for more information on adding custom functionality to existing Grunt tasks.