Using the Siebel Smart Answer Administration Tool

This chapter provides instructions for using the Siebel Smart Answer Administration Tool. It includes the following topics:

About Siebel Smart Answer Administration Tool

The Siebel Smart Answer Administration Tool provides all the required functionality for creating and managing a corpus, creating and maintaining categories, building, tuning and exporting a knowledge base, and monitoring the run-time effectiveness of the exported knowledge base. The purpose of this section is to describe each of the major features of the Siebel Smart Answer Administration Tool and provide examples of how to use each in order to maximize the accuracy of the knowledge base for use in Siebel Contact Center, Siebel Email Response, and Siebel Self Service Center.

Logging in to the Siebel Smart Answer Administration Tool

The Siebel Smart Answer Administration Tool requires a user name and password, so that the system will not let the same user log in more than once at the same time and so that the corpus and knowledge base are protected.

To log in to the Siebel Smart Answer Administration Tool

Obtain a username and password from your system administrator.
Add users.

Note: If you are the system administrator, see Using the Siebel Smart Answer Administration Tool for instructions on how to add users.
Select the desired language.
Enter your username and password, and click OK.

Note: The path to the configuration file and the knowledge base endpoint must remain untouched, unless advised by your administrator to use a different configuration or knowledge base endpoint. In particular if the Smart Answer Server is installed to a non-default port, then make sure to enter the knowledge base endpoint as KB name:port number.
Check for the status message: Connected to Server at the end of the screen. This message will appear as soon as the Siebel Smart Answer Administration Tool is connected to the Siebel Smart Answer server and is ready to use. The first screen you will be presented with is the System Overview Tab, which contains information about the server in use as well as the currently logged in account.

Building a New Knowledge Base Model

Use the following procedure to build a new Siebel Smart Answer knowledge base model.

To build a new knowledge base model

Log into the Siebel Smart Answer Administration Tool by specifying a username, password, and the knowledge base you want to connect to.
Back up your existing corpus.
Prepare and import the corpus.
Validate the corpus.
Assign categories to your corpus entries.
Benchmark the knowledge base model.
Analyze and tune the knowledge base model.
Build the knowledge base model.
Export the categories to Siebel.
Monitor the activities of the run-time Call Center environment.

Backing Up and Restoring the Corpus

Before making any modifications to your corpus, you must back up the data associated with your corpus.

To back up and restore your corpus

Navigate to the Corpus Overview Tab and then click the Create Backup button.
Save the .sab file to a place where you can easily access it again in case you want to restore that version of the corpus.
To restore the corpus, navigate to the Corpus Overview Tab and select Restore Backup.
Browse to the location where you save the .sab file and select the file.

Note: This will completely restore your corpus and categories to the state of the backup file and will delete the current state stored in the Siebel Smart Answer Administration Tool.

If you want the knowledge base model in the Siebel Smart Answer production environment to match the restored version in the Siebel Smart Answer Administration Tool, you must build a knowledge base model. For more information on how to build a knowledge base model, see Building a New Knowledge Base Model.

Building a Corpus

Building a corpus is one of the most critical parts of the knowledge base model building process since the quality of the corpus determines the quality of the performance of Smart Answer.

Building, Preparing, and Importing a Corpus for the Siebel Smart Answer Knowledge Base Model

Use the following procedure to build, prepare, and import a corpus for the Siebel Smart Answer knowledge base model.

To build, prepare, and import a corpus for the Siebel Smart Answer knowledge base model

Carefully select emails and any other documents that represent your business domain to ensure that Siebel Smart Answer has enough content to learn the concept models for the categories you will later use as part of the Siebel Call Center. The documents can be of mixed language and mixed format.
The supported formats include:
- Raw Text
- HTML
- Microsoft Word
- PDF
- PostScript
- XML
- E-mail
- CSV
- ENV
The supported languages include:
- English
- French
- Italian
- German
- Spanish
Note: The corpus can be heterogeneous and it is not required for the corpus to contain documents in the same format or in the same language.

Navigate to the Corpus Tab and then click the Import button. A popup box will appear that requires the name of the corpus, the location of the corpus, and the file type of the corpus. Create a new Corpus by completing the necessary fields.

Some fields are described in the following table:

Corpus Import Fields	Input Type
Input knowledge base	Directory or File
Type	Drop Down List

During the import process, a status screen will appear that indicates the number of entries processed. When the import is complete if there were any errors, both the file and line number of the file will be listed. Carefully read the error screen and correct each error by either removing the files or editing them so that they are compatible with the expected format.
After the errors are corrected, it is not necessary to re-import the entire corpus. Instead navigate to the Corpus Browse tab and select Add Entry for each entry that had errors. This completes the process of preparing and importing a corpus.

After the corpus is imported the corpus entries are summarized on the Corpus Browse tab. Each corpus entry is parsed so that multiple fields are associated with it. These include, a unique identifier, the language of the corpus entry, an optional Categories, To, From, and Subject, and then the Message field. In the case the entry is an email, the To, From, Subject, and Message will all be populated. For non-email corpus entries all content will be associated with the Message field.

Modifying Corpuses

Use the following procedure to modify your corpus.

To modify your corpus

Navigate to the Corpus Browse Tab.
The entries can be filtered either by language or by whether the corpus entry has a category assigned to it.
To edit an entry, highlight the row of the entry and select Edit. In this screen the category, to, from, subject, message, and language information for a category can be modified and saved.
To delete an entry, highlight the row of the entry and select Delete.
To add an entry, click on Add and browse to new the corpus entry on your file system.

Assigning and Managing Categories for Corpus Entries

After all the corpus entries have been added, updated, and verified, the next step is to assign categories to each corpus entry. This section will cover the following topics on creating categories for corpus entries:

Multilingual corpus categories
Bulk import of category assignments
Using suggested categories
Assigning categories and parent categories manually

Extraction of Categories from a Corpus

If the corpus entries are organized in a directory or folder structure, during the import process the Siebel Smart Answer Administration Tool will automatically assign categories to corpus entries. Each corpus entry will be assigned the name of its parent folder or directory.

Hierarchical Knowledge Bases for a Multilingual Corpus

If a corpus is in more than one language, the Siebel Smart Answer Administration Tool will detect the language of each entry and organizes the content into groups by language. When you browse the imported multilingual category set, the top-level nodes of the category hierarchy are the languages in your corpus, the categories themselves have the language name appended with a hyphen, and the entries are leaf-level nodes in the hierarchy.

Caution: If you are manually creating your corpus and it consists of multiple languages you must organize your entries by language otherwise the categorization might not work.

Bulk Importing of Category Assignments

Bulk importing of corpus entries and category assignments is most appropriate when a call center has a large set of existing emails and they have already been labeled with one or more categories. For example, as soon as a customer request has been resolved, a call center site might store all service requests with its resolution category in a database table. During the Siebel Smart Answer corpus preparation, the database can be queried for the customer email content and the resolution category. The results of the query can be quickly formatted into a bulk import CSV file.

Expected Format for a Bulk Import CSV file

The supported fields for a Siebel Smart Answer CSV import for email is a subset of those exported by Outlook Express and include: Subject, Body, From: (Name), From: (Address), To: (Name), To: (Address).

The only mandatory field for import is the Body field. An example of an import file with the fields Subject, Content, and Categories might look like the following:

Line 1: Subject, Content, Categories

Line 2: stains, I can not get the ink stains out of my cotton shirt using your 
product, apparel

Line 3: coffee, The coffee tasted bitter after sitting in the freezer for two weeks, 
food

Note: If you want to add a document as an entry to the CVS bulk import, you can fill or specify the Body and Categories field and dump the text of the document into the Body field.

Bulking Import Corpus Entries with Category Assignments

Use the following procedure to bulk import corpus entries with category assignments.

To bulk import corpus entries with category assignments

Navigate to the Corpus Status Tab, and then click Import.
Select the file type CSV.
For more information on generating a bulk CSV file, see Expected Format for a Bulk Import CSV file.
Browse to the CSV file, and click OK to start the import process.

Starting Your Category Sets with Suggested Categories

For call centers that are starting from scratch and have no predefined categories for their corpus and entries, the Siebel Smart Answer Administration Tool has the ability to suggest categories for their corpus entries.

To generate suggested categories for your corpus entries

Navigate to the Category tab and select Start Clustering, where you will be prompted to fill in the number of categories that you want your corpus to have.

Note: The suggested number of categories for a corpus is generally the square root of the number of corpus entries. Additionally it is recommended that each category have roughly the same number of corpus entries for best performance of the knowledge base. (For a very large corpus (more than 1000 entries) this can take up to an hour.
Fill in the number of desired categories, and click OK.

Note: For a very large corpus of more than 1000 entries, this process can take up to an hour. You will know the clustering process is complete when the status information has Clustering Result Available: Yes.
To generate suggested keywords to use as topics for each cluster, select Create Clusters from Categories.

Note: A pop up box will appear to let you know when this process is complete and will state Successfully created categories and assigned to entries. This can take up to a minute. Select OK to dismiss the box.
Navigate to the Category View/Edit Tab.
Select List view to see the category to corpus assignments. The clusters will initially be named Cluster [1-n].
Highlight the first category set, Cluster1, and select Edit.

Note: The Description Field will list the common keywords for the category and can be used as a guideline for naming the category.
Pick the keyword that makes the most sense for your business domain and type it into the Category Name field.
Select Save to commit this keyword as the new category name.
Repeat Step 2 through Step 8 until all the clusters have been renamed with human understandable category names.

Creating Hierarchical Categories

If you have a large number of categories that can be grouped together, it makes sense to organize them into a hierarchy.

To create hierarchical categories

Navigate to the Tree View of the Category Tab.
Highlight the category that you would like to assign a parent node.
Select a parent from the drop down list of Parent Categories, and then Save to commit the change.
Repeat Step 2 and Step 3 for all categories for which you would like to assign parents.

Automatically Creating Categories Using Feedback

As soon as a category is added to the catalog attached to the Siebel Smart Answer knowledge base and agents manually assigning the category to service requests, a feedback request will be sent to Siebel Smart Answer. The feedback request will alert Siebel Smart Answer that the given service request content and the assigned category are new and valid corpus entries. You do not have to manually enter the category in the Siebel Smart Answer Administration Tool. The next time the knowledge base is retrained, the category will be part of the Siebel Smart Answer run-time category set. Over time as more service requests are attached to the new category, the category confidence will increase.

Manually Assigning Leaf and Parent Categories

Manually assigning categories to corpus entries is the most time consuming and error prone method for building a Siebel Smart Answer corpus and is not recommended for a corpus with more than a few hundred entries. If you have more than a few hundred and must assign categories to each of them, see Starting Your Category Sets with Suggested Categories.

To manually form a hierarchical category set, you must decide which categories will be the parent nodes and those that will be the leaf or child nodes.

Note: You must first create a new category.

To create a new parent or leaf node

Navigate to the Category - Tree View Tab.

Create a new category by completing the necessary fields.

Some fields are described in the following table:

Category Name	Text
Parent Category	Select from Dropdown list
Category Description	Text

After all the categories are added into the Siebel Smart Answer Administration Tool, you can proceed to manually assign them to your corpus entries.

Assigning Categories Manually to Corpus Entries

Use the following procedure to assign categories manually to corpus entries.

To manually assign categories to corpus entries

Navigate to the Corpus Browse Tab.
Highlight the corpus entry and select Edit.
In the categories box select the plus sign to add a category.
If you make a mistake you can highlight the category and delete it by clicking the x.
Repeat Step 2 through Step 4 for all categories you want to add to the entry.
Select Close to save the assignment and move on to the next corpus entry.

Creating a Knowledge Base Model Using the Benchmark and Analyze Work Flow

The process for creating a knowledge base model for use in the Siebel Smart Answer run-time system is an iterative process of benchmarking the model, analyzing the model, and tuning it. If during the analysis process you make any changes to the categories or corpus entries, the benchmark, and analyze process must be repeated to reflect your changes. You must carefully review the analysis and benchmarking results to make sure that your changes are having a positive impact on the Siebel Smart Answer knowledge base model.

Benchmarking the Knowledge Base Model

The Knowledge Base benchmark is the most informative set of statistics to predict how well the Siebel Smart Answer system will perform in the Siebel Smart Answer run time. The Siebel Smart Answer Administration Tool will compute the expected accuracy of each category by randomly sampling the corpus and reserving 20% for testing and using the rest for training. It will repeat this process five different times and then average the scores.

Running a benchmark

Navigate to the Knowledge Base Benchmarking Tab and select Run Benchmark.

Note: The benchmarking process can take several hours for a large corpus.
Review the final scores shown in the Benchmarking report.
Review for each category the accuracy, the total incorrectly classified entries (Incorrect), the total of correctly classified entries (Correct), and the total number of entries that were not classified as the expected category (Missing).

Modifying the corpus using Benchmark Results

The knowledge base benchmark is the best way to measure the expected performance of your knowledge base model in the call center run time environment. When first building a knowledge base it is expected that you must adjust your knowledge base inputs - the categories, corpus entries, and category assignments, at least once before the knowledge base model will be ready for export in the Siebel Smart Answer environment. The Knowledge Base Benchmarking Tab lists each of the categories with their accuracy scores as well as their corresponding entries.

To adjust and analyze your knowledge base model

Navigate to the Knowledge Base Benchmarking Tab.
Select a category row and select View… Miscategorized entries to display all the entries that were incorrectly assigned to the selected category.
Inspect each entry to verify that:
1. It really belongs to the category currently assigned to it and not to the one it was categorized as during the benchmarking process. If not, re-label the entry with correct category by clicking Edit entry at the end of the screen and changing the category name using the plus and minus signs.
2. The content of the entry is informative and meaningful for the currently assigned category. If not then modify the contents of the entry by clicking Edit entry.
3. For instructions on how to delete an entry because it is noisy or contains irrelevant information, see Modifying Corpuses.

Repeating the Benchmarking Process

In order to validate that the changes made to the corpus during the Analyze and Tune process were productive, a new benchmark must be run to measure your modifications to the corpus and/or category set.

To repeat the benchmark process

Navigate to the Knowledge Base Benchmark Tab and select Run Benchmark.
For more information on the benchmarking process, see Benchmarking the Knowledge Base Model.
Examine the results as described previously and repeat the Knowledge Base Benchmark, and Analyze workflow until the scores for each category are at least 50% or higher.

Note: Keep in mind that the feedback mechanism will quickly tune the knowledge base model to learn the operating environment and that the initial knowledge base model is only intended as a starting point from which Siebel Smart Answer will learn and refine the categorization models inside the knowledge base model. So if you can not reach the 50% accuracy mark, you can rely on the feedback mechanisms to do the correction in the production environment.
Whenever any changes are made to the categories or the corpus inside the Siebel Smart Answer Administration Tool, always build the knowledge base model in order for the changes to take affect in the Siebel Smart Answer run-time system.
For more information on how to build a Siebel Smart Answer knowledge base model, see Building a New Knowledge Base Model.

Building a Knowledge Base Model for a Siebel Smart Answer Production Environment

After you have completed the Benchmark, Analyze and Tune workflow, you must build a knowledge base model in order push the knowledge base model into the Siebel Smart Answer production environment.

Caution: Building a Knowledge Base will automatically make the new knowledge base the one used in production. Do not build a knowledge base unless you want the most recent changes to your corpus in the Siebel Smart Answer Administration Tool to be reflected in the Siebel Smart Answer production environment.

To build a knowledge base model for your Siebel Smart Answer production environment

Navigate to the Knowledge Base Status tab and select Build Knowledge Base to start the knowledge base model training.
Wait for the Training Status to be set to: Not Training.

Note: The training process can take several hours and scales with the size of your corpus. Do not build a new knowledge base model during high call center load.
As soon as the knowledge base model is built, it will automatically be the default model for your production environment.

Tuning Thresholds for Your Knowledge Base Model

The knowledge base benchmark is the best way to predict the performance of your knowledge base model in the call center run time environment. When you finish making the corpus adjustments, as determined by the benchmark results, your next step is to adjust the category thresholds. Each of the categories has a target accuracy to simplify this process.

To tune thresholds for your knowledge base model

Navigate to the Knowledge Base Analysis Tab.
Set the target accuracy of each category with either the arrow keys or manually percentage values.
With the modified category selected, click the 'Calculate Threshold' button.
1. This will set the Confidence threshold to a level so that, based on the benchmark results, the accuracy of entries above the threshold will meet or exceed the target accuracy.
2. If the Confidence Threshold sets itself to 100%, this means that Target Accuracy can not be achieved.
  
  Note: You must select the Push Changes option to commit and save the changes to a category. When you select the Reset option, all of the categories will revert to their current state in the Smart Answer Server.
Click the Calculate Expected Accuracy button.

Note: This may take a few minutes to complete for a large corpus.
Inspect the results the Expected Accuracy Results.
1. Actual Accuracy of each category is the accuracy of benchmark results above the confidence threshold.
2. Exceeds Threshold of each category is the percentage of benchmark results that are above the confidence threshold.
Select Push Changes to save the changes of all categories to the Smart Answer Server.

Querying the Knowledge Base Model

After the final knowledge base model is built, a good practice is to validate the knowledge base model by testing it on a small sample of emails for which the desired category is already known. There are two methods for doing this:

Single query mode
Batch query mode

Validate Your Knowledge Base Model with Single Query Mode

Single query mode is useful for dynamic validation of your knowledge base model. In this mode you can type in text or copy in email text to see how the knowledge base model will categorize it.

To validate your knowledge base model using single query mode

Navigate to the Query - Single Query Tab.
Type in or copy and paste the text of an email into the box labeled: Enter message to be analyzed here, and select Run Query.
Review the category assignments and the confidence scores for your query in the results table at the end of the screen.
Select Clear to enter another query.
Repeat Step 1 through Step 4.

Validating Your Model with Batch Query Mode

If you have several test queries, they can be placed in a CSV file with at least two fields: Categories, and Body. The file can be thought of as a key or a test to see how well your knowledge base model does on new emails that were not part of the training set. For more information on the available fields, see Expected Format for a Bulk Import CSV file.

To validate your knowledge base model using batch query mode

Navigate to the Query - Batch Query Tab.
Select Load File and browse to your CSV file with the tests emails and their expected category assignments.
Review the results of the batch query at the end of your screen. The format is the same as for the Knowledge Base Analysis Tab.
For more information on the benchmarking process, see Benchmarking the Knowledge Base Model.

Exporting the Knowledge Base

After the final knowledge base model is built and has been tuned adequately, the category set needs to be exported for import into Siebel so that the catalog inside Siebel and the category set inside the Siebel Smart Answer Administration Tool are the same.

To export your knowledge base categories for import into Siebel Smart Answer

Navigate to the Category Overview tab and select Export Categories.
Save the knowledge base .xml file to the shared Siebel File system under the location configured while installing the Siebel Smart Answer connector. For more information on configuring the knowledge base file import location, see the Siebel Smart Answer Connector Guide on the Siebel Bookshelf.

For more information on how to import the knowledge base file, see Importing the Knowledge Base File.

Scheduling Maintenance for the Knowledge Base Model

In order to incorporate feedback into the Siebel Smart Answer knowledge base model, the NLP engine needs to rebuild the knowledge base model each night. Additionally, any changes made to the catalog or feedback requests sent for new categories from within the Siebel application, will not be immediately reflected when the Siebel Smart Answer Engine is run. The scheduled maintenance will:

Synchronize the Siebel Smart Answer Administration Tool data model with the Siebel Catalog data model.
Incorporate the Siebel Smart Answer feedback from the last 24 hours into the new knowledge base data model.
Re-adjust thresholds based on Target Accuracy for the new model.
Rebuild a new model that will become the production Siebel Smart Answer knowledge base model.
If there are more than 500,000 entries in the knowledge base corpus, the Smart Answer Engine will remove duplicate entries and retain the most recent entries for each category. This action ensures that the categorization time is less than a second. It also tunes the knowledge base model to the most recent activities in the call center.

To schedule a maintenance window for the Siebel Smart Answer knowledge base model

Navigate to the Knowledge Base Status tab.
Modify the Auto-rebuild time and select Set.

Synchronizing the Knowledge Base Categories and Thresholds

Each time a new knowledge base model is pushed to the Siebel Smart Answer Manager, the confidence thresholds must be updated for auto-response inside Siebel Email Response. Because the knowledge base model will change at least once a day due to scheduled maintenance, synchronization must be done at least once a week. Synchronization will also push any updates made in the Siebel application to the categories into the Siebel Smart Answer Administration Tool and the knowledge base.

Note: If changes are made to the categories inside the Smart Answer Administration Tool, then a knowledge base file must be exported and then imported into Siebel.

To synchronize thresholds and categories

Navigate to the Administration - Catalog screen inside Siebel.
Select the active knowledge base.
Select Smart Answer Synchronize.

About Dashboard and Reports

The Dashboard tab in the Siebel Smart Answer Administration Tool is designed to give the user an at a glance summary of the effectiveness of the call center.

Messages per Minute and Hourly messages are two views of how much request traffic is being seen by the CSRs and how quickly they are responding to service requests.
Top five categories of the day is an up to date pie chart of the top five most frequently used categories of the day.
CSR optimization has two metrics: CSR productivity and CSR assist.
CSR productivity is a measure of how often the Siebel Smart Answer engine auto-responds to customer requests and consequently increases the productivity of the CSR.
CSR assist is a measure of how often the CSR uses the auto-suggestions from the Siebel Smart Answer engine.
Service Affecting Messages is a graph of how many service requests were classified as customer complaints or with a highly negative sentiment. In the run time Siebel Smart Answer engine service affecting messages are classified with the category service_affecting. It is up to the call center manager to impose policies or create specialized work flows to take advantage of this information in case escalation is required. For a definition of service affecting messages, see Glossary.