Analyze Tuning Impact on Similar Questions

You can review how your tuning changes affect the health of your knowledge base by analyzing the impact on similar questions, or questions that share similar words or concepts to the tuned question. For example, for the tuned question Can I add a user to my family plan? you might see the following similar questions.

• Can I change my family booster?

• Can I add a line to my family plan?

• Can I add a family member to my current booster plan?

MSQ calculates whether the relevancy of the top answers returned to similar questions remains the same, improves, or diminishes after tuning the original question. It uses an algorithm comprised of several metrics to generate a score for each result before and after the tuning changes. Once these scores are calculated, the system compares these scores to each similar question before and after tuning.

1. Select Analyze Change (or select Next).

   At the Analyze Change screen you see a list of the similar questions in the Impact of Tuning Changes pane.

2. At the How Changes Impact drop-down menu, select one of the following options. (MSQ calculates certain characteristics (such as click-through rates, answer positions, etc.) that provides values for each option.)

   • Clicked Document. This is the number of times a user clicked through on an answer after asking a question. This is calculated by using metrics such as index positions and the number of answer click-throughs.

   • Linked To Case. This is the number of incidents (cases) linked to an answer in a time period.

     This is calculated by using metrics such as the answer relevancy score, index positions, and the number of incidents linked to the answer.

   • Popular Document. These are the articles users opened most often in a time period.

     The metric for this option is calculated by information such as the search relevancy score and the number of times the article was opened or clicked on.

3. See how the Search results are impacted by your tuning changes in the Impact of Tuning Changes pane. For each similar question in the list, you see:

   • Improved.

     This is a visual indication that represents whether the results set for this question improved. The icons are:

     • The dash indicates the tuning changes added no change to the clicked article.

     • The up arrow indicates the tuning changes may have added positive value and made the search better.

     • The down arrow indicates the tuning changes may have detracted from the value and made the search worse.

   • Estimated Improvement. This is a numeric value that represents the positive or negative change. This is calculated using different algorithms based on the option you selected in the How Changes Impact drop-down menu.

4. When you have reviewed the tuning impact on all similar questions:

   • If the tuning changes affected your knowledge base in a negative way (detracted from the value), select Cancel to return to the previous screens to modify or cancel your changes.

   • If the tuning changes affected your knowledge base in a positive way (improved the value), select Publish to save all changes.