Known Performance Issues

What impacts performance the most?

There are many variables that define what is going to be the performance of the back-end work queue, for example, how long is it going to take for the system to recalculate specific projects. It depends ultimately on four variables:

1. Number of charges to be recalculated across all projects

2. Number of charge recalculation triggers for a particular project

3. Speed of creation of an individual charge record

4. Number of computing threads

If one of the first three variables is extreme, it will dramatically slow down the system. You may have only one project in the system with only several charges, but if you set up your system in a way that charges get recalculated 1000x a day, this will slow down the system although at first glance the setup seems simple. You may have only 1 recalculation a day, but with hundreds of thousands of charges to be recalculated, you may get similarly poor performance results. You may have a low number of charges, and a low number of recalculation triggers, but an extremely heavy scripted charge record where it takes minutes to create a record where you get similarly poor results. In the same manner, if there is such a large project (either from # of charges, # of recalculations, or both) it can occupy all computing threads for a long time, and then even really small projects return high estimated finish times.

As you can see, it is basically impossible to define what is a safe setup, and what number of charges and recalculation triggers will ensure a smooth experience. It is a combination of all these variables.

Based on our experience we can say that generally, it is the forecast charges that are bottle-necking the system. From the nature of their existence, their purpose, and their goal - to provide the latest data about the project's future and reflect all relevant changes that took place on the project as soon as possible. In other words – forecast charges get deleted and recreated often and there may be a lot of them.

As you can see from the text above, the list of the changes/triggers is quite extensive, and each individual trigger can be hit often – especially when the project is edited. That means forecast charges may get recalculated often. If we combine it with a greater amount of charges coming from all projects (#1 above) the performance results may not be ideal.

How to mitigate possible performance bottlenecks?

Possible mitigation measures are related to the first three variables listed above and focus in particular on forecast charges. Check all three variables and strive to reduce all the numbers. The questions you often ask are:

1. Do we need forecast charges on these projects? Are these projects being actively used?

   Even unused or not pending projects are recalculated when a trigger is hit and with that, it takes some calculating capacity that could have been used for active projects.

2. Do we need forecast charges to be recalculated that often?

   This includes possible imports or scheduled scripts that can be run through all projects resulting in great amounts of charges to be recalculated for all affected projects

   Looking at the list of charge recalculation triggers above it is clear these triggers may get hit often – do we really need the data to be recalculated X times a day?

3. Do we have scripts/workflows linked to the charge record that could slow down its creation? Is there any way we could optimize it? Could we run a scheduled one-time-a-day script rather than having it as a user event so the project gets recalculated more often?

Forecast Charge Run On Demand Preference

The project preference ‘Forecast Charge Run On Demand’ was introduced in 2017. The preference is available to limit the recreation of forecast charges. The Forecast Charge Run on Demand preference is disabled by default. When enabled, forecasts will only be updated when manually generated or with the regular midnight charge run. Enabling this preference can help alleviate potential performance issues from frequent forecast updates.

Keep in mind that by enabling the preference you are adding more projects to the regular midnight charge run. As described above, the midnight charge run is looking at the timestamp of the last forecast recalculation which, if the preference is enabled, does not happen that often.

Midnight Run

Midnight run consumes projects that are in 'In Progress' state and possibly other project statuses as set up by the customer.

We've experienced scenarios where the midnight run took a long time colliding with the morning hours. What we've usually seen in such cases is that there were many projects set in 'In Progress' state while we saw these projects were not touched in months. Try to regularly update the status of projects and change any old projects from ‘In Progress’ to another status to alleviate the midnight charge run. Or manage project statuses and their midnight charge run as close to reality as possible.

General Notices