Retry

Default Retry Strategy

See here for information on the settings used for the default retry strategy.

oci.retry.DEFAULT_RETRY_STRATEGY = <oci.retry.retry.ExponentialBackOffWithDecorrelatedJitterRetryStrategy object>

A retry strategy which has all options enabled and which will use the default settings for those options. These defaults are:

  • 8 total attempts - 1 original and 7 retries

  • Total allowed elapsed time for all requests of 600 seconds (10 minutes)

  • Exponential backoff with jitter using a base time of 1 second and an exponent of 2

  • The maximum wait time between calls is 30 seconds

  • Exponential backoff with de-correlated jitter of 1000 milliseconds is used

  • Retries on the following exception types:-

    • timeouts and connection errors
    • HTTP 409/IncorrectState, 429s (throttles)
    • Any HTTP 5xx except 501

Retry

Classes

RetryStrategyBuilder(**kwargs) A class which can build a retry strategy based on provided criteria.
NoneRetryStrategy A strategy that does not retry
ExponentialBackoffRetryStrategyBase(…) A base retry strategy from which other retry strategies inherit.
ExponentialBackoffWithFullJitterRetryStrategy(…) A retry strategy which does exponential backoff and full jitter.
ExponentialBackoffWithEqualJitterRetryStrategy(…) A retry strategy which does exponential backoff and equal jitter.
ExponentialBackoffWithFullJitterEqualForThrottlesRetryStrategy(…) A retry strategy that does exponential backoff and full jitter for most retries, but uses exponential backoff with equal jitter for throttles.
ExponentialBackOffWithDecorrelatedJitterRetryStrategy(…) A retry strategy which does exponential backoff with decorrelated jitter.

Data

DEFAULT_RETRY_STRATEGY A retry strategy which has all options enabled and which will use the default settings for those options.
GLOBAL_RETRY_STRATEGY A retry strategy which can be set by the user to modify the SDK retry behavior globally.
class oci.retry.RetryStrategyBuilder(**kwargs)

A class which can build a retry strategy based on provided criteria. Criteria can be provided at construction time or afterwards via using the add_* (to add/enable criteria) and no_* (to disable/remove criteria) methods.

Trying to build a strategy when there are no enabled checks will result in a oci.retry.NoneRetryStrategy being produced.

This builder is intended as a convenience, but callers are also able to bypass this and construct retry strategies directly.

Methods

__init__(**kwargs) Creates a new builder and initializes it based on any provided parameters.
add_max_attempts([max_attempts])
add_service_error_check(**kwargs)
add_total_elapsed_time([…])
get_retry_strategy()
no_max_attemps()
no_service_error_check()
no_total_elapsed_time()
__init__(**kwargs)

Creates a new builder and initializes it based on any provided parameters.

Parameters:
  • max_attempts_check (optional) (Boolean) – Whether to enable a check that we don’t exceed a certain number of attempts. If not provided this defaults to True (with a default max-attempts = 8)
  • service_error_check (optional) (Boolean) – Whether to enable a check that will retry on connection errors, timeouts and service errors which match given combinations of HTTP statuses and textual error codes. If not provided this defaults to True (i.e. this check will be done)
  • total_elapsed_time_check (optional) (Boolean) – Whether to enable a check that we don’t exceed a certain amount of time when retrying. This is intended for scenarios such as “keep retrying for 5 minutes”. If not provided this defaults to True (i.e. this check will be done with the total elapsed time of 600 seconds)
  • max_attempts (optional) (int) – If we are checking that we don’t exceed a certain number of attempts, what that number of attempts should be. This only applies if we are performing a check on the maximum number of attempts and will be ignored otherwise. If we are performing a check on the maximum number of attempts and this value is not provided, we will default to a maximum of 8 attempts
  • total_elapsed_time_seconds (optional) (int) – If we are checking that we don’t exceed a certain amount of time when retrying, what that amount of time should be (in seconds). This only applies if we are performing a check on the total elapsed time and will be ignored otherwise. If we are performing a check on the total elapsed time and this value is not provided, we will default to 600 seconds (10 minutes)
  • service_error_retry_config (optional) (dict) –

    If we are checking on service errors, we can configure what HTTP statuses (e.g. 429) to retry on and, optionally, whether the textual code (e.g. TooManyRequests) matches a given value.

    This is a dictionary where the key is an integer representing the HTTP status, and the value is a list(str) where we will test if the textual code in the service error is a member of the list. If an empty list is provided, then only the numeric status is checked for retry purposes.

    If we are performing a check on service errors and this value is not provided, then by default we will retry on HTTP 409/IncorrectState, 429’s (throttles) without any textual code check.

  • service_error_retry_on_any_5xx (optional) (Boolean) – If we are checking on service errors, whether to retry on any HTTP 5xx received from the service. If we are performing a check on service errors and this value is not provided, it defaults to True (retry on any 5xx except 501)
  • retry_base_sleep_time_seconds (optional) (int) – For exponential backoff with jitter, the base time to use in our retry calculation in seconds. If not provided, this value defaults to 1 second
  • retry_exponential_growth_factor (optional) (int) – For exponential backoff with jitter, the exponent which we will raise to the power of the number of attempts. If not provided, this value defaults to 2
  • retry_max_wait_between_calls_seconds (optional) (int) – For exponential backoff with jitter, the maximum amount of time to wait between retries. If not provided, this value defaults to 30 seconds
  • decorrelated_jitter (optional) (int) – The random De-correlated jitter value in seconds (default 1) to be used when using the backoff_type BACKOFF_DECORRELATED_JITTER_VALUE
  • backoff_type (optional) (str) – The type of backoff we want to do (e.g. full jitter). The convenience constants in this retry module: BACKOFF_DECORRELATED_JITTER_VALUE, BACKOFF_FULL_JITTER_VALUE,`BACKOFF_EQUAL_JITTER_VALUE``, and BACKOFF_FULL_JITTER_EQUAL_ON_THROTTLE_VALUE can be used as values here. If no value is specified then the value BACKOFF_DECORRELATED_JITTER_VALUE will be used. This will use exponential backoff and a random de-correlated jitter.
add_max_attempts(max_attempts=8)
add_service_error_check(**kwargs)
add_total_elapsed_time(total_elapsed_time_seconds=600)
get_retry_strategy()
no_max_attemps()
no_service_error_check()
no_total_elapsed_time()
class oci.retry.NoneRetryStrategy

A strategy that does not retry

Attributes

__init__(*args, **kwargs) Initialize self.

Methods

make_retrying_call(func_ref, *func_args, …) Calls the function given by func_ref.
__init__(*args, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

make_retrying_call(func_ref, *func_args, **func_kwargs)

Calls the function given by func_ref. Any positional (*func_args) and keyword (**func_kwargs) arguments are passed as-is to func_ref.

Parameters:func_ref (function) – The function that we should call with retries
Returns:the result of calling func_ref
class oci.retry.ExponentialBackoffRetryStrategyBase(base_sleep_time_seconds, exponent_growth_factor, max_wait_between_calls_seconds, checker_container, **kwargs)

A base retry strategy from which other retry strategies inherit. Implementors can create a subclass of this to define their own retry logic. This is primarily useful when an implementor wishes to customize the sleep strategy used - to customize the checking logic on whether a retry should be done, implementors can define their own subclasses of oci.retry.retry_checkers.BaseRetryChecker and provide this in the checker container.

Methods

__init__(base_sleep_time_seconds, …) Creates a new instance of an exponential backoff with full jitter retry strategy.
add_circuit_breaker_callback(…)
do_sleep(attempt, exception)
make_retrying_call(func_ref, *func_args, …) Calls the function given by func_ref.
__init__(base_sleep_time_seconds, exponent_growth_factor, max_wait_between_calls_seconds, checker_container, **kwargs)

Creates a new instance of an exponential backoff with full jitter retry strategy.

Parameters:
  • base_sleep_time_seconds (int) – The base amount to sleep by, in seconds
  • exponent_growth_factor (int) – The exponent part of our backoff. We will raise take this value and raising it to the power of attemps and then multiply this with base_sleep_time_seconds
  • max_wait_between_calls_seconds (int) – The maximum time we will wait between calls, in seconds
  • checker_container (retry_checkers.RetryCheckerContainer) – The checks to run to determine whether a failed call should be retried
add_circuit_breaker_callback(circuit_breaker_callback)
do_sleep(attempt, exception)
make_retrying_call(func_ref, *func_args, **func_kwargs)

Calls the function given by func_ref. Any positional (*func_args) and keyword (**func_kwargs) arguments are passed as-is to func_ref.

Parameters:func_ref (function) – The function that we should call with retries
Returns:the result of calling func_ref
class oci.retry.ExponentialBackoffWithFullJitterRetryStrategy(base_sleep_time_seconds, exponent_growth_factor, max_wait_between_calls_seconds, checker_container, **kwargs)

A retry strategy which does exponential backoff and full jitter. Times used are in seconds and the strategy can be described as:

random(0, min(base_sleep_time_seconds * exponent_growth_factor ** (attempt), max_wait_seconds))

Methods

__init__(base_sleep_time_seconds, …) Creates a new instance of an exponential backoff with full jitter retry strategy.
add_circuit_breaker_callback(…)
do_sleep(attempt, exception)
make_retrying_call(func_ref, *func_args, …) Calls the function given by func_ref.
__init__(base_sleep_time_seconds, exponent_growth_factor, max_wait_between_calls_seconds, checker_container, **kwargs)

Creates a new instance of an exponential backoff with full jitter retry strategy.

Parameters:
  • base_sleep_time_seconds (int) – The base amount to sleep by, in seconds
  • exponent_growth_factor (int) – The exponent part of our backoff. We will raise take this value and raising it to the power of attempts and then multiply this with base_sleep_time_seconds
  • max_wait_between_calls_seconds (int) – The maximum time we will wait between calls, in seconds
  • checker_container (retry_checkers.RetryCheckerContainer) – The checks to run to determine whether a failed call should be retried
add_circuit_breaker_callback(circuit_breaker_callback)
do_sleep(attempt, exception)
make_retrying_call(func_ref, *func_args, **func_kwargs)

Calls the function given by func_ref. Any positional (*func_args) and keyword (**func_kwargs) arguments are passed as-is to func_ref.

Parameters:func_ref (function) – The function that we should call with retries
Returns:the result of calling func_ref
class oci.retry.ExponentialBackoffWithEqualJitterRetryStrategy(base_sleep_time_seconds, exponent_growth_factor, max_wait_between_calls_seconds, checker_container, **kwargs)

A retry strategy which does exponential backoff and equal jitter. Times used are in seconds and the strategy can be described as:

exponential_backoff_sleep = min(base_sleep_time_seconds * exponent_growth_factor ** (attempt), max_wait_seconds)
sleep_with_jitter = (exponential_backoff_sleep / 2) + random(0, exponential_backoff_sleep / 2)

Methods

__init__(base_sleep_time_seconds, …) Creates a new instance of an exponential backoff with equal jitter retry strategy.
add_circuit_breaker_callback(…)
do_sleep(attempt, exception)
make_retrying_call(func_ref, *func_args, …) Calls the function given by func_ref.
__init__(base_sleep_time_seconds, exponent_growth_factor, max_wait_between_calls_seconds, checker_container, **kwargs)

Creates a new instance of an exponential backoff with equal jitter retry strategy.

Parameters:
  • base_sleep_time_seconds (int) – The base amount to sleep by, in seconds
  • exponent_growth_factor (int) – The exponent part of our backoff. We will raise take this value and raising it to the power of attemps and then multiply this with base_sleep_time_seconds
  • max_wait_between_calls_seconds (int) – The maximum time we will wait between calls, in seconds
  • checker_container (retry_checkers.RetryCheckerContainer) – The checks to run to determine whether a failed call should be retried
add_circuit_breaker_callback(circuit_breaker_callback)
do_sleep(attempt, exception)
make_retrying_call(func_ref, *func_args, **func_kwargs)

Calls the function given by func_ref. Any positional (*func_args) and keyword (**func_kwargs) arguments are passed as-is to func_ref.

Parameters:func_ref (function) – The function that we should call with retries
Returns:the result of calling func_ref
class oci.retry.ExponentialBackoffWithFullJitterEqualForThrottlesRetryStrategy(base_sleep_time_seconds, exponent_growth_factor, max_wait_between_calls_seconds, checker_container, **kwargs)

A retry strategy that does exponential backoff and full jitter for most retries, but uses exponential backoff with equal jitter for throttles. This provides a reasonable distribution of retry times for most retryable error cases, but for throttles guarantees some sleep time

Methods

__init__(base_sleep_time_seconds, …) Creates a new instance of the retry strategy.
add_circuit_breaker_callback(…)
do_sleep(attempt, exception)
make_retrying_call(func_ref, *func_args, …) Calls the function given by func_ref.
__init__(base_sleep_time_seconds, exponent_growth_factor, max_wait_between_calls_seconds, checker_container, **kwargs)

Creates a new instance of the retry strategy.

Parameters:
  • base_sleep_time_seconds (int) – The base amount to sleep by, in seconds
  • exponent_growth_factor (int) – The exponent part of our backoff. We will raise take this value and raising it to the power of attemps and then multiply this with base_sleep_time_seconds
  • max_wait_between_calls_seconds (int) – The maximum time we will wait between calls, in seconds
  • checker_container (retry_checkers.RetryCheckerContainer) – The checks to run to determine whether a failed call should be retried
add_circuit_breaker_callback(circuit_breaker_callback)
do_sleep(attempt, exception)
make_retrying_call(func_ref, *func_args, **func_kwargs)

Calls the function given by func_ref. Any positional (*func_args) and keyword (**func_kwargs) arguments are passed as-is to func_ref.

Parameters:func_ref (function) – The function that we should call with retries
Returns:the result of calling func_ref
class oci.retry.ExponentialBackOffWithDecorrelatedJitterRetryStrategy(base_sleep_time_seconds, exponent_growth_factor, max_wait_between_calls_seconds, checker_container, decorrelated_jitter=1, **kwargs)

A retry strategy which does exponential backoff with decorrelated jitter. Times used are in seconds and the strategy can be described as:

min(base_sleep_time_seconds * exponent_growth_factor ** (attempt) + random(0, 1), max_wait_seconds))

Methods

__init__(base_sleep_time_seconds, …[, …]) Creates a new instance of an exponential backoff with Decorrelated jitter retry strategy.
add_circuit_breaker_callback(…)
do_sleep(attempt, exception)
make_retrying_call(func_ref, *func_args, …) Calls the function given by func_ref.
__init__(base_sleep_time_seconds, exponent_growth_factor, max_wait_between_calls_seconds, checker_container, decorrelated_jitter=1, **kwargs)

Creates a new instance of an exponential backoff with Decorrelated jitter retry strategy.

Parameters:
  • base_sleep_time_seconds (int) – The base amount to sleep by, in seconds
  • exponent_growth_factor (int) – The exponent part of our backoff. We will raise take this value and raising it to the power of attempts and then multiply this with base_sleep_time_seconds
  • max_wait_between_calls_seconds (int) – The maximum time we will wait between calls, in seconds
  • checker_container (retry_checkers.RetryCheckerContainer) – The checks to run to determine whether a failed call should be retried
  • decorrelated_jitter (optional) (int) – The amount of time in seconds (default 1) to be used as de-correlated jitter between subsequent retries.
add_circuit_breaker_callback(circuit_breaker_callback)
do_sleep(attempt, exception)
make_retrying_call(func_ref, *func_args, **func_kwargs)

Calls the function given by func_ref. Any positional (*func_args) and keyword (**func_kwargs) arguments are passed as-is to func_ref.

Parameters:func_ref (function) – The function that we should call with retries
Returns:the result of calling func_ref
oci.retry.DEFAULT_RETRY_STRATEGY = <oci.retry.retry.ExponentialBackOffWithDecorrelatedJitterRetryStrategy object>

A retry strategy which has all options enabled and which will use the default settings for those options. These defaults are:

  • 8 total attempts - 1 original and 7 retries

  • Total allowed elapsed time for all requests of 600 seconds (10 minutes)

  • Exponential backoff with jitter using a base time of 1 second and an exponent of 2

  • The maximum wait time between calls is 30 seconds

  • Exponential backoff with de-correlated jitter of 1000 milliseconds is used

  • Retries on the following exception types:-

    • timeouts and connection errors
    • HTTP 409/IncorrectState, 429s (throttles)
    • Any HTTP 5xx except 501
oci.retry.GLOBAL_RETRY_STRATEGY = None

A retry strategy which can be set by the user to modify the SDK retry behavior globally. Initially set to None, users can pass to it a Retry Strategy which can be:-

  • A retry strategy built using oci.retry.RetryStrategyBuilder
  • The oci.retry.DEFAULT_RETRY_STRATEGY
  • oci.retry.NoneRetryStrategy() which will disable retries at SDK level

A helpful environment variable OCI_SDK_DEFAULT_RETRY_ENABLED is also provided to enable/disable default retries for the SDK

Retry Checkers

Classes

BaseRetryChecker(**kwargs) The base class from which all retry checkers should derive.
LimitBasedRetryChecker([max_attempts]) A retry checker which can retry as long as some threshold (# of attempts/tries) has not been breached.
RetryCheckerContainer(checkers, **kwargs) A container which holds at least one retry checker.
TimeoutConnectionAndServiceErrorRetryChecker([…]) A checker which will retry on certain exceptions.
TotalTimeExceededRetryChecker([…]) A retry checker which can retry as long as some upper time limit (in seconds) has not been breached.
class oci.retry.retry_checkers.BaseRetryChecker(**kwargs)

The base class from which all retry checkers should derive. This has no implementation but just defines the contract for a checker. Implementors can extend this class to define their own checking logic.

Methods

__init__(**kwargs) Initialize self.
should_retry([exception, response]) Determines if a retry should be performed based on either an exception or a response.
__init__(**kwargs)

Initialize self. See help(type(self)) for accurate signature.

should_retry(exception=None, response=None, **kwargs)

Determines if a retry should be performed based on either an exception or a response.

Parameters:
  • exception (Exception) – An exception received from the service
  • response (Response) – The Response received from a service call
Returns:

True if we should retry, and False otherwise
Return type:

Boolean
class oci.retry.retry_checkers.LimitBasedRetryChecker(max_attempts=8, **kwargs)

A retry checker which can retry as long as some threshold (# of attempts/tries) has not been breached. It is the responsiblity of the caller to track how many attempts/tries it has done - objects of this class will not track this.

If not specified, the default number of tries allowed is 8. Tries are also assumed to be one-based (i.e. the first attempt/try is 1, the second is 2 etc)

Methods

__init__([max_attempts]) Initialize self.
should_retry([exception, response]) Determines if a retry should be performed based on either an exception or a response.
__init__(max_attempts=8, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

should_retry(exception=None, response=None, **kwargs)

Determines if a retry should be performed based on either an exception or a response.

Parameters:
  • exception (Exception) – An exception received from the service
  • response (Response) – The Response received from a service call
Returns:

True if we should retry, and False otherwise
Return type:

Boolean
class oci.retry.retry_checkers.RetryCheckerContainer(checkers, **kwargs)

A container which holds at least one retry checker. This lets us chain together different retry checkers into an overall evaluation of whether we should retry a request.

Checkers are evaluated in the order they appear in the provided list of checkers, and if one checker reports failure we consider this to be an overall failure and no more retries should happen.

Methods

__init__(checkers, **kwargs) Initialize self.
add_checker(checker)
should_retry([exception, response]) Determines if a retry should be performed based on either an exception or a response.
__init__(checkers, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

add_checker(checker)
should_retry(exception=None, response=None, **kwargs)

Determines if a retry should be performed based on either an exception or a response. We will retry if all the checkers held in this container indicate that they should retry; if any checker indicates that the call should not be retried then we will not retry.

Parameters:
  • exception (Exception) – An exception received from the service
  • response (Response) – The Response received from a service call
Returns:

True if we should retry, and False otherwise
Return type:

Boolean
class oci.retry.retry_checkers.TimeoutConnectionAndServiceErrorRetryChecker(service_error_retry_config={-1: [], 409: ['IncorrectState'], 429: []}, retry_any_5xx=True, **kwargs)

A checker which will retry on certain exceptions. Retries are enabled for the following exception types:

  • Timeouts from the requests library (we will always retry on these)
  • ConnectionErrors from the requests library (we will always retry on these)
  • Built-in ConnectionErrors from Python 3
  • Service errors where the status is 500 or above (i.e. a server-side error, except 501)
  • Service errors where a status (e.g. 429) and, optionally, the code meet a given criteria

Attributes

RETRYABLE_STATUSES_AND_CODES dict() -> new empty dictionary

Methods

__init__([service_error_retry_config, …]) Initialize self.
should_retry([exception, response]) Determines if a retry should be performed based on either an exception or a response.

The last item is configurable via dictionary where the key is some numeric status representing a HTTP status and the value is a list of strings with each string representing a textual error code (such as those error codes documented at https://docs.cloud.oracle.com/Content/API/References/apierrors.htm). If an empty list is provided, then only the numeric status is checked for retry purposes. For a populated array, we are looking for where the numeric status matches and the code from the exception appears in the array. As an example:

{
    400: ['QuotaExceeded'],
    500: []
}

If no configuration is provided, then the default for service errors is to retry on HTTP 409/IncorrectState, 429’s and 5xx’s (except 501) without any code checks. If a specific 5xx code (e.g. 500, 502) is provided in the dictionary, then it takes precedence over the option to retry on any 500, for example, it is possible to retry on only 502s (either by status or by status and matching some code) by disabling the general “retry on any 5xx” configuration and placing an entry for 502 in the dictionary

RETRYABLE_STATUSES_AND_CODES = {-1: [], 409: ['IncorrectState'], 429: []}
__init__(service_error_retry_config={-1: [], 409: ['IncorrectState'], 429: []}, retry_any_5xx=True, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

should_retry(exception=None, response=None, **kwargs)

Determines if a retry should be performed based on either an exception or a response.

Parameters:
  • exception (Exception) – An exception received from the service
  • response (Response) – The Response received from a service call
Returns:

True if we should retry, and False otherwise
Return type:

Boolean
class oci.retry.retry_checkers.TotalTimeExceededRetryChecker(time_limit_seconds=600, **kwargs)

A retry checker which can retry as long as some upper time limit (in seconds) has not been breached. This is intended for scenarios such as “keep retrying for 5 minutes”. It is the responsiblity of the caller to track the total time elapsed - objects of this class will not track this.

If not specified, the default time limit is 600 seconds (10 minutes).

Methods

__init__([time_limit_seconds]) Initialize self.
should_retry([exception, response]) Determines if a retry should be performed based on either an exception or a response.
__init__(time_limit_seconds=600, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

should_retry(exception=None, response=None, **kwargs)

Determines if a retry should be performed based on either an exception or a response.

Parameters:
  • exception (Exception) – An exception received from the service
  • response (Response) – The Response received from a service call
Returns:

True if we should retry, and False otherwise
Return type:

Boolean