Planning for Controlled Updates in a Production Environment

Software and OS updates can pose a problem for complex production environments that have mission critical applications that require minimal downtime. One solution might be to lock an environment to a single tested Oracle Linux release and update level to avoid updating the OS often. However, this approach increases the risk from security vulnerabilities and can make integration testing more difficult.

We recommend that you implement a software update strategy to ensure that the OS and underlying software packages on production systems are often updated in a way that you can manage the risk of application breakages because of software updates.

The following guidelines can help you to implement a software update strategy that's in line with best practice but protects the production systems from unexpected changes.

• Create a local ULN mirror.

  One of the challenges associated with rolling out updates on systems is that even if you have tested the updates in an integration and testing environment, if you don't manage the source of the updated packages, changes to packages can occur between the period of integration testing and the moment when you roll the package updates out to the production environment.

  By creating a local ULN mirror, you can control when and how often channels are synchronized to the mirror server. The selection of packages is static between synchronization periods, which gives you an opportunity to test a set of packages and then update the production environment to a known working set.

  By using ULN for the mirror service, you can mirror channels that contain Ksplice updates so that you can take advantage of an offline Ksplice service. With the offline Ksplice, you can use in-memory kernel updates to avoid reboots. At the same time, you can test these updates in an integration environment first, before applying the updates to the production environment.

• Consider a staged update strategy based on risk and threat mitigation.
  Not all updates are equal. You can time synchronization of ULN Mirror channels depending on requirements. Based on those requirements, you can configure systems to perform different update types on differing schedules. For example, you can work with a strategy similar to the following:

  • Schedule Oracle Ksplice updates for the kernel and user space to run at least weekly. Optionally, you can vet these updates within an integration test environment first.
  • For security related package updates, follow a monthly maintenance schedule and in line with alerts from security tools or errata notifications. Use the dnf update --security command for these types of update.
  • Apply at least a quarterly maintenance schedule to run full package updates that use a ULN mirror snapshot. Vet the updates on an integration test environment first before implementing these on production servers.

By performing regular atomic updates it's easier to resolve integration issues as they arise and you better protect an environment from potential security issues. Using an integration test environment and a Yum or ULN mirror is critical to maintaining stability of a platform and protecting it from compromise.