Using Software Distribution Mirrors

6 Using Software Distribution Mirrors

Distribution mirrors are alternative sources of software packages to repositories on the Oracle Linux yum server or Unbreakable Linux Network. These are selected repositories that you locally replicate from the public server. The local repositories become the package sources for client systems that exist in the local network.

Distribution mirrors are useful in complex infrastructures and are important when developing a controlled update strategy for a mission critical production environment. Distribution mirrors are deployed to provide the following services:

Provide access to yum repositories or ULN channels for systems that don't have access to a public network.
Improve software download times and reducing bandwidth overhead for larger infrastructure
Set up network-based installation infrastructure
Cater for a snapshot style update strategy where testing can be performed against a controlled software distribution environment before the updates are implemented on production systems.

A server that functions as a software distribution mirror contains yum repositories or ULN channels. The repositories or channels can be made available to client systems in the internal network through various methods such as using local web server, a file transfer server, and so on.

The software distribution mirror must be synchronized with the official Oracle Linux sources. If required, you can control synchronization to occur at strategic intervals so that you can test system updates against a known set of package versions before you roll them out to all the infrastructure.

Note:

If you're considering mirroring ULN channels on a local server, check also Oracle Linux Manager that's based on the Spacewalk open source software. Oracle Linux Manager provides tools to help with system maintenance, installation, and package management. For more information, see Oracle^® Linux Manager & Spacewalk for Oracle^® Linux Documentation.

Prerequisites for the Local Distribution Mirror

The system that you set up as a local distribution mirror must meet the following criteria:

Must have Internet access to connect to the official Oracle Linux sources.
Has at least 6 GB of memory to create the yum metadata.
Must be configured to provide access to the mirrored repositories by system clients.
Has enough disk space to store copies of the packages that it hosts.

When calculating for the needed disk space, consider the following:
- Disk space requirements depend on the repositories or channels that you choose to mirror. Other factors are the number of clients to be serviced, including their platforms, operating systems, and other specific packages that each client might be using and which would require updates.
- Disk space that's used for a mirror is only consumed and is never released. Thus, disk requirements aren't static and can increase over time.
- Packages in the repositories or channels are also updated on a regular schedule and further affects the storage requirements on the local yum server.
For guidance in estimating the disk size requirements, run the following command:
```
sudo dnf repoinfo [repo-ID]
```
Part of the command output includes the size of a specific repository, for example:
```
...
Repo-id            : ol8_x86_64_baseos_latest
...
Repo-size          : 29 G
...

Repo-id            : ol8_addons
...
Repo-size          : 4.8 G
...
```
Because repositories are dynamic and grow over time, always plan to allocate substantially greater disk space than what Repo-size specifies. Optionally, you can also create a dedicated file system and mount this to the directory that hosts the mirrored repositories.

How to Set Up a Distribution Mirror

Systems can be configured to distribute packages and provide updates to client systems within a local network without the need for clients to access the public servers through the Internet.

You can select any method to provide access to the local repositories in the mirror server. This task uses HTTP as an example.

Ensure that the latest version of the yum-utils is installed on the system.
```
sudo dnf install -y yum-utils
```
Install the Apache HTTP server.
```
sudo dnf install -y httpd
```
Create a base directory for the local repositories, for example:
```
sudo mkdir -p /var/www/html/yum
```
You can create the base directory anywhere. However, the repository owner must have read and write permissions on that location.
If you created a dedicated file system for the mirror, them mount that file system to the base directory.
If you created a base directory in a different location than /var/www/html, create a symbolic link in /var/www/html that points to the repository.
For example, if the base directory is /var/yum, type:
```
sudo ln -s /var/yum /var/www/html/yum
```
If SELinux is enabled in enforcing mode, do the following steps:
1. Define the default file type of the repository root directory hierarchy as httpd_sys_content_t.
```
sudo /usr/sbin/semanage fcontext -a -t httpd_sys_content_t "/var/repos(/.*)?"
```
2. Apply the file type to the entire repository.
```
sudo /sbin/restorecon -R -v /var/yum
```
Edit the HTTP server configuration file, /etc/httpd/conf/httpd.conf, as follows:
1. Specify the resolvable domain name or the IP address of the server in the argument to ServerName.
```
ServerName system-mirror:80
```
2. Verify that in the <Directory "/var/www/html"> section, the setting of the Options directive specifies Indexes and FollowSymLinks, for example:
```
Options Indexes FollowSymLinks
```
  With this setting, you can browse the directory hierarchy.
Start the HTTP server and configure it to start after a reboot.
```
sudo systemctl start httpd
sudo systemctl enable httpd
```
If you enabled a firewall on the system, configure it to enable incoming HTTP connection requests on TCP port 80.
```
sudo firewall-cmd --add-service=http
sudo firewall-cmd --permanent --add-service=http
```
Choose how you want the local mirror to function to serve clients in the local network:
- Set up the system as a yum mirror.
- Set up the system as a ULN mirror.

Setting Up a Local Yum Mirror

A system that functions as a local yum repository mirrors repositories from the public Oracle Linux yum server.

When Oracle Linux is installed on this system, that system automatically contains the repositories that are required by the system's OS. These repositories are found in the system's /etc/yum/repos.d directory. The repositories are defined in different /etc/yum/repos.d/*.repo files.

By mirroring these default repositories, the system can function as a local yum server to service clients that have the same OS and platform as the mirror.

However, you might want the local yum mirror to also service clients that use different OS releases for other platforms. In this case, you would need to define other repositories that are required by those clients.

How to Configure the Local Yum Server

Setting up a system to function as a local yum server involves mirroring required repositories from the public Oracle Linux yum server.

The yum mirror must meet the requirements described in Prerequisites for the Local Distribution Mirror. Also, you must have completed the procedure in How to Set Up a Distribution Mirror.

You can mirror any repository available on the Oracle Linux yum server, if you have the definition for the repository configured in /etc/yum.repos.d. Mirroring repositories that the system already has available is uncomplicated. However, for other repositories, you might need to be more specific about the which repositories you want to mirror. Moreover, you might need to provide other repository configuration.

Mirror all the current system's enabled repositories to the base directory.
```
sudo dnf reposync --delete --download-metadata -p /var/www/html/yum 
```
--delete

Remove from the mirror any package that's removed upstream. Using this option is highly recommended.

--download-metadata

Include all repository metadata in the synchronization.
If you run the command for the first time, the process might take a long while to complete. At the end of the process, the system becomes ready to provide packages to client systems with compatible OS and platforms as the mirror.

Set the local mirror to host repositories for heterogeneous clients.

Create the required repositories for mixed clients.

Suppose that the server is running the latest Oracle Linux 8 release, but must provide packages for Oracle Linux 9 and Oracle Linux 7 clients. You would do the following:

Create /etc/yum.repos.d/9-mirror.repo with entries similar to the following example:

[ol9_baseos_latest]
name=Oracle Linux 9 BaseOS Latest  ($basearch)  
baseurl=https://yum$ociregion.$ocidomain/repo/OracleLinux/OL9/baseos/latest/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0

[ol9_appstream]
name=Oracle Linux 9 Application Stream Packages ($basearch) 
baseurl=https://yum$ociregion.$ocidomain/repo/OracleLinux/OL9/appstream/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0

Create /etc/yum.repos.d/7-mirror.repo with entries similar to the following example:

[ol7_latest]
name=Oracle Linux 7 Latest ($basearch)
baseurl=https://yum$ociregion.$ocidomain/repo/OracleLinux/OL7/latest/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0

[ol7_optional_latest]
name=Oracle Linux $releasever Optional Latest ($basearch)
baseurl=https://yum$ociregion.$ocidomain/repo/OracleLinux/OL7/optional/latest/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0

[ol7_addons]
name=Oracle Linux $releasever Add ons ($basearch)
baseurl=https://yum$ociregion.$ocidomain/repo/OracleLinux/OL7/addons/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0

Full yum configurations for different releases are available at https://yum.oracle.com/mirror/. Navigate to the correct distribution and architecture and download the appropriate *.repo files.

Important:

All entries must have enabled=0 to prevent packages from these repositories to be installed on the local yum mirror itself.

Mirror each repository in the *.repo files to the base directory.

sudo dnf reposync --repoid ol9_baseos_latest --delete --download-metadata -p /var/www/html/yum
...

sudo dnf reposync --repoid ol97_latest --delete --download-metadata -p /var/www/html/yum
...

Automate the regular client package updates from mirrored repositories through a cron script or systemd timer unit.
For example, create a file at /etc/cron.daily/yum-mirror-update with the following content:
```
#!/bin/bash
# Regularly update yum repos
dnf reposync --delete --download-metadata -p /var/www/html/yum
```
Ensure that the file is executable.
```
sudo chmod +x /etc/cron.daily/yum-mirror-update
```
- If the yum mirror services mixed clients, change the script to resemble the following:
```
#!/bin/bash

REPOS=(repo-IDs)

for REPO in ${REPOS[@]}
do
    dnf reposync --repo=$REPO --delete --download-metadata -p /var/www/html/yum 
done
```
  repo-IDs represents a comma separated list of the IDs of repositories that are required by all the clients of the mirror. These IDs are contained in corresponding /etc/yum.repos.d/*.repo files you created for those clients. In this procedure's example, the repo-IDs would represent Oracle Linux 8 repositories for clients that are compatible with the mirror. In addition, you would include ol9_baseos_latest, ol9_appstream, ol7_latest, ol7_optional_latest,ol7_addons, and so on, for the other clients.
Configure clients appropriately to access these repositories.

See How to Configure Client Access to the Local Mirror.

How to Use rsync to Mirror the Oracle Linux Yum Server

Oracle provides an rsync interface to the Oracle Linux yum server repositories at the yum-rsync.oracle.com domain that maps directly to the URL structure of the Oracle Linux yum server.

You must fulfill the requirements as described in Prerequisites for the Local Distribution Mirror. Additionally, you must complete the procedure as provided in How to Set Up a Distribution Mirror.

With the rsync interface, you can easily mirror the Oracle Linux yum server for broader usage without any requirement for complex system configuration. This approach is helpful for large enterprises that want to mirror entire repository structures for all architectures. The rsync interface is an alternative method to running the reposync command to synchronize mirrored repositories.

Install rsync on the system.
```
sudo dnf install -y rsync
```
Use rsync to mirror all the repositories that you intend to mirror.
For example, to mirror all the Oracle Linux 8 repositories for all architectures, you can recursively mirror everything at the rsync://yum-rsync.oracle.com/repo/OracleLinux/OL8/ endpoint.
```
rsync -arv rsync://yum-rsync.oracle.com/repo/OracleLinux/OL8 /var/www/html/yum/
```
You can mirror a particular repository for a particular architecture by providing a more specific URL. For example, to mirror the current Oracle Linux 9 baseos repository for the x86_64 architecture, you would type:
```
mkdir -p /var/www/html/yum/OL9/baseos/latest
rsync -arv rsync://yum-rsync.oracle.com/repo/OracleLinux/OL9/baseos/latest/x86_64 /var/www/html/yum/OL9/baseos/latest/
```

How to Mirror Repositories From an ISO

The local yum mirror can be configured to mirror repositories from an ISO image to make them available to clients.

You must fulfill the requirements as described in Prerequisites for the Local Distribution Mirror. You must also complete the procedure as provided in How to Set Up a Distribution Mirror.

This task assumes that you're mirroring the repositories from an Oracle Linux 8 image. It also assumes that to provide access to the mirror, you're using a web server.

Mount the ISO image at an appropriate location so you can copy its contents.
```
sudo mount -o loop,ro OL8.iso /mnt
```
Create a directory to host the repositories from the ISO.
```
sudo mkdir -p /var/www/html/yum/8_ISO
```

Copy the repositories from the ISO to the new directory.

sudo cp -r /mnt/BaseOS /var/www/html/yum/8_ISO/
sudo cp -r /mnt/AppStream /var/www/html/yum/8_ISO/

Configure clients appropriately to access these repositories.

See How to Configure Client Access to the Local Mirror.

Setting Up a Local ULN Mirror

A system that functions as a local ULN server mirrors channels in the Unbreakable Linux Network.

When you register an Oracle Linux system with ULN, that system is automatically subscribed to default channels in ULN, depending on the system's OS release and architecture. As such, the system can become a mirror to service clients that have the same OS and platform as the mirror.

However, you might also want the local ULN mirror to service clients that use different OS releases for other platforms. In this case, you would need to subscribe to any other channels that are required by those clients.

Note:

Mirroring ULN channels is often slower than mirroring yum repositories. Only consider creating a ULN mirror for channels that aren't otherwise available on the Oracle Linux yum server. Where possible, set up mirrors of Oracle Linux yum server repositories instead.

How to Configure the Local ULN Mirror

Setting up the system to be a local ULN mirror involves replicating channels from Unbreakable Linux Network.

The designated ULN mirror must meet the requirements described in Prerequisites for the Local Distribution Mirror. Additionally, you must have completed the followng tasks:

For each step in this procedure, you can use either the ULN web interface or the uln-channel command. To display options that you can use with the uln-channel command, type uln-channel -h.

Enable the system as a yum server.
As a yum server, the system can subscribe to channels for OS versions and platforms other than the system's own OS and platform.
- Using the ULN web interface
1. On a browser, log in at https://linux.oracle.com with the proper credentials.
2. On the Systems tab, click the link named for the system designated to be a ULN mirror.
3. On the System Details page, click Edit.
4. On the Edit System Properties page, select the Yum Server check box.
5. Click Apply Changes.
- Using the uln-channel command
1. On the system's terminal window, type:
```
sudo uln-channel --enable-yum-server
```
2. If prompted, specify the appropriate ULN user name and password.
Subscribe the system to the channels that you intend to mirror.
- Using the ULN web interface
1. On the System Details page of the designated ULN mirror, click Manage Subscriptions.
2. On the System Summary page, select channels from the list of available or subscribed channels and click the arrows to move the channels between the lists.
  
  Note:
  
  If you have an Oracle Linux Support account and you want the mirror to host Ksplice packages for local Ksplice Offline clients, subscribe to the Ksplice for Oracle Linux channels for the architectures and Oracle Linux releases that you want to support.
3. When you have finished selecting channels, click Save Subscriptions.
- Using the uln-channel command
1. On the system's terminal window, type:
```
sudo uln-channel -a -c channel [-c channel …]
```
2. If prompted, specify the appropriate ULN user name and password.
3. (Optional) To verify that the subscriptions completed successfully, type:
```
sudo uln-channel -l
```
Protect the system's own repositories when other mirrored repositories are being updated.

See How to Localize Subscriptions for the ULN Mirror.
Mirror the ULN Channels to the location of the base directory for the mirror, by using the dnf reposync command.
```
sudo dnf reposync --delete --download-metadata -p /var/www/html/yum
```
Consider creating a cron script or systemd service and timer to run this command regularly. For example, create a file at /etc/cron.daily/uln-mirror-update with the following content:
```
#!/bin/bash
# Regularly update yum repos
dnf reposync --delete --download-metadata -p /var/www/html/yum
```
Ensure that the file is executable.
```
sudo chmod +x /etc/cron.daily/uln-mirror-update
```

How to Localize Subscriptions for the ULN Mirror

Localizing the ULN mirror's channel subscriptions prevents the mirror's packages from being updated that would cause package collisions and damage package dependencies.

Ensure that you have subscribed to required channels to serve clients running different OS versions on different platforms, as described in How to Configure the Local ULN Mirror.

This task is required for ULN mirrors that serve heterogeneous clients. In this case, the mirror subscribes to multiple channels, including channels the mirror itself doesn't need. You would need to configure the mirror to prevent its own channel subscriptions from being updated with packages targeted for other clients.

Suppose that the mirror is an Oracle Linux 9 system but is also serving Oracle Linux 8 clients on the x86_64 platform. The following steps would localize the Oracle Linux 9's channel subscriptions:

Identify the channels to which the server is subscribed.

sudo dnf repolist

...
ol8_addons                       Oracle Linux 8 Addons (x86_64)
ol8_appstream                    Oracle Linux 8 Application Stream (x86_64)
ol8_baseos_latest                Oracle Linux 8 BaseOS Latest (x86_64)
...

In addition to the system's own Oracle Linux 9 channels, the output would include Oracle Linux 8 channels intended for clients.

Edit /etc/dnf/plugins/spacewalk.conf to disable repository updates inapplicable to the server.
Use the following format:
```
[repo_id]
enabled=0
```
For the current example, you would specify the following on the file:
```
[ol8_addons]
enabled=0

[ol8_appstream]
enabled=0

[ol8_baseos_latest]
enabled=0
```
Note:

If you subsequently subscribe the system to any other incompatible channels on ULN, you must also disable those channels in /etc/dnf/plugins/spacewalk.conf.
Configure the mirror to be a client of itself.

See How to Configure Client Access to the Local Mirror.

How to Configure Client Access to the Local Mirror

Clients require access to the local repository mirror to receive updates and errata fixes.

A local mirror must be configured where the clients connect. See previous sections in Using Software Distribution Mirrors.

Perform this task on all the clients in the local network. Use this same procedure to configure the local ULN mirror as a client of itself.

Note:

On Oracle Linux 8 and later clients, use the dnf command. On earlier clients, use the yum command.

Import the GPG key.
```
sudo gpg --import /etc/pki/rpm-gpg/RPM-GPG-KEY
```
The location of the GPG key might differ depending on the Oracle Linux release that's installed on the system. You can also download and import the GPG keys directly from the Oracle Linux yum server. See https://yum.oracle.com/faq.html#a10 for more information.
Disable any existing yum repositories configured in the /etc/yum.repos.d directory.

Choose from one of the following methods:
- Edit each /etc/yum.repos.d/*.repo file to specify an enabled=0 setting for each entry in the file.
- Perform a global disable operation.
```
cd /etc/yum.repos.d
sudo dnf config-manager|yum-config-manager --disable \*
```
- Remove the .repo extension from the file names to cause yum operations to ignore these files.
```
/etc/yum.repos.d> sudo for i in *.repo; do mv $i $i.disabled; done
```
Create a local *.repo file, such as /etc/yum.repos.d/local-yum.repo, and populate it with repository entries from the local mirror.
Tip:

To distinguish the local repositories from the public yum repositories or ULN channels, prefix the names of their entries with a string such as local_.

The following example shows entries for an Oracle Linux 8 client:
```
[local_ol8_baseos_latest]
name=Oracle Linux 8 BaseOS Latest ($basearch)
baseurl=http://local_mirror/repo-location/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY
gpgcheck=1
enabled=1

[local_ol8_appstream]
name=Oracle Linux 8 Application Stream ($basearch)
baseurl=http://local_mirror/repo-location/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY
gpgcheck=1
enabled=1

[local_ol8_addons]
name=Oracle Linux 8 Addons ($basearch)
baseurl=http://local_mirror/repo-location/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY
gpgcheck=1
enabled=1
```
For local_mirror, you can specify either the local server's resolvable host name or its IP address.

Ensure that the following configurations are correct:
- All the entries have an enabled=1 setting.
- The baseurl points to the correct mirror location that contains the repositories that each client requires. The locations depend on how you organized the repositories in the mirror's base directory, such as /var/www/html/yum.
- The correct GPG key file must exist at the path that's specified for the gpgkey parameter. You can download the GPG keys used to sign all the Oracle Linux release packages from the Oracle Linux yum server. See https://yum.oracle.com/faq.html#a10 for more information.
Test the configuration.
1. Clear the yum metadata cache.
```
sudo dnf|yum clean metadata
```
2. Verify that the relevant repositories are listed for the client.
```
sudo dnf|yum repolist
```
  If the client can not connect to the local yum server, check that the firewall settings on the local yum server enable incoming TCP connections to the HTTP port, which is typically port 80.
After confirming that the correct repositories are configured on the client, obtain updates from the local server.
```
sudo dnf|yum update
```