Oracle Cloud Infrastructure Documentation

Login Troubleshooting

If you are unable to log into an Oracle Linux instance on Oracle Cloud Infrastructure, review the following information.

Logging into an Oracle Linux Instance Fails with a "fork: retry: Resource temporarily unavailable" Error

The fork failed: Resource temporarily unavailable error on Oracle Linux usually indicates the system is unable to create new processes due to resource limitations such as a full process table, insufficient memory, or reaching user or system limits.

Potential Causes

  • User or system resource limits were reached - A user might have a limit on the number of processes they can create, or the number of open files they can have. Or, the system might be unable to create new processes because it's running out of resources or has reached its maximum number of processes.
  • Insufficient system memory or swap space - The system might not have enough memory or swap space to allocate for new processes.
  • A misbehaving service or process - A runaway process or service could be consuming excessive resources, preventing new processes from starting.
  • Too many file descriptors are open - The system might be unable to open new files because it has reached its maximum number of open file descriptors.
  • Misconfiguration or error in the /etc/security/limits.conf file - if there is a misconfiguration or error in the limits.conf file, the higher values might not be applied and a user account could hit a default limit triggering the error.

Troubleshooting

  • Monitor resource usage and identify and terminate unnecessary processes to free up memory and swap space
    • Use top or vmstat to identify processes consuming high CPU or memory, for example:
      $ vmstat

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
1  0      0 12607316   4336 2737552    0    0   109   302  182  267  3  1 96  0  0

$ top

top - 16:04:04 up  1:07,  2 users,  load average: 0.01, 0.13, 0.10
Tasks: 180 total,   1 running, 179 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.2 sy,  0.2 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15700.7 total,  12305.2 free,    717.2 used,   2678.4 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  14673.0 avail Mem
PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
12331 opc       20   0   54424   5040   4252 R   0.7   0.0   0:00.07 top
11972 root      30  10 2095708 292468 123100 S   0.3   1.8   0:05.65 wlp-agen+
...
    • Use ps aux to list all processes and their resource usage, for example:
      $ ps aux

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.1  0.1 179460 17548 ?        Ss   14:56   0:04 /usr/lib/syste
root           2  0.0  0.0      0     0 ?        S    14:56   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   14:56   0:00 [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   14:56   0:00 [rcu_par_gp]
...
    • Terminate processes that are no longer needed or causing problems using kill <PID> or kill -9 <PID>, for example:
      [opc@<oracle-linux-instance-name> ~]$ kill -9 12346
  • Check and adjust user resource limits
    • Use ulimit -a to check current user limits, for example:
      $ ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 62425
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
...
    • Use ulimit to temporarily increase a limit. For example, to temporarily increase the number of open files a user can have, use ulimit -n <number>:
      $ ulimit -n 2096

$ ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 62425
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 2096
...
    • Check the /etc/security/limits.conf file for correct formatting, no typing errors, and that the values set for a user account are what you expect. If there is a misconfiguration or error, the system might not apply higher values and a user account could hit a default limit triggering the error.

      For example, the following is a correct entry in the /etc/security/limits.conf file to set the maximum number of processes for a single user or all users:

      #<domain>      <type>  <item>         <value>

*                hard    nproc           2048
oracle           hard    nproc           65356
root             hard    nproc           unlimited
      Note

      The limits.conf file can be used to set custom user limits for different system resources instead of using the system default limits. See the limits.conf(5) manual page for details.
  • Increase system limits
    • Edit /etc/sysctl.conf to increase system-wide limits. For example to set the maximum number of open files:
      fs.file-max = 204708
    • Reboot the system for the changes to take effect.
  • Check for memory issues
    • Use free -h to check the current memory usage and swap space, for example:
      $ free -h
total        used        free      shared  buff/cache   available
Mem:           15Gi       727Mi        11Gi        16Mi       2.8Gi        14Gi
Swap:         4.0Gi          0B       4.0Gi
    • If memory is low, consider increasing swap space or upgrading RAM.