3000000000001000 - Server Disk Space Shortage Error

This alarm indicates that one of the following conditions has occurred:

A file system has exceeded a failure threshold, which means that more than 90% of the available disk storage has been used on the file system.
More than 90% of the total number of available files have been allocated on the file system.
A file system has a different number of blocks than it had when installed.

Recovery

Run syscheck.
Examine the syscheck output to determine if the file system /var/TKLC/lsms/free is low on space. If it is, continue to the next step; otherwise go to Step 4.
If possible, recover space on the free partition by deleting unnecessary files:
1. Log in to the server generating the alarm as the root user:
  Login: root
  
  Password:<Enter root password>
2. Change to the /var/TKLC/lsms/free directory: # cd /var/TKLC/lsms/free
3. Confirm that you are in the /var/TKLC/lsms/free directory: # pwd /var/TKLC/lsms/free
4. When the pwd command is executed, if /var/TKLC/lsms/free is not output, go back to Sub-step b
5. List files to be deleted and delete them using the rm command
6. Re-run syscheck
If the alarm is cleared, the problem is solved. If the alarm is not cleared, go to the next Step.
If the file system mounted on /var/TKLC/lsms/logs is the file system that syscheck is reporting to be low on space, execute the following steps:
1. Log in to the server generating the alarm as the root user:
  Login: root
  
  Password:<Enter root password>
2. Change to the /var/TKLC/lsms/logs directory: # cd /var/TKLC/lsms/logs
3. Confirm that you are in the /var/TKLC/lsms/logs directory: # pwd /var/TKLC/lsms/logs
4. When the pwd command is executed, if /var/TKLC/lsms/logs is not output, go back to Sub-step b
5. Look for files with names matching: logs_(hostname)_(date/timestamp).tar, where (hostname) is replaced by the server’s hostname, and (date/timestamp) is any date or timestamp. # ls logs_'hostname'_*.tar. Any files listed may be safely deleted, so for each file listed in the ls output, execute an rm command: # rm <filename> where <filename> is replaced by the name of the file to be deleted.
6. Re-run syscheck
  If the alarm is cleared, the problem is solved. If the alarm is not cleared, go to the next Step.
Core files can occupy a large amount of disk space and may the cause of this alarm. To collect and remove any core files from the server:
1. Log in to the server generating the alarm as the root user:
  Login: root
  
  Password:<Enter root password>
2. Change directory to /var/TKLC/core and list the core files. # cd /var/TKLC/core # ls -l
Note:
The ls command shown above will list any core files found and then compresses and renames the file, adding a ".gz" extension. If any core files are found, transfer them off the system and save them for examination by Oracle Engineering. Once a copy of a compressed file has been saved, it is safe to delete it from the server.
1. Re-run syscheck
  If the alarm is cleared, the problem is solved. If the alarm is not cleared, go to the next Step.
Execute the following Sub-steps if the file system reported by syscheck is /tmp, otherwise skip to Step 7:
1. Log in to the server generating the alarm as the root user:
  Login: root
  
  Password:<Enter root password>
2. Change to the /tmp directory: # cd /tmp
3. Confirm that you are in the /tmpdirectory: # pwd /tmp
4. When the pwd command is executed, if /tmp is not output, go back to Step 5.
5. Look for possible candidates for deletion: # ls *.iso *.bz2 *.gz *.tar *.tgz *.zip
6. If any deletable files exist, the output of the ls will show them. For each of the files listed, execute the rm command to delete the file: # rm <filename>
7. Run syscheck
  If the alarm is cleared, the problem is solved. If the alarm is not cleared, go to Step 4.
8. Upon a reboot, the system will clean the /tmp directory.
  To reboot the system issue the # shutdown -r now command.
9. Re-run syscheck
  If the alarm is cleared, the problem is solved. If the alarm is not cleared, go to the next Step.
Execute the following steps if the file system reported by syscheck is /var, otherwise skip to Step 10:
1. Log in to the server generating the alarm as the root user:
  Login: root
  
  Password:<Enter root password>
2. Change to the /var/tmp directory: # cd /var/tmp
3. Confirm that you are in the /var/tmp directory: # pwd, then /var/tmp
4. When the pwd command is executed, if /var/tmp is not output, go back to Step 5.
5. Since all files in this directory can be safely deleted, execute the rm * command to delete all files from the directory: # rm -i *.
6. Re-run syscheck
  If the alarm is cleared, the problem is solved. If the alarm is not cleared, go to Step 10.
Execute the following steps if the file system reported by syscheck is /var/TKLC, otherwise skip to Step 10.
1. Log in to the server generating the alarm as the root user:
  Login: root
  
  Password:<Enter root password>
2. Change to the /var/TKLC/upgrade directory: # cd /var/TKLC/upgrade
3. Confirm that you are in the /var/TKLC/upgrade directory: # pwd, then /var/TKLC/upgrade
4. When the pwd command is executed, if /var/tmp is not output, go back to Step 5.
5. Since all files in this directory can be safely deleted, execute the rm * command to delete all files from the directory: # rm -i *.
6. Re-run syscheck
  If the alarm is cleared, the problem is solved. If the alarm is not cleared, go to Step 10.
For any other file system, execute the following command, where <mountpoint> is the file system’s mount point: # find <mountpoint> -type f -exec du -k {} \; | sort -nr > /tmp/file_sizes.txt
This will produce a list of files in the given file system sorted by file size in the file /tmp/file_sizes.txt.
Note: The find command above may take a few minutes to complete if the given mountpoint contains many files. Do not delete any files unless care certain that they are not needed. Continue to Step 10.
Run savelogs to gather all application logs (see Saving Logs Using the LSMS GUI or Command Line).
Contact the My Oracle Support (MOS).