To properly boot after installing RHEL AS 5 on an X8440 Server Module with a fully populated DIMM configuration, RHEL AS 5 you must install the latest software patch.
RHEL AS 5 panics after booting with 96 Gbytes (4-Gbyte DIMMs per CPU) on an X8440 Server Module. (CR 6557717)
Download and install the latest RHEL AS 5 patch. For more information, refer to the RHEL AS 5 Release Notes Updates at the following site:
http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/release-notes/RELEASE-NOTES-x86_64-en.html
Alternatively, you can remove memory from the system so that no more than 64 Gbytes of memory is installed. Or, you can use kernel arguments dom0_mem=512M mem=64G when booting.