This chapter presents a collection of miscellaneous programming topics for systems programmers who want to understand more about the 64-bit Solaris operating environment.
Most of the new features of the 64-bit environment are extensions of generic 32-bit interfaces, though several new features are unique to 64-bit environments.
64-bit applications are described using Executable and Linking Format (ELF64), which allows large applications and large address spaces to be described completely.
The SPARC Compliance Definition, Version 2.4, contains details of the SPARC V9 ABI. It describes the 32-bit SPARC V8 ABI and the 64-bit SPARC V9 ABI. You can obtain this document from SPARC International at www.sparc.com.
Following is a list of the SPARC V9 ABI features.
The SPARC V9 ABI allows all 64-bit SPARC instructions and 64-bit wide registers to be used to their full effect. Many of the new relevant instructions are extensions of the existing V8 instruction set. See The SPARC Architecture Manual, Version 9.
The basic calling convention is the same. The first six arguments of the caller are placed in the out registers %o0-%o5. The SPARC V9 ABI still uses a register window on a larger register file to make calling a function a "cheap" operation. Results are returned in %o0. Because all registers are now treated as 64-bit quantities, 64-bit values can now be passed in a single register, rather than a register pair.
The layout of the stack is different. Apart from the increase in the basic cell size from 32-bit to 64-bit, various "hidden" parameter words have been removed. The return address is still %o7 + 8.
%o6 is still referred to as the stack pointer register %sp, and %i6 is the frame pointer register %fp. However, the %sp and %fp registers are offset by a constant, known as the stack bias, from the actual memory location of the stack. The size of the stack bias is 2047 bits.
Instruction sizes are still 32 bits. Address constant generation therefore takes more instructions. The call instruction can no longer be used to branch anywhere in the address space, since it can only reach within plus or minus 2 Gbytes of %pc.
Integer multiply and divide functions are now implemented completely in hardware.
Structure passing and return are accomplished differently. Small data structures and some floating point arguments are now passed directly in registers.
User traps allow certain traps from non-privileged code to be handled by a user trap handler (instead of delivering a signal).
All data types are now aligned to their size.
Many basic derived types are larger. Thus many system call interface data structures are now of different sizes.
Two different sets of libraries exist on the system: those for 32-bit SPARC applications and those for 64-bit SPARC applications.
An important feature of the SPARC V9 ABI for developers is the stack bias. For 64-bit SPARC programs, a stack bias of 2047 bytes must be added to both the frame pointer and the stack pointer to get to the actual data of the stack frame. See the following figure.
For more information on stack bias, please see the SPARC V9 ABI.
For 64-bit applications, the layout of the address space is closely related to that of 32-bit applications, though the starting address and addressing limits are radically different. Like SPARC V8, the SPARC V9 stack grows down from the top of the address space, while the heap extends the data segment from the bottom.
The diagram below shows the default address space provided to a 64-bit application. The regions of the address space marked as reserved might not be mapped by applications. These restrictions might be relaxed on future systems.
The actual addresses in the figure above describe a particular implementation on a particular machine, and are given for illustrative purposes only.
By default, 64-bit programs are linked with a starting address of 0x100000000. The whole program is above 4 gigabytes, including its text, data, heap, stack, and shared libraries. This helps ensure that 64-bit programs are correct by making it so the program will fault in the lower 4 gigabytes of its address space, if it truncates any of its pointers.
While 64-bit programs are linked above 4 gigabytes, you can still link them below 4 gigabytes by using a linker mapfile and the -M option to the compiler or linker. A linker mapfile for linking a 64-bit SPARC program below 4 gigabytes is provided in /usr/lib/ld/sparcv9/map.below4G.
See the ld(1) linker man page for more information.
Different code models are available from the compiler for different purposes to improve performance and reduce code size in 64-bit SPARC programs. The code model is determined by the following factors:
Positionability (absolute versus position-independent code)
Code size ( < 2 gigabytes)
Location (low, middle, anywhere in address space)
External object reference model (small or large)
The following table describes the different code models available for 64-bit SPARC programs.
Table 6-1 Code Model Descriptions
Code Model |
Positionability |
Code Size |
Location |
External Object Reference Model |
---|---|---|---|---|
abs32 |
Absolute |
< 2 gigabytes |
Low (low 32 bits of address space) |
None |
abs44 |
Absolute |
< 2 gigabytes |
Middle (low 44 bits of address space) |
None |
abs64 |
Absolute |
< 2 gigabytes |
Anywhere |
None |
pic |
PIC |
< 2 gigabytes |
Anywhere |
Small (<= 1024 external objects) |
PIC |
PIC |
< 2 gigabytes |
Anywhere |
Large (<= 2**29 external objects) |
Shorter instruction sequences can be achieved in some instances with the smaller code models. The number of instructions needed to do static data references in absolute code is the fewest for the abs32 code model and the most for the abs64 code model, while abs44 is in the middle. Likewise, the pic code model uses fewer instructions for static data references than the PIC code model. Consequently, the smaller code models can reduce the code size and perhaps improve the performance of programs that do not need the fuller functionality of the larger code models.
To specify which code model to use, the -xcode=<model> compiler option should be used. Currently, for 64-bit objects, the compiler uses the abs64 model by default. You can optimize your code by using the abs44 code model; you will use fewer instructions and still cover the 44-bit address space that the current UltraSPARC platforms support.
See the SPARC V9 ABI and compiler documentation for more information on code models.
A program compiled with the abs32 code model must be linked below 4 gigabytes using the -M /usr/lib/ld/sparcv9/map.below4G option.
The following interprocess communication (IPC) primitives continue to work between 64-bit and 32-bit processes:
The System V IPC primitives, such as shmop(2), semop(2), msgsnd(2)
mmap(2) on shared files
pipe(2) between processes
door_call(3DOOR) between processes
rpc(3NSL) between processes on the same or different machines using the external data representation described in xdr(3NSL)
Although all these primitives allow interprocess communication between 32-bit and 64-bit processes, you might need to take explicit steps to ensure that data being exchanged between processes is correctly interpreted by all of them. For example, two processes sharing data described by a C data structure containing variables of type long
cannot do so without understanding that a 32-bit process views this variable as a 4-byte quantity, while a 64-bit process views this variable as an 8-byte quantity.
One way to handle this difference is to ensure that the data has exactly the same size and meaning in both processes. Build the data structures using fixed-width types, such as int32_t
and int64_t
.
A family of derived types that mirrors the system derived types is available in <sys/types32.h>. These types possess the same sign and sizes as the fundamental types of the 32-bit system but are defined in such a way that the sizes are invariant between the ILP32 and LP64 compilation environments.
Sharing pointers between 32-bit and 64-bit processes is substantially more difficult. Obviously, pointer sizes are different, but more importantly, while there is a 64-bit integer quantity (long long
) in existing C usage, a 64-bit pointer has no equivalent in a 32-bit environment. In order for a 64-bit process to share data with a 32-bit process, the 32-bit process can only "see" up to 4 Gbytes of that shared data at a time.
The XDR routine xdr_long(3NSL) might seem to be a problem; however, it is still handled as a 32-bit quantity over the wire to be compatible with existing protocols. If the 64-bit version of the routine is asked to encode a long
value that does not fit into a 32-bit quantity, the encode operation fails.
64-bit binaries are stored in files in ELF64 format, which is a direct analog of the ELF32 format, except that most fields have grown to accommodate full 64-bit applications. ELF64 files can be read using elf(3ELF) APIs; for example, elf64_getehdr(3ELF).
Both 32-bit and 64-bit versions of the ELF library, libelf(), support both ELF32 and ELF64 formats and their corresponding APIs. This allows applications to build, read, or modify both file formats from either a 32-bit or a 64-bit system (though a 64-bit system is still required to execute a 64-bit program).
In addition, Solaris provides a set of GELF (Generic ELF) interfaces that allow the programmer to manipulate both formats using a single, common API. See elf(3ELF).
All of the system ELF utilities, including ar(1), nm(1), ld(1) and dump(1), have been updated to accept both ELF formats.
The /proc interfaces are available to both 32-bit and 64-bit applications. 32-bit applications can examine and control the state of other 32-bit applications. Thus, an existing 32-bit debugger can be used to debug a 32-bit application.
64-bit applications can examine and control 32-bit or 64-bit applications. However, 32-bit applications are unable to control 64-bit applications, because the 32-bit APIs do not allow the full state of 64-bit processes to be described. Thus, a 64-bit debugger is required to debug a 64-bit application.
The 64-bit version of the Solaris system is implemented using a 64-bit kernel. Applications that examine or modify the contents of the kernel directly must be converted to 64-bit applications and linked with the 64-bit version of libkvm().
Before doing this conversion and cleanup work, you should examine why the application needs to look directly at kernel data structures in the first place. It is possible that in the time since the program was first ported or created, additional interfaces have been made available on the Solaris platform, to extract the needed data with system calls. See sysinfo(2), kstat(3KSTAT), sysconf(3C), and proc(4) as the most common alternative APIs. If these interfaces can be used instead of libkvm(), use them and leave the application as 32-bit for maximum portability. As a further benefit, most of these APIs are probably faster and might not require the same security privileges needed to access kernel memory.
The 32-bit version of libkvm returns a failure from any attempt to use kvm_open(3KVM) on a 64-bit kernel or crash dump. Similarly, the 64-bit version of libkvm returns failure from any attempt to use kvm_open(3KVM) on a 32-bit kernel crash dump.
Because the kernel is a 64-bit program, applications that open /dev/ksyms to examine the kernel symbol table directly need to be enhanced to understand ELF64 format.
The ambiguity over whether the address argument to kvm_read(3KVM) or kvm_write(3KVM) is supposed to be a kernel address or a user address is even worse for 64-bit applications and kernel. All applications using libkvm that are still using kvm_read() and kvm_write() should transition to use the appropriate kvm_kread(3KVM), kvm_kwrite(3KVM), kvm_uread(3KVM) and kvm_uwrite(3KVM) routines. (These routines were first made available in Solaris 2.5.)
Applications that read /dev/kmem or /dev/mem directly can still run, though any attempt they make to interpret data they read from those devices might be wrong; data structure offsets and sizes are almost certainly different between 32-bit and 64-bit kernels.
The sizes of many kernel statistics are completely independent of whether the kernel is a 64-bit or 32-bit program. The data types exported by named kstats (see kstat(3KSTAT)) are self-describing, and export signed or unsigned, 32-bit or 64-bit counter data, appropriately tagged. Thus, applications using libkstat need not be made into 64-bit applications to work successfully with the 64-bit kernel.
If you are modifying a device driver that creates and maintains named kstats, you should try to keep the size of the statistics you export invariant between 32-bit and 64-bit kernels by using the fixed-width statistic types.
In the 64-bit environment, the stdio facility has been extended to allow more than 256 streams to be open simultaneously. The 32-bit stdio facility continues to have a 256 streams limit.
64-bit applications should not rely on having access to the members of the FILE data structure. Attempts to access private implementation-specific structure members directly can result in compilation errors. Existing 32-bit applications are unaffected by this change, but any direct usage of these structure members should be removed from all code.
The FILE structure has a long history, and a few applications have looked inside the structure to glean additional information about the state of the stream. Because the 64-bit version of the structure is now opaque, a new family of routines has been added to both 32-bit libc and 64-bit libc to allow the same state to be examined without depending on implementation internals. See, for example, __fbufsize(3C).
The following sections discuss advantages and disadvantages of 64-bit performance.
Arithmetic and logical operations on 64-bit quantities are more efficient.
Operations use full-register widths, the full-register set, and new instructions.
Parameter passing of 64-bit quantities is more efficient.
Parameter passing of small data structures and floating point quantities is more efficient.
64-bit applications require more stack space to hold the larger registers.
Applications have a bigger cache footprint from larger pointers.
64-bit applications do not run on 32-bit platforms.
The following sections discuss system call issues.
The EOVERFLOW return value is returned from a system call whenever one or more fields of the data structure used to pass information out of the kernel is too small to hold the value.
A number of 32-bit system calls now return EOVERFLOW when faced with large objects on the 64-bit kernel. While this was already true when dealing with large files, the fact that daddr_t
, dev_t
, time_t
, and its derivative types struct timeval
and timespec_t
now contain 64-bit quantities might allow more EOVERFLOW return values to be observed by 32-bit applications.
Some ioctl(2) calls have been rather poorly specified in the past. Unfortunately, ioctl() is completely insulated from compile-time type checking; therefore, it can be a source of bugs that are difficult to track down.
Consider two ioctl() calls--one that manipulates a pointer to a 32-bit quantity (IOP32), the other that manipulates a pointer to a long quantity (IOPLONG).
The following code sample works as part of a 32-bit application:
int a, d; long b; ... if (ioctl(d, IOP32, &b) == -1) return (errno); if (ioctl(d, IOPLONG, &a) == -1) return (errno);
Both ioctl(2) calls work correctly when this code fragment is compiled and run as part of a 32-bit application.
Both ioctl() calls also return success when this code fragment is compiled and run as a 64-bit application. However, neither ioctl() works correctly. The first ioctl() passes a container that is too big, and on a big-endian implementation, the kernel will copy in or copy out from the wrong part of the 64-bit word. Even on a little-endian implementation, the container probably contains stack garbage in the upper 32-bits. The second ioctl() will copy in or copy out too much, either reading an incorrect value, or corrupting adjacent variables on the user stack.