test

Solaris 64-bit Developer's Guide

Chapter 6 Advanced Topics

This chapter presents a collection of miscellaneous programming topics for systems programmers who want to understand more about the 64-bit Solaris operating environment.

What's New for Applications

Most of the new features of the 64-bit environment are extensions of generic 32-bit interfaces, though several new features are unique to 64-bit environments.

64-bit: ABI Features

64-bit applications are described using Executable and Linking Format (ELF64), which allows large applications and large address spaces to be described completely.

ABI Features: SPARC V9

The SPARC Compliance Definition, Version 2.4, contains details of the SPARC V9 ABI. It describes the 32-bit SPARC V8 ABI and the 64-bit SPARC V9 ABI. You can obtain this document from SPARC International at www.sparc.com.

Following is a list of the SPARC V9 ABI features.

Stack Bias

An important feature of the SPARC V9 ABI for developers is the stack bias. For 64-bit SPARC programs, a stack bias of 2047 bytes must be added to both the frame pointer and the stack pointer to get to the actual data of the stack frame. See the following figure.

Graphic

For more information on stack bias, please see the SPARC V9 ABI.

Address Space Layout: SPARC V9

For 64-bit applications, the layout of the address space is closely related to that of 32-bit applications, though the starting address and addressing limits are radically different. Like SPARC V8, the SPARC V9 stack grows down from the top of the address space, while the heap extends the data segment from the bottom.

The diagram below shows the default address space provided to a 64-bit application. The regions of the address space marked as reserved might not be mapped by applications. These restrictions might be relaxed on future systems.

Graphic

The actual addresses in the figure above describe a particular implementation on a particular machine, and are given for illustrative purposes only.

Placement of Text and Data

By default, 64-bit programs are linked with a starting address of 0x100000000. The whole program is above 4 gigabytes, including its text, data, heap, stack, and shared libraries. This helps ensure that 64-bit programs are correct by making it so the program will fault in the lower 4 gigabytes of its address space, if it truncates any of its pointers.

While 64-bit programs are linked above 4 gigabytes, you can still link them below 4 gigabytes by using a linker mapfile and the -M option to the compiler or linker. A linker mapfile for linking a 64-bit SPARC program below 4 gigabytes is provided in /usr/lib/ld/sparcv9/map.below4G.

See the ld(1) linker man page for more information.

Code Models

Different code models are available from the compiler for different purposes to improve performance and reduce code size in 64-bit SPARC programs. The code model is determined by the following factors:

The following table describes the different code models available for 64-bit SPARC programs.

Table 6-1 Code Model Descriptions

Code Model 

Positionability 

Code Size 

Location 

External Object Reference Model 

abs32

Absolute 

< 2 gigabytes 

Low (low 32 bits of address space) 

None 

abs44

Absolute 

< 2 gigabytes 

Middle (low 44 bits of address space) 

None 

abs64

Absolute 

< 2 gigabytes 

Anywhere 

None 

pic

PIC 

< 2 gigabytes 

Anywhere 

Small (<= 1024 external objects) 

PIC

PIC 

< 2 gigabytes 

Anywhere 

Large (<= 2**29 external objects) 

Shorter instruction sequences can be achieved in some instances with the smaller code models. The number of instructions needed to do static data references in absolute code is the fewest for the abs32 code model and the most for the abs64 code model, while abs44 is in the middle. Likewise, the pic code model uses fewer instructions for static data references than the PIC code model. Consequently, the smaller code models can reduce the code size and perhaps improve the performance of programs that do not need the fuller functionality of the larger code models.

To specify which code model to use, the -xcode=<model> compiler option should be used. Currently, for 64-bit objects, the compiler uses the abs64 model by default. You can optimize your code by using the abs44 code model; you will use fewer instructions and still cover the 44-bit address space that the current UltraSPARC platforms support.

See the SPARC V9 ABI and compiler documentation for more information on code models.

Note -

A program compiled with the abs32 code model must be linked below 4 gigabytes using the -M /usr/lib/ld/sparcv9/map.below4G option.

Interprocess Communication

The following interprocess communication (IPC) primitives continue to work between 64-bit and 32-bit processes:

Although all these primitives allow interprocess communication between 32-bit and 64-bit processes, you might need to take explicit steps to ensure that data being exchanged between processes is correctly interpreted by all of them. For example, two processes sharing data described by a C data structure containing variables of type long cannot do so without understanding that a 32-bit process views this variable as a 4-byte quantity, while a 64-bit process views this variable as an 8-byte quantity.

One way to handle this difference is to ensure that the data has exactly the same size and meaning in both processes. Build the data structures using fixed-width types, such as int32_t and int64_t.

A family of derived types that mirrors the system derived types is available in <sys/types32.h>. These types possess the same sign and sizes as the fundamental types of the 32-bit system but are defined in such a way that the sizes are invariant between the ILP32 and LP64 compilation environments.

Sharing pointers between 32-bit and 64-bit processes is substantially more difficult. Obviously, pointer sizes are different, but more importantly, while there is a 64-bit integer quantity (long long) in existing C usage, a 64-bit pointer has no equivalent in a 32-bit environment. In order for a 64-bit process to share data with a 32-bit process, the 32-bit process can only "see" up to 4 Gbytes of that shared data at a time.

The XDR routine xdr_long(3NSL) might seem to be a problem; however, it is still handled as a 32-bit quantity over the wire to be compatible with existing protocols. If the 64-bit version of the routine is asked to encode a long value that does not fit into a 32-bit quantity, the encode operation fails.

ELF and System Generation Tools

64-bit binaries are stored in files in ELF64 format, which is a direct analog of the ELF32 format, except that most fields have grown to accommodate full 64-bit applications. ELF64 files can be read using elf(3ELF) APIs; for example, elf64_getehdr(3ELF).

Both 32-bit and 64-bit versions of the ELF library, libelf(), support both ELF32 and ELF64 formats and their corresponding APIs. This allows applications to build, read, or modify both file formats from either a 32-bit or a 64-bit system (though a 64-bit system is still required to execute a 64-bit program).

In addition, Solaris provides a set of GELF (Generic ELF) interfaces that allow the programmer to manipulate both formats using a single, common API. See elf(3ELF).

All of the system ELF utilities, including ar(1), nm(1), ld(1) and dump(1), have been updated to accept both ELF formats.

/proc

The /proc interfaces are available to both 32-bit and 64-bit applications. 32-bit applications can examine and control the state of other 32-bit applications. Thus, an existing 32-bit debugger can be used to debug a 32-bit application.

64-bit applications can examine and control 32-bit or 64-bit applications. However, 32-bit applications are unable to control 64-bit applications, because the 32-bit APIs do not allow the full state of 64-bit processes to be described. Thus, a 64-bit debugger is required to debug a 64-bit application.

libkvm and /dev/ksyms

The 64-bit version of the Solaris system is implemented using a 64-bit kernel. Applications that examine or modify the contents of the kernel directly must be converted to 64-bit applications and linked with the 64-bit version of libkvm().

Before doing this conversion and cleanup work, you should examine why the application needs to look directly at kernel data structures in the first place. It is possible that in the time since the program was first ported or created, additional interfaces have been made available on the Solaris platform, to extract the needed data with system calls. See sysinfo(2), kstat(3KSTAT), sysconf(3C), and proc(4) as the most common alternative APIs. If these interfaces can be used instead of libkvm(), use them and leave the application as 32-bit for maximum portability. As a further benefit, most of these APIs are probably faster and might not require the same security privileges needed to access kernel memory.

The 32-bit version of libkvm returns a failure from any attempt to use kvm_open(3KVM) on a 64-bit kernel or crash dump. Similarly, the 64-bit version of libkvm returns failure from any attempt to use kvm_open(3KVM) on a 32-bit kernel crash dump.

Because the kernel is a 64-bit program, applications that open /dev/ksyms to examine the kernel symbol table directly need to be enhanced to understand ELF64 format.

The ambiguity over whether the address argument to kvm_read(3KVM) or kvm_write(3KVM) is supposed to be a kernel address or a user address is even worse for 64-bit applications and kernel. All applications using libkvm that are still using kvm_read() and kvm_write() should transition to use the appropriate kvm_kread(3KVM), kvm_kwrite(3KVM), kvm_uread(3KVM) and kvm_uwrite(3KVM) routines. (These routines were first made available in Solaris 2.5.)

Applications that read /dev/kmem or /dev/mem directly can still run, though any attempt they make to interpret data they read from those devices might be wrong; data structure offsets and sizes are almost certainly different between 32-bit and 64-bit kernels.

libkstat

The sizes of many kernel statistics are completely independent of whether the kernel is a 64-bit or 32-bit program. The data types exported by named kstats (see kstat(3KSTAT)) are self-describing, and export signed or unsigned, 32-bit or 64-bit counter data, appropriately tagged. Thus, applications using libkstat need not be made into 64-bit applications to work successfully with the 64-bit kernel.

Note -

If you are modifying a device driver that creates and maintains named kstats, you should try to keep the size of the statistics you export invariant between 32-bit and 64-bit kernels by using the fixed-width statistic types.

Changes to stdio

In the 64-bit environment, the stdio facility has been extended to allow more than 256 streams to be open simultaneously. The 32-bit stdio facility continues to have a 256 streams limit.

64-bit applications should not rely on having access to the members of the FILE data structure. Attempts to access private implementation-specific structure members directly can result in compilation errors. Existing 32-bit applications are unaffected by this change, but any direct usage of these structure members should be removed from all code.

The FILE structure has a long history, and a few applications have looked inside the structure to glean additional information about the state of the stream. Because the 64-bit version of the structure is now opaque, a new family of routines has been added to both 32-bit libc and 64-bit libc to allow the same state to be examined without depending on implementation internals. See, for example, __fbufsize(3C).

Performance Issues

The following sections discuss advantages and disadvantages of 64-bit performance.

64-bit Application Advantages

64-bit Application Disadvantages

System Call Issues

The following sections discuss system call issues.

What Does EOVERFLOW Mean?

The EOVERFLOW return value is returned from a system call whenever one or more fields of the data structure used to pass information out of the kernel is too small to hold the value.

A number of 32-bit system calls now return EOVERFLOW when faced with large objects on the 64-bit kernel. While this was already true when dealing with large files, the fact that daddr_t, dev_t, time_t, and its derivative types struct timeval and timespec_t now contain 64-bit quantities might allow more EOVERFLOW return values to be observed by 32-bit applications.

Beware ioctl()

Some ioctl(2) calls have been rather poorly specified in the past. Unfortunately, ioctl() is completely insulated from compile-time type checking; therefore, it can be a source of bugs that are difficult to track down.

Consider two ioctl() calls--one that manipulates a pointer to a 32-bit quantity (IOP32), the other that manipulates a pointer to a long quantity (IOPLONG).

The following code sample works as part of a 32-bit application: 

	int a, d;
	long b;
	...
	if (ioctl(d, IOP32, &b) == -1)
			return (errno);
	if (ioctl(d, IOPLONG, &a) == -1)
			return (errno);

Both ioctl(2) calls work correctly when this code fragment is compiled and run as part of a 32-bit application.

Both ioctl() calls also return success when this code fragment is compiled and run as a 64-bit application. However, neither ioctl() works correctly. The first ioctl() passes a container that is too big, and on a big-endian implementation, the kernel will copy in or copy out from the wrong part of the 64-bit word. Even on a little-endian implementation, the container probably contains stack garbage in the upper 32-bits. The second ioctl() will copy in or copy out too much, either reading an incorrect value, or corrupting adjacent variables on the user stack.