Debugging a Program With dbx

Working Around the Eight Megabyte Limit

Note -

On V9, you can work around the eight megabyte limit by using the setenv command to set the USE_FASTTRAPS environment variable to 1. This workaround makes dbx run more slowly and use more memory.

dbx provides some possible workarounds to users who have run into this limit. These workarounds require the use of a utility called rtc_patch_area which is included with Sun WorkShop.

This utility creates object files or shared object files that can be linked into your program to create patch areas for RTC to use.

There are two situations that can prevent dbx from finding patch areas within 8 megabytes of all the loads and stores in an executable image:

Case 1: The statically linked a.out file is too large.

Case 2: One or more dynamically linked shared libraries is too large.

When dbx runs into this limitation, it prints a message telling how much patch space it needs and directs you to the appropriate case (Case 1 or Case 2).

Case 1

In Case 1, you can use rtc_patch_area to make one or more object files to serve as patch areas and link them into the a.out.After you have seen a message like the following:

Enabling Error Checking... dbx: warning: rtc: cannot find patch space within 8Mb (need 6490432 bytes for ./a.out)
dbx: patch area too far (8Mb limitation); Access checking disabled
         (See `help rtc8M', case 1)

Create an object file patch.o for a patch area with a size less than or equal to 8 megabytes:
```
rtc_patch_area -o patch.o -size 6490432
```
The -size flag is optional; the default value is 8000000.

If the size request from the error message is satisfied, continue to the next step. Otherwise, repeat step 1 and create more .o files as needed.

Relink the a.out, adding the patch.o files to the link line.

Try RTC again with the new binary.

If RTC still fails, you may try to reposition the patch.o files on the link line.

An alternate workaround is to divide the a.out into a smaller a.out and shared libraries.

Case 2

If you are running RTC on an attached process, see case 2a. Otherwise, the only workaround is to rebuild the shared library with extra patch space:

After you have seen a message like the following::

Enabling Error Checking... dbx: warning: rtc: cannot find patch space within 8Mb (need 563332 bytes for ./sh1.so)
dbx: patch area too far (8Mb limitation); Access checking disabled
         (See `help rtc8M', case 2)

Create an object file patch.o for a patch area with a size less than or equal to 8 megabytes:
```
rtc_patch_area -o patch.o -size 563332
```
The -size flag is optional; the default value is 8000000.

If the size request from the error message is satisfied, continue to the next step. Otherwise, repeat step 1 and create more .o files as needed.

Relink sh1.so, adding the patch.o files to the link line.

Try RTC again with the new binary; if dbx requests patch space for another shared library, repeat steps 1-3 for that library.

Case 2a

In Case 2a, if the shared library patch space requested by dbx is eight megabytes or less, you can use rtc_patch_area to make a shared library to serve as a patch area and link it into the a.out.

After you have seen a message like the following:

Enabling Error Checking... dbx: warning: rtc: cannot find patch space within 8Mb (need 563332 bytes for ./sh1.so)
dbx: patch area too far (8Mb limitation); Access checking disabled
         (See `help rtc8M', case 2)

Create a shared object patch1.so for a patch area:
```
rtc_patch_area -so patch1.so -size 563332
```
The -size flag is optional; the default value is 8000000.

2) Relink the program with patch1.so placed adjacent to sh1.so in the link line. If dbx still requests patch space for the same sh1.so, then try placing patch1.so immediately before sh1.so.

It may be necessary to use full pathnames instead of the ld -l option to get the desired shared library ordering.

Try RTC again with the new binary; if dbx requests patch space for another shared library, repeat steps 1-2 for that library.

If the patch space requested by dbx is more than eight megabytes for a given shared library, follow the steps in case 2 above.