C H A P T E R  5

Helpful Testing and Debugging Hints

This chapter contains information to consider when you are designing FCode source code for PCI. It includes the following sections:


Accessing a PCI Device's Configuration Space Registers

It is not necessary to do anything extra to access your device's configuration space registers. They are always accessible.


Base Address Register Setting

The base address registers in the configuration space are set by the CPU PROM.

The CPU PROM (not the PCI card's FCode PROM) allocates the base address for memory and/or I/O space on your PCI device and for the FCode PROM.


System Cache Line Size

To determine the system's cache line size from the FCode to write into the cache-line-size configuration space register of your PCI device, look in the cache-line-size register in the configuration space; it refers to the cache line size supported by the PCI device.


Sun Ultra 30 UPA/PCI-Related Nodes

The PCI-related nodes on the Sun Ultra 30 UPA/PCI system are /pci@1f,4000 and /pci@1f,2000. pcia and pcib as needed for the NVRAM variables pcia-probe-list and pcib-probe-list are determined in the following manner.

Each PCI bus has a property named "slot-names" which gives information about slots on that PCI bus. It could sometimes indicate which NVRAM variable corresponds to it.

To get a human-readable value for that property, do the following:


ok " </pci-bus-node>" select-dev
ok " slot-names" get-my-property drop decode-int .h cr  type 

For example, for a PCI bus at /pci@1f,2000, typing:


ok " /pci@1f,2000" select-dev
ok " slot-names" get-my-property drop decode-int .h cr  type 

will display something like:


6 
pcia slot 1pcia slot 2

This is an indication that devices under /pci@1f,2000 relate to pcia.

In a Sun Ultra 30 UPA/PCI system with 4 plug-in PCI slots, only slot 1 is physically present for pci@1f,2000. It can also support 66 MHz, 64-bit PCI devices.


ok " /pci@1f,4000" select-dev
ok " slot-names" get-my-property drop decode-int .h cr  type 
34
pcib slot 2pcib slot 4pcib slot 5

This is an indication that devices under /pci@1f,4000 relate to pcib.

In a Sun Ultra 30 UPA/PCI system with 4 plug-in PCI slots:

Note that the value of the slot-names property differs for different systems. Some systems may not indicate which PCI bus is which by the value of the "slot-names" property.

Also in different releases of the PROM for the same system, the value of the "slot-names" property may change. You may need to refer to the system documentation for details about PCI buses on the system.

Alternatively, you can find which NVRAM variable refers to which PCI bus by setting the NVRAM variables to different values or by plugging PCI card(s) in different slot(s).


Finding and Using Physical Addresses

To find and use physical addresses to access, for instance, configuration space registers on a Sun Ultra 30 UPA/PCI system, bypass the MMU with the choice of the correct ASI space. The arguments for the space {c,d,w,l,x} command include an address and ASI code (and data for a write operation).

On Sun Ultra 30 UPA/PCI systems, a PCI device's configuration registers are viewed using the following address:

1fe.0100.0000 + X

where the 32-bit value of X is represented in bit format as:

0000.0000.bbbb.bbbb.dddd.dfff.rrrr.rrrr 

Also:

So, if the bus number is 81, device number is 0, and function number is 1, then X will be 81.0100, giving you a configuration register base.

You'll access the 0th configuration register at 1fe.0181.0100 (physical address). On Sun Ultra 30 UPA/PCI systems, you can use ASI 0x15 for a noncacheable address being accessed by MMU bypass. If you are accessing a little-endian device, use ASI 0x1d.

You can get bus number, device number, and function number from my-space after selecting that device or from the "reg" property value for that device. Look in IEEE 1275/PCI binding for the "reg" property format.

In general, for any system, use the map-in command to get physical addresses for registers in any space (configuration space, 32-bit memory space, and others). map-in requires the phys.lo, phys.mid, phys.hi, and length arguments. phys.lo, phys.mid, and phys.hi values can be taken from the corresponding "reg" property.

In the case of configuration space, getting the physical address is easy since phys.lo and phys.mid are always zero. phys.hi is just the configuration space address.

Here is an example of getting the physical address of the configuration space registers, using on-board Ethernet on a Sun Ultra 30 UPA/PCI system:


ok " /pci@1f,4000/network@1,1" begin-select-dev
ok pwd
/pci@1f,4000/network@1,1
ok .properties
.
.
reg  (Config Space ---->)00000900 00000000 00000000 00000000 00000000      
(32bit memory space ---->) 02000910 00000000 00000000 00000000 00007020
.
.
.
ok 0 0 900 100 " map-in" $call-parent constant my-cfg-vaddr
ok my-cfg-vaddr . fff80900
ok my-cfg-vaddr map?  VA:fff80900
G:0 W:1 P:1 E:1 CV:0 CP:0 L:0 Soft1:1 PA[40:13]:ff00800 PA:1fe01000000 
Diag:0 Soft2:0 IE:0 NFO:0 Size:0 V:1
PA:1fe01000900 
 

Hence the physical address for the base of configuration registers is 1fe.0100.0900 for this device. For plug-in PCI devices, the registers' physical address may vary if the device is plugged into a different slot or if other devices are present. Similarly, using the "reg" entry for memory or I/O space, you can find a physical address for those spaces.


Controlling PCI Slot Probing on an Ultra 30 UPA/PCI System

You can control probing of PCI slots on your Sun Ultra 30 UPA/PCI system as follows. On Sun Ultra 30 UPA/PCI systems, during normal system initialization, there are NVRAM variables which indicate to the CPU PROM what slots to probe and in what order. On the Sun Ultra 30 UPA/PCI system they are: pcia-probe-list and pcib-probe-list. The default value for pcia-probe-list is 1,2; for pcib-probe-list, it is 3,2,4,5. To disable slot 4 probing on pcib, during normal initialization after a reset, change pcib-probe-list to:


ok setenv pcib-probe-list 3,2,5

After a reset, to probe slot 4 on pcib manually:


ok 4 probe-pci-slot /pci@1f,4000

Note that not all CPU PROMs have the probe-pci-slot command. Also, in future PROMs, behavior of this command may change, including the possibility of its deletion.


Using 3.x Tokenizer and 3.x CPU PROMs

Here are some points to consider while using the 3.x tokenizer: While you are testing FCode under CPU OpenBoot PROM 3.x versions, make sure that you have OpenBoot PROMs version 3.1 or later. Pre-3.1 PROMs will need the following NVRAM patch:


ok nvedit
0:: nl-move( src dst len -- ) rot n->l rot n->l rot n->l (move); 
1: ['] nl-move is move
2: ['] l>>a 2 la+ dup l@ h# 1000 invert and swap l!
3: ['] lrshift 2 la+ dup l@ h# 1000 invert and swap l!
4: ^C
ok nvstore
ok setenv use-nvramrc? true
ok reset-all

Also note that, while using the 2.x or 3.x tokenizer, literals or numbers that have bit 31 set to 1 will extend this bit (1) to bit 63 on 3.x CPU PROMs. For example, typing 8000.0000 constant xxx will in reality be giving a value as ffff.ffff.8000.0000. When such words or constants are used in address manipulation or otherwise, your code should clip them to a 32-bit value:

To return only 8000.0000, type:

ff ff ff ff bljoin constant x-num
: clip-num ( n -- l )  x-num and ; 
8000.0000 clip-num constant  xxx 

or use xxx clip-num wherever "xxx" is being used.


PCI Device Configuration Register Access

To find the address to use for configuration register access on your PCI device, look in the format for the physical address of the "reg" property. Use the phys.hi cell of the first entry in the "reg" property as the base address for the configuration space. The first entry in the "reg" property must be the configuration space entry (bbbb.bbbb.dddd.dfff.0000.0000 binary). Using this or any other method, obtain the values of bbbb.bbbb, ddddd and fff for your device. Then use:


ok "< parent-pci-bus-node>" select-dev
ok <bbbb.bbbb.dddd.dfff>XX config-l@

Where XX is the offset for that register configuration. For example, if the bus number is 1000.0001 (0x81), the device number is 0.0000, and the function number is 001 (0x01), then use:


ok " /pci@1f,2000" select-dev
ok 81.0100 config-l@ ( to read device id and vendor id)
ok 81.0104 config-w@  ( to read command register )
ok 81.0130 config-l@  ( to read the expansion PROM base address  register)


Boot Software Roles

Three types of software are involved during a boot: the kernel, FCode, and the operating environment driver. This section describes the normal Solaris operating environment boot scenario, including the functions of each and the order in which they begin.

At power on, the CPU PROM begins execution. It probes all on-board devices and plug-in cards, thus interpreting the FCodes on all FCode PROMs. In the FCode probing process, FCode PROMs generate properties for devices. Some FCode PROMs execute commands to reset the device and perform other initialization.

Then, the CPU PROM boots over the specified boot device (using its FCode boot driver), loads bootblk (or inetboot for network booting), and passes control to the bootblk code. Then, the bootblk code loads the kernel and modules, and passes control to the kernel. The kernel at some point starts to use the operating environment's device driver.

So, the order is:


Enabling Access to a PCI Device's Memory Space Locations

If you are loading FCode and can't access memory space locations, how do you go about enabling access to memory space locations for your PCI device?

Look in the format for the physical address of the reg property. Using that (or any other method), obtain the values of bbbb.bbbb, ddddd and fff for your device. Then use:


ok " <parent-pci-bus-node>" select-dev
ok 3 <bbbb.bbbb.dddd.dfff>04 config-w!

This will write to the configuration space command register and thus enable access to memory and I/O space. This sets bit[0] and bit[1] of the command register. In the same way, you may set other bits in the command register if needed by your application. If the bus number is 1000.0001 (0x81), the device number is 0.0000, and the function number is 001 (0x01), you will then use:


ok " /pci@1f,2000" select-dev
ok 81.0104 config-w@ 3 or 81.0104 config-w!

Normally, your FCode driver's open routine should enable such access. FCode can use the value returned by my-space and add an offset of 4 to get the address of the command register. Then it can set various bits in the command register to enable the desired access. The close routine should disable that access.


Expansion FCode PROM

If you are unable to access your expansion FCode PROM, how can access to it be enabled?

To enable access, look in the format for the physical address of the "reg" property. Using that (or any other method), obtain the values of bbbb.bbbb, ddddd and fff for your device. Then use:


ok "< parent-pci-bus-node>" select-dev
ok <bbbb.bbbb.dddd.dfff>04 config-w@ 3 or  <bbbb.bbbb.dddd.dfff>04 config-w!
ok <bbbb.bbbb.dddd.dfff>30 config-l@ 1 or  <bbbb.bbbb.dddd.dfff>30 config-l!

This will first enable memory and I/O space access. Then, it will read the value from the expansion PROM configuration space base address register (at offset 0x30) or 1 to it, and write the value in the expansion PROM base address register to enable access to your FCode PROM. This sets bit[0] of the expansion PROM base address register. In other words, if the bus number is 1000.0001 (0x81), the device number is 00000, and the function number is 001 (0x01), then use:


ok " /pci@1f,2000" select-dev
ok 81.0104 config-w@ 3 or 81.0104 config-w!
ok 81.0130 config-l@ 1 or 81.0130 config-l!

If for any reason the FCode needs to access PROM data (for example, to access Vital Product Data stored in the PROM), then the FCode should enable PROM access by using the value returned by my-space and adding an offset of 0x30 as the register address. The FCode should read the value from the address, or the value with 1, and write the result back to that address.

Also, since the FCode would have been copied in memory, the devices' memory and I/O spaces may not be enabled. So, the FCode must enable them, using:


ok my-space h# 30 + dup config-l@ 1 or swap config-l! ( enable PROM access)  
ok my-space h# 4 + dup config-w@ 3 or swap config-w!  ( enable I/O, memory access)

Using the example above, you can disable expansion PROM access as:


ok my-space h# 30 + dup config-l@ 1 invert and swap config-l! ( disable PROM access)  


Packaging Error With Ethernet FCode

In trying to load the FCode from Ethernet, the code seems to load without errors; however, when you try to build the package, you get an error:


ok 4000 dload /stand/cheerio.o
Boot device: /pci@1f,4000/network@1,1:,|stand|cheerio.o  File and args: 
ok 0 0 " 0,1" " /pci@1f,2000" begin-package
ok 4000 1 byte-load
Unimplemented FCode token before address 4004
Warning: FCode sequence resulted in a net stack depth change of 1   
ok 

The error may be due to either of two causes:

Solution: Dump the download image beginning at 4000. For example:


4000 60 dump

and see where f1 or fd starts. It is the beginning of the FCode data for the byte-load. For instance, if the FCode data starts at X, use the address X in


ok X 1 byte-load

f1 or fd is the beginning of the FCode header, 8 bytes long, as:

f1, <reserved byte>,<2 reserved bytes>,<4 bytes of FCode length>


Note - If you begin your FCode source with fcode-version1, the first FCode data is fd, but if you use fcode-version2 or fcode-version3, the first FCode data is f1.



Solution: Change your FCode to handle two numbers returned from my-address.

One way to do this is:


my-address      constant my-bus-addr-mid  constant my-bus-addr-low
 
: my-bus-addr ( -- paddr.low paddr.mid ) 
my-bus-addr-low my-bus-addr-mid
;

Then use my-bus-addr in the creation of the "reg" property.