First, log in to the server as root and see if the command rscadm status reports a good status. If it does, then RSC is working. Log in to the server as root and use the rscadm show command to check that serial port configuration variables are set correctly.
If the problem is with a modem:
Log in to the server as root and use the command rscadm modem_setup to check and verify modem configuration. It is very important that the modem is configured to use a fixed baud rate for incoming calls.
Verify that the modem is working properly by attaching it to another computer.
Attach a terminal or another system to the serial port and verify that the serial port is working. Set the configuration variable serial_hw_handshake false and then reset RSC before trying this.
Run SunVTS to check the serial port; the external serial port test requires a serial loopback connector.
Run SunVTS to check the SSP and paddle card.