First, check to make sure that the file in question exists and is readable and that you have permission to write to it.
You can use ls /var/nis/trans.log to display the transaction log.
You can use nisls -l and niscat to check for existence, permissions, and readability.
You can use syslog to check for relevant messages.
The most likely cause of inability to truncate an existing log file for which you have the proper permissions is lack of disk space. (The checkpoint process first creates a duplicate temporary file of the log before truncating the log and then removing the temporary file. If there is not enough disk space for the temporary file, the checkpoint process cannot proceed.) Check your available disk space and free up additional space if necessary.