To install Messaging Server in a non-global sparse root zone, you first need to install/upgrade the applicable OS patches and shared components in the global zone. You are unable to do so in the sparse root zone, because the /usr directory (where the shared components reside) is a read-only directory in the sparse root zone.
Follow pre-installation requirements as described in Preparing for Installation.
Verify that you are about to install the shared components and OS patches in the global zone and not the sparse root zone.
Run the installer in the global zone and only install/upgrade the OS patches and the Shared Components:
commpkg install -upgradeSC y
See the Installing Shared Components.
Do not install Messaging Server in the global zone. To do this, add a ~ (tilde) to the component number you want to install in the sparse zone. For example, if you plan to install Messaging Server 64–bit in the sparse zone, you select ~3 during the global zone installation. The installer will know to only install dependencies and not the product itself.
Once you have the shared components and OS patches installed, install Messaging Server in the sparse root zone by following the steps in the To Begin Installationsection.