For more information, see Using Your Assigned Administrative Rights in Securing Users and Processes in Oracle Solaris 11.4.
rcap.max-rss=value
The following line in the /etc/project file sets an RSS cap of 10 GBytes for a project named db.
db:100::db,root::rcap.max-rss=10737418240
Note that the system might round the specified cap value to a page size.