ProcessRestartMemoryLimit

You can set this parameter at the individual server process level, or the domain level.

See ProcessRestartMemoryLimit.