Whenever a process attempts to increase its memory size, it is subject to the memory limits (total and per-process) of the lnode to which it is attached. There are five means by which a process can attempt to increase its memory size:
Calling an allocation routine, such as malloc(3C), which results in an invocation of the sbrk(2) system call. If a memory limit is exceeded, the call will return an error with errno set to ENOMEM.
Expanding the stack and causing a stack fault, which would normally cause an extra page of memory to be given to the process. If a memory limit is exceeded, the process will be sent a SIGSEGV signal.
Using fork(2). The child address space is duplicated while it is still owned by the parent process. During duplication, the new address space will not exceed the memory.plimit since the parent must already be within this limit; however, the allocation is subject to the memory.limit.
Using exec(2). During an exec, memory usage first decreases as the old address space is discarded. However, if the address space of the new program is larger, and causes a limit to be exceeded, the exec can fail.