A custom function is a function written by a programmer in Java or .NET, conforming to a particular interface that makes them callable from a rulebase. Custom functions are a method of performing custom processing to return a calculated value, and are used when the required calculations cannot readily be performed with existing rulebase functions.
For a custom function to be available while writing and running your rulebase, it must be installed in a project folder at Development\Extensions\<extension name>. A file called extensions.xml contains the definition of the available custom functions, and the custom function itself is located within a \lib subfolder.
For full details on the structure of the custom function files, see the Oracle Policy Automation Developer's Guide.