10. Contingency Plan

The managers should determine the operational risks and develop a procedure to ensure that the business is not disrupted in an emergency situation.

The people responsible for developing a contingency plan for the bank are:

• The Contingency Planning Officer-for the overall branch plan
• The Data center manager-for technology contingency, including communications, hardware, system software and third party software
• The Senior Operations Manager of the bank, who is responsible for Oracle FLEXCUBE
• The members of the Internal Control department

The Contingency plan should address the following issues:

• Operation Risk assessment
• Contingency planning
• Software errors outside of normal working hours
• Contingency plan distribution list
• Persons who can authorize the emergency procedure
• Contact points in the event of hardware and software problems
• Potential exposure and containment measures
• Emergency back-up plan

10.1 Operation Risk Assessment

Oracle FLEXCUBE runs on Client Server Architecture, with ORACLE FORMS on the client side and ORACLE 7.3 (with or without distribution option) on the Server (UNIX). The Database can either be centralized in a single server or distributed in more than one server connected through telecommunication network.

There are some risks involved in the implementation of a new system and the maintenance of an existing one. Some risks are controllable while others are not. But the degree of non-controllable risks, such as natural disasters, can be minimized. This chapter deals with risks and their corresponding protective measures. The risk analysis is geared toward the security of hardware and software. Tight security and back-up systems are the most important elements. Adequate training for the personnel who will be dealing with the computer is also very important. The basic contents of the risk assessment are as follows:

• Brief Description of Location and Operations.
• Major Causes of the Operational Risks.

This section contains the following topics

• Section 10.1.1, "Brief Description of Location and Operations"
• Section 10.1.2, "Major Causes of the Operational Risks"

10.1.1 Brief Description of Location and Operations

A brief description of the location and the operations of the bank should be indicated in the assessment.

10.1.2 Major Causes of the Operational Risks

The following type of critical factors, which present operational risks, are to be considered:

• Political or Civil Unrest
  • Political or Civil disturbances
  • Strikes and Riots
  • Insurrection etc

• People Related Risks (Internal)
  • Illness or Injury
  • Non-adherence to established procedures
  • Shortage of training in established procedures
  • Unpleasant working conditions
  • Deliberate or Negligent acts

The risk involved can be loss of customers, fraud, processing errors, delays which include information modification, loss of information, data omission, damages to hardware, etc.

• People Related Risks (External)
  • Intrusion
  • Kidnapping or extortion
  • Theft of equipment, etc

• Utility Related Risk
  • Electricity
  • Communication
  • Voltage stabilizer
  • Air Conditioning etc

Power loss causes processing errors and delays, and a total memory loss in computers. Irregular or faulty power lines can alter the data being processed and/or cause permanent damage to the computer.

• Neighbourhood Hazards
  • Proximity to chemical or explosive operations
  • Nearby building or floor that constitutes a fire hazard to the operation.
  • Potential risk of leakage or burst in the water pipes on the premises.
  • High crime areas.

10.2 Contingency Planning

The Data Center is a supporting unit that provides the Data Processing services for the contingency planning process. These services are in the area of the Data Center

This is considered to be a medium risk area and its disruption implies that the bank, Customer Service, Trade and Treasury are rendered inoperable with a resultant loss of the bank s reputation and customers. Potential threats to the Data Center (and consequently to the other units of Operations) include:

• People related risk, e.g. illness, strike, sabotage
• Natural disasters, e.g. fire, power failure, etc. rendering the premises and facilities unavailable
• Equipment failure, e.g. failure of Main or Back-up Hardware, peripherals, workstations, printers, etc.
• Software Failure, e.g. abnormal termination of a program due to a database corruption or program bug
• Support Line failure, e.g. failure of the link between the Oracle Financial Services Support team and the bank

The above risk factors should be analyzed and preventive measures should be described in detail. Some of the recommended preventive measures are:

• Employees should be cross-trained at different jobs so that they can act as a back-up for each other in case of absence due to illness, etc.
• Regular medical check-ups should be arranged for the employees
• Fire detection and suppression equipment, e.g. smoke detectors, fire extinguishers, Halon, etc. should be placed at all vulnerable locations
• Main and Back-up Hardware systems, printers and workstations should be serviced and maintained regularly
• Service contract - The bank should evaluate the need for a maintenance contract and document for hardware and software
• A private, leased data-line should exist between Oracle Financial Services (formerly i-flex) and the bank. A dial-up facility should also exist as a back-up to access Oracle Financial Services (formerly i-flex)

The contingency plan should be drawn to cover those functions critical to the business of the bank. The contingency plan should contain personnel lists (organization chart, distribution list, authorizers of emergency measures, succession list and alternate account managers), action plans, contact names and addresses and contingency planning documentation.

10.3 Software Errors Outside Normal working Hours

In case of a problem arising outside normal working hours, e.g. during the End of Day, month- end, year- end runs, etc., the following is the recommended emergency procedure:

• The Systems Manager or the back-up person should be summoned if not already available on the premises. No action should be taken until the Systems Manager or the back-up person arrives.
• The Senior Operations Manager of the bank should be informed about the matter.
• If the problem has occurred earlier and if the method of resolution is known, it should be applied (all occurrences of problems and their resolution should be entered in the Data Center software logbook). The Systems Manager should ensure that the program overlay and the statement number are exactly the same as in the logbook.
• The Senior Operations Manager of the Bank and the Internal Control Head may be asked to come to the bank to allow the Systems Manager to enter in the SUPERVISOR mode and correct the problem.
• If the Senior Operations Manager of the bank and the Internal Control Head are not available, the Systems Manager should contact the respective back-ups to gain access to the supervisor mode. Secret passwords to the supervisor mode are lodged in the main vault and may be withdrawn and opened if the regular and alternate custodians are unavailable.
• If the error happens for the first time and the bank has a technical person for Oracle FLEXCUBE maintenance, the Systems Manager should consult the person, identify the bug and fix it as per procedure. If this is not possible, Oracle Financial Services should be contacted for instructions. When the bug is fixed, recovery programs 100 and 101 should be run.
• If the error happens for the first time and the bank has a maintenance and support agreement with Oracle Financial Services the Systems Manager should report/handle the problem as per the support procedure in the support Manual. The bank should take action as per the instructions of Oracle Financial Services
• In all cases, details of the events (error description, communications with Oracle Financial Services etc. should be carefully noted in the Logbook of the Data Center and reviewed by the Internal Control Unit.

This section contains the following topics

• Section 10.3.1, "Contingency Plan Distribution List"
• Section 10.3.2, "People who can Authorize Emergency Measures"
• Section 10.3.3, "Contact Points in Case of Hardware and Software Problems"

10.3.1 Contingency Plan Distribution List

The recommended contingency plan distribution list is as follows.

• The Senior Management of the branch
• The Manager of the Data Center and the Systems Manager of the bank
• The Audit Division of the bank
• The Senior Operations Manager of the bank
• The System Administrative Manager
• The Control Head of the bank
• The head of the Liability Management Group
• The Manager of the Off-site Back-up location

The contingency plan distribution list should be finalized by the senior management of the bank.

10.3.2 People who can Authorize Emergency Measures

A list should be prepared, giving the following information about the officers who can authorize emergency measures:

• Name and function of the officer
• Telephone extension number in the office
• Direct telephone number (if available) in the office
• Residential telephone number

10.3.3 Contact Points in Case of Hardware and Software Problems

The following information for contact points in the event of Hardware and Software Problems should be listed:

• Name address of the Organization to be contacted
• Contact person, office telephone number and residential telephone numbers
• AX/Telex numbers of the Branch or Organization
• Port Numbers of the Oracle Financial Services Support division
• Dialing codes for the country

10.4 Potential for Exposure and Containment Measures

The Contingency Plan should clearly describe the Safety, Preventive and Containment measures. It should include guidelines and the procedures to be followed in case of an emergency related to Oracle FLEXCUBE.

This section contains the following topics

• Section 10.4.1, "Hardware Failure"
• Section 10.4.2, "Containment Measures"
• Section 10.4.3, "Disk Drive Failure"
• Section 10.4.4, "Remote Branch Communications/Modem Failure"
• Section 10.4.5, "Software Failure"
• Section 10.4.6, "Utilities Failure"

10.4.1 Hardware Failure

Hardware failure can be caused by the following situations:

• Failure of the Main machine
• Failure of the Back-up machine
• Failure of the Main and Back-up machines
• Failure of the Disk Drive
• Failure of a Terminal
• Failure of the Remote Branch terminal
• Failure of the Remote Branch communication or Modem
• Failure of the Printer

10.4.2 Containment Measures

Main machine failure

When failure of the main machine has an immediate effect on the processing, the following steps should be taken:

• The Manager of the Data Center should inform the user departments about the situation and the time required to switch on the back-up system.
• The matter should be reported to the Senior Operations Manager of the bank.
• The Manager of the Data Center should establish the cause of the problem before switching on the Back-up system.
• The Manager of the Data Center should connect all the workstations and peripherals to the back-up system and inform all the departments.
• Contact the local dealers for immediate repair.
• Record all information about the problem in the Trouble Report Log, The details to be included are nature of the problem, time of occurrence and name of the person reporting the problem. The time of contacting the Customer Engineer and the arrival of the Customer Engineer should also be recorded.
• After consultation with Customer Engineer determine the time required to get the main system running. If the repairs are expected to extend beyond 24 hours and the off-site back-up facility exists, inform the off-site back-up location of a possible need for use of their system.
• The following reports should be generated at the end of each day, to provide for an emergency. The reports will be useful if the Back-up machine fails before the Main machine is repaired.
  • Accounting Journal
  • Trial Balance
  • General Ledger
  • Currency Position
  • Limits Reports
  • Accrual reports
  • End of Day Account/GL Balances

Produce the accrual reports to avoid manual calculations. This program should only be run against a copy of data.

Back-up machine failure

If the Back-up machine is inoperative, the situation does not affect the normal operations, but it requires immediate repair to avoid a complete breakdown. The following steps should be carried out:

• The Senior Operations Manager of the bank should be apprised of the situation. Since the failure of the back-up machine does not disrupt the normal operations, it is not necessary to inform the user departments.
• Contact the local dealers for immediate repair.
• Enter all relevant information about the problem in the Trouble Report Log. The details should include the nature of the problem, the time of occurrence and the name of the person who reported the problem. The time of contacting the Customer Engineer and the arrival of the Customer Engineer should also be recorded.
• After consultation with the Customer Engineer determine the time required to get the back-up system running. If the repairs are expected to extend beyond 24 hours and if an off-site back-up facility exists, inform the off-site location of a possible need to use their system.
• If repairs are expected to take more than one day, the key reports (as outlined above, under Main Machine Failure), should be generated. The reports will be useful if the Main Machine fails before the back-up machine is repaired.

Failure of Main and Back-up Machines

The failure of both the Main and the Back-up machines is considered an unlikely event. The stationery required for manual operations, should be stored off-site. In such a situation the following steps should be taken:

• The Manager of the Data Center should inform all the user departments of the situation.
• Report to the Senior Operations Manager of the bank.
• Contact the local dealers for immediate repair.
• Record all relevant information in the Trouble Report Log. The details should include the nature of the problem, the time of occurrence and the name of the person who reported it. The time of contacting the Customer Engineer and the arrival of the Customer Engineer should also be recorded.
• After consultation with the Customer Engineer, determine the time required to get at least one machine running. If the repairs are expected to extend beyond 8 hours and the off-site back-up facility exists then inform the off-site back-up location, of a need to use their system. The necessary equipment should be made available.
• If the Senior Operations Manager of the bank decides to go off-site to continue processing, the following steps are recommended:
  • The Data Center Manager should ensure that all the required material on the checklist is ready to be taken to the back-up site.
  • The heads of all the departments should arrange to collect the input media, Database maintenance forms and tickets from the users and advise the systems department of completion.
  • The Data Center Manager should approve of the material checklist before the materials can be transported to the off-site back-up facility under proper custody. The Data Center Manager should monitor the whole process and if any problems are encountered they should be reported to the Senior Branch Operations Manager.

At the back-up site, Oracle FLEXCUBE Database should be restored from the latest back-up tapes from the on site library (ideally, the backup of post end of previous day). All the daily transactions should be input if necessary and EOD should be run. Extra copies of reports should be prepared where necessary. The bank should use the latest daily central liability printouts (stored off-site) to pay checks over the counter and maintain manual ledger to record all movements.

In case both the machines break down in the middle of business hours, earlier payments should be taken into account from daily printouts (last business day central liability report) before further payments are made.

The Senior Branch Operations Manager should use discretion in determining the specific operation needs. The process detailed below provides a guideline for manual processing. This can be followed if the time to get at least one system up is likely to exceed 2 working days.

• The Processing priorities should be as follows:
  • Clearing Operations
  • Teller transactions
  • Trading Exchange/Transactions
  • Import/Export Operations
  • Funds Transfers
  • Deposits
  • Credit/Financial Control processing

• The following reports printed and kept off-site will assist in manual operations.
  • Limit Reports
  • Account Balances Report

• Balance controlling should be centralized with the Referral Clerk or person designated by the Senior Branch Operations Manager.
• Adequate stationery for departmental proofs, tickets, ledgers and manifolds, should be maintained to facilitate manual processing.
• Current Account balances should be updated manually.
• For all accounts having movements, a run-up of old balances, debits, credits and new balances should be taken.
• Accounting entry should be done through departmental proofs and tickets. For this purpose the trial balance and the Currency Position of the previous day should be used. Ledgers should be maintained, showing balance B/F, debits and credits movements and new balance.
• The Control department should comprehensively recheck all movements through source documents and tickets. Control should ensure that all source documents and tickets are microfilmed.

10.4.3 Disk Drive Failure

Main Machine Disk Drive Failure

This does not constitute a critical issue. If the disk fails the processing should be switched on to the back-up machine as per the recommended procedure described above in Main machine failure. The main machine equipment should be repaired/replaced immediately.

Back-up Machine Disk Drive failure

Although there is no impact on the operation when the disk drive fails on the back-up machine, urgent repairs/replacements must be ensured in order to bring the probability of general system failure back to the desired level.

Both Main and Back-up machines Disk Drive failure

Failure of both Main and Back-up machines Disk Drives constitutes a critical situation and is considered an unlikely event. Should the situation arise the branch should follow the recommended procedure described above in situation where both main and back-up machine failure occurs simultaneously.

Note

If the disk drive on the main machine is physically scratched or damaged by any means, the latest back-up tape should be loaded on the back-up machine and re-input or re- pro­cessing has to be done.

Terminal Failure

Any malfunctioning terminal can be replaced by the Data Center staff or Hardware Engineer with an available terminal. The failure of one or more is not critical to Branch operations. The recommended steps to be followed in case of terminal failure are given below:

• Any malfunction of a terminal should be notified to the Data Center Manager who should perform first level of trouble shooting.
• The terminal should be repaired or replaced by the local vendors as soon as possible.

Remote Branch Terminal Failure

Any malfunctioning remote branch terminal should be repaired or replaced by the Hardware Engineer with the available terminal at the branch. The failure of one or more terminals is not critical to remote Branch operations if some remote terminals are in working order. The recommended steps to be followed in case of failure of all the remote terminals, are given below:

• Any malfunction of terminal should be notified to the Hardware Engineer.
• The terminal should be repaired or replaced by the local vendors as soon as possible.
• The transactions should be transmitted to the main branch by telex/phone or FAX. The main branch should be responsible for the day s posting after receipt of transactions.

10.4.4 Remote Branch Communications/Modem Failure

Central Server

If the communications or modem between the main system and remote branch fails, it automatically cut off the hardware in use in the branch, except in case of a distributed setup, where there is a server in the remote branch. The recommended steps to be followed, where the remote branch connects to the main system, in case of communications/modem failure are given below:

• Contact the local communications expert to check communication line and for modem check.
• If not fixed, the senior officer in the remote branch should decide to telephone the main branch depending on the expected down time and the main branch should arrange to process these inputs.
• For modem failures, back-up modem may be arranged from local vendors.

In case of a distributed set-up, the remote node can still function.

Printer Failure

Any malfunctioning printer can be replaced by Data Center staff/Hardware Engineer with available printer. The failure of one or more printers is not critical to Branch operations. The printers can be used as back-ups for each other without any adverse effects. The recommended steps to be followed in case of printer failure are as follows:

• Any malfunction of printer should be notified to the Data Center Manager who should perform first level of trouble shooting.
• The printer should be repaired or replaced by the local vendors as soon as possible.

10.4.5 Software Failure

Software failure can be caused by the following situations:

• Database/Files Damage
• Oracle FLEXCUBE Software Damage
• Oracle FLEXCUBE Software Bug

The recommended steps to be followed in the above cases are:

• The Data Center Manager should inform all the user departments of the situation and the expected time for correction.
• Report to the Senior Branch Operations Manager.
• The Data Center Manager should inform the Systems Manager.
• If the Branch has a maintenance and support agreement with Oracle Financial Services then the Systems Manager should report/handle the problem as per the support procedure in the Support Manual. The bank should take action as per Oracle Financial Services s instructions.
• On the Trouble Report Log, a record should be made of the time the problem occurred, problem description, for further investigation and action taken.

10.4.6 Utilities Failure

Utilities failures can be caused by the following situations: power supply failure and variation in Power supply/Temperature/Humidity.

Power Supply Failure

The recommended steps to be followed in case of a power supply failure are as follows:

• If advanced warning of power failure is announced Data Center Manager should make alternative arrangements.
• The Data Center Manager should contact technicians
• Power off all the machines
• Report to Senior Branch Operations Manager
• If power remains off and back-up site arrangement exists then back-up site should be contacted for possible use of their system.
• If power remains off for more than 8 hours prepare to move to back-up site.

Variation of Power Supply/Temperature/Humidity

The recommended steps to be followed in case of a variation in Power Supply/Temperature/Humidity are as follows:

The Data Center Manager should contact technicians and ensure:

• Power off all the machines.
• Report to Senior Branch Operations Manager.
• If variation in power/temperature/humidity lasts over 2 hours and back-up site arrangement exists then back-up site should be contacted for possible use of their system.
• If variation in power/temperature/humidity lasts over 8 hours, prepare to move to back-up site.