10. Contingency Plan
The managers should determine the operational risks and develop a
procedure to ensure that the business is not disrupted in an emergency
situation.
The people responsible for developing a contingency plan for the bank
are:
- The Contingency Planning Officer-for the overall branch plan
- The Data center manager-for technology contingency, including communications,
hardware, system software and third party software
- The Senior Operations Manager of the bank, who is responsible for
Oracle FLEXCUBE
- The members of the Internal Control department
The Contingency plan should address the following issues:
- Operation Risk assessment
- Contingency planning
- Software errors outside of normal working hours
- Contingency plan distribution list
- Persons who can authorize the emergency procedure
- Contact points in the event of hardware and software problems
- Potential exposure and containment measures
- Emergency back-up plan
10.1 Operation Risk Assessment
Oracle FLEXCUBE runs on Client Server Architecture, with ORACLE FORMS
on the client side and ORACLE 7.3 (with or without distribution option)
on the Server (UNIX). The Database can either be centralized in a single
server or distributed in more than one server connected through telecommunication
network.
There are some risks involved in the implementation of a new system
and the maintenance of an existing one. Some risks are controllable while
others are not. But the degree of non-controllable risks, such as natural
disasters, can be minimized. This chapter deals with risks and their
corresponding protective measures. The risk analysis is geared toward
the security of hardware and software. Tight security and back-up systems
are the most important elements. Adequate training for the personnel
who will be dealing with the computer is also very important. The basic
contents of the risk assessment are as follows:
- Brief Description of Location and Operations.
- Major Causes of the Operational Risks.
This section contains the following topics
10.1.1 Brief Description of Location and Operations
A brief description of the location and the operations of the bank
should be indicated in the assessment.
10.1.2 Major Causes of the Operational Risks
The following type of critical factors, which present operational
risks, are to be considered:
- Political or Civil Unrest
- Political or Civil disturbances
- Strikes and Riots
- Insurrection etc
- People Related Risks (Internal)
- Illness or Injury
- Non-adherence to established procedures
- Shortage of training in established procedures
- Unpleasant working conditions
- Deliberate or Negligent acts
The risk involved can be loss of customers, fraud, processing errors,
delays which include information modification, loss of information, data
omission, damages to hardware, etc.
- People Related Risks (External)
- Intrusion
- Kidnapping or extortion
- Theft of equipment, etc
- Utility Related Risk
- Electricity
- Communication
- Voltage stabilizer
- Air Conditioning etc
Power loss causes processing errors and delays, and a total memory
loss in computers. Irregular or faulty power lines can alter the data
being processed and/or cause permanent damage to the computer.
- Neighbourhood Hazards
- Proximity to chemical or explosive operations
- Nearby building or floor that constitutes a fire hazard to the operation.
- Potential risk of leakage or burst in the water pipes on the premises.
- High crime areas.
10.2 Contingency Planning
The Data Center is a supporting unit that provides the Data Processing
services for the contingency planning process. These services are in
the area of the Data Center
This is considered to be a medium risk area and its disruption implies
that the bank, Customer Service, Trade and Treasury are rendered inoperable
with a resultant loss of the bank s reputation and customers. Potential
threats to the Data Center (and consequently to the other units of Operations)
include:
- People related risk, e.g. illness, strike, sabotage
- Natural disasters, e.g. fire, power failure, etc. rendering the premises
and facilities unavailable
- Equipment failure, e.g. failure of Main or Back-up Hardware, peripherals,
workstations, printers, etc.
- Software Failure, e.g. abnormal termination of a program due to a
database corruption or program bug
- Support Line failure, e.g. failure of the link between the Oracle
Financial Services Support team and the bank
The above risk factors should be analyzed and preventive measures
should be described in detail. Some of the recommended preventive measures
are:
- Employees should be cross-trained at different jobs so that they
can act as a back-up for each other in case of absence due to illness,
etc.
- Regular medical check-ups should be arranged for the employees
- Fire detection and suppression equipment, e.g. smoke detectors, fire
extinguishers, Halon, etc. should be placed at all vulnerable locations
- Main and Back-up Hardware systems, printers and workstations should
be serviced and maintained regularly
- Service contract - The bank should evaluate the need for a maintenance
contract and document for hardware and software
- A private, leased data-line should exist between Oracle Financial
Services (formerly i-flex) and the bank. A dial-up facility should also
exist as a back-up to access Oracle Financial Services (formerly i-flex)
The contingency plan should be drawn to cover those functions critical
to the business of the bank. The contingency plan should contain personnel
lists (organization chart, distribution list, authorizers of emergency
measures, succession list and alternate account managers), action plans,
contact names and addresses and contingency planning documentation.
10.3 Software Errors Outside Normal working Hours
In case of a problem arising outside normal working hours, e.g. during
the End of Day, month- end, year- end runs, etc., the following is the
recommended emergency procedure:
- The Systems Manager or the back-up person should be summoned if not
already available on the premises. No action should be taken until the
Systems Manager or the back-up person arrives.
- The Senior Operations Manager of the bank should be informed about
the matter.
- If the problem has occurred earlier and if the method of resolution
is known, it should be applied (all occurrences of problems and their
resolution should be entered in the Data Center software logbook). The
Systems Manager should ensure that the program overlay and the statement
number are exactly the same as in the logbook.
- The Senior Operations Manager of the Bank and the Internal Control
Head may be asked to come to the bank to allow the Systems Manager to
enter in the SUPERVISOR mode and correct the problem.
- If the Senior Operations Manager of the bank and the Internal Control
Head are not available, the Systems Manager should contact the respective
back-ups to gain access to the supervisor mode. Secret passwords to the
supervisor mode are lodged in the main vault and may be withdrawn and
opened if the regular and alternate custodians are unavailable.
- If the error happens for the first time and the bank has a technical
person for Oracle FLEXCUBE maintenance, the Systems Manager should consult
the person, identify the bug and fix it as per procedure. If this is
not possible, Oracle Financial Services should be contacted for instructions.
When the bug is fixed, recovery programs 100 and 101 should be run.
- If the error happens for the first time and the bank has a maintenance
and support agreement with Oracle Financial Services the Systems Manager
should report/handle the problem as per the support procedure in the
support Manual. The bank should take action as per the instructions of
Oracle Financial Services
- In all cases, details of the events (error description, communications
with Oracle Financial Services etc. should be carefully noted in the
Logbook of the Data Center and reviewed by the Internal Control Unit.
This section contains the following topics
10.3.1 Contingency Plan Distribution List
The recommended contingency plan distribution list is as follows.
- The Senior Management of the branch
- The Manager of the Data Center and the Systems Manager of the bank
- The Audit Division of the bank
- The Senior Operations Manager of the bank
- The System Administrative Manager
- The Control Head of the bank
- The head of the Liability Management Group
- The Manager of the Off-site Back-up location
The contingency plan distribution list should be finalized by the
senior management of the bank.
10.3.2 People who can Authorize Emergency Measures
A list should be prepared, giving the following information about
the officers who can authorize emergency measures:
- Name and function of the officer
- Telephone extension number in the office
- Direct telephone number (if available) in the office
- Residential telephone number
10.3.3 Contact Points in Case of Hardware and Software
Problems
The following information for contact points in the event of Hardware
and Software Problems should be listed:
- Name address of the Organization to be contacted
- Contact person, office telephone number and residential telephone
numbers
- AX/Telex numbers of the Branch or Organization
- Port Numbers of the Oracle Financial Services Support division
- Dialing codes for the country
10.4 Potential for Exposure and Containment Measures
The Contingency Plan should clearly describe the Safety, Preventive
and Containment measures. It should include guidelines and the procedures
to be followed in case of an emergency related to Oracle FLEXCUBE.
This section contains the following topics
10.4.1 Hardware Failure
Hardware failure can be caused by the following situations:
- Failure of the Main machine
- Failure of the Back-up machine
- Failure of the Main and Back-up machines
- Failure of the Disk Drive
- Failure of a Terminal
- Failure of the Remote Branch terminal
- Failure of the Remote Branch communication or Modem
- Failure of the Printer
10.4.2 Containment Measures
Main machine failure
When failure of the main machine has an immediate effect on the processing,
the following steps should be taken:
- The Manager of the Data Center should inform the user departments
about the situation and the time required to switch on the back-up system.
- The matter should be reported to the Senior Operations Manager of
the bank.
- The Manager of the Data Center should establish the cause of the
problem before switching on the Back-up system.
- The Manager of the Data Center should connect all the workstations
and peripherals to the back-up system and inform all the departments.
- Contact the local dealers for immediate repair.
- Record all information about the problem in the Trouble Report Log,
The details to be included are nature of the problem, time of occurrence
and name of the person reporting the problem. The time of contacting
the Customer Engineer and the arrival of the Customer Engineer should
also be recorded.
- After consultation with Customer Engineer determine the time required
to get the main system running. If the repairs are expected to extend
beyond 24 hours and the off-site back-up facility exists, inform the
off-site back-up location of a possible need for use of their system.
- The following reports should be generated at the end of each day,
to provide for an emergency. The reports will be useful if the Back-up
machine fails before the Main machine is repaired.
- Accounting Journal
- Trial Balance
- General Ledger
- Currency Position
- Limits Reports
- Accrual reports
- End of Day Account/GL Balances
Produce the accrual reports to avoid manual calculations. This program
should only be run against a copy of data.
Back-up machine failure
If the Back-up machine is inoperative, the situation does not affect
the normal operations, but it requires immediate repair to avoid a complete
breakdown. The following steps should be carried out:
- The Senior Operations Manager of the bank should be apprised of the
situation. Since the failure of the back-up machine does not disrupt
the normal operations, it is not necessary to inform the user departments.
- Contact the local dealers for immediate repair.
- Enter all relevant information about the problem in the Trouble Report
Log. The details should include the nature of the problem, the time of
occurrence and the name of the person who reported the problem. The time
of contacting the Customer Engineer and the arrival of the Customer Engineer
should also be recorded.
- After consultation with the Customer Engineer determine the time
required to get the back-up system running. If the repairs are expected
to extend beyond 24 hours and if an off-site back-up facility exists,
inform the off-site location of a possible need to use their system.
- If repairs are expected to take more than one day, the key reports
(as outlined above, under Main Machine Failure), should be generated.
The reports will be useful if the Main Machine fails before the back-up
machine is repaired.
Failure of Main and Back-up Machines
The failure of both the Main and the Back-up machines is considered
an unlikely event. The stationery required for manual operations, should
be stored off-site. In such a situation the following steps should be
taken:
- The Manager of the Data Center should inform all the user departments
of the situation.
- Report to the Senior Operations Manager of the bank.
- Contact the local dealers for immediate repair.
- Record all relevant information in the Trouble Report Log. The details
should include the nature of the problem, the time of occurrence and
the name of the person who reported it. The time of contacting the Customer
Engineer and the arrival of the Customer Engineer should also be recorded.
- After consultation with the Customer Engineer, determine the time
required to get at least one machine running. If the repairs are expected
to extend beyond 8 hours and the off-site back-up facility exists then
inform the off-site back-up location, of a need to use their system.
The necessary equipment should be made available.
- If the Senior Operations Manager of the bank decides to go off-site
to continue processing, the following steps are recommended:
- The Data Center Manager should ensure that all the required material
on the checklist is ready to be taken to the back-up site.
- The heads of all the departments should arrange to collect the input
media, Database maintenance forms and tickets from the users and advise
the systems department of completion.
- The Data Center Manager should approve of the material checklist
before the materials can be transported to the off-site back-up facility
under proper custody. The Data Center Manager should monitor the whole
process and if any problems are encountered they should be reported to
the Senior Branch Operations Manager.
At the back-up site, Oracle FLEXCUBE Database should be restored from
the latest back-up tapes from the on site library (ideally, the backup
of post end of previous day). All the daily transactions should be input
if necessary and EOD should be run. Extra copies of reports should be
prepared where necessary. The bank should use the latest daily central
liability printouts (stored off-site) to pay checks over the counter
and maintain manual ledger to record all movements.
In case both the machines break down in the middle of business hours,
earlier payments should be taken into account from daily printouts (last
business day central liability report) before further payments are made.
The Senior Branch Operations Manager should use discretion in determining
the specific operation needs. The process detailed below provides a guideline
for manual processing. This can be followed if the time to get at least
one system up is likely to exceed 2 working days.
- The Processing priorities should be as follows:
- Clearing Operations
- Teller transactions
- Trading Exchange/Transactions
- Import/Export Operations
- Funds Transfers
- Deposits
- Credit/Financial Control processing
- The following reports printed and kept off-site will assist in manual
operations.
- Limit Reports
- Account Balances Report
- Balance controlling should be centralized with the Referral Clerk
or person designated by the Senior Branch Operations Manager.
- Adequate stationery for departmental proofs, tickets, ledgers and
manifolds, should be maintained to facilitate manual processing.
- Current Account balances should be updated manually.
- For all accounts having movements, a run-up of old balances, debits,
credits and new balances should be taken.
- Accounting entry should be done through departmental proofs and tickets.
For this purpose the trial balance and the Currency Position of the previous
day should be used. Ledgers should be maintained, showing balance B/F,
debits and credits movements and new balance.
- The Control department should comprehensively recheck all movements
through source documents and tickets. Control should ensure that all
source documents and tickets are microfilmed.
10.4.3 Disk Drive Failure
Main Machine Disk Drive Failure
This does not constitute a critical issue. If the disk fails the processing
should be switched on to the back-up machine as per the recommended procedure
described above in Main machine failure. The main machine equipment should
be repaired/replaced immediately.
Back-up Machine Disk Drive failure
Although there is no impact on the operation when the disk drive fails
on the back-up machine, urgent repairs/replacements must be ensured in
order to bring the probability of general system failure back to the
desired level.
Both Main and Back-up machines Disk Drive failure
Failure of both Main and Back-up machines Disk Drives constitutes
a critical situation and is considered an unlikely event. Should the
situation arise the branch should follow the recommended procedure described
above in situation where both main and back-up machine failure occurs
simultaneously.
Note
If the disk drive on the main machine is physically
scratched or damaged by any means, the latest back-up tape should be
loaded on the back-up machine and re-input or re- processing has
to be done.
Terminal Failure
Any malfunctioning terminal can be replaced by the Data Center staff
or Hardware Engineer with an available terminal. The failure of one or
more is not critical to Branch operations. The recommended steps to be
followed in case of terminal failure are given below:
- Any malfunction of a terminal should be notified to the Data Center
Manager who should perform first level of trouble shooting.
- The terminal should be repaired or replaced by the local vendors
as soon as possible.
Remote Branch Terminal Failure
Any malfunctioning remote branch terminal should be repaired or replaced
by the Hardware Engineer with the available terminal at the branch. The
failure of one or more terminals is not critical to remote Branch operations
if some remote terminals are in working order. The recommended steps
to be followed in case of failure of all the remote terminals, are given
below:
- Any malfunction of terminal should be notified to the Hardware Engineer.
- The terminal should be repaired or replaced by the local vendors
as soon as possible.
- The transactions should be transmitted to the main branch by telex/phone
or FAX. The main branch should be responsible for the day s posting after
receipt of transactions.
10.4.4 Remote Branch Communications/Modem Failure
Central Server
If the communications or modem between the main system and remote
branch fails, it automatically cut off the hardware in use in the branch,
except in case of a distributed setup, where there is a server in the
remote branch. The recommended steps to be followed, where the remote
branch connects to the main system, in case of communications/modem failure
are given below:
- Contact the local communications expert to check communication line
and for modem check.
- If not fixed, the senior officer in the remote branch should decide
to telephone the main branch depending on the expected down time and
the main branch should arrange to process these inputs.
- For modem failures, back-up modem may be arranged from local vendors.
In case of a distributed set-up, the remote node can still function.
Printer Failure
Any malfunctioning printer can be replaced by Data Center staff/Hardware
Engineer with available printer. The failure of one or more printers
is not critical to Branch operations. The printers can be used as back-ups
for each other without any adverse effects. The recommended steps to
be followed in case of printer failure are as follows:
- Any malfunction of printer should be notified to the Data Center
Manager who should perform first level of trouble shooting.
- The printer should be repaired or replaced by the local vendors as
soon as possible.
10.4.5 Software Failure
Software failure can be caused by the following situations:
- Database/Files Damage
- Oracle FLEXCUBE Software Damage
- Oracle FLEXCUBE Software Bug
The recommended steps to be followed in the above cases are:
- The Data Center Manager should inform all the user departments of
the situation and the expected time for correction.
- Report to the Senior Branch Operations Manager.
- The Data Center Manager should inform the Systems Manager.
- If the Branch has a maintenance and support agreement with Oracle
Financial Services then the Systems Manager should report/handle the
problem as per the support procedure in the Support Manual. The bank
should take action as per Oracle Financial Services s instructions.
- On the Trouble Report Log, a record should be made of the time the
problem occurred, problem description, for further investigation and
action taken.
10.4.6 Utilities Failure
Utilities failures can be caused by the following situations: power
supply failure and variation in Power supply/Temperature/Humidity.
Power Supply Failure
The recommended steps to be followed in case of a power supply failure
are as follows:
- If advanced warning of power failure is announced Data Center Manager
should make alternative arrangements.
- The Data Center Manager should contact technicians
- Power off all the machines
- Report to Senior Branch Operations Manager
- If power remains off and back-up site arrangement exists then back-up
site should be contacted for possible use of their system.
- If power remains off for more than 8 hours prepare to move to back-up
site.
Variation of Power Supply/Temperature/Humidity
The recommended steps to be followed in case of a variation in Power
Supply/Temperature/Humidity are as follows:
The Data Center Manager should contact technicians and ensure:
- Power off all the machines.
- Report to Senior Branch Operations Manager.
- If variation in power/temperature/humidity lasts over 2 hours and
back-up site arrangement exists then back-up site should be contacted
for possible use of their system.
- If variation in power/temperature/humidity lasts over 8 hours, prepare
to move to back-up site.