Table of Contents

• Title and Copyright Information
• My Oracle Support
• What's New in This Guide
• 1 Introduction
  • 1.1 Overview
  • 1.2 Scope and Audience
  • 1.3 Documentation Admonishments
  • 1.4 Manual Organization
  • 1.5 Related Publications
  • 1.6 Locate Product Documentation on the Oracle Help Center Site
• 2 LSMS Overview
  • 2.1 Introduction
  • 2.2 LSMS Connectivity
  • 2.3 LSMS Hardware Overview
  • 2.4 LSMS Software Overview
    • 2.4.1 Operating System Layer
    • 2.4.2 Base Platform Software
    • 2.4.3 Support/Base Utilities Components
    • 2.4.4 LSMS Applications
  • 2.5 Overview of High Availability
    • 2.5.1 Hardware Redundancies
    • 2.5.2 Software Availability Design
  • 2.6 Enhancing High Availability with a Shadow LSMS
  • 2.7 Understanding the Surveillance Feature
    • 2.7.1 Controlling the Surveillance Feature
      • 2.7.1.1 Starting the Surveillance Feature
      • 2.7.1.2 Stopping the Surveillance Feature
      • 2.7.1.3 Determining the Surveillance Status
      • 2.7.1.4 Returning the Surveillance Feature to Last Valid State
    • 2.7.2 Understanding Surveillance Notifications
  • 2.8 Understanding the Service Assurance Feature
  • 2.9 Understanding the SNMP Agent Process
    • 2.9.1 Stopping the SNMP Agent
    • 2.9.2 Starting the SNMP Agent
    • 2.9.3 Determining the Status of the SNMP Agent
    • 2.9.4 Logging SNMP Agent Actions
  • 2.10 Configuring the SNMP Agent
  • 2.11 SNMP Global Mode
  • 2.12 SNMPv3 Access View Management
  • 2.13 SNMPv3 Group Management
  • 2.14 SNMPv3 User Management
  • 2.15 NMS Configuration
  • 2.16 Autonomous Events Trap Forwarding
  • 2.17 Resynchronization
  • 2.18 Connectivity between LSMS and NMS
• 3 Routine Procedures
  • 3.1 Introduction
  • 3.2 Using Login Sessions
    • 3.2.1 Logging In to LSMS Server Command Line
    • 3.2.2 Logging in from One Server to the Mate’s Command Line
    • 3.2.3 Starting a Web-Based LSMS GUI Session
    • 3.2.4 Logging Into the LSMS Console Window
    • 3.2.5 Modifying Title Bar in LSMS Console Window
  • 3.3 Powering On the LSMS
  • 3.4 Powering Off the LSMS
  • 3.5 Managing the System Clock
    • 3.5.1 Automatically Controlling the LSMS Time Using NTP
      • 3.5.1.1 Configuring the LSMS as an NTP Client
      • 3.5.1.2 Verifying NTP Service
      • 3.5.1.3 Troubleshooting NTP Problems
        • 3.5.1.3.1 Reference Time Off By More Than Twenty Minutes
        • 3.5.1.3.2 Violation of Maximum Oscillator Frequency in Network
    • 3.5.2 Manually Controlling the LSMS Time Without an External NTP Source
  • 3.6 Managing User Accounts
    • 3.6.1 Non-Configurable Permission Groups
    • 3.6.2 Configurable Permission Groups (LSMS Command Class Mgmt)
      • 3.6.2.1 Permission Group Processing
    • 3.6.3 Managing User Accounts on the Primary and Secondary Servers
      • 3.6.3.1 Adding a User
      • 3.6.3.2 Deleting a User
      • 3.6.3.3 Changing a User Password
      • 3.6.3.4 Setting the System Level Password Timeout Using the Command Line
      • 3.6.3.5 Setting the System Level Password Timeout Using the GUI
      • 3.6.3.6 Setting the User Level Password Timeout Using the Command Line
      • 3.6.3.7 Setting the User Level Password Timeout Interval Using the GUI
      • 3.6.3.8 Displaying All LSMS User Accounts
      • 3.6.3.9 Viewing the Active User List
      • 3.6.3.10 Terminating an Active User Session
    • 3.6.4 Activating the SPID Security Feature
  • 3.7 Accessing GUI on Java8
• 4 Preventive Maintenance
  • 4.1 Introduction
  • 4.2 Recommended Daily Monitoring
    • 4.2.1 Once a Day Monitoring Activities
  • 4.3 LSMS Database Defragmentation
  • 4.4 Using Backup Procedures
    • 4.4.1 Understanding How the LSMS Backs Up File Systems and Databases
    • 4.4.2 Understanding the Backup Results
    • 4.4.3 Backing Up the LSMS Manually
    • 4.4.4 Stopping an Automatic or Manual Backup
    • 4.4.5 Checking for Running Backups
  • 4.5 Using Restore Procedures
  • 4.6 Additional Tools for Monitoring the LSMS Hardware and the Network
    • 4.6.1 Verifying Active Server Network Interfaces and NPAC Connections
      • 4.6.1.1 Using the ifconfig Command
      • 4.6.1.2 Using the traceroute Utility
  • 4.7 Managing Automatic File Transfers
    • 4.7.1 Displaying Remote Locations Used for Automatic File Transfers
    • 4.7.2 Adding a New Remote Location for Automatic File Transfers
    • 4.7.3 Deleting a Remote Location for Automatic File Transfers
    • 4.7.4 Displaying Previously Scheduled Automatic File Transfers
    • 4.7.5 Scheduling an Automatic File Transfer
    • 4.7.6 Removing a Scheduled Automatic File Transfer
• 5 Restarting Software Processes
  • 5.1 Introduction
  • 5.2 Automatically Restarting Software Processes
    • 5.2.1 Automatically Monitoring and Restarting EAGLE Agent Processes
    • 5.2.2 Automatically Monitoring and Restarting NPAC Agent Processes
    • 5.2.3 Automatically Monitoring and Restarting OSI Process
    • 5.2.4 Automatically Monitoring and Restarting the Service Assurance Process
    • 5.2.5 Automatically Monitoring and Restarting the rmtpmgr Process
    • 5.2.6 Automatically Monitoring and Restarting the rmtpagent Process
    • 5.2.7 Automatically Monitoring and Restarting Other Processes
• 6 Managing Server States
  • 6.1 Introduction
  • 6.2 Understanding Server States
  • 6.3 Understanding Switchover
  • 6.4 Understanding Automatic Switchover
  • 6.5 Managing Server States Manually
    • 6.5.1 Using the lsmsmgr Interface to Determine the Server Status
    • 6.5.2 Using the hastatus Command to Determine the Server Status
    • 6.5.3 Manually Switching Over from the Active Server to the Standby Server
    • 6.5.4 Inhibiting a Standby Server
    • 6.5.5 Starting a Server
• 7 Recovering from Site Failures
  • 7.1 Introduction
  • 7.2 Choosing a Disaster Backup Strategy
  • 7.3 Synchronizing Data Between the Main LSMS and Shadow LSMS
  • 7.4 Preparing for a Disaster Situation
  • 7.5 Determining When to Switch to Shadow LSMS
  • 7.6 Disaster Recovery Procedure Overview
  • 7.7 Performing Disaster Recovery with an Active Shadow LSMS
  • 7.8 Performing Disaster Recovery with an Inactive Shadow LSMS
  • 7.9 Performing Disaster Recovery without a Shadow LSMS
  • 7.10 Returning Operation from Shadow LSMS to Main LSMS
  • 7.11 Resynchronizing After an Outage Between an NPAC and the LSMS
  • 7.12 Reconnecting Network Elements
    • 7.12.1 Reconnecting Network Elements Procedures
• 8 Verifying Recovery
  • 8.1 Introduction
  • 8.2 Verifying that the LSMS Is Fully Functional
    • 8.2.1 Verifying the State of the Servers
    • 8.2.2 Verifying the Processes Running on the Active Server
    • 8.2.3 Verifying the GUI Operability on the Active Server
• 9 Field Replaceable Units
  • 9.1 Introduction
  • 9.2 E5-APP-B Card FRUs and Part Numbers
  • 9.3 Removing and Replacing E5-APP-B Cards
    • 9.3.1 Removing an E5-APP-B Card
    • 9.3.2 Replacing an E5-APP-B Card
  • 9.4 Removing and Replacing a Drive Module Assembly
    • 9.4.1 Removing a Drive Module Assembly
    • 9.4.2 Replacing a Drive Module Assembly
• A Commands
  • A.1 Introduction
  • A.2 Entering LSMS and Third-Party Application Commands
    • A.2.1 autoxfercfg
    • A.2.2 chglct
    • A.2.3 chkfilter
    • A.2.4 eagle
    • A.2.5 hastatus
    • A.2.6 import
    • A.2.7 keyutil
    • A.2.8 lsms
    • A.2.9 lsmsdb
    • A.2.10 lsmsSNMP
    • A.2.11 lsmssurv
    • A.2.12 massupdate
    • A.2.13 measdump
    • A.2.14 npac_db_setup
    • A.2.15 npacimport
    • A.2.16 report
    • A.2.17 resync_db_setup
    • A.2.18 SAagent
    • A.2.19 savelogs
    • A.2.20 service mysql status
    • A.2.21 spidsec
    • A.2.22 sup
    • A.2.23 sup_db_setup
    • A.2.24 survNotify
    • A.2.25 syscheck
• B Automatic Monitoring of Events
  • B.1 Introduction
  • B.2 Overview of Monitored Events
  • B.3 Overview of GUI Notifications
    • B.3.1 Format of GUI Notifications
      • B.3.1.1 Variables Used in GUI Notification Format Descriptions
    • B.3.2 Variables Used in Message Text String of GUI Notifications
    • B.3.3 Examples of GUI Notifications
    • B.3.4 Logging GUI Notifications
  • B.4 Overview of Surveillance Notifications
    • B.4.1 Variables Used in Surveillance Notification Format Descriptions
    • B.4.2 Variables Used in Message Text String of Surveillance Notifications
    • B.4.3 Example of a Surveillance Notification
    • B.4.4 Logging Surveillance Notifications
  • B.5 Overview of Traps
    • B.5.1 Logging SNMP Agent Actions
  • B.6 Event Descriptions
  • B.7 Platform Alarms
    • B.7.1 How to Decode Platform Alarms
    • B.7.2 Platform Alarms
    • B.7.3 Alarm Recovery Procedures
    • B.7.4 Major Platform Alarms
      • B.7.4.1 3000000000000001 – Server fan failure
      • B.7.4.2 3000000000000002 - Server Internal Disk Error
      • B.7.4.3 3000000000000008 - Server Platform Error
      • B.7.4.4 3000000000000010 - Server File System Error
      • B.7.4.5 3000000000000020 - Server Platform Process Error
      • B.7.4.6 3000000000000080 - Server Swap Space Shortage Failure
      • B.7.4.7 3000000000000100 - Server provisioning network error
      • B.7.4.8 3000000000001000 - Server Disk Space Shortage Error
      • B.7.4.9 3000000000002000 - Server Default Route Network Error
      • B.7.4.10 3000000000004000 - Server Temperature Error
      • B.7.4.11 3000000000008000 - Server Mainboard Voltage Error
      • B.7.4.12 3000000000010000 - Server Power Feed Error
      • B.7.4.13 3000000000020000 - Server Disk Health Test Error
      • B.7.4.14 3000000000040000 - Server Disk Unavailable Error
      • B.7.4.15 3000000000080000 - Device Error
      • B.7.4.16 3000000000100000 - Device Interface Error
      • B.7.4.17 3000000400000000 - Multipath device access link problem
      • B.7.4.18 3000000800000000 – Switch Link Down Error
      • B.7.4.19 3000001000000000 - Half-open Socket Limit
      • B.7.4.20 3000002000000000 - Flash Program Failure
      • B.7.4.21 3000004000000000 - Serial Mezzanine Unseated
      • B.7.4.22 3000000008000000 - Server HA Keepalive Error
      • B.7.4.23 3000000010000000 - DRBD block device can not be mounted
      • B.7.4.24 3000000020000000 - DRBD block device is not being replicated to peer
      • B.7.4.25 3000000040000000 - DRBD peer needs intervention
    • B.7.5 Minor Platform Alarms
      • B.7.5.1 5000000000000001 - Server Disk Space Shortage Warning
      • B.7.5.2 5000000000000002 - Server Application Process Error
      • B.7.5.3 5000000000000004 - Server Hardware Configuration Error
      • B.7.5.4 5000000000000008 - Server RAM Shortage Warning
      • B.7.5.5 5000000000000020 - Server Swap Space Shortage Warning
      • B.7.5.6 5000000000000040 - Server Default Router Not Defined
      • B.7.5.7 5000000000000080 – Server temperature warning
      • B.7.5.8 5000000000000100 - Server Core File Detected
      • B.7.5.9 5000000000000200 - Server NTP Daemon Not Synchronized
      • B.7.5.10 5000000000000400 - Server CMOS Battery Voltage Low
      • B.7.5.11 5000000000000800 - Server Disk Self Test Warning
      • B.7.5.12 5000000000001000 - Device Warning
      • B.7.5.13 5000000000002000 - Device Interface Warning
      • B.7.5.14 5000000000004000 - Server Reboot Watchdog Initiated
      • B.7.5.15 5000000000008000 - Server HA Failover Inhibited
      • B.7.5.16 5000000000010000 - Server HA Active To Standby Transition
      • B.7.5.17 5000000000020000 - Server HA Standby To Active Transition
      • B.7.5.18 5000000000040000 - Platform Health Check Failure
      • B.7.5.19 5000000000080000 - NTP Offset Check Failure
      • B.7.5.20 5000000000100000 - NTP Stratum Check Failure
      • B.7.5.21 5000000020000000 – Server Kernel Dump File Detected
      • B.7.5.22 5000000040000000 – TPD Upgrade Failed
      • B.7.5.23 5000000080000000 - Half Open Socket Warning Limit
      • B.7.5.24 5000000000200000 - SAS Presence Sensor Missing
      • B.7.5.25 5000000000400000 - SAS Drive Missing
      • B.7.5.26 5000000000800000 - DRBD failover busy
      • B.7.5.27 5000000001000000 - HP disk resync
    • B.7.6 Saving Logs Using the LSMS GUI or Command Line
• C Downloading Files from an NPAC to the LSMS
  • C.1 Overview
    • C.1.1 NPAC-LSMS Download Procedure Summary
  • C.2 Prerequisite Information
  • C.3 File Naming Conventions
  • C.4 NPAC-LSMS Download Procedure
  • C.5 Copying Files to Other Server If Switchover Occurs
    • C.5.1 Copying All Downloaded Files After Switchover
    • C.5.2 Copying Partially Downloaded Files After Switchover
• D Worksheets
  • D.1 Introduction
  • D.2 Recovery Preparation Worksheet
• E Query Server Maintenance Procedures
  • E.1 Introduction
  • E.2 LSMS Maintenance Procedures
    • E.2.1 Automatic Monitoring of Query Servers
    • E.2.2 Modify the MySQL Port for Query Servers
    • E.2.3 Check Connection Status of Directly Connected Query Servers
    • E.2.4 Maintain the Binary Log on Query Servers
    • E.2.5 Check MySQL Replication Status on Query Servers
    • E.2.6 Start MySQL Replication on Query Servers
    • E.2.7 Stop MySQL Replication on Query Servers
    • E.2.8 Check for Running Backups
    • E.2.9 Reload a Query Server Database from the LSMS
    • E.2.10 Reload a Query Server Database from Another Query Server
    • E.2.11 Clean Up After Failed or Interrupted Snapshot
  • E.3 Automated System Check
  • E.4 Query Server Error Log
  • E.5 Retrieving Information from LNP Database Fields
  • E.6 LNP Database Tables and Fields
  • E.7 Query Server Database Structure
• F Resolution Of Known Customer Issues