• The RDBMS is used to persist the NMS electrical network data model along with outage information, switch plans and analytical information.

• The supporting Oracle RDBMS is often deployed on (RDBMS) dedicated hardware, but can also be deployed on the same server as NMS Services and/or WebLogic.

• The UNIX file system is used to hold executables, RDBMS data files, certain electrical and customer model initialization files (generally extracted from enterprise GIS and CIS respectively) and assorted configuration information - for multiple tiers.

• The Oracle RDBMS listener process is used to service requests from NMS services, NMS Perl, Python, and SQL*Plus scripts, WebLogic JDBC connections, and (potentially) other project specific applications.

• NMS installations do not typically encrypt Oracle NMS RDBMS data "at rest" or "in flight."

• Encrypting all in-transit Oracle RDBMS data has an NMS performance impact (roughly 5% in our internal NMS tests). Because of the internal nature of the typical infrastructure supporting NMS environments (NMS is typically deployed behind multiple firewalls) - in-transit Oracle RDBMS data is not encrypted by default. Still the performance penalty for encrypting all in-transit data should not have a major impact on overall NMS performance. If the NMS RDBMS is exposed to a difficult to monitor/manage set of users - such as other business owners/users outside the core NMS team/users- encrypting in-transit data should be considered as a viable option to help protect NMS model data from unauthorized access.

• Oracle RDBMS data "in flight" can be encrypted via the Oracle Enterprise Edition RDBMS supported Native Network Encryption option. This includes any data exchanged between Oracle RDMS instances via Data Guard.

• Either a subset (specific columns within specific tables) of Oracle RDBMS data "at rest" or entire application tablespaces can also be encrypted via the Oracle Enterprise Edition RDBMS Transparent Data Encryption (TDE) option. If the TDE option is deployed it is recommended to be used for entire tablespaces rather than individual tables and columns - for efficiency. With modern hardware (with on-chip encryption/decryption) the overhead is typically very low (low single digits). Once configured this option is transparent to RDBMS users. The Oracle TDE option is included with Oracle EE RDBMS.

• By using the Oracle Enterprise Edition RDBMS Advanced Security option all data "at rest" or "on-the-wire" can be encrypted and sensitive data can also be redacted. The Advanced Security option is a separate Oracle EE RDBMS license option.

• Oracle RDBMS "in flight" data includes data handled by Oracle Data Guard if a backup Oracle NMS RDBMS is in play. The Oracle Data Guard Redo Transport Authentication mechanism leverages the Oracle RDBMS listener as a redo log data transport mechanism.

• Oracle NMS executable files installed under the nmsadmin Unix user name should never have more than "-rwxr-x---" file permission. The nmsadmin Unix user should have a umask of at least 0027. The default Unix group id for the nmsadmin username should only be used for Unix accounts that support NMS. For example the Unix group for nmsadmin should NOT be the generic Unix "users" group name (or similar). Suggest an NMS specific Unix group name of oranms (or similar) be used for the default Unix group for any nmsadmin type accounts - and only for those accounts.

• If other enterprise systems require access to the production NMS RDBMS recommend the principle of least privilege be applied - via an alternate RDBMS schema with limited privilege synonyms to the production schema. For example, Data Manipulation Language (DML) access and not Data Definition Language (DDL) access. If only read-only access is required - than configure accordingly. Using this approach should minimize the damage any outside account could potentially have on the production NMS RDBMS. Note this principle can apply to any external integration - including CIS integration.

• Single server supporting a single Oracle RDBMS instance.

• Two servers in a failover cluster supporting a single Oracle RDBMS instance.

• Two servers in a Real Application Cluster supporting a single Oracle RDBMS.

• Disaster Recovery and/or Business Continuity requires at least one additional RDBMS instance - in any of the above 3 configurations. Typically replicated using Oracle Data Guard or Oracle Active Data Guard.

• NMS services (written in C/C++) are used to manage the NMS real-time operational electrical network data model.

• A significant percentage of NMS project specific configuration is managed within NMS services via a range of configuration options. The vast majority of these configuration options are dependent on the NMS Oracle RDBMS model and configuration tables. These options tend to dictate how the NMS model looks and how it reacts to specific input stimuli. These options are mostly determined during the project configuration phase of an NMS deployment but can be modified by NMS admins after initial go-live.

• NMS services are tied together by the internal ISIS messaging bus. ISIS is a real-time synchronous/asynchronous high-performance publication/subscription in-memory messaging bus. NMS services use ISIS to publish relevant updates to each other and to the NMS CORBA gateway to get the messages to WebLogic. WebLogic provides an independent mechanism to cache messages from the NMS CORBA gateway and provides routine updates to NMS clients.

• ISIS only runs on the node where NMS services run. By default, ISIS runs on the loopback (localhost) network. In this mode, ISIS is only used to coordinate messages between NMS service processes and no ISIS ports are accessible from any external nodes. NMS ISIS messages are not encrypted.

• Recommend that a minimal number of (non NMS production account related) Unix accounts be configured on the Unix host (physical or virtual) where production NMS services run. Ideally only required Unix accounts and production NMS admin accounts should have access to these hosts. Minimizing the possibility of someone other than a proper NMS admin accessing production NMS services hosts minimizes the potential attack surface.

• NMS services talk to the Oracle NMS RDBMS listener via Oracle Call Interface APIs. NMS run-time Perl, Python, and SQL*Plus scripts also communicate to the RDBMS via the Oracle NMS RDBMS listener process.

• The Oracle NMS admin account has access to the Oracle RDBMS (for all of the above mechanisms) via an Oracle wallet installed on the Oracle NMS nmsadmin Unix account. The Oracle wallet stores encrypted RDBMS credentials for NMS daemon process access.

• If NMS services and WebLogic are running on distinct nodes (or under distinct UNIX usernames) there must be a mechanism in place to "serve" the NMS model map (*.mad and *mac) files from where NMS services are running to where WebLogic is running. These map files contain quasi-static coordinate and graphic symbology information for individual NMS electrical model partitions.

• Using the Network File System (NFS) is not an acceptable option for serving the NMS model map files to WebLogic. NFS suffers from race conditions with the NMS message bus that can yield inconsistent data for the Java clients.

• The preferred method to serve the NMS model map files is to use an HTTP file server. NMS comes with lighttpd embedded and auto-configured. The HTTP server is generally configured to serve a single directory of unencrypted NMS model map files to specific IP addresses where the WebLogic instances that support this NMS instance run. The NMS system release includes lighttpd in the 3rdparty products package and supporting scripts (nms-lighttpd and nms-lighttpd-oem) to automatically configure, start and stop each lighttpd instance.

• Lighttpd can be configured to leverage HTTPS.

• Lighttpd can be configured to limit access to specific file types.

• Lighttpd can be configured to only allow access from specific hosts.

• NMS services communicate to Enterprise Java Beans within WebLogic via an NMS CORBA adapter. CORBA uses the Internet Inter-Orb Protocol (IIOP) to move data from NMS CORBA adapter (corbagateway) to WebLogic. IIOP traffic is not encrypted.

• NMS services use Simple Object Access Protocol (SOAP) over HTTP to send synchronous request messages to WebLogic that require a distinct response. These messages are authenticated and can also be encrypted via SSL. Oracle Wallet functionality is used to hold authentication credentials for the NMS services that send these SOAP messages to WebLogic.

• Oracle Wallet functionality is used to hold authentication credentials for the NMS services that send SOAP messages to WebLogic. The nms-env-config script is typically used to configure all Oracle Wallet credentials required to support NMS Services..

• The node that hosts NMS services is typically supplied with a daily or weekly set of routine GIS (map) and CIS (customer model) updates. These updates are commonly in the form of flat files and are often transported via SFTP (Secure File Transfer Protocol) from the appropriate project specific servers to the NMS services server.

• Oracle recommends the NMS admin account pull routine external updates (GIS, CIS, and so on) into NMS - rather than those accounts push updates into NMS. Minimizing external account access to NMS admin accounts further reduces the potential attack surface.

• Not pictured in Fig. 2 above is a common connection from one of the NMS corbagateway processes to a Simple Mail Transport Protocol (SMTP) server. This is a not always but often used configuration to send utility customer interruption notices and texts to utility internal account reps or other interested parties (via the NMS Service Alert product component option). Outgoing SMTP traffic from CORBA Publisher (corbagateway in publisher mode) is not encrypted.

• Typically NMS services are deployed on hardware vendor supported failover (active/passive) clustered infrastructure.

• NMS services can also be deployed on a standalone server - possibly using one or more Disaster Recover/Business Continuity sites for backup.

• For development, test, training, or model validation environments, NMS is often deployed on a server supporting other (similar category) NMS instances. Each NMS instance can be configured to run on a (mostly) distinct set of ports. This generally includes NMS instance specific ports for ISIS, the CORBA Object Request Broker (ORB), SOAP over HTTP and the NMS to WebLogic HTTP server. For a node supporting multiple NMS instances, each NMS instance would be deployed under a unique Unix user name (for example, nmsdev, nmstrain, nmstest). Multiple (non-production) NMS instances are often hosted on a single Oracle RDBMS (under separate schemas).

• In a dual-environment NMS configuration, two Unix user names (for example, nmsadmin1 and nmsadmin2) will each have an independent set of NMS executable files for a given NMS instance.

• WebLogic Managed Servers are used to host NMS specific Enterprise Java Beans (EJBs) that generally cache updates from NMS services or the RDBMS. Caching model updates to improve access for many (hundreds) of NMS clients is a primary goal for this tier.

• Separate WebLogic Managed Servers (within the same WebLogic domain) are also used to support Web Service based integration to external systems - typically via the NMS MultiSpeak Adapter (AMI, AVL, SCADA).

• Separate WebLogic Managed Servers (in different WebLogic domains) can be used to support NMS Operations Mobile Adapter (mobile) clients or NMS Flex (browser) clients. These NMS OMA/Flex specific WebLogic Managed Servers communicate to a matching internal WebLogic Managed Server - often behind a firewall. All communication between WebLogic Servers is initiated from the internal WebLogic server via a set of JMS queues to the "external" WebLogic Managed Server. This allows communication from the inside out without opening a hole in the firewall from the outside in. In either case, if different WebLogic domains are used, domain trust must be established between the WebLogic Managed Servers involved.

• WebLogic uses CORBA to communicate to the NMS services CORBA adapter (corbagateway). CORBA IIOP traffic to/from the NMS CORBA adapter (corbagateway) is not encrypted.

• WebLogic uses the Java Database Connectivity (JDBC) Thin Client driver (for a non-RAC RDBMS) or Active Gridlink driver (for RAC RDBMS) to communicate to the NMS Oracle RDBMS. These JDBC connections are generally not encrypted but can be via the Oracle Enterprise Edition RDBMS Native Network Encryption option.

• WebLogic uses an NMS instance specific port to communicate to an HTTP or HTTPS server on the NMS services node to pull in NMS map files to serve to NMS clients (lighttpd is typically used as the HTTP/HTTPS server).

• One WebLogic managed server (or WebLogic cluster) instance is typically used to support NMS Java clients. If necessary it is possible to configure multiple WebLogic managed servers (on different nodes - typically behind some type of load balancing switch) to support a larger number or different categories of NMS Java clients.

• If you have distinct categories of NMS end users different WebLogic Managed Servers can also be configured to support those specific categories of users.

• For example, you may have one set of NMS end users that are allowed to send outbound SCADA control (open/close) SCADA requests. A specific WebLogic server (or WebLogic cluster) can be configured to specifically allow outbound controls. Other (less privileged) NMS end users can be forced to login via an alternate WebLogic server (or WebLogic cluster) that does not allow outbound controls.

• This type of configuration typically requires some form of network segmentation. This is to ensure that ONLY qualified/privileged/appropriate NMS end users with physical access to a given control room (for example) – are allowed to initiate outbound controls. This also requires very NMS specific corba-gw and WebLogic configuration – to ensure that non-privileged NMS users cannot initiate outbound controls.

• NMS end users can also use this scheme for other reasons. One WebLogic server to support primary control center users and another to support “non-control-center” users – for example. This can be a mitigation strategy to provide more consistent (dedicated) resources for core (control room) NMS end user access. It also provides an additional access option for at least some NMS end-users. Even if one WebLogic server is not available (being maintained or reset), at least some NMS end users can continue to access NMS via the other – for example.

• NMS clients are authenticated and authorized via WebLogic. WebLogic typically maintains a connection to an enterprise (or operations specific) Lightweight Directory Access Protocol (LDAP) accessible set of Directory Services - for NMS end user authentication. LDAPv3 traffic should be configured to be encrypted.

• A typical production NMS installation serves a broad spectrum of utility users - often too many to put them all behind an internal firewall. For a more secure NMS implementation it is recommended that WebLogic managed servers (along with the RDBMS and NMS services that support their NMS Java clients) be separated from their NMS Java clients by an additional internal firewall. The firewall ensures the WebLogic Managed Servers supporting NMS Java clients are minimally accessible outside of the internal firewall. The only port that is required to be opened is the port configured for SSL(HTTP/t3s) access in WebLogic.

• A separate (additional) WebLogic Managed Server instance (or cluster) can also be used to communicate to other (external) systems. For external integration, NMS specific EJBs generally communicate to external systems via a Web Service layer on top of the NMS EJBs.

• This additional WebLogic managed server generally runs on the same node and WebLogic domain but on a different port than the WebLogic managed server that supports NMS Java clients. Running the WebLogic Managed Server used for external NMS integration on a separate port also hides this port from the general user population (if the recommended firewall between NMS end users and WebLogic is in place).

• Generally only one WebLogic managed server at a time can be used to integrate to a given external system - generally via some type of Web Service interface. The default Oracle NMS WebLogic managed server (supporting a MultiSpeak-based integration, for example) can be configured to utilize a WebLogic cluster.

• WebLogic is typically deployed on hardware vendor supported failover (active/passive) Unix cluster infrastructure. Using a Unix cluster is not necessary but provides protection against local hardware failure as well as a mechanism to support "rolling" infrastructure upgrades (upgrade one node of the Unix cluster at a time - within reason). Keep in mind that Unix clusters and WebLogic clusters are independent and have no direct dependency on each other.

• Unix clusters are also generally convenient if/when a WebLogic managed server is configured to integrate to other (project specific) EJB servers. Since only one WebLogic managed server at a time can be configured to integrate to external EJB servers - a Unix failover cluster provides a useful platform to host the "active" WebLogic managed server (floating IP addresses managed by the cluster - for example).

• If multiple concurrent WebLogic Servers are desired to support integration with an external system (typically a MultiSpeak integration), then the WebLogic Servers must be configured in a WebLogic cluster since the adapter uses an EJB singleton service to support external integration. A WebLogic singleton service requires either a single WebLogic instance or a WebLogic cluster.

• If the Oracle NMS Web Switching module is in play along with multiple concurrent WebLogic Managed Servers (for whatever reason), WebLogic Clustering must be used. This is because Oracle NMS also requires an EJB Singleton to support the Web Switching module.

• The WebLogic managed server(s) used to support NMS clients can be hosted on one or more (non-Unix-clustered) nodes. This is true even if/when WebLogic Enterprise Edition (or SOA Suite) is used in a WebLogic clustered configuration. WebLogic clusters do not have a dependency on hardware clusters - they do not need to share a common set of configuration or data files.

• If two or more WebLogic managed servers are used to support NMS clients some form of load balancer is generally required to route incoming traffic to the available WebLogic managed servers.

• In dual-environment configuration (for reduced downtime for NMS patching), each environment will use a different WebLogic managed server.

• NMS clients connect to the NMS WebLogic managed server and provide end users NMS electrical network model access and update options.

• The NMS Java clients support a very wide range of user interface configuration options to accommodate project specific business practices. Configuration options include Java files, RDBMS tables, NMS specific XML (jbot) files and property files.

• NMS Flex (browser) clients are a lighter weight NMS end user access option. They offer a subset of the functionality and configuration options offered by the NMS Java client.

• NMS clients only communicate to the NMS WebLogic Managed Server. Any subsequent communication to the RDBMS or NMS services is managed within WebLogic.

• The RDBMS access that an NMS client can configure is accomplished within WebLogic using a dedicated JDBC connection pool using a specific read-only RDBMS user/schema name. The read-only RDBMS user/schema has project defined private synonyms that reference a configurable subset of the primary NMS RDBMS schema.

• NMS Java clients receive updates from WebLogic by periodically polling the WebLogic managed server. The NMS Java clients use Remote Method Invocation (RMI) calls to facilitate this communication. The RMI calls between NMS Java clients and WebLogic should be configured as encrypted (t3s/https) for a production NMS deployment.

• NMS optional Operations Mobile Adapter (OMA) or Flex clients also receive updates from WebLogic by periodically polling a respective WebLogic Managed Server. Both NMS OMA and Flex clients make periodic RESTful calls using t3s/https to WebLogic to facility requests and/or updates.

• NMS Flex clients utilize very similar infrastructure to NMS OMA clients - including a dedicated WebLogic Flex Gateway WLFG. The WLFG can utilize WebLogic clustering. The WLFG can be deployed within the same domain as the "internal" WebLogic Managed Server that supports it or on a separate WebLogic domain (typically outside some internal firewall). If the WLFG is deployed on a separate WebLogic domain then domain trust must be configured between the WLFG domain and the internal WL Managed Server domain that supports it. The use of domain trust in this manner is entirely similar to OMA.

• NMS Java clients are authenticated via WebLogic. Typically, WebLogic uses a Secure LDAP connection to some form of Enterprise Directory Server (like Active Directory) for authentication, but any authentication mechanism supported by WebLogic can be used.

• NMS Java clients can also be configured to support two-factor authentication. Two-factor authentication requires project specific configuration.

• It is recommended that a firewall be placed in front of the WebLogic server that provides access for NMS Java, OMA, or Flex clients. Only a single port needs to be kept open in the firewall to allow clients access to WebLogic. Note that this firewall should also block all direct end-user access to the RDBMS and NMS services tiers that NMS WebLogic instance leverages.

• For a production NMS instance only HTTPS access for the NMS clients should be allowed. Note that HTTPS equates to the t3s protocol on WebLogic.

• NMS Java clients are typically accessed via Java Web Start. This scheme requires a web page (hosted by the NMS WebLogic managed server) that lists project configured NMS end user environments. Each link references a Java Network Launch Protocol (JNLP) file. Each JNLP file contains commands to download, cache and execute the appropriate Java application. Each time Java Web Start initiates a Java client application it automatically validates that the most current version of the Java client application is being used - making this a convenient option for administration. To avoid man-in-the-middle attacks customers are asked to use JNLP over an HTTPS connection to WebLogic when they download the NMS Java client.

• If a project wants more control over which workstations have NMS access the NMS Configuration Assistant application can be used to generate project specific NMS Java client installers. It is then up to local NMS administrators to distribute and manage these installers for the appropriate NMS end user workstations. This includes ensuring the appropriate version of the various NMS Java clients are installed. Changes to the project configuration will automatically be picked up the next time a client starts (NMS Java client configuration is handled separately from client versions). However, if there is a new product version of NMS installed, new installers will need to be generated, distributed, and installed.

• If NMS Java clients need to run over high-latency, low-bandwidth or even an external connection (typically via a WAN or VPN) NMS Java clients can also be deployed using Windows Remote Desktop Services (or similar) technology.

• This type of configuration allows the CPU, memory and network resources needed to run NMS Java clients to be more centralized - and possibly more performant.

• This type of configuration also provides a more secure deployment option as no NMS Java client software executes on the end-user hardware. Only the technology specific remote viewer client executes on the NMS end user workstation - generally over an encrypted protocol. For this type of deployment no NMS specific software need ever be installed on the end user workstation.

• The Oracle NMS Operations Mobile Application (OMA) is an (optional) NMS product that provides mobile (tablet) access to the real-time NMS electrical network model. OMA includes a Javacript based template that a given NMS customer can update/configure to match business processes. The Oracle NMS provided OMA template is also updated on a regular basis in an effort to minimize any project specific effort required to deploy and maintain for a given customer.

• The Oracle NMS OMA application has access to an (expanding) subset of NMS model data and functionality. OMA access typically includes the ability to integrate with the NMS Switching application to allow control room operators to more reliably coordinate field activity. This coordination allows an OMA user to update at least certain aspects of the NMS model directly (confirm a field switching operation as part of an existing NMS switch plan, for example). OMA can also be configured to directly manipulate the NMS model (open/close fuses, for example).

• OMA often uses standard cellular data technology to route mobile requests/responses via the Internet through an Oracle NMS WebLogic "Mobile Gateway".

• Since OMA users are typically "in-the-wild" (not on any kind of internal network behind a corporate firewall) special care must be taken to ensure only properly authorized OMA users get meaningful access to the NMS Mobile Gateway. OMA users communicate to the NMS Mobile Gateway using an encrypted RESTful protocol (over HTTPS).

• Once OMA client requests have landed in the NMS WebLogic Mobile Gateway they are placed on a JMS request queue. A (separate) properly configured Oracle NMS WebLogic Managed Server will monitor the JMS request queue and download validated OMA requests. If the request is valid it will be processed and a response will be placed on a matching JMS response queue back on the WebLogic Mobile Gateway and back to the OMA client.

• Both the OMA request and response JMS queues are managed on the NMS WebLogic Mobile Gateway and are always accessed from the (separate) internal NMS WebLogic Managed Server. This scheme eliminates the need to open up ports from the outside in. The only ports that are open are from the inside out. This is typically a more secure deployment model and also easier to shutdown if/when necessary.

• To shutdown OMA entirely an NMS admin can simply stop the internal WebLogic server from accessing the external JMS queues in the NMS WebLogic Mobile Gateway. No firewall changes (or network admin privileges) are required.

• The outward facing Oracle NMS WebLogic Mobile Gateway should be deployed on a separate (physical) server in a separate security zone from the (internal) Oracle NMS WebLogic Managed Server - with at least one firewall between them.

• The firewall in front of the internal NMS WebLogic Managed Server should have NO ports open from the outside in - only from the inside out.

• It is recommended to have some form of reverse proxy server and firewall between the NMS WebLogic Mobile Gateway and the OMA end users coming in from the Internet - to further isolate the WL Mobile Gateway.