This chapter describes the Oracle DIVArchive 7.6 release, and includes the following information:
The DIVArchive architecture enables integration of many different types of servers and technologies, for example Broadcast Video Servers, Storage Area Networks, and Enterprise Tape Libraries. The DIVArchive installation varies from site to site, therefore the exact configuration of your specific DIVArchive platform is not described in this book. Refer to your DIVArchive Site Configuration document for details on your specific DIVArchive system installation and configuration.
The Site Configuration is a document prepared by Oracle (or an affiliated system integrator) after installation at your site. It is a complete record of the configuration of the DIVArchive sub-systems and third party interfaces, site details and contacts, user names and passwords, and remote access codes.
The Site Configuration (for your site) is referenced at various points in this document, such as when user names and passwords are required for access to the DIVArchive software utilities.
Note:
The File System Interface is not released with DIVArchive 7.x and is only supported by special request.See Appendix A for DIVArchive options and licensing.
The following table lists the standard ports used by the DIVArchive system. If you need assistance contact Oracle Support.
Table 1-1 DIVArchive Port Utilization
| Service | Port Number | Description and Notes |
|---|---|---|
|
FTP |
|
Port depends on configuration |
|
SSH |
|
Linux hosts only |
|
HTTP |
|
DIVAdirector |
|
Oracle SQLNet |
|
Manager database access |
|
RDP (Microsoft Terminal Services) |
|
Remote Desktop access |
|
Oracle DIVArchive Robot Manager |
|
Robot Manager |
|
Oracle DIVArchive Manager |
|
Manager Unsecure Port Manager Secure Port |
|
Oracle DIVArchive Actor |
|
Actor |
|
Flip Factory |
|
Flip Factory |
|
DIVArchive AMC |
|
Avid AMC |
Refer to the DIVArchive Release Notes in the DIVArchive documentation library at https://docs.oracle.com/en/storage/#csm.
This section describe the media storage formats available in this DIVArchive release.
AXF (Archive eXchange Format) is an open format that supports interoperability among disparate content storage systems and ensures the content's long-term availability no matter how storage or file system technology evolves.
An AXF object is an IT-centric file container that can encapsulate any number, and any type, of files in a fully self-contained and self-describing package. The encapsulated package contains its own internal file system, which shields data from the underlying operating system and storage technology. It's like a file system within a file that can store any type of data on any type of storage media.
Tape groups or disk arrays used by complex object requests must be in an AXF format, because complex objects cannot be stored in Legacy format. Because all complex objects are written in the AXF format, any instance of a complex object will also be in the AXF format.
In DIVArchive, a Tape Group or Disk Array has a media format parameter that indicates which storage media format to use when creating Archived objects. You can set the media format to either DIVArchive Legacy Format or the AXF Format. This setting can be changed at any time and does not influence content already stored. It is possible to have more than one storage media format within tape groups and disk arrays.
A DIVArchive object instance is only written in one media format. Therefore, if an object spans tapes, each tape used as part of an object instance will be written in the same media format. An object can contain multiple instances, each of which can be stored in either Legacy or AXF format.
Although a tape group can contain more than one storage format, an individual tape has at most one storage media format. The format of a tape instance is the format of the tape on which the instance resides. All instances on a tape must have the same format.
The media format for an empty tape is assigned when the first object on that tape is written. The tape is assigned the format of the tape group that appears in the request. After the media format for a tape is assigned, you cannot change it unless all objects on the tape are deleted. After deletion of all objects from a tape, the tape's format becomes unassigned until content is again written to the tape. If the tape was in use, the tape format cannot change unless it is empty and cleared.
Both Legacy and AXF formatted tapes can exist in the same group. Nevertheless, objects in AXF format will only be written to AXF formatted tapes, and objects in Legacy format will only be written to Legacy formatted tapes, even though they are in the same tape group.
Note:
A Repack request will always write the destination tape in the same media format as the source tape.Similarly, tape spanning operations will always use the same format across all tapes storing spanned objects. If an instance spans across multiple tapes, then all tapes used to span the content will have the same format.
Unlike tapes, disks do not have a format. DIVArchive allows storing objects in different media formats on the same disk. If a disk contains objects in Legacy format, and that disk is then assigned to an AXF formatted array, it will still contain objects in Legacy format. However, new objects written to the disk will be in AXF format.
If a disk instance is non-complex and permanent (not a cache instance), it is stored in the format of the destination array. If a cache instance is non-complex, it is stored in the format of the group specified in the request.
You can use the Copy To Group, or Copy As New requests to migrate objects from Legacy media format to AXF media format (or back). However some AXF objects cannot be copied to the Legacy format; copying objects from Legacy format to AXF format does not present any issues. In DIVArchive the only limitation on copying an object instance from AXF format to Legacy format is the complex object feature.
Complex objects have significantly expanded the object component boundaries, allowing up to one million files and ten thousand folders per object.
Complex objects maintain information about files and folders in the archive. They store subtotals for each folder, including the total number of files and subfolders within the folder, and the total size of all files within the folder and within any subfolders.
DIVArchive uses the configurable Complex Object Threshold parameter during archival to determine whether a new object should be complex based on the number of components. This value is set in the manager.conf configuration file. If the number of components is greater than the Complex Object Threshold, the object becomes a complex object. After an object is identified as a complex object it will always be complex; even if it is copied using the Copy As command, or imported using the Export/Import Utility.
Oracle recommends that the threshold remain at the default value (1,000 components) unless there is a specific reason to adjust the value. Contact Oracle Support for assistance as required.
A complex object differs from a non-complex object in several key ways. For example, the file and folder metadata information of a complex object is stored in a file, not in the Oracle Database. The file contains the file names, folder names, checksums, and files sizes. The files are located in the Metadata Database root directory. Complex objects must be stored in AXF format whether on tape or on disk.
Complex objects can contain hundreds of thousands of files. However, some DIVArchive API commands (for example, GetObjectInfo) will not return the entire set of files. Instead, these commands return a single placeholder file which prevents downstream applications from being overwhelmed by file and folder information. Also, the entire set of files on a tape are not displayed in the Control GUI Object Properties and Tapes screens, only a single placeholder file is shown. The DIVArchive API includes a command to return all of the files and folders within a complex object. See the appropriate Oracle DIVArchive API documentation in the Oracle DIVArchive documentation libraries for details.
Oracle DIVAnet does not currently support replication of complex objects.
The following features do not support complex objects:
Delete on Source option
Verify on Restore (VFR) checksum feature
Verify on Archive (VFA) checksum feature
deleteFile API call
getObjectListbyFileName API call
GetByFilename API call (for Avid connectivity)
DeleteByFilename API call (for Avid connectivity)
When archiving complex objects using the FTP protocol, and an FTP Client with default settings (FileZilla is recommended), the transfer will typically fail when archiving any object with more than approximately 3,900 files.
Occasionally, during the directory scan, the Actor connection times out before the size of the object can be computed. More often, a request terminates in the middle of the transfer because the FTP server is consuming all of the available sockets.
You can add the following parameters in the Source/Destination Command Options or in the Options of the command itself to resolve timeout issues:
-transfer_timeout 1200 -list_timeout 600
See Appendix C for detailed parameter information.
Use the following procedure to include the parameters in the Source/Destination frame in the Configuration Utility:
Open the DIVArchive Configuration Utility.
Navigate to the System tab.
Double-click the desired Source/Destination in the Sources and Destinations frame to open the edit dialog box.
Add the two parameters (-transfer_timeout 1200 and -list_timeout 600) in the Connect Options field.
Click OK to save the changes.
Notify the Manager of the changes using the Control+N key combination.
Oracle recommends setting the following corresponding parameters in the FileZilla server under General Settings:
Connections Timeout = 600
No Transfer Timeout = 1200
Open the FileZilla server interface.
Click the Server Options icon on the tool bar.
Adjust the settings in the General Settings area.
If requests terminate unexpectedly during transfers, adjust the Windows Registry parameters as follows:
Open regedit.
Modify (or create) the following values under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters:
TcpTimedWaitDelay = 10 MaxUserPort = 90000
If the desired registry parameter does not exist, right-click and create a new double word (DWORD) value.
If the parameter does exist, double-click it and enter the values.
Restart the computer to enable the new registry values.
DIVArchive includes the core software components discussed in the following subsections. All core DIVArchive components support Oracle Linux 7 x86_64 and later. See the Oracle DIVArchive Supported Environments Guide in the Oracle DIVArchive Core documentation library for information about certain limitations when running in the Linux environment.
Long path names are supported on both Windows and Linux. Absolute path names are supported on both Windows and Linux to a maximum of 4000 characters. Relative path names are limited to 256 characters on Windows systems (only).
All Windows batch files (.bat) have corresponding shell scripts (.sh) in Linux. You must substitute Windows paths with Linux paths when operating on Linux. For example, the Windows path C:\DIVA\Program equates to /home/diva/DIVA/Program in the Linux environment.
Note:
Linux commands, paths, and file names are case-sensitive.Archive and restore operations of symbolic links are now supported in Linux. Shortcuts created using the Windows operating system are not represented as symbolic links because they are treated as files. Only symbolic links created on the UNIX platform are archived and represented as symbolic links in DIVArchive.
The Java and C++ APIs file list returned from a getFilesAndFolders call includes symbolic links, and the export and import operations type attribute contains the letter S to represent a symbolic link.
The following features require Windows-based Actors:
Oracle DIVArchive Avid Connectivity
Transcoder integration
Tape Reading Utility
On Linux actors, standard commands like DD and MT are alternatives of Tape Reading Utility. Linux Actors support QuickTime, GXF, and MXF and MPEG2 Transport stream wrappers for Oracle DIVArchive Partial File Restore (video). See Appendix A for Oracle DIVArchive options and licensing information.
Due to degraded performance, Windows IIS and FileZilla FTP sources and destinations cannot be used for complex objects. Oracle only supports Linux-based FTP servers when operating in a Linux environment. The Windows IIS and FTP servers cannot accommodate large numbers of files.
The DIVArchive Manager is the main component in a DIVArchive system. All archive operations are controlled and handled by the DIVArchive Manager. Operation requests are sent by initiator applications through the DIVArchive Client API. DIVArchive supports Main and Backup DIVArchive Managers (see Appendix A for Oracle DIVArchive options and licensing information).
The DIVArchive Actor is the data mover between devices in your production system. Actor supports interfacing and data transfer between many different types of devices.
All Actor operations are initiated and coordinated by the DIVArchive Manager through a TLS 1.2 secure connection. Key benefits of the distributed design of the DIVArchive Actors are:
You can expand the archive subsystem to increase the overall bandwidth by adding more Actors to the system. See Appendix A for Oracle DIVArchive options and licensing information.
You can share SAN based disk and tape drive resources among multiple Actors.
In combination with the DIVArchive Manager, multiple Actors provide scalability, load balancing, redundancy, and failover. You can take individual Actors offline for maintenance without shutting down the DIVArchive system.
Note:
UNC paths are supported for SMB Source/Destinations and managed disks if the UNC path is mounted directly on Windows Actors.DIVArchive 7.5 and later supports archive and restore of empty files and folders. Empty files and folders are only supported by AXF. When Legacy format is in use, DIVArchive reports an error if an empty file or folder is discovered during the transfer.
The DIVArchive Client APIs are a set of functions enabling external applications, acting as clients, to use the services offered by the DIVArchive system.
A library of client functions is provided with the selected API and must be linked to each DIVArchive client application. These functions encapsulate client commands into DIVArchive request messages sent over a TCP/IP connection to the DIVArchive Manager.
Currently available APIs include C++, Java, and Web Services (Oracle DIVA Enterprise Connect). Refer to the appropriate Oracle DIVArchive API documentation in the Oracle DIVArchive Additional Features documentation library, and the Oracle DIVA Enterprise Connect documentation library for more information.
The DIVArchive software is bundled with an Oracle database installation. The database stores all information relating to the DIVArchive system including its configuration. SQL queries used by the Manager are optimized to support configurations with up to 58 million components.
In DIVArchive 7.5 and later, the JDBC Thin Driver enables replacing the Oracle SID setting with the Oracle Service Name.
When installing DIVArchive in a 64-bit environment, the latest 64-bit DIVArchive Oracle release must be installed to use 64-bit support.
DIVArchive 7.6.2 supports Oracle Database 11.2.0.4 or greater.
The Oracle database is not intended to be modified directly by customers, but rather by using Oracle utilities. Direct modification of this database by customers through Oracle utilities is not supported by Oracle.
See the Oracle DIVArchive Database User's Guide in the Oracle DIVArchive Core documentation library for detailed information and "Secure Communication with Oracle Database".
DIVArchive stores object metadata separately from the Oracle database in the DIVArchive Metadata Database. The metadata database contains files stored in a file system local to the DIVArchive Manager. The files are located in the Metadata Database root folder. This storage method enables DIVArchive to effectively operate with large volumes of files, folders and other metadata.
The metadata database is very high performance, and has almost unlimited scalability. You must treat the metadata database with the same caution as the Oracle database, and it must be backed up at regular intervals through the DIVArchive Backup Service.
The DIVArchive Control GUI connects to both the DIVArchive Manager and the DIVArchive database. You use it to monitor, control, and supervise operations in DIVArchive. you can operate multiple Control GUI instances simultaneously from any computer that has TCP/IP connectivity to both the DIVArchive Manager and the DIVArchive database.
The Control GUI is not intended for the intensive archive operations of a DIVArchive system. Archive operations are typically initiated to DIVArchive from a Broadcast Automation or MAM (Media Asset Management) system.
See the Oracle DIVArchive Operations Guide in the Oracle DIVArchive Core documentation library for more information on using the interface.
The refresh rate for the Control GUI is set in the Manager Setting tab of the Configuration Utility in the GUI: Dashboard Refresh Delay field.
Additional modules are available to expand your DIVArchive system capabilities. Most of these options are currently covered in separate documents, but are briefly described here for completeness. See Appendix A for DIVArchive options and licensing information.
You can use DIVArchive to only manage disk storage, but storage capacity can be further expanded by adding one or more tape libraries. In these cases, the DIVArchive Robot Manager module provides an intermediate software layer for the DIVArchive Manager to interact with many different types of tape libraries. It is connected to the DIVArchive Manager through TCP/IP. See Appendix A for DIVArchive options and licensing information.
This distributed architecture provides substantial flexibility including:
Libraries controlled using a SCSI interface are limited by the cable length. Because the connection to the DIVArchive Robot Manager from the DIVArchive Manager is over TCP/IP, the library does not need to be colocated near the DIVArchive Manager host computer.
Enabling installation of multiple, or dissimilar, libraries by configuring additional DIVArchive Robot Manager modules.
Enabling rapid development to support new types or models of libraries.
You can restart the robotics interface without needing to restart the DIVArchive Manager.
The DIVArchive Robot Manager interfaces with the library using either a direct interface to the library itself (through native SCSI, or SCSI over Fibre Channel), or through an intermediate Ethernet connection to the manufacturer's own library control software.
VACP (Video Archive Communications Protocol) is developed by Harris Automation Solutions and used by some automation systems for interfacing to an archive system. DIVArchive has its own API for communicating with the DIVArchive Manager, which is not compatible with VACP.
To provide interoperability without the need to redevelop the archive interface at the automation level, this module is provided to act as an interface to convert VACP commands from the attached automation system to DIVArchive API commands on computers that have TCP/IP connectivity to DIVArchive.
The SPM (Oracle DIVArchive Storage Plan Manager) provides automatic migration and life cycling of material within the archive, based on the rules and policies defined in the SPM configuration. The DIVArchive DSM (Disk Space Monitor) works with SPM to delete material from SPM managed arrays (based on disk space watermarks).
The DIVArchive SNMP (Simple Network Management Protocol) interface supports status and activity monitoring of different DIVArchive components. DIVArchive MIB (Management Information Base) is provided to third party SNMP monitoring applications. The SNMP Agent uses the Windows SNMP Service and has not been ported to the Linux environment.
The DIVArchive Access Gateway provides DIVArchive client authentication and authorization. It can act as an intermediate gateway between DIVArchive components (for example the VACP converter) or third party applications and the DIVArchive Manager, and can restrict that component or application from access to the DIVArchive system.
Oracle DIVAnet is a powerful feature that allows multiple DIVArchive platforms to exchange archive resources and content, whether the archive systems are local to each other or remote.
The Access Gateway is used in DIVAnet installations and is the portal for multiple DIVArchive systems to communicate with each other. See the Oracle DIVAnet Installation, Configuration, and Operations Guide in the Oracle DIVAnet 2.2 documentation library for more information.
The DIVArchive DFM (Drop Folder Monitor) provides automatic monitoring of newly created files in multiple local directories or FTP folders (or combinations thereof). One file, or multiple files, per DIVArchive object are supported. When a new file is identified, DFM issues an archive request automatically to DIVArchive to archive the new file. After the files are successfully archived, they are then automatically deleted from the source. Refer to the Oracle DIVArchive Drop Folder Monitor (DFM) User's Guide in the Oracle DIVArchive Additional Features documentation library for more information.
When DFM is used in a Linux environment to monitor an FTP folder, you must configure it as in the following example:
diva
/ifs
/ifs/folder1
A correct DFM configuration with these parameters is:
ftp://diva:password@host_ip/folder1
An incorrect DFM configuration with these parameters is:
ftp://diva:password@host_ip/ifs/folder1
The DIVArchive Actor can integrate with a transcoder engine to provide real time transcoding of material as it is archived or restored, or to create objects from already existing content within the archive. Currently, integration to Bitscream products, and Telestream Flip Factory, and Telestream Vantage are supported. However, multiple transcoders are only supported for Vantage.
Note:
DIVArchive 7.5 ended Oracle support for Telestream Flip Factory. Oracle will provide best efforts to assist customers to transition to other transcoding solutions.Linux-based Actors only support Telestream Vantage for transcoding operations.
DIVArchive assumes a local transcoder address of 127.0.0.1 if a transcoder address is not specified in the transcoder's working directory.
The Promedia Carbon (formerly Rhozet) transcoder is supported in DIVArchive. You select the transcoder type tre from the Configuration Utility to use this transcoder. Both the Name and GUID are supported as options for Presets and Profiles format types.
The following sections describe general Avid connectivity with DIVArchive.
See the Oracle DIVArchive Avid Connectivity User's Guide in the Oracle DIVArchive Additional Features documentation library for more information. Also see Appendix A for DIVArchive options and licensing information.
The Avid DHM (Data Handler Module) interface support in DIVArchive enables finished content to be shared between post-production Avid environments and On Air Video servers. This eliminates the need for tape based content exchange. Timecode based Partial File Restores of content to On Air environments, and finished Avid Sequence submissions to On Air servers are key to the DHM functionality offered within DIVArchive. DHM support is implemented in DIVArchive TMC (Transfer Manager Communicator).
The Avid DET (Dynamically Extensible Transfer) interface support in DIVArchive allows storage expansion of Avid Unity infrastructures and enables editors to move native Avid content in and out of the DIVArchive storage system. Partially edited content stored within DIVArchive through the Avid DET interface can be later restored to Unity, and an editor can then resume editing at the point where they stopped. DIVArchive stores these files in native Avid format. DET support is implemented in DIVArchive TMC (Transfer Manager Communicator).
An interaction between the Avid Archive Manager solution and DIVArchive is implemented in a separate service called AMC (Archive Manager Communicator). AMC handles Archive, Restore, Partial File Restore, and Delete commands from the Avid Archive Manager using DIVArchive to store Avid content in its native MXF OP1 Atom format.
The DIVAprotect option is a utility that collects operational statistics from the DIVArchive system to monitor and maintain the archive's subcomponents (servers, media, drives, tapes, and so on). Analysis of these statistics allows both proactive and reactive maintenance of the DIVArchive system. See the Oracle DIVArchive DIVAprotect User's Guide in the Oracle DIVArchive Additional Features documentation library for more information.
The OTU (Object Transfer Utility) is an optional feature of the Control GUI providing a drag and drop interface to archive and restore content between DIVArchive and a (supported) source or destination server. See Appendix A for DIVArchive options and licensing information.
The following sections describe utilities available in the DIVArchive system
You use the DIVArchive Configuration Utility to configure the DIVArchive system. It connects directly to the DIVArchive database, and can be run on any computer with TCP/IP connectivity to the host running the DIVArchive database.
During configuration and troubleshooting of the library and its tape drives, DIVArchive provides both a command-line interface and GUI utility to send commands directly to the tape library through the Robot Manager. These utilities are not (and must not be) used while the DIVArchive Manager is running because this can adversely affect archive operations. See Appendix A for DIVArchive options and licensing information.
The DIVArchive Backup Service ensures reliability and monitoring of both the Oracle Database and Metadata Database backups.
The DIVArchive Backup Service component is installed as an integral part of the standard DIVArchive system installation. The component is typically installed on the same server as the DIVArchive Manager and Oracle Database. The DIVArchive Backup Service enables configuration of scheduled backups through its configuration file. The DIVArchive Backup Service manages and monitors the entire backup process.
See the Oracle DIVArchive Database User's Guide in the Oracle DIVArchive Core documentation library for more information.
This utility is provided on both Windows and Linux platforms. It assists in obtaining detailed device information such as serial numbers, firmware releases, and SCSI information from tape libraries or tape drives for use in the DIVArchive configuration.
Caution:
This utility must not be used while the DIVArchive Manager is running.This utility is provided on both Windows and Linux platforms and is primarily used with the Robot Manager Client utilities to send manual Eject commands to a tape drive connected to an Actor. This utility also provides advanced tape based operations, such as tape formatting, but should only be used under guidance from Oracle Support.
The Tape Reading Utility is only supported by Windows-based Actors. You must use standard commands, for example, DD and MT when operating in a Linux environment.
This utility allows DIVArchive C++ API commands to be executed using UNIX or DOS based scripts. It is designed to run automated tasks for testing rather than for any intensive uses. There is no Linux-based DIVAscript release.
The DIVArchive RDTU (Recover Damaged Tape Utility) is designed to recover object instances contained on a damaged tape. The utility can recover instances that have valid copies on other available media (that is, internal tape, or a connected disk or array) within a local or remote DIVArchive system.