BEA Logo BEA Tuxedo Release 7.1

  Corporate Info  |  News  |  Solutions  |  Products  |  Partners  |  Services  |  Events  |  Download  |  How To Buy

 

   Tuxedo Doc Home   |   Security   |   Topic List   |   Previous   |   Next   |   Contents

   Using BEA Tuxedo Security

Sending and Receiving Encrypted Messages

Message-based encryption provides end-to-end data privacy. For a diagram that illustrates how it works, see the figure BEA Tuxedo PKCS-7 End-to-End Encryption.

A message is encrypted just before it leaves the originating process, and remains encrypted until it is received by the final destination process. It is opaque at all intermediate transit points (including operating system message queues, system processes, and disk-based queues) and during network transmission over inter-server network links.

Writing Code to Send Encrypted Messages

The following flowchart provides the procedure for writing code to send encrypted messages.

Procedure for Sending Encrypted Messages

For details about these steps and insight into how the system encrypts a message buffer, see the following topics.

Step 1: Opening a Key Handle for Encryption

Call the tpkey_open(3c) function or TPKEYOPEN(3cbl) routine to make the digital certificate of the target recipient available to the originating process. The target recipient might be a client, a service, a server group, a gateway group, a server machine, or an entire domain of servers.

When the originating process calls tpkey_open() to open the key handle, it specifies either the TPKEY_ENCRYPT or TPKEY_AUTOENCRYPT flag to indicate that the handle will be used to encrypt a message buffer. Typically, a client makes this call after calling tpinit(), and a server makes this call as part of initializing through tpsvrinit().

Opening a key handle with the TPKEY_AUTOENCRYPT flag enables automatic encryption: subsequently, the originating process encrypts message buffers automatically whenever they are sent. Using the TPKEY_AUTOENCRYPT flag is beneficial for three reasons:

The following example code shows how to open an encryption key handle. TPKEY is a special data type defined in the atmi.h header file.

Opening an Encryption Key Handle-Example


main(argc, argv)
int argc;
char *argv[];
#endif

{
TPKEY tu_key;
.
.
.
if (tpkey_open(&tu_key, "TOUPPER", NULL,
NULL, 0, TPKEY_ENCRYPT) == -1) {
(void) fprintf(stderr, "tpkey_open tu failed
tperrno=%d(%s)\n", tperrno, tpstrerror(tperrno));
exit(1);
}
.
.
.
}


Step 2 (Optional): Getting Key Handle Information

You may want to get information about an encryption key handle to establish the validity of the key. To do so, call the tpkey_getinfo(3c) function or TPKEYGETINFO(3cbl) routine. While some of the information returned may be specific to a cryptographic service provider, a core set of attributes is common to all providers.

The default public key implementation supports three algorithms for bulk data encryption of message content:

Encryption strength is controlled by the ENCRYPT_BITS key attribute, and the algorithm is controlled by the ENCRYPT_ALG key attribute. When an algorithm with fixed key length is set in ENCRYPT_ALG, the value of ENCRYPT_BITS is automatically adjusted to match.

The following example code shows how to get information about an encryption key handle.

Getting Information About an Encryption Key Handle-Example


main(argc, argv)
int argc;
char *argv[];
#endif

{
TPKEY tu_key;
char principal_name[PNAME_LEN];
long pname_len = PNAME_LEN;
.
.
.
if (tpkey_getinfo(tu_key, "PRINCIPAL",
principal_name, &pname_len, 0) == -1) {
(void) fprintf(stdout, "Unable to get information
about principal: %d(%s)\n",
tperrno, tpstrerror(tperrno));
.
.
.
exit(1);
}
.
.
.
}


Step 3 (Optional): Changing Key Handle Information

To set optional attributes associated with an encryption key handle, call the tpkey_setinfo(3c) function or TPKEYSETINFO(3cbl) routine. Key handle attributes vary, depending on the cryptographic service provider.

The following example code shows how to change information associated with an encryption key handle.

Changing Information Associated with an Encryption Key Handle-Example


main(argc, argv)
int argc;
char *argv[];
#endif

{
TPKEY tu_key;
static const unsigned char rc2_objid[] = {
0x06, 0x08, 0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x03, 0x02
};
.
.
.
if (tpkey_setinfo(tu_key, "ENCRYPT_ALG", (void *) rc2_objid,
sizeof(rc2_objid), 0) == -1) {
(void) fprintf(stderr, "tpkey_setinfo failed
tperrno=%d(%s)\n",
tperrno, tpstrerror(tperrno));
return(1);
}
.
.
.
}


Step 4: Allocating a Buffer and Putting a Message in the Buffer

To allocate a typed message buffer, call the tpalloc(3c) function. Then put a message in the buffer.

Step 5: Marking the Buffer for Encryption

To mark, or register, the message buffer for encryption, call the tpseal(3c) function. By calling this function, you attach a copy of the encryption key handle to the message buffer. If you open the key with the TPKEY_AUTOENCRYPT flag, each message that you send is automatically marked for encryption without an explicit call to tpseal().

Note: In COBOL applications, use the AUTOENCRYPT settings member to encrypt a message buffer. See TPKEYOPEN(3cbl).

The following example code shows how to mark a message buffer for encryption.

Marking a Message Buffer For Encryption-Example


main(argc, argv)
int argc;
char *argv[];
#endif

{
TPKEY tu_key;
char *sendbuf, *rcvbuf;
.
.
.
if (tpseal(sendbuf, tu_key, 0) == -1) {
(void) fprintf(stderr, "tpseal failed tperrno=%d(%s)\n",
tperrno, tpstrerror(tperrno));
tpfree(rcvbuf);
tpfree(sendbuf);
tpterm();
(void) tpkey_close(tu_key, 0);
exit(1);
}
.
.
.
}


Step 6: Sending the Message

After the message buffer has been marked for encryption, transmit the message buffer using one of the following C functions or COBOL routines:

Step 7: Closing the Encryption Key Handle

Call the tpkey_close(3c) function or TPKEYCLOSE(3cbl) routine to release the encryption key handle and all resources associated with it.

How the System Encrypts a Message Buffer

Just before a message buffer is sent, the public key software encrypts the message and attaches an encryption envelope; the encryption envelope enables the target recipient to decrypt the message. If a sealed buffer is transmitted more than once, encryption is performed for each transmission. This process makes it possible to modify a message buffer after marking the buffer to be encrypted.

The public key software encrypts the content of the message buffer and generates an encryption envelope for the recipient of the encrypted message by performing the following two-step procedure.

  1. {message_buffer_data + buffer_type_string + buffer_subtype_string}session_key = encrypted_message

  2. {session_key}recipient's_public_key = encrypted_session_key = encryption_envelope_for_recipient

The notation {something}key means that something has been encrypted or decrypted using key. In Step 1, a message buffer is encrypted using the session key, and in Step 2, the session key is encrypted using the recipient's public key.

Multiple Message Recipients

More than one encryption envelope can be associated with a message buffer, which means that multiple recipients, with different private keys, can receive and decrypt an encrypted message. A recipient can be a person or a process. When a message is encrypted for multiple recipients, it is encrypted only once, but the session key is encrypted with the public key of each recipient. All encryption envelopes are attached to the encrypted message.

If several encryption envelopes are associated with one message buffer, all of them must use the same symmetric key algorithm and the same key size for that algorithm.

Encrypted Message Content

An encrypted message buffer is represented in the PKCS-7 format as a version 0 EnvelopedData content type. The EnvelopedData content type, as used by the BEA Tuxedo system, consists of the following items:

The following figure shows the envelope hierarchy for the EnvelopedData content type. The SignedData content type is part of the hierarchy only if the message to which it belongs has one or more associated digital signatures.

EnvelopedData Content Type

As shown in the preceding figure, a message buffer may be both signed and encrypted. No relationship is required between the number of digital signatures and the number of encryption envelopes associated with a message buffer.

When both processes are performed on a message buffer, signatures are generated first, on unencrypted data. The number of attached signatures and the identity of signing parties are then obscured by the bulk data encryption.

Note: A suitable decryption key must be available to access message data before signatures can be verified.

Writing Code to Receive Encrypted Messages

The procedure for writing code to receive encrypted messages consists of the following steps.

  1. Call tpkey_open() to open a key handle for the target recipient. tpkey_open returns a key handle to the recipient's private key and digital certificate.

  2. (Optional): Call tpkey_getinfo() to get information about the decryption key handle.

  3. (Optional): Call tpkey_setinfo() to change information associated with the decryption key handle.

  4. Call tpkey_close() to close the decryption key handle. tpkey_close() releases the key handle and all resources associated with it.

For details about these steps and insight into how the system decrypts a message buffer, see the following topics.

Step 1: Opening a Key Handle for Decryption

Call the tpkey_open(3c) function or TPKEYOPEN(3cbl) routine to make the private key and the associated digital certificate of the target recipient available to the receiving process. The receiving process might be a client, a service, a server group, a gateway group, a server machine, or an entire domain of servers.

An application administrator can configure the application's UBBCONFIG file such that decryption key handles are opened automatically when the application is booted. No more than one decryption key handle per server may be used with this method. See Initializing Decryption Keys Through the Plug-ins for details.

If an application is not configured to open a decryption key handle for the receiving process during booting, the receiving process initiates its own tpkey_open() call. Or, if the receiving process wants to open another decryption key handle, the receiving process makes an additional tpkey_open() call.

In order to access the target recipient's private key, the receiving process must prove its right to act as the target recipient. Proof requirements depend on the implementation of the public key plug-in interface. The default public key implementation requires a secret password from the calling process.

When the receiving process calls tpkey_open() to open the key handle, it specifies the TPKEY_DECRYPT flag to indicate that the handle will be used to decrypt a message buffer. Typically, a client makes this call after calling tpinit(), and a server makes this call as part of initializing through tpsvrinit().

The following example code shows how to open a decryption key handle. TPKEY is a special data type defined in the atmi.h header file.

Opening a Decryption Key Handle-Example


TPKEY tu_key;

tpsvrinit(argc, argv)
int argc;
char **argv;
#endif
{
char *tu_location;
.
.
.
if (tpkey_open(&tu_key, "TOUPPER", tu_location,
NULL, 0, TPKEY_DECRYPT) == -1) {
userlog("Unable to open private key: %d(%s)",
tperrno, tpstrerror(tperrno));
return(-1)
}
.
.
.
}


Step 2 (Optional): Getting Key Handle Information

You may want to get information about a decryption key handle to establish the validity of the key. To do so, call the tpkey_getinfo(3c) function or TPKEYGETINFO(3cbl) routine. While some of the information returned may be specific to a cryptographic service provider, a core set of attributes is common to all providers.

The following example code shows how to get information about a decryption key handle.

Getting Information About a Decryption Key Handle-Example


TPKEY tu_key;

tpsvrinit(argc, argv)
int argc;
char **argv;
#endif
{
char principal_name[PNAME_LEN];
long pname_len = PNAME_LEN;
.
.
.
if (tpkey_getinfo(tu_key, "PRINCIPAL",
principal_name, &pname_len, 0) == -1) {
(void) fprintf(stdout, "Unable to get information
about principal: %d(%s)\n",
tperrno, tpstrerror(tperrno));
.
.
.
exit(1);
}
.
.
.
}


Step 3 (Optional): Changing Key Handle Information

To set optional attributes associated with a decryption key handle, call the tpkey_setinfo(3c) function or TPKEYSETINFO(3cbl) routine. Key handle attributes vary, depending on the cryptographic service provider.

The following example code shows how to change information associated with a decryption key handle.

Changing Information Associated with a Decryption Key Handle-Example


TPKEY tu_key;

tpsvrinit(argc, argv)
int argc;
char **argv;
#endif
{
TM32U mybits = 128;
.
.
.
if (tpkey_setinfo(tu_key, "ENCRYPT_BITS", &mybits,
sizeof(mybits), 0) == -1) {
(void) fprintf(stderr, "tpkey_setinfo failed
tperrno=%d(%s)\n",
tperrno, tpstrerror(tperrno));
return(1);
}
.
.
.
}


Step 4: Closing the Decryption Key Handle

Call the tpkey_close(3c) function or TPKEYCLOSE(3cbl) routine to release the decryption key handle and all resources associated with it.

How the System Decrypts a Message Buffer

The public key software automatically decrypts an encrypted message buffer whenever it enters a BEA Tuxedo client process, server process, or any system process that needs to access the content of the message buffer. For automatic decryption to succeed, the receiving process must have opened a decryption key (type TPKEY_DECRYPT) corresponding to a recipient identified in one of the attached encryption envelopes.

Upon receiving an encrypted message, the public key software, operating on behalf of the receiving process, performs the following tasks.

  1. Reads the target recipient's name on the attached encryption envelope.

  2. To recover the session key, decrypts the recipient's encryption envelope using the recipient's private key and the public key algorithm.

  3. Decrypts the message using the recovered session key and the symmetric key algorithm.

  4. Uncompresses the message.

  5. Verifies digital signatures if any. (See How a Signed Message Is Received.)

  6. If the message buffer successfully passes the check performed in Step 5, the public key software decodes the message buffer data, buffer type string, and buffer subtype string, and then passes the plaintext message to the receiving process. This step reverses the encoding performed by the originating process. (The BEA Tuxedo encoded format allows a message buffer to be decrypted on any machine architecture.)

    Note: If none of the attached digital signatures can be verified or the message buffer cannot be decrypted, the receiving process does not receive the message buffer. Moreover, the receiving process has no knowledge of the message buffer.

If a system process is acting as a conduit (that is, if it is not reading the content of the message), then the message need not be decrypted. Bridges and Workstation Handlers (WSHs) are examples of system processes acting as conduits.

The WSH is a special example of a conduit. If a WSH is configured for data-dependent routing, it needs to read the received message buffer to determine how to route the buffer. The public key software makes a copy of the received message buffer, decrypts the copy, and then passes the decrypted copy to the WSH. The WSH analyzes the decrypted copy to determine how to route the buffer, and then routes the original message buffer unchanged to the appropriate server. (For more detail about the interaction between data-dependent routing and public key security, see Compatibility/Interaction with Data-dependent Routing.)

Discarding an Input Buffer's Encryption Envelopes

If a message buffer is passed to an ATMI function (such as tpacall()) as an input parameter, the public key software discards any encryption envelopes previously attached to the message. This behavior prevents the target recipients for the original message from receiving any modifications made by an intermediate process.

As an example of this process, consider the scenario shown in the following figure.

Forwarding a Signed and Encrypted Message-Example

A server process named Manager receives a signed and encrypted message buffer from a client process named Employee, decrypts and reads the received message buffer, signs and seals it for a service named Purchasing, and then forwards the message to Purchasing.

The following is a detailed description of how this operation is performed.

  1. The Workstation Handler (WSH) receives the signed and encrypted message buffer from the employee and forwards it as is.

    The WSH process is configured for data-dependent routing, which is briefly described in How the System Decrypts a Message Buffer. The public key software uses a decryption key previously opened for the WSH process to decrypt a copy of the received message buffer, and then passes the decrypted copy to the WSH. After analyzing the decrypted copy, the WSH routes the received message buffer to the Manager process as is.

    If the WSH process is not configured for data-dependent routing, the Employee process does not need to tpseal() the message buffer for the WSH process, and the WSH process does not need to open a decryption key.

    Regardless of how it is configured, the WSH does not verify digital signatures.

  2. When the message buffer arrives at the Manager process, the public key software:

    1. Decrypts the message buffer using a decryption key previously opened for the Manager process

    2. Verifies the employee's signature

    3. Passes the message without digital signature or encryption information to the Manager

      When a process receives a message buffer, it receives only the message content. Any digital signatures or encryption envelopes associated with the message buffer are not included.

  3. The Manager calls tpenvelope() repeatedly to find out about the digital signature and encryption information associated with the message buffer. tpenvelope() returns:

  4. The Manager calls tpkey_getinfo() with the signer's public key as an argument, to obtain more information about the signer, including the signer's principal name.

  5. If the Manager determines that the signer is a known employee and that the employee's request (as stated in the message content) is valid, the Manager proceeds as follows.

    1. Calls tpsign() to mark the message buffer for digital signature by the Manager.

    2. Calls tpseal() to mark the message buffer to be encrypted for Purchasing.

    3. Calls tpforward() (or some other function used to transmit data) to send the message to Purchasing.

Just before the message is transmitted, the public key software performs the following tasks.

  1. Generates a digital signature for the Manager

  2. Verifies the employee's digital signature

  3. Encrypts the message content and associated digital signatures

  4. Creates an encryption envelope for Purchasing

Replacing an Output Buffer's Encryption Envelopes

If a message buffer is passed to an ATMI function (such as tpgetrply()) as an output parameter, the public key software deletes any encryption information associated with the buffer. This information includes any pending seals, or seals from previous uses of the buffer. (A pending seal is a recipient's seal that is registered with a message buffer.)

New encryption information might be associated with the new buffer content after successful completion of the operation.

See Also