Creating Avro Schema Files

Payloads are generated internally within Siebel CRM Event Pub/Sub, just as they are when Avro serialization is not configured. Avro transformations are then applied according to the schemas, and the transformed and serialized messages are posted to Kafka.

Avro serialization supports various data types, for example string, record, array, and so on. Optionally, you can also specify default values for each field.

Avro fields in the schema file must follow these conventions:

• Field names must:
  • Start with a letter (A–Z or a–z) or an underscore (_).
  • Be followed by any combination of letters, digits (0–9), or underscores (_).
  • Not include hyphens (-), spaces, or special characters.
  • Examples:
    • Valid field names: userName, _id, age1
    • Invalid field names: 1name, user-name, user name
• Field values:
  • Strings must be UTF-8 encoded.

Mapping for Non-Avro Compliant Fields in Siebel JSON

Avro only supports field names that use the following characters: letters (a–z, A–Z), digits (0–9), and underscores (_). However, in Siebel CRM, some field names do not follow this format. For example, Last Name is a valid field name in Siebel CRM but is not compliant with Avro field naming rules.

To address this requirement, a field called siebelName has been introduced in the schema to support Siebel CRM fields that are not Avro-compliant. For example, if you want to include a field with a space, for example: Last Name, from Siebel CRM in your Avro JSON, you can add an element called Last_Name in the schema and set its siebelName property to Last Name. This approach allows you to map “Last Name” from the Siebel CRM payload to Last_Name in the Avro JSON, preserving the original value while maintaining Avro field naming conventions.

Note:

• If you do not configure the siebelName property for non-Avro compliant Siebel CRM field that you want to include, no transformation will occur.
• If the siebelName property is configured for a field, Kafka will scan the Siebel CRM payload (which is generated internally before transformation for Avro are applied) for a field which is mentioned as the value of siebelName. If the property is not configured, Kafka will search by the actual field name in the Siebel CRM payload.
• The Avro converted result will use the actual field name, not the siebelName.

Samples Avro Schema File and Output

Schema File 1

Output

{ "type": "record", "name": "AccountCreation", "fields": [ { "name": "Account", "type": { "type": "record", "name": "Account", "fields": [ { "name": "Name", "type": "string" }, { "name": "CSN", "type": "string" }, { "name": "Account_Holder", "siebelName": "Contact", "type": { "type": "record", "name": "Contact", "fields": [ { "name": "First_Name", "type": "string", "siebelName": "First Name" }, { "name": "Last_Name", "type": "string", "default": "NoName", "siebelName": "Last Name" } ] } } ] } } ] }

{ "Account": { "Name": "Demo Account", "CSN": 8836, "Account_Holder": { "First_Name": "John", "Last_Name": "Doe" } } }

The above sample schema is configured to write information of newly created accounts and their associated contacts, with Contact defined as a child business component of the Account.

In the sample:

1. The Contact business component includes the following fields in the message posted to Kafka:
   • First_Name, with the siebelName property set to First Name.
   • Last_Name, with the siebelName property set to and Last Name.
2. The Siebel CRM payload scan is based on First Name and Last Name.
3. The Avro compliant output will have First_Name and Last_Name.
4. The Account_Holder field is the same as Contact field in the Siebel JSON.
5. The default value of Last_Name is set to NoName. If there is no match for this field (for example, when Last Name for the record in Siebel CRM is empty), the final output will default to NoName.

Schema File 2	Input	Output
{ "type": "record", "name": "ContactAndKids", "fields": [ { "name": "MyContact", "siebelName": "Contact", "type": { "type": "record", "name": "Contact", "fields": [ { "name": "ContactId", "type": "string" }, { "name": "First_Name", "type": "string", "siebelName": "First Name" }, { "name": "Last_Name", "type": "string", "siebelName": "Last Name" }, { "name": "Account", "type": { "type": "array", "items": { "type": "record", "name": "Account", "fields": [ { "name": "Name", "type": "string" }, { "name": "Place", "type": "string", "siebelName": "Location" }, { "name": "ID", "type": "string", "siebelName": "CSN" }, { "name": "Quote", "type": { "type": "array", "items": { "type": "record", "name": "Quote", "fields": [ { "name": "Name", "type": "string" } ] } } } ] } } }, { "name": "Opportunity", "type": { "type": "array", "items": { "type": "record", "name": "Opportunity", "fields": [ { "name": "OptyId", "type": "string" }, { "name": "Name", "type": "string", "siebelName": "Name" } ] } } } ] } } ] }	{ "Contact": { "First Name": "Doe", "Last Name": "Jack", "Account": [ { "Name": "HM", "Location": "HQ", "City": "Boston", "CSN": "1-12I9", "City State": "Boston, MA" } ] } }	{ "MyContact": { "ContactId": "", "First_Name": "Doe", "Last_Name": "Jack", "Account": [ { "Name": "HM", "Place": "HQ", "ID": "1-12I9", "Quote": [ { "Name": "" } ] } ], "Opportunity": [ { "OptyId": "", "Name": "" } ] } }

The fields that are in the Siebel payload (generated internally before Avro transformations are applied according to the schema) will be populated with appropriate values and the fields that are not present in it will be left blank. In the above sample, the following fields that are not present in the Siebel CRM payload are left blank:

• Quote as a child of Account.
• Opportunity

Note: Avro serialization also supports the old Siebel payload format. In the old payload format for event publication and subscription, the primary business components and child business components are siblings in JSON. You must define your schema with this rule in mind. If you do not, the output will contain blank entries. For example:

Sample schema:

{
    "type": "record",
    "name": "ContactAndKids",
    "fields": [
        {
            "name": "MyContact",
            "siebelName": "Contact",
            "type": {
                "type": "record",
                "name": "Contact",
                "fields": [
                    {
                        "name": "ContactId",
                        "type": "string",
                        "siebelName": "Contact Id"
                    },
                    {
                        "name": "First_Name",
                        "type": "string",
                        "siebelName": "First Name"
                    },
                    {
                        "name": "Last_Name",
                        "type": "string",
                        "siebelName": "Last Name"
                    },
                    {
                        "name": "Account",
                        "type": {
                            "type": "array",
                            "items": {
                                "type": "record",
                                "name": "Account",
                                "fields": [
                                    {
                                        "name": "Name",
                                        "type": "string"
                                    },
                                    {
                                        "name": "Place",
                                        "type": "string",
                                        "siebelName": "Location"
                                    },
                                    {
                                        "name": "ID",
                                        "type": "string",
                                        "siebelName": "CSN"
                                    }
                                ]
                            }
                        }
                    }
          
                ]
            }
        }
    ]
}

Input Payload	Output Payload	Explanation
{ "Contact": { "Fields": { "Contact Id": "", "First Name": "John", "Last Name": "Dow" }, "BusinessCompName": "Contact" }, "Account": { "Fields": { "Name": "Hibbing Manufacturing" }, "BusinessCompName": "Account" }, "BusinessObjName": "Contact" }	{ "MyContact": { "ContactId": "", "First_Name": "John", "Last_Name": "Dow", "Account": [ { "Name": "", "Place": "", "ID": "" } ] } }	In this example, Account is defined as a child of Contact in the schema. However, in the old Siebel payload format, primary business components and child business components are represented as siblings in the JSON. As a result, the output contains blank entries for the Account fields.