G Additional collectd Configurations and Information
This appendix contains the following:
Manual Mapping
As an alternative to automatic data mapping, you can manually map metric data to handle monitoring requirements not covered with automatic mapping.
Note:
The metric schema automatically defined with automatic mapping is described in Metric Schema Mapping (collectd).- Define your own destination metric in Oracle Management Cloud using REST APIs.
Example: Generic Metric Collector Entity Type (Auto-mapping)
- Write a mapping metadata JSON file to map source collectd metrics to the destination metrics.
- Provide a path to the mapping metadata JSON file when adding the Generic Metric Collector entity.
Example: Generic Metric Collector Entity (Manual Mapping)
The following example shows a generic metric collector entity named collectd-myhost that is locally monitored on the Cloud agent host myhost.
Example G-1 Entity of Generic Metric Collector Type - Locally Monitored
{
"entities":
[
{
"name": "collectd-myhost",
"type": "omc_generic_metric_collector",
"displayName": "collectd-myhost",
"timezoneRegion": "PDT",
"properties":
{
"host_name":
{
"displayName": "Host Name",
"value": "myhost.mycompany.com"
},
"capability":
{
"displayName": "capability",
"value": "monitoring"
},
"omc_monitored":
{
"displayName": "Cloud Agent Monitored",
"value": "TRUE"
},
"omc_query_interface_path":
{
"displayName": "Query Interface Path",
"value": "/opt/collectd/bin/collectdctl"
},
"omc_product_name":
{
"displayName": "Product Name",
"value": "collectd"
},
"omc_product_vendor":
{
"displayName": "Product Vendor",
"value": "Florian octo Forster, et al."
},
"omc_product_version_query_arg":
{
"displayName": "Product Version Query Argument",
"value": "-h"
},
"omc_product_version_regex":
{
"displayName": "Product Version Regular Expression",
"value": "^collectd (.+), http"
},
"omc_metrics_query_arg":
{
"displayName": "Metrics Query Argument",
"value": "listval"
},
"omc_response_query_arg":
{
"displayName": "Response Query Argument",
"value": "listval"
},
"omc_use_exit_code_for_response":
{
"displayName": "Use exit code for response",
"value": "TRUE"
},
"omc_protocol":
{
"displayName": "Protocol",
"value": "https"
},
"omc_payload_format":
{
"displayName": "Payload Format",
"value": "json"
},
"omc_filter_expression":
{
"displayName": "Filter Expression",
"value": "{$.[?(@.host=='myhost.mycompany.com')]}"
},
"omc_mapping_metadata_file_path":
{
"displayName": "Mapping Metadata File Path",
"value": "/scratch2/agent/gmc/mapping_metadata_processes.json"
}
}
}
]
}Example: Mapping Metadata
The following example illustrates the how to map collectd metrics to Oracle Management Cloud metrics.
Example G-2 Mapping Metadata - Processes
{
"entityMetricMappings":
[
{
"entityTypeSourceFilter":
{
"value": "{$[?(@.plugin=='processes')]}"
},
"entityType": "%host_type%",
"entityName":
{
"value": "%$host%"
},
"metricGroupMappings":
[
{
"metricGroupSourceFilter":
{
"in":
{
"field": "type",
"values":
[
"fork_rate",
"ps_state"
]
}
},
"metricGroupName":
{
"join":
{
"values":
[
"collectd_processes",
"%$type%",
"%$type_instance%"
],
"delimiter": "_"
}
},
"metricNames":
[
{
"join":
{
"values":
[
"processes",
"%$type%",
"%$type_instance%"
],
"delimiter": "_"
}
}
],
"metricValues":
[
{
"value": "%$values[0]%"
}
]
}
]
}
],
"metricCollectionTimeMapping":
{
"value": "%$time%",
"timeFormat": "UNIX"
}
}Example: Destination Metric Definitions
The following example destination metric definition is posted to the entity model REST API
Path: /entityModel/metadata/entityTypes/omc_host/metricGroupTypes
Example G-3 Destination Metric Definitions - Custom Metric Groups on the Host Interface
[
{
"entityTypeName": "omc_host",
"metricGroupName": "collectd_processes_ps_state_running",
"metricGroupDisplayName": "Running processes",
"description": "Number of processes in running state",
"config": false,
"parentMGName": null,
"curationLevel": 1,
"columnList":
[
{
"metricColumnDisplayName": "Running processes",
"metricColumnName": "processes_ps_state_running",
"metricColumnClass": "NUM",
"baselineable": true,
"typeFormat": null,
"isKey": false,
"unitType": "OMC_SYS_STANDARD_GENERAL_NA",
"description": "Number of processes in running state",
"category": "Load"
}
]
},
{
"entityTypeName": "omc_host",
"metricGroupName": "collectd_processes_ps_state_sleeping",
"metricGroupDisplayName": "Sleeping processes",
"description": "Number of processes in sleeping state",
"config": false,
"parentMGName": null,
"curationLevel": 1,
"columnList":
[
{
"metricColumnDisplayName": "Sleeping processes",
"metricColumnName": "processes_ps_state_sleeping",
"metricColumnClass": "NUM",
"typeFormat": null,
"isKey": false,
"unitType": "OMC_SYS_STANDARD_GENERAL_NA",
"description": "Number of processes in sleeping state",
"category": "Load"
}
]
},
{
"entityTypeName": "omc_host",
"metricGroupName": "collectd_processes_ps_state_blocked",
"metricGroupDisplayName": "Blocked processes",
"description": "Number of processes in blocked state",
"config": false,
"curationLevel": 1,
"columnList":
[
{
"metricColumnDisplayName": "Blocked processes",
"metricColumnName": "processes_ps_state_blocked",
"metricColumnClass": "NUM",
"typeFormat": null,
"isKey": false,
"unitType": "OMC_SYS_STANDARD_GENERAL_NA",
"description": "Number of processes in blocked state",
"category": "Load"
}
]
},
{
"entityTypeName": "omc_host",
"metricGroupName": "collectd_processes_ps_state_stopped",
"metricGroupDisplayName": "Stopped processes",
"description": "Number of processes in stopped state",
"config": false,
"curationLevel": 1,
"columnList":
[
{
"metricColumnDisplayName": "Stopped processes",
"metricColumnName": "processes_ps_state_stopped",
"metricColumnClass": "NUM",
"typeFormat": null,
"isKey": false,
"unitType": "OMC_SYS_STANDARD_GENERAL_NA",
"description": "Number of processes in stopped state",
"category": "Load"
}
]
},
{
"entityTypeName": "omc_host",
"metricGroupName": "collectd_processes_ps_state_paging",
"metricGroupDisplayName": "Paging processes",
"description": "Number of processes in paging state",
"config": false,
"curationLevel": 1,
"columnList":
[
{
"metricColumnDisplayName": "Paging processes",
"metricColumnName": "processes_ps_state_paging",
"metricColumnClass": "NUM",
"typeFormat": null,
"isKey": false,
"unitType": "OMC_SYS_STANDARD_GENERAL_NA",
"description": "Number of processes in paging state",
"category": "Load"
}
]
},
{
"entityTypeName": "omc_host",
"metricGroupName": "collectd_processes_ps_state_zombies",
"metricGroupDisplayName": "Zombie processes",
"description": "Number of processes in zombie state",
"config": false,
"curationLevel": 1,
"columnList":
[
{
"metricColumnDisplayName": "Zombie processes",
"metricColumnName": "processes_ps_state_zombies",
"metricColumnClass": "NUM",
"typeFormat": null,
"isKey": false,
"unitType": "OMC_SYS_STANDARD_GENERAL_NA",
"description": "Number of processes in zombie state",
"category": "Load"
}
]
},
{
"entityTypeName": "omc_host",
"metricGroupName": "collectd_processes_fork_rate",
"metricGroupDisplayName": "Fork Rate",
"description": "Overall rate of creation of processes or threads by all CPUs",
"config": false,
"parentMGName": null,
"curationLevel": 1,
"columnList":
[
{
"metricColumnDisplayName": "Fork Rate",
"metricColumnName": "processes_fork_rate",
"metricColumnClass": "NUM",
"typeFormat": null,
"isKey": false,
"unitType": "OMC_SYS_STANDARD_RATE_SEC",
"description": "Overall rate of creation of processes or threads by all CPUs",
"category": "Load"
}
]
}
]Metric Schema Mapping (collectd)
The following table shows the Oracle Management Cloud metric schema derived from collectd metric identifiers. If you enable automatic mapping, collectd metrics are automatically mapped to Oracle Management Cloud metrics.
Note:
The maximum number of metric groups per entity type (auto-created by Oracle Management Cloud when using the automatic mapping) is 70. On the collectd side, this translates to 70 distinct type + type_instance combinations of metric identifiers per collectd plugin.Table G-1 Metric Schema Mapping
| Field | Value | Example |
|---|---|---|
|
parentTargetType(64) |
omc_target |
omc_target |
|
entityType(64) |
'_gmc_collectd_' <plugin> |
_gmc_collectd_redis |
|
typeDisplayName(128) |
<plugin in Title Case, replace underscore with space> |
Redis |
|
entityName(256) |
<host>'-'<plugin_instance> or <host> (if no plugin_instance) |
myhost.myco.com-6379 |
|
metricGroupName(64) |
<type>'_'<type_instance> or <type> (if no type_instance)
|
1. memcached_connections_clients 2. df_memory 3. uptime (no type_instance)
|
|
metricGroupDisplayName(256)
|
<metricGroupName in Title Case with underscores replaced by spaces>
|
Memcached Connections Clients |
|
metricColumnName(64) |
<dsname> |
1. value 2. used, free 3. value
|
|
metricColumnDisplayName(256) |
If <dsname> equals 'value' same as <metricGroupDisplayName> Else <metricGroupDisplayName>' '<dsname in Title Case> |
1. Memcached Connections Clients ('value' omitted) 2. Df Memory Used, Df Memory Free 3. Uptime (no type_instance)
|
|
metricColumnClass |
NUM |
NUM |
You can extract the plugin name, plugin_instance, type, and type instance by running the collectdctl listval command.
$ /opt/collectd/bin/collectdctl listval
myhost.mycompany.com/load/load
myhost.mycompany.com/processes/fork_rate
myhost.mycompany.com/processes/ps_state-blocked
myhost.mycompany.com/processes/ps_state-paging
myhost.mycompany.com/processes/ps_state-running
myhost.mycompany.com/processes/ps_state-sleeping
myhost.mycompany.com/processes/ps_state-stopped
myhost.mycompany.com/processes/ps_state-zombies
You can obtain the dsname (data source name) by running the collectdctl getval command..
$ /opt/collectd/bin/collectdctl getval myhost.mycompany.com/load/load
shortterm=1.100000e-01
midterm=8.000000e-02
longterm=6.000000e-02Send a Subset of collectd Metrics to Oracle Management Cloud
To minimize noise and increase efficiency of what is sent from collectd to Oracle Management Cloud, it is possible to write only a subset of metrics collected by collectd to Oracle Management Cloud. A PostCacheChain can be configured to specify this. Assume that following read plugins were initially enabled and were writing their output to the write_log plugin.
Initial Read Plugins Enabled
LoadPlugin cpu
LoadPlugin interface
LoadPlugin memory
LoadPlugin processesTo send the output from the processes plugin to Oracle Management Cloud, but not the other plugins. A PostCacheChain can be configured to achieve this, as shown in the following example..
Example: PostCacheChain Configuration to Selectively Send the Output of Processes to Oracle Management Cloud
LoadPlugin match_regex
PostCacheChain "PostCache"
<Chain "PostCache">
<Rule "write_omc">
<Match "regex">
Plugin "^processes$"
</Match>
<Target "write">
Plugin "write_http/omc"
Plugin "write_log"
</Target>
Target "stop"
</Rule>
# Default target
<Target "write">
Plugin "write_log"
</Target>
</Chain>Here we have specified a rule called "write_omc" with a regex to match the plugin's name (processes) to be sent to Oracle Management Cloud as well as to write_log (as before). The default write target is configured not to write to Oracle Management Cloud so that the output from the other remaining read plugins (cpu, interface, and memory) will not be sent to Oracle Management Cloud.
Receive Metrics from a Remote Generic Metric Collector
For environments where a local Cloud agent is not installed on the host running the Generic Metric Collector, it is still possible for Oracle Management Cloud to receive the collectd metrics even though it’s not actually being monitored by a Cloud agent.
-
Set up integration between the collectd collector and Cloud agent to only receive metrics from the collector but not monitor it. Do NOT add the Cloud agent host as a remote host in this case.
-
Add a generic metric collector entity to the Cloud agent as shown in the following sample JSON files. Both automatic and manual metric mapping samples are shown.
Automatic collectd--Oracle Management Cloud Metric Mapping
{ "entities": [ { "name": "<Your name for the collectd collector>", "type": "omc_generic_metric_collector", "displayName": "<Your display name for the collectd collector>", "timezoneRegion": "<Your timezone>", "properties": { "host_name": { "displayName": "Host Name", "value": "<Your name of the host where collectd is installed>" }, "omc_filter_expression": { "displayName": "Filter Expression", "value": "{$.[?(@.host=='<Value of the host field in the metric payload sent by collectd>')]}" }, "omc_auto_map": { "displayName": "Automatically Map Metrics", "value": "TRUE" }, "capability": { "displayName": "capability", "value": "monitoring" }, "omc_monitored": { "displayName": "Cloud Agent Monitored", "value": "FALSE" }, "omc_product_name": { "displayName": "Product Name", "value": "collectd" }, "omc_product_vendor": { "displayName": "Product Vendor", "value": "Florian octo Forster, et al." }, "omc_protocol": { "displayName": "Protocol", "value": "https" }, "omc_payload_format": { "displayName": "Payload Format", "value": "json" }, "omc_receiver_uri_path": { "displayName": "Receiver URI Path", "value": "/emd/receiver/gmc" } } } ] }Manual collectd--Oracle Management Cloud Metric Mapping{ "entities": [ { "name": "<Your name for the collectd collector>", "type": "omc_generic_metric_collector", "displayName": "<Your display name for the collectd collector>", "timezoneRegion": "<Your timezone>", "properties": { "host_name": { "displayName": "Host Name", "value": "<Your name of the host where collectd is installed>" }, "omc_filter_expression": { "displayName": "Filter Expression", "value": "{$.[?(@.host=='<Value of the host field in the metric payload sent by collectd>')]}" }, "omc_mapping_metadata_file_path": { "displayName": "Mapping Metadata File Path", "value": "<Path to your mapping metadata json file on the cloud agent host>" }, "capability": { "displayName": "capability", "value": "monitoring" }, "omc_monitored": { "displayName": "Cloud Agent Monitored", "value": "FALSE" }, "omc_product_name": { "displayName": "Product Name", "value": "collectd" }, "omc_product_vendor": { "displayName": "Product Vendor", "value": "Florian octo Forster, et al." }, "omc_protocol": { "displayName": "Protocol", "value": "https" }, "omc_payload_format": { "displayName": "Payload Format", "value": "json" }, "omc_receiver_uri_path": { "displayName": "Receiver URI Path", "value": "/emd/receiver/gmc" } } } ] } -
Configure the collectd write_http plugin to send metrics to the following URL:
https://<remote-cloud-agent-host>:<remote-cloud-agent-port>/emd/receiver/gmc
There is no need to configure the unixsock plugin or add the Cloud agent user to the collectd socket group.
Once this configuration is complete, the Cloud agent will not try to monitor the Generic Metric Collector entity. It will instead send an unknown response to Oracle Management Cloud so that the availability of the Generic Metric Collector will be unknown.
Availability (Up/Down) Status for Entities Monitored by collectd
For most entities monitored by collectd, availability is turned off by default, but can be enabled by setting the appropriate emd.properties file property. For a specific subset of entities where the concept of “availability” does not apply, availability processing cannot be enabled.
By default, availability processing is turned on for the following collectd plug-ins:
- processes
- postgresql
Availability processing is not applicable (and cannot be enabled) for the following collectd plug-ins:
- cpu
- df
- disk
- interface
- load
- memory
- swap
- vmem
Availability processing for all other collectd plug-ins is set to “not applicable” by default, but can be enabled via the following property settings in the emd.properties file.
Table G-2 Availability Properties
| Property | Description | Example |
|---|---|---|
| _gmcReceiver_downResponseMultiplicationFactor |
Sets multiplication factor for down detection to 4 times the collection interval instead of the default value of 3. |
|
| _gmcReceiver_downResponseWaitTimeSeconds |
Sets the wait time for down response to 35 seconds while default is 30 seconds. |
|
| _gmcReceiver_availabilityApplicableEntityTypes |
Enable sending availability (up/down) for collectd plug-ins. |
The following example illustrates enabling availability processing for collectd's plugin1.
|
|
_gmcReceiver_responseProcessorIntervalSeconds
|
Increases the response processor interval from the default 1 minute to 10 minutes. |
The following example sets the response processor interval to 10 minutes.
|
Troubleshooting collectd Metric Collection
-
Ensure that the generic metric collector entity was added successfully to the agent by the "omcli add_entity" command. If it is not showing up in the metric brower, run the
status_entity omclicommand.$ omcli status_entity agent <entityDefinitionJsonFilePath>Validation errors, if any, will be shown in the output.
-
Enable trace level logging in emd.properties. Set the following two properties
Logger._enableTrace=true Logger.sdklog.level=DEBUGand bounce the Cloud agent. Tail gcagent_sdk.trc in agent's log directory.
-
From the log file you should see the complete payload received by agent from collectd, which metrics are in turn being sent by receiver to Oracle Management Cloud, and which metrics are unmapped. Search for "gmcReceiver received payload" in the log file to see the full payload received. If this line is not seen in the log file, the agent may not be receiving data from collectd. So check if collectd is running and that read plugins are loaded and reading metrics. To get a list of identifiers against which read plugins are collecting metrics, run the
collectdctl listvalcommand. To check the data source names for each metric, run thecollectdctl getval <identifier>command as shown below.$ /opt/collectd/bin/collectdctl listval myhost.mycompany.com/load/load myhost.mycompany.com/processes/fork_rate myhost.mycompany.com/processes/ps_state-blocked myhost.mycompany.com/processes/ps_state-paging myhost.mycompany.com/processes/ps_state-running myhost.mycompany.com/processes/ps_state-sleeping myhost.mycompany.com/processes/ps_state-stopped myhost.mycompany.com/processes/ps_state-zombies $ /opt/collectd/bin/collectdctl getval myhost.mycompany.com/load/load shortterm=1.100000e-01 midterm=8.000000e-02 longterm=6.000000e-02Check that the write_http plugin has been configured correctly. Check the configured log file or syslog for any error message from the write_http plugin. Check if other software applications such as SELinux, antivirus, or a firewall may be blocking collectd’s ability to write metrics to the cloud agent's port.
- Search for payload level summary lines in the log file which starts with the "Source Metrics" line. These lines should give a summary count of statistics such as how many metrics are being received in each payload, how many have been sent to Oracle Management Cloud, or how many are unmapped.
Payload Level Summary Logging Example - gcagent_sdk.trc
2017-07-04 21:45:04,613 [401336:9A108C02] DEBUG - Source Metrics: 18 2017-07-04 21:45:04,613 [401336:9A108C02] DEBUG - SEND_METRIC_GROUP_CALLED: 18-
If the summary shows SEND_METRIC_GROUP_CALLED: <count>, that's normal.
-
If the summary shows NO_ASSOC_GMC_ENTITY_WITH_MONITORING_CAPABILITY: <count>, then check that omc_filter_expression of the generic metric collector (gmc) entity allows the payload to filter through. Ensure that the name of the host field (if any) specified in the omc_filter_expression property exactly matches the host field's value in the payload. Also ensure that the gmc entity has either standard or enterprise license. License can be checked from Oracle Management Cloud’s Administration UI.
-
If the summary shows UNMAPPED: <count>, then this will be accompanied by more detailed logging lines calling out the exact metrics that are unmapped. Unmapped metrics are only expected in the manual mapping case. The corresponding mapping metadata rule needs to be re-checked with the source data sent by collectd, particularly the filter expressions in metadata. Check if they match up with the metric data and structure.
-
If the summary shows METRIC_UPLOAD_RATE_LIMIT_EXCEEDED: <count>, then <count> metrics in the payload were down-sampled. They were not sent to up Oracle Management Cloud. This is expected if the sending interval is anything lower than once a minute (Interval 60 in the collectd.conf file).
-
If the summary shows WAITING_FOR_MAPPING_METADATA: <count>, then <count> metrics in the payload are waiting for auto-map processing to complete. This is a transient state only expected in the automatic mapping case. Auto-map processing can take a few minutes to a tens of minutes to complete.
-
-
Futher Steps - Automatic Mapping Case
-
If SEND_METRIC_GROUP_CALLED: <count> is seen, you should eventually start seeing entities on the monitoring service UI with type same as the collectd plugin name and entity name containing the collectd host's name (as obtained from the host field within the payload sent by collectd to Cloud agent). If you do not see such an entity, it’s possible that the entity has been created, but has not been granted Standard or Enterprise license. This can be fixed by adding a license from the License Administration UI. From the Oracle Management Cloud console, select the Administration > Entities Configuration > Licensing link. From this page, look at the Unlicensed Entities link. If it shows the auto-created entity, assign License Edition = Standard or Enterprise and click Save. To ensure this happens automatically in future, set the License Auto-Assignment to Standard or Enterprise.
-
Once the auto-created entity shows up on the list of entities in the monitoring service UI, drill down into the entity to see the auto-mapped metrics. Only the availability metric will be shown by default. On the Performance Charts tab, Click Options > Choose Metrics to select the auto-created metrics for viewing their charts. Metric alert rules can also be defined on these performance metrics and are expected to work similar to alerts on metrics natively collected by Oracle Cloud agent.
-
-
Further Step - Manual Mapping Case. This step is only applicable to manual mapping. If metrics are being sent from the agent to Oracle Management Cloud, check that the shape of the metrics sent (entity type, metric group and columns) are as expected and that the metrics are sent against the correct entity name. Search for the particular metric name or entity type in the log. Look for lines containing the string "Calling sendMetricGroup".
Send Metric Group Example - gcagent_sdk.trc
2018-03-04 11:25:04,610 [401336:9A108C02] DEBUG - Calling sendMetricGroup on targetID=_gmc_collectd_snmp.myhost.mycompany.com; PostMetricActionBinding [mergeKey=MergeKey [entityName=myhost.mycompany.com, metricGroupName=if_octets_eth0, MetricColumnWrappers=[MetricColumnWrapper [getColumnName()=rx, isNumeric()=true], MetricColumnWrapper [getColumnName()=tx, isNumeric()=true]], entityType=_gmc_collectd_snmp, collectionTS=Mon Mar 05 22:27:40 PST 2018], metricValues=[[6634.04566173485, 2624.41806371072]], gmcEntityName=collectd-slc11ciy-automap, response=null, autoMapRequests=nullFrom the above log line, the target entity type is _gmc_collectd_snmp; entity name is myhost.mycompany.com; metric group name is if_octets_eth0; metric column names are rx and tx; metric values are 6634.04566173485 and 2624.41806371072; and the associated gmc entity name is collectd-slc11ciy-automap.
-
When debugging is no longer required, turn off trace level logging and set the SDK log level to INFO. Set the following in emd.properties.
Logger._enableTrace=false Logger.sdklog.level=INFO