SMF services and various Oracle Solaris processes produce log files where you can look for error messages or gather more information about messages that were displayed on the screen. SMF service log files contain valuable debugging information.
Because OpenStack is typically installed across multiple systems, the log files that you need to check are also in various locations. For a more systematic troubleshooting, examine logs per node.
For general help in fixing problems with SMF services, see the troubleshooting section in Managing System Services in Oracle Solaris that applies to your operating system's release. Go to the appropriate library in http://docs.oracle.com/en/operating-systems/.
To view service logs, make sure you have the proper authorization. Assume the appropriate RBAC profile to view OpenStack service log files or to use the pfedit command to modify OpenStack service configuration files. The following profiles can be assigned:
OpenStack Block Storage Management
OpenStack Compute Management
OpenStack Identity Management
OpenStack Image Management
OpenStack Network Management
OpenStack Object Storage Management
OpenStack Management
To troubleshoot, use the following general commands:
To find out which OpenStack services are running on a specific node:
# svcs -a | grep openstack
To list which services might be in a maintenance mode:
# svcs -x
svc:/application/openstack/swift/swift-replicator-rsync:
default (OpenStack Swift Replication Service)
State: maintenance since Fri May 22 04:06:11 2015
Reason: Start method exited with $SMF_EXIT_ERR_FATAL.
See: http://support.oracle.com/msg/SMF-8000-KS
See: rsync(1)
See: rsyncd.conf(5)
See: /var/svc/log/application-openstack-swift-swift-replicator-rsync:default.log
Impact: This service is not running.
If a service is in maintenance mode, check the service log file.
To identify the log for a specific OpenStack service:
# svcs -L openstack-service
For example:
# svcs -L neutron-server /var/svc/log/application-openstack-neutron-neutron-server:default.log
With the proper authorization, you can combine options such as –Lv to list and view the log for the service.
To immediately identify instances of error that are recorded in a specific log, you can use common UNIX commands such as grep.
# grep keyword `svcs -L openstack-service`
You can search for the occurrences of error, warning, and other critical keywords to directly read the error messages.
To check EVS properties when troubleshooting networking issues, use the various evsadm subcommands such as evsadm show-prop.
An OpenStack list command line can provide information about the status of components. For example, nova list displays active instances on the compute node, while nova image-list shows available images and their status. For a sample output, see Displaying Information About Images.
The following logs often contain information helpful for troubleshooting:
nova-compute
nova-scheduler
cinder-scheduler
neutron-server
Aside from the SMF service log files, you can also check logs in the /var/log directory. Just like other Oracle Solaris processes, OpenStack services also generate their own log files in the /var/log/openstack-service directories.
For example, OpenStack image store log files are in /var/log/glance. Problems with creating and booting a VM instance might be recorded in the /var/log/zones directory. Messaging logs are stored as /var/log/rabbitmq/rabbit@hostname.log.
Most OpenStack configuration files are under the OpenStack service name in the /etc directory. For example, OpenStack networking configuration files are in /etc/neutron. Configuration files for Horizon are in /etc/openstack_dashboard. Those for Nova are in /etc/nova, and so on. You can use these files for troubleshooting a specific service by setting or uncommenting the following parameters in the service's configuration file:
debug=true
verbose=true
These parameters enable you to see more output from operations affected by that configuration file. See tables of configuration options in “Common Configuration Parameters for OpenStack” in http://www.oracle.com/technetwork/articles/servers-storage-admin/getting-started-openstack-os11-2-2195380.htmlGetting Started with OpenStack on Oracle Solaris 11.3 and in the OpenStack Configuration Reference on the OpenStack documentation site.