Troubleshooting Instances Using Instance Console Connections

The Oracle Cloud Infrastructure Compute service provides console connections that enable you to remotely troubleshoot malfunctioning instances, such as:

  • An imported or customized image that does not complete a successful boot
  • A previously working instance that stops responding

Two types of instance console connections exist: serial console connections and VNC console connections. Instance console connections are for troubleshooting purposes only. To connect to a running instance for administration and general use with Secure Shell (SSH) or Remote Desktop connection, see Connecting to an Instance.

To configure your console connection, follow these steps:

  1. Make sure you have the correct permissions.
  2. Complete the prerequisites, including creating your SSH key pair.
  3. Create the instance console connection.
  4. Connect to the serial console or connect to the VCN console.
  5. If you're trying to connect to the serial console and you think the connection isn't working, test your connection to the serial console using Cloud Shell.

Required IAM Policies

To use Oracle Cloud Infrastructure, you must be granted security access in a policy  by an administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don’t have permission or are unauthorized, verify with your administrator what type of access you have and which compartment  you should work in.

To create instance console connections, an administrator needs to grant user access to manage instance console connections and to read instances through an IAM policy. The resource name for instance console connections is instance-console-connection. The resource name for instances is instance. The following policies grant users the ability to create instance console connections:

Allow group <group_name> to manage instance-console-connection in tenancy
Allow group <group_name> to read instance in tenancy

Instance console connections also support network sources. The following policies grant users the ability to create instance console connections with a network source:

Allow group <group_name> to manage instance-console-connection in tenancy where request.networkSource.name='example-network-source'
Allow group <group_name> to read instance in tenancy where request.networkSource.name='example-network-source'

If you're new to policies, see Getting Started with Policies and Common Policies.

Prerequisites

Complete these prerequistes before creating the instance console connection.

Installing an SSH Client and a Command-line Shell (Windows)

Windows does not include an SSH client by default. If you are connecting from a Windows client, you need to install an SSH client. You can use PuTTY plink.exe with Windows PowerShell or software that includes a version of OpenSSH such as:

The instructions in this topic frequently use PuTTY and Windows PowerShell.

If you want to make the console connection from Windows with Windows PowerShell, PowerShell might already be installed on your Windows operating system. If not, follow the steps at the link. If you are connecting to the instance from a Windows client using PowerShell, plink.exe is required. plink.exe is the command link connection tool included with PuTTY. You can install PuTTY or install plink.exe separately. For installation information, see http://www.putty.org.

Creating SSH Key Pairs

To create the secure console connection, you need an SSH key pair. The method to use for creating key pairs depends on your operating system. When connecting to the serial console, you must use an RSA key. The instructions in this section show how to create an RSA SSH key pair.

Creating the SSH key pair for Linux

For detailed instructions about creating an SSH key pair to use on Linux, see Managing Key Pairs on Linux Instances.

To create an SSH key pair on the command line

If you're using a UNIX-style system, you probably already have the ssh-keygen utility installed. To determine whether the utility is installed, type ssh-keygen on the command line. If the utility isn't installed, you can download OpenSSH for UNIX from http://www.openssh.com/portable.html and install it.

  1. Open a shell or terminal for entering the commands.
  2. At the prompt, enter ssh-keygen and provide a name for the key when prompted. Optionally, include a passphrase.

    The keys will be created with the default values: RSA keys of 2048 bits.

Alternatively, you can type a complete ssh-keygen command, for example:

ssh-keygen -t rsa -N "" -b 2048 -C "<key_name>" -f <path/root_name>

The command arguments are shown in the following table:

Argument Description
-t rsa Use the RSA algorithm.
-N "<passphrase>"

A passphrase to protect the use of the key (like a password). If you don't want to set a passphrase, don't enter anything between the quotes.

A passphrase is not required. You can specify one as a security measure to protect the private key from unauthorized use. If you specify a passphrase, when you connect to the instance you must provide the passphrase, which typically makes it harder to automate connecting to an instance.

-b 2048

Generate a 2048-bit key. You don't have to set this if 2048 is acceptable, as 2048 is the default.

A minimum of 2048 bits is recommended for SSH-2 RSA.

-C "<key_name>" A name to identify the key.
-f <path/root_name> The location where the key pair will be saved and the root name for the files.

Creating the SSH key pair for Windows using PuTTY

If you are using a Windows client to connect to the instance console connection, use an SSH key pair generated by PuTTY.

To create the SSH key pair using PuTTY
Important

Ensure that you are using the latest version of PuTTY, see http://www.putty.org.
  1. Find puttygen.exe in the PuTTY folder on your computer, for example, C:\Program Files (x86)\PuTTY. Double-click puttygen.exe to open it.
  2. Specify a key type of SSH-2 RSA and a key size of 2048 bits:

    • In the Key menu, confirm that the default value of SSH-2 RSA key is selected.
    • For the Type of key to generate, accept the default key type of RSA.
    • Set the Number of bits in a generated key to 2048 if not already set.
  3. Click Generate.
  4. To generate random data in the key, move your mouse around the blank area in the PuTTY window.

    When the key is generated, it appears under Public key for pasting into OpenSSH authorized_keys file.

  5. A Key comment is generated for you, including the date and timestamp. You can keep the default comment or replace it with your own more descriptive comment.
  6. Leave the Key passphrase field blank.
  7. Click Save private key, and then click Yes in the prompt about saving the key without a passphrase.

    The key pair is saved in the PuTTY Private Key (PPK) format, which is a proprietary format that works only with the PuTTY tool set.

    You can name the key anything you want, but use the ppk file extension. For example, mykey.ppk.

  8. Select all of the generated key that appears under Public key for pasting into OpenSSH authorized_keys file, copy it using Ctrl + C, paste it into a text file, and then save the file in the same location as the private key.

    (Do not use Save public key because it does not save the key in the OpenSSH format.)

    You can name the key anything you want, but for consistency, use the same name as the private key and a file extension of pub. For example, mykey.pub.

  9. Write down the names and location of your public and private key files. You need the public key when creating an instance console connection. You need the private key to connect to the instance console connection using PuTTY.

Signing in to an instance from the serial console (optional)

To troubleshoot instances and see serial output using the serial console, you don't need to sign in. To connect to a running instance for administration and general use with Secure Shell (SSH) or Remote Desktop connection, see Connecting to an Instance.

If you want to sign in to an instance using an instance console connection, you can use Secure Shell (SSH) or Remote Desktop connection to sign in. If you want to sign in with a username and password, you need a user account with a password. Oracle Cloud Infrastructure does not set a default password for the opc user. Therefore, if you want to sign in as the opc user, you need to create a password for the opc user. Otherwise, add a different user with a password and sign in as that user.

Connecting Through Firewalls

If your system is behind a firewall, the system must be able to reach the console servers. The client system connecting to the serial console must be able to reach the serial or VCN console server (for example, instance-console.us-ashburn-1.oraclecloud.com) over SSH using port 443, directly or through a proxy.

Supported Instance Types

Serial console connections are supported on the following types of instances:

  • Virtual machine (VM) instances launched in September 2017 or later
  • Bare metal instances launched in November 2017 or later

VNC console connections are supported on the following types of instances:

  • VM instances launched on October 13, 2017 or later
  • Bare metal instances that use one of the following shapes:

    • BM.Standard2.52 - launched on February 21, 2019 or later
    • BM.Standard.E2.64 - launched after September 17, 2020 in the Oracle Cloud Infrastructure commercial realm
    • BM.Standard.E3.128 - launched on February 21, 2019 or later
    • BM.DenseIO2.52 - launched on February 21, 2019 or later
    • BM.GPU2.2 - launched on February 21, 2019 or later
    • BM.GPU3.8 - launched on February 21, 2019 or later
    • BM.GPU.4.8 - launched on February 21, 2019 or later
    • BM.HPC2.36 - launched on February 21, 2019 or later

Creating the Instance Console Connection

Before you can connect to the serial console or VNC console, you need to create the instance console connection.

Note

Instance console connections are limited to one client at a time. If the client fails, the connection remains active for approximately five minutes. During this time, no other client can connect. After five minutes, the connection is closed, and a new client can connect. During the five-minute timeout, any attempt to connect a new client fails with the following message:
channel 0: open failed: administratively prohibited: console access is limited to one connection at a time
Connection to <instance and OCID information> closed.
  1. Open the navigation menu. Under Core Infrastructure, go to Compute and click Instances.
  2. Click the instance that you're interested in.
  3. Under Resources, click Console Connection.

  4. Click Create Console Connection.
  5. Upload the public key portion for the SSH key. You have three options for adding the SSH key.
    • Generate SSH key pair: You can have Oracle Cloud Infrastructure generate an SSH key pair to use. If you are using PowerShell or PuTTY to connect to the instance from a Windows client, you cannot use the generated SSH key pair without first converting it to a .ppk file.
      To convert a generated .key private key file
      1. Open PuTTYgen.

      2. Click Load, and select the private key generated when you created the instance. The extension for the key file is .key.

      3. Click Save private key.

      4. Specify a name for the key. The extension for new private key is .ppk.

      5. Click Save.

    • Choose public key file: Browse to a public key file on your computer. If you followed the steps in Creating SSH Key Pairs in the Prerequisites section to create a key pair, use this option to navigate to the .pub file.
    • Paste public key: Paste the content of your public key file into the text box.
  6. Click Create Console Connection.

    When the console connection has been created and is available, the state changes to Active.

Connecting to the Serial Console

After you create the console connection for the instance, you can connect to the serial console using a Secure Shell (SSH) connection. When connecting to the serial console, you must use an RSA key. You can use the same SSH key for the serial console that was used when you launched the instance, or you can use a different SSH key.

When you are finished with the serial console and have terminated the SSH connection, you should delete the serial console connection. If you do not disconnect from the session, Oracle Cloud Infrastructure terminates the serial console session after 24 hours and you must reauthenticate to connect again.

Connecting from Mac OS X and Linux Operating Systems

Use an SSH client to connect to the serial console. Mac OS X and most Linux distributions include the SSH client OpenSSH by default.

To connect to the serial console for an instance using OpenSSH on Mac OS X or Linux
  1. On the instance details page in the Oracle Cloud Infrastructure Console, under Resources, click Console Connection.
  2. Click the Actions icon (three dots), and then click Copy Serial Console Connection for Linux/Mac.
  3. Paste the connection string into a terminal window on a Mac OS X or Linux system, and then press Enter to connect to the console.

    If you are not using the default SSH key or ssh-agent, modify the serial console connection string to include the identity file flag, -i, to specify the private key portion for the SSH key to use, for example id_rsa. Specify this flag for both the SSH connection and the SSH ProxyCommand, as shown in the following line:

    ssh -i /<path>/<ssh_key> -o ProxyCommand='ssh -i /<path>/<ssh_key> -W %h:%p -p 443...
  4. Press Enter again to activate the console. If the connection is active, a message appears in the console: IMPORTANT: Use a console connection to troubleshoot a malfunctioning instance.

  5. In the Oracle Cloud Infrastructure Console, reboot your instance. If the instance is functional and the connection is active, the serial output appears in your console. If serial output does not appear in the console, the instance operating system is not booting.

Connecting from Windows Operating Systems

The steps to connect to the serial console from Windows Powershell are different from the steps for OpenSSH. The following steps do not work in the Windows terminal.

Important

If you are connecting to the instance from a Windows client using PowerShell, plink.exe is required. plink.exe is the command link connection tool included with PuTTY. You can install PuTTY or install plink.exe separately. For more information, see Installing an SSH Client and a Command-line Shell (Windows).
To connect to the serial console for an instance on Microsoft Windows
  1. On the instance details page in the Oracle Cloud Infrastructure Console, under Resources, click Console Connection.
  2. Click the Actions icon (three dots). Depending on which SSH client you are using, do one of the following:
    • If you are using Windows PowerShell, click Copy Serial Console Connection for Windows.

    • If you are using OpenSSH, click Copy Serial Console Connection for Linux/Mac.

    Tip

    The copied connection string for Windows contains the parameter -i specifying the location of the private key file. The default value for this parameter in the connection string references an environment variable which might not be configured on your Windows client, or it might not represent the location where the private key file is saved. Verify the value specified for the -i parameter and make any required changes before proceeding to the next step.
  3. Paste the connection string copied from the previous step into a text file so that you can add the file path to the private key file.
  4. In the text file, replace $env:homedrive$env:homepath\oci\console.ppk with the file path to the .ppk file on your computer. This file path appears twice in the string. Replace it in both locations.
  5. Paste the modified connection string into the PowerShell window or your OpenSSH client, and then press Enter to connect to the console.
  6. Press Enter again to activate the console.
  7. In the Oracle Cloud Infrastructure Console, reboot your instance. If the instance is functional and the connection is active, the serial output appears in your client. If serial output does not appear in the client, the instance operating system is not booting.

Connecting from Cloud Shell

If you encounter issues when connecting to your instance's serial console using the steps for connection from Mac OS X, Linux, or Windows, test connecting to the serial console using Cloud Shell. Cloud Shell is a web browser-based terminal accessible from the Console, see Cloud Shell for more information. This procedure includes steps to access Cloud Shell. For an introductory walkthrough of using Cloud Shell, see Using Cloud Shell.

Note

You cannot use Cloud Shell for VNC console connections. You can only use it for serial console connections.

To connect to the serial console for an instance using Cloud Shell
  1. Sign in to the Console.
  2. Click the Cloud Shell icon in the Console header as shown in the following screenshot:

    Cloud Shell launch icon

    This action displays the Cloud Shell in a "drawer" at the bottom of the console as shown in the following screenshot:

    Cloud shell drawer example

  3. Run the following command in Cloud Shell to generate an SSH key pair:
    ssh-keygen -t rsa
  4. At the prompt to enter the file in which to save the key, press Enter to use the default location.
  5. At the passphrase prompt, press Enter for no passphrase, and then press Enter again to confirm.
  6. Run the following command to display the public key, and then copy the output:
    cat $HOME/.ssh/id_rsa.pub
  7. Open the navigation menu. Under Core Infrastructure, go to Compute and click Instances.
  8. Click the instance that you're interested in.
  9. Under Resources, click Console Connection.

  10. Click Create Console Connection.
  11. Select Paste SSH Key and paste the public key contents you copied in step 6.
  12. Click Create Console Connection.

    After the console connection state changes to Active proceed to the next step.

  13. Click the Actions icon (three dots), and then click Copy Serial Console Connection for Linux/Mac.
  14. Paste the connection string copied from the previous step to Cloud Shell, and then press Enter to connect to the console.

  15. Press Enter again to activate the console.

Connecting to the VNC Console

After you create the console connection for the instance, you need to set up a secure tunnel to the VNC server on the instance, and then you can connect with a VNC client.

Caution

The VNC console connection uses SSH port forwarding to create a secure connection from your local system to the VNC server attached to your instance's console. Although this method is a secure way to use VNC over the internet, owners of multiuser systems should know that opening a port on the local system makes it available to all users on that system until a VNC client connects. For this reason, we don't recommend using this product on a multiuser system unless you take proper actions to secure the port or you isolate the VNC client by running it in a virtual environment, such as Oracle VM VirtualBox.
To set up a secure tunnel to the VNC server on the instance using OpenSSH on Mac OS X or Linux
  1. On the instance details page in the Oracle Cloud Infrastructure Console, under Resources, click Console Connection.
  2. Click the Actions icon (three dots), and then click Copy VNC Connection for Linux/Mac.
  3. Paste the connection string copied from the previous step to a terminal window on a Mac OS X or Linux system, and then press Enter to set up the secure connection.
  4. After the connection is established, open your VNC client and specify localhost as the host to connect to and 5900 as the port to use.
Note

Mac OS X Screen Sharing.app Not Compatible with VNC Console Connections

The Mac OS X built-in VNC client, Screen Sharing.app does not work with VNC console connections in Oracle Cloud Infrastructure. Use another VNC client, such as Real VNC Viewer or Chicken.

To set up a secure tunnel to the VNC server on the instance using PowerShell on Windows
Important

If you are connecting to the VNC server on the instance from a Windows client using PowerShell, plink.exe is required. plink.exe is the command link connection tool included with PuTTY. You can install PuTTY or install plink.exe separately. For installation information, see http://www.putty.org.
  1. On the instance details page in the Oracle Cloud Infrastructure Console, under Resources, click Console Connection.
  2. Click the Actions icon (three dots), and then click Copy VNC Connection for Windows.
    Tip

    The copied connection string for Windows contains the parameter -i specifying the location of the private key file. The default value for this parameter in the connection string references an environment variable which might not be configured on your Windows client, or it might not represent the location where the private key file is saved. Verify the value specified for the -i parameter and make any required changes before proceeding to the next step.
  3. Paste the connection string copied from the previous step to Windows Powershell, and then press Enter to set up the secure connection.
  4. After the connection is established, open your VNC client and specify localhost as the host to connect to and 5900 as the port to use.

Note

Secure Connection Warning

When you connect, you might see a warning from the VNC client that the connection is not encrypted. Because you are connecting through SSH, the connection is secure, so this warning is not an issue.

Troubleshooting Instances from Instance Console Connections on Linux

The following tasks describe steps specific to instances running Oracle Autonomous Linux 7.x, Oracle Linux 8.x, and Oracle Linux 7.x, connecting from OpenSSH. Other operating system versions and SSH clients might require different steps.

After you are connected with an instance console connection, you can perform various tasks, such as:

  • Edit system configuration files.
  • Add or reset the SSH keys for the opc user.
  • Reset the password for the opc user.

These tasks require you to boot into a bash shell in maintenance mode.

To boot into maintenance mode
  1. Reboot the instance from the Console.
  2. When the reboot process starts, switch back to the terminal window, and you see Console messages start to appear in the window. As soon as the GRUB boot menu appears, use the up/down arrow key to stop the automatic boot process, enabling you to use the boot menu.
  3. In the boot menu, highlight the top item in the menu, and press e to edit the boot entry.
  4. In edit mode, use the down arrow key to scroll down through the entries until you reach the line that starts with linuxefi for instances running Oracle Autonomous Linux 7.x, Oracle Linux 8.x, and Oracle Linux 7.x.
  5. At the end of that line, add the following:
    init=/bin/bash
  6. Reboot the instance from the terminal window by entering the keyboard shortcut CTRL+X.

When the instance has rebooted, you see the Bash shell command line prompt, and you can proceed with the following procedures.

To edit the system configuration files
  1. From the Bash shell, run the following command to load the SELinux policies to preserve the context of the files you are modifying:
    /usr/sbin/load_policy -i
  2. Run the following command to remount the root partition with read/write permissions:
    /bin/mount -o remount, rw /
  3. Edit the configuration files as needed to try to recover the instance.
  4. After you have finished editing the configuration files, to start the instance from the existing shell, run the following command:
    exec /usr/lib/systemd/systemd

    Alternatively, to reboot the instance, run the following command:

    /usr/sbin/reboot -f
To add or reset the SSH key for the opc user
  1. From the Bash shell, run the following command to load the SELinux policies to preserve the context of the files you are modifying:
    /usr/sbin/load_policy -i
  2. Run the following command to remount the root partition with read/write permissions:
    /bin/mount -o remount, rw /
  3. From the Bash shell, run the following command to change to the SSH key directory for the opc user:
    cd ~opc/.ssh
  4. Rename the existing authorized keys file with the following command:
    mv authorized_keys authorized_keys.old
  5. Replace the contents of the public key file with the new public key file with the following command:
    echo '<contents of public key file>' >> authorized_keys
  6. Restart the instance by running the following command:
    /usr/sbin/reboot -f
To reset the password for the opc user
  1. From the Bash shell, run the following command to load the SELinux policies to preserve the context of the files you are modifying. This step is necessary to sign in to your instance using SSH and the Console.
    /usr/sbin/load_policy -i
  2. Run the following command to remount the root partition with read/write permissions:
    /bin/mount -o remount, rw /
  3. Run the following command to reset the password for the opc user:
    sudo passwd opc
  4. Restart the instance by running the following command:
    sudo reboot -f

Exiting the Instance Console Connection

To exit the serial console connection

When using SSH, the ~ character at the beginning of a new line is used as an escape character.

  • To exit the serial console, enter:

    ~.
  • To suspend the SSH session, enter:

    ~^z

    The ^ character represents the CTRL key

  • To see all the SSH escape commands, enter:

    ~?
To exit the VNC console connection
  1. Close the VNC client.
  2. In the Terminal or PowerShell window, type CTRL C

When you are finished using the console connection, delete the connection for the instance.

To delete the console connection for an instance
  1. Open the navigation menu. Under Core Infrastructure, go to Compute and click Instances.
  2. Click the instance that you're interested in.
  3. Under Resources, click Console Connection.
  4. Click the Actions icon (three dots), and then click Delete. Confirm when prompted.

Tagging Resources

You can add tags to your resources to help you organize them according to your business needs. You can add tags at the time you create a resource, or you can update the resource later with the desired tags. For general information about applying tags, see Resource Tags.
To manage tags for an instance console connection
  1. Open the navigation menu. Under Core Infrastructure, go to Compute and click Instances.
  2. Click the instance that you're interested in.

  3. Under Resources, click Console Connection.

  4. For the console connection that you're interested in, click the Actions icon (three dots) and then click Add Tags. To view existing tags, click View Tags.