10 Troubleshooting Oracle API Platform Cloud Service

Learn about common problems that you might encounter when using Oracle API Platform Cloud Service and find out how to solve them.

Troubleshoot Gateway Issues

Ensure that the Gateway is the latest version. If it's not, download new Gateway installer and install it. See Install the First Gateway Node for a Logical Gateway.

Note:

This Gateway Installer contains Critical Security Fixes. It's a mandatory upgrade for all customers.

Where can I find Gateway related documentation?

Use these links to find Gateway related information:

• Manage Gateways
• System Requirements for On-Premises Gateway Installation
• Prerequisites to Install a Gateway Node
• Install a Gateway Node
• Gateway Node Installer Actions
• gateway-props.json File
• Create a Gateway Node on Oracle Cloud Infrastructure
• REST API for the Gateway Controller in Oracle API Platform Cloud Service

Where can I find Gateway related logs?

• Installer Logs
• Startup Logs
• Logs for Gateway registration, api deployment, and polling
• Gateway Server Logs

Installer Logs

These cover all installation actions.

<INSTALL_ROOT_DIR>/logs/*

Startup Logs

These cover Gateway start and stop events.

<INSTALL_ROOT_DIR>/domain/gateway1/startWls.out

<INSTALL_ROOT_DIR>/domain/gateway1/startMServer.out

<INSTALL_ROOT_DIR>/domain/gateway1/startDb.out

Logs for Gateway registration, api deployment, and polling

<INSTALL_ROOT_DIR>/domain/gateway1/apics/logs/*.log* (.*)

Gateway Server Logs

These log Gateway backend routing:

<INSTALL_ROOT_DIR>/domain/gateway1/apics/logs/default.log

<INSTALL_ROOT_DIR>/domain/gateway1/servers/managedServer1/logs/managedServer1.log(.*)

<INSTALL_ROOT_DIR>/domain/gateway1/servers/managedServer1/trace/default.log(.*)

What are the pre-requisite checks to perform on the host machine before installation

Before installation here are a number of pre-requisites to perform, such as checking permissions, space requirements, and so on.

• Java
• Permissions
• Space requirements
• Firewall ports
• Check the mtu is 1500
• Check there are no existing gateway servers running in the same host
• Proxy settings of host machine
• Credentials
• OCI Security List
• Gateway Weblogic admin/DB password
• Host machine details

Java

Check that JAVA_HOME is pointing to the Oracle approved JDK 8 version, not to the JRE.

• If JAVA_HOME is not set, installation will fail.
• If JAVA_HOME is set to the JRE, during configure, apiplatform_gateway-services_template.jar will fail to create inside the <install loc>/build directory. This will result in starting the managed server failing.

You can set the JAVA_HOME using:

export JAVA_HOME=/u01/jdk1.8.0_211
export PATH=$PATH:$JAVA_HOME/bin
echo $JAVA_HOME

Permissions

Check that the target Gateway installation directory has write permissions. If not, the installer will not be able to copy the required files, and installation will fail.

Space requirements

Check that the target Gateway installation directory has enough space. If not, the installer will not be able to copy the required files, and installation will fail.

Firewall ports

Open the firewall ports defined in gateway-props.json. IIf they are not open, the API request to these ports will fail.

sudo su -c "firewall-offline-cmd --add-port=8011/tcp" root
sudo su -c "firewall-offline-cmd --add-port=8001/tcp" root
sudo su -c "firewall-offline-cmd --add-port=9021/tcp" root
sudo su -c "firewall-offline-cmd --add-port=9022/tcp" root
sudo su -c "/bin/systemctl restart firewalld" root

Check the mtu is 1500

If the mtu is more then 1500 communication to some services will fail, and this may lead to installation failure.

Check it using:

$ ifconfig

The reponse will be similar to this:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.249.134.107 netmask 255.255.248.0 broadcast 10.249.135.255
inet6 2606:b400:2010:6058:221:f6ff:fe79:12f9 prefixlen 64 scopeid 0x0<global>

You can set the mtu to 1500 as follows:

1. As root (sudo su -), run ifconfig -a to get the id/name of your network interface.
2. Run ifconfig <network interface name> mtu 1500.
3. Run ifconfig -a to confirm that the value has been updated.

Check there are no existing gateway servers running in the same host

Make sure there is no existing gateway servers running in the same host, and ensure no existing services are using the ports defined in gateway-props.json. If ports defined in gateway-props.json are used by existing services, Gateway servers will fail to start.

Find existing Gateway related services using:

$ ps -eaf | grep java | grep "NetworkServerControl" | grep GATEWAY_HOME
$ ps -eaf | grep java | grep "weblogic.NodeManager" | grep GATEWAY_HOME
$ ps -eaf | grep java | grep "weblogic.Server" | grep GATEWAY_HOME
$ ps -eaf | grep java | grep "logstash/runner.rb"

If you find existing services, do one of the following before performing a fresh installation:

• Kill the existing services by running : kill -9 <pid>.
• Use different value of ports in gateway-props.json.

Proxy settings of host machine

Be clear about the proxy setting of the Host machine where the Gateway is going to be installed. If proxies are not set properly, registration of the Gateway node (join action) may fail as the Gateway node may not communicate with the management server and the Gateway node may fail to route API calls to the backend.

In gateway-props.json, make sure the correct proxies values are set for managementServiceConnectionProxy and nodeProxy.

• managementServiceConnectionProxy: required if the gateway node needs a proxy to connect to the management service.

• nodeProxy: required if the gateway node needs a proxy to pass client requests to backend services.

For example:

managementServiceConnectionProxy" : ["http://proxy.example.com:80", "https://proxy.example.com:443"],
nodeProxy" : [ "http://proxy.example.com:80", "https://proxy.example.com:443" ]

Credentials

Check that credentials of gateway manager user and gateway runtime user are valid. In particular check that the password has not expired.

If the credentials are invalid, the Gateway join action will fail, that is, the Gateway node registration to Logical Gateway at management console will fail.

OCI Security List

If the Gateway host is in Oracle Cloud Infrastructure, make sure the ports defined in gateway-props.json are updated in the ingress and egress rules of VCN's subnet security list. If ingress and egress rules are not set properly, API calls to directed to the gateway ports will fail.

For more information, see Security Lists.

Gateway Weblogic admin/DB password

Make sure that the password for Gateway Weblogic Admin/DB doesn't have unsupported characters. For example $ should not be present in the password. If it is, it will bring up the servers and DB, however it will cause issues during the connection attempt while running operations such as join or status.

Host machine details

Make sure that the correct details of Gateway Host machine is entered in listenIpAddress and "publishAddress in gateway-props.json. If incorrect details are used install-configure-start actions will fail.

How do I use a custom temp directory for Gateway installation

The default temp directory used by the Gateway installer is /tmp. It expects the user who is performing the install operation to have read/write permission for the /tmp directory.

If for some reason, you don't have read/write permission to the /tmp directory you can specify your own custom temp directory in the JSON gateway properties file.

For example, to use a temp directory called gatewayTmpDir, add the following to gateway-props.json:

gatewayTmpDir" : "/u01/temp

Ensure that that this points to a valid directory, and that the user performing the installation has read/write access to it.

How do I set a custom hostname verifier

If you see the following error in the apics.log, you need to set a custom hostname verification:

javax.ws.rs.ProcessingException: javax.net.ssl.SSLKeyException: 
Hostname verification failed:
HostnameVerifier=weblogic.security.utils.SSLWLSHostnameVerifier

To set custom hostname verification

1. Log into the Administration Server console:

   http://<GW HOST IP>:<gatewayAdminServerPort>/console

2. In the Change Center of the Administration Console, click Lock & Edit.
3. In the left pane of the Console, expand Environment and select Servers.
4. In the Servers table, click the managed server name.
5. In the Settings for managed page, select SSL, and click Advanced.
6. In Hostname Verification select Custom Hostname Verification.
7. In Custom Hostname Verifier enter weblogic.security.utils.SSLWLSWildcardHostnameVerifier and click Save.
8. In the Change Center of the Administration Console, click Activate Changes, and restart the server using:

   ./APIGateway -f gateway-props.json -a stop
   ./APIGateway -fgateway-props.json -a start

I want to change the socket timeout values for backend services calls

If you get java.net.SocketTimeoutException while calling the backend REST service, you can change the socket timeout.

1. Log into the Administration Server console:

   http://<GW HOST IP>:<gatewayAdminServerPort>/console

2. Click OSCG.
3. Click the name of your managed server, then click Container Services.
4. Click DafGeneralInformation, select SocketTimeoutMs.
5. Enter the new socket timeout value, for example 60000 msecs. You may also want to change the ConnectTimeoutMs.
6. Click Update Attributes.
7. Restart the server using:

   ./APIGateway -f gateway-props.json -a stop
   ./APIGateway -fgateway-props.json -a start

I want to increase the maximum number of total connections to backend services

If you find that the Gateway node is unresponsive while performing the load testing, you can increase the maximum number of connections.

1. Log into the Administration Server console:

   http://<GW HOST IP>:<gatewayAdminServerPort>/console

2. Click OSCG.
3. Click the name of your managed server, then click Container Services.
4. Click DafGeneralInformation, select MaxTotalConnections.
5. Enter a new maximum number of connections, for example 6000.
6. Click Update Attributes.
7. Restart the server using:

   ./APIGateway -f gateway-props.json -a stop
   ./APIGateway -fgateway-props.json -a start

I want to increase callout retry times

If you find that the Gateway sometimes responds with HTTP 500, but the backend service does not receive any http request from the Gateway, you can increase the callout retry time. The exception will be ConnectionClosedException, NoHttpResponseException.

1. Log into the Administration Server console:

   http://<GW HOST IP>:<gatewayAdminServerPort>/console

2. Click OSCG.
3. Click the name of your managed server, then click Container Services.
4. Click DafGeneralInformation, select CallOutRetryTimes.
5. The default is 1, but you can make it a larger number.
6. Click Update Attributes.
7. Restart the server using:

   ./APIGateway -f gateway-props.json -a stop
   ./APIGateway -fgateway-props.json -a start

I want to change Gateway APIFirewall settings

If API Gateway keeps throwing a content size exceeds max message size error, you can change ApiFirewall attribute settings. You can do this from the Gateway Admin console (the changed properties only apply to that node), or from the API Platform Cloud Service console (where the changed values apply to all nodes).

The relevant firewall properties are:

• MaxMessageSize : Specifies the maximum size, in bytes, of the request, excluding attachments. The default value is set to 1024000. The maximum allowed value is 200MB.
• MaxUnboundedItems: Specifies the maximum number of unbounded items that a message can contain. The default value is set to 1024.
• MaxItemValueLength: Specifies the maximum size of a single message entity, such as an element, attribute or comment. The default value is 102400.
• MaxChildElementDepth: Specifies the maximum number of nested elements allowed in a message. The default value is 1024 nested elements.

From the Gateway Admin console:

1. Log into the Administration Server console:

   http://<GW HOST IP>:<gatewayAdminServerPort>/console

2. Click OSCG.
3. Click the name of your managed server, then click Container Services.
4. Click ApiFirewall, then change one or more of the values.
5. Click Update Attributes.
6. Restart the server using:

   ./APIGateway -f gateway-props.json -a stop
   ./APIGateway -fgateway-props.json -a start

From the API Platform Cloud Service console:

Note:

Properties you set here apply to all nodes.

1. From the Gateways List page, click the gateway for which you want to configure firewall properties
2. On the Settings tab, update the firewall properties that you want to change.
3. Click Save.

I want to change Gateway overload protection

You can either disable or customize Gatewy overload protection attributes. The default metrics collector is CPU Load.

To check the overload statistics:

1. Log into the Administration Server console:

   http://<GW HOST IP>:<gatewayAdminServerPort>/console

2. Click OSCG.
3. Click the name of your managed server, then click Container Services.
4. Click Overload Protection, then statistics.
5. On the Operations tab, select dumpOverloadStatisticsData and click invoke. The CPU details are output. For example:

   <date, time> The overload statistic snapshot:
   
   Current Overall System Status: NORMAL
   
   {
     "metrics" : [ {
       "name" : "CPU Load",
       "value" : "1.98%"
     } ],
     "serverloads" : [ {
       "name" : "managedServer1",
       "status" : "NORMAL",
       "metrics" : [ {
         "name" : "CPU Load",
         "value" : "1.98%"
       } ]
     } ]
   }
   SumSecondsLightOverload : 0
   SumSecondsMediumOverload : 0
   SumSecondsHighOverload : 0
   TotalRejectedRequests : 0

If you want to disable the overload protection attributes or change the values of the attributes:

1. Log into the Administration Server console, http://<GW HOST IP>:<gatewayAdminServerPort>/console.
2. Click OSCG.
3. Click the name of your managed server, then click Container Services.
4. Click Overload Protection, then Configuration.
5. Select the attributes you want to change. For example:
   • To disable Overload Protection, set:
     • EnableOverloadProtection: false
     • EnableLoadStatisticCollection: false
   • To change other attributes threshold, enter the new value.
6. Click Update Attributes.
7. Restart the server using:

   ./APIGateway -f gateway-props.json -a stop
   ./APIGateway -fgateway-props.json -a start

I want to stop the server shutting down due to overload panic action

If the reponse data size is large, you may find the Gateway node might go down. You can stop this by setting the Panic Action attribute to ignore.

1. Log into the Administration Server console:

   http://<GW HOST IP>:<gatewayAdminServerPort>/console

2. In the Change Center of the Administration Console, click Lock & Edit.
3. In the left pane of the Console, expand Environment and select Servers.
4. In the Servers table, click the managed server name.
5. In the Settings for managed page, select Overload.
6. In Panic Action, select Ignore, take no action and click Save.
7. In the Change Center of the Administration Console, click Activate Changes, and restart the server using:

   ./APIGateway -f gateway-props.json -a stop
   ./APIGateway -fgateway-props.json -a start

I want to enable the HTTP access log

By default the HTTP access log, used for debugging traffic through the Gateway, is not enabled.

1. Log into the Administration Server console:

   http://<GW HOST IP>:<gatewayAdminServerPort>/console

2. In the Change Center of the Administration Console, click Lock & Edit.
3. In the left pane of the Console, expand Logging then HTTP.
4. In the page, make sure that HTTP access log file enabled is checked.
5. Click Advanced.
6. In the Advanced window:
   • In Format, select Extended.
   • In Extended Logging Format Fields, enter this space-delimited string:

     c-ip date time time-taken cs-method cs-uri sc-status

7. Click Save, and in the Change Center of the Administration Console click Activate Changes
8. Restart the server using:

   ./APIGateway -f gateway-props.json -a stop
   ./APIGateway -fgateway-props.json -a start

Reset managementServiceConnectionProxy in already configured Gateway

If you find that the server is up and running but the gateway node is unable to poll the management tier to retrieve the latest APIs, artifacts, entities and so on for deployment, or a similar issue, work through these steps to resolve the problem.

1. Un-register the node either from APIPCS Management console or by running:

   ./APIGateway -f gateway-props.json -a unregister

2. Stop the Gateway servers:

   ./APIGateway -f gateway-props.json -a stop

3. If there is a managementServiceConnectionProxy proxy parameter in gateway-props.json, remove it.

   If there isn't a managementServiceConnectionProxy proxy parameter in gateway-props.json, add it. For example:

   managementServiceConnectionProxy" : ["http://proxy.example.com:80", "https://proxy.example.com:443"]

4. Re-run install-configure-start, and select y when prompted so that the existing installation location is cleaned:

   ./APIGateway -f gateway-props.json -a install-configure-start

5. Check in gateway-props.json that value of logicalGatewayId is assigned to the correct Logical Gateway where you want to register the node.
6. Run the status command to check that the connection from the Gateway Node reaches the API Platform Cloud Service Management Portal:

   ./APIGateway -f gateway-props.json -a status

   If connection is not successful, then the is issue with your network and you should consult your network team.

7. If the connection has been successfully established, use the Run join action to registerthe node with the Logical Gateway:

   ./APIGateway -f gateway-props.json -a join

8. Go to API Platform Cloud Service Management Portal and approve the node's request for registration:
   Wait for the next poll and monitor:

   <INSTALL_ROOT_DIR>/domain/gateway1/apics/logs/apics.log

   You should see the API automatically getting deployed in the new node (only APIs which are currently deployed in the Logical Gateway).

What to do when Gateway installation fails

• Logs to check:
  • <INSTALL_ROOT_DIR>/logs/install.log
• Check that the values used in gateway-props.json are all valid. See gateway-props.json File.
• Common reasons for failure:
  • Check that JAVA_HOME is configured and referencing a JDK on the system ( not a JRE).
  • Check that there is sufficient space in the installation directory, and that it has write permission.
  • Check that the default temp directory, /tmp, has write permission.
  • If the size of the downloaded installer is less than 1GB it is corrupted. For checksum, get in touch with Oracle Support.
  • gateway-props.json is corrupted, for example, it might not be formated properly, or it could have unknown characters that can't be encoded/decoded by Python.

The Gateway start action fails to start the servers

• Logs to check:
  • <INSTALL_ROOT_DIR>/logs/startl.log
  • <INSTALL_ROOT_DIR>/domain/gateway1/startWls.out
  • <INSTALL_ROOT_DIR>/domain/gateway1/startMServer.out
  • <INSTALL_ROOT_DIR>/domain/gateway1/startDb.out
• Check that the values used in gateway-props.json are all valid. See gateway-props.json File.
• Common reasons for failure:

  • Check that the ports defined in gateway-props.json are not being used by existing services.

    If they are, find the existing services and kill them.

  • Check that the database connection URL or port is not being used by any other existing gateway installation.

    If they are, modify gateway-props.json and change the port numbers to different values.Then run install-configure.

  Refer to the steps in What are the pre-requisite checks to perform on the host machine before installation.

The Gateway restart action fails to restart the servers

• Logs to check:
  • <INSTALL_ROOT_DIR>/logs/start.log
  • <INSTALL_ROOT_DIR>/domain/gateway1/startWls.out
  • <INSTALL_ROOT_DIR>/domain/gateway1/startMServer.out
  • <INSTALL_ROOT_DIR>/domain/gateway1/startDb.out
  • <INSTALL_ROOT_DIR>/domain/gateway1/apics/logs/apics.log
• Common reasons for failure:
  • Check in the apics.log for

    javax.net.ssl.SSLKeyException: Hostnameverification failed:
    HostnameVerifier=weblogic.security.utils.SSLWLSHostnameVerifier

    To stop it failing, see How do I set a custom hostname verifier.

The Gateway join actions fails

• Check these logs:
  • <INSTALL_ROOT_DIR>/logs/join.log
  • <INSTALL_ROOT_DIR>/domain/gateway1/apics/logs/apics.log
• Check that the values used in gateway-props.json are all valid. See gateway-props.json File.
• Common reasons for failure:
  • Check that the ports defined in gateway-props.json are not being used by existing services. If they are, find the existing services and kill them.
  • Check that the database connection URL or port is not being used by any other existing gateway installation. If they are, modify gateway-props.json and change the port numbers to different values.Then run install-configure.

  Refer to the steps in What are the pre-requisite checks to perform on the host machine before installation.

The Gateway fails to poll

• Check these logs:
  • <INSTALL_ROOT_DIR>/domain/gateway1/apics/logs/apics.log
• Common reasons for failure:
  • Failure to connect to the management portal due to a proxy issue.
  • The Gateway runtime user credential has expired.
  • The hostname verification has failed.
  • Network proxy issue.
  Refer to the steps in:

  • What are the pre-requisite checks to perform on the host machine before installation
  • How do I set a custom hostname verifier
  • How do I set a custom hostname verifier

Does Oracle API Platform Cloud Service support proxy with credentials

ERROR Property = managementServiceConnectionProxy is not of type list

Oracle API Platform Cloud Service doesn't currently support proxies that use usernames and passwords.

Slow Gateway performance

1. Upgrade Java to use the latest JDK.
2. Increase MaxTotalConnections:
   1. Log into the Administration Server console:

      http://<GW HOST IP>:<gatewayAdminServerPort>/console

   2. Click OSCG.
   3. Click the name of your managed server, then click Container Services.
   4. Click DafGeneralInformation, select MaxTotalConnections.
   5. Increase MaxTotalConnections based on the load. For example, if it is 4000 increate it to 6000.
   6. Click Update Attributes.
   7. Restart the server using:

      ./APIGateway -f gateway-props.json -a stop
      ./APIGateway -fgateway-props.json -a start

3. If out of memory, increase the JVM heap size of the Gateway managed server:
   1. Stop the Gateway servers

       ./APIGateway -f gateway-props.json -a stop

   2. The JVM heap size is set in the setDomainEnv.sh file which is under <Gateway Install location>/domain/gateway1/bin.
      The default values for the Gateway WebLogic server are:

      • Minimum heap size (Xms): 2G
      • Maximum heap size (Xmx): 4G

      If the application needs more than 4G heap memory, search for Xmx in setDomainEnv.sh and increase the value.

   3. Start the gateway servers:

      ./APIGateway -f gateway-props.json -a start

Create user in the Gateway realm to use Basic Auth

If you have an API with a basic authentication policy and try to call it, you may get a response of 401 Unauthorized. To resolve this, create a user and add them to a group in the Gateway WebLogic security realm. See Create a group and add the user to it for Basic Auth.

1. Log into the Administration Server console:

   http://<GW HOST IP>:<gatewayAdminServerPort>/console

2. Click Security Realms.
3. In the Summary of Security Realms page, under Realms, click myrealm.
4. In the Settings for myrealm page, click User and Groups and then click the Users tab.
5. In the Users page, click New.
6. In the Create a New User page, enter the user name, a description, the password, and confirm the password. For the provider, choose the default DefaultAuthenticator.
7. Click OK.

   Specify the new user in the policy. See Apply Basic Authentication Policies.

Create a group and add the user to it for Basic Auth

If you have an API with a basic authentication policy and try to call it, you may get a response of 401 Unauthorized. To resolve this, add a user to a group in the Gateway WebLogic security realm. See Create user in the Gateway realm to use Basic Auth.

1. Log into the Administration Server console:

   http://<GW HOST IP>:<gatewayAdminServerPort>/console

2. Click Security Realms.
3. In the Summary of Security Realms page, under Realms, click myrealm.
4. In the Settings for myrealm page, click User and Groups and then click the Groups tab.
5. In the Groups page, click New.
6. In the Create a New Group page, enter the group name and a description. For the provider, choose the default DefaultAuthenticator.

   Specify the group name in the policy. See Apply Basic Authentication Policies.

7. Click OK.
8. In the Users page, select the user name that you want to add to the group.

   Do not only select the check box, click the name as well.

9. On the Settings for <user_name> page, click the Groups tab, then select the group in the Available column and click the right arrow to move the group to the Chosen column.
10. Click Save.
11. Use the breadcrumbs at the top of the console to go back to the Users table, and repeat the same process for other users.

Change the Gateway WebLogic admin and Derby DB credentials

The API Platform Gateway WebLogic admin password and the Derby DB password have to be the same otherwise the server will not restart. So if you change either password, you must change the other.

1. To change the WebLogic admin password:
   1. Log into the Administration Server console:

      http://<GW HOST IP>:<gatewayAdminServerPort>/console

   2. Click Security Realms.
   3. In the Summary of Security Realms page, under Realms, click myrealm.
   4. In the Settings for myrealm page, click User and Groups and then click the Groups tab.
   5. Click on the Weblogic admin user and then Passwords.
   6. Enter the new password and confirm it, and click Save.
2. To change the Derby DB password:
   1. SSH to the Gateway host machine.
   2. Confirm that the DB is running in Gateway host machine:

      ps -eaf | grep NetworkServerControl

   3. Go to the Derby DB home lib folder:

      $ cd <GATEWAY_INSTALL_LOC>/GATEWAY_HOME/wlserver/common/derby/lib

   4. Run the ij utility:

      $ java -jar derbyrun.jar ij

   5. Once ij is up, run the connect command (using the appropriate hostname/ip, port and password):

      ij> CONNECT 'jdbc:derby://<HOSTNAME_OR_IP>:<PORT>/gatekeeper;user=gatekeeper;password=<WLS_ADMIN_PASSWORD>';

      For example:

      CONNECT 'jdbc:derby://slxxxxx.us.oracle.com:1527/gatekeeper;user=gatekeeper;password=welcome1';

      Note:

      • The hostname/ip is the value of listenIpAddress in gateway-props.json.
      • The DB User name is gatekeeper.
      • The password for the DB user is same as the WLS admin password.
   6. Once connected, update the password of the gatekeeper user.

      ij> CALL SYSCS_UTIL.SYSCS_SET_DATABASE_PROPERTY('derby.user.gatekeeper', '<WLS_NEW_ADMIN_PASSWORD>');

      For example:

      CALL SYSCS_UTIL.SYSCS_SET_DATABASE_PROPERTY('derby.user.gatekeeper', 'welcome123');

   7. Disconnect and then verify the password by trying to connect again:

      ij> disconnect;
      ij> CONNECT 'jdbc:derby://<HOSTNAME_OR_IP>:<PORT>/gatekeeper;user=gatekeeper;password=<WLS_NEW_ADMIN_PASSWORD>';

      For example:

      ij> CONNECT 'jdbc:derby://slxxxxx.us.oracle.com:1527/gatekeeper;user=gatekeeper;password=welcome123';

      If the password reset has been successful, then the connection will be made.

3. Change the JDB data source password:
   1. Login again to the WebLogic admin console.
   2. Click Lock & Edit.
   3. Go to Services, then Data Sources and open the Configuration tab.
   4. From the list of data sources, click wlng.datasource and open the Connection Pool tab.
   5. Enter the new password and confirm it.
   6. From the list of data sources, click wlng.localTX.datasource and open the Connection Pool tab.
   7. Enter the new password for this data source and confirm it.
   8. Click Activate Changes.
4. Verify the changes by restarting the Gateway servers:
   1. Restart from the WebLogic console, or start the Gateway servers from the Host machine:
      1. SSH to Gateway host machine.
      2. Set JAVA_HOME
      3. Stop the Gateway servers and the Derby DB:

         /APIGateway -f gateway-props.json -a stop

      4. Or pgrep java | xargs kill -9
      5. Start the Gateway servers and the Derby DB:

         /APIGateway -f gateway-props.json -a start

Cannot connect to Gateway Derby DB

Find out to check the connection, and learn about some of the root causes why a connection will fail.

• Check the Gateway Derby DB connection
• Root causes

Check the Gateway Derby DB connection

1. To check whether the DB is running in Gateway host machine:

   ps -eaf | grep NetworkServerControl

2. Go to the Derby DB home lib folder:

   $ cd
       <GATEWAY_INSTALL_LOC>/GATEWAY_HOME/wlserver/common/derby/lib

3. Run the ij utility:

   $ java -jar derbyrun.jar ij

4. Once ij is up, run the connect command (using the appropriate hostname/ip, port and password):

   ij> CONNECT 'jdbc:derby://<HOSTNAME_OR_IP>:<PORT>/gatekeeper;user=gatekeeper;password=<WLS_ADMIN_PASSWORD>';

   For example:

   CONNECT 'jdbc:derby://slxxxxx.example.com:1527/gatekeeper;user=gatekeeper;password=welcome1';

   Note:

   • The hostname/ip is the value of listenIpAddress in gateway-props.json.
   • The DB User name is gatekeeper.
   • The password for the DB user is same as the WLS admin password.
5. If the connection is not successful, check the parameters. Also, check the Derby DB log at:

   <GATEWAY_INSTALL_LOC>/GATEWAY_HOME/wlserver/common/derby/derby.log

6. If the connection is successful, run a basic command and then disconnect:

   ij> VALUES CURRENT_TIMESTAMP;
   ij> disconnect;
   ij> exit;

Root causes

• The DB password you specified is wrong.
  • Solution: Make sure the correct password (WebLogic admin password) is used.
• The DB password uses unsupported special characters, such as $.
  • Solution: Re-install the gateway using a password string which doesn't use unsupported special characters.
• The DB host machine, which is the listenIpAddress specified in gateway-props.json, is not reachable.
  • Solution: Use the correct private/public IP or hostname of the gateway node machine in gateway-props.json, and re-install.
• The DB port is already in use.
  • Solution: Use a different DB port in gateway-props.json, and re-install.
• Derby is failing to start or processes getting killed.
  • Solution: Derby DB is corrupted because of a forced stop. To fix it, restart the host machine and run the start operation.

Note:

• The DB connect hostname/ip is value of listenIpAddress in gateway-props.json.
• The DB User name is gatekeeper.
• The password for DB user is same as the WebLogic admin password.

Allow enabling cookies to be passed to the backend service

A flag has to be set for the Gateway to pass cookies to the backend server.

1. Upgrade Java to use the latest JDK.
2. Increase MaxTotalConnections:
   1. Log into the Administration Server console:

      http://<GW HOST IP>:<gatewayAdminServerPort>/console

   2. Click OSCG.
   3. Click the name of your managed server, then click Container Services.
   4. Click DafGeneralInformation.
   5. Set enableSouthCookie to true.
   6. Click Update Attributes.
   7. Restart the server using:

      ./APIGateway -f gateway-props.json -a stop
      ./APIGateway -fgateway-props.json -a start

Update Gateway threat protection configurations

1. Log into the Administration Server console:

   http://<GW HOST IP>:<gatewayAdminServerPort>/console

2. Click OSCG.
3. Click the name of your managed server, then click Reporter.
   • To see all current threat protections configurations, go to the Attributes tab
   • To get the full JSON configuration for a specific threat definition:
     1. Go to the Operations tab, select getFullThreatConfiguration and enter name of the threat protection.
     2. Click Invoke.
        An example of the JSON file of a threat configuration:

            {
                "name": "ApiFwFail",
                "actions": [
                    "ALARM",
                    "BLOCK"
                ],
                "trackedEntities": [
                    "IP",
                    "APP_INST_ACCOUNT"
                ],
                "maxTrackedEntities": 10000,
                "maxViolations": 10,
                "violationClearTime": 1,
                "description": "Generic API-Firewall Threat. Repeated API FW violations could be penetration tests, an alarm is sent when this happens, and further traffic is blocked"
            }

4. To update a threat attribute, on the Operations tab select updateThreatConfiguration and under Json: enter the updated threat attribute and click Invoke.

   For example, to update maxViolations of ApiFwFailto 20:

       {
           "name": "ApiFwFail",
           "actions": [
               "ALARM",
               "BLOCK"
           ],
           "trackedEntities": [
               "IP",
               "APP_INST_ACCOUNT"
           ],
           "maxTrackedEntities": 10000,
           "maxViolations": 20,
           "violationClearTime": 1,
           "description": "Generic API-Firewall Threat. Repeated API FW violations could be penetration tests, an alarm is sent when this happens, and further traffic is blocked"
       }

5. To mute all threats protections, to to the Operations tab, select select updateThreatConfiguration and under Json: enter:

    {
      "actions":[
      ]
    }

Show threat protection alarm description in logs

At runtime, the best way to see the effects of threat protection is to view the alarms generated by the gateway. Follow this example to have alarm descriptions appear in logs:

1. Log into the Administration Server console:

   http://<GW HOST IP>:<gatewayAdminServerPort>/console

2. Click OSCG.
3. Click the name of your managed server, then click Reporter and go to the Operations tab.
4. Click updateThreatConfiguration.
5. To update a threat attribute, on the Operations tab select updateThreatConfiguration and under Json: enter the updated threat attribute and click Invoke.

   For example, to add a description to the AppKeyLoginFail threat:

   {
     "name":"AppKeyLoginFail",
     "actions":[
       "ALARM",
       "BLOCK"
     ],
     "trackedEntities":[
       "IP"
     ],
     "maxTrackedEntities":10000,
     "maxViolations":3,
     "violationClearTime":30,
     "description":"Login-Fail Threat. Triggered for wrong x-app-key values, can
   only be armed for IP. Use this to prevent brute force appkey guessing"
   }

WebLogic vulnerability attack, 503 Service Unavailable, Gateway start getting killed

• Symptoms
• Diagnosis
• Quick solution
• Full solution

Symptoms

• An intermittent HTTP/1.1 503 Service Unavailable error while making called to deployed APIs.
• In the default.logs, you'll notice OVERLOAD PROTECTION has kicked in frequently and traffic is getting throttled.
• Gateway start keeps on getting killed.

Diagnosis

• Run the top command to identify CPU and memory usage.
• Check if there is an unknown process hogging CPU more than 100%, for example /tmp/javax/sshd2.
• Note the process ID.
• Check the process working directory pwdx <process id>.
• Get tids:

  $ ps -eLo pid,ppid,tid,pcpu,comm | grep <pid>

• Get thread dump of the process:

  $ kill -3 <pid> >> threaddumb.out

  or

  $ jstack -l <pid> >> threaddump.out

If getting thread dumps fails, it's a sign that the process is a malware.

Quick solution

1. Stop the Gateway:

   ./APIGateway -f gateway-props.json -a stop

2. Kill the CPU hogging process:

   sudo kill -9 <pid>

3. Delete the CPU hogging executable and its mother directory. Also, delete any file from the location found from running pwdx <pid>, for example:

   rm -rf /tmp/javax/sshd2
   rm -rf /tmp/javax

4. Clear the tmp directory:

   sudo rm -rf /tmp/*

5. Reboot the system.
6. Start the Gateway:

   ./APIGateway -f gateway-props.json -a start

Full solution

For a long term solution, you should:

• Decommission the Gateway node.
• Use a new Host.
• Download the latest installer.
• Install, configure, start a new Gateway node using the latest installer.
• Join to the specific logical Gateway.

Note:

• During installation, use use strong credentials for the Weblogic admin.
• Do not expose the WebLogic admin console to the internet, unless it is required.