10 Troubleshooting Oracle API Platform Cloud Service
Learn about common problems that you might encounter when using Oracle API Platform Cloud Service and find out how to solve them.
Troubleshoot Gateway Issues
Note:
This Gateway Installer contains Critical Security Fixes. It's a mandatory upgrade for all customers.
Where can I find Gateway related documentation?
Use these links to find Gateway related information:
- Manage Gateways
- System Requirements for On-Premises Gateway Installation
- Prerequisites to Install a Gateway Node
- Install a Gateway Node
- Gateway Node Installer Actions
- gateway-props.json File
- Create a Gateway Node on Oracle Cloud Infrastructure
- REST API for the Gateway Controller in Oracle API Platform Cloud Service
Where can I find Gateway related logs?
- Installer Logs
- Startup Logs
- Logs for Gateway registration, api deployment, and polling
- Gateway Server Logs
Installer Logs
These cover all installation actions.
<INSTALL_ROOT_DIR>/logs/*
Startup Logs
These cover Gateway start and stop events.
<INSTALL_ROOT_DIR>/domain/gateway1/startWls.out
<INSTALL_ROOT_DIR>/domain/gateway1/startMServer.out
<INSTALL_ROOT_DIR>/domain/gateway1/startDb.out
Logs for Gateway registration, api deployment, and polling
<INSTALL_ROOT_DIR>/domain/gateway1/apics/logs/*.log* (.*)
Gateway Server Logs
These log Gateway backend routing:
<INSTALL_ROOT_DIR>/domain/gateway1/apics/logs/default.log
<INSTALL_ROOT_DIR>/domain/gateway1/servers/managedServer1/logs/managedServer1.log(.*)
<INSTALL_ROOT_DIR>/domain/gateway1/servers/managedServer1/trace/default.log(.*)
What are the pre-requisite checks to perform on the host machine before installation
Before installation here are a number of pre-requisites to perform, such as checking permissions, space requirements, and so on.
- Java
- Permissions
- Space requirements
- Firewall ports
- Check the mtu is 1500
- Check there are no existing gateway servers running in the same host
- Proxy settings of host machine
- Credentials
- OCI Security List
- Gateway Weblogic admin/DB password
- Host machine details
Java
Check that JAVA_HOME is pointing to the Oracle approved JDK 8 version, not to the JRE.
- If JAVA_HOME is not set, installation will fail.
- If JAVA_HOME is set to the JRE, during configure,
apiplatform_gateway-services_template.jar
will fail to create inside the<install loc>/build
directory. This will result in starting the managed server failing.
You can set the JAVA_HOME using:
export JAVA_HOME=/u01/jdk1.8.0_211
export PATH=$PATH:$JAVA_HOME/bin
echo $JAVA_HOME
Permissions
Check that the target Gateway installation directory has write permissions. If not, the installer will not be able to copy the required files, and installation will fail.
Space requirements
Check that the target Gateway installation directory has enough space. If not, the installer will not be able to copy the required files, and installation will fail.
Firewall ports
Open the firewall ports defined in gateway-props.json. IIf they are not open, the API request to these ports will fail.
sudo su -c "firewall-offline-cmd --add-port=8011/tcp" root
sudo su -c "firewall-offline-cmd --add-port=8001/tcp" root
sudo su -c "firewall-offline-cmd --add-port=9021/tcp" root
sudo su -c "firewall-offline-cmd --add-port=9022/tcp" root
sudo su -c "/bin/systemctl restart firewalld" root
Check the mtu is 1500
If the mtu is more then 1500 communication to some services will fail, and this may lead to installation failure.
Check it using:
$ ifconfig
The reponse will be similar to this:
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.249.134.107 netmask 255.255.248.0 broadcast 10.249.135.255
inet6 2606:b400:2010:6058:221:f6ff:fe79:12f9 prefixlen 64 scopeid 0x0<global>
You can set the mtu to 1500 as follows:
- As root (sudo su -), run
ifconfig -a
to get the id/name of your network interface. - Run
ifconfig <network interface name> mtu 1500
. - Run
ifconfig -a
to confirm that the value has been updated.
Check there are no existing gateway servers running in the same host
Make sure there is no existing gateway servers running in the same host, and ensure no existing services are using the ports defined in gateway-props.json
. If ports defined in gateway-props.json
are used by existing services, Gateway servers will fail to start.
Find existing Gateway related services using:
$ ps -eaf | grep java | grep "NetworkServerControl" | grep GATEWAY_HOME
$ ps -eaf | grep java | grep "weblogic.NodeManager" | grep GATEWAY_HOME
$ ps -eaf | grep java | grep "weblogic.Server" | grep GATEWAY_HOME
$ ps -eaf | grep java | grep "logstash/runner.rb"
If you find existing services, do one of the following before performing a fresh installation:
- Kill the existing services by running
: kill -9 <pid>
. - Use different value of ports in
gateway-props.json
.
Proxy settings of host machine
Be clear about the proxy setting of the Host machine where the Gateway is going to be installed. If proxies are not set properly, registration of the Gateway node (join action) may fail as the Gateway node may not communicate with the management server and the Gateway node may fail to route API calls to the backend.
In gateway-props.json
, make sure the correct proxies values are set for managementServiceConnectionProxy
and nodeProxy
.
-
managementServiceConnectionProxy: required if the gateway node needs a proxy to connect to the management service.
-
nodeProxy: required if the gateway node needs a proxy to pass client requests to backend services.
For example:
managementServiceConnectionProxy" : ["http://proxy.example.com:80", "https://proxy.example.com:443"],
nodeProxy" : [ "http://proxy.example.com:80", "https://proxy.example.com:443" ]
Credentials
Check that credentials of gateway manager user and gateway runtime user are valid. In particular check that the password has not expired.
If the credentials are invalid, the Gateway join action will fail, that is, the Gateway node registration to Logical Gateway at management console will fail.
OCI Security List
If the Gateway host is in Oracle Cloud Infrastructure, make sure the ports defined in gateway-props.json
are updated in the ingress and egress rules of VCN's subnet security list. If ingress and egress rules are not set properly, API calls to directed to the gateway ports will fail.
For more information, see Security Lists.
Gateway Weblogic admin/DB password
Make sure that the password for Gateway Weblogic Admin/DB doesn't have unsupported characters. For example $
should not be present in the password. If it is, it will bring up the servers and DB, however it will cause issues during the connection attempt while running operations such as join
or status
.
Host machine details
Make sure that the correct details of Gateway Host machine is entered in listenIpAddress
and "publishAddress
in gateway-props.json
. If incorrect details are used install-configure-start actions will fail.
How do I use a custom temp directory for Gateway installation
The default temp directory used by the Gateway installer is /tmp
. It expects the user who is performing the install operation to have read/write permission for the /tmp
directory.
If for some reason, you don't have read/write permission to the /tmp
directory you can specify your own custom temp directory in the JSON gateway properties file.
For example, to use a temp directory called gatewayTmpDir
, add the following to gateway-props.json
:
gatewayTmpDir" : "/u01/temp
Ensure that that this points to a valid directory, and that the user performing the installation has read/write access to it.
How do I set a custom hostname verifier
If you see the following error in the apics.log, you need to set a custom hostname verification:
javax.ws.rs.ProcessingException: javax.net.ssl.SSLKeyException:
Hostname verification failed:
HostnameVerifier=weblogic.security.utils.SSLWLSHostnameVerifier
To set custom hostname verification
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- In the Change Center of the Administration Console, click Lock & Edit.
- In the left pane of the Console, expand Environment and select Servers.
- In the Servers table, click the managed server name.
- In the Settings for managed page, select SSL, and click Advanced.
- In Hostname Verification select Custom Hostname Verification.
- In Custom Hostname Verifier enter
weblogic.security.utils.SSLWLSWildcardHostnameVerifier
and click Save. - In the Change Center of the Administration Console, click Activate Changes, and restart the server using:
./APIGateway -f gateway-props.json -a stop ./APIGateway -fgateway-props.json -a start
I want to change the socket timeout values for backend services calls
If you get java.net.SocketTimeoutException
while calling the backend REST service, you can change the socket timeout.
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- Click OSCG.
- Click the name of your managed server, then click Container Services.
- Click DafGeneralInformation, select SocketTimeoutMs.
- Enter the new socket timeout value, for example 60000 msecs. You may also want to change the ConnectTimeoutMs.
- Click Update Attributes.
- Restart the server using:
./APIGateway -f gateway-props.json -a stop ./APIGateway -fgateway-props.json -a start
I want to increase the maximum number of total connections to backend services
If you find that the Gateway node is unresponsive while performing the load testing, you can increase the maximum number of connections.
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- Click OSCG.
- Click the name of your managed server, then click Container Services.
- Click DafGeneralInformation, select MaxTotalConnections.
- Enter a new maximum number of connections, for example 6000.
- Click Update Attributes.
- Restart the server using:
./APIGateway -f gateway-props.json -a stop ./APIGateway -fgateway-props.json -a start
I want to increase callout retry times
If you find that the Gateway sometimes responds with HTTP 500, but the backend service does not receive any http request from the Gateway, you can increase the callout retry time. The exception will be ConnectionClosedException, NoHttpResponseException.
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- Click OSCG.
- Click the name of your managed server, then click Container Services.
- Click DafGeneralInformation, select CallOutRetryTimes.
- The default is 1, but you can make it a larger number.
- Click Update Attributes.
- Restart the server using:
./APIGateway -f gateway-props.json -a stop ./APIGateway -fgateway-props.json -a start
I want to change Gateway APIFirewall settings
If API Gateway keeps throwing a content size exceeds max message size error, you can change ApiFirewall attribute settings. You can do this from the Gateway Admin console (the changed properties only apply to that node), or from the API Platform Cloud Service console (where the changed values apply to all nodes).
The relevant firewall properties are:
- MaxMessageSize : Specifies the maximum size, in bytes, of the request, excluding attachments. The default value is set to 1024000. The maximum allowed value is 200MB.
- MaxUnboundedItems: Specifies the maximum number of unbounded items that a message can contain. The default value is set to 1024.
- MaxItemValueLength: Specifies the maximum size of a single message entity, such as an element, attribute or comment. The default value is 102400.
- MaxChildElementDepth: Specifies the maximum number of nested elements allowed in a message. The default value is 1024 nested elements.
From the Gateway Admin console:
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- Click OSCG.
- Click the name of your managed server, then click Container Services.
- Click ApiFirewall, then change one or more of the values.
- Click Update Attributes.
- Restart the server using:
./APIGateway -f gateway-props.json -a stop ./APIGateway -fgateway-props.json -a start
From the API Platform Cloud Service console:
Note:
Properties you set here apply to all nodes.- From the Gateways List page, click the gateway for which you want to configure firewall properties
- On the Settings tab, update the firewall properties that you want to change.
- Click Save.
I want to change Gateway overload protection
You can either disable or customize Gatewy overload protection attributes. The default metrics collector is CPU Load
.
To check the overload statistics:
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- Click OSCG.
- Click the name of your managed server, then click Container Services.
- Click Overload Protection, then statistics.
- On the Operations tab, select dumpOverloadStatisticsData and click invoke. The CPU details are output. For example:
<date, time> The overload statistic snapshot: Current Overall System Status: NORMAL { "metrics" : [ { "name" : "CPU Load", "value" : "1.98%" } ], "serverloads" : [ { "name" : "managedServer1", "status" : "NORMAL", "metrics" : [ { "name" : "CPU Load", "value" : "1.98%" } ] } ] } SumSecondsLightOverload : 0 SumSecondsMediumOverload : 0 SumSecondsHighOverload : 0 TotalRejectedRequests : 0
If you want to disable the overload protection attributes or change the values of the attributes:
- Log into the Administration Server console,
http://<GW HOST IP>:<gatewayAdminServerPort>/console
. - Click OSCG.
- Click the name of your managed server, then click Container Services.
- Click Overload Protection, then Configuration.
- Select the attributes you want to change. For example:
- To disable Overload Protection, set:
- EnableOverloadProtection: false
- EnableLoadStatisticCollection: false
- To change other attributes threshold, enter the new value.
- To disable Overload Protection, set:
- Click Update Attributes.
- Restart the server using:
./APIGateway -f gateway-props.json -a stop ./APIGateway -fgateway-props.json -a start
I want to stop the server shutting down due to overload panic action
If the reponse data size is large, you may find the Gateway node might go down. You can stop this by setting the Panic Action attribute to ignore
.
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- In the Change Center of the Administration Console, click Lock & Edit.
- In the left pane of the Console, expand Environment and select Servers.
- In the Servers table, click the managed server name.
- In the Settings for managed page, select Overload.
- In Panic Action, select Ignore, take no action and click Save.
- In the Change Center of the Administration Console, click Activate Changes, and restart the server using:
./APIGateway -f gateway-props.json -a stop ./APIGateway -fgateway-props.json -a start
I want to enable the HTTP access log
By default the HTTP access log, used for debugging traffic through the Gateway, is not enabled.
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- In the Change Center of the Administration Console, click Lock & Edit.
- In the left pane of the Console, expand Logging then HTTP.
- In the page, make sure that HTTP access log file enabled is checked.
- Click Advanced.
- In the Advanced window:
- In Format, select Extended.
- In Extended Logging Format Fields, enter this space-delimited string:
c-ip date time time-taken cs-method cs-uri sc-status
- Click Save, and in the Change Center of the Administration Console click Activate Changes
- Restart the server using:
./APIGateway -f gateway-props.json -a stop ./APIGateway -fgateway-props.json -a start
Reset managementServiceConnectionProxy in already configured Gateway
If you find that the server is up and running but the gateway node is unable to poll the management tier to retrieve the latest APIs, artifacts, entities and so on for deployment, or a similar issue, work through these steps to resolve the problem.
- Un-register the node either from APIPCS Management console or by running:
./APIGateway -f gateway-props.json -a unregister
- Stop the Gateway servers:
./APIGateway -f gateway-props.json -a stop
-
If there is a managementServiceConnectionProxy proxy parameter in
gateway-props.json
, remove it.If there isn't a managementServiceConnectionProxy proxy parameter ingateway-props.json
, add it. For example:managementServiceConnectionProxy" : ["http://proxy.example.com:80", "https://proxy.example.com:443"]
- Re-run
install-configure-start
, and selecty
when prompted so that the existing installation location is cleaned:./APIGateway -f gateway-props.json -a install-configure-start
- Check in
gateway-props.json
that value oflogicalGatewayId
is assigned to the correct Logical Gateway where you want to register the node. - Run the status command to check that the connection from the Gateway Node reaches the API Platform Cloud Service Management Portal:
./APIGateway -f gateway-props.json -a status
If connection is not successful, then the is issue with your network and you should consult your network team.
- If the connection has been successfully established, use the Run join action to registerthe node with the Logical Gateway:
./APIGateway -f gateway-props.json -a join
- Go to API Platform Cloud Service Management Portal and approve the node's request for registration:
Wait for the next poll and monitor:
<INSTALL_ROOT_DIR>/domain/gateway1/apics/logs/apics.log
You should see the API automatically getting deployed in the new node (only APIs which are currently deployed in the Logical Gateway).
What to do when Gateway installation fails
- Logs to check:
<INSTALL_ROOT_DIR>/logs/install.log
- Check that the values used in
gateway-props.json
are all valid. See gateway-props.json File. - Common reasons for failure:
- Check that
JAVA_HOME
is configured and referencing a JDK on the system ( not a JRE). - Check that there is sufficient space in the installation directory, and that it has write permission.
- Check that the default temp directory,
/tmp
, has write permission. - If the size of the downloaded installer is less than 1GB it is corrupted. For checksum, get in touch with Oracle Support.
gateway-props.json
is corrupted, for example, it might not be formated properly, or it could have unknown characters that can't be encoded/decoded by Python.
- Check that
The Gateway start action fails to start the servers
- Logs to check:
<INSTALL_ROOT_DIR>/logs/startl.log
<INSTALL_ROOT_DIR>/domain/gateway1/startWls.out
<INSTALL_ROOT_DIR>/domain/gateway1/startMServer.out
<INSTALL_ROOT_DIR>/domain/gateway1/startDb.out
- Check that the values used in
gateway-props.json
are all valid. See gateway-props.json File. - Common reasons for failure:
-
Check that the ports defined in
gateway-props.json
are not being used by existing services.If they are, find the existing services and kill them.
-
Check that the database connection URL or port is not being used by any other existing gateway installation.
If they are, modify
gateway-props.json
and change the port numbers to different values.Then runinstall-configure
.
Refer to the steps in What are the pre-requisite checks to perform on the host machine before installation.
-
The Gateway restart action fails to restart the servers
- Logs to check:
<INSTALL_ROOT_DIR>/logs/start.log
<INSTALL_ROOT_DIR>/domain/gateway1/startWls.out
<INSTALL_ROOT_DIR>/domain/gateway1/startMServer.out
<INSTALL_ROOT_DIR>/domain/gateway1/startDb.out
<INSTALL_ROOT_DIR>/domain/gateway1/apics/logs/apics.log
- Common reasons for failure:
-
Check in the
apics.log
forjavax.net.ssl.SSLKeyException: Hostnameverification failed: HostnameVerifier=weblogic.security.utils.SSLWLSHostnameVerifier
To stop it failing, see How do I set a custom hostname verifier.
-
The Gateway join actions fails
- Check these logs:
<INSTALL_ROOT_DIR>/logs/join.log
<INSTALL_ROOT_DIR>/domain/gateway1/apics/logs/apics.log
- Check that the values used in
gateway-props.json
are all valid. See gateway-props.json File. - Common reasons for failure:
- Check that the ports defined in
gateway-props.json
are not being used by existing services. If they are, find the existing services and kill them. - Check that the database connection URL or port is not being used by any other existing gateway installation. If they are, modify
gateway-props.json
and change the port numbers to different values.Then runinstall-configure
.
Refer to the steps in What are the pre-requisite checks to perform on the host machine before installation.
- Check that the ports defined in
The Gateway fails to poll
- Check these logs:
<INSTALL_ROOT_DIR>/domain/gateway1/apics/logs/apics.log
- Common reasons for failure:
- Failure to connect to the management portal due to a proxy issue.
- The Gateway runtime user credential has expired.
- The hostname verification has failed.
- Network proxy issue.
Does Oracle API Platform Cloud Service support proxy with credentials
ERROR Property = managementServiceConnectionProxy is not of type list
Oracle API Platform Cloud Service doesn't currently support proxies that use usernames and passwords.
Slow Gateway performance
- Upgrade Java to use the latest JDK.
- Increase MaxTotalConnections:
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- Click OSCG.
- Click the name of your managed server, then click Container Services.
- Click DafGeneralInformation, select MaxTotalConnections.
- Increase MaxTotalConnections based on the load. For example, if it is 4000 increate it to 6000.
- Click Update Attributes.
- Restart the server using:
./APIGateway -f gateway-props.json -a stop ./APIGateway -fgateway-props.json -a start
- Log into the Administration Server console:
- If out of memory, increase the JVM heap size of the Gateway managed server:
- Stop the Gateway servers
./APIGateway -f gateway-props.json -a stop
- The JVM heap size is set in the
setDomainEnv.sh
file which is under<Gateway Install location>/domain/gateway1/bin
.The default values for the Gateway WebLogic server are:- Minimum heap size (
Xms
): 2G - Maximum heap size (
Xmx
): 4G
If the application needs more than 4G heap memory, search for
Xmx
insetDomainEnv.sh
and increase the value. - Minimum heap size (
- Start the gateway servers:
./APIGateway -f gateway-props.json -a start
- Stop the Gateway servers
Create user in the Gateway realm to use Basic Auth
If you have an API with a basic authentication policy and try to call it, you may get a response of 401 Unauthorized. To resolve this, create a user and add them to a group in the Gateway WebLogic security realm. See Create a group and add the user to it for Basic Auth.
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- Click Security Realms.
- In the Summary of Security Realms page, under Realms, click myrealm.
- In the Settings for myrealm page, click User and Groups and then click the Users tab.
- In the Users page, click New.
- In the Create a New User page, enter the user name, a description, the password, and confirm the password. For the provider, choose the default
DefaultAuthenticator
. - Click OK.
Specify the new user in the policy. See Apply Basic Authentication Policies.
Create a group and add the user to it for Basic Auth
If you have an API with a basic authentication policy and try to call it, you may get a response of 401 Unauthorized. To resolve this, add a user to a group in the Gateway WebLogic security realm. See Create user in the Gateway realm to use Basic Auth.
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- Click Security Realms.
- In the Summary of Security Realms page, under Realms, click myrealm.
- In the Settings for myrealm page, click User and Groups and then click the Groups tab.
- In the Groups page, click New.
- In the Create a New Group page, enter the group name and a description. For the provider, choose the default
DefaultAuthenticator
.Specify the group name in the policy. See Apply Basic Authentication Policies.
- Click OK.
- In the Users page, select the user name that you want to add to the group.
Do not only select the check box, click the name as well.
- On the Settings for <user_name> page, click the Groups tab, then select the group in the Available column and click the right arrow to move the group to the Chosen column.
- Click Save.
- Use the breadcrumbs at the top of the console to go back to the Users table, and repeat the same process for other users.
Change the Gateway WebLogic admin and Derby DB credentials
The API Platform Gateway WebLogic admin password and the Derby DB password have to be the same otherwise the server will not restart. So if you change either password, you must change the other.
- To change the WebLogic admin password:
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- Click Security Realms.
- In the Summary of Security Realms page, under Realms, click myrealm.
- In the Settings for myrealm page, click User and Groups and then click the Groups tab.
- Click on the Weblogic admin user and then Passwords.
- Enter the new password and confirm it, and click Save.
- Log into the Administration Server console:
- To change the Derby DB password:
- SSH to the Gateway host machine.
- Confirm that the DB is running in Gateway host machine:
ps -eaf | grep NetworkServerControl
- Go to the Derby DB home lib folder:
$ cd <GATEWAY_INSTALL_LOC>/GATEWAY_HOME/wlserver/common/derby/lib
- Run the
ij
utility:$ java -jar derbyrun.jar ij
- Once ij is up, run the connect command (using the appropriate hostname/ip, port and password):
ij> CONNECT 'jdbc:derby://<HOSTNAME_OR_IP>:<PORT>/gatekeeper;user=gatekeeper;password=<WLS_ADMIN_PASSWORD>';
For example:CONNECT 'jdbc:derby://slxxxxx.us.oracle.com:1527/gatekeeper;user=gatekeeper;password=welcome1';
Note:
- The hostname/ip is the value of
listenIpAddress
ingateway-props.json
. - The DB User name is
gatekeeper
. - The password for the DB user is same as the WLS admin password.
- The hostname/ip is the value of
- Once connected, update the password of the gatekeeper user.
ij> CALL SYSCS_UTIL.SYSCS_SET_DATABASE_PROPERTY('derby.user.gatekeeper', '<WLS_NEW_ADMIN_PASSWORD>');
For example:
CALL SYSCS_UTIL.SYSCS_SET_DATABASE_PROPERTY('derby.user.gatekeeper', 'welcome123');
- Disconnect and then verify the password by trying to connect again:
ij> disconnect; ij> CONNECT 'jdbc:derby://<HOSTNAME_OR_IP>:<PORT>/gatekeeper;user=gatekeeper;password=<WLS_NEW_ADMIN_PASSWORD>';
For example:
ij> CONNECT 'jdbc:derby://slxxxxx.us.oracle.com:1527/gatekeeper;user=gatekeeper;password=welcome123';
If the password reset has been successful, then the connection will be made.
- Change the JDB data source password:
- Login again to the WebLogic admin console.
- Click Lock & Edit.
- Go to Services, then Data Sources and open the Configuration tab.
- From the list of data sources, click
wlng.datasource
and open the Connection Pool tab. - Enter the new password and confirm it.
- From the list of data sources, click
wlng.localTX.datasource
and open the Connection Pool tab. - Enter the new password for this data source and confirm it.
- Click Activate Changes.
- Verify the changes by restarting the Gateway servers:
- Restart from the WebLogic console, or start the Gateway servers from the Host machine:
- SSH to Gateway host machine.
- Set JAVA_HOME
- Stop the Gateway servers and the Derby DB:
/APIGateway -f gateway-props.json -a stop
- Or
pgrep java | xargs kill -9
- Start the Gateway servers and the Derby DB:
/APIGateway -f gateway-props.json -a start
- Restart from the WebLogic console, or start the Gateway servers from the Host machine:
Cannot connect to Gateway Derby DB
Check the Gateway Derby DB connection
- To check whether the DB is running in Gateway host machine:
ps -eaf | grep NetworkServerControl
- Go to the Derby DB home lib folder:
$ cd <GATEWAY_INSTALL_LOC>/GATEWAY_HOME/wlserver/common/derby/lib
- Run the
ij
utility:$ java -jar derbyrun.jar ij
- Once ij is up, run the connect command (using the appropriate hostname/ip, port and password):
ij> CONNECT 'jdbc:derby://<HOSTNAME_OR_IP>:<PORT>/gatekeeper;user=gatekeeper;password=<WLS_ADMIN_PASSWORD>';
For example:CONNECT 'jdbc:derby://slxxxxx.example.com:1527/gatekeeper;user=gatekeeper;password=welcome1';
Note:
- The hostname/ip is the value of
listenIpAddress
ingateway-props.json
. - The DB User name is
gatekeeper
. - The password for the DB user is same as the WLS admin password.
- The hostname/ip is the value of
- If the connection is not successful, check the parameters. Also, check the Derby DB log at:
<GATEWAY_INSTALL_LOC>/GATEWAY_HOME/wlserver/common/derby/derby.log
- If the connection is successful, run a basic command and then disconnect:
ij> VALUES CURRENT_TIMESTAMP; ij> disconnect; ij> exit;
Root causes
- The DB password you specified is wrong.
- Solution: Make sure the correct password (WebLogic admin password) is used.
- The DB password uses unsupported special characters, such as
$
.- Solution: Re-install the gateway using a password string which doesn't use unsupported special characters.
- The DB host machine, which is the
listenIpAddress
specified ingateway-props.json
, is not reachable.- Solution: Use the correct private/public IP or hostname of the gateway node machine in
gateway-props.json
, and re-install.
- Solution: Use the correct private/public IP or hostname of the gateway node machine in
- The DB port is already in use.
- Solution: Use a different DB port in
gateway-props.json
, and re-install.
- Solution: Use a different DB port in
- Derby is failing to start or processes getting killed.
- Solution: Derby DB is corrupted because of a forced stop. To fix it, restart the host machine and run the start operation.
Note:
- The DB connect
hostname/ip
is value oflistenIpAddress
ingateway-props.json
. - The DB User name is
gatekeeper
. - The password for DB user is same as the WebLogic admin password.
Allow enabling cookies to be passed to the backend service
A flag has to be set for the Gateway to pass cookies to the backend server.
- Upgrade Java to use the latest JDK.
- Increase MaxTotalConnections:
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- Click OSCG.
- Click the name of your managed server, then click Container Services.
- Click DafGeneralInformation.
- Set
enableSouthCookie
totrue
. - Click Update Attributes.
- Restart the server using:
./APIGateway -f gateway-props.json -a stop ./APIGateway -fgateway-props.json -a start
- Log into the Administration Server console:
Update Gateway threat protection configurations
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- Click OSCG.
- Click the name of your managed server, then click Reporter.
- To see all current threat protections configurations, go to the Attributes tab
- To get the full JSON configuration for a specific threat definition:
- Go to the Operations tab, select
getFullThreatConfiguration
and enter name of the threat protection. - Click Invoke.
An example of the JSON file of a threat configuration:
{ "name": "ApiFwFail", "actions": [ "ALARM", "BLOCK" ], "trackedEntities": [ "IP", "APP_INST_ACCOUNT" ], "maxTrackedEntities": 10000, "maxViolations": 10, "violationClearTime": 1, "description": "Generic API-Firewall Threat. Repeated API FW violations could be penetration tests, an alarm is sent when this happens, and further traffic is blocked" }
- Go to the Operations tab, select
- To update a threat attribute, on the Operations tab select
updateThreatConfiguration
and underJson:
enter the updated threat attribute and click Invoke.For example, to update maxViolations of ApiFwFailto 20:
{ "name": "ApiFwFail", "actions": [ "ALARM", "BLOCK" ], "trackedEntities": [ "IP", "APP_INST_ACCOUNT" ], "maxTrackedEntities": 10000, "maxViolations": 20, "violationClearTime": 1, "description": "Generic API-Firewall Threat. Repeated API FW violations could be penetration tests, an alarm is sent when this happens, and further traffic is blocked" }
- To mute all threats protections, to to the Operations tab, select select
updateThreatConfiguration
and underJson:
enter:{ "actions":[ ] }
Show threat protection alarm description in logs
- Log into the Administration Server console:
http://<GW HOST IP>:<gatewayAdminServerPort>/console
- Click OSCG.
- Click the name of your managed server, then click Reporter and go to the Operations tab.
- Click
updateThreatConfiguration
. - To update a threat attribute, on the Operations tab select
updateThreatConfiguration
and underJson:
enter the updated threat attribute and click Invoke.For example, to add a description to the
AppKeyLoginFail
threat:{ "name":"AppKeyLoginFail", "actions":[ "ALARM", "BLOCK" ], "trackedEntities":[ "IP" ], "maxTrackedEntities":10000, "maxViolations":3, "violationClearTime":30, "description":"Login-Fail Threat. Triggered for wrong x-app-key values, can only be armed for IP. Use this to prevent brute force appkey guessing" }
WebLogic vulnerability attack, 503 Service Unavailable, Gateway start getting killed
Symptoms
- An intermittent HTTP/1.1 503 Service Unavailable error while making called to deployed APIs.
- In the default.logs, you'll notice
OVERLOAD PROTECTION
has kicked in frequently and traffic is getting throttled. - Gateway start keeps on getting killed.
Diagnosis
- Run the top command to identify CPU and memory usage.
- Check if there is an unknown process hogging CPU more than 100%, for example
/tmp/javax/sshd2
. - Note the process ID.
- Check the process working directory
pwdx <process id>
. - Get tids:
$ ps -eLo pid,ppid,tid,pcpu,comm | grep <pid>
- Get thread dump of the process:
$ kill -3 <pid> >> threaddumb.out
or
$ jstack -l <pid> >> threaddump.out
If getting thread dumps fails, it's a sign that the process is a malware.
Quick solution
- Stop the Gateway:
./APIGateway -f gateway-props.json -a stop
- Kill the CPU hogging process:
sudo kill -9 <pid>
- Delete the CPU hogging executable and its mother directory. Also, delete any file from the location found from running
pwdx <pid>
, for example:rm -rf /tmp/javax/sshd2 rm -rf /tmp/javax
- Clear the tmp directory:
sudo rm -rf /tmp/*
- Reboot the system.
- Start the Gateway:
./APIGateway -f gateway-props.json -a start
Full solution
For a long term solution, you should:
- Decommission the Gateway node.
- Use a new Host.
- Download the latest installer.
- Install, configure, start a new Gateway node using the latest installer.
- Join to the specific logical Gateway.
Note:
- During installation, use use strong credentials for the Weblogic admin.
- Do not expose the WebLogic admin console to the internet, unless it is required.