For server SSL/TLS, a server key pair needs to be generated. If the server is going to do client authentication, the client sertificates need to be installed in the trust store. These operations are described in Key and certificate handling.
SSL at the server side is activated with the property derby.drda.sslMode (default off) or the -ssl option for the server start command.
When the SSL mode is set to basic, the server will only accept SSL encrypted connections.
The properties javax.net.ssl.keyStore and javax.net.ssl.keyStorePassword need to be set with the proper values.
java -Djavax.net.ssl.keyStore=serverKeyStore.key \ -Djavax.net.ssl.keyStorePassword=qwerty \ -jar derbyrun.jar server start -ssl basic
When the server's SSL mode is set to peerAuthentication, then the server authenticates its clients' identity in addition to encrypting network traffic. In this situation, the server's trust store must contain a certificate for each client which will connect.
The javax.net.ssl.trustStore and javax.net.ssl.trustStorePassword need to be set in addition to the properties above.
See Running the client with SSL/TLS for client settings when the server does client authentication.
java -Djavax.net.ssl.keyStore=serverKeyStore.key \ -Djavax.net.ssl.keyStorePassword=qwerty \ -Djavax.net.ssl.trustStore=serverTrustStore.key \ -Djavax.net.ssl.trustStorePassword=qwerty \ -jar derbyrun.jar server start -ssl peerAuthentication