Oracle Cloud Infrastructure Documentation

Creating a Model Deployment with a Compute Target

Create a model deployment in Data Science with a compute target.

You can create a model deployment that uses a compute target for its infrastructure. When you select a compute target, the model deployment runs on the managed compute capacity provided by that compute target instead of provisioning separate on-demand infrastructure for the deployment.

Use this option when you want supported model deployments to share reusable managed capacity, improve GPU utilization, or use scale-to-zero behavior for supported deployments. Model deployment using a compute target supports Service Managed Container only.

Before You Begin

Before creating a model deployment with a compute target, do the following.

  • Create a compute target in the same region.
  • Confirm that the compute target has enough available capacity for the deployment.
  • Confirm that the model deployment uses a service-managed container flow.
  • Confirm that you have permissions to manage model deployments and use the required model artifacts. See Model Deployment Policies.

Invoking the Predict Endpoint

Model deployments that use compute targets support the /predict endpoint for request-response inference.

The caller must have permission to invoke the model deployment predict endpoint:

allow group <group-name> to use data-science-model-deployments in compartment <compartment-name>
    1. On the Projects list page, select the project that contains the model deployments that you want to work with. If you need help finding the list page or the project, see Listing Projects.
    2. On the project details page, select Model deployments.
    3. Select Create model deployment.
    4. On the Create model deployment page, enter the following information.
      • Compartment
      • Name (Optional): Enter a unique name for the model deployment (limit of 255 characters). If you don't provide a name, a name is automatically generated. Example: modeldeployment20200108222435
      • Description (Optional): Enter a description (limit of 400 characters) for the model deployment.
      • Default configuration (Optional): Enter one or more custom environment variables with their key values.
        • Custom environment variable key (Optional): Enter a custom environment variable key.
        • Value (Optional): Enter the value for the key.
      • Models: Select the model for the model deployment.
        1. Select Select.
        2. In the Select models panel, select Single Model.
        3. Find the model with either Select a model compartment (specify compartment and project) or Using OCID, then select the model from the list.
        4. Select Submit.
      • Compute resources: Select Compute target and then enter the following information.
        • Compartment: Select the compartment that contains the compute target.
        • Compute target: Select the compute target.
        • Routing policy: Select Least Request or Round Robin.
        • Number of instances
        • Number of GPUs
        • Min number of OCPUs
        • Max number of OCPUs
        • Min amount of memory (GB)
        • Max amount of memory (GB)
      • Autoscaling configuration (Optional): Select Enable autoscaling and then enter the following information. Model deployments that use compute targets support autoscaling. Autoscaling can adjust the number of model deployment replicas based on supported metrics.
        • Minimum number of instances
        • Maximum number of instances
        • Scaling metric type

          To use the custom scaling metric option, select Custom, and then specify the scale-in and scale-out queries.

          Important

          Include the following text in each MQL query to reference the resource OCID: {resourceId = "MODEL_DEPLOYMENT_OCID"}

          Example query:

          PredictRequestCount[5m]{resourceId = "MODEL_DEPLOYMENT_OCID"}.count()

          Scaling to Zero

          To enable scale to zero, set the minimum instance count to 0.

          When using scale to zero, use a metric that's available while the deployment is scaled down, such as PredictRequestCount.

          When a model deployment is scaled to zero, request buffering can be enabled so incoming requests wait while the deployment scales up instead of failing immediately.

          • Up to 100 requests can be buffered per model deployment.
          • Each request waits up to 5 minutes for a pod to become available.
          • If capacity does not become available within 5 minutes, the request returns 503.

          To enable request buffering, configure ENABLE_REQUEST_BUFFERING=true.

          Timeout

          If you want the model deployment to wait longer for compute target scale-out before failing, configure DEPLOYMENT_SCHEDULING_TIMEOUT

          Guidance:

          • Supported range: 15 to 120
          • Recommended value with default compute target autoscaling settings: 30
          • Unit in Minutes
        • Scale metric threshold
        • Metric namespace (Custom scaling metric type)
        • Advanced options (Optional): Autoscale the load balancer. Set the value of maximum bandwidth to be more than the minimum bandwidth value, and no more than twice the minimum bandwidth value.
          • Scale-in instance count step
          • Scale-out instance count step
          • Scale-in pending duration
          • Scale-out pending duration
          • Scale-in cooldown period in seconds
          • Scale-out cooldown period in seconds
      • Networking resources
        • Networking type: Select the relevant option.
          • Default Networking: Restricts traffic to Oracle services only. The system uses the existing service-managed network. The workload is attached by using a secondary VNIC to a preconfigured, service-managed VCN, and subnet. This provided subnet lets egress to the public internet through a NAT gateway, and access to other Oracle Cloud services through a service gateway.

            If you need access only to the public internet and OCI services, we recommend using this option. It doesn't require you to create networking resources or write policies for networking permissions.

          • Custom Networking: Select the VCN and subnet (by compartment) that you want to use.

            For egress access to the public internet, use a private subnet with a route to a NAT gateway.

            Note

            • Switching from custom networking to managed networking isn't supported after creation.
            • If you see the banner The specified subnet is not accessible. Select a different subnet., then create policies for custom networking.
        • Endpoint type: Select public or private.
      • Logging (Optional): Select Select to open the Select logging panel, enter the following information, and then select Submit to close the panel.
        Note

        Logging requires configured access or predict logging.
        • For access logs, select a compartment, log group, and log name.
        • For predict logs, select a compartment, log group, and log name.
      • Use a Custom Container Image
        • Set your BYOC environment (Optional): Select Select to open the Set your BYOC environment panel, enter the following information, and select Select again to close the panel.
          • Repository compartment
          • Repository: Enter the repository that contains the custom image.
          • Image: Enter the custom image to use at runtime.
          • Digest: Enter the image digest. For example: sha256:<digest>. The digest must match the exact image you're deploying.
          • Signature Id: Enter the image signature OCID that is generated when you sign the image. For more information about the signing process, see Signing Images for Security.

            The signature entry is optional, unless your tenancy or policy enforces image signature verification for custom images. If verification is enforced, you must provide all required signature fields. Otherwise, deployment will not proceed.

            If verification fails, the Console shows an error such as "Signature <ID> is not verified." or "Digest of the image provided does not match the digest on the signature." In this case, model deployment creation is blocked.

          • CMD: Enter commands for the container, one command per text box. For example, enter -\-host in one text-box and 0.0.0.0 in another. Do not use quotation marks at the end.
          • Entrypoint: Enter one or more entry point files to run when the container starts, such as /opt/script/entrypoint.sh. Don't use quotation marks at the end.
          • Server port: Enter the port for the inference web server. The default is 8080. Valid range: 1024–65535, except 24224, 8446, 8447.
          • Health check port: Enter the port for the container health check. The default is the server port. Valid range: 1024–65535, except 24224, 8446, 8447.
          • Predict API specification
      • Advanced options
        • Deployment mode (under Advanced options) (available for Standard compute infrastructure type) (Optional): Select the relevant option.
          • Real time inference (HTTPS): Select the load balancing bandwidth in Mbps or use the 10 Mbps default.

            Tips for load balancing:

            If you know the common payload size and the frequency of requests per second, you can use the following formula to estimate the bandwidth of the load balancer that you need. We recommend that you add an extra 20% to account for estimation errors and sporadic peak traffic.

            (Payload size in KB) * (Estimated requests per second) * 8 / 1024

            For example, if the payload is 1,024 KB and you estimate 120 requests per second, then the recommended load balancer bandwidth would be (1024 * 120 * 8 / 1024) * 1.2 = 1152 Mbps.

            Remember that the maximum supported payload size is 10 MB when dealing with image payloads.

            If the request payload size is more than the allocated bandwidth of the load balancer that was defined, then the request is rejected with a 429 status code.

          • Streaming mode: Select streams for input and output (by compartment).
        • Tags (Optional): Add tags to the model deployment. If you have permissions to create a resource, then you also have permissions to apply free-form tags to that resource. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you're not sure whether to apply tags, skip this option or ask an administrator. You can apply tags later.
    5. Select Create.

  • Use the oci data-science model-deployment create command and required parameters to create a model deployment:

    oci data-science model-deployment create --compartment-id <compartment-id> ... [OPTIONS]

    For a complete list of parameters and values for CLI commands, see the CLI Command Reference.

  • Use the CreateModelDeployment operation to create a model deployment.