Oracle Cloud Infrastructure Documentation

Creating a Model Deployment

After you store a Data Science model in the model catalog, it can be deployed as an HTTP endpoint as a model deployment.

Considerations

Consider using a custom container when creating a model deployment.

Steps

The steps on this page describe common options available when creating a model deployment.

To create specific types of model deployments, see the following pages.

  • Default networking: Use service-managed networking with or without internet access.
    • With internet access: Connect to both other OCI services and the internet.
    • Without internet access: Connect to other OCI services.
  • Custom networking: Configure networking in your tenancy, fully controlling the VCN, subnets, routing, and access policies.
  • Compute target: Use the processing power and memory provided by a compute target to run machine learning workloads.
    1. On the Projects list page, select the project that contains the model deployments that you want to work with. If you need help finding the list page or the project, see Listing Projects.
    2. On the project details page, select Model deployments.
    3. Select Create model deployment.
    4. On the Create model deployment page, enter the following information.
      • Compartment
      • Name (Optional): Enter a unique name for the model deployment (limit of 255 characters). If you don't provide a name, a name is automatically generated. Example: modeldeployment20200108222435
      • Description (Optional): Enter a description (limit of 400 characters) for the model deployment.
      • Default configuration (Optional): Enter one or more custom environment variables with their key values.
        • Custom environment variable key (Optional): Enter a custom environment variable key.
        • Value (Optional): Enter the value for the key.
      • Models: Select Select to open the Select models panel, select the relevant option, and then select Submit to close the panel.
        Important

        Model artifacts that exceed 400 GB aren't supported for deployment. Select a smaller model artifact for deployment.
        • Single Model: Find the model with either Select a model compartment (specify compartment and project) or Using OCID, then select the model from the list.
        • Model Groups: Specify compartment and project, then select the model group from the list.
      • Compute resources: Select Standard compute. To change the Compute shape, select Change shape and then follow these steps in the Select compute shape panel.
        Note

        For options related to Compute target, see Creating a Model Deployment with a Compute Target.
        • Select an instance type.
        • Select a shape series.
        • Select one of the supported Compute shapes in the series. Select the shape that best suits how you want to use the resource.
        • Expand the selected shape to configure OCPUs and memory.
          • Number of OCPUs
          • Amount of memory (GB): For each OCPU, select up to 64 GB of memory and a maximum total of 512 GB. The minimum amount of memory allowed is either 1 GB or a value matching the number of OCPUs, whichever is greater.
          • Enable Burstable Shape: Select if using burstable VMs, and then for Baseline utilization per OCPU, select the percentage of OCPUs that you usually want to use. The supported values are 12.5% and 50%. (For model deployments, only the value of 50% is supported.)
        • Select Select shape.
      • Autoscaling configuration (Optional): Select Enable autoscaling and enter the following information.
        Note

        The following options are related to the Standard compute infrastructure type. For options related to Compute target, see Creating a Model Deployment with a Compute Target.
        • Minimum number of instances
        • Maximum number of instances
        • Cooldown period in seconds
        • Scaling metric type

          To use the custom scaling metric option, select Custom, and then specify the scale-in and scale-out queries.

          Important

          Include the following text in each MQL query to reference the resource OCID: {resourceId = "MODEL_DEPLOYMENT_OCID"}

          Example query:

          PredictRequestCount[5m]{resourceId = "MODEL_DEPLOYMENT_OCID"}.count()

          Scaling to Zero

          To enable scale to zero, set the minimum instance count to 0.

          When using scale to zero, use a metric that's available while the deployment is scaled down, such as PredictRequestCount.

          When a model deployment is scaled to zero, request buffering can be enabled so incoming requests wait while the deployment scales up instead of failing immediately.

          • Up to 100 requests can be buffered per model deployment.
          • Each request waits up to 5 minutes for a pod to become available.
          • If capacity does not become available within 5 minutes, the request returns 503.

          To enable request buffering, configure ENABLE_REQUEST_BUFFERING=true.

          Timeout

          If you want the model deployment to wait longer for compute target scale-out before failing, configure DEPLOYMENT_SCHEDULING_TIMEOUT

          Guidance:

          • Supported range: 15 to 120
          • Recommended value with default compute target autoscaling settings: 30
          • Unit in Minutes
        • Scale-in threshold in percentage
        • Scale-out threshold in percentage
        • Advanced options (Optional): Autoscale the load balancer. Set the value of maximum bandwidth to be more than the minimum bandwidth value, and no more than twice the minimum bandwidth value.
          • Scale-in instance count step
          • Scale-out instance count step
          • Scale-in pending duration
          • Scale-out pending duration
      • Networking resources (available for Standard compute infrastructure type): Select the relevant option.
        • Default Networking: Restricts traffic to Oracle services only. The system uses the existing service-managed network. The workload is attached by using a secondary VNIC to a preconfigured, service-managed VCN, and subnet. This provided subnet lets egress to the public internet through a NAT gateway, and access to other Oracle Cloud services through a service gateway.

          If you need access only to the public internet and OCI services, we recommend using this option. It doesn't require you to create networking resources or write policies for networking permissions.

        • Default networking with internet: Allows outbound internet access through the Data Science NAT gateway.
          Note

          You can't use Default networking with internet in disconnected realms and Oracle development tenancies. If your tenancy or compartment has a Data Science security zone policy that denies public network access (for example, deny model_deploy_public_network—see Data Science security zone policy), the service-managed public internet access option is disabled. If you try to use this option, you receive a 404 NotAuthorizedOrNotFound error.
        • Custom Networking: Select the VCN and subnet (by compartment) that you want to use.

          For egress access to the public internet, use a private subnet with a route to a NAT gateway.

          Note

          • Custom networking must be used to use a file storage mount.
          • Switching from custom networking to managed networking isn't supported after creation.
          • If you see the banner The specified subnet is not accessible. Select a different subnet., then create policies for custom networking.
      • Endpoint type (available for Standard compute infrastructure type): Select the relevant option.
        • Public endpoint: Enables data access to a managed instance from outside a virtual cloud network (VCN).
        • Private endpoint: Specifies the private endpoint to use for the model deployment. Deployments that use private networking or private endpoints can't enable service-managed public internet access.
          • Private endpoint compartment
          • Private endpoint
      • Logging (Optional): Select Select to open the Select logging panel, enter the following information, and then select Submit to close the panel.
        Note

        Logging requires configured access or predict logging.
        • For access logs, select a compartment, log group, and log name.
        • For predict logs, select a compartment, log group, and log name.
      • Use a Custom Container Image
        • Set your BYOC environment (Optional): Select Select to open the Set your BYOC environment panel, enter the following information, and select Select again to close the panel.
          • Repository compartment
          • Repository: Enter the repository that contains the custom image.
          • Image: Enter the custom image to use at runtime.
          • Digest: Enter the image digest. For example: sha256:<digest>. The digest must match the exact image you're deploying.
          • Signature Id: Enter the image signature OCID that is generated when you sign the image. For more information about the signing process, see Signing Images for Security.

            The signature entry is optional, unless your tenancy or policy enforces image signature verification for custom images. If verification is enforced, you must provide all required signature fields. Otherwise, deployment will not proceed.

            If verification fails, the Console shows an error such as "Signature <ID> is not verified." or "Digest of the image provided does not match the digest on the signature." In this case, model deployment creation is blocked.

          • CMD: Enter commands for the container, one command per text box. For example, enter -\-host in one text-box and 0.0.0.0 in another. Do not use quotation marks at the end.
          • Entrypoint: Enter one or more entry point files to run when the container starts, such as /opt/script/entrypoint.sh. Don't use quotation marks at the end.
          • Server port: Enter the port for the inference web server. The default is 8080. Valid range: 1024–65535, except 24224, 8446, 8447.
          • Health check port: Enter the port for the container health check. The default is the server port. Valid range: 1024–65535, except 24224, 8446, 8447.
          • Predict API specification
      • Deployment mode (under Advanced options) (available for Standard compute infrastructure type) (Optional): Select the relevant option.
        • Real time inference (HTTPS): Select the load balancing bandwidth in Mbps or use the 10 Mbps default.

          Tips for load balancing:

          If you know the common payload size and the frequency of requests per second, you can use the following formula to estimate the bandwidth of the load balancer that you need. We recommend that you add an extra 20% to account for estimation errors and sporadic peak traffic.

          (Payload size in KB) * (Estimated requests per second) * 8 / 1024

          For example, if the payload is 1,024 KB and you estimate 120 requests per second, then the recommended load balancer bandwidth would be (1024 * 120 * 8 / 1024) * 1.2 = 1152 Mbps.

          Remember that the maximum supported payload size is 10 MB when dealing with image payloads.

          If the request payload size is more than the allocated bandwidth of the load balancer that was defined, then the request is rejected with a 429 status code.

        • Streaming mode: Select streams for input and output (by compartment).
      • Tags (under Advanced options) (Optional): Add tags to the model deployment. If you have permissions to create a resource, then you also have permissions to apply free-form tags to that resource. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you're not sure whether to apply tags, skip this option or ask an administrator. You can apply tags later.
    5. Select Create.

  • Use the oci data-science model-deployment create command and required parameters to create a model deployment:

    oci data-science model-deployment create --compartment-id <compartment-id> ... [OPTIONS]

    For a complete list of parameters and values for CLI commands, see the CLI Command Reference.

  • Use the CreateModelDeployment operation to create a model deployment.