Using Pretrained Models

Supported Languages

• Afrikaans

• Albanian

• Arabic

• Armenian

• Azerbaijani

• Basque

• Belarusian

• Bengali

• Bosnian

• Brazilian Portuguese

• Bulgarian

• Burmese

• Cantonese

• Catalan

• Cebuano

• Chinese

• Croatian

• Czech

• Danish

• Dutch

• Eastern Punjabi

• Egyptian Arabic

• English

• Esperanto

• Estonian

• Finnish

• French

• Georgian

• German

• Greek

• Hebrew

• Hindi

• Hungarian

• Icelandic

• Indonesian

• Irish

• Italian

• Japanese

• Javanese

• Kannada

• Kazakh

• Korean

• Kurdish (Sorani)

• Latin

• Latvian

• Lithuanian

• Macedonian

• Malay

• Malayalam

• Marathi

• Minangkabau

• Nepali

• Norwegian (Bokmal)

• Norwegian (Nynorsk)

• Persian

• Polish

• Portuguese

• Romanian

• Russian

• Serbian

• Serbo-Croatian

• Slovak

• Slovene

• Spanish

• Swahili

• Swedish

• Tagalog

• Tamil

• Telugu

• Thai

• Turkish

• Ukrainian

• Urdu

• Uzbek

• Vietnamese

• Welsh

Examples

The JSON for the first example is:

Sample Request

POST https://<region-url>/20210101/actions/batchDetectDominantLanguage

API Request format:

{
  "documents": [
    {
      "key": "doc1",
      "text": "OCI recently added new services to existing compliance program 
including SOC, HIPAA, and ISO to enable our customers to solve their use 
cases. We also released new white papers and guidance documents related to 
Object Storage, the Australian Prudential Regulation Authority (APRA), and the 
Central Bank of Brazil. These resources help regulated customers better 
understand how OCI supports their regional and industry-specific compliance 
requirements. Not only are we expanding our number of compliance offerings and 
regulatory alignments, we continue to add regions and services at a faster clip."
    }
  ]
}

Response JSON:

{
    "documents": [
        {
            "key": "1",
            "languages": [
                {
                    "code": "en",
                    "name": "English",
                    "score": 0.9999840921009815
                }
            ]
        }
    ],
    "errors": []
}}

Limitations

• Only one language is returned. In cases where the input is multi-lingual, the dominant language is returned.

Supported Features

• Text category

• Confidence scores

• Requests support single record and multi-record batches.

Supported Languages for Input Text

Examples

Sports and Games/Motor Sports 1.0000

The JSON for the first example is:

Sample Request

POST https://<region-url>/20210101/actions/batchDetectLanguageTextClassification

API Request format:

{
    "documents": [
        {
            "key": "1",
            "textClassification": [
                {
                    "label": "Sports and Games/Motor Sports",
                    "score": 1
                }
            ],
            "languageCode": "en"
        }
    ],
    "errors": []
}

Response JSON:

{
    "documents": [
        {
            "key": "1",
            "textClassification": [
                {
                    "label": "Sports and Games/Motor Sports",
                    "score": 1
                }
            ],
            "languageCode": "en"
        }
    ],
    "errors": []
}

Supported Categories

Limitations

• If your text is about more than one category, the major category of the text is identified and could differ from human interpretation of the text.

Supported Languages for Input Text

Use Cases

You could use the NER endpoint effectively in these scenarios:

Classifying content for news providers

It can be difficult to classify and categorize news article content. The NER model can automatically scan articles to identify the major people, organizations, and places in them. The extracted entities can be saved as tags with the related articles. Knowing the relevant tags for each article helps with automatically categorizing the articles in defined hierarchies, and enables content discovery.

Customer support

Recognizing relevant entities in customer complaints and feedback, product specifications, department details, or company branch details, helps to classify the feedback appropriately. The entities can then be forwarded to the person responsible for the identified product.

Similarly, there could be feedback tweets where you can categorize them all based on their locations, and the products mentioned.

Efficient search algorithms

You could use NER to extract entities that are then searched against the query, instead of searching for a query across the millions of articles and websites online. When run on articles, all the relevant entities associated with each article are extracted and stored separately. This separation could speed up the search process considerably. The search term is only matched with a small list of entities in each article, leading to quick and efficient searches.

It can be used for searching content from millions of research papers, Wikipedia articles, blogs, articles, and so on.

Content recommendations

Extracting entities from a particular article, and recommending the other articles that have the most similar entities mentioned in them is possible with NER. For example, it can be used effectively to develop content recommendations for a media industry client. It enables the extraction of the entities associated with historical content or previous activities. NER compares them with the label assigned to other unseen content to filter relevant entities.

Automatically summarizing job candidates

The NER model could facilitate the evaluation of job candidates, by simplifying the effort required to shortlist candidates with numerous applications. Recruiters could filter and categorize them based on identified entities like location, college degrees, employers, skills, designations, certifications, and patents.

Supported Entities

The following table describes the different entities that NER can extract. The entity type and subtype depends on the API that you call (detectDominantLanguageEntities or batchDetectDominantLanguageEntities).

NATIONALITIES, 
RELIGIOUS and 
POLITICAL GROUPS

("GB") - United Kingdom

("AU") - Australia 

("NZ") - New Zealand 

("SG") - Singapore 

("IN") - India

("US")  - United States

Examples

Red Bull Racing Honda [ORG] 1.0000
four-time [QUANTITY/NUMBER] 1.0000
Formula-1 World [EVENT] 0.9705
Oracle Cloud Infrastructure (OCI [ORG] 0.9811

OCI [ORG] 1.0000
SOC [ORG] 1.0000
HIPAA [ORG] 1.0000
ISO [ORG] 1.0000
Australian Prudential Regulation Authority [ORG] 1.0000
Central Bank of Brazil [ORG] 0.9998
OCI [ORG] 1.0000

The JSON for the first example is:

Sample Request

POST https://<region-url>/20210101/actions/batchDetectLanguageEntities

API Request format:

"{
    "documents": [
       

{             "key": "doc1",             "text": " Red Bull Racing Honda, the four-time Formula-1 World Champion team, has chosen Oracle Cloud Infrastructure (OCI) as their infrastructure partner."         }
    ]
}"

Response JSON:

{
    "documents": [
        {
            "key": "1",
            "entities": [
                {
                    "offset": 0,
                    "length": 15,
                    "text": "Red Bull Racing",
                    "type": "ORGANIZATION",
                    "subType": null,
                    "score": 0.9914557933807373,
                    "metaInfo": null
                },
                {
                    "offset": 16,
                    "length": 5,
                    "text": "Honda",
                    "type": "ORGANIZATION",
                    "subType": null,
                    "score": 0.6515499353408813,
                    "metaInfo": null
                },
                {
                    "offset": 27,
                    "length": 9,
                    "text": "four-time",
                    "type": "QUANTITY",
                    "subType": null,
                    "score": 0.9998091459274292,
                    "metaInfo": [
                        {
                            "offset": 27,
                            "length": 9,
                            "text": "four-time",
                            "subType": "UNIT",
                            "score": 0.9998091459274292
                        }
                    ]
                },
                {
                    "offset": 47,
                    "length": 5,
                    "text": "World",
                    "type": "LOCATION",
                    "subType": "NON_GPE",
                    "score": 0.5825434327125549,
                    "metaInfo": null
                },
                {
                    "offset": 79,
                    "length": 27,
                    "text": "Oracle Cloud Infrastructure",
                    "type": "ORGANIZATION",
                    "subType": null,
                    "score": 0.998045802116394,
                    "metaInfo": null
                },
                {
                    "offset": 108,
                    "length": 3,
                    "text": "OCI",
                    "type": "ORGANIZATION",
                    "subType": null,
                    "score": 0.9986366033554077,
                    "metaInfo": null
                }
            ],
            "languageCode": "en"
        }
    ],
    "errors": []
}

Limitations

• Sometimes, entities might not be separated or combined as you expect.

• NER uses the context of the sentence to identify entities. If the context is not present in the text processed, entities might not be extracted as you expect.

• Malformed text (structure and semantics) might reduce the performance.

• Age isn't a separate entity so age-related periods might be identified as a date entity.

Use Cases

Some business use cases are:

• Brand monitoring

• Monitoring market research

• Competitive market analysis

• Customer support tickets

• Employee feedback analysis

• Customer reviews

• Email analysis

Supported Features

• Key phrases

• Confidence scores

• Requests support single record and multi-record batches.

Supported Languages for Input Text

Examples

Red Bull Racing Honda 0.9997
Oracle Cloud Infrastructure 0.9583
infrastructure partner 0.9583
oci 0.9979

OCI 0.9999
new services 0.9998
existing compliance program 0.9998
including SOC 0.9998
use cases 0.9998
new white papers 0.9998
guidance documents 0.9998
Object Storage 0.9998
Australian Prudential Regulation Authority 0.9998
Central Bank of Brazil 0.9998
regulated customers 0.9998
industry-specific compliance requirements 0.9998
number of compliance offerings 0.9998
regulatory alignments 0.9998
faster rate 0.9998
ISO 0.9992
customers 0.9992
apra 0.9992
resources 0.9992
services 0.8186
HIPPA 0.9979
regions 0.9147

The JSON for the first example is:

Sample Request

POST https://<region-url>/20210101/actions/batchDetectLanguageKeyPhrases

API Request format:

{
  "documents": [
    {
      "key": "doc1",
      "text": "Red Bull Racing Honda, the four-time Formula-1 World Champion team, has chosen Oracle Cloud Infrastructure (OCI) as their infrastructure partner."
    }
  ]
}

Response JSON:

{
    "documents": [
        {
            "key": "1",
            "keyPhrases": [
                {
                    "text": "red bull racing honda",
                    "score": 0.9997546563973576
                },
                {
                    "text": "oracle cloud infrastructure",
                    "score": 0.9997546563973576
                },
                {
                    "text": "infrastructure partner",
                    "score": 0.9997546563973576
                },
                {
                    "text": "oci",
                    "score": 0.9979336625058923
                }
            ],
            "languageCode": "en"
        }
    ],
    "errors": []
}

Limitations

• Key phrases that are noun phrases with adjective modifiers are identified so words that don't follow this criteria could be ignored.

• This model is case insensitive.

• Text that contains multiple punctuation between words might be flagged as a key phrase.

• URLs that are well formed (begin with http, https, or www) are identified.

Aspect-Based Sentiment Analysis

Aspect-Based Sentiment Analysis (ABSA) extracts the individual aspects in the input document and classifies each of the aspects into one of the polarity classes: positive, negative, mixed, and neutral. With the predicted sentiment for each aspect, the Language API also provides a confidence score for each of the classes, and their corresponding offsets in the input.

Confidence scores closer to 1 indicate a higher confidence in the label's classification, while lower scores indicate lower confidence score. The range of the confidence score for each class is between 0–1, and the cumulative scores of all the four classes sum to 1.

For example, a restaurant review that says "Food is marginal, but the service is so bad.” contains positive sentiment toward the food aspect. Also, it has a strong negative sentiment toward the service aspect. Classifying the overall sentiment as negative would neglect the fact that food was good. ABSA addresses this problem by referring to an aspect as an attribute (or component) of an entity. Also, the screen of a phone or the picture quality of a camera.

If the input data is "I had a good day at work today", then a single aspect day is identified with 100% positive, 0% neutral, 0% mixed, and 0% negative sentiments.

Sentence Level Sentiment Analysis

The Language service also provides sentence level sentiment with confidence scores for each sentence in the text. Based on the use case, you can select either sentence or document sentiment, or ABSA, or both. For example, in a customer feedback analytics scenario, you may want to identify sentences that need human review for further action.

Use Cases

Some business use cases are:

• Brand monitoring

• Monitoring market research

• Employee feedback analysis

• Customer reviews and emails analysis

• Product surveys

For example, customer and employee raw survey responses can be processed using the sentiment analysis model. The results can then be aggregated for analysis and follow up, and to facilitate engagements.

Social media monitoring can be employed with sentiment analysis to specifically extract the overall mood swing of the customer. For example, when a new product is launched, or competitive market research is conducted.

Supported Features

• Analysis level: sentence and aspect

• English language

• Requests support single record and multi-record batches.

Supported Languages for Input Text

Aspect-Based Sentiment Analysis Example

1. OCI recently added new services to the existing compliance program including SOC, HIPAA, and ISO, to enable 
our customers to solve their use cases. NEUTRAL 0.5512
2. We also released new white papers and guidance documents related to Object Storage, the Australian Prudential 
Regulation Authority (APRA), and the Central Bank of Brazil. NEUTRAL 0.5512
3. These resources help regulated customers better understand how OCI supports their regional and industry-
specific compliance requirements. POSITIVE 0.6763
4. Not only are we expanding our number of compliance offerings and regulatory alignments, we continue to add 
regions and services at a faster rate. POSITIVE 0.4658

Sample Request:

API Request format:

POST https://<region-url>/20210101/actions/batchDetectLanguageSentiments?level=ASPECT 

Input JSON

{
“documents”: [
    {
        "key" : "doc1",
        "text" : "OCI recently added new services to existing compliance 
program including SOC, HIPAA, and ISO to enable our customers to solve their 
use cases. We also released new white papers and guidance documents related to 
Object Storage, the Australian Prudential Regulation Authority (APRA), and the 
Central Bank of Brazil. These resources help regulated customers better 
understand how OCI supports their regional and industry-specific compliance 
requirements. Not only are we expanding our number of compliance offerings and 
regulatory alignments, we continue to add regions and services at a faster clip."
    }
]
}

Response JSON:

{
    "documents": [
        {
            "key": "1",
            "documentSentiment": "Positive",
            "documentScores": {
                "Neutral": 0.44763687,
                "Positive": 0.46578798,
                "Mixed": 0.064058214,
                "Negative": 0.022516921
            },
            "sentences": [
                {
                    "offset": 0,
                    "length": 147,
                    "text": "OCI recently added new services to the existing compliance program including SOC, HIPAA, and ISO, to enable our customers to solve their use cases.",
                    "sentiment": "Neutral",
                    "scores": {
                        "Negative": 0.0154264,
                        "Mixed": 0,
                        "Neutral": 0.98231775,
                        "Positive": 0.0022558598
                    }
                },
                {
                    "offset": 148,
                    "length": 170,
                    "text": "We also released new white papers and guidance documents related to Object Storage, the Australian Prudential Regulation Authority (APRA), and the Central Bank of Brazil.",
                    "sentiment": "Neutral",
                    "scores": {
                        "Mixed": 0,
                        "Neutral": 0.97296304,
                        "Negative": 0.007886417,
                        "Positive": 0.019150572
                    }
                },
                {
                    "offset": 319,
                    "length": 137,
                    "text": "These resources help regulated customers better understand how OCI supports their regional and industry-specific compliance requirements.",
                    "sentiment": "Neutral",
                    "scores": {
                        "Neutral": 0.5864549,
                        "Positive": 0.35583654,
                        "Mixed": 0.02932497,
                        "Negative": 0.028383587
                    }
                },
                {
                    "offset": 457,
                    "length": 145,
                    "text": "Not only are we expanding our number of compliance offerings and regulatory alignments, we continue to add regions and services at a faster rate.",
                    "sentiment": "Positive",
                    "scores": {
                        "Negative": 0.022516921,
                        "Positive": 0.46578798,
                        "Mixed": 0.064058214,
                        "Neutral": 0.44763687
                    }
                }
            ],
            "aspects": [
                {
                    "offset": 325,
                    "length": 9,
                    "text": "resources",
                    "sentiment": "Positive",
                    "scores": {
                        "Positive": 0.9841423768960832,
                        "Negative": 0.01398839404953044,
                        "Neutral": 0,
                        "Mixed": 0.0018692290543864747
                    }
                }
            ],
            "languageCode": "en"
        }
    ],
    "errors": []
}

Sentence Level Sentiment Analysis Example

Positive: 0.9997686743736267,
Negative: 0.00023133518698159605,
Neutral: 0
Positive: 0.0043107867240906,
Negative: 
0.9956892132759094,
Neutral": 0
Positive: 0.9908866882324219,
Negative: 0,
Neutral: 0.009113257750868797
Positive: 0.0933981895446777,
Negative: 0,
Neutral: 0.906601857487112284

Sample Request:

API Request format:

POST https://<region-url>/20210101/actions/batchDetectLanguageSentiments?level=SENTENCE

Input JSON

{
    "documents": [
        {
            "key": "doc1",
            "text": "OCI recently added new services to existing compliance program including SOC, HIPAA, and ISO to enable our customers to solve their use cases. We also released new white papers and guidance documents related to Object Storage, the Australian Prudential Regulation Authority (APRA), and the Central Bank of Brazil. These resources help regulated customers better understand how OCI supports their regional and industry-specific compliance requirements. Not only are we expanding our number of compliance offerings and regulatory alignments, we continue to add regions and services at a faster clip."
        }
    ]
}

Response JSON:

{
    "documents": [
        {
            "key": "doc1",
            "documentSentiment": "positive",
            "documentScores": {
                "positive": 0.6763582,
                "mixed": 0.08708387,
                "neutral": 0.12376911,
                "negative": 0.11278882
            },
            "sentences": [
                {
                    "text": "OCI recently added new services to existing compliance program including SOC, HIPAA, and ISO to enable our customers to solve their use cases.",
                    "sentiment": "neutral",
                    "scores": {
                        "positive": 0.15475821,
                        "neutral": 0.5567636,
                        "mixed": 0.09907853,
                        "negative": 0.18939966
                    }
                },
                {
                    "text": "We also released new white papers and guidance documents related to Object Storage, the Australian Prudential Regulation Authority (APRA), and the Central Bank of Brazil.",
                    "sentiment": "neutral",
                    "scores": {
                        "mixed": 0.07148028,
                        "negative": 0.12318015,
                        "positive": 0.11138679,
                        "neutral": 0.6939528
                    }
                },
                {
                    "text": "These resources help regulated customers better understand how OCI supports their regional and industry-specific compliance requirements.",
                    "sentiment": "positive",
                    "scores": {
                        "negative": 0.11278882,
                        "neutral": 0.12376911,
                        "mixed": 0.08708387,
                        "positive": 0.6763582
                    }
                },
                {
                    "text": "Not only are we expanding our number of compliance offerings and regulatory alignments, we continue to add regions and services at a faster clip.",
                    "sentiment": "neutral",
                    "scores": {
                        "mixed": 0.0973028,
                        "positive": 0.18745653,
                        "negative": 0.1592837,
                        "neutral": 0.55595696
                    }
                }
            ],
            "aspects": [],
            "languageCode": "en"
        }
    ],
    "errors": []
}

The actual values, and input and output structure might vary by model version, see the SDK documentation.

Limitations

• The identified aspects might be partial matches or split aspects.

• When sentences are semantically or structurally incorrect, the aspects could differ from your expectations.

• Pronouns are not considered aspects.

• Sarcasm is not recognized.

Use Cases

Detecting and curating private information in user feedback

Many organizations collect user feedback is collected through various channels such as product reviews, return requests, support tickets, and feedback forums. You can use Language PII detection service for automatic detection of PII entities to not only proactively warn, but also anonymize before storing posted feedback. Using the automatic detection of PII entities allows you to proactively warn users about sharing private data, and applications to implement measures like storing masked data.

Scanning object storage for presence of sensitive data

Cloud storage solutions such as OCI Object Storage are widely used by employees to store business documents in the locations either locally controlled or shared by multiple teams. Ensuring that such shared locations do not store private information such as employee names, demographics and payroll information requires automatic scanning of all the documents for presence of PII. OCI language PII service provides batch API to process multiple text documents at scale for processing data at scale.

Examples

The JSON for the example is:

Sample Request

POST https://<region-url>/20210101/actions/batchDetectLanguagePiiEntities

API Request format:

{
  "documents": [
    {
      "languageCode": "en",
      "key": "1",
      "text": "Hello Support Team, I am reaching out to seek help with my credit card number 1234 5678 9873 2345 expiring on 11/23. There was a suspicious transaction on 12-Aug-2022 which I reported by calling from my mobile number +1 (423) 111-9999 also I emailed from my email id sarah.jones1234@hotmail.com. Would you please let me know the refund status? Regards, Sarah"
    }
  ],
  "compartmentId": "ocid1.tenancy.oc1..aaaaaaaadany3y6wdh3u3jcodcmm42ehsdno525pzyavtjbpy72eyxcu5f7q",
  "masking": {
    "ALL": {
      "mode": "MASK",
      "isUnmaskedFromEnd": true,
      "leaveCharactersUnmasked": 4
    }
  }
}

Response JSON:

{
    "documents": [
        {
            "key": "1",
            "entities": [
                {
                    "offset": 79,
                    "length": 19,
                    "type": "CREDIT_DEBIT_NUMBER",
                    "text": "1234 5678 9873 2345",
                    "score": 0.75,
                    "isCustom": false
                },
                {
                    "offset": 111,
                    "length": 5,
                    "type": "DATE_TIME",
                    "text": "11/23",
                    "score": 0.9992455840110779,
                    "isCustom": false
                },
                {
                    "offset": 156,
                    "length": 11,
                    "type": "DATE_TIME",
                    "text": "12-Aug-2022",
                    "score": 0.998766303062439,
                    "isCustom": false
                },
                {
                    "offset": 218,
                    "length": 2,
                    "type": "TELEPHONE_NUMBER",
                    "text": "+1",
                    "score": 0.6941494941711426,
                    "isCustom": false
                },
                {
                    "offset": 221,
                    "length": 14,
                    "type": "TELEPHONE_NUMBER",
                    "text": "(423) 111-9999",
                    "score": 0.9527066349983215,
                    "isCustom": false
                },
                {
                    "offset": 268,
                    "length": 27,
                    "type": "EMAIL",
                    "text": "sarah.jones1234@hotmail.com",
                    "score": 0.95,
                    "isCustom": false
                },
                {
                    "offset": 354,
                    "length": 5,
                    "type": "PERSON",
                    "text": "Sarah",
                    "score": 0.9918518662452698,
                    "isCustom": false
                }
            ],
            "languageCode": "en",
            "maskedText": "Hello Support Team, \nI am reaching out to seek help with my credit card number ***************2345 expiring on *1/23. There was a suspicious transaction on *******2022 which I reported by calling from my mobile number +1 **********9999 also I emailed from my email id ***********************.com. Would you please let me know the refund status?\nRegards,\n*arah"
        }
    ],
    "errors": []
}