Documentation sur Oracle Cloud Infrastructure

Intégration Label Studio

Le service d'étiquetage de données d'Oracle est en cours d'abandon. Nous vous recommandons de migrer vos jeux de données étiquetés vers Label Studio, un outil d'étiquetage open source et pris en charge par le marché.

Avec l'abandon du service d'étiquetage de données, suivez ces étapes pour convertir les exportations d'instantanés du service d'étiquetage de données en formats d'importation Label Studio et d'exportation JSON bruts Label Studio. Vous pouvez utiliser ces formats pour d'autres annotations dans Label Studio ou pour l'entraînement direct de modèles.

Flux de travail d'étiquetage pour NER personnalisé ou TXTC personnalisé

Annoter un nouveau jeu de données dans Label Studio.
  1. Chargez un jeu de données, soit un fichier texte, soit des jeux de données préannotés convertis à partir du service d'étiquetage de données.
  2. (Facultatif) Si nécessaire, annotez les intervalles d'entités.
  3. Exportez les données annotées.
    • Le jeu de données est exporté au format JSON brut à partir de Label Studio.
    • Ces données exportées peuvent être chargées dans le stockage d'objets pour traitement ultérieur.

Utilisez les données du flux de travail d'entraînement de modèle dans le pipeline Language.

Migration des données annotées du service d'étiquetage de données existantes vers Label Studio

Si vous avez déjà annoté des données dans le service d'étiquetage de données, un script de migration est fourni pour convertir le jeu de données existant dans un format compatible avec Label Studio.
  1. Exportez le jeu de données du service d'étiquetage de données au format JSONL vers un seau de stockage d'objets.
  2. Exécutez le script de conversion.
    Un script Python traite le jeu de données et le convertit en :
    • Label Studio Import Format pour permettre une utilisation directe dans Label Studio pour une annotation supplémentaire.
    • Label Studio Export Format, structuré pour une utilisation directe dans les workflows de formation sans modification supplémentaire.
  3. Chargez le fichier de format d'importation converti dans Label Studio pour toute mise à jour d'annotation.
  4. Exportez le jeu de données mis à jour dans le format JSONL brut vers le stockage d'objets.
Utilisez les données pour la formation en langue.

Formats de données

CNER

Les données CNER peuvent être stockées dans les formats suivants :
  • Format d'étiquetage de données
  • Format portable JSNOL (texte autonome)
  • Format d'exportation JSON brut Label Studio

Format d'étiquetage de données (Portable JSONL)

La principale différence entre l'étiquetage de données et le format JSONL portable est la façon dont le texte est stocké :

• Portable JSONL : Le texte réel est présent dans le même fichier sous sourceDetails['text'].

• Format d'étiquetage de données : Au lieu de texte, sourcePath est spécifié dans sourceDetails['path'], pointant vers un emplacement externe où le texte est stocké.

Exemples de formats différents :
CNER format JSONL portable
{"labelsSet": [{"name": "PER"}, {"name": "MISC"}, {"name": "LOC"}, {"name": "ORG"}], "annotationFormat": "ENTITY_EXTRACTION", "datasetFormatDetails": {"formatType": "TEXT"}}
{"sourceDetails": {"text": "His current band , Gigi , was formed in 1994 with Baron ( guitar ) , Thomas ( bass ) , Armand ( vocal ) and Ronald ( drum ) ."}, "annotations": [{"entities": [{"entityType": "TEXTSELECTION", "labels": [{"label_name": "ORG"}], "textSpan": {"offset": 19, "length": 4}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "PER"}], "textSpan": {"offset": 50, "length": 5}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "PER"}], "textSpan": {"offset": 69, "length": 6}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "PER"}], "textSpan": {"offset": 87, "length": 6}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "PER"}], "textSpan": {"offset": 108, "length": 6}}]}]}
{"sourceDetails": {"text": "Since then , Gigi has released six albums ."}, "annotations": [{"entities": [{"entityType": "TEXTSELECTION", "labels": [{"label_name": "ORG"}], "textSpan": {"offset": 13, "length": 4}}]}]}
{"sourceDetails": {"text": "He has also released his own solo albums : Nusa Damai ; Gitarku ; Samsara ; and Home , a tribute album to the December 2004 tsunami victims ."}, "annotations": [{"entities": [{"entityType": "TEXTSELECTION", "labels": [{"label_name": "MISC"}], "textSpan": {"offset": 43, "length": 10}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "MISC"}], "textSpan": {"offset": 56, "length": 7}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "MISC"}], "textSpan": {"offset": 66, "length": 7}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "MISC"}], "textSpan": {"offset": 80, "length": 4}}]}]}
{"sourceDetails": {"text": "Since Dewa Budjana started his professional career as a musician , he has mainly used a Parker Fly Delux as his main guitar , occasionally using Klein and Gibson guitars SG series instead ."}, "annotations": [{"entities": [{"entityType": "TEXTSELECTION", "labels": [{"label_name": "PER"}], "textSpan": {"offset": 6, "length": 12}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "MISC"}], "textSpan": {"offset": 88, "length": 16}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "ORG"}], "textSpan": {"offset": 145, "length": 16}}]}]}
{"sourceDetails": {"text": "Budjana also owns a double neck Klein guitar which was used on his latest album , Home , and pictured on the CD cover ."}, "annotations": [{"entities": [{"entityType": "TEXTSELECTION", "labels": [{"label_name": "PER"}], "textSpan": {"offset": 0, "length": 7}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "ORG"}], "textSpan": {"offset": 32, "length": 5}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "MISC"}], "textSpan": {"offset": 82, "length": 4}}]}]}
Format d'importation Label Studio
[{'data': {'text': 'His current band , Gigi , was formed in 1994 with Baron ( guitar ) , Thomas ( bass ) , Armand ( vocal ) and Ronald ( drum ) .'},
  'predictions': [{'result': [{'value': {'start': 19,
       'end': 23,
       'text': 'Gigi',
       'labels': ['ORG']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 50, 'end': 55, 'text': 'Baron', 'labels': ['PER']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 69, 'end': 75, 'text': 'Thomas', 'labels': ['PER']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 87, 'end': 93, 'text': 'Armand', 'labels': ['PER']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 108, 'end': 114, 'text': 'Ronald', 'labels': ['PER']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'}]}]},
 {'data': {'text': 'Since then , Gigi has released six albums .'},
  'predictions': [{'result': [{'value': {'start': 13,
       'end': 17,
       'text': 'Gigi',
       'labels': ['ORG']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'}]}]},
 {'data': {'text': 'He has also released his own solo albums : Nusa Damai ; Gitarku ; Samsara ; and Home , a tribute album to the December 2004 tsunami victims .'},
  'predictions': [{'result': [{'value': {'start': 43,
       'end': 53,
       'text': 'Nusa Damai',
       'labels': ['MISC']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 56, 'end': 63, 'text': 'Gitarku', 'labels': ['MISC']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 66, 'end': 73, 'text': 'Samsara', 'labels': ['MISC']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 80, 'end': 84, 'text': 'Home', 'labels': ['MISC']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'}]}]},
 {'data': {'text': 'Since Dewa Budjana started his professional career as a musician , he has mainly used a Parker Fly Delux as his main guitar , occasionally using Klein and Gibson guitars SG series instead .'},
  'predictions': [{'result': [{'value': {'start': 6,
       'end': 18,
       'text': 'Dewa Budjana',
       'labels': ['PER']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 88,
       'end': 104,
       'text': 'Parker Fly Delux',
       'labels': ['MISC']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 145,
       'end': 161,
       'text': 'Klein and Gibson',
       'labels': ['ORG']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'}]}]}]
Format d'exportation Label Studio (au format JSON brut)
[{'id': 141,
  'annotations': [{'id': 42,
    'completed_by': 4,
    'result': [{'value': {'start': 19,
       'end': 23,
       'text': 'Gigi',
       'labels': ['ORG']},
      'id': '467b95b0',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 50, 'end': 55, 'text': 'Baron', 'labels': ['PER']},
      'id': '8f12fa43',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 69, 'end': 75, 'text': 'Thomas', 'labels': ['PER']},
      'id': '936f63f0',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 87, 'end': 93, 'text': 'Armand', 'labels': ['PER']},
      'id': 'f02d2509',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 108, 'end': 114, 'text': 'Ronald', 'labels': ['PER']},
      'id': 'cb479df1',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T16:47:10.630009Z',
    'updated_at': '2025-01-19T16:47:10.630009Z',
    'draft_created_at': None,
    'lead_time': 0,
    'prediction': {'id': 95,
     'result': [{'value': {'start': 19,
        'end': 23,
        'text': 'Gigi',
        'labels': ['ORG']},
       'id': '467b95b0',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 50, 'end': 55, 'text': 'Baron', 'labels': ['PER']},
       'id': '8f12fa43',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 69, 'end': 75, 'text': 'Thomas', 'labels': ['PER']},
       'id': '936f63f0',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 87, 'end': 93, 'text': 'Armand', 'labels': ['PER']},
       'id': 'f02d2509',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 108,
        'end': 114,
        'text': 'Ronald',
        'labels': ['PER']},
       'id': 'cb479df1',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'}],
     'model_version': 'undefined',
     'created_ago': '0\xa0minutes',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T16:47:10.630009Z',
     'updated_at': '2025-01-19T16:47:10.630009Z',
     'model': None,
     'model_run': None,
     'task': 141,
     'project': 14},
    'result_count': 5,
    'unique_id': '54a60d2b-6960-48c9-baae-bf24b15ce273',
    'import_id': None,
    'last_action': None,
    'task': 141,
    'project': 14,
    'updated_by': 4,
    'parent_prediction': 95,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': 'example_file.json',
  'drafts': [],
  'predictions': [95],
  'data': {'text': 'His current band , Gigi , was formed in 1994 with Baron ( guitar ) , Thomas ( bass ) , Armand ( vocal ) and Ronald ( drum ) .'},
  'meta': {},
  'created_at': '2025-01-19T16:47:10.630009Z',
  'updated_at': '2025-01-19T16:47:10.630009Z',
  'inner_id': 1,
  'total_annotations': 1,
  'cancelled_annotations': 0,
  'total_predictions': 1,
  'comment_count': 0,
  'unresolved_comment_count': 0,
  'last_comment_updated_at': None,
  'project': 14,
  'updated_by': 4,
  'comment_authors': []},
 {'id': 141,
  'annotations': [{'id': 60,
    'completed_by': 4,
    'result': [{'value': {'start': 13,
       'end': 17,
       'text': 'Gigi',
       'labels': ['ORG']},
      'id': 'ce521973',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T16:47:10.630104Z',
    'updated_at': '2025-01-19T16:47:10.630104Z',
    'draft_created_at': None,
    'lead_time': 0,
    'prediction': {'id': 95,
     'result': [{'value': {'start': 13,
        'end': 17,
        'text': 'Gigi',
        'labels': ['ORG']},
       'id': 'ce521973',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'}],
     'model_version': 'undefined',
     'created_ago': '0\xa0minutes',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T16:47:10.630104Z',
     'updated_at': '2025-01-19T16:47:10.630104Z',
     'model': None,
     'model_run': None,
     'task': 141,
     'project': 14},
    'result_count': 1,
    'unique_id': '4a4b5631-4b51-4a09-99c4-ad8927074503',
    'import_id': None,
    'last_action': None,
    'task': 141,
    'project': 14,
    'updated_by': 4,
    'parent_prediction': 95,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': 'example_file.json',
  'drafts': [],
  'predictions': [95],
  'data': {'text': 'Since then , Gigi has released six albums .'},
  'meta': {},
  'created_at': '2025-01-19T16:47:10.630104Z',
  'updated_at': '2025-01-19T16:47:10.630104Z',
  'inner_id': 1,
  'total_annotations': 1,
  'cancelled_annotations': 0,
  'total_predictions': 1,
  'comment_count': 0,
  'unresolved_comment_count': 0,
  'last_comment_updated_at': None,
  'project': 14,
  'updated_by': 4,
  'comment_authors': []},
 {'id': 141,
  'annotations': [{'id': 96,
    'completed_by': 4,
    'result': [{'value': {'start': 43,
       'end': 53,
       'text': 'Nusa Damai',
       'labels': ['MISC']},
      'id': 'a013849e',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 56, 'end': 63, 'text': 'Gitarku', 'labels': ['MISC']},
      'id': '57423aa5',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 66, 'end': 73, 'text': 'Samsara', 'labels': ['MISC']},
      'id': 'eeee84bb',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 80, 'end': 84, 'text': 'Home', 'labels': ['MISC']},
      'id': '3df96d57',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T16:47:10.630126Z',
    'updated_at': '2025-01-19T16:47:10.630126Z',
    'draft_created_at': None,
    'lead_time': 0,
    'prediction': {'id': 95,
     'result': [{'value': {'start': 43,
        'end': 53,
        'text': 'Nusa Damai',
        'labels': ['MISC']},
       'id': 'a013849e',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 56,
        'end': 63,
        'text': 'Gitarku',
        'labels': ['MISC']},
       'id': '57423aa5',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 66,
        'end': 73,
        'text': 'Samsara',
        'labels': ['MISC']},
       'id': 'eeee84bb',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 80, 'end': 84, 'text': 'Home', 'labels': ['MISC']},
       'id': '3df96d57',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'}],
     'model_version': 'undefined',
     'created_ago': '0\xa0minutes',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T16:47:10.630126Z',
     'updated_at': '2025-01-19T16:47:10.630126Z',
     'model': None,
     'model_run': None,
     'task': 141,
     'project': 14},
    'result_count': 4,
    'unique_id': 'd2b8ec9a-cc62-4881-a42b-833546df7953',
    'import_id': None,
    'last_action': None,
    'task': 141,
    'project': 14,
    'updated_by': 4,
    'parent_prediction': 95,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': 'example_file.json',
  'drafts': [],
  'predictions': [95],
  'data': {'text': 'He has also released his own solo albums : Nusa Damai ; Gitarku ; Samsara ; and Home , a tribute album to the December 2004 tsunami victims .'},
  'meta': {},
  'created_at': '2025-01-19T16:47:10.630126Z',
  'updated_at': '2025-01-19T16:47:10.630126Z',
  'inner_id': 1,
  'total_annotations': 1,
  'cancelled_annotations': 0,
  'total_predictions': 1,
  'comment_count': 0,
  'unresolved_comment_count': 0,
  'last_comment_updated_at': None,
  'project': 14,
  'updated_by': 4,
  'comment_authors': []},
 {'id': 141,
  'annotations': [{'id': 7,
    'completed_by': 4,
    'result': [{'value': {'start': 6,
       'end': 18,
       'text': 'Dewa Budjana',
       'labels': ['PER']},
      'id': '2d675f56',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 88,
       'end': 104,
       'text': 'Parker Fly Delux',
       'labels': ['MISC']},
      'id': '13f6536b',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 145,
       'end': 161,
       'text': 'Klein and Gibson',
       'labels': ['ORG']},
      'id': '283b9e25',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T16:47:10.630165Z',
    'updated_at': '2025-01-19T16:47:10.630165Z',
    'draft_created_at': None,
    'lead_time': 0,
    'prediction': {'id': 95,
     'result': [{'value': {'start': 6,
        'end': 18,
        'text': 'Dewa Budjana',
        'labels': ['PER']},
       'id': '2d675f56',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 88,
        'end': 104,
        'text': 'Parker Fly Delux',
        'labels': ['MISC']},
       'id': '13f6536b',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 145,
        'end': 161,
        'text': 'Klein and Gibson',
        'labels': ['ORG']},
       'id': '283b9e25',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'}],
     'model_version': 'undefined',
     'created_ago': '0\xa0minutes',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T16:47:10.630165Z',
     'updated_at': '2025-01-19T16:47:10.630165Z',
     'model': None,
     'model_run': None,
     'task': 141,
     'project': 14},
    'result_count': 3,
    'unique_id': '21115ec1-4a96-43c4-b1ec-1734cd004160',
    'import_id': None,
    'last_action': None,
    'task': 141,
    'project': 14,
    'updated_by': 4,
    'parent_prediction': 95,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': 'example_file.json',
  'drafts': [],
  'predictions': [95],
  'data': {'text': 'Since Dewa Budjana started his professional career as a musician , he has mainly used a Parker Fly Delux as his main guitar , occasionally using Klein and Gibson guitars SG series instead .'},
  'meta': {},
  'created_at': '2025-01-19T16:47:10.630165Z',
  'updated_at': '2025-01-19T16:47:10.630165Z',
  'inner_id': 1,
  'total_annotations': 1,
  'cancelled_annotations': 0,
  'total_predictions': 1,
  'comment_count': 0,
  'unresolved_comment_count': 0,
  'last_comment_updated_at': None,
  'project': 14,
  'updated_by': 4,
  'comment_authors': []}]

TXTC personnalisé (CTXTC) - Classification de texte

Les jeux de données CTXTC prennent en charge plusieurs formats pour plus de flexibilité dans l'annotation et le traitement :
  • Format d'étiquetage de données
  • Format CSV
  • Format Label Studio
Format d'étiquetage de données
CTXTC format portablejsonl
Format CSV
CTXTC format CSV - ÉTIQUETTE UNIQUE
Format CSV CTXTC - ÉTIQUETTE MULTIPLE
Format d'importation Label Studio
CTXTC format CSV - ÉTIQUETTE UNIQUE
Format CSV CTXTC - ÉTIQUETTE MULTIPLE
[{'data': {'text': "I need to book a hotel in the east that has 4 stars. I can help you with that. What is your price range? That doesn't matter as long as it has free wifi and parking. If you'd like something cheap, I recommend the Allenbell. For something moderately priced, I would recommend the Warkworth House. Could you book the Wartworth for one night, 1 person? What day will you be staying? Friday and Can you book it for me and get a reference number ? Booking was successful. Reference number is : BMUKPTG6.  Can I help you with anything else today? I am looking to book a train that is leaving from Cambridge to Bishops Stortford on Friday. There are a number of trains leaving throughout the day.  What time would you like to travel? I want to get there by 19:45 at the latest. Okay! The latest train you can take leaves at 17:29, and arrives by 18:07. Would you like for me to book that for you? Yes please. I also need the travel time, departure time, and price. Reference number is : UIFV8FAS. The price is 10.1 GBP and the trip will take about 38 minutes. May I be of any other assistance? Yes. Sorry, but suddenly my plans changed. Can you change the Wartworth booking to Monday for 3 people and 4 nights? I have made that change and your reference number is YF86GE4J. Thank you very much, goodbye. You're welcome. Have a nice day!"},
  'predictions': [{'model_version': '1.3',
    'result': [{'id': '0',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['hotel', 'train']}}]}]},
 {'data': {'text': 'Howdy, I need a train heading into cambridge. I would be happy to help you find a train.  Where are you departing from? I am departing from norwich.  I need to leave after 18:45 on Wednesday. I have several options for you. Where is your destination? I will be heading to cabridge. The earliest after 18:45 is the TR8658, leaving Norwich at 19:16. Can I reserve you one or more seats on this train? yeah, i need one ticket. Booking was successful, the total fee is 17.6 GBP payable at the station. Your reference number is AXH1NM1I. Do you need assistance with anything else? I am also looking for a multi sport in the East. It looks like there is The Cherry Hinton Village Centre.  Can I get you more information about it? I would like to get the phone number, please. their phone number is 01223576412. anything else? Oh, and what is their postcode, please? Sure, the postcode is cb19ej. Can I help you find any other information about Cambridge? That is all for now thank you. enjoy your time in Cambridge!'},
  'predictions': [{'model_version': '1.3',
    'result': [{'id': '1',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['attraction', 'train']}}]}]},
 {'data': {'text': "What can you tell me about the Riverside Brasserie? It is a restaurant that serves modern european food near the centre of town. It is moderately priced. The phone number is 01223259988. Perfect. Can you help me with a reservation for 6 people at 14:30 this coming sunday? And please make sure I have a confirmation number to use. Your reservation is set! The table will be reserved for 15 minutes. Your reference number is LZLUDTVI. Is there anything else you need? I'm also looking for a place to stay.  In the south preferably. What price range were you thinking? No particular price range, but I would like it to be a 4 star hotel. There are no hotels that fit your criteria in the South, but there are two Guesthouses. Would you like to book one of those? Sure, that will work. Can you tell me more about them? Aylesbray Lodge Guesthouse and Rosa's Bed and Breakfast, both are rated at 4 stars and both include free parking and internet. Would you like a reservation for one of them? Can I get the postcode for both of them? Aylesbray postcode is cd17sr and Rosa's postcode is cb22ha. Is there anything else I can help you with today? No thanks. That's all the help I need. Take care. Bye. thank you! Enjoy your stay!"},
  'predictions': [{'model_version': '1.3',
    'result': [{'id': '2',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['hotel', 'restaurant']}}]}]},
 {'data': {'text': "I am looking for a specific hotel, its name is express by holiday inn cambridge. I have the Express by Holiday Inn Cambridge located on 15-17 norman way, coldhams business park. Their phone number is 01223866800. Would you like to know anything else? Yes, could you book the hotel room for me for 7 people? Yes, of course. What day would you like to stay? Monday, please. There will be 7 of us and we'd like to stay for 4 days. Here is the booking information:Booking was successful. Reference number is : 5F8G6J1G. Thank you. I would also like to book a train, please. Sure, which stations will you be traveling between? I will be going from cambridge to birmingham new street. What time would you like to leave?  The trains depart every hour. Whenever will get me there by 17:30. I do need to leave on Friday and I will need the travel time please. There are 11 results. Would you prefer the earliest departure time or latest?"},
  'predictions': [{'model_version': '1.3',
    'result': [{'id': '3',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['hotel', 'train']}}]}]}]
Format d'exportation Label Studio
CTXTC LABEL Studio Format d'exportation - ÉTIQUETTE UNIQUE
CTXTC LABEL Studio Format d'exportation - ÉTIQUETTE MULTIPLE
[{'id': 237,
  'annotations': [{'id': 110,
    'completed_by': 4,
    'result': [{'id': '0',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['hotel', 'train']},
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T19:15:56.447771Z',
    'updated_at': '2025-01-19T19:15:57.447771Z',
    'draft_created_at': None,
    'lead_time': 1.101,
    'prediction': {'id': 253,
     'result': [{'id': '0',
       'from_name': 'textClassification',
       'to_name': 'text',
       'type': 'choices',
       'value': {'choices': ['hotel', 'train']}}],
     'model_version': '1.3',
     'created_ago': '1\xa0minute',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T19:15:56.447771Z',
     'updated_at': '2025-01-19T19:15:57.447771Z',
     'model': None,
     'model_run': None,
     'task': 237,
     'project': 22},
    'result_count': 1,
    'unique_id': '6aa5eb24-4c5b-4e7a-9770-eca304ee1420',
    'import_id': None,
    'last_action': None,
    'task': 237,
    'project': 22,
    'updated_by': 4,
    'parent_prediction': 253,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': '0606dc5a-multiLabel_ethos_train_LabelStudio_Upload.json',
  'drafts': [],
  'predictions': [253],
  'data': {'text': "I need to book a hotel in the east that has 4 stars. I can help you with that. What is your price range? That doesn't matter as long as it has free wifi and parking. If you'd like something cheap, I recommend the Allenbell. For something moderately priced, I would recommend the Warkworth House. Could you book the Wartworth for one night, 1 person? What day will you be staying? Friday and Can you book it for me and get a reference number ? Booking was successful. Reference number is : BMUKPTG6.  Can I help you with anything else today? I am looking to book a train that is leaving from Cambridge to Bishops Stortford on Friday. There are a number of trains leaving throughout the day.  What time would you like to travel? I want to get there by 19:45 at the latest. Okay! The latest train you can take leaves at 17:29, and arrives by 18:07. Would you like for me to book that for you? Yes please. I also need the travel time, departure time, and price. Reference number is : UIFV8FAS. The price is 10.1 GBP and the trip will take about 38 minutes. May I be of any other assistance? Yes. Sorry, but suddenly my plans changed. Can you change the Wartworth booking to Monday for 3 people and 4 nights? I have made that change and your reference number is YF86GE4J. Thank you very much, goodbye. You're welcome. Have a nice day!"},
  'meta': {},
  'created_at': '2025-01-19T19:15:56.447771Z',
  'updated_at': '2025-01-19T19:15:57.447771Z',
  'inner_id': 2,
  'total_annotations': 1,
  'cancelled_annotations': 0,
  'total_predictions': 1,
  'comment_count': 0,
  'unresolved_comment_count': 0,
  'last_comment_updated_at': None,
  'project': 22,
  'updated_by': 4,
  'comment_authors': []},
 {'id': 237,
  'annotations': [{'id': 110,
    'completed_by': 4,
    'result': [{'id': '1',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['attraction', 'train']},
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T19:15:56.447847Z',
    'updated_at': '2025-01-19T19:15:57.447847Z',
    'draft_created_at': None,
    'lead_time': 1.101,
    'prediction': {'id': 253,
     'result': [{'id': '1',
       'from_name': 'textClassification',
       'to_name': 'text',
       'type': 'choices',
       'value': {'choices': ['attraction', 'train']}}],
     'model_version': '1.3',
     'created_ago': '1\xa0minute',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T19:15:56.447847Z',
     'updated_at': '2025-01-19T19:15:57.447847Z',
     'model': None,
     'model_run': None,
     'task': 237,
     'project': 22},
    'result_count': 1,
    'unique_id': 'adfa5ebc-bd45-4c9a-9f7e-1e797329dd85',
    'import_id': None,
    'last_action': None,
    'task': 237,
    'project': 22,
    'updated_by': 4,
    'parent_prediction': 253,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': '0606dc5a-multiLabel_ethos_train_LabelStudio_Upload.json',
  'drafts': [],
  'predictions': [253],
  'data': {'text': 'Howdy, I need a train heading into cambridge. I would be happy to help you find a train.  Where are you departing from? I am departing from norwich.  I need to leave after 18:45 on Wednesday. I have several options for you. Where is your destination? I will be heading to cabridge. The earliest after 18:45 is the TR8658, leaving Norwich at 19:16. Can I reserve you one or more seats on this train? yeah, i need one ticket. Booking was successful, the total fee is 17.6 GBP payable at the station. Your reference number is AXH1NM1I. Do you need assistance with anything else? I am also looking for a multi sport in the East. It looks like there is The Cherry Hinton Village Centre.  Can I get you more information about it? I would like to get the phone number, please. their phone number is 01223576412. anything else? Oh, and what is their postcode, please? Sure, the postcode is cb19ej. Can I help you find any other information about Cambridge? That is all for now thank you. enjoy your time in Cambridge!'},
  'meta': {},
  'created_at': '2025-01-19T19:15:56.447847Z',
  'updated_at': '2025-01-19T19:15:57.447847Z',
  'inner_id': 2,
  'total_annotations': 1,
  'cancelled_annotations': 0,
  'total_predictions': 1,
  'comment_count': 0,
  'unresolved_comment_count': 0,
  'last_comment_updated_at': None,
  'project': 22,
  'updated_by': 4,
  'comment_authors': []},
 {'id': 237,
  'annotations': [{'id': 110,
    'completed_by': 4,
    'result': [{'id': '2',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['hotel', 'restaurant']},
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T19:15:56.447870Z',
    'updated_at': '2025-01-19T19:15:57.447870Z',
    'draft_created_at': None,
    'lead_time': 1.101,
    'prediction': {'id': 253,
     'result': [{'id': '2',
       'from_name': 'textClassification',
       'to_name': 'text',
       'type': 'choices',
       'value': {'choices': ['hotel', 'restaurant']}}],
     'model_version': '1.3',
     'created_ago': '1\xa0minute',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T19:15:56.447870Z',
     'updated_at': '2025-01-19T19:15:57.447870Z',
     'model': None,
     'model_run': None,
     'task': 237,
     'project': 22},
    'result_count': 1,
    'unique_id': '2b0c6f75-82a7-4a0f-a5d5-a5a697ef6798',
    'import_id': None,
    'last_action': None,
    'task': 237,
    'project': 22,
    'updated_by': 4,
    'parent_prediction': 253,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': '0606dc5a-multiLabel_ethos_train_LabelStudio_Upload.json',
  'drafts': [],
  'predictions': [253]]

Scripts de conversion - CNER

Script : Convertir le format d'étiquetage de données en format Label Studio
Un script Python nommé cner_export_to_LS.py est fourni pour convertir le format JSONL d'étiquetage de données en formats d'importation et d'exportation Label Studio : 
import os
import argparse
import json
import uuid
from datetime import datetime
import random
import glob
 
 
def convert_to_label_studio_import(data):
    """ Convert portable JSONL format to Label Studio import format. """
    dd = []
    for i in data[1:]:
        ents = i['annotations'][0]['entities']
        ee = []
        text = i['sourceDetails']['text']
        for e in ents:
            start = e['textSpan']['offset']
            end = e['textSpan']['offset'] + e['textSpan']['length']
            ee.append({
                'value': {
                    'start': start,
                    'end': end,
                    'text': text[start:end],
                    'labels': [e['labels'][0]['label_name']]
                },
                'from_name': 'label',
                'to_name': 'text',
                'type': 'labels'
            })
        dd.append({
            'data': {'text': i['sourceDetails']['text']},
            'predictions': [{'result': ee}]
        })
    return dd
 
 
def convert_to_label_studio_raw_export(input_data):
    """ Convert portable JSONL format to Label Studio raw JSON export format. """
    # Extract text and annotations
    text = input_data['sourceDetails']['text']
    entities = input_data['annotations'][0]['entities']
     
    # Initialize variables
    task_id = 141
    project_id = 14
    now = datetime.now().isoformat() + "Z"
    unique_id = str(uuid.uuid4())
     
    # Process entities
    results = []
    for entity in entities:
        offset = entity['textSpan']['offset']
        length = entity['textSpan']['length']
        label_name = entity['labels'][0]['label_name']
        entity_text = text[offset:offset + length]
         
        result = {
            'value': {
                'start': offset,
                'end': offset + length,
                'text': entity_text,
                'labels': [label_name]
            },
            'id': str(uuid.uuid4())[:8],
            'from_name': 'label',
            'to_name': 'text',
            'type': 'labels',
            'origin': 'prediction'
        }
        results.append(result)
     
    # Construct the final output
    output = {
        'id': task_id,
        'annotations': [{
            'id': random.randint(1, 100),
            'completed_by': 4,
            'result': results,
            'was_cancelled': False,
            'ground_truth': False,
            'created_at': now,
            'updated_at': now,
            'draft_created_at': None,
            'lead_time': 0,
            'prediction': {
                'id': 95,
                'result': results,
                'model_version': 'undefined',
                'created_ago': '0\xa0minutes',
                'score': None,
                'cluster': None,
                'neighbors': None,
                'mislabeling': 0.0,
                'created_at': now,
                'updated_at': now,
                'model': None,
                'model_run': None,
                'task': task_id,
                'project': project_id
            },
            'result_count': len(results),
            'unique_id': unique_id,
            'import_id': None,
            'last_action': None,
            'task': task_id,
            'project': project_id,
            'updated_by': 4,
            'parent_prediction': 95,
            'parent_annotation': None,
            'last_created_by': None
        }],
        'file_upload': 'example_file.json',
        'drafts': [],
        'predictions': [95],
        'data': {'text': text},
        'meta': {},
        'created_at': now,
        'updated_at': now,
        'inner_id': 1,
        'total_annotations': 1,
        'cancelled_annotations': 0,
        'total_predictions': 1,
        'comment_count': 0,
        'unresolved_comment_count': 0,
        'last_comment_updated_at': None,
        'project': project_id,
        'updated_by': 4,
        'comment_authors': []
    }
     
    return output
 
 
def load_jsonl(file_path):
    """ Load a JSONL file and return a list of parsed JSON objects. """
    with open(file_path, 'r', encoding='utf-8') as f:
        return [json.loads(line) for line in f]
 
 
def save_jsonl(data, file_path):
    """ Save a list of JSON objects to a JSONL file. """
    with open(file_path, 'w', encoding='utf-8') as f:
        for entry in data:
            f.write(json.dumps(entry) + '\n')
 
 
def save_json(data, file_path):
    """ Save a JSON object to a JSON file. """
    with open(file_path, 'w', encoding='utf-8') as f:
        json.dump(data, f, indent=4)
 
 
def process_folder(input_folder, output_folder):
    """ Process all test, train, and dev files in the folder. """
    for split in ["test", "train", "dev"]:
        input_file = os.path.join(input_folder, f"{split}.jsonl")
        if not os.path.exists(input_file):
            print(f"Skipping {input_file}: File not found.")
            continue
 
        # Load input data
        data = load_jsonl(input_file)
         
        # Convert and save import format
        import_output_file = os.path.join(output_folder, f"{split}_import.jsonl")
        import_data = convert_to_label_studio_import(data)
        save_jsonl(import_data, import_output_file)
        print(f"Saved Label Studio import data to {import_output_file}")
         
        # Convert and save export format
        export_output_file = os.path.join(output_folder, f"{split}_export.json")
        export_data = [convert_to_label_studio_raw_export(item) for item in data]
        save_json(export_data, export_output_file)
        print(f"Saved Label Studio export data to {export_output_file}")
 
 
if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Convert JSONL files to Label Studio formats.")
    parser.add_argument("input_folder", type=str, help="Folder containing input JSONL files (test, train, dev).")
    parser.add_argument("output_folder", type=str, help="Folder to save converted files.")
    args = parser.parse_args()
 
    # Ensure output folder exists
    os.makedirs(args.output_folder, exist_ok=True)
 
    # Process input folder
    process_folder(args.input_folder, args.output_folder)

Instructions d'utilisation

  1. Placez tous les fichiers JSONL du service d'étiquetage de données (test, train, dev) dans un répertoire d'entrée.
  2. Exécutez le script, cner_conversion.ipynb : 
    import glob
 
 
def list_folders_in_directory(directory_path):
    """List all folders in a given directory using glob."""
    folder_paths = glob.glob(os.path.join(directory_path, "*/"))
    # folder_names = [os.path.basename(os.path.normpath(folder)) for folder in folder_paths]
    return folder_paths
 
 
def list_files_in_directory(directory_path, extension="*"):
    """List all files in a directory with an optional extension filter."""
    file_paths = glob.glob(os.path.join(directory_path, f"*.{extension}"))
    # file_names = [os.path.basename(file) for file in file_paths]
    return file_paths
 
directory_path = "/home/niksoni/cner_labelstudio_integeration/datasets/cner"
output_folder = "/home/niksoni/cner_labelstudio_integeration/datasets/cner/label_studio_formats"
folders = list_folders_in_directory(directory_path)
for folder in folders:
    folder_name = os.path.basename(os.path.normpath(folder))
    output_folder_path = f"{output_folder}/{folder_name}"
    os.makedirs(output_folder_path, exist_ok=True)
    files = list_files_in_directory(folder)
    for input_file in files:
        if not os.path.exists(input_file):
                    print(f"Skipping {input_file}: File not found.")
                    continue 
        data = load_jsonl(input_file)
        # Convert and save import format
        file_name = os.path.basename(input_file)
        import_output_file = os.path.join(output_folder_path, f"{file_name}_LS_import.json")
        import_data = convert_to_label_studio_import(data)
        save_json(import_data, import_output_file)
        print(f"Saved Label Studio import data to {import_output_file}")
         
        export_output_file = os.path.join(output_folder_path, f"{file_name}_LS_export.json")
        export_data = [convert_to_label_studio_raw_export(item) for item in data[1:]]
        save_json(export_data, export_output_file)
        print(f"Saved Label Studio export data to {export_output_file}")

    Le script génère les éléments suivants :

    • *_import.jsonl pour l'importation directe dans Label Studio
    • *_export.json pour une utilisation directe dans les flux de travail de formation
    Après la conversion, le dossier de sortie contient du contenu avec les structures suivantes :
    /input_folder/
│── test.jsonl
│── train.jsonl
│── dev.jsonl

/output_folder/
│── test_import.json
│── test_export.json
│── train_import.json
│── train_export.json
│── dev_import.json
│── dev_export.json

Scripts de conversion - CTXTC

Script : Convertir le format CSV en format Label Studio
Un script Python nommé ctxtc_export_to_LS.py est fourni pour convertir le format CSV en formats d'importation et d'exportation Label Studio :
import os
import argparse
import json
import uuid
# from datetime import datetime
import random
import glob
from datetime import datetime, timedelta
 
import pandas as pd
 
def load_jsonl(file_path):
    """ Load a JSONL file and return a list of parsed JSON objects. """
    with open(file_path, 'r', encoding='utf-8') as f:
        return [json.loads(line) for line in f]
 
 
def save_jsonl(data, file_path):
    """ Save a list of JSON objects to a JSONL file. """
    with open(file_path, 'w', encoding='utf-8') as f:
        for entry in data:
            f.write(json.dumps(entry) + '\n')
             
             
def save_to_json(data, file_path):
    with open(file_path, 'w') as f:
        json.dump(data, f, indent=4)
    print(f"Data successfully saved to {file_path}")
 
def load_from_json(file_path):
    with open(file_path, 'r') as f:
        data = json.load(f)
    print(f"Data successfully loaded from {file_path}")
    return data
 
 
# Function to convert dataframe to Label Studio format
def convert_to_label_studio(df, label_name="textClassification"):
    data = []
     
    for idx, row in df.iterrows():
        text = row['text']
        label = str(row['labels']).split("|")
         
        data_entry = {
            "data": {
                "text": text
            },
            "predictions": [
                {
                    "model_version": "1.3",  # This can be adjusted if needed
                    "result": [
                        {
                            "id": str(idx),
                            "from_name": label_name,  # Label name changed here
                            "to_name": "text",
                            "type": "choices",
                            "value": {
#                                 "score": 1.0,  # You can adjust the score based on confidence if needed
                                "choices": label
                            }
                        }
                    ]
                }
            ]
        }
        data.append(data_entry)
     
    return json.dumps(data, indent=4)
 
 
 
 
def convert_prediction_to_label_studio_export_format(input_data):
    # Generate placeholders
    current_time = datetime.utcnow()
    created_at = current_time.isoformat() + "Z"
    updated_at = (current_time + timedelta(seconds=1)).isoformat() + "Z"
    prediction_id = 253
    task_id = 237
    project_id = 22
    annotation_id = 110
    completed_by = 4
 
    # Extract values from input
    text = input_data['data']['text']
    predictions = input_data['predictions'][0]
    model_version = predictions['model_version']
    result = predictions['result']
 
    # Build the output structure
    converted_data = {
        'id': task_id,
        'annotations': [{
            'id': annotation_id,
            'completed_by': completed_by,
            'result': [
                {
                    **item,
                    'origin': 'prediction'
                }
                for item in result
            ],
            'was_cancelled': False,
            'ground_truth': False,
            'created_at': created_at,
            'updated_at': updated_at,
            'draft_created_at': None,
            'lead_time': 1.101,
            'prediction': {
                'id': prediction_id,
                'result': result,
                'model_version': model_version,
                'created_ago': '1\xa0minute',
                'score': None,
                'cluster': None,
                'neighbors': None,
                'mislabeling': 0.0,
                'created_at': created_at,
                'updated_at': updated_at,
                'model': None,
                'model_run': None,
                'task': task_id,
                'project': project_id
            },
            'result_count': 1,
            'unique_id': str(uuid.uuid4()),
            'import_id': None,
            'last_action': None,
            'task': task_id,
            'project': project_id,
            'updated_by': completed_by,
            'parent_prediction': prediction_id,
            'parent_annotation': None,
            'last_created_by': None
        }],
        'file_upload': '0606dc5a-multiLabel_ethos_train_LabelStudio_Upload.json',
        'drafts': [],
        'predictions': [prediction_id],
        'data': {
            'text': text
        },
        'meta': {},
        'created_at': created_at,
        'updated_at': updated_at,
        'inner_id': 2,
        'total_annotations': 1,
        'cancelled_annotations': 0,
        'total_predictions': 1,
        'comment_count': 0,
        'unresolved_comment_count': 0,
        'last_comment_updated_at': None,
        'project': project_id,
        'updated_by': completed_by,
        'comment_authors': []
    }
    return converted_data

Instructions d'utilisation

  1. Placez tous les fichiers JSONL du service d'étiquetage de données (test, train, dev) dans un répertoire d'entrée.
  2. Exécutez le script, ctxtc_conversion.ipynb : 
    import json
from ctxtc_conversion_to_LS import *
import glob
 
def list_folders_in_directory(directory_path):
    """List all folders in a given directory using glob."""
    folder_paths = glob.glob(os.path.join(directory_path, "*/"))
    # folder_names = [os.path.basename(os.path.normpath(folder)) for folder in folder_paths]
    return folder_paths
 
 
def list_files_in_directory(directory_path, extension="*"):
    """List all files in a directory with an optional extension filter."""
    file_paths = glob.glob(os.path.join(directory_path, f"*.{extension}"))
    # file_names = [os.path.basename(file) for file in file_paths]
    return file_paths
 
# cls_types = ["singleLabel", "multilabel"]
cls_types = ["multilabel"]
directory_path = "/home/niksoni/cner_labelstudio_integeration/datasets/ctxtc"
output_folder = "/home/niksoni/cner_labelstudio_integeration/datasets/ctxtc/label_studio_formats"
for cls_type in cls_types:
    cls_directory_path = f"{directory_path}/{cls_type}"
    folders = list_folders_in_directory(cls_directory_path)
    for folder in folders:
        folder_name = os.path.basename(os.path.normpath(folder))
        output_folder_path = f"{output_folder}/{cls_type}/{folder_name}"
        os.makedirs(output_folder_path, exist_ok=True)
        files = list_files_in_directory(folder)
        # print(files)
        for input_file in files:
            if not os.path.exists(input_file):
                        print(f"Skipping {input_file}: File not found.")
                        continue 
            df = pd.read_csv(input_file)
            json_data = convert_to_label_studio(df)
            json_data = json.loads(json_data)
            file_name = os.path.basename(input_file)
            # save_to_json(json_data,file_name)
            import_output_file = os.path.join(output_folder_path, f"{file_name}_LS_import.json")
            # import_data = convert_to_label_studio(data)
            save_to_json(json_data, import_output_file)
            print(f"Saved Label Studio import data to {import_output_file}")
 
            export_output_file = os.path.join(output_folder_path, f"{file_name}_LS_export.json")
            export_data = [convert_prediction_to_label_studio_export_format(item) for item in json_data]
            save_to_json(export_data, export_output_file)
            print(f"Saved Label Studio export data to {export_output_file}")

    Le script génère les éléments suivants :

    • *_import.jsonl pour l'importation directe dans Label Studio
    • *_export.json pour une utilisation directe dans les flux de travail de formation
    Après la conversion, le dossier de sortie contient du contenu avec les structures suivantes :
    /input_folder/
│── test.jsonl
│── train.jsonl
│── dev.jsonl

/output_folder/
│── test_import.json
│── test_export.json
│── train_import.json
│── train_export.json
│── dev_import.json
│── dev_export.json

Intégration avec la langue

Suivez ce flux de formation avec les données converties :
  1. Chargez un jeu de données, annoté manuellement à l'aide de label studio ou converti à partir du service d'étiquetage de données.
  2. Sélectionnez le jeu de données stocké dans le stockage d'objets pour l'entraînement de modèle.
    Le langage traite le jeu de données et poursuit le pipeline d'entraînement.

    Le modèle est validé, entraîné et déployé de la même manière qu'auparavant.

Conclusion

  • Le script de migration garantit une transition transparente du service d'étiquetage de données vers Label Studio.
  • Vous pouvez continuer à annoter des données existantes dans Label Studio.
  • Le jeu de données exporté est entièrement compatible avec les pipelines d'entraînement du service de langue.
  • Le format JSONL existant est toujours pris en charge pour la compatibilité descendante.
Pour plus de détails, voir la documentation Label Studio : Voir aussi :