Documentación de Oracle Cloud Infrastructure

Integración de Label Studio

El servicio Data Labeling de Oracle está en desuso. Le recomendamos que migre sus conjuntos de datos etiquetados a Label Studio, una herramienta de etiquetado de código abierto y compatible con el mercado.

Con el desuso de Data Labeling, siga estos pasos para convertir las exportaciones de instantáneas de Data Labeling a los formatos de importación de Label Studio y exportación de JSON raw de Label Studio. Puede utilizar estos formatos para realizar más anotaciones en Label Studio o en el entrenamiento directo del modelo.

Etiquetado de flujos de trabajo para NER personalizado o TXTC personalizado

Anote un nuevo juego de datos en Label Studio.
  1. Cargue un conjunto de datos, ya sea un archivo de texto o conjuntos de datos preanotados convertidos desde Data Labeling.
  2. (Opcional) Si es necesario, anote los intervalos de la entidad.
  3. Exporte los datos anotados.
    • El juego de datos se exporta en formato JSON raw desde Label Studio.
    • Estos datos exportados se pueden cargar en Object Storage para su procesamiento posterior.

Utilice los datos del flujo de trabajo de entrenamiento del modelo como parte del pipeline de idioma.

Migración de Datos Anotados de Etiquetado de Datos Existentes a Label Studio

Si ya ha anotado datos en Data Labeling, se proporciona un script de migración para convertir el conjunto de datos existente en un formato compatible con Label Studio.
  1. Exporte el juego de datos desde Data Labeling en formato JSONL a un cubo de almacenamiento de objetos.
  2. Ejecute el script de conversión.
    Un script de Python procesa el juego de datos y lo convierte en:
    • Formato de importación de Label Studio para activar el uso directo en Label Studio para más anotaciones.
    • Formato de exportación de Label Studio, estructurado para uso directo en flujos de trabajo de formación sin modificaciones adicionales.
  3. Cargue el archivo de formato de importación convertido en Label Studio para cualquier actualización de anotación.
  4. Exporte el juego de datos actualizado en formato Raw JSONL al almacenamiento de objetos.
Utilice los datos para la formación en idioma.

Formatos de Datos

CNER

Los datos del CNER se pueden almacenar en los siguientes formatos:
  • Formato de etiquetado de datos
  • Formato JSNOL portátil (texto autónomo)
  • Formato de exportación de JSON raw de Label Studio

Formato de etiquetado de datos (JSONL portátil)

La diferencia clave entre el etiquetado de datos y el formato JSONL portátil es cómo se almacena el texto:

• JSONL portátil: El texto real está presente en el mismo archivo en sourceDetails['text'].

• Formato de etiquetado de datos: en lugar de texto, se especifica sourcePath en sourceDetails['path'], que apunta a una ubicación externa donde se almacena el texto.

Ejemplos de diferentes formatos:
Formato JSONL portátil CNER
{"labelsSet": [{"name": "PER"}, {"name": "MISC"}, {"name": "LOC"}, {"name": "ORG"}], "annotationFormat": "ENTITY_EXTRACTION", "datasetFormatDetails": {"formatType": "TEXT"}}
{"sourceDetails": {"text": "His current band , Gigi , was formed in 1994 with Baron ( guitar ) , Thomas ( bass ) , Armand ( vocal ) and Ronald ( drum ) ."}, "annotations": [{"entities": [{"entityType": "TEXTSELECTION", "labels": [{"label_name": "ORG"}], "textSpan": {"offset": 19, "length": 4}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "PER"}], "textSpan": {"offset": 50, "length": 5}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "PER"}], "textSpan": {"offset": 69, "length": 6}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "PER"}], "textSpan": {"offset": 87, "length": 6}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "PER"}], "textSpan": {"offset": 108, "length": 6}}]}]}
{"sourceDetails": {"text": "Since then , Gigi has released six albums ."}, "annotations": [{"entities": [{"entityType": "TEXTSELECTION", "labels": [{"label_name": "ORG"}], "textSpan": {"offset": 13, "length": 4}}]}]}
{"sourceDetails": {"text": "He has also released his own solo albums : Nusa Damai ; Gitarku ; Samsara ; and Home , a tribute album to the December 2004 tsunami victims ."}, "annotations": [{"entities": [{"entityType": "TEXTSELECTION", "labels": [{"label_name": "MISC"}], "textSpan": {"offset": 43, "length": 10}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "MISC"}], "textSpan": {"offset": 56, "length": 7}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "MISC"}], "textSpan": {"offset": 66, "length": 7}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "MISC"}], "textSpan": {"offset": 80, "length": 4}}]}]}
{"sourceDetails": {"text": "Since Dewa Budjana started his professional career as a musician , he has mainly used a Parker Fly Delux as his main guitar , occasionally using Klein and Gibson guitars SG series instead ."}, "annotations": [{"entities": [{"entityType": "TEXTSELECTION", "labels": [{"label_name": "PER"}], "textSpan": {"offset": 6, "length": 12}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "MISC"}], "textSpan": {"offset": 88, "length": 16}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "ORG"}], "textSpan": {"offset": 145, "length": 16}}]}]}
{"sourceDetails": {"text": "Budjana also owns a double neck Klein guitar which was used on his latest album , Home , and pictured on the CD cover ."}, "annotations": [{"entities": [{"entityType": "TEXTSELECTION", "labels": [{"label_name": "PER"}], "textSpan": {"offset": 0, "length": 7}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "ORG"}], "textSpan": {"offset": 32, "length": 5}}, {"entityType": "TEXTSELECTION", "labels": [{"label_name": "MISC"}], "textSpan": {"offset": 82, "length": 4}}]}]}
Formato de importación de Label Studio
[{'data': {'text': 'His current band , Gigi , was formed in 1994 with Baron ( guitar ) , Thomas ( bass ) , Armand ( vocal ) and Ronald ( drum ) .'},
  'predictions': [{'result': [{'value': {'start': 19,
       'end': 23,
       'text': 'Gigi',
       'labels': ['ORG']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 50, 'end': 55, 'text': 'Baron', 'labels': ['PER']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 69, 'end': 75, 'text': 'Thomas', 'labels': ['PER']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 87, 'end': 93, 'text': 'Armand', 'labels': ['PER']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 108, 'end': 114, 'text': 'Ronald', 'labels': ['PER']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'}]}]},
 {'data': {'text': 'Since then , Gigi has released six albums .'},
  'predictions': [{'result': [{'value': {'start': 13,
       'end': 17,
       'text': 'Gigi',
       'labels': ['ORG']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'}]}]},
 {'data': {'text': 'He has also released his own solo albums : Nusa Damai ; Gitarku ; Samsara ; and Home , a tribute album to the December 2004 tsunami victims .'},
  'predictions': [{'result': [{'value': {'start': 43,
       'end': 53,
       'text': 'Nusa Damai',
       'labels': ['MISC']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 56, 'end': 63, 'text': 'Gitarku', 'labels': ['MISC']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 66, 'end': 73, 'text': 'Samsara', 'labels': ['MISC']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 80, 'end': 84, 'text': 'Home', 'labels': ['MISC']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'}]}]},
 {'data': {'text': 'Since Dewa Budjana started his professional career as a musician , he has mainly used a Parker Fly Delux as his main guitar , occasionally using Klein and Gibson guitars SG series instead .'},
  'predictions': [{'result': [{'value': {'start': 6,
       'end': 18,
       'text': 'Dewa Budjana',
       'labels': ['PER']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 88,
       'end': 104,
       'text': 'Parker Fly Delux',
       'labels': ['MISC']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'},
     {'value': {'start': 145,
       'end': 161,
       'text': 'Klein and Gibson',
       'labels': ['ORG']},
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels'}]}]}]
Formato de exportación de Label Studio (JSON sin formato)
[{'id': 141,
  'annotations': [{'id': 42,
    'completed_by': 4,
    'result': [{'value': {'start': 19,
       'end': 23,
       'text': 'Gigi',
       'labels': ['ORG']},
      'id': '467b95b0',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 50, 'end': 55, 'text': 'Baron', 'labels': ['PER']},
      'id': '8f12fa43',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 69, 'end': 75, 'text': 'Thomas', 'labels': ['PER']},
      'id': '936f63f0',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 87, 'end': 93, 'text': 'Armand', 'labels': ['PER']},
      'id': 'f02d2509',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 108, 'end': 114, 'text': 'Ronald', 'labels': ['PER']},
      'id': 'cb479df1',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T16:47:10.630009Z',
    'updated_at': '2025-01-19T16:47:10.630009Z',
    'draft_created_at': None,
    'lead_time': 0,
    'prediction': {'id': 95,
     'result': [{'value': {'start': 19,
        'end': 23,
        'text': 'Gigi',
        'labels': ['ORG']},
       'id': '467b95b0',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 50, 'end': 55, 'text': 'Baron', 'labels': ['PER']},
       'id': '8f12fa43',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 69, 'end': 75, 'text': 'Thomas', 'labels': ['PER']},
       'id': '936f63f0',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 87, 'end': 93, 'text': 'Armand', 'labels': ['PER']},
       'id': 'f02d2509',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 108,
        'end': 114,
        'text': 'Ronald',
        'labels': ['PER']},
       'id': 'cb479df1',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'}],
     'model_version': 'undefined',
     'created_ago': '0\xa0minutes',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T16:47:10.630009Z',
     'updated_at': '2025-01-19T16:47:10.630009Z',
     'model': None,
     'model_run': None,
     'task': 141,
     'project': 14},
    'result_count': 5,
    'unique_id': '54a60d2b-6960-48c9-baae-bf24b15ce273',
    'import_id': None,
    'last_action': None,
    'task': 141,
    'project': 14,
    'updated_by': 4,
    'parent_prediction': 95,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': 'example_file.json',
  'drafts': [],
  'predictions': [95],
  'data': {'text': 'His current band , Gigi , was formed in 1994 with Baron ( guitar ) , Thomas ( bass ) , Armand ( vocal ) and Ronald ( drum ) .'},
  'meta': {},
  'created_at': '2025-01-19T16:47:10.630009Z',
  'updated_at': '2025-01-19T16:47:10.630009Z',
  'inner_id': 1,
  'total_annotations': 1,
  'cancelled_annotations': 0,
  'total_predictions': 1,
  'comment_count': 0,
  'unresolved_comment_count': 0,
  'last_comment_updated_at': None,
  'project': 14,
  'updated_by': 4,
  'comment_authors': []},
 {'id': 141,
  'annotations': [{'id': 60,
    'completed_by': 4,
    'result': [{'value': {'start': 13,
       'end': 17,
       'text': 'Gigi',
       'labels': ['ORG']},
      'id': 'ce521973',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T16:47:10.630104Z',
    'updated_at': '2025-01-19T16:47:10.630104Z',
    'draft_created_at': None,
    'lead_time': 0,
    'prediction': {'id': 95,
     'result': [{'value': {'start': 13,
        'end': 17,
        'text': 'Gigi',
        'labels': ['ORG']},
       'id': 'ce521973',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'}],
     'model_version': 'undefined',
     'created_ago': '0\xa0minutes',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T16:47:10.630104Z',
     'updated_at': '2025-01-19T16:47:10.630104Z',
     'model': None,
     'model_run': None,
     'task': 141,
     'project': 14},
    'result_count': 1,
    'unique_id': '4a4b5631-4b51-4a09-99c4-ad8927074503',
    'import_id': None,
    'last_action': None,
    'task': 141,
    'project': 14,
    'updated_by': 4,
    'parent_prediction': 95,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': 'example_file.json',
  'drafts': [],
  'predictions': [95],
  'data': {'text': 'Since then , Gigi has released six albums .'},
  'meta': {},
  'created_at': '2025-01-19T16:47:10.630104Z',
  'updated_at': '2025-01-19T16:47:10.630104Z',
  'inner_id': 1,
  'total_annotations': 1,
  'cancelled_annotations': 0,
  'total_predictions': 1,
  'comment_count': 0,
  'unresolved_comment_count': 0,
  'last_comment_updated_at': None,
  'project': 14,
  'updated_by': 4,
  'comment_authors': []},
 {'id': 141,
  'annotations': [{'id': 96,
    'completed_by': 4,
    'result': [{'value': {'start': 43,
       'end': 53,
       'text': 'Nusa Damai',
       'labels': ['MISC']},
      'id': 'a013849e',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 56, 'end': 63, 'text': 'Gitarku', 'labels': ['MISC']},
      'id': '57423aa5',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 66, 'end': 73, 'text': 'Samsara', 'labels': ['MISC']},
      'id': 'eeee84bb',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 80, 'end': 84, 'text': 'Home', 'labels': ['MISC']},
      'id': '3df96d57',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T16:47:10.630126Z',
    'updated_at': '2025-01-19T16:47:10.630126Z',
    'draft_created_at': None,
    'lead_time': 0,
    'prediction': {'id': 95,
     'result': [{'value': {'start': 43,
        'end': 53,
        'text': 'Nusa Damai',
        'labels': ['MISC']},
       'id': 'a013849e',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 56,
        'end': 63,
        'text': 'Gitarku',
        'labels': ['MISC']},
       'id': '57423aa5',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 66,
        'end': 73,
        'text': 'Samsara',
        'labels': ['MISC']},
       'id': 'eeee84bb',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 80, 'end': 84, 'text': 'Home', 'labels': ['MISC']},
       'id': '3df96d57',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'}],
     'model_version': 'undefined',
     'created_ago': '0\xa0minutes',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T16:47:10.630126Z',
     'updated_at': '2025-01-19T16:47:10.630126Z',
     'model': None,
     'model_run': None,
     'task': 141,
     'project': 14},
    'result_count': 4,
    'unique_id': 'd2b8ec9a-cc62-4881-a42b-833546df7953',
    'import_id': None,
    'last_action': None,
    'task': 141,
    'project': 14,
    'updated_by': 4,
    'parent_prediction': 95,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': 'example_file.json',
  'drafts': [],
  'predictions': [95],
  'data': {'text': 'He has also released his own solo albums : Nusa Damai ; Gitarku ; Samsara ; and Home , a tribute album to the December 2004 tsunami victims .'},
  'meta': {},
  'created_at': '2025-01-19T16:47:10.630126Z',
  'updated_at': '2025-01-19T16:47:10.630126Z',
  'inner_id': 1,
  'total_annotations': 1,
  'cancelled_annotations': 0,
  'total_predictions': 1,
  'comment_count': 0,
  'unresolved_comment_count': 0,
  'last_comment_updated_at': None,
  'project': 14,
  'updated_by': 4,
  'comment_authors': []},
 {'id': 141,
  'annotations': [{'id': 7,
    'completed_by': 4,
    'result': [{'value': {'start': 6,
       'end': 18,
       'text': 'Dewa Budjana',
       'labels': ['PER']},
      'id': '2d675f56',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 88,
       'end': 104,
       'text': 'Parker Fly Delux',
       'labels': ['MISC']},
      'id': '13f6536b',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'},
     {'value': {'start': 145,
       'end': 161,
       'text': 'Klein and Gibson',
       'labels': ['ORG']},
      'id': '283b9e25',
      'from_name': 'label',
      'to_name': 'text',
      'type': 'labels',
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T16:47:10.630165Z',
    'updated_at': '2025-01-19T16:47:10.630165Z',
    'draft_created_at': None,
    'lead_time': 0,
    'prediction': {'id': 95,
     'result': [{'value': {'start': 6,
        'end': 18,
        'text': 'Dewa Budjana',
        'labels': ['PER']},
       'id': '2d675f56',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 88,
        'end': 104,
        'text': 'Parker Fly Delux',
        'labels': ['MISC']},
       'id': '13f6536b',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'},
      {'value': {'start': 145,
        'end': 161,
        'text': 'Klein and Gibson',
        'labels': ['ORG']},
       'id': '283b9e25',
       'from_name': 'label',
       'to_name': 'text',
       'type': 'labels',
       'origin': 'prediction'}],
     'model_version': 'undefined',
     'created_ago': '0\xa0minutes',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T16:47:10.630165Z',
     'updated_at': '2025-01-19T16:47:10.630165Z',
     'model': None,
     'model_run': None,
     'task': 141,
     'project': 14},
    'result_count': 3,
    'unique_id': '21115ec1-4a96-43c4-b1ec-1734cd004160',
    'import_id': None,
    'last_action': None,
    'task': 141,
    'project': 14,
    'updated_by': 4,
    'parent_prediction': 95,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': 'example_file.json',
  'drafts': [],
  'predictions': [95],
  'data': {'text': 'Since Dewa Budjana started his professional career as a musician , he has mainly used a Parker Fly Delux as his main guitar , occasionally using Klein and Gibson guitars SG series instead .'},
  'meta': {},
  'created_at': '2025-01-19T16:47:10.630165Z',
  'updated_at': '2025-01-19T16:47:10.630165Z',
  'inner_id': 1,
  'total_annotations': 1,
  'cancelled_annotations': 0,
  'total_predictions': 1,
  'comment_count': 0,
  'unresolved_comment_count': 0,
  'last_comment_updated_at': None,
  'project': 14,
  'updated_by': 4,
  'comment_authors': []}]

TXTC personalizado (CTXTC) - Clasificación de texto

Los conjuntos de datos CTXTC admiten varios formatos para ofrecer flexibilidad en las anotaciones y el procesamiento:
  • Formato de etiquetado de datos
  • Formato CSV
  • Formato de Label Studio
Formato de etiquetado de datos
Formato portablejsonl de CTXTC
Formato CSV
Formato CSV CTXTC - ETIQUETA ÚNICA
Formato CSV CTXTC - ETIQUETA MULTI
Formato de importación de Label Studio
Formato CSV CTXTC - ETIQUETA ÚNICA
Formato CSV CTXTC - ETIQUETA MULTI
[{'data': {'text': "I need to book a hotel in the east that has 4 stars. I can help you with that. What is your price range? That doesn't matter as long as it has free wifi and parking. If you'd like something cheap, I recommend the Allenbell. For something moderately priced, I would recommend the Warkworth House. Could you book the Wartworth for one night, 1 person? What day will you be staying? Friday and Can you book it for me and get a reference number ? Booking was successful. Reference number is : BMUKPTG6.  Can I help you with anything else today? I am looking to book a train that is leaving from Cambridge to Bishops Stortford on Friday. There are a number of trains leaving throughout the day.  What time would you like to travel? I want to get there by 19:45 at the latest. Okay! The latest train you can take leaves at 17:29, and arrives by 18:07. Would you like for me to book that for you? Yes please. I also need the travel time, departure time, and price. Reference number is : UIFV8FAS. The price is 10.1 GBP and the trip will take about 38 minutes. May I be of any other assistance? Yes. Sorry, but suddenly my plans changed. Can you change the Wartworth booking to Monday for 3 people and 4 nights? I have made that change and your reference number is YF86GE4J. Thank you very much, goodbye. You're welcome. Have a nice day!"},
  'predictions': [{'model_version': '1.3',
    'result': [{'id': '0',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['hotel', 'train']}}]}]},
 {'data': {'text': 'Howdy, I need a train heading into cambridge. I would be happy to help you find a train.  Where are you departing from? I am departing from norwich.  I need to leave after 18:45 on Wednesday. I have several options for you. Where is your destination? I will be heading to cabridge. The earliest after 18:45 is the TR8658, leaving Norwich at 19:16. Can I reserve you one or more seats on this train? yeah, i need one ticket. Booking was successful, the total fee is 17.6 GBP payable at the station. Your reference number is AXH1NM1I. Do you need assistance with anything else? I am also looking for a multi sport in the East. It looks like there is The Cherry Hinton Village Centre.  Can I get you more information about it? I would like to get the phone number, please. their phone number is 01223576412. anything else? Oh, and what is their postcode, please? Sure, the postcode is cb19ej. Can I help you find any other information about Cambridge? That is all for now thank you. enjoy your time in Cambridge!'},
  'predictions': [{'model_version': '1.3',
    'result': [{'id': '1',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['attraction', 'train']}}]}]},
 {'data': {'text': "What can you tell me about the Riverside Brasserie? It is a restaurant that serves modern european food near the centre of town. It is moderately priced. The phone number is 01223259988. Perfect. Can you help me with a reservation for 6 people at 14:30 this coming sunday? And please make sure I have a confirmation number to use. Your reservation is set! The table will be reserved for 15 minutes. Your reference number is LZLUDTVI. Is there anything else you need? I'm also looking for a place to stay.  In the south preferably. What price range were you thinking? No particular price range, but I would like it to be a 4 star hotel. There are no hotels that fit your criteria in the South, but there are two Guesthouses. Would you like to book one of those? Sure, that will work. Can you tell me more about them? Aylesbray Lodge Guesthouse and Rosa's Bed and Breakfast, both are rated at 4 stars and both include free parking and internet. Would you like a reservation for one of them? Can I get the postcode for both of them? Aylesbray postcode is cd17sr and Rosa's postcode is cb22ha. Is there anything else I can help you with today? No thanks. That's all the help I need. Take care. Bye. thank you! Enjoy your stay!"},
  'predictions': [{'model_version': '1.3',
    'result': [{'id': '2',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['hotel', 'restaurant']}}]}]},
 {'data': {'text': "I am looking for a specific hotel, its name is express by holiday inn cambridge. I have the Express by Holiday Inn Cambridge located on 15-17 norman way, coldhams business park. Their phone number is 01223866800. Would you like to know anything else? Yes, could you book the hotel room for me for 7 people? Yes, of course. What day would you like to stay? Monday, please. There will be 7 of us and we'd like to stay for 4 days. Here is the booking information:Booking was successful. Reference number is : 5F8G6J1G. Thank you. I would also like to book a train, please. Sure, which stations will you be traveling between? I will be going from cambridge to birmingham new street. What time would you like to leave?  The trains depart every hour. Whenever will get me there by 17:30. I do need to leave on Friday and I will need the travel time please. There are 11 results. Would you prefer the earliest departure time or latest?"},
  'predictions': [{'model_version': '1.3',
    'result': [{'id': '3',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['hotel', 'train']}}]}]}]
Formato de exportación de Label Studio
Formato de exportación de CTXTC LABEL Studio - ETIQUETA ÚNICA
Formato de exportación de CTXTC LABEL Studio - ETIQUETA MULTI
[{'id': 237,
  'annotations': [{'id': 110,
    'completed_by': 4,
    'result': [{'id': '0',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['hotel', 'train']},
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T19:15:56.447771Z',
    'updated_at': '2025-01-19T19:15:57.447771Z',
    'draft_created_at': None,
    'lead_time': 1.101,
    'prediction': {'id': 253,
     'result': [{'id': '0',
       'from_name': 'textClassification',
       'to_name': 'text',
       'type': 'choices',
       'value': {'choices': ['hotel', 'train']}}],
     'model_version': '1.3',
     'created_ago': '1\xa0minute',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T19:15:56.447771Z',
     'updated_at': '2025-01-19T19:15:57.447771Z',
     'model': None,
     'model_run': None,
     'task': 237,
     'project': 22},
    'result_count': 1,
    'unique_id': '6aa5eb24-4c5b-4e7a-9770-eca304ee1420',
    'import_id': None,
    'last_action': None,
    'task': 237,
    'project': 22,
    'updated_by': 4,
    'parent_prediction': 253,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': '0606dc5a-multiLabel_ethos_train_LabelStudio_Upload.json',
  'drafts': [],
  'predictions': [253],
  'data': {'text': "I need to book a hotel in the east that has 4 stars. I can help you with that. What is your price range? That doesn't matter as long as it has free wifi and parking. If you'd like something cheap, I recommend the Allenbell. For something moderately priced, I would recommend the Warkworth House. Could you book the Wartworth for one night, 1 person? What day will you be staying? Friday and Can you book it for me and get a reference number ? Booking was successful. Reference number is : BMUKPTG6.  Can I help you with anything else today? I am looking to book a train that is leaving from Cambridge to Bishops Stortford on Friday. There are a number of trains leaving throughout the day.  What time would you like to travel? I want to get there by 19:45 at the latest. Okay! The latest train you can take leaves at 17:29, and arrives by 18:07. Would you like for me to book that for you? Yes please. I also need the travel time, departure time, and price. Reference number is : UIFV8FAS. The price is 10.1 GBP and the trip will take about 38 minutes. May I be of any other assistance? Yes. Sorry, but suddenly my plans changed. Can you change the Wartworth booking to Monday for 3 people and 4 nights? I have made that change and your reference number is YF86GE4J. Thank you very much, goodbye. You're welcome. Have a nice day!"},
  'meta': {},
  'created_at': '2025-01-19T19:15:56.447771Z',
  'updated_at': '2025-01-19T19:15:57.447771Z',
  'inner_id': 2,
  'total_annotations': 1,
  'cancelled_annotations': 0,
  'total_predictions': 1,
  'comment_count': 0,
  'unresolved_comment_count': 0,
  'last_comment_updated_at': None,
  'project': 22,
  'updated_by': 4,
  'comment_authors': []},
 {'id': 237,
  'annotations': [{'id': 110,
    'completed_by': 4,
    'result': [{'id': '1',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['attraction', 'train']},
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T19:15:56.447847Z',
    'updated_at': '2025-01-19T19:15:57.447847Z',
    'draft_created_at': None,
    'lead_time': 1.101,
    'prediction': {'id': 253,
     'result': [{'id': '1',
       'from_name': 'textClassification',
       'to_name': 'text',
       'type': 'choices',
       'value': {'choices': ['attraction', 'train']}}],
     'model_version': '1.3',
     'created_ago': '1\xa0minute',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T19:15:56.447847Z',
     'updated_at': '2025-01-19T19:15:57.447847Z',
     'model': None,
     'model_run': None,
     'task': 237,
     'project': 22},
    'result_count': 1,
    'unique_id': 'adfa5ebc-bd45-4c9a-9f7e-1e797329dd85',
    'import_id': None,
    'last_action': None,
    'task': 237,
    'project': 22,
    'updated_by': 4,
    'parent_prediction': 253,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': '0606dc5a-multiLabel_ethos_train_LabelStudio_Upload.json',
  'drafts': [],
  'predictions': [253],
  'data': {'text': 'Howdy, I need a train heading into cambridge. I would be happy to help you find a train.  Where are you departing from? I am departing from norwich.  I need to leave after 18:45 on Wednesday. I have several options for you. Where is your destination? I will be heading to cabridge. The earliest after 18:45 is the TR8658, leaving Norwich at 19:16. Can I reserve you one or more seats on this train? yeah, i need one ticket. Booking was successful, the total fee is 17.6 GBP payable at the station. Your reference number is AXH1NM1I. Do you need assistance with anything else? I am also looking for a multi sport in the East. It looks like there is The Cherry Hinton Village Centre.  Can I get you more information about it? I would like to get the phone number, please. their phone number is 01223576412. anything else? Oh, and what is their postcode, please? Sure, the postcode is cb19ej. Can I help you find any other information about Cambridge? That is all for now thank you. enjoy your time in Cambridge!'},
  'meta': {},
  'created_at': '2025-01-19T19:15:56.447847Z',
  'updated_at': '2025-01-19T19:15:57.447847Z',
  'inner_id': 2,
  'total_annotations': 1,
  'cancelled_annotations': 0,
  'total_predictions': 1,
  'comment_count': 0,
  'unresolved_comment_count': 0,
  'last_comment_updated_at': None,
  'project': 22,
  'updated_by': 4,
  'comment_authors': []},
 {'id': 237,
  'annotations': [{'id': 110,
    'completed_by': 4,
    'result': [{'id': '2',
      'from_name': 'textClassification',
      'to_name': 'text',
      'type': 'choices',
      'value': {'choices': ['hotel', 'restaurant']},
      'origin': 'prediction'}],
    'was_cancelled': False,
    'ground_truth': False,
    'created_at': '2025-01-19T19:15:56.447870Z',
    'updated_at': '2025-01-19T19:15:57.447870Z',
    'draft_created_at': None,
    'lead_time': 1.101,
    'prediction': {'id': 253,
     'result': [{'id': '2',
       'from_name': 'textClassification',
       'to_name': 'text',
       'type': 'choices',
       'value': {'choices': ['hotel', 'restaurant']}}],
     'model_version': '1.3',
     'created_ago': '1\xa0minute',
     'score': None,
     'cluster': None,
     'neighbors': None,
     'mislabeling': 0.0,
     'created_at': '2025-01-19T19:15:56.447870Z',
     'updated_at': '2025-01-19T19:15:57.447870Z',
     'model': None,
     'model_run': None,
     'task': 237,
     'project': 22},
    'result_count': 1,
    'unique_id': '2b0c6f75-82a7-4a0f-a5d5-a5a697ef6798',
    'import_id': None,
    'last_action': None,
    'task': 237,
    'project': 22,
    'updated_by': 4,
    'parent_prediction': 253,
    'parent_annotation': None,
    'last_created_by': None}],
  'file_upload': '0606dc5a-multiLabel_ethos_train_LabelStudio_Upload.json',
  'drafts': [],
  'predictions': [253]]

Scripts de conversión - CNER

Script: convertir formato de etiquetado de datos a formato de Label Studio
Se proporciona un script de Python denominado cner_export_to_LS.py para convertir el formato JSONL de Data Labeling en formatos de importación y exportación de Label Studio: 
import os
import argparse
import json
import uuid
from datetime import datetime
import random
import glob
 
 
def convert_to_label_studio_import(data):
    """ Convert portable JSONL format to Label Studio import format. """
    dd = []
    for i in data[1:]:
        ents = i['annotations'][0]['entities']
        ee = []
        text = i['sourceDetails']['text']
        for e in ents:
            start = e['textSpan']['offset']
            end = e['textSpan']['offset'] + e['textSpan']['length']
            ee.append({
                'value': {
                    'start': start,
                    'end': end,
                    'text': text[start:end],
                    'labels': [e['labels'][0]['label_name']]
                },
                'from_name': 'label',
                'to_name': 'text',
                'type': 'labels'
            })
        dd.append({
            'data': {'text': i['sourceDetails']['text']},
            'predictions': [{'result': ee}]
        })
    return dd
 
 
def convert_to_label_studio_raw_export(input_data):
    """ Convert portable JSONL format to Label Studio raw JSON export format. """
    # Extract text and annotations
    text = input_data['sourceDetails']['text']
    entities = input_data['annotations'][0]['entities']
     
    # Initialize variables
    task_id = 141
    project_id = 14
    now = datetime.now().isoformat() + "Z"
    unique_id = str(uuid.uuid4())
     
    # Process entities
    results = []
    for entity in entities:
        offset = entity['textSpan']['offset']
        length = entity['textSpan']['length']
        label_name = entity['labels'][0]['label_name']
        entity_text = text[offset:offset + length]
         
        result = {
            'value': {
                'start': offset,
                'end': offset + length,
                'text': entity_text,
                'labels': [label_name]
            },
            'id': str(uuid.uuid4())[:8],
            'from_name': 'label',
            'to_name': 'text',
            'type': 'labels',
            'origin': 'prediction'
        }
        results.append(result)
     
    # Construct the final output
    output = {
        'id': task_id,
        'annotations': [{
            'id': random.randint(1, 100),
            'completed_by': 4,
            'result': results,
            'was_cancelled': False,
            'ground_truth': False,
            'created_at': now,
            'updated_at': now,
            'draft_created_at': None,
            'lead_time': 0,
            'prediction': {
                'id': 95,
                'result': results,
                'model_version': 'undefined',
                'created_ago': '0\xa0minutes',
                'score': None,
                'cluster': None,
                'neighbors': None,
                'mislabeling': 0.0,
                'created_at': now,
                'updated_at': now,
                'model': None,
                'model_run': None,
                'task': task_id,
                'project': project_id
            },
            'result_count': len(results),
            'unique_id': unique_id,
            'import_id': None,
            'last_action': None,
            'task': task_id,
            'project': project_id,
            'updated_by': 4,
            'parent_prediction': 95,
            'parent_annotation': None,
            'last_created_by': None
        }],
        'file_upload': 'example_file.json',
        'drafts': [],
        'predictions': [95],
        'data': {'text': text},
        'meta': {},
        'created_at': now,
        'updated_at': now,
        'inner_id': 1,
        'total_annotations': 1,
        'cancelled_annotations': 0,
        'total_predictions': 1,
        'comment_count': 0,
        'unresolved_comment_count': 0,
        'last_comment_updated_at': None,
        'project': project_id,
        'updated_by': 4,
        'comment_authors': []
    }
     
    return output
 
 
def load_jsonl(file_path):
    """ Load a JSONL file and return a list of parsed JSON objects. """
    with open(file_path, 'r', encoding='utf-8') as f:
        return [json.loads(line) for line in f]
 
 
def save_jsonl(data, file_path):
    """ Save a list of JSON objects to a JSONL file. """
    with open(file_path, 'w', encoding='utf-8') as f:
        for entry in data:
            f.write(json.dumps(entry) + '\n')
 
 
def save_json(data, file_path):
    """ Save a JSON object to a JSON file. """
    with open(file_path, 'w', encoding='utf-8') as f:
        json.dump(data, f, indent=4)
 
 
def process_folder(input_folder, output_folder):
    """ Process all test, train, and dev files in the folder. """
    for split in ["test", "train", "dev"]:
        input_file = os.path.join(input_folder, f"{split}.jsonl")
        if not os.path.exists(input_file):
            print(f"Skipping {input_file}: File not found.")
            continue
 
        # Load input data
        data = load_jsonl(input_file)
         
        # Convert and save import format
        import_output_file = os.path.join(output_folder, f"{split}_import.jsonl")
        import_data = convert_to_label_studio_import(data)
        save_jsonl(import_data, import_output_file)
        print(f"Saved Label Studio import data to {import_output_file}")
         
        # Convert and save export format
        export_output_file = os.path.join(output_folder, f"{split}_export.json")
        export_data = [convert_to_label_studio_raw_export(item) for item in data]
        save_json(export_data, export_output_file)
        print(f"Saved Label Studio export data to {export_output_file}")
 
 
if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Convert JSONL files to Label Studio formats.")
    parser.add_argument("input_folder", type=str, help="Folder containing input JSONL files (test, train, dev).")
    parser.add_argument("output_folder", type=str, help="Folder to save converted files.")
    args = parser.parse_args()
 
    # Ensure output folder exists
    os.makedirs(args.output_folder, exist_ok=True)
 
    # Process input folder
    process_folder(args.input_folder, args.output_folder)

Instrucciones de uso

  1. Coloque todos los archivos JSONL de Data Labeling (prueba, entrenamiento, desarrollo) en un directorio de entrada.
  2. Ejecute el script, cner_conversion.ipynb: 
    import glob
 
 
def list_folders_in_directory(directory_path):
    """List all folders in a given directory using glob."""
    folder_paths = glob.glob(os.path.join(directory_path, "*/"))
    # folder_names = [os.path.basename(os.path.normpath(folder)) for folder in folder_paths]
    return folder_paths
 
 
def list_files_in_directory(directory_path, extension="*"):
    """List all files in a directory with an optional extension filter."""
    file_paths = glob.glob(os.path.join(directory_path, f"*.{extension}"))
    # file_names = [os.path.basename(file) for file in file_paths]
    return file_paths
 
directory_path = "/home/niksoni/cner_labelstudio_integeration/datasets/cner"
output_folder = "/home/niksoni/cner_labelstudio_integeration/datasets/cner/label_studio_formats"
folders = list_folders_in_directory(directory_path)
for folder in folders:
    folder_name = os.path.basename(os.path.normpath(folder))
    output_folder_path = f"{output_folder}/{folder_name}"
    os.makedirs(output_folder_path, exist_ok=True)
    files = list_files_in_directory(folder)
    for input_file in files:
        if not os.path.exists(input_file):
                    print(f"Skipping {input_file}: File not found.")
                    continue 
        data = load_jsonl(input_file)
        # Convert and save import format
        file_name = os.path.basename(input_file)
        import_output_file = os.path.join(output_folder_path, f"{file_name}_LS_import.json")
        import_data = convert_to_label_studio_import(data)
        save_json(import_data, import_output_file)
        print(f"Saved Label Studio import data to {import_output_file}")
         
        export_output_file = os.path.join(output_folder_path, f"{file_name}_LS_export.json")
        export_data = [convert_to_label_studio_raw_export(item) for item in data[1:]]
        save_json(export_data, export_output_file)
        print(f"Saved Label Studio export data to {export_output_file}")

    El script genera:

    • *_import.jsonl para importación directa en Label Studio
    • *_export.json para uso directo en flujos de trabajo de formación
    Después de la conversión, la carpeta de salida contiene contenido con las siguientes estructuras:
    /input_folder/
│── test.jsonl
│── train.jsonl
│── dev.jsonl

/output_folder/
│── test_import.json
│── test_export.json
│── train_import.json
│── train_export.json
│── dev_import.json
│── dev_export.json

Scripts de conversión - CTXTC

Script: convertir formato CSV a formato de Label Studio
Se proporciona un script de Python denominado ctxtc_export_to_LS.py para convertir el formato CSV en formatos de importación y exportación de Label Studio:
import os
import argparse
import json
import uuid
# from datetime import datetime
import random
import glob
from datetime import datetime, timedelta
 
import pandas as pd
 
def load_jsonl(file_path):
    """ Load a JSONL file and return a list of parsed JSON objects. """
    with open(file_path, 'r', encoding='utf-8') as f:
        return [json.loads(line) for line in f]
 
 
def save_jsonl(data, file_path):
    """ Save a list of JSON objects to a JSONL file. """
    with open(file_path, 'w', encoding='utf-8') as f:
        for entry in data:
            f.write(json.dumps(entry) + '\n')
             
             
def save_to_json(data, file_path):
    with open(file_path, 'w') as f:
        json.dump(data, f, indent=4)
    print(f"Data successfully saved to {file_path}")
 
def load_from_json(file_path):
    with open(file_path, 'r') as f:
        data = json.load(f)
    print(f"Data successfully loaded from {file_path}")
    return data
 
 
# Function to convert dataframe to Label Studio format
def convert_to_label_studio(df, label_name="textClassification"):
    data = []
     
    for idx, row in df.iterrows():
        text = row['text']
        label = str(row['labels']).split("|")
         
        data_entry = {
            "data": {
                "text": text
            },
            "predictions": [
                {
                    "model_version": "1.3",  # This can be adjusted if needed
                    "result": [
                        {
                            "id": str(idx),
                            "from_name": label_name,  # Label name changed here
                            "to_name": "text",
                            "type": "choices",
                            "value": {
#                                 "score": 1.0,  # You can adjust the score based on confidence if needed
                                "choices": label
                            }
                        }
                    ]
                }
            ]
        }
        data.append(data_entry)
     
    return json.dumps(data, indent=4)
 
 
 
 
def convert_prediction_to_label_studio_export_format(input_data):
    # Generate placeholders
    current_time = datetime.utcnow()
    created_at = current_time.isoformat() + "Z"
    updated_at = (current_time + timedelta(seconds=1)).isoformat() + "Z"
    prediction_id = 253
    task_id = 237
    project_id = 22
    annotation_id = 110
    completed_by = 4
 
    # Extract values from input
    text = input_data['data']['text']
    predictions = input_data['predictions'][0]
    model_version = predictions['model_version']
    result = predictions['result']
 
    # Build the output structure
    converted_data = {
        'id': task_id,
        'annotations': [{
            'id': annotation_id,
            'completed_by': completed_by,
            'result': [
                {
                    **item,
                    'origin': 'prediction'
                }
                for item in result
            ],
            'was_cancelled': False,
            'ground_truth': False,
            'created_at': created_at,
            'updated_at': updated_at,
            'draft_created_at': None,
            'lead_time': 1.101,
            'prediction': {
                'id': prediction_id,
                'result': result,
                'model_version': model_version,
                'created_ago': '1\xa0minute',
                'score': None,
                'cluster': None,
                'neighbors': None,
                'mislabeling': 0.0,
                'created_at': created_at,
                'updated_at': updated_at,
                'model': None,
                'model_run': None,
                'task': task_id,
                'project': project_id
            },
            'result_count': 1,
            'unique_id': str(uuid.uuid4()),
            'import_id': None,
            'last_action': None,
            'task': task_id,
            'project': project_id,
            'updated_by': completed_by,
            'parent_prediction': prediction_id,
            'parent_annotation': None,
            'last_created_by': None
        }],
        'file_upload': '0606dc5a-multiLabel_ethos_train_LabelStudio_Upload.json',
        'drafts': [],
        'predictions': [prediction_id],
        'data': {
            'text': text
        },
        'meta': {},
        'created_at': created_at,
        'updated_at': updated_at,
        'inner_id': 2,
        'total_annotations': 1,
        'cancelled_annotations': 0,
        'total_predictions': 1,
        'comment_count': 0,
        'unresolved_comment_count': 0,
        'last_comment_updated_at': None,
        'project': project_id,
        'updated_by': completed_by,
        'comment_authors': []
    }
    return converted_data

Instrucciones de uso

  1. Coloque todos los archivos JSONL de Data Labeling (prueba, entrenamiento, desarrollo) en un directorio de entrada.
  2. Ejecute el script, ctxtc_conversion.ipynb: 
    import json
from ctxtc_conversion_to_LS import *
import glob
 
def list_folders_in_directory(directory_path):
    """List all folders in a given directory using glob."""
    folder_paths = glob.glob(os.path.join(directory_path, "*/"))
    # folder_names = [os.path.basename(os.path.normpath(folder)) for folder in folder_paths]
    return folder_paths
 
 
def list_files_in_directory(directory_path, extension="*"):
    """List all files in a directory with an optional extension filter."""
    file_paths = glob.glob(os.path.join(directory_path, f"*.{extension}"))
    # file_names = [os.path.basename(file) for file in file_paths]
    return file_paths
 
# cls_types = ["singleLabel", "multilabel"]
cls_types = ["multilabel"]
directory_path = "/home/niksoni/cner_labelstudio_integeration/datasets/ctxtc"
output_folder = "/home/niksoni/cner_labelstudio_integeration/datasets/ctxtc/label_studio_formats"
for cls_type in cls_types:
    cls_directory_path = f"{directory_path}/{cls_type}"
    folders = list_folders_in_directory(cls_directory_path)
    for folder in folders:
        folder_name = os.path.basename(os.path.normpath(folder))
        output_folder_path = f"{output_folder}/{cls_type}/{folder_name}"
        os.makedirs(output_folder_path, exist_ok=True)
        files = list_files_in_directory(folder)
        # print(files)
        for input_file in files:
            if not os.path.exists(input_file):
                        print(f"Skipping {input_file}: File not found.")
                        continue 
            df = pd.read_csv(input_file)
            json_data = convert_to_label_studio(df)
            json_data = json.loads(json_data)
            file_name = os.path.basename(input_file)
            # save_to_json(json_data,file_name)
            import_output_file = os.path.join(output_folder_path, f"{file_name}_LS_import.json")
            # import_data = convert_to_label_studio(data)
            save_to_json(json_data, import_output_file)
            print(f"Saved Label Studio import data to {import_output_file}")
 
            export_output_file = os.path.join(output_folder_path, f"{file_name}_LS_export.json")
            export_data = [convert_prediction_to_label_studio_export_format(item) for item in json_data]
            save_to_json(export_data, export_output_file)
            print(f"Saved Label Studio export data to {export_output_file}")

    El script genera:

    • *_import.jsonl para importación directa en Label Studio
    • *_export.json para uso directo en flujos de trabajo de formación
    Después de la conversión, la carpeta de salida contiene contenido con las siguientes estructuras:
    /input_folder/
│── test.jsonl
│── train.jsonl
│── dev.jsonl

/output_folder/
│── test_import.json
│── test_export.json
│── train_import.json
│── train_export.json
│── dev_import.json
│── dev_export.json

Integración con el idioma

Siga este flujo de formación con los datos convertidos:
  1. Cargue un juego de datos, anotado manualmente mediante el estudio de etiquetas o convertido desde Data Labeling.
  2. Seleccione el juego de datos almacenado en Object Storage para el entrenamiento del modelo.
    Language procesa el conjunto de datos y continúa el pipeline de formación.

    El modelo se valida, entrena y despliega de la misma manera que antes.

Conclusión

  • El script de migración garantiza una transición perfecta de Data Labeling a Label Studio.
  • Puede continuar anotando datos existentes en Label Studio.
  • El conjunto de datos exportado es totalmente compatible con los pipelines de formación de idioma.
  • El formato JSONL existente sigue siendo compatible con versiones anteriores.
Para obtener más información, consulte la documentación de Label Studio: Consulte también lo siguiente: