MySQL HeatWave User Guide
See MySQL 9.4 Error Message Reference for Lakehouse error messages.
As of MySQL 9.2.2, a detailed summary of warnings is available when loading data to Lakehouse. This includes warnings related to the schema inference stage of loading data.
The summaries include the following:
Warnings for Lakehouse Load and
Lakehouse Schema Inference.
The number of warnings.
The code of each warning with a description.
If warning messages were filtered, the number of warnings that were not recorded due to the filters.
Refer to the following samples of warning summaries.
A warning summary for Lakehouse Load that
includes the number of filtered warnings that are not recorded
for the summary.
Warning 6434 [WARNINGS SUMMARY] Lakehouse Load had 27 warning(s) out of which 27 were not recorded (due to max_error_count limit or filtering rules) Warning 6434 [WARNINGS SUMMARY] 22 warnings with code: 6053(ER_LH_WARN_EXCEEDS_MAX_TRUNCATING) Warning 6434 [WARNINGS SUMMARY] 5 warnings with code: 6052(ER_LH_WARN_EXCEEDS_MIN_TRUNCATING)
A warning summary for Lakehouse Schema
Inference with Autopilot that includes the number of
filtered warnings that are not recorded for the summary.
{"message": "[WARNINGS SUMMARY] Lakehouse Schema Inference had 18 warning(s) out of which 1 were not recorded (due to max_error_count limit or filtering rules)", "table_name": "lineitem", "schema_name": "autoschema_tmp_db", "condition_no": 1}
{"message": "[WARNINGS SUMMARY] 16 warnings with code: 6062(ER_LH_COLUMN_MISMATCH_ERR)", "table_name": "lineitem", "schema_name": "autoschema_tmp_db", "condition_no": 2}
{"message": "[WARNINGS SUMMARY] 1 warnings with code: 6095(ER_LH_WARN_INFER_SKIPPED_LINES)", "table_name": "lineitem", "schema_name": "autoschema_tmp_db", "condition_no": 3}
{"message": "[WARNINGS SUMMARY] 1 warnings with code: 6063(ER_LH_COLUMN_MAX_ERR)", "table_name": "lineitem", "schema_name": "autoschema_tmp_db", "condition_no": 4}
As of MySQL 9.1.2, the following details are included in Lakehouse error and warning messages related to data files in the Object Storage bucket to help you understand them better:
Byte offset text message is added to the existing
error/warning messages in csv or JSON
file formats.
Error/warning position details are added to JSON data type
related errors/warnings in csv, JSON,
parquet or AVRO file formats.
Row number and character position details are added to
warning messages(Non strict mode) in csv
and JSON file formats.
Row number is added to warning messages(Non strict mode) in
parquet or AVRO file formats.
Refer to the following sample error messages, for more information.
Error message - Byte offset (17) is added to
csv or JSON error message.
ERROR HY000: Column 1 of {object: "mtrtest/datalake_integer.test_3.tbl" in
namespace: "OCI_OBJECT_STORE_SOURCE_NAMESPACE", bucket:
"OCI_OBJECT_STORE_SOURCE_BUCKET"} at byte offset 17 : Invalid integer value.
Warning message - Row number (13), character position (1),
and byte offset (66) are added to csv or
JSON warning message.
Warning 6057 Row 13, char 1, column 1 of {object: "mtrtest/datalake_int_basic/0.tbl" in
namespace: "OCI_OBJECT_STORE_SOURCE_NAMESPACE", bucket:
"OCI_OBJECT_STORE_SOURCE_BUCKET"} at byte offset 66 : Integer value was truncated.
Warning message - Position (11), character (12), are added
to csv, JSON, parquet
or AVRO warning message.
Warning 6121 Row 20, column 3 of {object:
"datalake_avro_json/datalake_json_error.avro.tbl" in namespace:
"OCI_OBJECT_STORE_SOURCE_NAMESPACE", bucket: "OCI_OBJECT_STORE_SOURCE_BUCKET"} :
Invalid escape character in string at position 11(char 12) in this column.
Some Lakehouse error and warning messages include the URL to the external object. The URL can contain up to 1024 characters, and exceed the error message limit of 512 characters. As of MySQL 9.0.1-u1, the truncated message will include a MySQL command to access the full error message. For example:
Error message:
ERROR HY000: Message was truncated, use
<SELECT data FROM performance_schema.error_log
WHERE thread_id=15 AND data LIKE 'ERR_LH_test.t1_long_name_6045%'
ORDER BY LOGGED DESC>
to obtain more detailed information about the warning/error
Warning message:
Warning 6045 Message was truncated, use
<SELECT data FROM performance_schema.error_log
WHERE thread_id=15 AND data LIKE 'WARN_LH_test.t1_long_name_6045_0%'
ORDER BY LOGGED DESC>
to obtain more detailed information about the warning/error
As of MySQL 9.0.1-u1, it is possible to filter out Lakehouse warning messages from the console, MySQL Shell and error log following a load command. The console Total Warnings will include both the displayed and the filtered warnings count. This does not filter error messages.
Use the
lakehouse_filter_warning_codes_list
session variable to filter warning messages by error code. For
example:
mysql> SET @@session.lakehouse_filter_warning_codes_list = '6017,5099,6018';
Use the
lakehouse_filter_warning_modes_list
session variable to filter warning messages by mode. For
example:
mysql> SET @@session.lakehouse_filter_warning_modes_list =
'NUMERIC_TRUNCATION,STRING_TRUNCATION,EMPTY_FILES,DUPLICATE_FILES';
The supported modes are:
Mode: AVRO_FILE_BLOCK_HEADER filters the
following warning messages:
ER_LH_AVRO_CANNOT_PARSE_HEADER
ER_LH_AVRO_HEADER_METADATA_ERR
ER_LH_FORMAT_HEADER_NO_MAGIC_BYTES
ER_LH_AVRO_HEADER_NO_SCHEMA
ER_LH_AVRO_INVALID_BLOCK_RECORD_COUNT
ER_LH_AVRO_INVALID_BLOCK_SIZE
ER_LH_AVRO_NO_CODEC_IN_HEADER
Mode: DUPLICATE_FILES filters the
following warning messages:
ER_LH_DUPLICATE_FILE
Mode: EMPTY_FILES filters the following
warning messages:
ER_LH_EMPTY_FILE
Mode: EMPTY_MISSING_COLUMNS filters the
following warning messages:
ER_LH_COLUMN_MISMATCH_ERR
ER_LH_COLUMN_MAX_ERR
ER_LH_WARN_COL_MISSING_NOT_NULLABLE
ER_LH_COL_IS_EMPTY_WARN
Mode: INFER_SKIPPED filters the following
warning messages:
ER_LH_INFER_FILE_HAS_NO_DATA
ER_LH_WARN_INFER_SKIPPED_FILES
ER_LH_WARN_INFER_SKIPPED_LINES
Mode: MISSING_FILES filters the following
warning messages:
ER_LH_NO_FILES_FOUND
Mode: NULL_COLUMNS filters the following
warning messages:
ER_LH_COL_NOT_NULLABLE
Mode: NUMERIC_PARSING for the
int,
decimal,
float, and
double data types filters the
following warning messages:
ER_LH_BAD_VALUE
ER_LH_DECIMAL_OOM_ERR
ER_LH_DECIMAL_UNKNOWN_ERR
ER_LH_PARQUET_DECIMAL_CONVERSION_ERR
ER_LH_REAL_IS_NAN
Mode: NUMERIC_TRUNCATION for the
int,
decimal,
float, and
double data types filters the
following warning messages:
ER_LH_DECIMAL_PRECISION_EXCEEDS_SCHEMA
ER_LH_EXCEEDS_MAX
ER_LH_EXCEEDS_MIN
ER_LH_OUT_OF_RANGE
ER_LH_WARN_DECIMAL_ROUNDING
ER_LH_WARN_EXCEEDS_MIN_TRUNCATING
ER_LH_WARN_EXCEEDS_MAX_TRUNCATING
ER_LH_WARN_TRUNCATED
Mode: STRING_PARSING for the
char,
enum,
text, and
varchar data types filters
the following warning messages:
ER_LH_BAD_VALUE
ER_LH_CANNOT_CONVERT_STRING
Mode: STRING_TRUNCATION for the
char,
text, and
varchar data types filters
the following warning messages:
ER_LH_STRING_TOO_LONG
Mode: TEMPORAL_PARSING for the
date,
datetime,
time, and
timestamp data types filters
the following warning messages:
ER_LH_BAD_VALUE
ER_LH_DATETIME_FORMAT
Mode: TEMPORAL_TRUNCATION for the
date,
datetime,
time, and
timestamp data types filters
the following warning messages:
ER_LH_WARN_TRUNCATED
Mode: PARQUET_SCHEMA filters the
following warning messages:
ER_LH_PARQUET_CANNOT_LOCATE_SCHEMA
ER_LH_PARQUET_SCHEMA_MISMATCH