mlm_insights.mlm_spark.execution_engine package

Submodules

mlm_insights.mlm_spark.execution_engine.spark_engine module

class mlm_insights.mlm_spark.execution_engine.spark_engine.SparkEngine

Bases: ExecutionEngine

Class for Spark execution engine functionality.

parse_data_frame_result(profile_dataframe: DataFrame) → Profile

Method to parse the profile dataframe based on different Execution engines As Spark converts to a bytearray whereas Dask provides a byte string. We need to handle this distinction between different Execution engines

Parameters

profile_dataframe: DataFrame

Pandas profile dataframe

Returns

Profile

Contains data summary and includes profile header, information about the features, metrics, SFCs.