mlm_insights.core.execution_engine.interfaces package¶
Submodules¶
mlm_insights.core.execution_engine.interfaces.execute_engine module¶
- class mlm_insights.core.execution_engine.interfaces.execute_engine.ExecutionEngine(engine_type: str)¶
Bases:
ABCAbstract Base Class for Execution engine functionality.
This can be implemented for different execution engines like ‘dask’, ‘spark’ for engine specific configurations.
- classmethod create_client(engine_detail: EngineDetail, **kwargs: Any) Any¶
Factory Method to create engine specific client.
Parameters¶
- engine_detail: EngineDetail
Engine Detail object can hold Any type like ‘dask’, ‘spark’ as string
Returns¶
- Any
Engine specific Client
- engine_type: str = 'native'¶
- get_schema_provider(input_schema: Dict[str, FeatureType]) SchemaProvider¶
Method to convert engine specific schema using user schema.
Parameters¶
- input_schema: Dict[str, FeatureType]
key-value pair
key: attribute name from data set
value: attribute data type and variable type
Returns¶
- SchemaProvider
Engine specific schema
- parse_data_frame_result(profile_dataframe: DataFrame) Profile¶
Method to parse the profile dataframe based on different Execution engines.
As Spark converts to a bytearray whereas Dask provides a byte string, we need to handle this distinction between different Execution engines.
Parameters¶
- profile_dataframe: DataFrame
pandas profile dataframe
Returns¶
- Profile
Contains data summary and includes profile header, information about the features, metrics, SFCs.