mlm_insights.core.data_sources.filters package¶
Submodules¶
mlm_insights.core.data_sources.filters.contains_filter module¶
- class mlm_insights.core.data_sources.filters.contains_filter.ContainsFilter(**kwargs: Any)¶
Bases:
OCIObjectStorageDataSourceFilter
This class implements the OCIObjectStorageDataSourceFilter class for contains filter-type to search/filter out the list of file location which contains the filter value in file location .
- filter_files(files_location: List[str], filter_value: str, object_storage: Any) List[str] ¶
Parameters¶
- files_location :
List of files present on the file path in the oci object system
- filter_value :
A filter/search string if file name contains the string
- kwargs:
Key-value pair for dynamic arguments
Returns¶
- List[str]:
List of files present on the file path in the oci object system
mlm_insights.core.data_sources.filters.date_range_filter module¶
- class mlm_insights.core.data_sources.filters.date_range_filter.DateRangeFilter(**kwargs: Any)¶
Bases:
OCIObjectStorageDataSourceFilter
This class implements the OCIObjectStorageDataSourceFilter class for date range filter-type to search/filter out the list of file location which were last modified in between the provided date range . Two arguments are taken as input in this filter : start and end, start being older date than end.
- filter_files(files_location: List[str], filter_value: Any, object_storage: OCIFileSystem) List[str] ¶
Parameters¶
- files_location:
List of files to filter from
- filter_value:
dictionary containing the arguments for filters
- object_storage:
OCIFileSystem object
Returns¶
- List[str]:
List of files present on the file path in the oci object system
Raises¶
- DataSourceException: DataSourceException
Exception if the file list is empty
mlm_insights.core.data_sources.filters.file_extension_filter module¶
- class mlm_insights.core.data_sources.filters.file_extension_filter.FileExtensionFilter(**kwargs: Any)¶
Bases:
OCIObjectStorageDataSourceFilter
This class implements the OCIObjectStorageDataSourceFilter class for file extension filter-type to search/filter out the list of file location based on the file extension provided by user.
- filter_files(files_location: List[str], filter_value: str, object_storage: Any) List[str] ¶
Parameters¶
- files_location :
List of files present on the file path in the oci object system
- filter_value :
string argument for file extension filter
- object_storage:
OCIFileSystem object
Returns¶
- List[str]:
List of files present on the file path in the oci object system
mlm_insights.core.data_sources.filters.last_n_days_filter module¶
- class mlm_insights.core.data_sources.filters.last_n_days_filter.LastNDaysFilter(**kwargs: Any)¶
Bases:
OCIObjectStorageDataSourceFilter
This class implements the OCIObjectStorageDataSourceFilter class for last_n_days filter-type to search/filter out the list of file location modified in last n days , where n is the filter value provided by user .
- filter_files(files_location: List[str], filter_value: str, object_storage: OCIFileSystem) List[str] ¶
Parameters¶
- files_location:
List of files to filter from
- filter_value:
string argument for last n days filter
- object_storage:
OCIFileSystem object
Returns¶
- List[str]:
List of files present on the file path in the oci object system based on last n days filter type
mlm_insights.core.data_sources.filters.oci_object_storage_data_source_filter module¶
- class mlm_insights.core.data_sources.filters.oci_object_storage_data_source_filter.OCIObjectStorageDataSourceFilter(name: str = '', **kwargs: Any)¶
Bases:
ABC
Abstract Base Class for OCI Object Storage DataSource Filters . The child class must implement filter_files methods to write the filtering logic .
- abstract filter_files(files_location: List[str], filter_value: Any, object_storage: Any) List[str] ¶
Read and Return the List of files based on the filter_value and implemented filtering logic
Parameters¶
- files_location :
List of files present on the file path in the oci object system
- filter_value :
A string to filter/search the files
- kwargs:
Key-value pair for dynamic arguments
Returns¶
- List[str]:
List of files present on the file path in the oci object system
mlm_insights.core.data_sources.filters.partition_based_date_range_filter module¶
- class mlm_insights.core.data_sources.filters.partition_based_date_range_filter.PartitionBasedDateRangeFilter(**kwargs: Any)¶
Bases:
OCIObjectStorageDataSourceFilter
This class implements the OCIObjectStorageDataSourceFilter class for date range filter-type to search/filter out the list of file location partitioned on basis of date range. Two arguments are taken as input in this filter : start, end and date_format, start being older date than end and date format is the regex required to search the date in object location.
- filter_files(files_location: List[str], filter_value: Any, object_storage: OCIFileSystem) List[str] ¶
Parameters¶
- files_location:
List of files to filter from
- filter_value:
dictionary containing the arguments for filters
- object_storage:
OCIFileSystem object
Returns¶
- List[str]:
List of files present on the file path in the oci object system
Raises¶
- DataSourceException: DataSourceException
Exception if the file list is empty
mlm_insights.core.data_sources.filters.prefix_filter module¶
- class mlm_insights.core.data_sources.filters.prefix_filter.PrefixFilter(**kwargs: Any)¶
Bases:
OCIObjectStorageDataSourceFilter
This class implements the OCIObjectStorageDataSourceFilter class for prefix filter-type to search/filter out the list of file location which start with the filter value in file location .
- filter_files(files_location: List[str], filter_value: str, object_storage: Any) List[str] ¶
Parameters¶
- files_location :
List of files present on the file path in the oci object system
- filter_value :
A filter/search string if file name prefixes the string
Returns¶
- List[str]:
List of files present on the file path in the oci object system matching the filter
mlm_insights.core.data_sources.filters.suffix_filter module¶
- class mlm_insights.core.data_sources.filters.suffix_filter.SuffixFilter(**kwargs: Any)¶
Bases:
OCIObjectStorageDataSourceFilter
This class implements the OCIObjectStorageDataSourceFilter class for suffix filter-type to search/filter out the list of file location which end with the filter value in file location .
- filter_files(files_location: List[str], filter_value: str, object_storage: Any) List[str] ¶
Parameters¶
- files_location :
List of files present on the file path in the oci object system
- filter_value :
A filter/search string if file name suffixes the string
Returns¶
- List[str]:
List of files present on the file path in the oci object system matching the filter