Part IV Data Engineering
The section explains the methods for developing your data in AI Data Platform.
Data engineers focus on building and maintaining the systems that data analysts use to access and manipulate data. They use big data technologies like Apache Spark and programming languages including Python and SQL to process and manage data located in object storage, databases, and data warehouses. They are responsible for the initial stages of the data analytics and data science workflow, such as collecting, storing, and transforming data. Their work ensures that the data is accessible and is of high quality so that other data scientists and analysts can use it for their work. Data Engineers also use CI/CD principles for data pipelines and code to manage version control and promote collaboration with data scientists, analysts, and other stakeholders.
Topics: