Run Commands

Developer Workflow

Learn how to manage and process data in your CI workflow.

April 4, 2024

In general, the developer workflow for HPE ML Data Management involves adding data to versioned data repositories, creating pipelines to read from those repositories, executing the pipeline’s code, and writing the pipeline’s output to other data repositories. Both the data and pipeline can be iterated on independently with HPE ML Data Management handling the code execution according to the pipeline specfication. The workflow steps are shown below.

Developer workflow

Data Workflow #

Adding data to HPE ML Data Management is the first step towards building data-driven pipelines. There are multiple ways to add data to a HPE ML Data Management repository:

For more information, see Load Your Data Into HPE ML Data Management.

Pipeline Workflow #

The fundamental concepts of HPE ML Data Management are very powerful, but the manual build steps mentioned in the pipeline workflow can become cumbersome during rapid-iteration development cycles. We’ve created a few helpful developer workflows and tools to automate steps that are error-prone or repetitive: