Pachyderm Worker
Learn about the concept of a Pachyderm worker.
December 4, 2023
About #
HPE ML Data Management workers are kubernetes pods that run the docker image (your user code) specified in the pipeline specification. When you create a pipeline, HPE ML Data Management spins up workers that continuously run in the cluster, waiting for new data to process.
Each datum goes through the following processing phases inside a HPE ML Data Management worker pod:
| Phase | Description |
|---|---|
| Downloading | The HPE ML Data Management worker pod downloads the datum contents into HPE ML Data Management. |
| Processing | The HPE ML Data Management worker pod runs the contents of the datum against your code. |
| Uploading | The HPE ML Data Management worker pod uploads the results of processing into an output repository. |
