Run Commands

Export via Egress

Learn how to push pipeline results to an external datastore using the egress pipeline spec attribute.

December 4, 2023

The egress field in the HPE ML Data Management pipeline specification enables you to push the results of a pipeline to an external datastore such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. After the user code has finished running, but before the job is marked as successful, HPE ML Data Management pushes the data to the specified destination.

ℹī¸

Make sure that your cluster has been configured to work with your object store.

Pick the egress protocol that applies to your storage:

Cloud PlatformProtocolExample
Google Cloud Storagegs://gs://gs-bucket/gs-dir
Amazon S3s3://s3://s3-endpoint/s3-bucket/s3-dir
Azure Blob Storagewasb://wasb://default-container@storage-account/az-dir

Example #

"egress": {
   "URL": "s3://bucket/dir"
},