Global Config
You can set global defaults for your cluster that are passed down to all pipeline specs. These defaults provide a consistent experience for your data scientists and help manage your cluster.
Before You Start #
- You must have a running HPE Machine Learning Data Management cluster
- You should be familiar with the Pachyderm Pipeline Specification (PPS)
Cluster Defaults #
Set Global Cluster Defaults #
pachctl update defaults --cluster <<EOF
{
"createPipelineRequest": {
"resourceRequests": {
"cpu": 1,
"memory": "256Mi",
"disk": "1Gi"
},
"resourceLimits": {
"cpu": 1,
"memory": "256Mi",
"disk": "1Gi"
}
}
}
EOF
- Log in to Console
- Navigate to the Project Dashboard homepage.
- Select Cluster Defaults.
- Input your desired default Pipeline Specification (PPS) settings in JSON format.
{ "createPipelineRequest": { "resourceRequests": { "cpu": 1, "memory": "256Mi", "disk": "1Gi" }, "sidecarResourceRequests": { "cpu": 1, "memory": "256Mi", "disk": "10Gi" } } }
- Optionally select the Prettify button to make the JSON more readable.
- Select Continue.
- Choose from one of the following options:
- Save Cluster Defaults
- Only new and edited pipelines apply the new defaults
- No pipeline regeneration or datum reprocessing occurs
- Save Cluster Defaults and Regenerate Pipelines
- Only pipelines with impacted specs are regenerated
- Previously processed datums are not reprocessed
- Save Cluster Defaults, Regenerate Pipelines, and Reprocess Datums
- Only pipelines with impacted specs are regenerated
- Previously processed datums are reprocessed
- Save Cluster Defaults
- Select Save.
Note
--regenerate
flag is set.View Global Cluster Defaults #
- Open a terminal window.
- Input the following command:
pachctl inspect defaults --cluster | jq
Project Defaults #
Set Project-Level Defaults #
pachctl update defaults --project <<EOF
{
"createPipelineRequest": {
"resourceRequests": {
"cpu": 1,
"memory": "256Mi",
"disk": "1Gi"
},
"resourceLimits": {
"cpu": 1,
"memory": "256Mi",
"disk": "1Gi"
}
}
}
EOF
- Log in to Console
- Navigate to the Project Dashboard homepage.
- Scroll to the project you wish to add defaults to and select View Project.
- Select Project Defaults from the sidebar.
- Input your desired default Pipeline Specification (PPS) settings in JSON format.
{ "createPipelineRequest": { "resourceRequests": { "cpu": 1, "memory": "256Mi", "disk": "1Gi" }, "sidecarResourceRequests": { "cpu": 1, "memory": "256Mi", "disk": "10Gi" } } }
- Optionally select the Prettify button to make the JSON more readable.
- Select Continue.
- Choose from one of the following options:
- Save Project Defaults
- Only new and edited pipelines apply the new defaults
- No pipeline regeneration or datum reprocessing occurs
- Save Project Defaults and Regenerate Pipelines
- Only pipelines with impacted specs are regenerated
- Previously processed datums are not reprocessed
- Save Project Defaults, Regenerate Pipelines, and Reprocess Datums
- Only pipelines with impacted specs are regenerated
- Previously processed datums are reprocessed
- Save Project Defaults
- Select Save.
Note
--regenerate
flag is set.View Project-Level Defaults #
- Open a terminal window.
- Input the following command:
pachctl inspect defaults --project | jq
Override Cluster or Project Defaults #
Users can override global cluster or project defaults in their pipeline specs when necessary. For example, let’s say that you set a global default for resource requests to look like the following:
{
"resourceRequests": {
"cpu": 1,
"memory": "256Mi",
"disk": "1Gi",
},
}
A user could disable needing any CPU by setting the following in their pipeline spec:
{
"resourceRequests": {
"cpu": null,
},
}
Any keys not specified in the pipeline spec will apply the global defaults in the following way: project > cluster.