Amazon SageMaker Service

2023/06/02 - Amazon SageMaker Service - 4 updated api methods

Changes  This release adds Selective Execution feature that allows SageMaker Pipelines users to run selected steps in a pipeline.

DescribePipelineExecution (updated) Link ¶
Changes (response)
{'SelectiveExecutionConfig': {'SelectedSteps': [{'StepName': 'string'}],
                              'SourcePipelineExecutionArn': 'string'}}

Describes the details of a pipeline execution.

See also: AWS API Documentation

Request Syntax

client.describe_pipeline_execution(
    PipelineExecutionArn='string'
)
type PipelineExecutionArn

string

param PipelineExecutionArn

[REQUIRED]

The Amazon Resource Name (ARN) of the pipeline execution.

rtype

dict

returns

Response Syntax

{
    'PipelineArn': 'string',
    'PipelineExecutionArn': 'string',
    'PipelineExecutionDisplayName': 'string',
    'PipelineExecutionStatus': 'Executing'|'Stopping'|'Stopped'|'Failed'|'Succeeded',
    'PipelineExecutionDescription': 'string',
    'PipelineExperimentConfig': {
        'ExperimentName': 'string',
        'TrialName': 'string'
    },
    'FailureReason': 'string',
    'CreationTime': datetime(2015, 1, 1),
    'LastModifiedTime': datetime(2015, 1, 1),
    'CreatedBy': {
        'UserProfileArn': 'string',
        'UserProfileName': 'string',
        'DomainId': 'string',
        'IamIdentity': {
            'Arn': 'string',
            'PrincipalId': 'string',
            'SourceIdentity': 'string'
        }
    },
    'LastModifiedBy': {
        'UserProfileArn': 'string',
        'UserProfileName': 'string',
        'DomainId': 'string',
        'IamIdentity': {
            'Arn': 'string',
            'PrincipalId': 'string',
            'SourceIdentity': 'string'
        }
    },
    'ParallelismConfiguration': {
        'MaxParallelExecutionSteps': 123
    },
    'SelectiveExecutionConfig': {
        'SourcePipelineExecutionArn': 'string',
        'SelectedSteps': [
            {
                'StepName': 'string'
            },
        ]
    }
}

Response Structure

  • (dict) --

    • PipelineArn (string) --

      The Amazon Resource Name (ARN) of the pipeline.

    • PipelineExecutionArn (string) --

      The Amazon Resource Name (ARN) of the pipeline execution.

    • PipelineExecutionDisplayName (string) --

      The display name of the pipeline execution.

    • PipelineExecutionStatus (string) --

      The status of the pipeline execution.

    • PipelineExecutionDescription (string) --

      The description of the pipeline execution.

    • PipelineExperimentConfig (dict) --

      Specifies the names of the experiment and trial created by a pipeline.

      • ExperimentName (string) --

        The name of the experiment.

      • TrialName (string) --

        The name of the trial.

    • FailureReason (string) --

      If the execution failed, a message describing why.

    • CreationTime (datetime) --

      The time when the pipeline execution was created.

    • LastModifiedTime (datetime) --

      The time when the pipeline execution was modified last.

    • CreatedBy (dict) --

      Information about the user who created or modified an experiment, trial, trial component, lineage group, project, or model card.

      • UserProfileArn (string) --

        The Amazon Resource Name (ARN) of the user's profile.

      • UserProfileName (string) --

        The name of the user's profile.

      • DomainId (string) --

        The domain associated with the user.

      • IamIdentity (dict) --

        The IAM Identity details associated with the user. These details are associated with model package groups, model packages, and project entities only.

        • Arn (string) --

          The Amazon Resource Name (ARN) of the IAM identity.

        • PrincipalId (string) --

          The ID of the principal that assumes the IAM identity.

        • SourceIdentity (string) --

          The person or application which assumes the IAM identity.

    • LastModifiedBy (dict) --

      Information about the user who created or modified an experiment, trial, trial component, lineage group, project, or model card.

      • UserProfileArn (string) --

        The Amazon Resource Name (ARN) of the user's profile.

      • UserProfileName (string) --

        The name of the user's profile.

      • DomainId (string) --

        The domain associated with the user.

      • IamIdentity (dict) --

        The IAM Identity details associated with the user. These details are associated with model package groups, model packages, and project entities only.

        • Arn (string) --

          The Amazon Resource Name (ARN) of the IAM identity.

        • PrincipalId (string) --

          The ID of the principal that assumes the IAM identity.

        • SourceIdentity (string) --

          The person or application which assumes the IAM identity.

    • ParallelismConfiguration (dict) --

      The parallelism configuration applied to the pipeline.

      • MaxParallelExecutionSteps (integer) --

        The max number of steps that can be executed in parallel.

    • SelectiveExecutionConfig (dict) --

      The selective execution configuration applied to the pipeline run.

      • SourcePipelineExecutionArn (string) --

        The ARN from a reference execution of the current pipeline. Used to copy input collaterals needed for the selected steps to run. The execution status of the pipeline can be either Failed or Success .

      • SelectedSteps (list) --

        A list of pipeline steps to run. All step(s) in all path(s) between two selected steps should be included.

        • (dict) --

          A step selected to run in selective execution mode.

          • StepName (string) --

            The name of the pipeline step.

ListPipelineExecutionSteps (updated) Link ¶
Changes (response)
{'PipelineExecutionSteps': {'SelectiveExecutionResult': {'SourcePipelineExecutionArn': 'string'}}}

Gets a list of PipeLineExecutionStep objects.

See also: AWS API Documentation

Request Syntax

client.list_pipeline_execution_steps(
    PipelineExecutionArn='string',
    NextToken='string',
    MaxResults=123,
    SortOrder='Ascending'|'Descending'
)
type PipelineExecutionArn

string

param PipelineExecutionArn

The Amazon Resource Name (ARN) of the pipeline execution.

type NextToken

string

param NextToken

If the result of the previous ListPipelineExecutionSteps request was truncated, the response includes a NextToken . To retrieve the next set of pipeline execution steps, use the token in the next request.

type MaxResults

integer

param MaxResults

The maximum number of pipeline execution steps to return in the response.

type SortOrder

string

param SortOrder

The field by which to sort results. The default is CreatedTime .

rtype

dict

returns

Response Syntax

{
    'PipelineExecutionSteps': [
        {
            'StepName': 'string',
            'StepDisplayName': 'string',
            'StepDescription': 'string',
            'StartTime': datetime(2015, 1, 1),
            'EndTime': datetime(2015, 1, 1),
            'StepStatus': 'Starting'|'Executing'|'Stopping'|'Stopped'|'Failed'|'Succeeded',
            'CacheHitResult': {
                'SourcePipelineExecutionArn': 'string'
            },
            'AttemptCount': 123,
            'FailureReason': 'string',
            'Metadata': {
                'TrainingJob': {
                    'Arn': 'string'
                },
                'ProcessingJob': {
                    'Arn': 'string'
                },
                'TransformJob': {
                    'Arn': 'string'
                },
                'TuningJob': {
                    'Arn': 'string'
                },
                'Model': {
                    'Arn': 'string'
                },
                'RegisterModel': {
                    'Arn': 'string'
                },
                'Condition': {
                    'Outcome': 'True'|'False'
                },
                'Callback': {
                    'CallbackToken': 'string',
                    'SqsQueueUrl': 'string',
                    'OutputParameters': [
                        {
                            'Name': 'string',
                            'Value': 'string'
                        },
                    ]
                },
                'Lambda': {
                    'Arn': 'string',
                    'OutputParameters': [
                        {
                            'Name': 'string',
                            'Value': 'string'
                        },
                    ]
                },
                'QualityCheck': {
                    'CheckType': 'string',
                    'BaselineUsedForDriftCheckStatistics': 'string',
                    'BaselineUsedForDriftCheckConstraints': 'string',
                    'CalculatedBaselineStatistics': 'string',
                    'CalculatedBaselineConstraints': 'string',
                    'ModelPackageGroupName': 'string',
                    'ViolationReport': 'string',
                    'CheckJobArn': 'string',
                    'SkipCheck': True|False,
                    'RegisterNewBaseline': True|False
                },
                'ClarifyCheck': {
                    'CheckType': 'string',
                    'BaselineUsedForDriftCheckConstraints': 'string',
                    'CalculatedBaselineConstraints': 'string',
                    'ModelPackageGroupName': 'string',
                    'ViolationReport': 'string',
                    'CheckJobArn': 'string',
                    'SkipCheck': True|False,
                    'RegisterNewBaseline': True|False
                },
                'EMR': {
                    'ClusterId': 'string',
                    'StepId': 'string',
                    'StepName': 'string',
                    'LogFilePath': 'string'
                },
                'Fail': {
                    'ErrorMessage': 'string'
                },
                'AutoMLJob': {
                    'Arn': 'string'
                }
            },
            'SelectiveExecutionResult': {
                'SourcePipelineExecutionArn': 'string'
            }
        },
    ],
    'NextToken': 'string'
}

Response Structure

  • (dict) --

    • PipelineExecutionSteps (list) --

      A list of PipeLineExecutionStep objects. Each PipeLineExecutionStep consists of StepName, StartTime, EndTime, StepStatus, and Metadata. Metadata is an object with properties for each job that contains relevant information about the job created by the step.

      • (dict) --

        An execution of a step in a pipeline.

        • StepName (string) --

          The name of the step that is executed.

        • StepDisplayName (string) --

          The display name of the step.

        • StepDescription (string) --

          The description of the step.

        • StartTime (datetime) --

          The time that the step started executing.

        • EndTime (datetime) --

          The time that the step stopped executing.

        • StepStatus (string) --

          The status of the step execution.

        • CacheHitResult (dict) --

          If this pipeline execution step was cached, details on the cache hit.

          • SourcePipelineExecutionArn (string) --

            The Amazon Resource Name (ARN) of the pipeline execution.

        • AttemptCount (integer) --

          The current attempt of the execution step. For more information, see Retry Policy for SageMaker Pipelines steps.

        • FailureReason (string) --

          The reason why the step failed execution. This is only returned if the step failed its execution.

        • Metadata (dict) --

          Metadata to run the pipeline step.

          • TrainingJob (dict) --

            The Amazon Resource Name (ARN) of the training job that was run by this step execution.

            • Arn (string) --

              The Amazon Resource Name (ARN) of the training job that was run by this step execution.

          • ProcessingJob (dict) --

            The Amazon Resource Name (ARN) of the processing job that was run by this step execution.

            • Arn (string) --

              The Amazon Resource Name (ARN) of the processing job.

          • TransformJob (dict) --

            The Amazon Resource Name (ARN) of the transform job that was run by this step execution.

            • Arn (string) --

              The Amazon Resource Name (ARN) of the transform job that was run by this step execution.

          • TuningJob (dict) --

            The Amazon Resource Name (ARN) of the tuning job that was run by this step execution.

            • Arn (string) --

              The Amazon Resource Name (ARN) of the tuning job that was run by this step execution.

          • Model (dict) --

            The Amazon Resource Name (ARN) of the model that was created by this step execution.

            • Arn (string) --

              The Amazon Resource Name (ARN) of the created model.

          • RegisterModel (dict) --

            The Amazon Resource Name (ARN) of the model package that the model was registered to by this step execution.

            • Arn (string) --

              The Amazon Resource Name (ARN) of the model package.

          • Condition (dict) --

            The outcome of the condition evaluation that was run by this step execution.

            • Outcome (string) --

              The outcome of the Condition step evaluation.

          • Callback (dict) --

            The URL of the Amazon SQS queue used by this step execution, the pipeline generated token, and a list of output parameters.

            • CallbackToken (string) --

              The pipeline generated token from the Amazon SQS queue.

            • SqsQueueUrl (string) --

              The URL of the Amazon Simple Queue Service (Amazon SQS) queue used by the callback step.

            • OutputParameters (list) --

              A list of the output parameters of the callback step.

              • (dict) --

                An output parameter of a pipeline step.

                • Name (string) --

                  The name of the output parameter.

                • Value (string) --

                  The value of the output parameter.

          • Lambda (dict) --

            The Amazon Resource Name (ARN) of the Lambda function that was run by this step execution and a list of output parameters.

            • Arn (string) --

              The Amazon Resource Name (ARN) of the Lambda function that was run by this step execution.

            • OutputParameters (list) --

              A list of the output parameters of the Lambda step.

              • (dict) --

                An output parameter of a pipeline step.

                • Name (string) --

                  The name of the output parameter.

                • Value (string) --

                  The value of the output parameter.

          • QualityCheck (dict) --

            The configurations and outcomes of the check step execution. This includes:

            • The type of the check conducted.

            • The Amazon S3 URIs of baseline constraints and statistics files to be used for the drift check.

            • The Amazon S3 URIs of newly calculated baseline constraints and statistics.

            • The model package group name provided.

            • The Amazon S3 URI of the violation report if violations detected.

            • The Amazon Resource Name (ARN) of check processing job initiated by the step execution.

            • The Boolean flags indicating if the drift check is skipped.

            • If step property BaselineUsedForDriftCheck is set the same as CalculatedBaseline .

            • CheckType (string) --

              The type of the Quality check step.

            • BaselineUsedForDriftCheckStatistics (string) --

              The Amazon S3 URI of the baseline statistics file used for the drift check.

            • BaselineUsedForDriftCheckConstraints (string) --

              The Amazon S3 URI of the baseline constraints file used for the drift check.

            • CalculatedBaselineStatistics (string) --

              The Amazon S3 URI of the newly calculated baseline statistics file.

            • CalculatedBaselineConstraints (string) --

              The Amazon S3 URI of the newly calculated baseline constraints file.

            • ModelPackageGroupName (string) --

              The model package group name.

            • ViolationReport (string) --

              The Amazon S3 URI of violation report if violations are detected.

            • CheckJobArn (string) --

              The Amazon Resource Name (ARN) of the Quality check processing job that was run by this step execution.

            • SkipCheck (boolean) --

              This flag indicates if the drift check against the previous baseline will be skipped or not. If it is set to False , the previous baseline of the configured check type must be available.

            • RegisterNewBaseline (boolean) --

              This flag indicates if a newly calculated baseline can be accessed through step properties BaselineUsedForDriftCheckConstraints and BaselineUsedForDriftCheckStatistics . If it is set to False , the previous baseline of the configured check type must also be available. These can be accessed through the BaselineUsedForDriftCheckConstraints and BaselineUsedForDriftCheckStatistics properties.

          • ClarifyCheck (dict) --

            Container for the metadata for a Clarify check step. The configurations and outcomes of the check step execution. This includes:

            • The type of the check conducted,

            • The Amazon S3 URIs of baseline constraints and statistics files to be used for the drift check.

            • The Amazon S3 URIs of newly calculated baseline constraints and statistics.

            • The model package group name provided.

            • The Amazon S3 URI of the violation report if violations detected.

            • The Amazon Resource Name (ARN) of check processing job initiated by the step execution.

            • The boolean flags indicating if the drift check is skipped.

            • If step property BaselineUsedForDriftCheck is set the same as CalculatedBaseline .

            • CheckType (string) --

              The type of the Clarify Check step

            • BaselineUsedForDriftCheckConstraints (string) --

              The Amazon S3 URI of baseline constraints file to be used for the drift check.

            • CalculatedBaselineConstraints (string) --

              The Amazon S3 URI of the newly calculated baseline constraints file.

            • ModelPackageGroupName (string) --

              The model package group name.

            • ViolationReport (string) --

              The Amazon S3 URI of the violation report if violations are detected.

            • CheckJobArn (string) --

              The Amazon Resource Name (ARN) of the check processing job that was run by this step's execution.

            • SkipCheck (boolean) --

              This flag indicates if the drift check against the previous baseline will be skipped or not. If it is set to False , the previous baseline of the configured check type must be available.

            • RegisterNewBaseline (boolean) --

              This flag indicates if a newly calculated baseline can be accessed through step properties BaselineUsedForDriftCheckConstraints and BaselineUsedForDriftCheckStatistics . If it is set to False , the previous baseline of the configured check type must also be available. These can be accessed through the BaselineUsedForDriftCheckConstraints property.

          • EMR (dict) --

            The configurations and outcomes of an Amazon EMR step execution.

            • ClusterId (string) --

              The identifier of the EMR cluster.

            • StepId (string) --

              The identifier of the EMR cluster step.

            • StepName (string) --

              The name of the EMR cluster step.

            • LogFilePath (string) --

              The path to the log file where the cluster step's failure root cause is recorded.

          • Fail (dict) --

            The configurations and outcomes of a Fail step execution.

            • ErrorMessage (string) --

              A message that you define and then is processed and rendered by the Fail step when the error occurs.

          • AutoMLJob (dict) --

            The Amazon Resource Name (ARN) of the AutoML job that was run by this step.

            • Arn (string) --

              The Amazon Resource Name (ARN) of the AutoML job.

        • SelectiveExecutionResult (dict) --

          The ARN from an execution of the current pipeline from which results are reused for this step.

          • SourcePipelineExecutionArn (string) --

            The ARN from an execution of the current pipeline.

    • NextToken (string) --

      If the result of the previous ListPipelineExecutionSteps request was truncated, the response includes a NextToken . To retrieve the next set of pipeline execution steps, use the token in the next request.

StartPipelineExecution (updated) Link ¶
Changes (request)
{'SelectiveExecutionConfig': {'SelectedSteps': [{'StepName': 'string'}],
                              'SourcePipelineExecutionArn': 'string'}}

Starts a pipeline execution.

See also: AWS API Documentation

Request Syntax

client.start_pipeline_execution(
    PipelineName='string',
    PipelineExecutionDisplayName='string',
    PipelineParameters=[
        {
            'Name': 'string',
            'Value': 'string'
        },
    ],
    PipelineExecutionDescription='string',
    ClientRequestToken='string',
    ParallelismConfiguration={
        'MaxParallelExecutionSteps': 123
    },
    SelectiveExecutionConfig={
        'SourcePipelineExecutionArn': 'string',
        'SelectedSteps': [
            {
                'StepName': 'string'
            },
        ]
    }
)
type PipelineName

string

param PipelineName

[REQUIRED]

The name of the pipeline.

type PipelineExecutionDisplayName

string

param PipelineExecutionDisplayName

The display name of the pipeline execution.

type PipelineParameters

list

param PipelineParameters

Contains a list of pipeline parameters. This list can be empty.

  • (dict) --

    Assigns a value to a named Pipeline parameter.

    • Name (string) -- [REQUIRED]

      The name of the parameter to assign a value to. This parameter name must match a named parameter in the pipeline definition.

    • Value (string) -- [REQUIRED]

      The literal value for the parameter.

type PipelineExecutionDescription

string

param PipelineExecutionDescription

The description of the pipeline execution.

type ClientRequestToken

string

param ClientRequestToken

[REQUIRED]

A unique, case-sensitive identifier that you provide to ensure the idempotency of the operation. An idempotent operation completes no more than once.

This field is autopopulated if not provided.

type ParallelismConfiguration

dict

param ParallelismConfiguration

This configuration, if specified, overrides the parallelism configuration of the parent pipeline for this specific run.

  • MaxParallelExecutionSteps (integer) -- [REQUIRED]

    The max number of steps that can be executed in parallel.

type SelectiveExecutionConfig

dict

param SelectiveExecutionConfig

The selective execution configuration applied to the pipeline run.

  • SourcePipelineExecutionArn (string) -- [REQUIRED]

    The ARN from a reference execution of the current pipeline. Used to copy input collaterals needed for the selected steps to run. The execution status of the pipeline can be either Failed or Success .

  • SelectedSteps (list) -- [REQUIRED]

    A list of pipeline steps to run. All step(s) in all path(s) between two selected steps should be included.

    • (dict) --

      A step selected to run in selective execution mode.

      • StepName (string) -- [REQUIRED]

        The name of the pipeline step.

rtype

dict

returns

Response Syntax

{
    'PipelineExecutionArn': 'string'
}

Response Structure

  • (dict) --

    • PipelineExecutionArn (string) --

      The Amazon Resource Name (ARN) of the pipeline execution.