EMR Auto-Tune with a Dedicated Role

Auto-Tune on EMR (on EC2) reads your cluster and resizes it. If your job's role must stay minimal, you can give these permissions to a separate role. The definity agent then uses that role only for its own EMR calls.

You don't share any credentials with definity.

How it works

1. The job runs as its usual role, called the base role. This is the cluster's instance profile role, or the step's runtime role (--execution-role-arn).
2. You set spark.definity.aws.roleArn to a second role, the definity role.
3. When the agent needs AWS (Auto-Tune, cluster info), it asks AWS STS to act as the definity role.
4. AWS allows this only if the base role may assume the definity role, and the definity role trusts the base role.
5. Every change shows in CloudTrail as assumed-role/<definity role>/definity-agent.

Why a separate role

• The job's role stays small. The job's own code gets no EMR permissions, only permission to assume one role.
• Clear audit. Every cluster change by definity shows in CloudTrail under the definity role.
• Easy to control. Limit the definity role to some clusters, or delete it to turn Auto-Tune off, without touching your job roles.
• One role for many jobs. Many runtime roles can share one definity role.

Setup (one time)

1. Create the definity role

Create a role, for example definity-autotune, with this permissions policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "elasticmapreduce:Describe*",
        "elasticmapreduce:List*",
        "elasticmapreduce:GetManagedScalingPolicy",
        "elasticmapreduce:ModifyInstanceGroups",
        "elasticmapreduce:ModifyInstanceFleet",
        "elasticmapreduce:AddInstanceGroups",
        "elasticmapreduce:AddInstanceFleet",
        "elasticmapreduce:PutManagedScalingPolicy",
        "elasticmapreduce:RemoveManagedScalingPolicy",
        "elasticmapreduce:PutAutoScalingPolicy",
        "elasticmapreduce:RemoveAutoScalingPolicy"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:DescribeTags",
      "Resource": "*"
    }
  ]
}

To limit the role to your clusters, set the EMR statement's Resource to arn:aws:elasticmapreduce:<region>:<account>:cluster/*, or add a tag condition.

Actions	Used for
Describe*, List*, GetManagedScalingPolicy, ec2:DescribeTags	Finding the cluster and reading its setup
ModifyInstanceGroups, ModifyInstanceFleet, AddInstanceGroups, AddInstanceFleet	Resizing the cluster and adding task nodes
PutManagedScalingPolicy, RemoveManagedScalingPolicy	Changing managed scaling
PutAutoScalingPolicy, RemoveAutoScalingPolicy	Changing custom autoscaling (the cluster must have an --auto-scaling-role)

2. Trust the base role

In the definity role's trust policy, allow the base role to assume it:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<account>:role/<instance profile role>",
          "arn:aws:iam::<account>:role/<runtime role>"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

List the instance profile role for clusters without runtime roles. List each runtime role that runs steps monitored by definity.

3. Let the base role assume the definity role

Add this policy to the same base role, or roles, named in step 2:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::<account>:role/definity-autotune"
    }
  ]
}

Nothing else is needed. On clusters with runtime roles, steps 2 and 3 name the runtime role, and the instance profile needs no change.

4. Configure the agent

Set the role on the job, as a Spark conf or as a definity server override:

spark.definity.aws.roleArn=arn:aws:iam::<account>:role/definity-autotune

Notes

• No secrets. You share only the role ARN. The agent uses temporary STS credentials.
• No external ID or sts:TagSession. The role is assumed from inside your own account.