Blogfoon

Blogfoon is a personal space for thoughtful notes on technology, coding, travel, and ordinary life.

  • Published:

Watchtower works well for lightweight asynchronous CD setups: it can watch your Docker images, pull newer versions, and restart containers automatically. But when the images live in a private Amazon ECR registry, a common problem appears quickly:

no basic auth credentials. Proceeding to next.

The issue is that ECR does not use a fixed username/password model. Authentication depends on temporary AWS credentials, so Watchtower needs extra help before it can pull from ECR reliably. The practical fix is to use amazon-ecr-credential-helper, Docker’s credential helper for ECR.

The general approach

The overall idea is already documented in Watchtower’s private registry guidance, but parts of that setup are outdated and need a small adjustment.

A workable setup looks like this:

  • build the amazon-ecr-credential-helper binary first
  • store that binary inside a Docker volume
  • mount the volume into the Watchtower container
  • provide a Docker config file that tells Docker to use the helper for ECR
  • pass the required AWS environment variables into Watchtower

That way, Watchtower can authenticate against ECR and pull updated images without relying on static Docker login credentials.

Step 1: Build the credential helper

Instead of installing the helper directly on the host, it can be compiled inside a temporary container and written into a shared Docker volume.

Create a Dockerfile with the following content:

FROM golang:1.24
ENV CGO_ENABLED=0
RUN go install github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login@latest
WORKDIR /go/bin/

This differs from the older example in the documentation. The image version used there is too old and may fail to install the helper correctly.

Next, create a volume, build the image, and run a temporary container so the compiled docker-credential-ecr-login binary ends up in that volume:

# 1. 创建一个用于存储二进制文件的卷
$ docker volume create helper

# 2. 使用上面的 Dockerfile 构建镜像
$ docker build -t aws-ecr-dock-cred-helper .

# 3. 运行容器，将 /go/bin (包含二进制文件) 挂载到 helper 卷
# 容器启动后即退出，但二进制文件已保存在卷中
$ docker run -d --rm --name aws-cred-helper --volume helper:/go/bin aws-ecr-dock-cred-helper

# 4. 然后查看是否存在 helper，如果存在就表示已经可以了
$ docker volume ls

If the helper volume is present, the binary is ready to be shared with Watchtower.

Step 2: Prepare Docker credential configuration

In your working directory, for example alongside docker-compose.yml, create .docker/config.json.

Replace <AWS_ACCOUNT_ID> and <AWS_ECR_REGION> with your actual AWS account ID and ECR region:

{
  "credsStore" : "ecr-login",
  "HttpHeaders" : {
    "User-Agent" : "Docker-Client/19.03.1 (XXXXXX)"
  },
  "auths" : {
    "<AWS_ACCOUNT_ID>.dkr.ecr.<AWS_ECR_REGION>.amazonaws.com" : {}
  },
  "credHelpers": {
    "<AWS_ACCOUNT_ID>.dkr.ecr.<AWS_ECR_REGION>.amazonaws.com" : "ecr-login"
  }
}

This tells Docker to use the ECR credential helper for the target registry instead of expecting standard login credentials.

Step 3: Wire everything into Watchtower

Once the helper binary and Docker config are ready, you can mount them into Watchtower through Docker Compose.

services:
  myapp:
    image: 1651561651.dkr.ecr.us-east-1.amazonaws.com/myapp:latest
    container_name: myapp
    restart: unless-stopped
    # 通过配置 label 能告诉 watchtower 需要关注哪些镜像的更新
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

  watchtower:
    image: containrrr/watchtower
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./config.json:/config.json
      - helper:/go/bin
    environment:
      - HOME=/
      - PATH=$PATH:/go/bin
      - WATCHTOWER_POLL_INTERVAL=10 # 间隔时间 10s 用于测试，实际可以根据所需调整
      - WATCHTOWER_MONITOR_ONLY=true
      - WATCHTOWER_LABEL_ENABLE=true
      - WATCHTOWER_CLEANUP=true
      - AWS_REGION=us-east-1
      - AWS_ACCESS_KEY_ID=AKxxxxx
      - AWS_SECRET_ACCESS_KEY=xxxxx
      # 最重要的是这个，需要将我们之前的 helper 挂载进去

volumes:
  helper:
    external: true

A few details matter here:

  • /var/run/docker.sock lets Watchtower monitor and update local containers.
  • ./config.json:/config.json provides the Docker credential configuration.
  • helper:/go/bin makes the compiled docker-credential-ecr-login binary available inside the container.
  • PATH=$PATH:/go/bin ensures the helper can actually be found and executed.
  • AWS_REGION, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY give the helper the AWS credentials it needs.
  • WATCHTOWER_LABEL_ENABLE=true means only containers with the proper label will be monitored.

For testing, WATCHTOWER_POLL_INTERVAL=10 checks every 10 seconds. In a real deployment, that interval should be adjusted to something more reasonable.

After startup, check the Watchtower logs. If the setup is correct, it should be able to pull from ECR without throwing the authentication error.

One deployment pitfall to watch for

In actual testing, the initial deployment order can matter.

If Watchtower was started before the helper was mounted correctly, simply changing the mount afterward may not be enough. In that case, it is safer to fully clean up and repeat the process:

  • stop and remove the Compose stack with docker compose down
  • remove the helper volume with docker volume rm -f helper
  • rebuild and rerun the earlier steps from the beginning

Doing this avoids cases where the volume was not mounted correctly or the helper was not picked up as expected.

With the helper binary shared through a Docker volume, the proper Docker config in place, and AWS credentials exposed to Watchtower, ECR-backed containers can be updated automatically just like images from a normal registry.

Related Posts