Capability Overview
AWS has shipped a production-grade, event-driven infrastructure stack that enables multi-turn reinforcement learning (RL) for Amazon Nova models on SageMaker HyperPod. Unlike standard RLHF, which scores isolated responses, this system optimises over entire agent interaction sequences — teaching Nova to orchestrate tools, recover from mid-workflow failures, and chain multi-step reasoning. The architecture links three compute surfaces: SageMaker HyperPod P5 pods running vLLM and GRPO weight updates; ECS Fargate containers hosting the reward environment; and the Nova Forge SDK managing conversation state routing between them. AWS Step Functions and EventBridge tie it together, automatically triggering training runs when data lands in S3.
For defenders, the significance is not the Wordle demo — it is the production blueprint this represents. The same infrastructure, pointed at enterprise agentic workflows, trains models to autonomously query databases, call APIs, and recover from errors at scale. The trust surface for what shapes a model’s learned behaviour has grown substantially.
Attack Surface Analysis
Reward signal as an attack vector. The GRPO training loop depends entirely on reward scores returned by ECS Fargate workers. These workers are network-reachable services with a clearly defined API surface. An attacker who gains code execution inside a reward container — through a vulnerable dependency, a compromised container image, or an insider — can inject crafted reward scores. Because multi-turn RL optimises over sequences, even subtly biased rewards can steer learned policies toward attacker-desired behaviours without any direct weight manipulation.
S3-triggered pipeline as an injection point. Training is initiated by an EventBridge rule on S3 object creation. Any principal with write access to the trigger bucket can launch a training run. This is a low-friction vector for supplying poisoned datasets or initiating unsanctioned fine-tuning jobs that consume GPU budget and alter model behaviour simultaneously.
Ephemeral compute supply chain exposure. HyperPod pods are provisioned ephemerally per training run, pulling container images and model artefacts at runtime. A compromised upstream ECR image or a tampered Nova checkpoint in S3 propagates directly into the training job. Because the model is being further trained, any backdoor introduced at this stage persists into the fine-tuned artefact.
Conversation-state manipulation. The Nova Forge SDK routes multi-turn conversation state between model and reward environment. Intercepting or replaying state payloads mid-training — possible if message transit is not authenticated end-to-end — can corrupt trajectory rollouts used for policy gradient updates.
Excessive agency carry-over to production. Agents explicitly trained to recover from failures and bypass mid-process obstacles may learn generalised strategies that exceed intended boundaries. A model trained to retry after an API denial may generalise this to circumventing access controls in production deployments.
Framework Mapping
- AML.T0020 / LLM03: Poisoned datasets uploaded to S3 feed directly into training sequences, shaping multi-step policy.
- AML.T0018 / LLM05: Compromised reward containers or base images introduce backdoors into fine-tuned Nova checkpoints.
- AML.T0031: Sustained reward manipulation erodes model integrity across repeated training runs without leaving obvious artefacts.
- LLM08: Multi-turn RL explicitly trains agents for autonomous action sequences, amplifying excessive agency risk if guardrails are not co-trained.
- AML.T0010: Ephemeral provisioning of images and artefacts at training time is a textbook supply chain exposure window.
Threat Scenarios
Scenario 1 — Insider reward poisoning. A disgruntled ML engineer with ECS task update permissions modifies the Wordle reward worker to return inflated scores for responses containing a specific token sequence. Over subsequent training runs, the Nova model learns to embed that sequence in agentic outputs — a persistent, covert backdoor.
Scenario 2 — S3 trigger abuse by compromised CI/CD. An attacker who pivots from a developer’s compromised CI/CD pipeline uploads a crafted dataset to the training S3 bucket. EventBridge fires, HyperPod provisions GPUs, and an unsanctioned training run executes on enterprise-grade compute — modifying the production Nova checkpoint without any human approval gate.
Scenario 3 — Supply chain via base image. The ephemeral HyperPod pod pulls a vLLM base image from a registry where a dependency was silently trojaned. The training job completes normally, but the resulting model checkpoint carries a backdoor that activates on a specific input pattern in production.
Defender Checklist
- Apply strict least-privilege IAM policies to the S3 training-trigger bucket; log and alert on all writes from non-pipeline principals
- Enforce container image signing (AWS Signer or Notary) for all ECS Fargate reward worker images and HyperPod base images
- Deploy network segmentation between ECS reward workers and HyperPod pods; reward API calls should traverse an authenticated, TLS-enforced endpoint only
- Instrument Nova Forge SDK message routing with structured logging; establish baseline reward score distributions and alert on statistical anomalies before weight updates are committed
- Require human-in-the-loop approval gates (via Step Functions manual approval states) before training runs execute on production model checkpoints
- Pin all container image digests and model artefact checksums; validate integrity at pod startup, not just at image push time
- Review trained agent policies for excessive agency before production deployment — red-team the model’s tool-use behaviour specifically around permission boundaries and retry logic