In-depth analysis · About 7 min read
AWS’s multi-turn RL guide makes reward design, decoding and the full bill part of the result
A SageMaker AI Wordle walkthrough reports gpt-oss-20b gains, while showing why evaluation settings, reward mistakes and AgentCore costs must stay visible.
- Published here
- Source published
- Source checked
AWS has published a follow-up hands-on guide to multi-turn reinforcement learning on SageMaker AI. The October23 technical article trains gpt-oss-20b to play Wordle, then explains its reward failures, operating problems and evaluation conditions. This is new implementation evidence about an existing service, rather than another announcement of training-session general availability or a new base model.
Listen to the news
Read this article aloud or keep listening to the news.
Preparing the reader…
Device voice
Speech is handled by your browser or device. AISubHub does not send the text to an audio API. Voice availability and playback with a locked screen vary by device.
Resuming may repeat a short phrase.
At a glance
- On128 held-out words, AWS reports61%→84% solves at medium reasoning and temperature1.0; low reasoning at the same temperature gives15%→69%.
- The 100-step run used 105,486 rollouts in 12h40m and its model token charges total $338.50; approximately $79 for AgentCore is separate.
- The guide shows how a badly weighted reward preferred slower solutions, and why a37% training-validation endpoint is not the84% serving evaluation.
The training target is an episode, not a model answer
A multi-step agent needs feedback on the sequence of decisions, not just an example response to imitate. In AWS’s architecture, SageMaker AI owns the training job and policy model; an AgentCore runtime hosts a Strands agent, the environment and scoring. The agent asks the policy model for a guess, receives Wordle feedback and reports one reward when the game ends. GRPO compares several rollouts for the same word and updates a LoRA adapter. Wordle’s five letters and six guesses make the outcome easy to score; it is a teaching task, not evidence of a production workflow.
The example separates640 training words from128 held-out words. Each training row carries a secret word for the environment, never as an answer hint to the policy. There are no target responses or labeled trajectories. Training uses100 steps,128 words per step and eight games per word, with low reasoning effort and temperature1.0. This separation matters: a working reward and environment provide the learning signal, but remain customer responsibilities.
A reward can improve while the intended behavior gets worse
The first scoring design accumulated bonuses after each guess. It could give a losing game+0.49, a first-guess solution+0.99 and a third-guess solution+1.61, rewarding a longer path over a better outcome. The final design makes the episode outcome dominant: a solution earns an outcome from1.55 on the first guess down to1.05 on the sixth; loss is−0.5 and no valid guess is−1.5. Clue consistency, repetition and information gain contribute smaller shaping adjustments.
Shaping is clipped between−1 and1 and weighted0.04 for solved games,0.15 otherwise. A previous solved-game weight of0.15 could still make slower completion preferable; narrowing it keeps adjacent success ranges apart and the worst solution above the success threshold. AISubHub’s interpretation is that the operational lesson is to inspect real trajectories and compare reward ordering against the intended behavior. A rising average reward alone cannot establish that an agent learned the right objective.
The headline gain depends on reasoning and sampling
On128 held-out words, the base model solves61% and the trained model84% at medium reasoning effort and temperature1.0, an improvement of23 percentage points. The trained84% averages two runs,80% and88%. At low effort with temperature1.0, the comparison is15%→69%, or54 points. At temperature0, the chart instead shows2%→37% for low effort and16%→30% for medium. These are four distinct conditions, not interchangeable estimates of one universal gain.
During training, validation uses low reasoning and a service-fixed temperature0. Its solve-rate curve peaks at48% at step90 and ends at37%, whereas the serving headline uses medium effort and sampling. The source explicitly connects the difference to decoding settings. Low effort used63–83% fewer sample tokens per game than medium in the authors’ evaluations; that is not a guaranteed reduction in total project cost. The held-out set is small, and AWS did not establish results for arbitrary tools, languages or business tasks.
Training tokens are not the entire bill
The reported100-step run lasted12h40m and processed105,486 rollouts. Its table lists491,616,806 prefill tokens at$0.12 per million,$58.99;159,737,098 sample tokens at$0.30,$47.92; and643,308,554 training tokens at$0.36,$231.59. The total is$338.50, rounded to about$339. These are article-time GPT-OSS-20B list prices in us-east-1. We preserve the source’s$ symbol without independently assigning a currency or treating the figures as a current, universal quote.
AgentCore CPU and memory add approximately$79 separately, and the evaluation jobs approximately$1.20. Prefill includes rereading conversation history; sampled tokens include reasoning, while backward-pass training is the largest listed charge. Longer trajectories therefore affect several meters. The guide also warns that AgentCore, MLflow and S3 resources can persist after training. AISubHub sees a useful accounting example, but has not verified a bill or established an all-in cost that includes every retained resource and downstream deployment.
Runtime failures can erase an otherwise valid experiment
The guide records failures that a high-level accuracy claim would hide. The job role needs the job.sagemaker.amazonaws.com trust principal and both AssumeRole and TagSession permissions. A handler decorator alone does not start the runtime server; the sample needs BedrockAgentCoreApp and an outermost entrypoint decorator. Data, runtime and job must use the same Region. Redeploying the runtime during training interrupts rollouts and can fail the job.
Thirty-two concurrent rollouts also exposed cached-credential expiry; the authors describe proactive refresh and re-signing after an authorization failure. Token budgets could leave no room for a tool call, and an overly large learning rate could turn structured calls into plain text. Their mitigations are implementation lessons, not reliability guarantees. We read the included code examples but did not install the dependencies, call AWS APIs, run the smoke test, train an adapter or reproduce the reported fixes.
What transfers beyond this Wordle walkthrough
The guide’s model table is limited to its writing date: Nova2Lite and GPT-OSS-20B in us-east-1 and us-west-2, and Gemma4 31B instruction-tuned and Qwen3.6 27B in us-west-2, using LoRA. It does not establish support for all models, Regions or accounts. Nor does this follow-up guide reannounce those models or the separate training-sessions service launch covered previously.
The transferable design is to keep task outcome, shaping incentives, serving configuration and total resource cost under separate review. A business workflow would need its own environment, defensible reward and representative held-out evaluation before claiming benefit. Our reading covers the complete article, all tables and seven necessary static figures. Linked documentation, papers, repository and licenses were not audited, and no independent performance, pricing or security test was performed. These reported results remain AWS’s bounded Wordle experiment.
Why it matters
The guide provides concrete ways to examine a multi-turn training project: test reward ordering, compare identical decoding conditions and budget both model work and the agent runtime. Whether those choices improve another workflow must be measured on that workflow.
Limits and availability
AWS reports a single Wordle experiment with128 held-out words; the headline trained rate averages two runs. We reviewed the complete guide and seven static figures, not linked documentation, repositories or licenses, and ran no training, API, billing or deployment test. Prices and model/Region conditions are source-time statements.
Sources and evidence
Source published: · Source checked:
AI-assisted editorial articles checked against linked primary sources. Vendor claims are attributed; inclusion is not endorsement.