openfront-ai devlog

Self-play RL on OpenFront.io, real engine, ~two days in. DESIGN.md · README

Jul 25, 2026 - 4×A100 scale-up (single pod)

Moved live ppo_v11 from 2×A100 to one 4×A100 80GB SXM pod. Kept the validated util stack, raised envs/shard for VRAM fill, and revised ETAs from measured update cadence.

Recipe

• Pod: ppo-v11-4xA100 (secure cloud), NUM_GPUS=4, NUM_ENVS=38, MAX_ENVS=40, same util clap defaults (rollout 48 / BPTT 24 / epochs 8-12 balance / persistent+recurrent).
• Old 2× pod (ppo-v11-2xA100) terminated after HF sync; RunPod now shows only ppo-v11-4xA100. One oftrain + one HF sync-loop; no duplicate trainers.
• Launch fixes that unblocked master resumes: clap ArgAction::Set needs --flag=true (not bare --amp); --foveate/--pinned-h2d stay flag-only so do not pass a trailing true.

Util / VRAM (stable)

• 2× baseline: ~100% util, ~55/80 GB at 24 envs/shard (48 envs total).
• 4× @ 32 envs/shard (96 total): ~95-100% util, ~47-52/80 GB; brief warm stretch looked ~10-12 updates/hr.
• 4× @ 38 envs/shard (152 total), settled: ~95-100% util, ~47-52/80 GB, single trainer. Update wall ~7.5-9 min (~7-8/hr), similar to late 2×, but ~3× env-steps per update (152 vs 48). Autoscale will not grow further on util alone (already at the 0.92 target); VRAM headroom to 40 remains.

Revised ETAs

As of Jul 27 ~21:00 UTC: stage 16 / u806. Stuck ~114 updates (~21 h) on 3-nation Easy since u692. WR briefly hit the 0.75 gate at u801, then 0.625 by u806. Cadence ~5.4 updates/hr; progression is WR-gated:

Caveats: nation-count jumps dominate calendar risk. A second trough like u801→u806 pushes clear past Jul 28. Medium lobby resets to 90 bots / 4n at s36.

Jul 24-25, 2026 - V11 soak: util stack, watch truth, clip bugs

First multi-day ppo_v11 run on 2×A100. Validated the util clap defaults, found greedy watch is not the train policy, fixed a client-render nations rewrite that truncated win clips at ~60% land, and measured real stage pace (fast bots-only, then a 10h stage-12 hole, then into the 1-nation band).

Util stack as clap defaults

• Live recipe mirrors scripts/pod_train_v11.sh: rollout 48, BPTT 24, epochs 8 (balance up to 12), amp/fp16/compact rollout, persistent + work-conserving actors, recurrent policy, autoscale target util 0.92 / max-envs 32.
• Resume across rollout/BPTT retunes is allowed; recurrent still requires persistent actors.
• Bool clap defaults use ArgAction::Set where needed (--amp true, not bare --amp).
• --max-episode-ticks is one source of truth: ofcore::DEFAULT_MAX_EPISODE_TICKS = 21000 for train, watch, and ofhub showcase. Watch --max-steps is derived so the decision cap cannot undercut the tick budget. OpenFront msPerTick=100 ⇒ 21000 ticks ≈ 35 in-game minutes.

Curriculum pace (what actually happened)

Advances (update → stage): 87→1, 113→2, 139→3, 208→4, 219→5, 231→6, 241→7, 251→8, 260→9, 269→10, 279→11, 296→12, 398→13, 425→14, 449→15.

Estimate update after s12: do not assume the 1 h/stage honeymoon holds. With recent ~3-5.5 h/stage, surpass-V10 (s28) slips to ~Jul 27-28; Medium (s68) to ~Aug 1-3. Another s12-shaped hole pushes Medium later.

Watch / eval truths

• Greedy ≠ train. Argmax watch freezes near spawn (~52 tiles) and dies on both native and Node. Stochastic sampling is what training / WR windows measure. Default is now stochastic (--watch-stochastic=true); ofshowcase daemon always passes it. Argmax is debug-only via --watch-stochastic=false. Watch / showcase clips default to --engine node so the GameRecord matches the TS client replay (native↔TS desync was shipping false death videos). Training stays native. Tick budget is the shared DEFAULT_MAX_EPISODE_TICKS=21000.
• FFA win is real 80% land. Replayed watch records with nations=0 restored: agent land share ~80.05-80.34%, fallout 0, reason land_share. Knife-edge clears look "on top but not painted" on large maps.
• Clip bug (nations rewrite). Client schema rejects nations: 0 (min(1) or "disabled"|"default"). Rewriting 0→1 for render spawned a nation that hit 80% mid-replay; the renderer stopped on that win modal while the agent was still ~60%. Fix: patch bots-only lobbies to nations: "disabled", and dismiss early client win/death modals until tick >= recorded end_tick (see scripts/render_client_replay.py). Native outcome is truth — do not seed-hunt when the client desyncs.
• Clip truth: native watch can record a real 80%-land win while the TS client replay desyncs and flashes You Died mid-video. Do not dismiss that and ship — refuse the render. Win clips must show client You-Won at end_tick. Asia u911 probe: native win@15261, client You Died@~6140 (desync).
• Stage-15 stochastic sample (u456, 6 maps): 2 wins (Asia, World) / 4 nation-losses. Train window still ~85% WR / 0 DR; nations are the new failure mode on some seeds/maps.

Scale: what helps before multipod

• Live pod: 2×A100 80GB @ ~97-100% util, ~55/80 GB VRAM, 24 envs/shard, MAX_ENVS=32, collect≈train (epochs already at max 12). Progression is mostly WR-gated, not idle-GPU-gated.
• Raising envs on the same 2 GPUs: small (~10-20%) at best.
• Best single-pod lever: more GPUs on one host (NUM_GPUS=4 etc.). oftrain is single-node multi-GPU (cuda:0..N-1 + NCCL, not a multi-machine launcher.
• Multipod only after one host's GPU ceiling; needs multi-node DDP we do not run in the current pod recipe.

Prose lint

scripts/lint_emdash.sh bans U+2014 and HTML — / — forms across the repo (excluding openfront/ and patches/). Prefer ASCII - / commas / colons. Test: scripts/tests/lint_emdash_test.sh.

Jul 23, 2026 - V11: obs + capacity + LSTM + BC restart

V10 proved the Easy mastery ladder and dense anti-spiral reward can climb. It also exposed that we have been squeezing a ~12M GRU policy that is still structurally blind on diplomacy geometry, attack ownership, and unit identity. V11 ships the missing DESIGN.md entity stack, replaces the GRU with a full LSTM, grows capacity, and unfreezes behavior cloning on the new schema - one from-scratch generation, not a mid-run migrate.

V10 killed (Jul 23)

• Pod qvrke1ylmfse4y (ppo-v8-4xA40-fresh, 4×A40): oftrain stopped, final ofhf push, pod terminated. Fleet empty.
• Final checkpoint: update 12581, stage 25 (Onion Easy ~22–26 bots / 3 nations band), ~44.5M env steps, window WR ~0.30 vs 95% gate, conversion ~0.63, lr 2.8e-5.
• Peak earlier in the ramp was stage ~27 overnight Jul 18–19; later grind oscillated in the mid-20s under the 95% Onion-nation gates. Curriculum worked; the bottleneck moved to representation / memory / capacity, not “needs another Φ term.”
• Weights remain on HF ppo_v10/ (latest + policy_update12580). No V10→V11 warm start - shapes and recurrence both change.

What we learned (audit → V11)

Pulled from V8–V10 training, the BC verdict / moratorium, the AlphaStar / OA5 contrast, and a pass over live obs vs DESIGN.md.

Observation is strong spatially, blind where humans look

• Neighbor pack was promised and never shipped. DESIGN player tokens include shared-border length with self, terrain mix along that border, and a rich relation continuum. Live P_FEAT=21 has troops/gold/tiles/ally/embargo/attack aggregates/income/doomsday - but no border-length or border-terrain features. Relation scores that matter for diplo sit mostly in the reward (v10-diplo-panic etc.), not in the policy’s player stream. The agent cannot see “who am I touching and on what terrain?”
• Attack fronts are anonymous. TR_ATTACK_SRC / TR_ATTACK_RETREAT rasterize every active attack with intensity, but they are shared planes - not ego-split (own / ally / enemy) the way every other unit-bearing transient is. Land invasions between two other bots look like fog blobs.
• No unit instances. DESIGN head 4 is a pointer over own-unit tokens (upgrade / delete / cancel_boat / move_warship). Live code still does nearest-in-region snap (translate::nearest_own_unit). The policy cannot name a specific city or warship; it points at a tile region and hopes.
• Diplomacy continuum is reward-only. Soft relation / panic / targeting pressure shaped the return; the obs mostly exposes binary ally/embargo/traitor bits. AlphaStar kept strategy targets in the observation and SL prior; we kept stacking mid-run Φ.

Capacity and memory are below the systems we cite

• Policy is ~11–12M params (GC=256, BLOCKS=4, HIDDEN=512) plus a 256-wide GRU (RECURRENT_HIDDEN). AlphaStar / OA5 sat in the tens–hundreds of millions with deep LSTMs. The Jul 10 small-vs-full A/B already hinted full policy learns more smoothly; V10 did not re-test size under the new curriculum.
• The GRU is not carrying the load. We added recurrence in V8.x hoping for commitment memory (attack/retreat flip-flops, massing neighbors). In practice the GRU has not unlocked those behaviors; boat/tempo/closeout still had to be reward-engineered. V11 replaces it with a full LSTM (cell + hidden), wider, with BPTT already wired (--bptt-chunk-len). Hypothesis: gated cell state over multi-minute games matters more than another Φ coefficient.

Reward / curriculum are ahead of representation

• V9 sparse-win cleared bots-only then died at first nations - scaffolding kind mattered more than “sparse vs dense” slogans (AlphaStar / OA5).
• V10’s 100-stage Easy ramp + ungate finish rewards + timeout stick were the right curriculum/reward moves: conversion → ~98%, climb to mid-20s. Further Φ stacking is low priority.
• Next wins are obs + architecture + imitation bootstrap, then reuse V10’s ladder (possibly remapped) under the new schema.

Past BC failed for reasons V11 can avoid

Honest read from bc_v1…bc_v6 (not “BC never works”):

• Ceiling was low on a blind obs. Holdout action acc saturated ~0.60–0.70; temporal BC was worse. Human archive ~300 games, noop/UI default heavy (quantity bucket 25%). Imitating clicks without neighbor geometry / unit identity / ego attack fronts teaches “looks human,” not “wins.”
• Warm start helped early PPO once (ppo_v5 off bc_v4), then stopped being the bottleneck - later plateaus were reward/curriculum/obs. The Jul 8 moratorium was correct: we never isolated BC from simultaneous recipe changes, and v7 broke the weights anyway.
• Infra ate the signal (z-cache ~2× not 10×, OOM, EDQUOT). V11 BC must ride the Rust AE path / compact caches, not resurrect the Python mmap pile.
• Unfreeze condition for V11: BC only on the new V11 schema (neighbor pack + ego fronts + unit tokens + LSTM policy), feedforward first, outcome-conditioned winner bucket, then a short A/B: PPO --init bc_v11 vs scratch on identical Easy/2 seeds. If warm start does not clear stages 0–3 faster, keep BC as an eval prior and do not block the PPO track.

V11 scope (all of it)

Keep from V10: dense v10-anti-spiral-style reward (profile bump to v11-… only if knobs change), 100-stage Easy mastery ladder (remap stage indices; do not restart the curriculum philosophy), foveation, compact/fp16 rollout, persistent actors, autoscale, HF sync-loop. Drop: mid-run Φ stacking as the default response to plateaus; GRU; nearest-unit snap for unit actions; AE v3.1 encoders.

Encoder / schema baseline

• AE: ae_v32_nostatic_d8c32 (fine) + ae_v32_nostatic_d16c32 (coarse). Static buildings are exact 6-plane bypass; latent is owners+terrain(+fallout) only. C_GRID starts at 95 before ego-split attack expansion.
• Schema id suggestion: v11_lstm_entities in sidecar / checkpoint meta so resume refuses V10 weights loudly.
• Webbot / ONNX export must track unit pointer + LSTM state; do not ship V11 to showcase until both are wired.

BC unfreeze rules (so we do not repeat Jul 8)

1. No BC until V11 featurization + policy forward are green on bot + human holdout batches (shape tests, legal masks, pointer masks).
2. Train bc_v11 feedforward on the new schema only; downsample noops; mask unused heads; winner-bucket at deploy.
3. Success metric is not chasing 0.70→0.75 action acc. Success is: short PPO A/B where warm start beats scratch to clear Easy bots-only stages under identical seeds/config.
4. If A/B is flat, freeze bc_v11 as a playable prior and continue PPO from scratch - do not grind the archive for weeks.
5. Temporal / seq BC stays shelved until LSTM PPO demonstrably uses memory (commitment metrics: attack flip rate, alliance hold time).

Build order

1. Feat: neighbor pack + ego-split attack fronts in ofcore::feat / batch encode; update webbot parity tests.
2. Units: top-K unit token builder + legality masks; pointer head in policy.rs; translate path for upgrade/delete/ warship/boat-cancel.
3. LSTM + size: replace RecurrentCore GRU with LSTM; bump GC/BLOCKS/HIDDEN/RECURRENT defaults; fix checkpoint serialize for (h, c).
4. AE wire-up: finish/export v3.2 weights; point pod_train_v11.sh at them (do not leave v3.1 hardcodes).
5. BC: Rust BC loop (or recovered Python) → bc_v11 → isolation A/B.
6. PPO launch: ppo_v11 on 4×A40, V10 curriculum + reward, from BC init if A/B wins else scratch. Kill criteria pre- registered: stage-0–3 wall-clock vs V10 baseline; attack flip rate; unit-pointer usage share.

Non-goals for V11

• Another sparse-win parallel (V9 answered).
• New boat/tempo/embargo Φ terms before the obs/LSTM stack lands.
• Self-play league (still post-bot-curriculum).
• Migrating ppo_v10 weights into V11.

Implementation tracks this plan on branch work toward scripts/pod_train_v11.sh + schema bump; this entry is the contract.

Jul 18–19, 2026 - V10: Easy mastery ladder that actually climbs

After V9 died on first nations and the dense 15-stage V10 ladder stalled around stage 2–3 (~30–50% WR), we rebuilt the curriculum around a long Easy Onion ramp, high early gates, and denser late Easy→Impossible steps. Overnight result: 0 → 27 on the new 100-stage curve.

Why V10, not another sparse bet

V9 proved win-only + high gates can clear bots-only, then collapsed the moment nations appeared. The dense V8.x reward stack (closeout Φ, boats, tempo, embargo/combat thrash, soft death) was already teaching useful priors - the failure mode was lobby difficulty, not missing shaping. So V10 keeps v10-anti-spiral-v1 (V8.6-like dense reward + survival / anti-diplo / combat priors) and spends the degrees of freedom on a much longer Easy mastery curriculum, inspired by josh-freeman / AlphaFront starting at Easy/2 and reporting 100% WR there.

Reward / episode fixes that unblocked finish

Before the ramp rewrite, mid-lobby episodes often reached ~45% land and then idled. Diagnosis: win rate ≠ conversion. Many washouts never hit closeout; among closeout episodes, timeouts-after-closeout were the “does nothing” failure. Shipped in sequence:

• Ungate finish rewards (closeout Φ / tempo / boat / combat not hard-gated on stage ≥5)
• Max episode ticks 21000 (35 min); survival taper 45→80% land
• --v10-timeout-closeout 20 stick after closeout
• Radically larger win terminal: --v85-extra-win-bonus 200, --v84-fast-win-coef 40
• One-shot --v10-closeout-entry 25 on first cross of 45% land

After those: timeout-after-closeout ~0–1%, converted episodes rewarding ~310, but the dense lobby gate (then 70%) still was not clearing. The curriculum, not the reward, was the remaining bottleneck.

100-stage curve

Shipped in #50 / #51 / #52 (db79da8). Geometry:

win_at holds 95% through the Onion micro-ramp, then smoothsteps to 65% by stage 99 - no 95→70 cliff at the old “legacy stage 0” handoff. Closeout conversion gate @48, eight-map bridge @54.

Overnight result (Jul 18 eve → Jul 19 morning)

Stage 26 (26 bots / 3 nations) cleared at ~90% wins. Stage 27 is the first real slowdown on the 95% bar - still converting when it reaches 45% land, not yet winning often enough to graduate. One demote 27→26 and re-advance; no death spiral.

What we learned

• Match AlphaFront’s start, not their whole stack. Easy/2 bots-only with a high win gate is the missing first mile. Dense reward on top of that climbs; sparse win-only on a hard lobby does not.
• Do not jump 0→2 nations. An explicit 1-nation band (stages 15–19) is load-bearing. The old ramp skipped it and pretended nations were a small step; they are not.
• High gates on easy lobbies, decay later. 95% through Onion mastery forces competence before maps/difficulty expand. A hard drop to 70% at the first dense lobby invited “advance on luck, demote on death.”
• Win rate and conversion answer different questions. Low WR with mid conversion meant washouts; high conversion with mid WR means finish / endgame. Instrument both before changing reward.
• Finish incentives must fire before the curriculum gets hard. Ungating closeout/tempo + timeout stick + big win bonus removed the post-45%-land idle without waiting for stage ≥5.
• Sidecar expands need an explicit remap. 15→35 used +ramp_len; 35→100 uses proportional remap. Forcing stage 0 after a curriculum rewrite beats preserving a stuck dense index.
• Disk full kills trainers silently. Pod filled to 80G/80G; crash-looped with No space left on device on log tee / git fetch. Trim validation dirs + truncate logs before redeploy; watch the overlay (~87% again by morning).
• HF sync-loop is the recovery path. ofhf sync-loop on djmango/openfront-rl / ppo_v10 kept latest + numbered policy updates current through the overnight climb (occasional xet/commit retry noise; uploads continue).

Ops pointers

• Pod: 4×A40 RunPod, tmux v10-production, scripts/pod_train_v10.sh, HEAD db79da8+
• Checkpoints: HF ppo_v10
• Showcase: openfrontai.skg.gg - defaults RUN_NAME=ppo_v10; rebuild via docker/homelab_deploy.sh when clips go stale
• Devlog proxy: skg.gg/pages/openfront-devlog (Cloudflare Pages pulls docs/devlog.html from master)

Jul 14, 2026 - V9: sparse win/loss as a parallel to V8.x

Hypothesis: we have been doing a lot of reward engineering (V8.3 closeout → V8.4 boats/tempo → V8.5 urgency → V8.6 attack-fair) to force the behaviors we want. Instead, try an extremely gradual curriculum with sparse win/loss only, and require near-perfect win rate to graduate. Parallel run, not a migrate-from-V8.

Why parallel, not another V8.x reward profile

V8.3–V8.6 share the same curriculum schedule and keep stacking dense terms (strength, delta, closeout Φ, boat outcomes, tempo, embargo/combat thrash, death overrides). That path is valuable, but each knob is another hand-written prior. Zeroing V8.x coeffs is not enough for win-only: vecenv still always applies strength, strength-delta, waste, death, and place terminal. So V9 is a new schedule + a hard sparse branch in reward assembly, not --migrate-v86-to-v9.

What AlphaStar and OpenAI Five actually did

Neither system was pure win/loss with no scaffolding. The useful contrast is what kind of scaffolding they used.

Takeaways we are acting on:

• Win-only can work, but expects worse sample efficiency unless early games are easy enough that the policy almost always wins.
• OA5 shaped once and froze; they did not keep stacking mid-run terms the way V8.4→8.5→8.6 did. If V9 stalls, prefer a cheap bootstrap (BC / light strategy targets) over more dense shaping.
• AlphaStar’s scaffold was imitation + strategy targets, not per-tick gold/kill crumbs. Different from both OA5 and our V8.x stack.
• Do not use win_at: 1.0 with current should_advance (mean > gate over WINDOW=40): perfect 40/40 still fails > 1.0. Use 0.975 for “essentially locked.”

V9 design

Stage sketch (high gates)

Bots are always present and always far outnumber nations (food). Typical ratios: 30 bots / 5 nations, 200 bots / 30 nations. Early stages are bots-only; nations are layered on with Exact(n) only - never Nations::Default (that dumps the full map manifest, e.g. World ~72).

Env targets: 24 early, taper to 8 on late broad-pool stages (same spirit as V8.3). Rehearsal 25% still applies; rehearsal eps do not count toward the gate.

Implementation notes

• CurriculumSchedule::V9 + V9_ENV_TARGETS in rust/ofcore/src/curriculum.rs.
• RewardConfig.v9_sparse_win → profile v9-sparse-win-v1; sparse_terminal_reward(won).
• vecenv early-returns the sparse path before strength/delta/shaping.
• CLI: --v9-curriculum and --v9-sparse-win must be used together (mutually exclusive with V8.x schedule flags).
• Sidecar writes reward_profile for V9; resume requires matching v9-sparse-win-v1. No migrate-from-V8 path.
• Launch: NUM_GPUS=4 bash scripts/pod_train_v9.sh (or V9_MODE=1 bash scripts/pod_train_v8.sh). Smoke: bash scripts/tests/pod_train_v9_test.sh.

What success / failure looks like

• Good: stage 0–4 clear with locked win rates; policy invents attack/boat/build without being told; later stages stay above 0.90 without new shaping knobs.
• Expected pain: much slower wall-clock than V8.x; long plateaus if a stage jump is still too hard under sparse credit.
• If stuck: split the failing stage further (smaller bot/map step) before adding reward terms. Only then consider a cheap AlphaStar-style bootstrap (BC init or light strategy targets), not another boat/tempo stack.

PR: #35 (sully/v9-sparse-win). Intended to A/B against live ppo_v86 on a second pod.

Launch notes (Jul 14 evening)

• Worktree: /workspace/oftrain-v9-sparse-win on sully/v9-sparse-win.
• 4×A40 secure was out of stock; took 3×A40 (ppo-v9-sparse-win-3xA40 / wq7acsw606cc9q).
• Custom dockerArgs that curl+train before sshd is ready left the first pod SSH-stuck (“pod not ready”); relaunched with the image default entrypoint, then started training over SSH via /root/start_v9.sh (clone branch + SKIP_SYNC=1 NUM_GPUS=3 bash scripts/pod_train_v9.sh).
• HF_TOKEN must be exported in the SSH launcher - RunPod --env did not land in the login shell for this create path.
• Account balance was ~$29 with V8 already at $1.76/hr; V9 adds $1.32/hr. Top up before a long overnight if both stay up.
• First updates healthy: profile v9-sparse-win-v1, gamma=0.9997, ret-clip=2, stage 0, ~210–217 steps/s on 3×A40, recent_reward=-1.0 (sparse terminals; early losses expected). Checkpoints at update 5/10 under rust/checkpoints/ppo_v9/.
• Curriculum fix (same evening): stage 0–2 were nation-only (bots: 0, Exact(1→3)) - nations are harder than bots, so the sparse ladder never saw a win (~650 eps, win_rate 0, land share decaying). Reladdered so bots are always on and always ≫ nations (~6:1+, e.g. 30/5, 200/30); never Nations::Default. Stages 0–1 bots-only, then nations layered on. Fresh restart from stage 0 on the same pod after rebuild.

Jul 15, 2026 - V9 sparse-win killed (postmortem)

Overnight run on 3×A40. Bots-only stages worked; the first nation step killed the policy and it never recovered. Experiment stopped; pod terminated.

What happened

Timeline:

• ~u0–25: all losses; random policy dying to 15 Easy bots.
• ~u26: first +1 win under sparse terminals.
• ~u300–700: real learning on bots-only - land share up to ~0.75, window WR often 0.4–0.8, cleared stage 0 (win_at=0.975) and stage 1 (win_at=0.95).
• ~u750+: stage 2 introduces 2 nations beside 30 bots. Win rate collapses to ~0; deaths 67–89%; max land share falls back to ~0.1–0.2. ~750 further updates with no recovery.

What we learned

• Sparse + bots-only can work. The policy invented enough expand/fight behavior to lock near-perfect win rates with only terminal +1/−1 and no dense shaping - matching the OA5 “win-only is possible but slow” note, on a trivial enough stage.
• Nations are a cliff under sparse credit. The jump from Exact(0) → Exact(2) (still 30 bots) was enough to erase competence. High gates got us onto stage 2; they did not keep us alive there. Splitting further (1 nation, then 2) might soften it, but the overnight evidence is that sparse alone does not bridge “bots food” → “competent nation opponents” at our sample budget.
• Always bots ≫ nations (and never Nations::Default) is the right lobby shape either way - kept for V8.3 via #38 and V9 via #40. The first V9 sketch (nation-only early stages) was simply wrong.
• Do not start sparse self-play cold. Self-play as 1v1 + bots is a good later mix on dense V8 once a checkpoint can farm bots; it is a bad bootstrap for sparse V9 (two weak peers, non-stationary, no easy +1s). Prefer scripted bots/nations until the policy is competent.
• Prefer V8 dense path for now. V9’s hypothesis (“curriculum instead of reward engineering”) failed at the first real opponent step. Resume effort on ppo_v86 / dense shaping; if sparse is revisited, want BC or strategy-target bootstrap before nation stages - not another boat/tempo knob stack on V9.

Code stays on master (schedule, --v9-sparse-win, pod_train_v9.sh). Run is dead; do not relaunch without a new plan.

Jul 14, 2026 - persistent actors + GPU-util autoscale via restart

Clarified game-time vs wall-clock, then fixed the inverse --persistent-actors / --auto-scale-envs relationship and deployed it onto live ppo_v86.

Game time: 15000 ticks is ~25 minutes, not 3–4

OpenFront’s config is msPerTick() = 100 → 10 ticks/s of normal game time. So:

Half-hour games are fine and desired - we want episodes long enough to actually win. The earlier “collectors install in 3–4 minute games” figure was pod wall-clock collect stall (one PPO update’s collect phase), not match length. Shortening max_episode_ticks for util would train the wrong horizon.

Why persistent and autoscale looked inversely proportionate

Persistent owners pin actor/learner CUDA state to long-lived OS threads. Live mid-run spawn_worker growth (the legacy autoscale path) is unsafe under that ownership model. The trainer therefore used to print a warning and cfg.auto_scale_envs = false whenever both flags were set.

Meanwhile V8.3 stage env targets drop late stages to ~8–12 envs/GPU for host memory / engine tail latency. With autoscale forced off, stage 8 stuck at ~10 envs/shard → long collects, GPUs often idle (VRAM full, util spiky/uneven). It looked like “persistent ⇒ bad util” and “more envs ⇒ need shorter episodes,” but the real coupling was autoscale disabled + low stage floor.

Curriculum stage env changes already used a safe path: checkpoint → restart_request.json → clean exit → pod_train_v8.sh relaunches at the new count. Util autoscale just wasn’t wired into that path.

What we changed

• Restart-based autoscale: with --persistent-actors, GPU-util growth sets requested_env_target, writes restart_request.json with reason=gpu_util_autoscale, and exits. Same supervisor path as stage env-target resizes. Legacy (non-persistent) collectors still live-spawn.
• Stage targets as floors: within a stage, resume keeps max(previous envs_per_shard, stage_floor) so autoscale gains stick. Cold start still honors the stage schedule. Explicit requested_env_target always wins (stage shrink/grow or autoscale).
• min_envs alignment: default floor is CLI --num-envs, not stage_env_targets[--stage] (often an early-stage 24). After resolve, if min_envs > live count, snap down so we don’t jump from 10→24 on the first check.
• Pod defaults: V8.3+ sets AUTO_SCALE_ENVS=1, MAX_ENVS=20, TARGET_GPU_UTIL=0.85, step 2. Override with AUTO_SCALE_ENVS=0.

PRs: #30 (core), #31 (min_envs), #32 (minibatch / restart consume), #33 (clamp to max-envs).

Deploy lessons (live ppo_v86 on 4×A40)

First autoscale check correctly requested 10 → 26 envs/shard (reason=gpu_util_autoscale). Then three failure modes stacked:

1. Recurrent minibatch gate: startup required minibatches ≤ every stage_env_targets entry (late floors are 8). Growing to 26 recomputed minibatches=13 → instant Error: recurrent PPO requires minibatches <= envs per shard. Fix: check only the resolved live env count after sizing, not the whole stage table.
2. Restart-request pin loop: oftrain left restart_request.json on failed boot; pod_train re-read it forever (26 → 26 every ~1s). Fix: supervisor mvs the request to .last when consuming it.
3. OOM at 26×4=104 envs: prime/rollout allocated ~43 GiB/GPU and died. Fix: default MAX_ENVS=20; clamp resolved startup count to --max-envs so a stale requested_env_target cannot OOM the next boot. Cleared the pending 26 request and relaunched at 16.

Live status after redeploy

• Tree: master 4c0e01fe under /tmp/v86-validation, tmux v86-production.
• Resume: ppo_v86/latest.safetensors @ update 651, stage 8, profile v8.6-attack-fair-v1.
• Sizing: 16 envs/shard (64 total), --auto-scale-envs --max-envs 20, max_ticks=15000.
• First post-fix update completed (~142s collect-bound; util still uneven - expected until/unless autoscale climbs 16→18→20).

Next: let util-driven restarts step toward 20; watch OOM; leave reward settling alone until env count moves. Still do not cut episode ticks.

Jul 14, 2026 - V8.6: attack-fair reward fixes

V8.5 mid-run “unlearning” audit → reward-only ppo_v86 that resumes the same weights.

What the audit found

• Sticky combat refresh: every reinforce rewrote last_attack_decision, so nearly every retreat paid premature −0.10 and AttackRetreat churn −0.05.
• Tempo / dominant-loss mismatch: tempo drains from share 0.30, but loss-weight relief only at 0.55 - and ~80% of stage-8 eps never reach 0.55.
• Strength asymmetry: W_DELTA_GAIN=5 vs W_DELTA_LOSS=6.5 taxes the troop-burn-before-land shape of every attack.
• Death too cheap: W_DEATH=1 vs positive place terminal.
• Boat merge → Destroyed: landing into an existing sourced attack has no new attack id and ~0 refund.
• First-step windfall: prev_strength=0 at reset → free +gain×share.

V8.6 design (still checkpoint-compatible)

Launch: SKIP_SYNC=1 V86_MODE=1 → ppo_v86, seed from ppo_v85.

Jul 14, 2026 - V8.5: win-first urgency, embargo + combat outcomes

Overnight ppo_v84 audit → reward-only ppo_v85 that resumes the same weights.

What V8.4 taught overnight

• Boats improved: useful landing fraction ~53–59% on stages 7–8; boat reward positive.
• ~80% of stage-8 episodes never reach dominance (0.55); median max land ~0.09. Tempo (gated on 0.55) only fired on ~14% of episodes.
• Embargo thrash remains the main dumb pattern - even on closeout winners (~7.5 embargo pairs/ep). Attack↔retreat secondary.
• On winners, terminal (~42) already dwarfs shaping; on the other 80%, PPO mostly trains strength/death/boat crumbs because win never arrives.

V8.5 design (still checkpoint-compatible)

Relations are now emitted in native entities obs (reward-only; policy tensors unchanged). Embargo-stop is priced from the pre-step relation score. Combat outcomes use a sticky decision window (same length as churn window).

Jul 14, 2026 - V8.3 → V8.4: what we learned about boats, greedy, and churn

Live ppo_v83 on 4×A40 (optimized trainer), then design of reward-only ppo_v84 that resumes the same weights.

V8.3 closeout worked - then the real failure mode showed up

Migrated from V8.2 stage 5 into V8.3’s inserted Easy closeout stage (Onion/Pangaea/Caucasus, 10 Easy bots, Exact(6) nations, win_at 0.45) with land-share potential Φ from 0.45→0.80 and a dual advancement gate (wins + conversion). Early episodes were deaths at ~0.15–0.18 max land; within ~15 updates it was hitting ~0.80 land and converting. At advance: 14/25 wins, 14/14 conversions among qualifiers. Across 57 finished stage-5 episodes: 26 wins, 27 closeout reaches, 26/27 converted. Optimized infra held ~80–100% GPU util after the NCCL SHM / fused BPTT / cross-map batching work.

Stage 6 (30 Easy, broader maps, 16 envs/shard) is where playtesting hurt: under greedy the mode action is often boat, and coordinate overlays show destinations are effectively random. Stochastic training still explores attack/build, so train metrics look “kind of OK” while deployment-style play looks brain-dead.

Greedy vs sampling is not the training bug

Superseded (V11): play / watch / showcase are stochastic, matching PPO rollouts and WR windows. Greedy argmax freezes near spawn and dies — do not use it for demos. Mixing greedy trajectories into training still does not invent a better mode. Temperature does not exist in this stack; exploration is softmax + entropy floor. Argmax remains debug-only (--watch-stochastic=false / PLAY_GREEDY=1).

Why it does not “just learn boats are bad”

1. Launch is nearly free under strength. Fielded troops aboard transports still count in the military share (v6.2 fixed cancel-farming). So boat does not create an immediate strength_delta slap for a stupid destination.
2. Real cost is delayed past credit assignment. Land/die/cancel happens many decisions later; BPTT was 16. The action head learns “boat is an okay spam,” the tile head never sees which (x,y) caused pain.
3. Legal ≠ good. Boat tile mask is reachable non-owned land, not useful shore. Random among legal cells is reward-equivalent until much later.
4. Waste is tiny and wrong-shaped. W_WASTE=0.01 only when translate fails. A successfully launched stupid boat is not waste.
5. Win is too sparse to train the tile head. “Random boats slightly lower P(win)” loses to dense strength that treats launch as neutral.

Longer GRU/BPTT helps a little; it does not invent a boat-quality signal. Bigger GRU also breaks the checkpoint. So V8.4 keeps hidden size 256, bumps BPTT/rollout to 32/64, and adds an explicit outcome term.

Churn is a band-aid; embargo thrash dominated

Across 161 V8.3 episodes the inverse-pair counts were roughly: embargo↔stop 333, attack↔retreat 211, boat↔cancel 70. Window=2 only caught adjacent undos; we bumped to 16. That still does not explain boat addiction - and explicit churn is not the fundamental fix for free reversible actions (mask/gate inverses or price outcomes).

V8.4 design (reward-only, resume from ppo_v83)

Boat outcome is categorical at resolution (not launch/cancel Δtroops), so it does not reintroduce the old churn farm. Pending transports are tracked by unit id; classification uses sourced land attacks, cancel flags, and refund heuristics. Metrics: reward/boat_outcome, reward/tempo, boats/*.

What we are not doing yet: hard-masking boats when land attack exists (user wants learning, not forbidding), bigger GRU, mixing greedy into PPO.

Operational notes

• Launch: SKIP_SYNC=1 V84_MODE=1 from a tree with this branch; seeds once from local/HF ppo_v83.
• Negative boat rewards must use --flag=-0.20 form; bare -0.20 is parsed by clap as an unexpected flag.
• Shallow clones lack openfront/resources/maps - symlink the maps tree from a populated checkout before spawn.
• Do not play/watch greedy for demos — use stochastic (train-matched). Argmax is debug-only.
• NCCL on this A40 host still needs NCCL_P2P_DISABLE=1 / NCCL_IB_DISABLE=1 (SHM collectives).

Jul 14, 2026 - curriculum parity closeout: what we learned

Outcome gate on curriculum-parity-v4 (48 self-play records, 8 bot-count buckets × 6 seeds, 20k-tick / 40-min timer horizon), plus the gate speed work that made iterating on it bearable.

The bar that matters

Exact tick/hash parity is still the wrong instrument for RL. Training needs the native engine to produce the same kind of game as TS: same winner when one exists, similar terminal timing, similar land share, and - when both sides honestly fail to crown a winner before the record ends - matching final rankings. The archive of 78 real 400-bot human games answers a different question (extreme bot density) and remains a poor curriculum proxy.

What closed the last gaps on v4

• Water-path duplicate stalls. TS PathFinderStepper keeps consecutive MiniMapTransformer duplicates as one-tick stalls; native was collapsing them, so transport boats arrived a tick early and everything downstream drifted.
• Docked warship heal-before-undock. TS applies passive+active heal, then handles manual patrol undock; native undocked first and permanently lost 5 HP on that tick.
• Warship patrol random bounds. TS nextInt(-range/2, range/2) floors both ends. Rust's integer division rounded odd negative bounds toward zero (-225/2 → -112 vs TS -113), shifting the PRNG draw and sending RF warship 241 down a different path by tick ~3115.
• Duplicate TrainStation lookup. TS findStation(unit) returns the first inserted station for a unit; Rust's station_by_unit.insert overwrote with the newest. Moroccan train trade gold then missed a +35k payout and gold desynced at tick 8245.
• Agreed-stalemate policy. With maxTimerValue=40 on a 20k-tick record, max_timer never fires (~24k ticks needed). Soft-identical boards that both report winner=None are now a pass when final tick, land tiles, and rankings match (ignoring leftover alive bits on zero-tile players).

Result: 48/48 on curriculum-parity-v4. Earlier mid-run scoreboard was ~36 → 43 → 44 → 46 → 48 as each class of bug landed.

Why the gate felt "CPU isn't even pinned"

Because it wasn't. The shell had CURRICULUM_JOBS=4 exported on a 20-core box while the script default is nproc. Separately:

• TS oracle was already a cache hit - Node was not the cost.
• Native replayed every record to the full 20k-tick horizon even after a land-share win at tick ~5k (often 40%+ wasted ticks on winner games).
• cargo run re-checked/rebuilt the binary every invocation.

Fixes shipped: early-exit after a winner on both TS and native (oracle fingerprint rotates once), native outcome cache keyed by engine fingerprint + record-set hash, exec the prebuilt outcome_gate binary, default TS oracle workers to all cores (was capped at 4). Measured on this host: cold compare ~691s (still bounded by the longest late/no-winner game), cached re-run ~6ms.

Tick-by-tick parity, out of curiosity

Soft field agreement (alive,tiles,troops,gold,numUnits, and unit dumps when needed) is the useful zoom lens when an outcome fails - it is not the gate. On the road to 48/48 we routinely saw:

• Early curriculum games (0-10 bots, small maps) soft-agree for thousands of ticks after the neighbor-order fix; many now soft-agree through the whole record once the boat/warship/rail bugs above landed.
• Mid/high bot games can still soft-diverge (gold, unit tiles, shell range) long before rankings disagree enough to flip the outcome. That is expected: autonomous bot AI amplifies any PRNG/path difference every tick.
• Exact game.hash() bit-identity remains fragile (IEEE noise, intentional early-exit changing finalTick on winner games) and is not what training needs.

Tooling that made the grind possible: scripts/bisect_parity.sh (coarse then fine), OF_DUMP_TICKS_FROM / unit dumps including patrolTile, and clearing leaked dump env vars before coarse passes so bisects stop lying about first divergence.

What we are expanding next

v4's map coverage was thin at the high end (mostly the seven legacy curriculum names, 6 seeds/bucket). The generator now fans mid/late buckets across the broad V8.2-ish pool available on the pinned openfront (Europe, Britannia, GreatLakes, continents, EastAsia, MiddleEast, Japan, Caribbean, Iceland, FourIslands, …) and we are spinning a larger curriculum-parity-v5 set (128 games, 16 seeds/bucket) to see whether 48/48 was "these 48" or "the class of games the trainer actually samples."

Jul 13, 2026 - Rust PPO CUDA diagnosis, persistent ownership, and the path past Python

The goal was not "restart when CUDA dies." It was a stable trainer that can be made faster without reintroducing hidden races. All production validation below was deliberately returned to one A40 until the ownership model proved itself.

What failed, and why the early fixes were misleading

The four-A40 run repeatedly reached a few updates and then failed with cudaErrorAssert. Because CUDA reports errors at the next synchronization, traces named innocent operations such as memcpy_and_sync, copy_device_to_device, and reduction kernels. Several plausible optimizations passed CPU tests and then merely changed the time-to-crash:

• CUDA-resident rollout tensors crossing collector/learner threads;
• fine/coarse AE tensor sharing before actor ownership was persistent;
• full-AE and conv-only bf16 inference;
• pipelined env groups and temporary non-blocking pinned H2D buffers.

Those variants were reverted rather than protected by recovery logic. A restart loop is useful for host failures, but it is actively harmful for deterministic GPU corruption: optimizer state is not restored, repeated warm restarts damaged entropy/reward, and a new process only delayed the same fault.

The diagnostic method that replaced blind soak/restart cycles

• NVIDIA Compute Sanitizer: memcheck with stream-ordered race tracking ran six instrumented updates and reported no invalid global-memory access. racecheck, initcheck, synccheck, CUDA core dumps, and cuda-gdb are available on the pod for narrower kernel bugs. PyTorch CSAN exists only in the Python frontend, so it cannot directly instrument tch-rs.
• CUDA_LAUNCH_BLOCKING=1: the exact workload became stable, proving an asynchronous ordering/lifetime failure. TORCH_USE_CUDA_DSA was not treated as a magic environment variable; it is a PyTorch build-time option.
• Named phase synchronization: opt-in --cuda-sync-diagnostics records the last successful actor AE/build, policy act, D2H, bootstrap, learner batch-build, forward/backward, gradient, optimizer, and weight-copy boundary with device/shard/update context.
• Single-GPU first: one GPU removed cross-device all-reduce as a confound while retaining actor/learner overlap and the same CUDA stream ownership problem.

Root cause: CUDA state crossed ephemeral thread-local streams

std::thread::scope created new collector, batch-builder, and backward OS threads every update/minibatch. LibTorch's default CUDA stream is thread-local. Host join() only waited for the Rust function; it did not guarantee kernels queued on that thread's CUDA stream were complete. Three unsafe ownership boundaries followed:

1. actor policy/AE tensors returned from a collector thread and were refreshed or reused by a new collector thread;
2. ShardBatch CUDA tensors were built on one temporary learner thread and consumed by another;
3. VarStore::copy could return before the learner-to-actor D2D refresh completed.

The immediate correctness fix synchronizes collector exit, batch-builder handoff, initialization, and actor weight refresh. The durable architecture is opt-in --persistent-actors: one persistent actor thread owns each GPU's actor policy, frozen AEs, terrain cache, and env workers; on one GPU, a persistent learner owns the trainable policy, optimizer, batch tensors, backward, and PPO update. Only CPU-owned rollouts and a packed 43 MiB Vec<f32> weight snapshot cross channels. No CUDA tensor crosses an OS-thread boundary.

Measured one-A40 results

The live run advanced stage 1 → 2 at update 262. Stage-1 recent reward reached roughly 41-49; entropy stayed mostly 2.5-3.5. Value MSE sometimes spiked at curriculum/non-stationarity boundaries but repeatedly recovered instead of exploding. Stage 2 is intrinsically heavier: larger maps increase both frozen-AE work and policy training, and episodes take longer before reward telemetry becomes representative. The 64-env A/B was rejected: it produced no fleet-throughput gain and increased tail/CPU pressure; 48 envs/GPU remains the measured optimum.

Exact speedups retained in the candidate stack

• compact fp16 host-owned foveated rollouts; no actor CUDA tensor enters a Step;
• episode-static land/magnitude cached on the actor device while fallout/owners/structures stay fresh;
• u8 owner slots (87.5% less owner H2D) and bit-packed fallout (96.875% less fallout H2D);
• two packed action-result D2H transfers instead of eight synchronizations;
• one shared fine/coarse AE input upload, gated to persistent actor ownership;
• single foveation reused by all policy heads;
• uniform compact fast path plus stable exact-shape bucketing for stage-2 mixed maps;
• pooled immutable compact-host payloads instead of six allocations per env/step;
• fused native tile trim/LUT/fallout/defense/ego/local preparation;
• persistent two-group actor pipeline, now safe because the actor remains on one stream-owning thread.

Not retained: stale coarse-latent caching, cross-thread resident rollouts, or any change to reward/PPO math. Frozen AE convolution weights now have cached bf16 copies under persistent actor ownership; embedding, GroupNorm, and returned latents remain fp32. Performance claims use fleet policy decisions/s, not Python's game-ticks/s (which must be divided by decision_ticks).

Four-GPU scale-out and old-model evidence (Jul 13 evening)

• Latest master engine parity integrated: UnitGrid ordering, nation assist/alliance behavior, raw attack budgets, warship repair/targeting, defense-post queries, ports/trade ships, and spawn validity. These change the stage-4 MDP for correctness, not benchmark cosmetics.
• Persistent multi-GPU learners: one owner thread per GPU, deterministic shared minibatch shuffles, shard-ordered CPU gradient averaging, global clipping, and identical optimizer steps. Repeated 1-vs-2 tests measured zero gradient, replica, and final-weight difference.
• Work-conserving actors: independently ready envs are shape-bucketed into bounded microbatches instead of waiting on the slowest member of a lockstep half.
• Production topology: GPUs 0-2 train continuously; GPU 3 performs nonblocking eval and best-checkpoint promotion. Three-GPU stage-4 production sustains roughly 76-82 decisions/s versus 20-29 on one GPU. A six-update four-training-GPU stress passed at 74-83 decisions/s with only 8 envs/GPU.
• Async eval: stage-4 eval runs without pausing training and atomically maintains best_eval.safetensors. A two-seed pass promoted score 0.88; larger passes remain the confidence gate.
• Old-model paired benchmark: the engine supports one controllable policy, so this is an indirect same-seed scripted-bot comparison. Over eight stage-1 seeds, current Rust beat ppo_v7 placement score by +0.458 (95% bootstrap CI [0.25, 0.625]). Versus ppo_v5 it led score by +0.167 but the CI crossed zero and v5 recorded one win; that comparison remains inconclusive.

Plan and gates

1. Finish the final one-GPU stage-2 soak. Gate: zero CUDA errors, zero NaN-guard events, healthy entropy, bounded/recovering value loss, periodic checkpoints, and enough completed episodes to judge reward. The candidate lives in draft PR #14, stacked on CUDA lifecycle PR #13.
2. Paired Python benchmark: done. Same A40, stage 3, 48 envs, rollout 32, two epochs, minibatch 128, fine+coarse AE, foveation, eval off. Python no-compile stabilized at 392 game-ticks/s = 26.1 decisions/s (roll=43.6s, update=83.0s). Rust cached-bf16 stabilized at 34.2-34.7 decisions/s (collect~44.4s, train~18s). Python compile was not a usable comparison on this dynamic workload: it remained in Dynamo tracing for many minutes without completing update 1.
3. Keep one GPU until the paired benchmark and soak pass. Then extend persistent learner ownership to multi-GPU gradient synchronization. Do not simply turn four GPUs back on around the legacy cross-thread learner path.
4. Scale by measured stage. Keep 48 envs/GPU in early/mid curriculum; reduce env count for Asia/World if memory or tail latency demands it. Optimize decisions/s/$, not snapshot utilization.
5. Quality gate against older models. Fixed-seed greedy eval, stage-clear time, recent reward, entropy, value-loss band, KL/clip fraction, and direct matches against frozen v5/v7 checkpoints. Speed alone does not promote a trainer.

Status snapshot (July 6th evening)

One PPO run is live: ppo_v3, trained from scratch on curriculum v2, now at stage 4 (30 bots, three maps) with a 60-70% rolling win rate. Its rival ppo_v2c, warm-started from v2b weights, was retired Jul 6 evening after the fresh run overtook it by a full curriculum stage. Both produced genuine engine wins on stage 0 (1v1 vs one nation on Onion), the first time the win condition has actually fired, partly because the curriculum now makes wins reachable and partly because win detection was silently broken until today (bugs). A parallel experiment is training a behavior-cloning warm start from archived human games. The full observe→act→reward loop, a 7-map win-gated curriculum, restart-proof cloud training, and a complete visualization suite (real-client replay videos with a live model-debug overlay, plus live play against the agent) are in place.

On the observation side, the encoder bake-off (AE v3.1) is concluded: benchmarking the bot-trained AE on real human games exposed a 16-point border-accuracy gap, and parallel ablations found the fix: latent resolution, not channel count. Final standings on uniform-crop eval: d8 (64ch @ 1/8) at 89.3% human / 96.1% bot borders, and the compact d8c32 (32ch @ 1/8) at 88.2% / 95.5%, one point behind at half the policy input. d8c32 cleared the pre-registered ≥88% bar, so it is the encoder for PPO v4. The channel-count control (c96) was pruned unfinished: its host kept dying and the resolution result had already answered the question. First deployment-style eval of ppo_v3 (fixed-seed local games, sampled actions): 2 wins / 2 losses on the first four stage-4 games, consistent with its training roll-win.

Jul 7 ~00:45: v4.2 deployed mid-run. Watching v4 replays exposed the phantom-boat exploit (43-78% of decisions were boats the engine silently discarded - see bugs). v4.2 fixes it three ways (honest masks, valid-tile snapping, wasted-intent penalty) and resumed ppo_v4 from its own checkpoint - no retrain needed, since no tensor shapes changed. The new episode/wasted curve opened at ~85-93 discarded intents per episode (stage 4), the measured size of the exploit; the penalty now prices those at noop-minus-w_waste. Watch items: wasted falling, boat's action-mix share normalizing, and a transient roll-win dip while the value head absorbs the reward shift (resumed at update 260, roll-win 0.28).

Numbers: 375k bot + 420k human full-state snapshots · AE border accuracy 71.8% → 89.3% (human) · 5.9M-param policy · 11 curriculum stages · first 2 engine wins Jul 6.

Run ledger

What is running, what is done, and what is stale. All RL/BC still loads frozen ae_v3.pt until the v3.1 winner is wired into rl/obs.py.

Is PPO actually improving? Yes, but the signal was hard to read. Placement-gated curriculum v1 looked like progress (roll-score ~0.65 on Pangaea) while the agent never won. Curriculum v2 reset both runs to 1v1, win detection was broken until 12:06, and both policies now win stage 0 in replay. By evening v3 had cleanly won the head-to-head: stage 4 with 60-70% wins vs v2c stuck at stage 3 with 7-17%, which is why v2c was retired. Neither lineage has been trained with the new encoder or BC warm start yet.

July 5th through July 7th, 2026

Complete chronological record (newest entries at bottom of list).

Timeline

• Jul 5 eve: headless datagen + tile AE v1. TS runner boots the real engine in Node, plays bot games, dumps packed tile grids + full entity state every 10 ticks.
• Jul 5 night: AE v2: one unified latent over ALL state. Spatial recon excellent; exact facts not (alliance pairs F1 0.67, troop ordering 0.81 after four tuning runs).
• Jul 5 23:14: AE v3: spatial-only + bypass split. The pivotal correction. 1.17M params, 64ch latent per 16x16 region, structures at P/R 1.0 through the latent.
• Jul 5 23:25 → Jul 6 00:36: RL scaffold → vectorized PPO → curriculum v1. Engine bridge over stdio, obs builder on the frozen AE, factorized masked heads, strength-index reward.
• Jul 6 morning: human-game pipeline (fetch archive, hash-verified deterministic replay to snapshots, HF upload) and restart-proof cloud training (full-state checkpoints, supervised restarts, HF checkpoint sync). Training hosts became disposable.
• Jul 6 10:27–11:32: GameRecord replays (bit-identical in the real client) and the live-lobby agent client: fight the policy from a normal browser.
• Jul 6 11:07–11:19: curriculum v2: map pools, 25% rehearsal, win-gated advancement, 1v1 → 1v3 start for dense wins.
• Jul 6 11:40: behavior cloning from human games (placement-conditioned; optional temporal transformer).
• Jul 6 12:06: win-detection bug found and fixed. Every win-rate stat before this is invalid.
• Jul 6 12:20: training speed overhaul: per-batch padding, pipelined rollout, bf16, binary IPC, per-stage decision cadence.
• Jul 6 13:04–13:39: real-client replay videos (actual game graphics, agent crowned as AGENT) and the MODEL debug overlay everywhere: rendered videos, browser replays, live matches.
• Jul 6 aft: AE reckoning: benchmarked the bot-trained encoder on human games (border accuracy 71.8% vs 87.5% on bot data), retrained on a bot+human mix, then designed and ran the v3.1 ablation matrix in parallel. Latent resolution won. New encoder + benchmark table uploaded to HF.
• Jul 7 early: the phantom-boat audit. Replay sidecars showed the agent picking boat on 43-78% of decisions (p(boat) up to 0.79, p(noop) 0.01) with nearly all of them silently discarded by the engine - the cap is 3 transports in flight. Root cause: a whole class of actions was "legal" per the mask but a guaranteed no-op at execution, and a discarded intent was reward-identical to noop with occasional upside - a free lottery ticket the policy learned to farm (ppo_v3's version was 61% build on random land). Fixed everywhere at once: masks now match engine state (canBoat = boat slot free + own shore, canExpand = neutral land actually borders, hasSilo now respects silo cooldown + spawn immunity), the translator snaps tile picks to tiles that can work (boats never target own/ally land, builds only own territory, ports own shore), and whatever still slips through is counted by the bridge (wasted, exact engine calls per intent) and penalized at W_WASTE=0.01 per discarded intent, so noop strictly dominates doomed actions. New episode/wasted TB curve. Tagged v4.2 and hot-deployed to the running ppo_v4 pod at ~00:45: trainer killed, repo synced, resumed from the same checkpoint (update 260, stage 4, win-window intact) - resume-safe because nothing about the network changed. First v4.2 episodes measured the exploit at ~85-93 wasted intents per episode.
• Jul 6 eve: v4: the whole policy stack moved to 1/8 resolution (d8c32 encoder, 8x8 tile regions, 150x250 grid budget) - the clean cut, no space-to-depth shims. GPU-side featurization (ego pooling, fallout unpack, AE encode, local crop batched in encode_grids), which also fixed a stale-fallout bug (dynamic fallout was riding inside the per-episode cached terrain tensor). Raw 64x64 local owner-crop bypass. Learned spawn placement end to end: spawn action + legal-region mask in PPO, spawn-phase snapshots + labels in replay sidecars (formatVersion 2) for BC. BC prefeaturize cache (scripts/prefeaturize_bc.py): 1.45 ms/sample single-threaded vs ~15-20 ms before. PPO v4 fixes: entropy anneal, stage LR warmdown, persisted win-gate window, periodic fixed-seed greedy eval. Pod supervisors got commit assertions and crash-loop backoff.

July 6th, 2026 - architecture and curriculum

Observation stack, reward, curriculum design (written this day).

Observation: compress the map, bypass the rest

The core architectural finding so far. Three iterations:

• v1: AE over tile ownership + terrain. Worked; too narrow.
• v2: one unified AE over all state (tiles, players, units, diplomacy). Spatial recon excellent, but tiny exact facts fought the bottleneck: alliance pairs peaked at F1 0.67 and troop ordering 0.81 across four tuning attempts (loss weights, pooled-vector capacity, per-slot latents, graph-to-encoder).
• v3 (current): compress only what is big: the map. Tile accuracy 99.4% (border-tile 87–99% by map), structures precision/recall 1.0 per class. Everything small and exact bypasses the latent raw: pairwise diplomacy, per-player scalars, transient units (nukes in flight with impact points), attack aggregates, legality masks.

The lesson: a one-bit fact reconstructed at 95% is strictly worse than reading the bit. Autoencoders are for high-dimensional state; never make exact small state fight the map for latent capacity.

AE details that held up: border-weighted cross-entropy (borders blur first and matter most), owner relabeling to static per-game spawn slots (any player count, fixed channels), fully-convolutional training on random crops (one model, any map size), and rarity-weighted BCE detection for structures: count regression collapses to all-zeros on 99.9%-empty grids.

AE v3.1: the border-accuracy push

The trigger: overall tile accuracy saturates near 99% for every model (water and player interiors are easy), so the honest metric is border-tile accuracy, and it wasn't in the training logs at all. Benchmarking the bot-trained v3 on the newly replayed human games exposed a 16-point domain gap: 87.5% border accuracy on bot data, 71.8% on human. Human territories are gnarly (naval invasions, enclaves, 50+ player fronts); nation bots grow blobs.

Two fixes ran in sequence. First, retraining v3 unchanged on a bot+human mix recovered the domain gap (80.1% human), so data was part of the problem. Then v3.1 attacked the architecture with seven changes at once plus two ablations to find the real constraint:

• decoder conditioned on static terrain (land + magnitude, never dynamic fallout), side information the policy gets for free, so the latent stops re-encoding coastlines
• nearest-upsample + 3x3 convs replacing ConvTranspose (no checkerboard), plus a full-res 3x3 refinement block before the classifier (was a bare 1x1, zero neighborhood context)
• border-density rejection sampling for training crops (uniform crops waste most gradient on ocean), focal loss on hard pixels, warmup+cosine LR, and native border-accuracy logging
• ablations: latent channels 64→96 at 1/16 res, vs. latent grid at 1/8 res

Result: resolution is the constraint, not capacity. Uniform-crop eval, 256 samples:

At the same 1/16 resolution, the v3.1 decoder/loss work helped bot borders but couldn't beat plain mixed-data retraining on human borders. Borders are high-frequency spatial detail, and a 64-number vector summarizing a 16x16 patch simply cannot store where a ragged front cuts through. Halving the patch to 8x8 bought ~9 points on both domains. Structure detection stayed at precision/recall 1.0 throughout. The catch: the 1/8 latent is 4x the policy input; a 1/8 x 32ch run (2x baseline) is in flight to see if the win survives halved channels, alongside the 96ch control.

Complementary hedge (decided): the policy will also receive a raw local owner-map crop around its own territory, exact borders where the agent acts most, latent for global context. The latent doesn't have to be pixel-perfect everywhere.

Action space

Full intent surface from day one, legality masking only (never curricular). Factorized masked heads over a shared conv trunk. One flat softmax is impossible (tile arguments alone are millions of options):

Masks come from exact engine legality calls each step; the policy can only pick what the engine would accept. Where a tile argument can still fail at execution (boat destination unreachable by water, build site too close to another structure), the bridge counts the discarded intent and the env charges w_waste for it - the mask keeps the policy honest, the penalty keeps the engine honest. Full intent table in DESIGN.md.

Reward

Iterated from raw land occupancy to a composite strength index: 0.45 x land + 0.25 x military + 0.30 x economy (shares). Land alone mis-scores legit strategies like tiny-island gold-stacking.

• dense: w_str x strength x timeweight, timeweight 0.5→1.0 over the first 8000 ticks. Strong late is worth double strong early, and it doesn't net to zero on death like pure deltas
• delta term for immediate credit (v4: symmetric 5.0; v5: gain 5.0 / loss 6.5)
• terminal: w_place x place^-pow (v4: pow=0.7; v5: pow=1.5), power law: 1st ≈ w_place, 2nd much lower at v5; the dead rank behind everyone still alive
• win bonus w_win = 30 (doubled when the curriculum became win-gated)
• small flat death penalty
• wasted-intent penalty w_waste = 0.01 per intent the engine silently discarded (doomed boat, invalid build site, expand with no neutral border): without a price these are noop-with-upside and the policy farms them (see bugs)

Curriculum

Eleven stages over seven maps (Onion → Pangaea → Caucasus → BlackSea → BetweenTwoSeas → World → Asia). Three principles, all added after watching v1 fail to progress:

1. Dense win signal first. Stage 0 is 1v1 vs a single nation on Onion; stage 1 is 1v3. Only then tribe bots and full rosters. Before this the win bonus essentially never fired.
2. Map pools, not single maps. The map is sampled per episode, so no overfitting one layout.
3. Rehearsal against forgetting. 25% of episodes replay an earlier stage's map at the current bots/difficulty. Old maps come back harder. Rehearsal trains but doesn't count toward advancement.

Advancement is win-gated: rolling win rate over the last 40 on-stage episodes must exceed 0.5. The agent must be winning more often than not, not merely surviving.

Prior art: AlphaFront

josh-freeman/openfront-rl (HF mischievers/openfront-rl-agent, "v19b") is the other PPO agent on this engine we're aware of, and a lot of our scaffolding follows from what they figured out first. The main fork is observation and tile choice: they keep a compact scalar view (~96 floats) and route spatial decisions through hand-coded heuristics (closest border tile, pickBuildTile()); we put map structure in the observation stack and learn tile selection with a spatial pointer (the bridge still snaps to legal tiles). Different tradeoffs: compact obs plus heuristics is simpler to train and deploy; richer spatial input is heavier but can in principle learn geography, chokepoints, and nuke aim.

What they showed works here, and we adopted: PPO converges on this engine at ~10 ticks/decision (we chose the same cadence independently); win-rate-gated advancement is viable. Still on our borrow list from their setup: LR warmdown on stage transitions; tiny flat generated maps as an even softer stage 0; a mid-game "beat all bots" gate so advancement doesn't wait on long endgames. On our side we've added a human-replay pipeline for behavior cloning: extra imitation signal that's awkward to fold into a purely scalar observation.

July 6th, 2026 - evening

Throughput engineering, progress graphs.

Performance

GPU util was poor and env stepping was the bottleneck:

• Per-batch dynamic grid padding. Everything was padded to the largest map (75x125 regions); Onion is 32x32, ~9x wasted conv compute on small maps. Obs stay native-size; collate pads to the batch max; pointer heads convert global↔local region indices.
• Double-buffered rollout. Env stepping (CPU/Node) and inference (GPU) were serial; now one env group steps while the other is on the GPU.
• bf16 autocast + TF32 + fused AdamW, plus a --compile flag.
• Binary tile IPC. Tile grids crossed the bridge as gzip+base64 JSON every step; now a JSON header + raw bytes on stdout.
• Per-stage decision cadence. Early stages decide every 15 engine ticks instead of 10.
• fp16 rollout storage + pinned staging + prefetch (Jul 6, 8a0da05). The single main thread was serializing fp32 copies, minibatch collation, and GPU compute; observations now ride as fp16, transfers go through pinned buffers, and a worker thread collates the next minibatch during the current backward. 3.6x on stage-3 maps (~590 → ~2100 game-ticks/s).
• v4.1 async rollout/update overlap (Jul 6 night, b8c331c). A collector thread builds rollout k+1 (env stepping + encode/act on a snapshot policy) while the main thread runs the PPO update on rollout k. One rollout of staleness; old_logp stays anchored to the acting snapshot so the clipped ratio math is unchanged. The former ~35s rollout phase now hides inside the update phase (perf/rollout_wait_s tracks what's left).

Progress graphs (Jul 6 night)

Generated by scripts/make_progress_graphs.py from the surviving TB event files and BC jsonl logs. ppo_v3's curves died with its pod (lesson: archive events before terminating); it appears as annotated reference lines from the devlog record.

• Curriculum: ppo_v4 cleared stages 0→3 in ~2h from scratch and reached stage 4 at ~3.6h - territory v2c never reached and roughly v3's pace despite v4 carrying the 1/8-resolution stack, learned spawns, and two mid-run restarts.
• Throughput: the per-update tps window decays as the curriculum moves to bigger maps (Node engine tick cost), so compare like-for-like: at stage 3-4 the fp16+prefetch fix took sustained tps from ~590 to ~2100, and v4.1 hides the rollout phase entirely.
• BC: the cache killed the data bottleneck (~1.5ms/sample vs 15-20ms), which moved the limiter to the GPU: bc_v4 runs ~56 ex/s at batch 24 x accum 4 on the 5090 (vs the pilot's 34 ex/s where the GPU sat idle waiting on gzip+JSON). Loss/accuracy now measure the model, not the pipeline. Loss is not yet trending down - that judgment (and the temporal transformer comparison) needs the full 60k-step run.

July 6th, 2026 - late night

Phantom-boat fix and the Rust port (~midnight).

Escaping the GIL: Rust hot paths + free-threaded Python (Jul 7 early am)

Every performance fix this project has shipped was some flavor of "get work off the single Python thread" (VecEnv processes, GPU featurization, pinned prefetch, the v4.1 async collector). Two structural moves land the endgame:

• rust/ofrs (PyO3 + rayon, 9f262a6). The two stable, allocation-heavy loops are now Rust with the GIL fully released: cache-bc frame decode (zstd + split) and the pad+stack collate that runs per PPO minibatch (2.6x single-threaded, and it no longer blocks other threads at all - which is what the prefetch thread actually needed). rl/native.py keeps numpy fallbacks so nothing requires the toolchain; pod scripts build the wheel opportunistically at boot. Formats are frozen (CACHE_FORMAT=1, obs v4), which is what makes these safe to freeze in a compiled language.
• Free-threaded CPython works, today, on our stack. torch dropped 3.13t but ships cp314t wheels (2.10+, CPU/CUDA/macOS); numpy and zstandard have cp314t wheels; orjson doesn't (we already fall back to stdlib json). Verified locally on 3.14.5t: torch 2.11 imports with the GIL genuinely off, rl.bc trains, and the BC sampler scales 322 → 1104 samples/s at 4 threads (vs 356 → 598 with the GIL - 1.7x from thread parallelism the GIL was eating). Two caveats from the bench: threads must own their samplers (shared hot objects ping refcounts across cores), and 8 threads regressed on the laptop (memory bandwidth), so pods should tune worker count.
• Migration stance: pods stay on the image's GIL python while runs are hot; the RunPod images pin torch to the driver. Next fresh pod generation moves to 3.14t (cu130 cp314t wheels exist), at which point thread-based feeding replaces the DataLoader-process workaround and the PPO collector/updater threads stop time-slicing.

Round two (8aa7fbd): the whole BC sampler is Rust now. The DataLoader process workers turned out to be a regression in production - 16 workers sat idle while the main process choked unpickling ~50MB raw batches off the IPC queues (stall 300s per 50-step window on bc2). The fix was to delete the abstraction, not tune it:

• ofrs::Sampler - sample_batch(n) does everything rl/bc_data.py did per sample (zstd frame/entity/step decode, serde JSON, the full obs-v4 featurization, label→head targets, legality forcing) in Rust, rayon across step draws, GIL held only to hand back numpy dicts. Laptop: 639 → 2631 samples/s vs the Python path; parity proven exactly (every array, 11.8k samples) by scripts/test_ofrs_parity.py. sample_windows(batch, k) draws a whole seq micro-batch in parallel for the temporal run. The trainer feeds from a plain thread pool again - no processes, no pickle.
• vec.py IPC off pickle - env workers ship a binary frame (u32 header-len ++ json ++ raw array buffers); the parent parses with ofrs::unpack_arrays. Numpy fallback speaks the same protocol.
• collate fully native - the ~8 small per-key np.stack calls now go through ofrs::stack_f32/f16 (parallel copy, GIL released), so the PPO prefetch thread's collate no longer serializes against the collector.
• Measured on the pods: bc2 14.9 → ~105 ex/s sustained (GPU 36% → ~90%, OOM-tuned to micro-batch 24 x accum 4); bc3 seq 4.5 → ~10.5 ex/s (window draws remain rejection-sampling-bound; next lever is a smarter end-picker, not more Rust); rl4 restarted clean on the new IPC with expandable_segments after an OOM at update 267 (fragmentation after 1135s uptime, not the new code path).

July 7th, 2026

Overnight check-in through the morning session.

Overnight results (Jul 7 morning)

• bc_v4 (bc2): step 6.5k → 17k overnight (~1M examples at ~95-105 ex/s). Holdout: acted-action 0.74-0.75, player 0.65, quantity 0.62, build 0.67-0.79, nuke 0.75; best-checkpoint metric climbed to 0.406. tile_region stays ~0-0.07 - the ~37k-class tile head is the clear weak spot, not new tonight. Known issue: throughput decays over hours (100 → 25 ex/s, RSS 23GB, stall climbing) and a trainer restart fully restores it - smells like slow accumulation in the feed path; restart-on-decay is the mitigation until it's profiled.
• bc_seq_v4 (bc3): ~3k steps at ~10 ex/s. First holdout evals: acted-action ~0.50, player 0.59-0.60. Behind the feedforward run at the same wall-clock, as expected at 5x less data; the temporal-vs-feedforward call needs matched sample counts, not matched hours.
• ppo_v4 (rl4): reached update 433 / 1.35M steps, stage 4. Headline concern: rolling win rate fell 0.28 → 0.00 over ~120 updates while placement score held 0.72-0.84 and entropy slid 9.2 → 2.8. The agent consistently places 2nd and has stopped converting to wins - looks like entropy collapse into a safe-second strategy (the win bonus vanishes from its experience, so nothing pushes it over the top). The binary-IPC deploy is exonerated: a pod-side round-trip probe preserves the win bool, and the decline started before the deploy. Candidate levers for the next session: entropy-coef floor or schedule, and the v5 asymmetric strength shaping.

PPO v5: breaking the safe-second attractor (Jul 7)

ppo_v4 was killed at update 433 (final checkpoint archived to HF) and ppo_v5 launched on the same pod with three MDP/optimizer changes aimed squarely at the collapse, plus a BC warm start:

• Adaptive entropy floor (--ent-floor 3.5): a multiplicative controller on the entropy coef (x1.3 per update when measured entropy is under the floor, capped at x30, hysteresis at 1.4x floor). v4's entropy slid 9.2 → 2.8 at a barely-annealed coef of 0.009 - the schedule alone cannot prevent collapse, only a feedback loop can. The scale persists through checkpoints and multiplies the existing anneal.
• Sharper placement curve: PLACE_POW 0.7 → 1.5. Terminal rewards for 1st/2nd/3rd go from 45/9.2/6.9 to 45/5.3/2.9 - second place lost 42% of its value, so "safe second" is no longer a comfortable optimum even before the win bonus enters experience.
• Loss-aversion delta shaping (v5 wishlist): W_DELTA splits into GAIN 5.0 / LOSS 6.5. v4 games showed hyper-expansion then collapse (103k tiles at t8001, dead by t10711); symmetric shaping priced a lost tile the same as a gained one, so defense was never worth learning.
• BC warm start: INIT_BC=bc_v4 pulls bc_best.pt (holdout 0.406) from HF, loads the 70 shared Policy tensors and folds the winner-bucket conditioning into the head biases.

Early read (30 min in): stage 0 cleared in ~25 updates, stage 2 by update 64 - v4 needed hours for the same ground. Entropy holds ~4.8 with the floor controller live (ecoef oscillating 0.010-0.048 as the controller breathes). Deliberately NOT in v5: the LSTM/temporal policy - bc3's seq-vs-feedforward comparison is the evidence gate for that, and it hasn't reached matched sample counts yet.

bc2 decay hunt + encode as the real feed cost (Jul 7)

• Instrumentation first: bc.py now logs per-phase feed seconds (smp/enc/col) plus CUDA allocated/reserved every 50 steps. First readings overturned the "sampler accumulation" theory: smp ~0s (the Rust sampler is essentially free), col ~5s, enc 50-65s per 50-step window - the frozen-AE encode is the entire feed cost, and it shares the GPU with the trainer.
• bf16 autocast on the frozen AE encode (rl/obs.py): free conv throughput - the latents already ship as fp16 to PPO. bc2 recovered to ~85-93 ex/s; bc3's seq run is fully encode-bound (enc 153s of a 163s window, 8 frames per sample) and sits at GPU ~95%: the next real lever there is caching AE latents in cache-bc (the encoder is finally frozen at d8c32, so the cache won't be invalidated - v4's reason to skip it is gone).
• Decay watch - caught in the act (30 min after restart): bc2 slid 93 → 35 ex/s again, and the instrumentation pins it: enc grew 49 → 135s/window while smp stayed 0.0 and col ~9s; CUDA reserved is flat at 27.5GB; GPU util 31% with clocks at 2872/3090MHz, 43°C, 202W. So it is NOT fragmentation, NOT thermal throttle, NOT the sampler - something CPU-side inside encode_batch (the single-threaded np.stack H2D staging of full-res owner grids) slows as the process ages. RSS was already 11.7GB at 34 min uptime (23GB overnight): prime suspect is glibc heap/arena churn from the hundreds of MB of short-lived numpy buffers per batch - consistent with a restart fully fixing it. Experiment running: MALLOC_ARENA_MAX=2 on bc2. Real fix if confirmed: have ofrs return pre-stacked batch arrays so the Python-side memcpy churn disappears.

v5.1 denser rewards + bc2 decay root-caused (Jul 7 midday)

ppo_v5 status at the time: not stuck so much as slow - roll-win crept 0.00 → 0.17 and roll-score 0.29 → 0.85 on stage 4 by update 260, but the gate needs >0.5 wins, and the run was also OOM-crash-looping every ~1h (CUDA OOM in group_norm/conv2d at 96 envs on the stage-4 map pool; the supervisor restarts and resumes, so progress survives but momentum doesn't). Two changes:

• v5.1: structures join the strength blend. The old blend (land/mil/eco 45/25/30) had a real mis-incentive: buying a building read as a strength loss - your gold share drops and the building counts nowhere, so the loss-averse delta actively punished construction. Now K_BUILD=0.15 adds a structure-value share (blend 40/20/25/15): each player's finished structures are valued by in-game cost ceilings (City/Port/Factory/Silo 1.0, DefensePost 0.25, SAM 3.0, × level) and shared over living players' totals. Completing a building now pays an immediate W_DELTA_GAIN spike, losing one to a nuke costs W_DELTA_LOSS, and holding a built-up economy earns dense reward every step - the reward is denser exactly where the action space was silent.
• bc2 decay: root-caused and fixed fundamentally. The faulthandler dumps caught the prefetch thread in np.stack at rl/obs.py:449 - the terrain_static stack, ~460MB of fresh float32 per batch (19MB/sample × 24), freed immediately. Allocations that size go through glibc's mmap path, so every batch paid mmap+munmap page faults; on a host with badly fragmented physical memory (65M compaction stalls, 115B failed page migrations in vmstat) the fault path itself degrades - a fresh-allocation memcpy microbenchmark on the degraded box ran ~300MB/s cold vs 1.6GB/s warm. That's why restarts fixed it (fresh page tables, warm buffers) and MALLOC_ARENA_MAX didn't (arenas don't govern mmap'd allocations). Fix: encode_grids now stacks the four big full-res arrays into persistent per-thread staging buffers (np.stack(out=...)), keyed by array name+shape and grown to the largest batch seen - the pages fault once and stay warm, so there is nothing left to decay.
• Per-stage win gates + curriculum re-grounding (afternoon). Stage-4 telemetry showed the safe-second signature reforming under v5.1: roll-score climbed 0.29 → 0.84 and saturated while roll-win stalled at 0.17 - the agent reliably places 2nd but doesn't close out games. Diagnosis: the flat 0.5 gate let the BC-warm-started run clear stages 0-3 at 21/40 wins each, so it never mastered the close-out kill where wins are cheap (1v1 Onion). The gate is now per-stage - 0.9 / 0.8 / 0.75 / 0.65 / 0.55 for stages 0-4, tapering to 0.3 by stage 10 where winning is genuinely hard - and ppo_v5's checkpoint was reset to stage 0 (weights and optimizer kept, win window cleared): it must now re-earn the curriculum under dominance gates, with rehearsal keeping the old maps in the mix.

Reward MDP reference (Jul 7)

The dense + terminal reward is fully specified in rl/curriculum.py and applied in rl/vec.py. These panels are generated from the live constants by scripts/make_reward_graphs.py (re-run after any tweak):

• Terminal panel: v5's sharper PLACE_POW is the main anti-safe-second lever - 2nd place drops from 9.2 to 5.3 before any win bonus. The win bonus (+30) still dominates a outright win (45 total for 1st+win vs 5.3 for 2nd-only), but the agent has to actually win to see it; placement alone now pushes harder for the top.
• Dense panel: most steps are small (strength ~0.1-0.3 at timeweight 1.0 gives ~0.002-0.006/step); the delta term is where expansion and collapse show up (±0.05 strength swing ≈ ±0.25-0.33 per step). The heatmap is the tuning surface for "how aggressive should expansion be?" Loss-aversion tilts the agent toward defending a lead.
• Scenario panel: synthetic strength traces (not real games) show total return under the old vs new MDP. Safe-second timeout pays similarly in dense terms but v5 cuts the terminal; expand-then-collapse gets punished twice (asymmetric deltas on the way down + worse placement). Waste spam shows why W_WASTE exists: without it, doomed boats are free lottery tickets.
• Strength panel: the composite index is not raw land - eco islands and military rushes can score honestly. Tuning K_LAND/K_MIL/K_ECO shifts which strategies the dense signal rewards mid-game.

July 7th, 2026 - evening

v5.1 freeze postmortem, the OOM root cause, temporal-BC verdict, v6 plan.

ppo_v5.1 frozen: the MDP worked, the crashes ate the wins

ppo_v5 is frozen (run killed tonight; final snapshot at update 550, global_step 1,725,216, stage 4, roll-win ~0.25 against the 0.55 gate - it advanced from ~530 during shutdown). The verdict on v5.1 is that the reward and gate design did their job: after the curriculum reset it re-earned stages 0-4 rapidly under the dominance gates, the structure-value reward (K_BUILD) is visibly expressed in play (see the behavior analysis), and it cleared the stage-5 win gate twice. Both advances were lost to CUDA OOM crashes before a checkpoint captured them - the run kept dying at exactly the moment it succeeded. The frozen weights become the v6 warm start.

OOM root cause: collate padding, not fragmentation (36a8efb)

• The real cause. NOT fragmentation - expandable_segments was already on. collate() pads every update minibatch to its largest grid, so one BetweenTwoSeas sample (132x223 latent cells) padded all 128 samples to 29k cells, and loss.backward() needed multi-GiB activation allocations - precisely at stage-5 entry, where the big maps first appear. The crash wasn't random; it was structural to advancing.
• The fix: zero padding waste. Update minibatches are now split by native grid shape, with a 1.6M latent-cell budget per forward/backward (MAX_UPD_PIX in rl/ppo.py); sub-batch losses are weighted by sample fraction so the accumulated gradients are numerically identical to the padded batch. Result: GPU 30.2 → ~25GB, throughput ~353 → ~430 game-ticks/s, zero OOMs after deploy.
• Checkpoint on curriculum advance (same commit): a stage advance now flags an immediate save at the end of the in-flight update instead of waiting out the 10-update cadence. A cleared gate can no longer be lost to whatever kills the process next.

The "always 20% attacks" mystery: resolved by measurement

Replays kept reading as if the agent always attacks with ~20% of its troops. New quantity-mix telemetry (TB + analyzer) settles it: the 5-bucket quantity head ({5,10,25,50,100}%) is alive and differentiated, favoring 10% and 25% attacks, with 100% almost never sampled - and a stream of 10-25% attacks reads as "about 20%". Coincidentally 20% is the engine default when troops is null, but our intents always carry explicit counts. Nothing was broken; the eyeball estimate landed on the mixture mean.

Behavior analysis: where the policy actually is

scripts/analyze_policy_behavior.py, run on the rl4 GPU against the update-490 checkpoint, 4 sampled episodes per stage:

• Stage 3 win 3/4, stage 4 win 3/4. The winning pattern: expand early, mass troops, then snowball 200k → 900k tiles with streams of 10-25% boat+attack intents while building continuously. One stage-4 win ended with 59 cities, 21 ports, 6 silos - the structure reward is clearly expressed.
• Stage 5: 0/4. Dies tick 3,400-5,200 before establishing any economy (zero structures in 3 of the 4 deaths), even on rehearsal Onion. The wall is the Easy → Medium difficulty jump, not map size.

bc_seq_v4 killed + the BC slow decay finally fixed (9c030f7)

• Temporal BC loses, decisively. bc_seq_v4 (temporal BC, seq=8) killed on the bc3 A100: holdout action accuracy 0.48-0.60 vs the feedforward bc_v4's 0.65-0.70, at 8x the encode cost per sample; the A100 sat at ~0% utilization (~10 ex/s). Lesson: temporal BC is only worth revisiting after AE-latent caching makes sequence windows nearly free.
• BC slow-decay root cause, one level deeper. glibc malloc's dynamic mmap threshold caps at 32MB while batch buffers are 50-80MB, so every batch paid mmap/munmap plus zero-page faults; bc_v4 decayed 88 → 12 ex/s over ~5h, with the stall moving to collate after the earlier obs.py staging fix. Fix: MALLOC_MMAP_THRESHOLD_=256MB and MALLOC_TRIM_THRESHOLD_=256MB in both pod launchers; bc_v4 restarted at ~100 ex/s.

Next: v6 (plan written)

Highlights of the written plan:

• Action space 14 → 21, covering every mechanical human intent: upgrade_structure, move_warship, cancel_boat, delete_unit, embargo_stop, target_player, alliance_extension. Warship joins BUILD_TYPES.
• Nuke head 3 → 5: Atom/Hydrogen × arc direction + MIRV. The arc flips the parabolic flight path for SAM evasion; MIRV ignores it.
• Quantity head becomes a scalar: a 0-1 Beta distribution replaces the 5-bucket softmax.
• BC sidecar re-replay for new-action labels.
• Extend-and-copy warm start from frozen ppo_v5.
• Per-update state.json to make the rolling win window restart-proof.
• AE-latent RAM-LRU cache to make BC ~10x faster: encode is 90% of BC wall time, re-computing a frozen deterministic function.

July 8th, 2026 - daytime

v6 first night, the honest cache verdict, the v6 label pipeline, and the 100x scaling plan.

ppo_v6 launched: stages 0-3 in one night, then the stage-4 wall

ppo_v6 went live on rl4 at ~11pm Jul 7, warm-started from frozen v5 via --init-extend: 62 tensors copied verbatim, the action head extended 14 → 21, build 6 → 7, nuke 3 → 5 (new logits initialized near zero so existing behavior is preserved), and the Beta quantity head fresh - the one component with no v5 ancestor. It cleared stages 0-3 overnight under the dominance gates (0.8 / 0.75 / 0.65 win) and hit stage 4 by morning. There it sits at the wall: update ~743, roll-win 0.10 against the 0.55 gate, roll-score 0.88 - it dominates games but is weak at converting them, the same safe-second pattern v5.1 showed at this stage. Entropy settled 8.9 → 7.6 over the night with the floor controller live - no collapse, the extended heads are being explored.

Live-play stack verified against v6: scripts/play_live.sh works end-to-end with the new action space - the agent joins a real lobby and plays full games.

Behavior analysis: a genuinely new playstyle on night one

scripts/analyze_policy_behavior.py against the update-190 checkpoint, 4 sampled episodes per stage: wins 100 / 75 / 75 / 25 / 0% on stages 0-4. The interesting part is not the win curve, it's that v6 is not playing like v5.1:

• Boat-dominated: boats are 54-73% of actions across stages (v5.1 mixed boats and land attacks much more evenly), and cancel_boat - a brand-new action - is already used as an active hedge, up to 14% of actions at stage 4: launch, reassess, recall.
• Warships are real: 13% of builds, with move_warship following coherently after them rather than firing at random.
• Build mix shifted: Defense Posts (48%) + Factories (22%) dominate, where v5.1 leaned cities/ports - consistent with the loss-averse deltas finally having a defensive action vocabulary to express themselves through.
• The Beta quantity head is healthy and differentiated on night one: boats commit 20-50% of troops, attacks 10-30%, with contextual shifts rather than one fixed fraction - the scalar head is doing on day one what the 5-bucket softmax took v5 its whole life to approximate (the "always 20%" mystery).
• Nukes fully dormant: 0 launches, 1 silo built all night. The 5-way nuke head has yet to enter the policy's experience; worth watching once economies get bigger.

Boat-churn and the outcome-only reward rule

The boat numbers above are not a playstyle, they're an exploit. 54-73% boats with cancel_boat at 14% of stage-4 actions is launch/recall churn: the agent sends boats out and calls them home in loops, and the roll-score 0.88 / win 0.10 split is the signature of learned busywork rather than closing out games. The rule this hardens into (rl/curriculum.py header): punish illegal or wasted actions, reward outcomes (state), never individual actions. Any reward attached to an action invites pairs of legal actions that cancel each other out and farm the shaping instead of the game.

The audit found no explicit per-action bonus to delete - the MDP was already outcome-based (strength blend + deltas + terminal placement/win) plus the W_WASTE illegality tax, and no-ops were never penalized. The churn reward was hiding in measurement: the strength blend's military share read player.troops(), the home pool only, and the engine deducts troops from the pool the moment a boat or attack launches, refunding them (minus the 25% retreat malus) on recall. So "send boat" scored as a strength loss and "cancel_boat" as a gain - a per-action reward pair in disguise - and boats doubled as a free troop bank, since troops parked on them dodge the maxTroops regen throttle. Fix: strengths() now counts troops in the field (aboard transports, committed to outgoing attacks) toward the military share. Launching and recalling are reward-neutral state moves; only real losses - combat deaths, the retreat malus - move the blend. W_WASTE still prices intents the engine silently discards (a cancel_boat with no boat in flight stays wasted), constants and the reward-MDP table are otherwise unchanged. Lands on rl5 at its next restart from master; watch the boat share of the action mix and the win conversion at stage 4.

The honest cache verdict: promised 10x, delivered ~2x

The v6 plan promised the AE-latent cache would make BC ~10x faster, on the theory that encode was 90% of wall time. Live numbers: bc2 peaked at 128-145 ex/s at 33-48% cache hits, bc_v6 runs 110-118 ex/s at 57-66% hits - call it ~2x over the ~60-100 ex/s pre-cache baselines. The profiler found why: at BATCH=96 the GPU fwd/bwd step itself caps at ~196 ex/s, because conv FLOPs scale with the padded batch-max grid. Encode was only part of the story; the model step was always going to cap throughput near there, and the cache merely exposed it.

• Fixes landed for the next restart (63d6e1a): disk-persisted z-cache (restarts stay warm instead of re-encoding from zero), a GPU-resident hit path (cache hits never round-trip through host memory), and area-bucketed batches (batchmates share grid sizes, so padding waste collapses - the collate-padding lesson again, this time for throughput instead of OOM).
• In flight today: bf16 autocast + torch.compile per shape bucket + dynamic area-based batch sizing (256+ effective) in rl/bc.py. Expected compound on top of the ~196 ex/s cap: ~500-650 ex/s ≈ 5-6x. Measured by a 500-step A/B on bc3 after its natural supervisor restart.
• Lesson: verify the ceiling before promising the multiplier - the second bottleneck is always waiting behind the first. "Encode is 90% of wall time" was true and still didn't make 10x true, because removing it just promoted the model step to the new ceiling.

The v6 label pipeline: 303 games re-replayed, 8h → ~2.5h

The seven new v6 actions need BC labels, which means a full re-replay of the human archive: 303 games to formatVersion 3 sidecars. First estimate was ~8 hours - because the replay driver ran 16 workers each batching 4 games, leaving a 128-core box ~95% idle. Fixed to 64 per-game workers plus a restart-safe --rebc skip that ignores already-refreshed games (d1ff455): ~2.5h wall for the full archive.

• One casualty: an EIO crash on the mfs network volume during prefeaturize. Recovered incrementally thanks to the skip logic - 212 games refreshed in the second pass, 0 empty sidecars. The cache-bc-v6 tars (29GB) are on HF, so no future pod ever replays this archive again.
• bc_v6 is the consumer: fine-tuning on bc3 from bc_v4's step-55k checkpoint via its own --init-extend (e6245da - the PPO head-extension machinery reused for BC). Step ~30k by midday, holdout evals already at bc_v4-final levels, and the Beta quantity target trains to MAE 0.096 - the scalar labels are cleaner than the old bucket argmax ever was.

The scaling plan: BC to ~5x, PPO to 10x then 100x

Written today after profiling both trainers. Where the time actually goes: on BC the feed is solved and the GPU step is the ceiling (above); on PPO the rollout (52s) already hides inside the update (82s) via the async collector, and the engine is coasting (384 cores, wait 0.0s) - so scaling means scaling the learner first, then actors. PPO currently does ~400 game-ticks/s at stage 4 on one 32GB GPU; target ~4,000 (10x), stretch ~40,000 (100x). Three phases, each shippable alone:

• Phase A - learner code wins (days, ~1.5-2x, no new hardware): bf16 autocast + torch.compile on the update path, epochs 4 → 2 with minibatch retuning (PPO tolerates this at our batch sizes; KL/clip-frac logging added so the gate is measurable).
• Phase B - one fat node (a week, ~8-15x cumulative): DDP the update across a 4-8x H100/A100 pod over the existing minibatch loop (the pixel-budget sub-batching already shards cleanly), scale envs 96 → 384+ with rollout inference sharded across the same GPUs. No algorithm change - pure scale, de-risking Phase C.
• Phase C - async multi-node actor-learner (2-4 weeks, the 100x path): evolve, not rewrite - rl/actor.py (extracted rollout + bridge + encode, shipping trajectories as zstd fp16 latents, ~1-2MB/step, version-tagged), rl/learner.py (the existing update loop + staleness handling: PPO's own ratio clipping + a hard version cutoff first, V-trace only if measured staleness demands it), rl/transport.py (ZMQ push/pull, no broker). Actors stateless and spot-friendly. Cost sketch: 4x H100 learner + 4 cheap actor pods ≈ $16-20/hr chasing ~30-60x. Gated on v6.1 running stably for ~a week.

Executing now: v6.1 = Phase A + Phase B together, launched as one new run on a 4x H100 pod (rl5) resuming ppo_v6's weights - same action space, direct resume, stage continuity preserved. See the run ledger for the verification gates.

v7 direction (parallel, non-blocking): foveated two-stream observation. Uniform /16 resolution was rejected by the old ablation - border accuracy 89.3% at /8 vs 78.5% at /16, "resolution is the constraint" - but that measured uniform resolution. Strategy-level context doesn't need pixel borders: v7 pairs a coarse /16 global stream (4x fewer cells on the maps that dominate late training) with a fine /8 stream restricted to own territory + the border band, extending the existing exact-borders local-crop machinery in rl/obs.py. Near tile targets resolve on the fine stream; far targets (nukes, cross-map boats) tolerate coarse regions. It cuts update AND rollout AND (later) transport cost - the single biggest code lever - but it changes the observation space and trunk input, so it needs an AE /16 fine-tune and an eval-parity gate (stage 4-6 win rate within noise) first. That's why it's v7's headline change, deliberately NOT in v6.1.

Rejected outright: feeding raw full-res maps into the policy (no AE) - it multiplies the measured bottleneck (per-latent-cell update FLOPs) ~64x, and weight pruning doesn't reduce dense conv time on GPUs. Submanifold sparse conv over active cells stays a stretch option (~3-5x ceiling, heavy complexity), unscheduled unless post-foveation profiling still shows dead compute.

Naming note (resolved late Jul 8): the "v7" tag briefly meant two different things - the foveation idea above, and the same-day full-state observation expansion in the next section. Since no v7 run had launched, the scope was merged rather than split: v7 = obs expansion + foveation, one version, one from-scratch retrain. Both changes break every checkpoint anyway, so shipping them separately would have cost two retrains and two BC-cache formats for no benefit. See the amended v7 scope.

July 8th, 2026 - evening: browser-side live play

ONNX in the browser, headless Chromium agent, MODEL overlay restored, spawn re-roll bug.

Live play on the homelab showcase no longer needs a server GPU/CPU model process. The pipeline:

• scripts/export_onnx.py exports the frozen AE encoder + policy to ONNX (dynamic map axes; TransformerEncoder nested-tensor / MHA fast-path disabled so the trace matches PyTorch numerically).
• openfront/src/client/webbot/: TypeScript ports of observation featurization (obsCore.ts / features.ts), categorical+Beta sampling, and intent translation, plus an onnxruntime-web session. Inference runs in a dedicated Web Worker so the main-thread WebSocket ping loop stays alive (~5s forward pass on World would otherwise trip the server's 60s ping timeout).
• URL trigger: ?webbot=<gameID> short-circuits the normal UI (Main.ts) and joins the lobby as AgentRL. Bot identity is isolated to sessionStorage so a human tab and a bot tab in the same browser don't fight over player_persistent_id.
• scripts/webbot_launcher.py: headless Chromium (Playwright) opens that URL; ofshowcase hub's /play launches it instead of rl.play. A small HTTP sidecar on --debug-port serves window.__webbotDebug so the existing MODEL overlay keeps working.
• Deploy: forked djmango/OpenFrontIO (upstream submodule we can't push to), pinned via openfront.commit + Dockerfile remote URL, rebuilt the showcase image on the homelab.

First live bug (spawn re-roll loop). During the spawn phase the action mask forced spawn every decision tick. Engine SpawnExecution treats every spawn intent as a re-roll: relinquish all tiles, then re-place. The bot teleported every ~10 ticks until a pick landed where getSpawnTiles returned empty (SpawnExecution: cannot spawn AgentRL), leaving it with 0 tiles → isAlive()===false → game over. Fix: gate spawn on !player.hasSpawned() (noop while waiting out the spawn timer), plus a translator guard that drops spawn intents once the agent already owns tiles. Python rl.play / PPO never hit this the same way because training episodes typically place once and the spawn phase ends quickly under the curriculum's start delay.

July 8th, 2026 - evening

BC verdict + freeze, playable v6_bc baseline, and v7 full-state observation expansion (plan, implementation, BC-regen ops saga).

BC was never really great - freeze, kill, keep a playable prior

Honest read across the whole BC line (bc_v1…bc_v6, plus the killed temporal bc_seq_v4):

• Ceiling is low. Feedforward holdout action accuracy saturates around 0.60-0.70 and then stops moving. bc_v6 spent ~35k steps (of a planned 60k) moving action acc from 0.60 → 0.62 and loss from 8.41 → 7.55 - two percentage points of action accuracy for a full A100 day. Temporal BC was worse (0.48-0.60) at 8x the encode cost. The human archive is small (~300 games), noisy, and heavily no-op / UI-default (quantity bucket 25% dominates because the client slider defaults there), so the prior learns "looks like a human clicking around" more than "wins games."
• As a PPO warm start it helped once, then stopped being the bottleneck. ppo_v5 cleared early curriculum stages fast off bc_v4/bc_best, which is the real win - BC buys a legal-action prior and a non-random spawn, not a mid-game strategy. Every later plateau (safe-second on stage 4, boat-churn, passivity after the fielded-troops fix) was a reward / curriculum / obs problem, not a "BC wasn't accurate enough" problem. Grinding BC from 0.62 → 0.65 would not have moved those.
• Infra cost dominated the learning signal. The z-cache was supposed to be 10x and delivered ~2x (cache verdict); glibc mmap churn, CUDA OOM on padded large-map batches, and finally an EDQUOT loop (disk z-cache budget 450GB on a 120GB volume - mfs reports cluster free space in df, so the quota is invisible until writes fail; 732b6c5 defaults Z_CACHE_GB=60) ate more calendar time than the accuracy curve justified.
• v7 makes the current BC weights unloadable anyway. The evening's obs expansion moves C_GRID 43→89 / P_FEAT 12→21 / N_SCALARS 8→11 (v7 scope), so there is no warm-start path from bc_v6 into v7.

What we did: froze bc_v6 at step 35,600 (ckpt already on HF as bc_v6/), terminated the bc3 A100 pod (fleet = rl5 only), and published a playable pure-BC baseline:

• v6_bc = bc_v6/bc_best.pt folded into a v6-shape Policy with the winner placement bucket (same fold as ppo --init). On HF as v6_bc/policy.pt.
• Play: bash scripts/play_live.sh --run-name v6_bc (now the script default). Or short PPO: python -m rl.ppo --init runs/rl/v6_bc/policy.pt … on a v6-shape checkout (pre-v7 C_GRID=43) - not against the v7 obs tree.
• No bc_v7; see the moratorium below.

BC moratorium (decided late Jul 8): no further BC runs until further notice. The trigger for the moratorium is honesty about evidence: the one claimed BC win - ppo_v5 clearing early curriculum fast off a BC warm start - was never isolated from the reward/curriculum changes that shipped alongside it. Before any bc_v7 is even considered, the existing BC artifact has to prove it does something useful. The unfreeze experiment is cheap and fully specified: on a v6-shape checkout, run two short PPO starts under identical seeds/config - ppo --init runs/rl/v6_bc/policy.pt vs from-scratch - and compare wall-clock to clear stages 0-3 and stage-4 roll-win at a fixed step budget. If the warm start doesn't show a clear, repeatable advantage, BC stays a frozen toolbox item indefinitely. Consequences now: ppo_v7 launches from scratch with no BC warm start; the homelab archive re-replay keeps running (raw sidecars are cheap, shape-agnostic, and useful for eval/analysis regardless), but the BC-specific prefeaturize + cache-bc/ upload steps are deferred, not queued.

Net: BC stays in the toolbox as a cheap prior hypothesis, not as a research track. Next accuracy wins come from obs (v7, now including foveation), reward (outcome-only / fielded troops), and scale (rl5 DDP) - not from another week of imitating 300 human games.

Closing the remaining blind spots (v7 part 1: full-state expansion)

Scope note (amended late Jul 8): v7 is now two changes shipping as one version - this full-state expansion (implemented, below) and the foveated two-stream resolution (designed, to build). Both break every checkpoint shape, so merging them costs nothing and saves a second from-scratch retrain.

v6's transient/entity streams omitted several things a human player sees that only started to matter once the bots got good enough to punish blindness to them. A plan was written and then implemented end to end in one session:

• Other players' land attacks - previously only boat/nuke arrivals showed up on the grid; ground invasions between two bots were invisible. New attack-front planes (TR_ATTACK_SRC, TR_ATTACK_RETREAT) rasterize every active attack's origin, any player, with log(troops) intensity, plus whether it's retreating.
• Unit ownership on the map - every transient unit plane is now ego-relative-split (own/ally/enemy) using the same clut ego-classing the static ownership grid already used, instead of one undifferentiated presence plane per unit type.
• Troop counts on boats, via intensity-encoded (not just binary) Transport/Warship/Train planes.
• Alliance-expiry timers - a countdown scalar per ally, ticks-to-expire normalized over a 3000-tick window.
• New unit types: SAM Missile, MIRV Warhead, Train, each a first-class transient plane; Missile Silo/SAM Launcher cooldown planes; an active-train-station flag on City/Port/Factory (rail network stays lightweight - no separate graph/edge representation, just the flag and the unit plane).
• Trade-ship destinations, per-unit health/cooldown, tile defense bonus (decoded from the engine's raw tile-state bit, exposed both as a region-averaged grid channel and a 5th local-crop plane alongside the existing exact-borders bypass), target-painting (a marked flag, mirrors the existing embargo-by-self column), troop/gold income rates, and the doomsday clock (in-clock flag + ticks, both per-player and a global clock-enabled scalar).

Architecture: every addition goes through the existing bypass, none through the AE. The frozen spatial AE (ae/model_v3.py) compresses only the map (ownership + terrain + static structures); everything above is either small-and-exact (player features, global scalars) or already-transient (unit planes), so it rides the same raw-channel/vector paths v3 established for nukes-in-flight and diplomacy. Consequence: this part needs no AE retrain, but no BC/PPO checkpoint survives - every shape constant moved (N_TRANSIENT 8→53, C_GRID 43→89, P_FEAT 12→21, N_SCALARS 8→11, N_LOCAL 4→5), so v7 is a from-scratch retrain, not a warm start (PPO only - BC is on moratorium). Note the foveation half of v7 does need an AE fine-tune; see below.

Engine surface + backward-compatible replay

obsCore.ts (live play) and datagen/common.ts (replay) both gained the same new fields, kept in parity, plus a hasFn feature-detection helper in datagen/common.ts: replay_all.sh checks out the archive's exact historical engine commit per bucket before replaying, and several of the new APIs (maxHealth(), troopIncreaseRate(), inDoomsdayClock(), hasTrainStation()) postdate some of those commits. Calling a method that doesn't exist yet at that commit would crash the replay instead of just omitting the field, so every new accessor is guarded and defaults sanely for old commits. The same defensive pattern runs on the read side: rl/obs.py pulls every new field with .get() + a zero/null default, so training keeps working against not-yet-re-replayed archive data during the migration window. The cached-frame format bumped (CACHE_FORMAT 1→2, adds a packed defense-bonus plane) with a stale-cache rebuild guard in scripts/prefeaturize_bc.py, and the Rust fast-path sampler (rust/ofrs/src/sampler.rs) stays frozen at the pre-v7 schema - rl/bc_data.py compares its compiled-in (P_FEAT, N_TRANSIENT, N_SCALARS) against the live Python constants and falls back to the pure-Python sampler on mismatch, so training is always correct, just slower until the Rust side is ported to match.

Foveated two-stream resolution (v7 part 2: designed, to build)

Decision (late Jul 8): the foveated observation is part of v7, not deferred to a later version. The original plan (see the daytime scaling entry) treated foveation as a follow-on because it changes the observation space and trunk input - but the full-state expansion above already breaks every checkpoint shape and forces a from-scratch retrain. Deferring foveation would mean a second from-scratch retrain and a second BC-cache format a few weeks later for zero benefit. Nothing is wasted by merging now: no v7 run has launched, the v7 code push is still held back, and prefeaturize hasn't started (and is deferred anyway under the BC moratorium).

Why it's worth it. The v3.1 ablation rejected uniform /16 latents - border accuracy 78.5% at /16 vs 89.3% at /8, "resolution is the constraint" - but that measured uniform resolution everywhere. Strategy-level context doesn't need pixel borders: what needs /8 precision is own territory and the contested border band; what's three map-widths away only needs to exist as a region. Foveation keeps the /8 fidelity exactly where the ablation said it matters and pays /16 everywhere else. On the giant maps that dominate late curriculum stages this is ~4x fewer global latent cells, and it cuts update AND rollout AND (later, Phase C) transport cost - the single biggest code lever identified in the scaling plan, bigger than anything left in Phase A.

Design - two grid streams replacing the single /8 stream:

• Coarse global stream, /16 over the full map: AE latents at /16 plus the same transient/bypass channel stack as today, region-averaged to /16. Every tile of the map is represented; nothing is ever fully invisible.
• Fine stream, /8, restricted to own territory + the border band: extends the existing exact-borders local-crop machinery in rl/obs.py (the same philosophy that already gives the policy pixel-exact borders and the 5-plane local crop). Carries the full C_GRID=89 channel stack at /8, plus one coverage-mask channel so the trunk can tell "empty" from "outside the fovea". Cells outside coverage are zeroed and masked out of any fine-stream pointer logits.
• Tile pointer heads go two-level: near targets (attacks, expansion, structure placement - the actions the ablation showed need precision) resolve on the fine stream; far targets (nuke aim points, cross-map boat destinations) resolve on the coarse stream, where a /16 region is already the honest precision of those actions. Concretely: the tile head picks a coarse cell first, then refines within the fine stream when the picked cell has fine coverage - one softmax over coarse cells, one masked refinement over the covered fine cells. Exact head wiring is the main open implementation question; the mask plumbing (rl/env.py legality masks per region) must follow whichever wiring wins.

What it requires:

• AE /16 fine-tune. The frozen encoder (d8c32) is /8-native. The coarse stream needs a /16 head - fine-tuned from d8c32 (a downsampling stage + short fine-tune, not a from-scratch AE), trained on the same bot+human mix. Quality gate below.
• Obs plumbing in rl/obs.py + the bridge: emit both streams; the fine stream's bounding box / coverage set is derived from ownership + border dilation, recomputed per decision step (it already is, for local crops). Bridge serialization gains the coarse stream; step payload shrinks on big maps.
• Trunk/conv input change: two spatial encoders (share conv weights where shapes allow) pooled and concatenated - the same fusion pattern as today, one more stream.

Gates before ppo_v7 launches on this:

• AE quality gate: the fine stream keeps d8c32's measured quality by construction (same encoder, same /8). The /16 fine-tune must hit region-level fidelity - structure precision/recall ~1.0 and ownership-region accuracy comparable to ae_v3_mix's /16 numbers - but is explicitly NOT held to the 89.3% border bar; borders live on the fine stream now. Benchmark table goes to HF next to the d8 one.
• Early-curriculum sanity: the original plan's eval-parity gate (swap obs under a trained policy, stage 4-6 win rate within noise) can't run - v7 is from scratch, there is no trained v7-shape policy to swap under. The honest replacement: ppo_v7 must clear stages 0-3 in wall-clock comparable to the v6 lineage (which did it in ~2h on 1 GPU), and per-update cost on the big-map stages must come in measurably under v6.1's. If either fails, the fallback is running v7 with the fine stream covering the whole map (= v7 obs at uniform /8, foveation off via config), which must remain a supported degenerate case.

Still rejected (unchanged from the daytime entry): raw full-res maps into the policy with no AE (~64x the measured per-latent-cell update FLOPs; pruning doesn't reduce dense conv time on GPUs), and submanifold sparse conv stays an unscheduled stretch option (~3-5x ceiling, heavy complexity) unless post-foveation profiling still shows dead compute.

v7's first mid-run fix: coarse tile picks only searched a quarter of their cell

Symptom (Jul 8 late): ppo_v7 on stage 1 showed boat-heavy noise again - lots of boats the engine discards, no boat-and-cancel play. Familiar smell (the v4 phantom-boat exploit, the v6 boat-churn), but the mechanism this time was new, and it was a genuine bug in the foveation change, not a reward hole.

Root cause: in the two-level tile head, only precision actions (REFINE_TILE: spawn/build/upgrade/cancel_boat/delete) refine down to the fine /8 grid. Boat, nuke, and warship targets stop at the coarse /16 pick, and _coarse_local_to_global maps that cell to its top-left /8 region - which the translator then treated as the whole target, searching just that 8x8-tile quadrant for a valid destination. Three quarters of every coarse cell was unreachable by boat, ever. Picks whose valid tiles sat in the other quadrants either snapped to nothing (translator-side wasted penalty, noop-equivalent) or forced the policy toward whatever cells happened to have targets in the lucky quadrant. The policy cannot learn its way around this: the information that a cell's target sits in the wrong quadrant isn't in the action space.

Fix (translator-only, no tensor shapes touched): region_tile gains a span parameter; boat/nuke/warship snaps search the full 2x2 region block (span=2), matching the /16 granularity the head actually decided at. Refine actions keep exact /8 snapping. Bonus fix while in there: boat_tile now prefers shoreline candidates (the engine resolves destinations via targetTransportTile → closestShore(owner, dst, 50), so a shore pick always resolves; inland only works when the owner has shore within 50 tiles by land), falling back to any valid tile. Fewer boats die at canBuild. The remaining engine-side reject (water-component reachability) stays priced by the wasted-intent penalty, and translator-failed snaps were already counted as wasted in vec.py - that incentive hole was closed in v4.2 and stayed closed.

Deploy: v4.2-style hot patch - repo synced on the pod, trainer killed, relaunched by the crash loop with --resume from the last checkpoint. Watch items: episode/wasted falling, boat share of the action mix normalizing, stage-1 roll-win resuming its climb.

Jul 9 audit: the entropy controller was the real stage-1 killer

A systematic pass over the training paths after the boat fix found the actual reason stage-1 roll-win decayed to 0.00 - and it wasn't boats. The adaptive entropy-floor controller was pegged at its 30x cap and structurally unable to come down. The log tells it cleanly: at updates 17-22 the policy started winning stage 0 (roll-win 0.40 → 0.85), which naturally dropped entropy to 3.3-3.5 - just under the 3.5 floor. The controller read commitment as collapse and ratcheted the coefficient 1.3x/update, 0.013 → 0.298 in 13 updates. The huge bonus then held entropy at 3.7-3.9: above the floor, but below the floor×1.4 = 4.9 release threshold - a dead band the bonus itself kept entropy inside, so the scale never decayed. For 100+ updates the entropy term (~1.1) dwarfed the policy gradient (~0.01-0.03); the optimizer was actively keeping the policy random.

Root causes and fixes (all in rl/ppo.py): (1) dead band: the scale now decays slowly (/1.05 per update) anywhere above the floor, not just above floor×1.4; (2) cap 30 → 5 (ENT_SCALE_MAX), and a resumed scale is clamped to the new cap so the pegged run recovers on redeploy; (3) no startup grace for fresh runs: the spawn-heavy-rollout grace only applied to resumes - now unconditional; (4) floor 3.5 → 2.5: 3.5 was tuned for the pre-v7 single full-grid tile softmax. v7's two-level head is structurally lower-entropy (coarse has 1/4 the cells, refinement ≤ ln 4), and a winning stage-0 policy sat at 3.3-3.5 - the old floor actively fought commitment.

Also shipped in the same pass:

• Update-path throughput: _fine_to_coarse_mask ran a per-sample Python loop with a GPU-syncing nonzero() inside every evaluate() minibatch (~24k syncs/update; upd 70-100s vs roll 14s). Replaced with a batched scatter.
• act/evaluate logprob consistency: when the fine mask falls back (no legal fine cells anywhere), the sampled fine cell can live outside the sampled coarse cell; act() now scores the coarse factor of the cell evaluate() will reconstruct (identical in all non-degenerate cases). Verified by 200-trial sampled parity test.
• Per-action coarse content masks: new coarse_has_land/water obs keys prune coarse tile picks - boats/nukes can no longer point at pure-ocean /16 cells, warship moves at landlocked ones. Less doomed-pick space to learn away by penalty.
• Webbot parity: the browser translator got the same span-2 + shore-preference fix (openfront submodule), since master's webbot serves v7 exports.

Env fleet auto-sizing (same night): post-fix GPU util read 20-40% on the 4x H200s with CPU load 17/128 - the box was barely working. Rollout runs on a collector thread fully overlapped with the update, so with roll ~16s vs upd ~68s the 384 envs sat idle ~75% of every cycle. The trainer now writes a suggested_envs hint into state.json every update (current envs scaled by the EMA'd upd/roll ratio, damped to ≤2.5x per restart, capped at 8/core and 2048), and pod_train.sh ENVS=auto re-reads it at every relaunch - restarts are the only point the fleet can resize, so the loop self-tunes across them. MINIBATCH=auto scales with the fleet (ROLLOUT×ENVS/12) so optimizer steps/update stay constant and bigger fleets mean bigger kernels, not more per-step overhead. ppo_v7 relaunched at envs 768 / minibatch 2048 (cold default; history takes over from here).

The real update-time culprit (found while scaling): more envs alone made updates WORSE (68s → 160s at 768) - v7's grid_fine is a per-sample coverage crop whose shape drifts every step, and the update's exact-shape sub-batch grouping (fine for the old per-map grids) splintered each minibatch into hundreds of single-digit sub-batches: the update was kernel-launch overhead, not compute. Fine dims now bucket up to multiples of 8 (collate zero-pads within the bucket, policy masks padded cells): >20x fewer sub-batches for bounded padding waste. A latent np.split abort also surfaced (auto minibatch rarely divides B_total evenly) - now array_split. Result at envs 1024 / minibatch 2730: update 68s → ~33s, game-ticks/s 2,650 → ~10,800 (4x), GPU util 20-40% → 36-72% at 380-460W. Rollout (~48s) is now the long pole; the auto-sizer will trim the fleet toward balance on future restarts.

BC-regen ops saga: three infra bugs before the first game replayed

Re-replaying the human archive (303 games, three engine-commit buckets) to backfill the new BC fields should have been routine - the same datagen/replay_all.sh pipeline as the v6 label regen. Getting it running cleanly surfaced three unrelated environment bugs:

• Submodule remote mismatch. .gitmodules points the openfront submodule at the djmango/OpenFrontIO fork, but the working repo's local origin had long since been repointed at the upstream openfrontio/OpenFrontIO - and at least one archived game's engine commit only exists on upstream's v32 branch, not the fork. A fresh clone (on the homelab) follows .gitmodules literally, so git checkout <historical-commit> failed with "pathspec did not match any file(s)", silently leaving the submodule on the wrong commit - the replay then ran the wrong engine version against an old game and desynced at tick 90. Fix: git remote set-url origin to the upstream URL on every fresh checkout of this repo, matching what the primary dev machine already runs. Worth eventually reconciling .gitmodules itself so new clones don't repeat this.
• macOS AppleDouble junk survives tar transfer. Copying the untracked webbot/ directories (needed because the submodule's working tree loses them on every historical checkout) via tar/scp from a Mac carried ._* resource-fork shadow files into the Linux submodule tree - harmless to the TypeScript build but noisy in git status; swept with a plain find -name '._*' -delete.
• The homelab's login shell is fish, not bash. Bash heredocs and && chains piped through a bare ssh host 'command' run under the remote login shell, so anything beyond trivial one-liners needs ssh host bash -s <<'EOF' ... EOF (stdin piped to an explicit bash) instead - the more common ssh host bash -c "..." also breaks past a couple of nesting levels of quoting once tmux is in the mix.

Earlier in the session, before settling on the homelab, the same job was attempted in an isolated git worktree on the primary dev machine to avoid touching the main repo's in-progress submodule state; that surfaced a fourth, homelab-irrelevant issue - the local Shell tool's background-process backgrounding has a hard ~15-18 minute lifetime independent of &/disown, which doesn't matter once the job runs inside a detached tmux session on a persistent remote host instead.

Once past those three bugs: a single-game validation replay passed clean (hash-verified, 10,751 ticks), and the full backfill launched in a tmux session on the homelab (20 vCPU, 47GB RAM - replay_all.sh defaults its worker count to nproc, one game per worker). Steady-state load ~18-25/20 cores. In progress as of this entry: bucket 1 of 3, single-digit desync failures expected and tolerated (the pipeline skips a failed game rather than blocking the batch) - one seen so far out of the first ~16 games, consistent with the occasional non-deterministic edge case rather than a systemic issue.

Found two GPU pods already running; kept one

Checking readiness for the BC/PPO retrain surfaced openfront-bc3 (1x A100) and openfront-rl5 (4x H100, ppo_v61) both still billing. bc3 was mid-restart into a torch.compile / EDQUOT crash loop on the stale bc_v6 run - already low-value (flat learning curve, see BC verdict) and about to be obsoleted by the v7 shape change regardless, so it was terminated after archiving bc_v6/ + publishing the playable v6_bc prior. rl5 is mid-flight on real progress (ppo_v61, update ~976, stage 4) and shares no state with the v7 retrain, so it stays running untouched - stopping a live, valuable, unrelated run to save ~$12/hr would be the wrong trade. General lesson for this kind of work: check what's already running and billing before assuming a clean slate - the pod list doesn't show up in any code diff.

Remaining before ppo_v7 can start (amended late Jul 8 for the merged scope + BC moratorium): the path no longer runs through BC data at all. In order: (1) implement the foveated two-stream obs (coarse /16 stream, fine-stream coverage mask, two-level tile head, bridge serialization); (2) run the AE /16 fine-tune off d8c32 and pass its quality gate; (3) push the combined v7 code to origin/master - push still deliberately held back, because pod_train.sh/pod_bc.sh bootstrap with git reset --hard origin/master and rl5's next restart would load v7 shapes against a v6.1 checkpoint and crash; the push therefore coincides with deciding rl5's fate (freeze ppo_v61 or pin its pod to the pre-v7 commit); (4) launch ppo_v7 from scratch, no BC warm start. The homelab archive re-replay keeps running to completion (the refreshed maps//bc/ sidecars get uploaded to djmango/openfront-human-games for archival/eval value), but prefeaturize + cache-bc/ are deferred under the moratorium and no longer gate anything. Until the push: local master stays at v7 for the homelab regen; rl5 stays on the last pushed commit.

GPU utilization push: 20% → 72%, then killed to move the hot path to Rust

Starting point: ppo_v7 on the 4x H200 pod was pegged at 20-40% GPU (140-195W of a ~700W card) with only 384 envs and CPU load ~13/128. The rollout/update loop is sequential and only the update touches the GPU heavily, so an idle CPU phase (rollout) was starving the accelerator every cycle.

What actually moved the needle, in order:

1. Env fleet auto-sizing. Rollout runs on a collector thread fully overlapped with the update; envs idle whenever roll < upd. The trainer now writes a suggested_envs hint into state.json every update (EMA'd upd/roll ratio, damped ≤2.5x per restart, capped 8/core), and pod_train.sh ENVS=auto/MINIBATCH=auto adopt it at every crash-loop restart.
2. Sub-batch bucketing. Scaling envs alone made updates worse (68s → 160s): v7's grid_fine is a per-sample coverage crop whose shape drifts every step, and the update's exact-shape grouping splintered each minibatch into hundreds of single-digit sub-batches - pure kernel-launch overhead. Bucketing fine dims to multiples of 8 cut that >20x.
3. Crash-loop unhang. A rank that dies by signal (SIGSEGV/SIGABRT) left orphaned env worker processes holding the log pipe open, so the restart loop's tee never saw EOF and the run sat dead in "restarting" indefinitely - this is what actually ended the session (see below). Fixed with a direct redirect + explicit reap, but not in time to save this run.
4. Batched coverage/content masks. Stack-sampled the main rank's Python stacks (RunPod drops CAP_SYS_PTRACE, so py-spy can't attach; used faulthandler + SIGUSR1 instead) and found _fine_coverage running .any()/nonzero()/int() per sample - 5+ GPU syncs and 3 host transfers each, ~2.5k syncs per act batch at 256 envs/rank. Rewrote as one batched kernel per op plus a host-side box search, verified bit-identical against the old path.

Result: envs 384 → 1024, update time 68s → ~25-30s, game-ticks/s 2,650 → ~10,000-13,000 (4-5x), GPU utilization 20-40% → peaks of 72% (avg still 12-50%, noisy across the 4 ranks). Never reached the 90% target.

Why it plateaued below 90%, and why that's a dead end in Python: even after killing every measured Python-side sync, the rollout half of the loop is still bound by VecEnv's Node.js game-engine subprocesses talking to the trainer over pipes - that's wall-clock the GPU cannot touch no matter how the featurizer or update batching are tuned. Every fix so far has been "find the next accidental sync/serialization in the Python glue"; there's no more slack to cut on that side, and each fix has been getting smaller (auto-sizing and bucketing were 2-4x each, batched coverage was incremental). The single-GPU control run (NPROC=1) that was supposed to establish the true non-DDP baseline never got readings before the pod's tmux session died with the rank's SIGSEGV and the crash loop hung (bug #3 above) - so the DDP-vs-single-GPU question is still open, but at this point it's moot: the environment step/serialize path itself, not DDP or the featurizer, is the actual ceiling, and that only moves by rewriting rl/vec.py's engine bridge (currently Node.js subprocesses + JSON/pipe IPC) in Rust with a real shared-memory or FFI hot path instead of pipes. This is the plan going forward - see the game engine's existing rust/ofrs native-path precedent (escaping the GIL) - rather than continuing to squeeze the Python side.

Run status: ppo_v7 killed at update 355 (stage 1, step 5.32M, roll-win 0.25). Final checkpoint + state.json synced to djmango/openfront-rl on HF before teardown - resumable once the Rust-backed vec env lands. The 4x H200 pod is being torn down; no GPU spend while the port is in progress.

July 9th-10th, 2026 - the Rust port: trainer, native engine, and the utilization ceiling that moved

PPO trainer ported to Rust/tch, then the game engine itself went native in-process. IPC is dead. Sustained GPU util 55% → 82% on a single A100, engine ticks 12-14k → 130-150k/s standalone. And one honest learning-signal finding that matters more than any of the speed numbers.

The trainer port (rust/oftrain, tch-rs/libtorch)

Full port of the PPO training path (rl/ppo.py + rl/vec.py + rl/obs.py + curriculum + translator + policy) to a Rust cargo workspace: ofcore (Python-free featurization/curriculum/translate), oftrain (bridge client, threaded vec-env - one OS thread per env, no GIL so no multiprocessing/pickle - tch policy net, PPO loop, checkpointing, nvidia-smi-polling util sampler). BC warm-start and AE training explicitly out of scope; policy sees raw pooled grid planes instead of an AE latent for now.

Two libtorch linking footguns worth never re-deriving: (1) tch 0.24 is hard-pinned to torch 2.11.0 - and not just by version string: its C++ shim calls ATen ops that don't exist in older torch headers, so it fails to compile (not link) against a pod's stock torch 2.8. Fix both locally and on pods: dedicated venv with pinned torch==2.11.0 (CUDA pods: +cu128 wheel), LIBTORCH pointed at it. (2) The big one: CUDA silently not linked. Nothing in our Rust references a torch_cuda symbol directly (kernels self-register via static initializers), so the linker's --as-needed drops libtorch_cuda.so entirely - Device::Cuda(0) "works" and then every real op panics. Known upstream (tch-rs #907/#1015). Fix: oftrain/build.rs re-asserts -Wl,--no-as-needed -ltorch_cuda -lc10_cuda from the binary crate; readelf -d to verify. Plus the returning NVRTC cameo: lgamma (Beta quantity head) needs libnvrtc-builtins from the separate nvidia pip package on LD_LIBRARY_PATH.

Rounds 1-3 on RunPod: the utilization number that lied, then the one that didn't

First A100 runs looked great - "98-100% GPU util" - until scaling to 4 and 8 GPUs exposed that those were instantaneous snapshots at the logging moment, not time-averaged utilization. The honest metric (min_mean_util, cumulative time-average across the run) said 36-41%: the GPU burst to 100% during the update and sat idle through the whole CPU/IPC-bound rollout. Same trap the Python ppo_v7 push fell into from the other side. Fixes that actually moved it: pipelined actor/learner (collect rollout N+1 on background threads with a frozen actor copy while the learner trains on N - classic one-step-lag IMPALA-style overlap, done with scoped threads and disjoint actor/learner struct borrows), GPU-resident minibatching (build full batch tensors once, index_select on-device per minibatch), flattened gradient sync for multi-GPU (manual replica averaging, "poor man's DDP" - 4-GPU scaled ~5x over 1-GPU). Result: 66-68% sustained, 330 → 560 decisions/s, and a measured dead end: fwdbwd was ~1s per 512-sample minibatch of genuine saturated compute, and the rollout side was pinned to the wall-clock of Node.js subprocesses talking JSON over pipes - the exact same ceiling Python hit. One instrumentation bug worth remembering: timing both phases from the same join point made collect_s == train_s on every line - time the inner work directly, and let the join cost land on the phase that actually waits.

Going native: the engine moves in-process

Instead of optimizing the pipe protocol (shared memory/FFI to the same Node engine - real work, throwaway destination), the call was to go all-in on the native Rust engine port (rust/engine, the rust-ofrs-fast workstream): the environment becomes a plain Rust object (RlSession mirroring bridge/env.ts's reset/step/countWasted), and oftrain grows a GameEngine trait with two backends - Node (the old bridge, kept for A/B) and Native (in-process, zero IPC). Porting the RL surface meant porting the full observation head (obsCore.ts entities/legality field-for-field), the TS percentage-based FFA win check (80% of total map tiles, checked every 10 ticks - not last-alive), missing intent executions (targetPlayer), and spawn-phase termination for singleplayer. One Rust wart: Game held a raw pointer for the execution list, making it !Send; documented unsafe impl Send after confirming single-thread ownership per session.

The outcome-parity campaign: hash parity was the wrong bar

Exact hash parity (every tick, every record, bit-identical) was costing more than it bought for RL purposes - what training needs is outcome parity: same winner, similar terminal conditions. New gate: replay all 78 archived human-game records through both engines, compare winner identity, terminal tick (±20%), winner land share (±10%); target 55/78 (70%). The TS side became a cached oracle (one-time expensive TS replay per record, resumable per-record shards, parallel child processes, fingerprinted by record-set hash + parity commit) so iterating on the native side never re-pays the TS cost. Then a long divergence-fixing grind, one root cause per commit, ~19 landed: win-condition semantics (FFA ordering, team aggregation, timers, bot suppression), random-boat target filtering and troop caps, nation defense-post sampling bounds, alliance-triggered attack retreats returning all troops, relation clamping to TS bounds + per-tick decay toward neutral + cross-alliance-only boosts, reciprocal alliance requests canceling in-flight nukes, manual boat retreat execution, island invasion target selection, city rail-connectivity placement scoring, and the warship suite (construction, patrol, shell targeting/flight, damaged/docked healing lifecycle, trade piracy + post-piracy patrol replan). The full-78 gate run is grinding in a pinned worktree as of this writing; per-cluster spot checks pass.

Native hot path: 4.9x single-thread, ~10x the old IPC ceiling

Standalone benchmark (rust/engine/examples/bench_rl_session.rs, no policy, no IPC): raw RlSession stepping went 9.3k → 45k ticks/s single-threaded, 51-58k → 130-152k ticks/s across 8 threads. The embarrassing hot spot: a scalar per-tile loop re-encoding the full tile plane to bytes on every step, eaten immediately by a decode back to u16 on the trainer side - ~78% of step time paying for a serialization format whose only consumer no longer exists. Fixes: memcpy-fast-path encoding (unit-tested byte-identical), Value::Array built straight from iterators (no intermediate Vecs), and a zero-copy tile_state() -> &[u16] accessor so the native backend skips bytes entirely. Correctness pinned by bit-identical obs JSON on fixed seeds before/after, and parity tests failing identically (same tick, same hash) on base vs. optimized - pre-existing oracle staleness, not regression. A rayon batched-stepping path exists behind a parallel feature but plain thread-sharding measured 20-40% faster - memory-bandwidth-bound, not scheduler-bound.

A100 validation: 82% sustained, and the law the sweep taught us

Fresh 1x A100 SXM pod, --engine native, 64 envs x 32 rollout, full policy: 81-82% sustained time-averaged utilization (the honest metric), flat through a 150-update / ~30min stability run, zero crashes, ~173 steps/s, collect_s ≈ train_s - collection and training genuinely overlapped and balanced. Every config swept told the same story from a different angle: 96 envs → 76-78% (collection scales linearly, no fixed overhead to amortize); 128 envs + foveation → 58-64% despite fitting more envs (cheaper obs = less GPU work = balance tips back to collection-bound); 192 envs + small policy (GC=128/BLOCKS=2) → highest raw throughput of anything tried (~285 steps/s) at the lowest utilization (37%); 256 envs full-res → OOM (the 250x150x63 obs batch alone is ~77GB at that scale - real ceiling, not fragmentation). The law: in this pipelined architecture, utilization tracks the ratio of GPU-bound train compute to CPU-bound collect compute - anything that makes the network cheaper buys throughput and costs utilization. Chasing the util percentage with a bigger network would be goodharting; decisions/s/$ is the metric that matters, and the small-policy config wins it by a mile if it learns as well per sample. Also landed flag-gated and GPU-verified: --amp (manual bf16 - tch has no bf16 autocast; halves activation memory, needs a clean same-minibatch A/B for speed), --foveate (real 48x48 crop centered on own-tile centroid, fully vectorized coordinate math, two broadcasting bugs caught by a dedicated unit test), --gc/--blocks, --pinned-h2d. Confirmed unreachable in tch-rs 0.24 and correctly skipped, not hacked: channels-last, fused/foreach optimizer, CUDA graphs.

The finding that outranks the speed work: stage 0 never wins

New OFTRAIN_DEBUG_EPISODES diagnostic dumped per-episode results and exposed that every episode in every config runs to tick-limit truncation: place 1/2, ~50 tiles, won=false, so the curriculum win-window never fills and training is permanently stuck at stage 0. Not an engine bug and not a PPO-sign bug (win-check math, entropy bonus sign, grad clipping all audited correct, hyperparameters match rl/ppo.py): the FFA win condition requires 80% of all map tiles, and the policy plateaus around half the map, stops expanding into neutral land, and its entropy collapses from ~7 to <0.05 within 2-5 updates on this low-variance, nearly-solved 2-player stage - then never recovers across 150 updates. Inherited characteristic of the ported curriculum/reward design (ent_coef=0.01 is rl/ppo.py's own default), but at Rust speeds it's now the blocker: the pipeline can push samples 10x faster at a policy that isn't going anywhere. Fix candidates: higher/slower-annealing entropy coefficient for early stages, reward shaping that keeps paying for expansion into neutral land after the opponent is matched, or stage-0 win semantics that are actually reachable (e.g. relative-strength or opponent-elimination based).

Jul 10, ~2am: the stall investigated and root-caused - three bugs, none of them the theory

The "curriculum/exploration tuning problem" framing above didn't survive contact with one question: the Python trainer won stage 0 on this exact curriculum, so what's actually different? A node-vs-native A/B (same Rust trainer, 12 CPU updates each, per-episode diagnostics on) plus a line-by-line diff against rl/ppo.py found three concrete divergences, in ascending order of embarrassment:

1. Episode budget: 3000 ticks vs Python's 15000. The Rust CLI default silently shipped at 3000 max_episode_ticks; Python's is 15000. Every "plateaued at half the map" episode was truncated at a fifth of the budget the Python wins were earned under. The 80%-ownership win was never reachable, entirely independent of policy quality. Fixed: default now 15000.
2. The entropy-floor controller was never ported. rl/ppo.py --ent-floor (default 2.5, on by default!) scales the entropy coef up 1.3x/update (cap 5x) whenever measured entropy dips below the floor, with hysteresis and startup grace. The Rust port had only the linear anneal - so the collapse to ~0 nats that Python's controller exists specifically to catch just... ran to completion, every run. Fixed: controller ported faithfully (including the "decay anywhere above the floor" rule that keeps a transient spike from trapping the scale), driven by minibatch-mean entropy like Python rather than a last-minibatch snapshot.
3. The native engine never left the spawn phase. The big one, and the answer to "is it our bridge?": in TS singleplayer, GameRunner adds no spawn timer - the spawn phase ends when SpawnExecution sees the human place a spawn (endSpawnPhase()). The native SpawnExecution port dropped exactly that call (the replay path has a timer, so replays/parity never noticed), so every native RL episode spent its entire life in the spawn phase: legality returned {actions: {}} (policy entropy pinned at ln-uniform ~6.9 over everything, Beta quantity head literally never active), vecenv's random-spawn fallback fired every step (which is also why native episodes burned ~2x ticks per decision), and no real game - no PlayerExecutions, no win check - ever ran. The tell in retrospect: node-engine logs showed ent≈2-3.8, entq≈-0.1 while native showed ent=6.916, entq=+0.000 from the very first update. Every native-engine GPU number from yesterday (the 82% included) was measured on spawn-phase noise. The throughput conclusions survive (the compute mix barely changes); the learning conclusions were never about learning at all. Fixed: execute_player_spawn reports success and singleplayer+human spawns end the phase, matching TS.

Post-fix A/B, same 12-update CPU config: native now behaves like node in kind - entropy 3.2 vs 3.8, quantity head active on both (entq -0.33 vs -0.10), episodes at the same tick cadence, mean reward 13.9 vs 10.9 and 1st-place 17/24 vs 10/24 (native's nations look somewhat softer than node's - the outcome gate, still grinding through all 78 records in its pinned worktree, is the instrument that quantifies exactly this). Neither leg wins inside 2000-tick test episodes with 12 updates of a tiny CPU net - wins need the real budget and a real training run, which is the very next thing to spend a pod on. Engine unit tests: identical pass/fail (47/35, all failures pre-existing missing-fixture paths) before and after the spawn fix.

Lesson worth writing down: all three bugs were silent defaults and omissions in a port, not logic errors - the code that existed was correct, the problem was code and config that didn't exist. Port audits should diff the flag surface and controller inventory against the original, not just the math that made it across.

Why the gate took 4 hours (it shouldn't have) and what the numbers actually mean

outcome_gate was strictly sequential with zero progress output until the whole 78-record report printed at the end - a run silently ran for ~4 hours with no way to tell if it was almost done or hung, and there was no per-record timeout at all. Root cause wasn't the engine (still 130-150k ticks/s standalone) - these 78 records are real archived multiplayer games, uniformly 400 bots + ~125 human players each (every single one - checked the full set), individually taking 100-1150s to fully replay tick-by-tick. Fixed: outcome_gate now runs records across a worker-thread pool (--jobs, defaults to core count) with a per-record wall-clock cap via a channel recv_timeout (--record-timeout-seconds) and streams N/total record -> category (Ns) progress to stderr as each one finishes. Also uploaded the cached TS oracle to djmango/openfront-human-games on HF (outcome-oracle/<commit>.json) and wired an automatic fetch into run_outcome_gate.sh - the ~1hr+ TS-engine regeneration only has to happen once, ever, per parity commit, not per machine.

The result, and why it's not the alarm it looks like: every record checked so far is missing_winner - not "close but outside tolerance," wildly different final states. One example (Two Lakes, Team mode, 400 bots): TS's real game ended at tick 7231 with Team Red at 95.05% land share; replaying the exact same human intents natively at the same tick leaves Red at 14.8%, with unaffiliated "Bot" entities alone holding 36.4% of the map - a different game, not a near-miss. But every one of the 78 records uses 400 bots, and the RL curriculum never goes above 150 (stages run 0, 0, 5, 10, 30, 30, 50, 80, 80, 120, 150). Bot AI acts autonomously each tick (it's not in the recorded intent stream), so any small per-bot decision difference between native and TS compounds every tick across however many bots are alive - at 400 independent agents over 5,000-13,000 ticks that compounds into a completely different map, while at the curriculum's actual bot counts the same per-bot error has far less surface area to snowball on. This test set answers "does native match TS at the most extreme bot density OpenFront supports," not "does native match TS at the bot counts the curriculum actually trains on" - it's the wrong instrument for the question that matters for training, and a 0%-ish pass rate on it shouldn't be read as "training is broken."

What it does establish, cleanly: bot AI has a real, likely systemic behavioral gap between native and TS (worth its own investigation - see next steps), and the existing 78-record human- game archive can't validate curriculum-relevant parity because it has no low-bot-count games at all. Both are now tracked as their own next-step items rather than blocking anything.

Jul 10, afternoon: all six next steps, done in parallel - the stall is fixed, native is default, here's what's real and what's not

Two subagents (curriculum-representative parity + bot-AI tick-level investigation), three A100 pods (stall-fix validation, model-size A/B, clean AMP A/B), one 4-GPU sanity check. Every item landed real evidence, not vibes.

1-2. Curriculum-representative parity + bot-AI root cause (done together, they compound)

First pass at curriculum-bot-count parity (0/5/10/30/50/80/120/150, `datagen/gen_curriculum_parity.ts`, `scripts/run_curriculum_parity_gate.sh`) used the same 4500-tick/6-min-timer budget as the archive and landed 22/40 (55%) with every failure a narrow-margin wrong_winner in a crowded field, not a systemic miss - a completely different failure mode from the 400-bot catastrophe. But that generator had every game hit the exact same max_timer cutoff at tick ~3910 regardless of bot count - i.e. every "outcome" was a coin-flip snapshot of an unfinished race, not a genuine win. Reran with a much longer horizon (20000 ticks / 40min timer) so games can actually resolve: 0/5/10 bots (the curriculum's first 5-6 stages) now hit real, complete dominance in both engines (land shares 0.93-1.0, not snapshots) - 0 bots 2/2 exact match, 5/10 bots near-misses where both engines agree on a genuine winner and near-identical land share, just occasionally disagreeing on which nation among several that converge to ~100% simultaneously. 30+ bots still shows real winner-identity divergence even at full dominance.

In parallel, a tick-level side-by-side trace (new `tick_dump` binary + TS dump script) at ~50 bots found the actual bug: first divergence appeared at tick 303 - two nations with byte-identical troop counts landed 1 tile apart, pointing at conquest-order/PRNG mechanics rather than a formula error. Root cause: GameMap::for_each_neighbor4 iterated neighbors west/east/north/south instead of TS's north/south/west/east - silently shifting which specific tile received which PRNG draw during every border-tile conquest-priority computation, in every attack, everywhere in the engine, for as long as it's existed. Fixed, plus two related missing is_impassable filters in attack/bot-AI decision code (one of which was a real invariant violation - impassable tiles could become "owned"). Effect: first divergence moved from tick ~303 to ~1497 (5x later, and everything in between is now byte-identical), total tile-attribution error at tick 6000 dropped 46% (756k -> 410k). This is now on master and is what the parity numbers above already reflect.

Bottom line: curriculum-relevant parity (bots 0-10, stages 0-3ish) is good - real wins, right magnitude, occasional coin-flip on exact winner identity in tight races. Higher bot counts (30+) still have a real, if much smaller than before, systemic gap worth continued attention as training reaches those stages, but nothing here blocks starting or continuing training now.

3. Stall-fix validation - confirmed, end to end, on a real A100

Single-GPU pod, --engine native, full 15000-tick budget, entropy floor on, all three 2026-07-10 fixes in place. Result: entropy holds 2-5 nats the whole run (entropy-floor controller visibly engaging - ecoef climbs to 0.02-0.05 when entropy dips, decays back down otherwise), the quantity head stays active (entq nonzero throughout, never pinned at 0.000), win rate climbs into the 80-100% range in bursts, and the curriculum advanced twice - stage 0 (Onion, 0 bots) → stage 1 → stage 2 (Onion+Pangaea, 5 bots) - within ~2200s / 166 updates. This is the thing that was completely impossible before yesterday's fixes (every episode timed out at 1/5 budget with zero win-rate signal, forever). The run stopped cleanly sometime after update 166 with no error in the log and the GPU idle/released - not a crash we could find a cause for, but not before it had already proven the thing we needed proven.

4. Model size re-asked with learning curves, not just throughput

Full policy (11.2M params, GC=256/BLOCKS=4) vs small (2.6M params, GC=128/BLOCKS=2), identical 64-env/rollout-32 config, run in parallel on separate pods for the same wall-clock. Small is genuinely faster per step (~280-380 steps/s vs full's ~150-160, matching yesterday's throughput numbers) but its learning curve was rockier: it dipped to a 0% win-rate window partway through (full policy dipped too, recovered faster) before eventually also reaching stage 2. Both networks are capable of learning this task; full policy got there first and more smoothly at this wall-clock budget despite the throughput disadvantage. Verdict: not the clean "smaller = more learning/hour" win hypothesized - model size trades throughput for stability/sample-efficiency here, and the net effect on learning-per-hour needs a longer, cleaner run (both to full convergence, same seed) to call definitively. Foveation's case is similarly still open - same throughput-vs-capacity tradeoff logic applies.

5. Clean AMP A/B

Same config, same minibatch count (8), only --amp flipped, 8 updates each, on a fresh pod so nothing else could confound it. fwdbwd_s: 0.505s -> 0.318s per minibatch (37% faster), steps/s 154.9 -> 217.3 (+40%), no NaN, losses in the same healthy range as the non-AMP run. This is a clean, real win - worth turning on by default for future training once the model-size question above is settled (AMP's memory savings also directly relax the env-count ceiling that caused the 256-env OOM on 2026-07-09).

6. Native the default; multi-GPU confirmed working

--engine native is now the default (--engine node still available as the parity-testing fallback) - justified by items 1-3 above: curriculum-relevant parity is real, the stall is fixed, and native is still ~10x faster ticking regardless. Also ran a 4-GPU sanity check (256 total envs, native engine, all fixes in place): shards scale correctly across all 4 GPUs (81-100% per-GPU during compute), ~520-550 steps/s (~3.4x single-GPU), 64% sustained min_mean_util (lower than single-GPU's 76-82%, matching the known per-shard-thread-overhead pattern from earlier multi-GPU work), and it also advanced the curriculum (stage 0 -> 1) before being torn down. Nothing broke moving to multi-GPU with today's changes.

Remaining next steps, in order of expected value

1. Kick off a real training run. Everything that was blocking one is resolved: native is default, the stall is fixed and validated, parity is good at the bot counts that matter early, AMP is a clean win. Turn on --amp, pick full policy (safer choice per the model-size A/B above) or small policy (if willing to accept rockier-but-faster), and let it run for real instead of 150-400-update smoke tests - single GPU first, watch the curriculum advance past stage 2, *then* scale.
2. Settle model size / foveation with a longer, cleaner A/B. Today's runs showed both sizes can learn but weren't run to convergence or matched by seed - re-run full vs small (and foveated) to the same wall-clock or update budget with a fixed seed and compare updates-to-stage-N, not just steps/s.
3. Investigate the pod1 validation run's silent stop (stopped cleanly after update 166, no error in the log, GPU released) - probably benign (pod inactivity policy, disconnect) but worth confirming before relying on multi-hour unattended runs.
4. Keep chipping at bot-AI parity for 30+ bots as training reaches those curriculum stages - the neighbor-order fix closed most of the gap but a smaller residual (troop-level, first appearing ~tick 1498 in the tick-level trace) remains undiagnosed.
5. Profile the collection phase now that engine ticking is ~10x too fast to be the bottleneck: collection scaled linearly with env count, so the cost is per-step obs decode/featurize/act - the next 76→90%+ util move is either actor/learner CUDA stream separation or moving featurization onto the GPU, not more engine speed.
6. Scale to 8 GPUs once a real run is underway and stable on 1-4 - the plumbing is confirmed working (this session's 4-GPU sanity check advanced the curriculum too), just hasn't been pushed further yet.

Jul 10, 2026 - ppo_v8 launch plan (8x A100, written now, not started)

Everything upstream of "press go" is validated: native engine, stall fixes, curriculum-relevant parity, AMP. This is the concrete runbook for the first real (not smoke-test) run, sized for 8 GPUs. Explicitly not executed yet - writing it down first so launch is a checklist, not a series of decisions made live on an $12/hr clock.

Prerequisites - done as of Jul 10, afternoon

1. Checkpoint resume: implemented. --resume <path> loads the VarStore weights plus a <ckpt>.state.json sidecar (curriculum stage, entropy-floor scale, learning rate, total env steps, win-rate window, update counter) written atomically (tmp+rename, matching rl/ppo.py's policy.pt.tmp pattern) alongside every checkpoint. Every save also refreshes a fixed-name latest.ot/.state.json pointer so a wrapper never has to parse filenames. Known, documented gap: AdamW's momentum/ variance state is not restored - tch-rs 0.24 exposes no optimizer state_dict save/load at all (checked the crate source directly) - so it rebuilds over the first few dozen post-resume updates. Deliberate, not silent.
   Smoke-tested end to end: short run → checkpoint → resume confirms the update counter and decisions_total continue (not reset to 0), and a synthetic mid-curriculum state (stage=2, ent_scale=3.5, lr_now=1.5e-4) injected into the sidecar correctly reaches the training loop (ecoef in the log reflects the resumed scale exactly: 0.01 × 3.5 = 0.035).
2. HF sync + restart-loop wrapper: implemented (scripts/pod_train_v8.sh, the Rust-flavored pod_train.sh equivalent) - bootstraps repo/toolchain/CUDA-libtorch-venv/ build, verifies libtorch_cuda.so is actually linked before spending GPU-hours, seeds --resume from a local-or-HF checkpoint, crash-loops with backoff, and background-syncs latest.ot/.state.json to djmango/openfront-rl every 10 minutes.
3. Found and fixed a real crash bug while smoke-testing resume - see below, this is likely what actually killed the pod1 validation run.
4. Model-size/foveation decision: still open (unchanged from the earlier next-steps list) - the plan below defaults to the safe choice (full policy, no foveation) pending a longer seed-matched A/B, exactly as before.

The bug that (probably) actually killed pod1's validation run

Testing --resume against a synthetic stage-2 state (maps=[Onion, Pangaea], matching exactly what pod1 had just advanced to before it went quiet) reproduced an immediate panic: obs 2 grid shape mismatch (v1 requires uniform batch shape). batch.rs's build_obs assumed every env in a batch shares one grid resolution - true only for single-map stages with no rehearsal draw. Every curriculum stage past stage 1 samples multiple maps of different native sizes, and even single-map early stages mix in differently-sized past-stage maps via rehearsal sampling - so a mixed-shape batch isn't a rare edge case, it's the guaranteed outcome of training past stage 1 for more than a few minutes. This fully explains why the run "stopped cleanly with no error" right after the stage 0 → 1 → 2 advance logged.

Fix: pad every item to the batch's max (gh, gw), top-left aligned, zero-fill. grid_valid (previously always an all-ones placeholder - the code comment already said so) is now 1 only over each item's real region. This needed no architecture change: policy.rs already multiplies grid_valid into every tile-legality mask and the foveation coverage math, so padded cells were already correctly excludable from action selection by design - only the batcher itself needed to stop assuming uniformity. Same-shape batches (still the common case for early stages) take an unchanged fast path. New regression test (mixed_shape_batch_pads_instead_of_panicking) pins down padding placement and the grid_valid mask precisely. Stress-tested with a real 20-update run at stage 3 (Pangaea+Caucasus, naturally sampling both) - no crash, no NaN.

A second finding while investigating: the sizing math only covers early stages

The "64 envs/GPU" ceiling from 2026-07-09 was established entirely on Onion (the smallest map, 512x512 native → 64x64 after /REGION alignment). Checked the other curriculum maps' actual dimensions: Asia is 2000x1200 → 250x150 - exactly GW_MAX/GH_MAX, and 9.2x more grid cells than Onion. Stage 6 onward (bots=50+) mixes in World/Asia; since the padding fix above makes batches size themselves to the largest map actually sampled, a batch containing even one Asia-scale episode costs ~9x the raw grid memory of an all-Onion batch - and the intermediate conv-tower activations (GC=256 channels vs 63 input channels) scale by the same spatial factor on top of a channel multiplier, so the realistic multiplier is higher than 9.2x, not lower. The 64-envs/GPU figure is only proven safe through roughly stage 5; expect to need fewer envs (or foveation, which caps the fine-grid cost regardless of map size) once training reaches stage 6+. This is exactly the kind of thing checkpoint resume now makes cheap to handle reactively: if stage 6+ OOMs, relaunch with a smaller --num-envs and it resumes from the last synced checkpoint with nothing lost - no need to get this perfectly right preemptively. Added as an explicit watch item in the monitoring section below.

Config decisions and why

Sizing for 8 GPUs

Single-GPU ceiling found 2026-07-09: 64 envs x 32 rollout (2048 samples/update) fits at 46% GPU memory and sustains 81-82% utilization; 256 envs at the same config OOMs (~77GB just for the observation batch tensor, before activations). AMP roughly halves activation memory but the raw observation-batch storage is the dominant cost and isn't shrunk by AMP alone, so don't scale envs up to compensate for AMP's savings without re-testing for OOM first - keep per-GPU envs at the proven-safe 64 for the initial launch.

The multi-GPU architecture is data-parallel replicas (each GPU shard runs its own independent copy of the policy + its own 64 envs, gradients averaged across shards every update - "poor man's DDP", no cross-GPU memory sharing), so per-GPU memory pressure at 8 GPUs is identical to 1 GPU. Confirmed working at 4 GPUs on 2026-07-10 (256 total envs, ~520-550 steps/s, 64% sustained util, curriculum advanced). Launch config:

--engine native --num-envs 64 --num-gpus 8 --rollout-len 32 --epochs 2 --minibatches 8 \
--amp --pinned-h2d --stage 0 --device cuda:0 --ckpt-every 50 --log-every 1 \
--resume <latest checkpoint if any>

= 512 total envs across 8 shards. Expect steps/s roughly 2x the 4-GPU number (~1000-1100) if scaling stays similarly sub-linear (4-GPU was ~3.4x single-GPU, not the full 4x - per-shard-thread overhead); sustained util likely similar-or-slightly-lower than 4-GPU's 64% (more shards = more overhead), which is fine per the established "utilization% isn't the metric, decisions/s/$ is" rule from 2026-07-09's sweep.

Launch runbook (once prereqs are done)

1. Provision: runpodctl create pod --gpuType "NVIDIA A100-SXM4-80GB" --gpuCount 8 --containerDiskSize 60 --volumeSize 200 --vcpu 64 --mem 500 --startSSH --ports "22/tcp" --secureCloud (bump vcpu/mem/volume vs the single/4-GPU pods used this week - 512 envs means ~512+ Rust OS threads plus whatever the native engine itself spawns per session).
2. Everything else - repo clone, Rust toolchain, the CUDA-linked libtorch venv (must be exactly torch 2.11.0 - tch 0.24's C++ shim calls ATen ops that don't exist in the pod's stock torch 2.8, fails to compile not just link), build, the readelf CUDA-link check, resume-seed from HF, and the crash-loop - is now one command:

   RUN_NAME=ppo_v8 NUM_GPUS=8 bash scripts/pod_train_v8.sh

   or as the pod's start command:

   bash -c "curl -fsSL https://raw.githubusercontent.com/djmango/openfront-ai/master/scripts/pod_train_v8.sh | RUN_NAME=ppo_v8 NUM_GPUS=8 bash"

3. Set EXTRA_ARGS to override the frozen config decisions above if deliberately deviating (default is --amp --pinned-h2d); OFTRAIN_DEBUG_EPISODES=1 isn't wired into the wrapper (it's a stderr firehose at 512 envs) - for the first stretch, tail /tmp/train_ppo_v8.log and run the binary manually with that env var set for a short supervised burst before trusting the unattended wrapper.

Monitoring, gates, and stop conditions

• Every checkpoint (--ckpt-every): confirm it actually synced to HF (the wrapper's background loop logs [hf-sync] synced latest checkpoint every HF_SYNC_INTERVAL_SECONDS) - a checkpoint that only exists on the pod's local disk isn't a safety net.
• Approaching stage 6 (bots=50, World/Asia enter the map pool): watch GPU memory% closely for an OOM as batches start sizing themselves to Asia-scale (~9x Onion's grid cells, see above) - if it OOMs, just relaunch with a smaller NUM_ENVS; the wrapper's crash-loop + resume means this is a config tweak and a restart, not a lost run.
• Curriculum stage index: the single clearest learning-progress signal - it should be monotonically increasing over hours, not stuck. Stuck for a long time at any one stage (no fixed threshold yet - use judgment, but "stuck" means many multiples of the 2026-07-10 stage-0->2 pace of ~2200s/166 updates on 1 GPU, adjusted for the 8-GPU throughput multiplier) is a stop-and-investigate signal, not a push-through signal - it was a real bug the last three times a stage looked stuck.
• Entropy (ent in the log): should stay in the low single digits, not collapse toward 0 and stay there for many updates (the pre-fix failure mode) - if it does despite the floor controller, that's a stop-and-investigate signal, not something to wait out.
• ecoef: watching it spike (entropy-floor engaging) and decay back down is healthy; pinned at the 5.0x cap for a long stretch means the floor controller is fighting a losing battle and something upstream (LR, clip range, reward scale) may need attention.
• min_mean_util / steps/s: sanity-check against the sizing section's expectations, but per the 2026-07-09 finding, don't chase a utilization number for its own sake - decisions/s and curriculum-stage progress are what matter.
• Cost: 8x A100 SXM ballparks ~$12/hr on RunPod (matches the 8-GPU pod pricing from 2026-07-09's session) - budget/time-box the run and check in on progress rather than leaving it fully unattended for the first day, until the resume/crash-recovery path has actually been proven under a real crash, not just the smoke test in the prerequisites.

July 10th, 2026 - auto-scaling env workers to a GPU-utilization set point (mechanics validated, GPU-driven convergence not)

Note on where this entry lives: the task that produced it was written against rust/DEVLOG.md, which by the time this branch's worktree was created had already been reduced to a "moved, see here" pointer and then deleted outright in the very next commit (both same-day, both from the concurrent engine-parity thread's work in this repo). Filing the entry here instead, matching that removal commit's own instruction to "write new entries" in this file.

Opt-in (--auto-scale-envs, off by default) automatic growth of the per-shard env-worker count toward a target GPU-utilization set point (--target-gpu-util, default 0.95), instead of the manual trial-and-error that found --num-envs 4 giving ~40% (dipping to ~1%, IPC-latency-bound) vs --num-envs 64 giving 98-100% on the same A100 box (see the rounds 1-3 entry above). New module oftrain/src/autoscale.rs: a pure, `tch`-free, unit-tested decision function next_env_count(current, gpu_util_frac, target, min, max, step) with a 3-point hysteresis band (only grow when utilization reads below target - 0.03; hold inside the band or above target) so noise in a single sample near the target doesn't cause resize churn every --autoscale-check-every (default 5) updates, plus cpu_env_cap_per_shard (logical CPUs minus a small reserved margin, divided across shards) backing --max-envs 0's "derive automatically" default. Twelve unit tests pin the boundaries: below/at/above target, already at min/max, gpu_util_frac=None, a zero step, and the nonsensical min > max case (resolved by raising max to min rather than left as something that could later hang or panic).

Design decisions: grow-only, and the cfg.num_envs correctness fix

Scale-down: not implemented (v1). Low GPU utilization realistically means "this run started with too few envs for this box" - a one-time mis-sizing to correct, not an ongoing overshoot signal the way memory pressure would be - so there's no symmetric case v1 needs to react to. Shrinking safely also means tearing down EnvWorker threads/bridge subprocesses mid-run without leaving shards at different env counts for even one update (see below); real complexity with no concrete driving need yet. Documented as a deliberate scope cut in the module doc, not a silent gap - if real runs show oscillation worth reacting to (thermal throttling, GPU contention from another process), the function signature already carries everything a shrink decision would need.

The actual correctness risk, and how it was fixed. train_update previously did let n = cfg.num_envs; and used that fixed n to size every GAE/minibatch buffer and index every per-env slot in the rollout. Growing ActorShard.workers/cur_obs at runtime (which is exactly what auto-scaling does) while train_update keeps trusting the original config value would have silently misindexed or corrupted those buffers the very next update after any scale-up - the kind of bug that doesn't panic, just quietly reads garbage. Audited every cfg.num_envs read in the crate (rg 'num_envs' oftrain/src): most are doc-comments/logging (left alone), but two were real sizing/indexing uses. Fix: train_update now derives n from the actual collected rollout's buffer width (pending[0].buffer[0].len()) instead of the config, guarded by a debug_assert_eq! inside each shard's own batch-build thread that would trip immediately if a future change ever let shards' env counts drift apart. collect_rollout itself turned out to already be correct - it was already deriving its per-call env count from actor.workers.len(), not the config - so growth is picked up there for free with no change needed. The other real one: the update loop's total_env_steps/steps/s stats used a startup-time total_envs constant that would have silently under-reported throughput after any scale-up; now derived per-update from the actual collected rollout widths (live_total_envs).

Why growth is uniform across shards, not per-shard-independent. Growing every shard by the same amount, spawning all of one growth step's new workers before committing any of them, and rolling back the whole attempt (closing any partially-spawned workers, keeping the old count) if even one spawn_worker call fails anywhere - this keeps the "every shard has the same env count" property always true by construction, which is exactly what lets train_update use one shared n derived from shard 0's data for every shard's minibatch/index-tensor math without per-shard bookkeeping. The alternative (independent per-shard counts) would need heterogeneous per-shard batch sizes threaded through the minibatch-index-tensor logic for a case (a spawn failing on exactly one shard) that should be rare and is now just refused outright instead of partially applied. The resize itself is placed at the one safe point in the loop: after this update's pending/next_pending swap, before the next iteration spawns its collect_rollout calls - never mid-rollout, where collect_rollout's already-cached n = actor.workers.len() from the top of that call would desync from a worker list growing underneath it.

What was actually validated (CPU-only sandbox, no GPU or CUDA device present)

cargo build --release -p oftrain --features native-engine succeeds; cargo test -p oftrain passes all 20 tests (12 new in autoscale.rs plus the 8 pre-existing policy/batch tests, unaffected); cargo clippy -p oftrain --features native-engine --no-deps shows the identical warning set before and after this change (a full non---no-deps clippy run fails identically on unmodified master too - pre-existing deny-level lints in the untouched ofcore/engine crates, confirmed by stashing this diff and re-running). A plain CPU smoke test (--device cpu --engine native --num-envs 4 --auto-scale-envs --max-envs 8 --autoscale-check-every 1 --updates 3) ran clean with the [autoscale] enabled: ... banner and no crash/hang, correctly taking the no-GPU-signal branch (hold steady, no resize) every update since a CPU-only device never starts a GpuUtilSampler - exactly the designed degrade path, not a heuristic scaling-without-a-GPU-signal fallback (see the module doc's rationale for why CPU headroom is used only as a ceiling, never as the growth signal itself). To actually exercise the growth code path end to end without a GPU, temporarily patched one line to force gpu_util_frac = Some(0.1) (reverted before committing, no trace left in the diff) and re-ran: real new EnvWorker threads spawned and joined the shard cleanly, 4 -> 8 -> 12 across two growth steps then correctly held at the --max-envs 12 ceiling, and decisions_total tracked the growing env count exactly right update over update (24, 72, 144, 216, 288 for rollout-len 6 at 4/4/8/12/12/12 envs respectively) with no crash and no misindexing - direct evidence the cfg.num_envs fix above actually works, not just that it compiles. A 2-shard CPU run (--num-envs 2 --num-gpus 2) with autoscale on also completed clean, confirming the uniform-growth bookkeeping doesn't break plain multi-shard CPU mode. Also checked: setting --max-envs below --num-envs logs a clear warning and clamps instead of hanging or panicking.

What is explicitly NOT validated: real GPU-driven convergence. Every test above ran with gpu_util_frac either None (real no-GPU degrade path) or hand-forced to a fixed value (mechanics-only, no real utilization signal, no real training-induced load pattern). Whether the grow-on-low-utilization decision actually converges toward the 95% target the way the manual --num-envs 4 vs 64 comparison suggests it should - and how fast, and whether the default hysteresis band/step size (3 points, +4 envs) are sane for a real training loop's utilization noise - is completely unconfirmed. Next step, on an actual CUDA pod: --engine native --device cuda --num-envs 4 --auto-scale-envs --min-envs 4 --max-envs 64 --autoscale-check-every 5 --autoscale-step 4 --rollout-len 32 --updates 100, watching the [autoscale] log lines and min_mean_util% to confirm the count climbs from 4 toward the previously-manual 64 and utilization tracks it up toward the 95% target instead of oscillating, stalling short, or overshooting the CPU-derived cap before the GPU signal even gets a chance to matter.

July 12th, 2026 - v8 launch: a RunPod community-cloud host with a broken CUDA init, and a real A40-vs-A100 price/perf number

First real multi-GPU RunPod launch of the Rust oftrain trainer (v8), mixing --node-fraction 0.2 (1 Node/TS env per 5, hedging native's known parity gaps at higher bot counts - see the bot-ai-parity docs) into an otherwise-native run. Hit a genuinely confusing CUDA init failure on the first pod that took an extended live bisection to root-cause; documenting the dead ends as much as the fix, since the dead ends are what would waste time again.

The failure: cuInit() itself returns CUDA_ERROR_UNKNOWN, only from the compiled binary

Symptom: oftrain --engine native --num-gpus 4 panicked in tch::nn::VarStore::new immediately on startup with libtorch's generic "CUDA unknown error ... Setting the available devices to be zero", crash-looping every ~15s. nvidia-smi inside the same container showed all 4 GPUs healthy; torch.cuda.device_count() from a plain python3 -c "import torch; ..." in the exact same venv, same LD_LIBRARY_PATH returned 4 correctly, every time, interactively. Only the compiled Rust binary failed - and it failed identically whether launched from the crash-loop wrapper or run by hand over SSH, ruling out a spawn-context/race explanation.

Dead end 1: missing device nodes. /dev/nvidia* only had numbers 1/3/6/7, not 0-3, on the first pod - looked exactly like the known "container device-passthrough numbering doesn't match NVML's clean 0..N-1 view" class of bug. Turned out irrelevant: `nvidia-smi -L` and `torch.cuda.device_count()` both worked fine despite it.

Dead end 2: missing split-pip-wheel library directories. The launch script's LD_LIBRARY_PATH only listed torch/lib and nvidia/cuda_nvrtc/lib, not the dozen other nvidia-*-cu12 packages' own lib/ dirs (each CUDA component - cublas, cudnn, cufft, etc. - is its own separate pip package in a modern torch wheel). Expanded LD_LIBRARY_PATH to include all of them - `ldd ./target/release/oftrain` already resolved every single library either way, and the expanded path made no difference at runtime.

Dead end 3: PyTorch's own RTLD_GLOBAL preload dance. Python's torch/__init__.py explicitly ctypes.CDLL(path, mode=RTLD_GLOBAL)s each split nvidia-*-cu12 package's .so in dependency order before its C extension touches CUDA - a plain Rust binary's normal ELF DT_NEEDED/lazy-bind linking doesn't replicate this, and it's a real, separate bug class (see tch-rs issue #1015 for the ROCm/HIP version of exactly this problem, hit and fixed for real in this same session's ROCm port). Implemented the same preload (oftrain/src/main.rs::preload_cuda_deps, kept in the codebase - harmless no-op elsewhere, and cheap insurance if this resurfaces) - zero effect on this specific failure.

The actual diagnostic: bypass torch/tch entirely. dlopen("libcuda.so.1") + dlsym("cuInit") + calling it directly, before any tch/torch code ran at all, returned 999 (CUDA_ERROR_UNKNOWN) - the CUDA driver API's most basic possible call failed, for this compiled binary, on this specific host, full stop. Not a torch bug, not a linking bug, not a preload-order bug. Kept as an opt-in diagnostic flag (OFTRAIN_EXPLICIT_CUINIT=1) for next time this shows up.

The fix: different host, not different code. Noticed (while chasing an unrelated setarch -R permission error, which turned out to be a red herring - the identical restriction exists on the host that ended up working fine, so it's apparently just how RunPod containers are hardened in general, unrelated to the CUDA failure) that this was worth testing on a different physical machine rather than continuing to debug in place. Terminated the pod, relaunched on secure cloud instead of community cloud (different specific host, since machineId changed) - worked on the very first attempt, zero code changes. Community-cloud machines are individually-operated third-party hardware with far less standardization than RunPod's own secure-cloud datacenters; this is the concrete cost of that variance materializing, not a general "community cloud is broken" claim - the working host also happened to share the exact same setarch restriction. Takeaway for next launch: if a fresh community-cloud pod fails CUDA init with everything else looking healthy (`nvidia-smi`/`ldd`/env all fine), don't sink time bisecting it - terminate and retry, preferring secure cloud, before assuming it's a real code bug.

A40 vs A100 price/perf, with a real number instead of a TFLOPS-sheet guess

The working run: 4x A40 (secure cloud, $1.76/hr total), --node-fraction 0.2 --num-envs 64 --rollout-len 32 --amp --pinned-h2d (the frozen v8 launch config). Steady-state (updates 6-13, past initial env-spawn ramp-up): ~320-340 steps/s, GPU util spiking to 100% across all 4 during the compute-heavy part of each update. Compared against this repo's only prior multi-GPU throughput numbers (rounds 1-3, all on A100 SXM secure-cloud pods, same --num-envs 64 --rollout-len 32 --amp config): single A100 ~217 steps/s (with AMP), 4x A100 ~520-550 steps/s. On raw steps/s, A100 is faster (as its ~2x-higher memory bandwidth and ~2x TFLOPS over A40 would suggest) - but on steps per dollar, using each GPU's live secure-cloud price at the time of writing (A40 $0.44/GPU/hr, A100 SXM $1.49/GPU/hr):

• A40 x4: 330 steps/s ÷ $1.76/hr ≈ 675,000 steps/$
• A100 x4: 535 steps/s ÷ $5.96/hr ≈ 323,000 steps/$ (worse - multi-GPU scaling is sub-linear for both, but A100's per-hour cost isn't)
• A100 x1: 217 steps/s ÷ $1.49/hr ≈ 525,000 steps/$ (still worse than 4x A40)

RTX 4090 was not empirically tested for this comparison - secure-cloud 4090 capacity showed zero availability at the exact time this was checked (SUPPLY_CONSTRAINT from podFindAndDeployOnDemand), and community-cloud 4090 is where the CUDA-init failure above was first hit, so re-testing it wasn't worth the risk of re-landing on a similarly broken host mid-comparison. 4090's raw TFLOPS (~330, more than double A40's ~150) suggests it could beat A40 on both raw steps/s and steps/$ if secure-cloud stock ever firms up - worth a real re-test if/when it does, rather than assuming the TFLOPS ratio carries over directly (this exact session's A40-vs-A100 numbers show it doesn't scale cleanly with TFLOPS alone - memory bandwidth and per-shard-thread overhead both matter too). Current guidance: A40 secure cloud is the best-value option with real, current data behind it; re-check RTX 4090 secure-cloud stock before the next launch and re-run this comparison if it's available.

A more fundamental infra fix: don't tie the container's lifecycle to the training script

While iterating on the NaN guard above, pod relaunches started failing in a new, much worse way: ssh.runpod.io proxy connections returned "container not running" or "container not found", repeatedly, across multiple fresh pods, multiple GPU types (A40, then A100 SXM to rule out one bad host), and both with and without a persistent volume - i.e. not a host-specific or GPU-type-specific problem, and not something retrying/recreating pods fixed (several cycles of terminate-and-relaunch all hit the identical symptom, which is itself the sign something structural was wrong, not "unlucky" - see the retrospective note below).

The actual design flaw: every earlier launch set dockerArgs to run the bootstrap + training script directly as the container's own foreground process. That makes the training script's process the container's entire lifecycle - if it exits for any reason (a bug in a newly-added preflight check, a transient error, anything), the container exits too, and SSH access (which depends on sshd running inside that same container) disappears along with it. Confirmed directly: removing the dockerArgs override entirely (falling back to the base runpod/pytorch image's own default CMD ["/start.sh"]) made the "container not found" failures worse, not better - that default script sets up nginx/Jupyter/filebrowser/sshd and appears to hang or take far longer than any prior custom dockerArgs ever needed, on this account/image combination specifically.

The fix: decouple training from the container's foreground process entirely. dockerArgs now: (1) explicitly starts sshd itself (service ssh start || /usr/sbin/sshd, not relying on the base image's own startup script for it), (2) launches the bootstrap+training script as a fully detached background job (nohup ... & disown, output redirected to a file), then (3) uses sleep infinity as the container's actual foreground process - stable and content-free, so nothing the training script does can ever bring the container down. Worked on the very first attempt: SSH available within ~60-70s (vs. multi-minute hangs/failures on every attempt before this), stable throughout a 20+ minute build+train cycle, survived the training process itself hitting the NaN-guard warnings above without any container-level disruption.

Retrospective note on process, not just the technical fix: several relaunch cycles in a row hit the identical "container not found" symptom before the actual structural cause (dockerArgs-tied lifecycle) was identified and fixed - repeatedly terminating and recreating pods without changing the underlying approach wastes time and money without new information once the same failure recurs on genuinely different hardware. The lesson generalizes past this one incident: when a class of failure repeats across multiple "fresh start" attempts, stop retrying the same approach and go looking for what's structurally common to all of them instead.

Value-loss explosion mid-run: root-cause attempt, symptom guarded, exact trigger still unconfirmed

Around update 9 of the same run, v (value loss) jumped from a healthy ~0.05-0.14 range to 56.4 trillion, stayed in the tens-to-hundreds-of-billions for several more updates (743K, 39.4M, 202M, 150M, 4.8B, 84.4B, 254B, 1.27T), then hit outright NaN at update 18 - entropy fell from ~3.7 to ~1.6 over the same span, a classic value-divergence-plus-premature-exploitation pattern. Killed the run (no checkpoint existed yet - --ckpt-every defaults to 200, this never got past 18) and read code rather than keep guessing live on GPU-hours.

Ruled out by direct comparison, not speculation: the leading hypothesis going in was a native-vs- Node-engine reward-scale mismatch, since this was the first-ever run to combine --node-fraction engine-mixing with a real, full 4-GPU/256-env-scale run - exactly the kind of brand-new code path most likely to hide a real bug. Traced the reward computation (oftrain/src/vecenv.rs::apply → ofcore::curriculum::strengths) back to its terrain/land-total dependency and compared the IS_LAND_BIT bit position across all three implementations that could disagree: TS's GameMapImpl.ts (7), native's engine/src/map.rs (7), and the shared ofcore::feat constant Rust code on both backends is meant to match (7) - identical everywhere, so a terrain-decode mismatch corrupting land_total (and hence every tile-share term in the reward) is not the cause. Also checked whether the Rust port had silently dropped a value-loss safeguard rl/ppo.py has: it hasn't - neither implementation clips or normalizes the value-loss target (ret), only the advantage (correctly, and only after ret is already derived from the pre-normalization advantage - standard GAE, not a bug), and gradient clipping (clip_grad_norm(0.5)) is present and correctly matches Python's - it just bounds the step size given a gradient, not the loss magnitude producing that gradient, so an unclipped squared-error value loss can still produce a step that overshoots badly enough to compound across updates even when every single step is clip-bounded.

Exact trigger still unconfirmed - the pod was killed before enough live diagnostic data could be gathered (no checkpoint, no detailed per-episode reward trace at the tick this happened). Rather than guess at a fix for an unconfirmed root cause, guarded the symptom directly and instrumented for a future recurrence:

• Skip applying a minibatch's optimizer step outright (zero the grads, log a warning, move on) if any shard's loss comes back non-finite - turns "one bad rollout eventually NaNs the entire run and whatever checkpoint it produces" into "one bad rollout costs one discarded minibatch". 5 new unit tests pin the boundary precisely - a merely huge-but-finite loss (like the actual 1.27 trillion seen live) must not trigger this, only true NaN/Inf should.
• A new warning whenever any single return exceeds |1000|, logging the exact global env-worker index (matching engine_for_idx's own indexing scheme) that produced it - if this recurs, that index directly answers whether the anomalous return came from a Node- or native-engine worker, closing exactly the evidence gap this investigation hit.

Next real run should watch for either warning firing; if the global-index warning ever points consistently at Node-mixed workers, that confirms the original hypothesis after all and narrows the follow-up to comparing reward-relevant fields between the two engines' observation payloads directly.

Follow-up: relaunched with the guard, node-fraction hypothesis revised

Relaunched the same config after fixing the pod-stability issue below. The exact same instability recurred - if anything worse (v peaked at 16.5 trillion this time, vs 56.4 trillion before, but across more updates) - and this time did not reach NaN: the guard fired 13 times over the first ~21 updates, and v settled into a ~700K-2.7M oscillation (still far from the healthy ~0.05-0.14 baseline, but stable, not climbing further) while training kept progressing (eps_done 0 → 217, steps/s holding ~330-350 throughout). The extreme-return diagnostic's global env-worker indices this time were 129 (Node-mixed), 138 (native), 182 (native) - computed directly against engine_for_idx(idx, default, 0.2)'s own formula. A mix, not all-Node - this is real evidence *against* the native-vs-Node-engine reward-mismatch hypothesis from the original incident, not for it. Revised conclusion: this looks like a genuine, reward-scale/hyperparameter-inherent PPO value-function instability at early stage-0 training (all three flagged occurrences were at t=31, the rollout's own bootstrap-value timestep specifically), independent of --node-fraction. Worth a real fix later (return normalization, or a smaller/warmup-scheduled vf_coef), but the guard already converts it from run-ending to merely-annoying, which was the actual goal.

Reference and open threads

Tooling, bugs, lessons, infra, recipes, backlog.

Visualization & tooling

Determinism is the superpower: the bridge mirrors the client's createGameRunner() init exactly (same PseudoRandom ID stream, no spawn timer), so agent games saved as GameRecords replay bit-identically in the real client, and verify_record.ts proves it by re-simulating from intents alone.

• oftrain --watch: one stochastic episode (train-matched; default) → GameRecord + .debug.json sidecar (every decision: action, probs, value). Full tick budget 21000.
• scripts/render_client_replay.py: headless Chromium replays the record in the actual game client → WebM with real graphics. Agent renders as AGENT with the gold spawn ring, crown when first, "You Won!" modal; camera starts fit to the whole map.
• ofshowcase archive: archive-API shim so a normal browser replays agent records interactively.
• MODEL overlay (RlDebugOverlay.ts): in-client panel synced to the sim tick: chosen action, value, probability bars, recent-actions log. Works in rendered videos, browser replays, and live matches; auto-probes the agent's feed on localhost:8988, zero setup.
• rl/play.py + bridge/play.ts: the agent joins a real lobby over the live websocket protocol; you fight it while watching it think.
• patches/client-replay-tooling.patch: all client hooks as an uncommitted-submodule patch, auto-reapplied if the submodule resets.

Bugs

• Wins were never counted. vec.py checked winner[1] == "Agent" (username); the engine emits the clientID ("AGENTRL1"). Every "the agent never wins" observation before Jul 6 12:06 was partly measurement error. Found only by watching a replay where the agent visibly won while the stats said otherwise. Validate the reward path end-to-end with a game you can see.
• Case-insensitive macOS hid a Linux crash. Map dir lookup lowercased only the first char (BlackSea → blackSea); dirs are fully lowercase. Fine on Mac, ENOENT on Linux.
• Resume logic fought the supervisor. Supervisor passed a stale --stage 4; resume took max(cli, checkpoint), so a manually reset checkpoint kept snapping back, and a still-running trainer kept re-saving its old stage over the reset. Checkpoint stage is now authoritative; kill the whole process group before editing checkpoints.
• Cropped-map index crash. Transport/nuke target coords weren't bounds-checked against the 16px-cropped grid; a boat targeting BlackSea's trimmed edge strip crashed the obs builder.
• Phantom boats: "legal" actions that always no-op. The boat mask was just troops > 100; the engine additionally requires a free boat slot (max 3), an own shore to launch from, and a non-own/non-ally destination with a reachable shore - and silently discards the intent otherwise, with troops never deducted. A discarded boat was therefore reward-identical to noop but occasionally landed and grabbed land, so the policy-gradient math correctly learned to spam it: 43-78% of decisions in v4 stage 2-3 replays, p(noop) driven to 0.01. Same shape in expand (fizzles without a neutral border), build (61% of one v3 episode's decisions, snapped to random land the engine rejected), and nukes (silo cooldown/spawn immunity ignored by the mask). The bridge comment said "illegal tile picks become no-ops the policy learns to avoid" - but there was no signal to learn from: the failure and the noop pay identically. Fix: honest masks + valid-tile snapping + a per-discarded-intent penalty (Jul 7 timeline entry). Caught by computing the action mix from replay debug sidecars, not from any training curve.
• Headless-browser quirks. Vite binds IPv6 ::1 while headless Chromium resolves localhost to IPv4 (force --host 127.0.0.1); the client rejects software WebGL (needs --use-angle=metal --enable-gpu --ignore-gpu-blocklist).

Lessons

• Only compress what's big. The AE is for the map; exact small state rides the bypass. One decision dissolved four v2 loss-weight tuning runs the same evening.
• Detection beats regression on sparse targets. Structure counts under MSE collapse to zero; rarity-weighted BCE occupancy hits P/R 1.0.
• Make wins reachable before making them valuable. A win bonus is useless if the setting never produces wins. 1v1 → 1v3 staging turned the win signal from theoretical to dense.
• Warm starts inherit stale habits. ppo_v2c resumed v2b weights (trained under the old placement reward and pre-fix win counting) onto the new curriculum; ppo_v3 started from scratch on the same code. In one day the fresh run reached stage 4 at 60-70% wins while v2c stalled at stage 3 at 7-17% with entropy stuck near 8, exploring without committing. When the reward or curriculum changes materially, retrain; a checkpoint tuned for the old objective is a local optimum in the new one.
• Rehearse or forget. Revisit cleared maps at current difficulty for a fixed fraction of episodes.
• Exploit determinism everywhere. Bit-identical replay gives verification, visualization, and live mirroring from one mechanism.
• Pad to the batch, not the maximum. Most of a "GPU too slow" problem was wasted convolution.
• Watch the agent play. The replay tooling caught the win-detection bug and the camera problems; curves never would have.
• An action that can silently fail is a free lottery ticket. If a "legal" action's failure mode is exactly a noop (no cost, no signal) and its success mode has upside, the policy will rationally spam it forever and it will look like pathological behavior. Every action needs one of: an honest mask, or a price on the discarded intent. "The policy will learn to avoid illegal picks" is only true if avoiding them pays better than trying them.
• The loss curve lies; log the metric you care about. The mixed-retrain loss looked flat for hours while border accuracy was still climbing. Overall tile accuracy is 99% saturated and borders are a sliver of pixels. Border accuracy cost one line to log natively (the mask already existed for the loss weighting).
• Benchmark on the distribution you'll deploy on. The encoder looked done at 87% border accuracy until it met real human games and dropped 16 points. Bot data underrepresents exactly what humans do: allying, betraying, invading by sea.
• Spatial precision can't be bought with channels. +50% latent channels does far less for borders than halving the patch each latent vector must summarize. And don't make the latent re-encode static side-information (terrain) the downstream consumer already has.
• Treat training hosts as disposable. Checkpoint everything, sync to HF, restore on boot, supervise with auto-restart. Every ops incident so far violated some part of this.
• Auto-restart is not auto-recover. The temporal-BC supervisor relaunched an OOM-crashing trainer into the same wall for hours, and a separate pod ran stale code all day because its git pull had silently failed on an untracked file. A restart loop needs two cheap guards: back off (or change something, or alert) when the child dies instantly twice in a row, and assert the deployed commit matches origin/master at launch.
• Sequence models multiply memory before they multiply quality. A seq-8 BC window is 8 full observations through the frozen encoder per sample: activation memory scales with window length, so batch size must scale down by the same factor. Budget it before launch instead of discovering it as a 19.9 GiB allocation on a 24 GiB card.

Infrastructure

Homelab for datagen and some training; cloud GPUs for long PPO and AE runs. scripts/pod_train.sh supervises RL jobs (idempotent deploy, auto-restart). Datasets are prefeaturized once; checkpoints live on Hugging Face: openfront-snapshots, openfront-human-games (285 hash-verified replayed human games + raw intent records), openfront-tile-autoencoder. Ops scar tissue from the AE push, mostly self-inflicted: the homelab mixed run was host-OOM-killed at step 22.7k (CPU evals + HF upload + 6 DataLoader workers on one box); remote trainers hit ulimit -n with 16 workers memory-mapping 459 game caches; a billing outage restarted cloud machines mid-run (tmux gone; the interrupted ablations were relaunched from scratch since AdamW state isn't checkpointed, only weights). Git history was scrubbed once with git filter-repo after large blobs snuck in; data dirs are gitignored now.

v4 training recipe & decision gates

Three arms, two launched together (Jul 6 eve), the third conditional:

• Arm A - pure RL (ppo_v4): from scratch, stage 0. Baseline and control; also the only arm learning spawn placement without supervision.
• Arm B - BC (bc_v4): feedforward pilot, 60k steps on the cache-bc pipeline. At ~1.5ms/sample CPU the run is GPU-bound; verdict lands in hours, not days. Saves bc_best.pt on holdout improvement (acted-action + tile accuracy - the honest heads; plain action accuracy is inflated by forced-legality noops).
• Arm C - BC→RL (ppo_v4_bcinit, conditional): launched when B plateaus. rl/ppo.py --init loads the shared Policy weights and folds the winner-bucket conditioning into the head biases - verified exact (max output diff ~1e-7 across all heads). Same hypers/envs/pod class as A; judged at equal PPO wall-clock on the eval/win curves and time-to-stage-N.

Gates. B is learning iff holdout tile_region accuracy climbs and holdout loss falls (10k steps ≈ 1M samples ≈ 5x the entire old pilot's data; flat there = dead, kill the pod, A continues regardless). B has plateaued (launch C) when holdout metrics are flat across two consecutive eval windows (~10k steps) or the cosine tail begins. C's first ~30min will look bad while the untrained value head catches up - judge after the critic settles. If C's entropy climbs back toward ~7 in the first hour, the BC prior got washed out: add a KL-to-prior regularizer (AlphaStar-style) before rerunning.

Watch items. BC first log line should read several hundred ex/s (under ~100 = cache problem, stop). Late-curriculum OOM lever: MINIBATCH=64 env var (1/8 grids cost ~4x v3's conv activations at World/Asia). Rollout buffer RAM ~20GB at Asia-size grids with 96x32; halve --rollout if the pod is RAM-tight. Fixed-seed eval pauses training for minutes at late stages (~2% overhead at eval-every 300). Old v1 sidecars carry no spawn labels; re-replay with BC=1 REBC=1 datagen/replay_all.sh feeds the next BC run.

Deliberately skipped: BC value-head pretraining on placements (scale-mismatched with PPO returns, muddies A-vs-C), fp16 rollout buffers (RAM insurance we may not need), latent precompute to disk (at 1/8 a latent is ~2.4MB/snapshot ≈ 200GB corpus - the GPU-batched encode from the raw cache IS that idea, done right).

Open threads

• V11 (active plan, Jul 23): see V11 plan - neighbor pack, ego-split fronts, unit tokens + pointer, bigger model, full LSTM replacing GRU, BC unfreeze on the new schema. V10 killed.
• v5 wishlist (drafted Jul 6 night, from watching v4 launch day):
  • Memory. Superseded by V11: V8.x shipped a 256 GRU; it did not unlock commitment memory. V11 replaces it with a full LSTM (cell + hidden), wider, BPTT already wired.
  • BC temporal transformer, judged properly. Still shelved under V11 rules: feedforward bc_v11 first; seq BC only after LSTM PPO shows memory value. See BC unfreeze.
  • Reward MDP audit + asymmetric strength shaping. v4 games show the shape: hyper-aggressive expansion, big holdings, then collapse (s2 seed 7: 103k tiles at t8001, dead by t10711). W_DELTA treats gains and losses symmetrically; weight losses heavier than gains (loss-aversion shaping) so defense is learned before stage 4 punishes its absence. While in there: audit timeweight and the noop-heavy action mix (35% noop at stage 2) against the actual returns.
  • Async rollout/update overlap. Post-fix the phases are still sequential (~33s rollout with GPU half idle, ~48s update with 46 cores idle). Overlapping them is another ~1.5-2x. Same surgery should batch the two act-groups' encodes.
• PPO v4 wishlist: shipped (Jul 6 eve). Learned spawn placement (PPO + BC), d8c32 encoder at full 1/8 with the local owner-crop bypass, periodic fixed-seed greedy eval (eval/* curves), win-gate window persisted in the checkpoint, entropy anneal (0.01 → 0.002 over 4k updates) and stage LR warmdown (x0.85 per advance). v4 trains from scratch - the 1/8 grid, new action ("spawn"), and 43ch observation change the MDP and the tensor shapes anyway.
• Spawn supervision coverage: only sidecars regenerated with --bc --rebc (formatVersion 2) carry spawn labels; old v1 sidecars still train everything else. Re-replay the archive on a pod to unlock BC spawn supervision everywhere.
• BC warm start: Jul 8 moratorium unfreeze experiment was never run on v6-shape. Superseded Jul 23: run the isolation A/B on V11 schema (bc_v11 vs scratch) per V11 BC rules - not on obsolete v6 weights.
• Warm-start refreshes: future same-architecture encoder updates on new data should resume from the current checkpoint (and checkpoint optimizer state, see billing-restart scar) instead of 40k fresh steps.
• Curriculum progression: with wins now measured correctly, watch whether stages clear; tune WIN_AT / WINDOW if stage 0 saturates or stalls.
• Remaining action heads: upgrade_structure, delete_unit, move_warship, cancel_attack/boat are schema'd but not all wired.
• Self-play league: past-policy opponents replacing scripted bots once the agent clears the bot curriculum.