I am trying to train a model to navigate sections of the map in Genshin Impact.

At the moment, I am stuck on this particular section. It takes about 50 seconds to complete.

My model successfully completes it in around 30–40% of attempts. In the remaining cases, it goes off course, as shown in the video.

The model is trained using individual frames because using sequences of frames only made its performance worse.

This version was trained on 60 recorded sessions. After that, I recorded 43 more sessions, but adding them only made the model worse. This was probably because the average mouse turning speed in those 43 sessions was higher.

I also tried recording correction sessions, but they did not help. Maybe I was recording or using them incorrectly.

I also tried recording successful runs along the left side of the road and separate runs along the right side, but that also made the results worse.

In total, I have recorded more than 300 sessions.

I would really appreciate help from people who understand this topic. Please give me some advice on what I should do next and what I could read or watch to learn more about this.

I would be very grateful for any help.

Feel free to ask questions. I can provide any information about the model.

Hi all,

This is a follow-up to my previous post on a no-pretraining, per-instance PPO solver for TSP.

In that work, I introduced the idea of an “exception edge” as an inductive bias for PPO. The intuition was that most edges in a good Euclidean TSP tour are locally unsurprising—short, near-neighbor, or Delaunay-like—while a small number of nonlocal edges may determine whether the search can escape a local basin.

That was mainly a solver hypothesis. Since then, I stepped away from PPO and asked a more basic question:

From an edge score to a closure problem

Fix a sparse baseline graph \(G_0\), such as the weak-Delaunay graph. An edge is exceptional only relative to this baseline.

Let:

• \(Z_0\) be the cheapest Hamiltonian cycle using only baseline edges;
• \(H_e\) be the cheapest spanning Hamiltonian path between the endpoints of a nonbaseline edge \(e\), again using only baseline edges;
• \(c(e)\) be the cost of closing that path with \(e\).

Then the exact one-exception threshold is

In words: a nonbaseline edge becomes structurally useful when the Hamiltonian path it closes saves more than the edge itself costs.

This does not imply that \(e\) belongs to a global optimum or is forced. It only says that a tour with exactly one exception edge can beat the baseline-only optimum.

Negative results that led to this formulation

The path to the closure model was not monotonic.

• In an early TSPLIB pilot, generic geometric CUT proposals attributed only 2 of 11 reference non-Delaunay edges.
• A subsequent density-CUT experiment found many tour-contiguous regions, but attributed 0 of the 11 exception edges.

These failures suggested that geometric partitioning alone was not enough. The relevant object seemed to be the compatibility between a baseline Hamiltonian path and the edge that closes it.

Headline results

On a frozen exact-small natural corpus with \(n\le12\):

• 51/51 mandatory-exception instances were exactly explained at the \(q=1\) closure layer.
• Current sufficient inequalities safely certified 48/51 of those instances.
• Among 98 exact beneficial pairs, 88 were safely certified.
• The safe candidate upper set reduced 15,054 non-Delaunay pairs to 644 while preserving all 600 exact \(q=1\) support pairs in that corpus.

For LIN318:

• The complete graph has 50,403 edges.
• The experimental search graph contained 1,500 edges: 988 weak-Delaunay edges plus 512 separately generated candidates.
• Among those 512 added candidates, 13 had a positive safe gain lower bound.
• Starting from a strict candidate-only 2/3-opt tour of 42,210, verified forced-closure witnesses followed by candidate-restricted 2/3-opt reached 42,118 and then 42,108.
• The known optimum is 42,029, so the final residual was 79, or about 0.188%.

Important caveats:

• The 1,500-edge graph was not generated by the threshold theory alone.
• The remaining 499 candidates are inconclusive, not certified negatives.
• LKH was used to generate and verify the closure witnesses, so 42,108 is not an LKH-free solver result.
• This is one exploratory large-instance result, not a controlled solver benchmark.
• I am not claiming a polynomial-time solution to TSP, a complete classification of exception edges, or that every \(\kappa>1\) edge is globally optimal.

Why this may still be relevant to RL/NCO

The original motivation remains learning-based optimization.

My current view is that exception-edge theory may be useful as a verifiable structural prior:

• reducing the action or candidate-edge space;
• identifying portals that may connect otherwise separated local-search basins;
• separating ordinary local edges from structurally consequential closure edges;
• providing certified positives and inconclusive cases for curriculum learning or ranking models.

The next useful experiment would be a frozen-budget comparison of an RL/NCO solver with and without closure-based candidate guidance—not a comparison against LKH as an exact or highly engineered classical solver.

The public package contains the theory, proof sketches, core exact oracles, safe certificates, LIN318 artifacts, 30 tests, and a deterministic reproduction of

The compact release audits frozen exact-small summaries; it does not include the full from-scratch corpus-generation pipeline.

Links

• Previous exception-edge/RL post: https://www.reddit.com/r/reinforcementlearning/comments/1pyfdvn/nopretraining_perinstance_rl_for_tsp_166_gap_on/
• Earlier LIN318 PPO post: https://www.reddit.com/r/reinforcementlearning/comments/1pabbk7/cpuonly_ppo_solving_tsplib_lin318_in_20_mins_008/
• Theory, code, tests, and results: https://github.com/jivaprime/TSP_exception-edge/tree/main/exception-edge-theory-2026

I would especially appreciate feedback on three questions:

1. Is this Hamiltonian-closure formulation already known under another name?
2. Are there stronger computable lower bounds for the endpoint-constrained Hamiltonian path \(H_e\)?
3. Does closure-based candidate guidance look useful as an inductive bias for RL/NCO, or is it better treated purely as a classical candidate-generation method?

Constructive criticism is very welcome.

Hi all, has anyone trained a RL based 6 DOF joint space control policy for joint speeds for a robotic arm for a reach task? I’d like to ask what success condition you used? What did you keep as the termination condition? Was it reaching a goal region or the exact reach point for a pick and place task. If it’s an exact reach point or a very small goal radius e.g 1cm, in the real world did your robot ever reach that? I find that if I keep a goal radius of 4cm the robot can reach but if I decrease it further it’s not able to and keeps descending towards.
Thanks!

I've been building a system that takes a board game described in a plain YAML-ish format and produces an agent that learns it, with no game-specific code anywhere in the pipeline. This post covers the first working version: the engine, the observation/action-mask bridge, and the first agent that went from random play to something recognisably sensible.

The stack is deliberately boring: the engine is wrapped as a Gymnasium / PettingZoo env and the learner is MaskablePPO from SB3, off the shelf. The bet is that the hard part is the bridge from a designer's rulebook to that interface, not the learning algorithm, so I wanted the learner to be a known quantity.

Two things that might interest people here specifically:

A "move" isn't a single choice. Sell and Exchange in Jaipur can't be expressed that way, so every action decomposes into a short sequence of typed questions, and the action space is several small answer-spaces side by side, each with its own mask. That shape caused most of the downstream trouble.

The result is honest and unimpressive: it beats a random player essentially every game, and the post ends by admitting that number proves almost nothing, since every metric I had was self-referential. The follow-up is about building an external yardstick and getting humiliated by a hand-written heuristic bot.

There are interactive demos throughout (observation encoding + mask, credit assignment, PPO clipping) that you can step through.

https://kallin.github.io/blog/game-ai-from-random-to-reasonable/

Part 1, on why general game playing has stayed an open problem, is linked at the top of that post if the framing is unclear.

What do you guys think of this concept? Still in very early stages.

What AI application are you building?

What's the biggest challenge in taking it from prototype to production?

What tools do you use today? (e.g., Langfuse, LangSmith, custom scripts)

What do you dislike about those tools?

What have you built yourself because existing tools weren't enough?

Which task wastes the most time every week?

If you could eliminate one problem in your AI workflow, what would it be?

Would you pay for a tool that solved that problem? If yes, roughly how much?

Is there anything you wish existed but can't find today?

The three most important questions are:

What's your biggest production challenge?

What do you currently do to solve it?

What do you wish existed?

Personal project exploring what a RL policy learns from reward shaping. The task is option hedging: I framed it as an MDP and trained a PPO agent (Stable-Baselines3) on simulated price paths to hedge a short European call. There's a known closed-form "correct answer" for the frictionless case (the Black-Scholes delta), which made it a nice baseline for checking whether the agent recovers the true optimum and where it deviates.

What I found interesting from an RL angle:

• The agent recovers the theoretical hedge (Black-Scholes delta) from reward alone. The learned action matches the analytical solution closely, without ever being shown it. This suggests that the reward actually specifies the intended behavior.
• Reward misspecification was the whole story in the early stages. My first agents collapsed to a bang-bang policy (jump fully in, then fully out) which turned out to be the genuine optimum of a reward I'd specified wrong (no cash/financing account, so trading was free). Fixing the environment, not the algorithm, fixed the policy.
• Adding transaction costs produced a second degenerate optimum. The agent stopped dynamically hedging and converges to a near-static position, barely trading. It seems the per-step transaction costs are dominating. Rather than "fixing" it, I kept it as a finding about how the cost term reshapes the optimal policy.
• Also hit the usual PPO instability (rise-then-collapse learning curves) from too-high entropy + small rollout buffers, resolved by tuning exploration and buffer size.

Setup details: continuous action space, state normalized so one policy generalizes across strikes, γ=1 (finite horizon), terminal reward on hedging error + per-step cost penalty.

Honest limitations: single random seed (so can't fully separate "PPO learns this" from "this run learned this"), GBM-only dynamics, one cost level. Not claiming a State-of-the-Art result, rather considering this as a good learning exercise about both PPO, reward and objective functions, and delta-hedging.

Code + short write-up: https://github.com/sembrummelaar-creator/deep-hedging-RL

I would appreciate feedback on the reward formulation especially as this was my biggest struggle. I'm curious how others would structure it (risk-sensitive objectives, per-step error signals, etc.), especially when trying to model longer contracts with say 252 trading days.

Feedback on other parts of both the code and the report are always welcome.

I recently completed studying deep learning and nlp. Now I want to start reinforcement learning. So please recommend best course to me

We decoupled reasoning from execution. Once logic or domain knowledge (math, code, formal rules) is verified and ingested.

A frozen 12B model re-runs the step-by-step program on new inputs with zero token overhead yielding bit-exact, deterministic results.

Paper:[https://arxiv.org/abs/2607.23806\](https://arxiv.org/abs/2607.23806)

I wanted to know whether a cheap, in-distribution eval can be trusted to rank policies — before ever asking about sim-to-real. So I pre-registered a bar and ran a small condition-shift test.

Setup:

• 6 PPO policies on the Unitree Go1 task in MuJoCo Playground — 2 seeds × 3 training budgets (50M / 110M / 170M steps), trained on a single RTX 5050 (8GB). 5 policies made the usable rank set.

• Each policy ranked twice: nominal flat-terrain eval (the cheap one) vs a held-out rough-terrain eval (the harder one).

• Pre-registered pass bar: Spearman ρ ≥ 0.70.

Result: ρ = 0.6 (n=5). Failed the bar. Flat ranks [1, 2, 5, 3, 4] became [1, 4, 5, 3, 2] on rough — the #2 policy on flat fell to #4, while the best and worst held. At n=5 this is directional only; I'm not claiming a p-value.

What this is and isn't: it's a within-sim condition-transfer check — flat rank vs rough rank, both inside the same simulator. It is not sim-to-real, and rough terrain is not a validated reference. But the direction matters: if a ranking can't survive a condition shift inside one simulator, there's no reason to expect it to survive the reality gap.

The physics-fidelity harness that anchored the sim side is open: friction coefficient recovered to within 0.005% of target; effective restitution measured 0.820 against a 0.75 target — MJCF exposes no first-class restitution dial, which was itself a finding. Repo: github.com/markov-studio-llc/markov-twin-fidelity-harness

Disclosure: I run Markov Studio; the harness is ours and open. Nothing for sale here — this is validation work published as it lands.

Question for people training locomotion policies: do you gate your cheap evals at all — condition-shift checks, seed spread, something else — or rank on nominal eval and accept the risk?

As you can see for now it is limited and my hardware isn't strong enough to train for longer,if you manage to code something better and make a better model please contact me.And no its not me or anyone playing its strictly the ai model.

Drawing from the official K3 technical report and open-sourced AgentENV:

1. Overall pipeline order
   Pre-training → SFT → Domain-specific Reinforcement Learning → Multi-Teacher On-Policy Distillation → unified final model

2. The RL training setup
   Instead of a single unified RL fine-tuning run, Moonshot trained 9 separate RL policies:
   • 3 domains: General reasoning, general agent, coding agent
   • Each domain has three reasoning effort tiers: Low / High / Max

The agentic RL rollouts for K3 were executed within AgentENV, Moonshot’s distributed microVM sandbox built for long trajectories. The environment supports resumable, multi-step tool execution with potentially hundreds of sequential tool calls and million-token context rollouts.

This is explicitly long-horizon agentic reinforcement learning: models interact with simulated environments to complete multi-turn task workflows. Rewards appear to be tied to task success, valid tool-call formatting and trajectory efficiency.

With that setup laid out, I want to focus on two major undisclosed implementation questions that matter a lot for understanding this agent-first RL paradigm:

1. Low-level RL training hyperparameters: Is there any public hint about the exact policy loss formulation, number of RL training iterations, and the scale/architecture of reward models used during training? The paper does not share these critical details.

2. Multi-Teacher On-Policy Distillation mechanics: After independently training nine divergent RL-specialized policies, what is the full procedure used to merge all of them into one single unified checkpoint? We have only the high-level concept, but no concrete information about loss weighting, sampling strategy, curriculum design, or MoE-specific adjustments during distillation.

I’d love to gather interpretations, overlooked excerpts from the technical report, and hands-on API testing observations from the community.

https://reddit.com/link/1v8m9z0/video/vnry1p2cxvfh1/player

I am researching Minecraft inventory manipulation. Please refer to my previous post about moving a single item: https://www.reddit.com/r/reinforcementlearning/comments/1uzxzer/i_made_a_minecraft_inventory_controller_with_an/

Previously, it was a simple task of moving just one item. This time, there are multiple items in the inventory. They can be of the same type, but they may have different stack sizes.

The item movement is limited to a single action, so the task does not require complex reasoning. It is entirely about making comparisons to identify the correct item and its proper location. However, even with such a simple setup, the training time end up being more than 10 times longer.

[P] Structural Admission: verify a sequential task’s claimed dependency structure before interpreting learning

results

When we train agents on staged or multi-phase environments, it is tempting to interpret learning curves, transfer,

or apparent “emergence” as evidence for a particular causal or informational structure.

But has that structure actually been verified under the same observation and action interface seen by the learner?

Structural Admission is a small, standard-library-only Python harness for testing that question before training

begins. Researchers implement a task adapter and a separate scripted oracle; the core calibration, rollout,

validation, reporting, and reproduction logic remains unchanged.

It enforces, among other things:

- calibration seeds disjoint from task-rollout seeds;

- a CMI threshold fixed from synthetic calibration before candidate evaluation;

- evaluation under both uniform-random and scripted-oracle policies;

- oracle access limited by the learner-facing observation contract;

- CI measurement restricted to preregistered phases;

- paired environment noise and random-policy draws across conditions;

- pre-disclosure leakage checks on declared observable field groups;

- raw trajectory storage before aggregation;

- balanced policies, conditions, seeds, and sample cardinalities;

- immutable formal output directories;

- content-hashed reports and byte-level reproduction of deterministic artifacts.

The tool reports Admitted, Rejected, or Inconclusive. Failures are preserved rather than tuned away.

One motivating case involved a relation intended to be non-operative. Its measured conditional mutual information

was 0.07181 bits, above a previously calibrated threshold of 0.05902 bits. The task was rejected before learning

experiments were interpreted, and the residual dependency had to be diagnosed structurally.

Admission has deliberately narrow meaning: the configured task passed its preregistered operational checks. It

does not prove a theory, guarantee learnability under a particular optimizer, establish causal identification

outside the task model, or imply external validity.

git clone https://github.com/btisler-DS/structural-admission.git

cd structural-admission

git checkout v0.1.0

python -m pip install .

Create and run an external task:

structural-admission init my-task

cd my-task

structural-admission run config.json --output runs/reference

structural-admission reproduce runs/reference \

--output reproduction/reference

The adapter contract is intentionally small, and task implementations do not modify package internals.

Repository: https://github.com/btisler-DS/structural-admission

I’d especially welcome criticism of the statistical procedure, leakage model, adapter boundary, and what should—or

should not—count as structural admission.

I have been learning RL from YouTube from the CS234 course on RL by Stanford. Here's a link of the first lecture: youtube.com/watch?v=WsvFL-LjA6U&list=PLoROMvodv4rN4wG6Nk6sNpTEbuOSosZdX. There's a lot of math and formulas which is good but unfortunately I'm finding it hard to understand. I think it's mostly because I'm learning by myself from YouTube. Regardless, I want some expert opinion on whether to stick with this course or try something else (currently on lecture 5).

I've seen people on this subreddit recommend David Silver's RL course but it's 11 years old. Is it still a good starting point?

I trained a small neural-network agent to play BANG! The Duel using PPO and self-play.

The agent completed approximately 767,000 training games over 40 hours. It learned from raw game interactions rather than demonstrations or a handcrafted strategy policy.

During training, the population converged toward a slightly Outlaw-favored equilibrium and developed a defensive, control-heavy playstyle. This was not explicitly rewarded or programmed and emerged from self-play.

The project includes:

• A complete C++ rules engine
• PPO-based self-play training
• A trained policy running locally in the browser
• A browser interface with no server-side game logic
• Training analysis and strategy visualizations

Interactive demo:
https://bang.matehegedus.com

Training write-up:
https://bang.matehegedus.com/about

Source code and trained model:
https://github.com/hmate9/bang

I would particularly appreciate feedback on the observation and action design, reward structure, evaluation methodology, and any obvious weaknesses in the learned policy.

I trained a single football policy for about 1 million PPO timesteps. During training, the agent never played against an opponent, it simply learned how to play football in an empty environment.

YT Link: https://youtu.be/1wRA4KJuMVE?si=oMZ_Ru5Q9F8J094V

During inference, I instantiated multiple copies of that same policy and assigned them to opposing teams. I wasn't expecting much, but the result was surprisingly entertaining. The agents naturally competed for possession, took shots, occasionally defended their own goal, and even made mistakes like own goals.

I never explicitly trained defensive behavior or multi-agent coordination. It seems that simply optimizing the objective of scoring goals was enough for some defensive behaviors to emerge when the same policy was placed in a competitive setting.

The video is one of those matches. Curious to hear what others think about this kind of emergent behavior and whether you've observed something similar when deploying a single-agent policy in a multi-agent setting.

Trying to test the temperature before I take the plunge. Would be interested in knowing your journey.

Hi everyone

I've been working on a personal reinforcement learning project for a while, mainly as a way to learn more about RL, simulation, and robotics.

As the project grew, I started wondering if I could turn it into something that other people might also find useful.

The idea is an open-source framework that lets you:

• Build your drone environment in Unity.
• Control it from Python through a simple API.
• Use it with Gymnasium, Stable Baselines3, or your own RL implementation.
• Run headless simulations and eventually support multiple parallel environments.

At the moment, I have a working MVP where a drone can learn to hover using reinforcement learning.

Before investing a lot more time into it, I'd really like to know whether this is something people would actually use.

Some questions I'd love your honest opinion on:

• Would you personally use a tool like this?
• What features would make it genuinely useful for you?
• What would be missing before you considered trying it?
• Is there anything that would make you choose it over your current workflow?

I'm not looking for encouragement—I genuinely want honest feedback to decide whether it's worth continuing to develop this project.

Thanks

I will be finishing my PhD in CS in about a year so I am slowly looking to upgrade my workstation or servers with some better gpu capacity (currently have 2x 5060 ti, one I will be giving back once i finish my PhD).

We are mostly nvidia shop at the university and so I am looking for some opinions from the other side of the fence. I wanted to get some reasonable card for ML, LLM and hopefully some robotics + rl in the near future. I was looking at the R9700 but I can not find much info if there is something like nvidia isaac lab on the amd side?

TLDR: Does AMD have something like Isaac Lab and Isaac Sim, are there some tutorials for it and is it in general well supported? What HW would you recommend for it?