Reinforcement learning lab

Frozen Lake

The agent starts on S and has to reach G. Dark water is a hole. Train a Q-table, a policy gradient, or a small neural network, then watch the policy try to cross.

Click a square to paint it. S tile and Begin here are the same square: it becomes S, and every episode starts there.

  • Start 0
  • Ice 0
  • Hole 0
  • Goal 1
  • Agent

Setting up the lake.

Arrow keys walk the live lake. Playback replays one recorded episode, including slips.

Training

γ discounts the future. At 0.99, a reward six steps away is still worth about 94% of a reward received now.

Tile rewards

Paid when the agent lands on that tile. The opening square pays nothing until a later step lands there. Same tile again is added when that landing is a tile the agent has already occupied in this episode. Defaults: 0 on ice and holes, 1 on the goal, 0 for a repeat visit.

For One episode. At 1 every action is random. At 0 the agent picks the largest number in the current row.

Results

Greedy success

—

Average reward

—

Average steps

—

Greedy success means the agent reached the goal, with exploration turned off. Average reward is the sum of the tile rewards along the way. With the defaults, a success pays 1 and every other episode pays 0.

Point at a tile after training to read the four action values.

Learning curves

Q-table

Starting values, drawn at random between 0 and 0.1. Holes and the goal stay at 0.

State Tile Left Down Right Up
No Q-table yet.