Skip to main content
Data Science

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

3
  • $\begingroup$ I think I am starting to understand. So indeed, during differentiation, wrapping $\pi(\tau)$ inside of $log$ allows us to A: detach the "Probability of the next state under the dynamics" and B: detach the "probability of the initial state", from C: the "probability of action given state" all of which sit inside $\pi(\tau)$. We then can focus on the latter, without having to know the values of probabilities A or B. $\endgroup$ Commented Sep 10, 2018 at 1:09
  • $\begingroup$ without this trick, we would need to know values for A and B because they would stay as coefficients due to the chain rule. That would mean we would need to know precise distributions which would be incredibly difficult to estimate (and the A would have to be known for every state) $\endgroup$ Commented Sep 10, 2018 at 1:10
  • $\begingroup$ In this case it's a little unclear why S. Levine used $\pi (\tau)$ showing that we worked with a trajectory (as I would expect) and David Silver used $\pi (s, a)$ to only show a single step, not a trajectory. Edit actually, I think I now remember David mentioning "this was done for simplification, the trajectories would be a homework". $\endgroup$ Commented Sep 10, 2018 at 1:43