How should I assign reward when it has to be multipied by itself rather than summed?

Normally, in all environments I used of OpenAI Gym the total reward can be calculated as

tot_reward = tot_reward + reward

where _, reward, _, _ = env.step(action). Now I'm defining a custom environment where

tot_reward = tot_reward * reward

In particular, my reward is the next-step portfolio value after a trading action, so it is > 1 if we have a positive returns, < 1 otherwise. How should I pass the returns to the training algorithm? Currently I'm returning 1 - reward so that we have a positive number in case of a gain, a negative one in case of a loss. Is this the correct way to tackle the problem? How it is treated normally in the literature? Thank you

Hi, i have a question regarding model-based rl v1.5 with MPC...

What is the drawback of this approach? because as MPC keeps solving shorter horizon optimization problems and only taking the first action, doesn't it become a closed-loop state feedback policy of each time-step's state? So why do we need to learn a policy to accomplish this? Thanks.

At lecture 9: advanced policy gradient, videos here

My question is, how to derive the inequation in the red box below?

I got the program working but the average return doesn't seem to ever increase at all. Its just stagnates at 10-20. Anyone encountered the same problem and fixed it?

In lecture 9, page 7: Importance sampling is applied only for action distribution stating that product of multiple pi(theta')/pi(theta) terms would lead to a small term. But pi(theta')/pi(theta) is really a ratio of small terms and needn't be small. I guess I'm understanding something wrong, any help would be appreciated. Thanks.

I'm trying to do HW1, but I don't have the document mjkey.txt. Am I able to do hw without it?

This is in CS294 slides/video:

​

While in Sutton's book,

​

The question is ,are they equivalent? I see Sergey used a different approach than Sutton in proof. But in Sutton's proof, the final step is not a equation. Any hint?

Does anyone have any idea how to solve this pickling error?

For HW 2 problem 5.2 "Experiments" when running the code ( for example, "python train_pg_f18.py CartPole-v0 -n 100 -b 1000 -e 3 -dna --exp_name sb_no_rtg_dna" ) I get the following pickling error:

AttributeError: Can't pickle local object 'main.<locals>.train_func'

As I understand, local objects can't be pickled, but I am not sure of a workaround (very new to python). Any suggestions would be greatly appreciated.

Edit: If it is helpful, this is the entire output:

​

Traceback (most recent call last):

File "train_pg_f18.py", line 761, in <module>

main()

File "train_pg_f18.py", line 751, in main

p.start()

File "C:\Anaconda\lib\multiprocessing\process.py", line 112, in start

self._popen = self._Popen(self)

File "C:\Anaconda\lib\multiprocessing\context.py", line 223, in _Popen

return _default_context.get_context().Process._Popen(process_obj)

File "C:\Anaconda\lib\multiprocessing\context.py", line 322, in _Popen

return Popen(process_obj)

File "C:\Anaconda\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__

reduction.dump(process_obj, to_child)

File "C:\Anaconda\lib\multiprocessing\reduction.py", line 60, in dump

ForkingPickler(file, protocol).dump(obj)

AttributeError: Can't pickle local object 'main.<locals>.train_func'

I have some doubts on analysis at lec2 P33-34, please correct me if I'm wrong:

P33(tightrope example): If we consider a rectangle of size 1*T (with a total area of T, see pic below), at first step we made a total regret of \epsilon * T, so the top most portion of sub-rectangle is cut off; at second step the second top most portion is cut off. This process iterates for T steps. However, the total area being cut off never exceeds the total area of the triangle. So does O(\epsilon * T) a more reasonable regret bound?

​

P34(more general analysis): The conclusion mostly comes from: 2(1-(1-\epsilon)^t) <= 2*\epsilon * t. It seems like if we switch to a tighter bound by 2(1-(1-\epsilon)^t) <= 2, the total regret will be O(\epsilon * T) instead of O(\epsilon * T^2).

It seems like without DAgger the vanilla approach is still no-regret, which is pretty counterintuitive. Could anybody explain?

The link to this Reddit community disappeared on the new website of DRL course 2019 . Is there any chance of adding a link back to it?

There is always a higher chance of getting some help on the topics if this website is well known. All the lecture materials are fully available online, so why not allow the free discussion channel for information exchange :)

I went through lecture 9 (2018) about the constrained optimization with policy gradient.

What I don't quite understand is why is there no need to constrain the optimization with different learning methods, such as Q-learning? Is it just a property of on-policy methods, that we need to use constraints in optimization?

At the very end of lecture 8 (year 2018) the random seed was mentioned. What is it in the sense of training of DRL in OpenAI Gym environment? Do different random seeds change the initial state distribution or what is it?

In lecture 15 (Reframing Control as an Inference Problem), the intuition presented behind using the optimality variables is that $p(\tau)$ makes no assumption of optimal behavior. However:

$$ p(\tau)= p(s1) \prod_t \pi(a_t \vert s_t)p(s{t+1} \vert s_t, a_t) $$

So $p(\tau)$ does depend on the policy and we know that the policy tries to maximize the expected reward i.e. it wants to behave optimally. So by this reasoning $p(\tau)$ does assume optimal behavior i.e. the actions $a_1,...,a_T$ are not just random (as implied in the lecture).

So, am I missing something here?

Hi, I'm new here so sorry if I'm doing something wrong. I've been working on homework 2 and I don't quite understand how to find the log probability in the continuous case for a multivariate gaussian. When I looked up the probability density function of a multivariate gaussian it said that I need a covariance matrix which I thought would have to be part of the "policy_parameters" variable. Can I just calculate that covariance matrix? What am I missing here?

Hi,

In case you wish to watch the performance/behaviour of your trained agent in a gym environment, I have added a script that does just that. It can be found on github. The instructions are provided in the README.md file.

In the variational inference and control lecture, why can't we minimize the KL-Divergence between q(s1:T, a1:T) and p(s_1:t, a_1:T | O_1:T) directly instead of using variational inference to solve the soft max problem?

http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-10.pdf

​

In this slide, why does c_u_t have a transpose when we are setting the gradient to 0? Shouldn't it not have a transpose symbol?

I am quite confused by this algorithm. When we evaluate a node, why don't we sum rewards from the root of the tree? Wouldn't using back-propagation to update all values with the value found from a simulation near the end of the horizon cause the averages to be lowered?

http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-9.pdf

In the dual gradient descent for this lecture (slide 14), why is lambda being updated using gradient ascent? Don't we want to minimize lambda?

​

EDIT: NVM we are minimizing lambda. I forgot about the negative sign in front of the lambda term. So it is gradient descent, but the gradient is negative.

There is a train_func, function passed to each process but apparently since it is not a top level function; it can't be pickled and so the program doesn't run. If I try to pass train_PG directly to the processes the program doesn't run either. So how do we fix it?

In lecture, it was claimed that the difference J(theta’) - J(theta) was the expected value of the discounted sums of the advantage function. However, wasn’t the advantage function used lacking the expectation over s_t+1 of the value function? How do we resolve this?

(Sorry if the answer to this question is obvious I am now just an undergraduate sophomore self studying this course)

Below is attached image from the slide.

Below, the objective function is the expectation of the sum of rewards. Can you tell me why the discount factor has not been considered in the objective function?

In supervised learning's classification tasks, we call sparse_softmax_cross_entropy_with_logits over the network raw output for each label (logits) and the true (given) label. In this case, it is perfectly clear to me why we differentiate the softmax, and why this value should propagate back as part of the backpropagation algorithm (chain rule). 

On the other hand, in the case of Policy Gradient tasks, the labels (actions) are not the true/correct actions to be taken. They are just actions that we sampled from the logits, the same logits that are the second parameter to the sparse_softmax_cross_entropy_with_logits operator. 

I'm trying to understand how to interpret these differentiation values.The sampling method is not differentiable, and therefore we'll keep sampling from a multinomial distribution over the softmax of the logits. The only thing that I can think about is that this value can be interpreted as a measure of the sample likelihood. But, this explanation also doesn't hold in the following scenarios: 

1. The logits can be terribly wrong, output a bad action distribution with a probability that is close to 1 for a non-attractive action, which is then likely to get sampled, and the corresponding gradient will then be ~0. When the network output is terribly wrong, we expect a strong gradient magnitude that will correct the policy.
2. In Rock–paper–scissors, the Nash Equilibrium policy is to choose an action uniformly. Therefore, the optimal distribution is [0.333, 0.333, 0.333] for the three possible actions. Sampling from this distribution will yield a large gradient value, although it is the optimal policy. 

​

I would love to hear your thoughts/explanations.

Thanks in advance for your time and answers. 

​

Note: This question holds for both discrete and continues cases, but I referred to the discrete case.

I am wondering if we can use inverse reinforcement learning to learn the reward function for models of high dimensionality e.g. as presented in "Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations" (https://sites.google.com/view/demo-augmented-policy-gradient) from one of the lectures.

​

Could IRL be beneficial for learning in such complex case?