Skip to content

Exploring Variational Deep Q Networks. A study undertaken for the University of Cambridge's R244 Computer Science Masters Course. Inspired by https://arxiv.org/abs/1711.11225/.

License

Notifications You must be signed in to change notification settings

bellthomas/VDQN

Repository files navigation

Exploring Variational Deep Q Networks

This study provides a research-ready implementation of Tang and Kucukelbir's Variational Deep Q Network, a novel approach to maximising the efficiency of exploration in complex learning environments using Variational Bayesian Inference, using the Edward PPL. Alongside reference implementations of both Traditional and Double Deep Q Networks, a small novel contribution is presented --- the Double Variational Deep Q Network, which incorporates improvements to increase the stability and robustness of inference-based learning. Finally, an evaluation and discussion of the effectiveness of these approaches is discussed in the wider context of Bayesian Deep Learning.

The full report is available here: Exploring VDQNs.

Using the Framework

These steps assume a fresh Ubuntu installation is being used. The process may deviate slightly if a different system is used. A helper script is included for installing the required dependencies on Linux; the process is near-identical for macOS.

git clone https://github.com/HarriBellThomas/VDQN.git
cd VDQN
./init.sh
source env/bin/activate

There are four main files to be aware of:

  • run.py --- this is the main entrypoint for running a single instance of one of the four algorithms. It accepts a number of CLI arguments for configuring the parameters it used.
  • DQN.py --- this is the source file containing the implementations for both DQN and DDQN.
  • VDQN.py --- this is the source file containing the implementations for both VDQN and DVDQN.
  • drive.py --- this script is the driver used for running experiments at scale. It constructs a collection of 80 experiments, and iteratively loops through them.

The run.py script can be used as follows:

python3 run.py
    --algorithm DQN|DDQN|VDQN|DVDQN \
    --environment CartPole-v0 \
    --episodes 200 \
    --timesteps 200 \
    --lossrate 1e-2

About

Exploring Variational Deep Q Networks. A study undertaken for the University of Cambridge's R244 Computer Science Masters Course. Inspired by https://arxiv.org/abs/1711.11225/.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published