Skip to content

Code and materials for the paper S. Phelps and Y. I. Russell, Investigating Emergent Goal-Like Behaviour in Large Language Models Using Experimental Economics, working paper, arXiv:2305.07970, May 2023

License

Notifications You must be signed in to change notification settings

phelps-sg/llm-cooperation

Repository files navigation

llm-cooperation

GitHub Workflow Status

This repo contains code, explanations and results of experiments to ascertain the propensity of large-language models to cooperate in social dilemmas. The experiments are described in the following papers.

S. Phelps and Y. I. Russell, Investigating Emergent Goal-Like Behaviour in Large Language Models Using Experimental Economics, working paper, May 2023, arXiv:2305.07970

S. Phelps and R. Rannson, Of Models and Tin Men - a behavioural economics study of principal-agent problems in AI alignment using large-language models, working paper, July 2023, arXiv:2307.11137

Getting started

  1. Install miniforge.
  2. In a shell:
export OPENAI_API_KEY='<my key>'
make install
make run

Configuration

To run specific experiments and parameter combinations follow instructions below.

  1. In a shell:
mkdir ~/.llm-cooperation
cat > ~/.llm-cooperation/llm_config.py << EOF


grid = {
        "temperature": [0.1, 0.6],
        "model": ["gpt-3.5-turbo", "gpt-4"],
        "max_tokens": [300]
}


num_replications = 3


experiments = ["dictator", "dilemma"]
EOF
  1. Edit $HOME/.llm-cooperation/llm_config.py with required values.

  2. In a shell:

export OPENAI_API_KEY='<key>'
make run

Contributing

If you have a new experiment then please submit a pull request. All code should have corresponding tests and all experiments should be replicable.