Error when using SimpleTransformer: TypeError("forward() missing 1 required positional argument: 't'")

[joelrorseth]

Hi, I am trying to run SimpleTransformer, but I cannot seem to resolve the following error (which is being thrown for each run):
TypeError("forward() missing 1 required positional argument: 't'")

I have double checked my code against the documentation and test file for SimpleTransformer, but I cannot spot any issue. I have also disabled the GPU, since I suspect SimpleTransformer may have issues running on GPU. Here is my config, does anything look wrong, or could this be a bug?

DATETIME_START = "2016-02-24"
RELEVANT_COLS = ["Lane 1 Flow (Veh/5 Minutes)", "day_of_week"]
TARGET_COLS = ["Lane 1 Flow (Veh/5 Minutes)"]

def build_config(project_name, wandb_config, file_path, train_end, valid_end):
    return {
        "model_name": "SimpleTransformer",
        "model_type": "PyTorch",
        "use_decoder": True,
        "model_params": {
            "number_time_series": 2,
            "seq_length": wandb_config["forecast_history"],
            "output_seq_len": wandb_config["out_seq_length"],
            "d_model": 128,
            "n_heads": 8,
            "dropout": 0.1,
            "forward_dim": 2048,
            "sigmoid": False
        },
        "dataset_params": {
            "class": "default",
            "training_path": file_path,
            "validation_path": file_path,
            "test_path": file_path,
            "batch_size": wandb_config["batch_size"],
            "forecast_history": wandb_config["forecast_history"],
            "forecast_length": wandb_config["out_seq_length"],
            "train_end": train_end,
            "valid_start": int(train_end+1),
            "valid_end": int(valid_end),
            "test_start": int(valid_end) + 1,
            "target_col": TARGET_COLS,
            "relevant_cols": RELEVANT_COLS,
            "scaler": "StandardScaler",
            "interpolate": False
        },
        "early_stopping": {
            "patience": 2
        },
        "training_params": {
            "criterion": "MSE",
            "optimizer": "Adam",
            "optim_params": {},
            "lr": wandb_config["lr"],
            "epochs": 10,
            "batch_size": wandb_config["batch_size"]
        },
        "GCS": False,
        "sweep": True,
        "wandb": {
            "project": project_name,
            "name": None,
            "tags": None
        },
        "forward_params": {
            "t": {}
        },
        "takes_target": True,
        "metrics": ["MSE"],
        "inference_params": {
            "datetime_start": DATETIME_START,
            "hours_to_forecast": 150,
            "test_csv_path": file_path,
            "decoder_params": {
                "decoder_function": "simple_decode",
                "unsqueeze_dim": 1
            },
            "dataset_params": {
                "file_path": file_path,
                "forecast_history": wandb_config["forecast_history"],
                "forecast_length": wandb_config["out_seq_length"],
                "relevant_cols": RELEVANT_COLS,
                "target_col": TARGET_COLS,
                "scaling": "StandardScaler",
                "interpolate_param": False
            }
        }
    }

Thanks for your help!

[isaacmg]

Do you have the full traceback (e.g. line number it fails on)? It is hard to know otherwise.

[joelrorseth]

Sorry about that, there really wasn't any more information in the actual output, but some details were logged to debug.log under the wandb directiory:

2021-07-06 19:13:35,168 INFO    Thread-22 :99618 [wandb_setup.py:_flush():69] setting env: {'project': 'sample-transformer', 'entity': 'joelrorseth', 'root_dir': '/project/6001748/rorsethj/time-series/flow-forecast/SimpleTransformer', 'run_id': 'fsm7yosh', 'sweep_param_path': '/project/6001748/rorsethj/time-series/flow-forecast/SimpleTransformer/wandb/sweep-8vuerads/config-fsm7yosh.yaml', 'sweep_id': '8vuerads'}
2021-07-06 19:13:35,169 INFO    Thread-22 :99618 [wandb_setup.py:_flush():69] setting login settings: {}
2021-07-06 19:13:35,170 INFO    Thread-22 :99618 [wandb_init.py:_log_setup():336] Logging user logs to /project/6001748/rorsethj/time-series/flow-forecast/SimpleTransformer/wandb/run-20210706_191335-fsm7yosh/logs/debug.log
2021-07-06 19:13:35,170 INFO    Thread-22 :99618 [wandb_init.py:_log_setup():337] Logging internal logs to /project/6001748/rorsethj/time-series/flow-forecast/SimpleTransformer/wandb/run-20210706_191335-fsm7yosh/logs/debug-internal.log
2021-07-06 19:13:35,171 INFO    Thread-22 :99618 [wandb_init.py:init():369] calling init triggers
2021-07-06 19:13:35,172 INFO    Thread-22 :99618 [wandb_init.py:init():376] wandb.init called with sweep_config: {'batch_size': 2, 'forecast_history': 2, 'lr': 0.01, 'out_seq_length': 1}
config: {}
2021-07-06 19:13:35,173 INFO    Thread-22 :99618 [wandb_init.py:init():418] starting backend
2021-07-06 19:13:35,173 INFO    Thread-22 :99618 [backend.py:_multiprocessing_setup():71] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2021-07-06 19:13:35,182 INFO    Thread-22 :99618 [backend.py:ensure_launched():123] starting backend process...
2021-07-06 19:13:35,203 INFO    Thread-22 :99618 [backend.py:ensure_launched():128] started backend process with pid: 107880
2021-07-06 19:13:35,206 INFO    Thread-22 :99618 [wandb_init.py:init():423] backend started and connected
2021-07-06 19:13:35,211 INFO    Thread-22 :99618 [wandb_run.py:_config_callback():749] config_cb None None {'batch_size': 2, 'forecast_history': 2, 'lr': 0.01, 'out_seq_length': 1}
2021-07-06 19:13:35,213 INFO    Thread-22 :99618 [wandb_init.py:init():465] updated telemetry
2021-07-06 19:13:35,214 INFO    Thread-22 :99618 [wandb_init.py:init():484] communicating current version
2021-07-06 19:13:40,216 INFO    Thread-22 :99618 [wandb_init.py:init():497] communicating run to backend with 30 second timeout
2021-07-06 19:14:01,137 INFO    Thread-22 :99618 [wandb_init.py:init():522] starting run threads in backend
2021-07-06 19:14:06,158 INFO    Thread-22 :99618 [wandb_run.py:_console_start():1524] atexit reg
2021-07-06 19:14:06,172 INFO    Thread-22 :99618 [wandb_run.py:_redirect():1371] redirect: SettingsConsole.REDIRECT
2021-07-06 19:14:06,172 INFO    Thread-22 :99618 [wandb_run.py:_redirect():1376] Redirecting console.
2021-07-06 19:14:06,173 ERROR   Thread-22 :99618 [wandb_run.py:_redirect():1441] Failed to redirect.
Traceback (most recent call last):
  File "/project/6001748/rorsethj/time-series/flow-forecast/venv/lib/python3.7/site-packages/wandb/sdk/wandb_run.py", line 1434, in _redirect
    out_redir.install()
  File "/project/6001748/rorsethj/time-series/flow-forecast/venv/lib/python3.7/site-packages/wandb/sdk/lib/redirect.py", line 605, in install
    _WSCH.add_fd(self._pipe_read_fd)
  File "/project/6001748/rorsethj/time-series/flow-forecast/venv/lib/python3.7/site-packages/wandb/sdk/lib/redirect.py", line 555, in add_fd
    self._register()
  File "/project/6001748/rorsethj/time-series/flow-forecast/venv/lib/python3.7/site-packages/wandb/sdk/lib/redirect.py", line 540, in _register
    old_handler = signal.signal(signal.SIGWINCH, lambda *_: None)
  File "/cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib/python3.7/signal.py", line 47, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread
2021-07-06 19:14:06,175 INFO    Thread-22 :99618 [wandb_init.py:init():546] run started, returning control to user process
2021-07-06 19:14:07,229 INFO    Thread-22 :99618 [wandb_setup.py:_flush():69] setting env: {'project': 'sample-transformer', 'entity': 'joelrorseth', 'root_dir': '/project/6001748/rorsethj/time-series/flow-forecast/SimpleTransformer', 'run_id': 'fsm7yosh', 'sweep_param_path': '/project/6001748/rorsethj/time-series/flow-forecast/SimpleTransformer/wandb/sweep-8vuerads/config-fsm7yosh.yaml', 'sweep_id': '8vuerads'}
2021-07-06 19:14:07,230 INFO    Thread-22 :99618 [wandb_setup.py:_flush():69] setting login settings: {}
2021-07-06 19:14:07,231 INFO    Thread-22 :99618 [wandb_init.py:_log_setup():336] Logging user logs to /project/6001748/rorsethj/time-series/flow-forecast/SimpleTransformer/wandb/run-20210706_191407-fsm7yosh/logs/debug.log
2021-07-06 19:14:07,232 INFO    Thread-22 :99618 [wandb_init.py:_log_setup():337] Logging internal logs to /project/6001748/rorsethj/time-series/flow-forecast/SimpleTransformer/wandb/run-20210706_191407-fsm7yosh/logs/debug-internal.log
2021-07-06 19:14:07,233 INFO    Thread-22 :99618 [wandb_init.py:init():369] calling init triggers
2021-07-06 19:14:07,234 INFO    Thread-22 :99618 [wandb_init.py:init():376] wandb.init called with sweep_config: {'batch_size': 2, 'forecast_history': 2, 'lr': 0.01, 'out_seq_length': 1}
config: {'model_name': 'SimpleTransformer', 'model_type': 'PyTorch', 'use_decoder': True, 'model_params': {'number_time_series': 2, 'seq_length': 2, 'output_seq_len': 1, 'd_model': 128, 'n_heads': 8, 'dropout': 0.1, 'forward_dim': 2048, 'sigmoid': False}, 'dataset_params': {'class': 'default', 'training_path': '../data/sample.csv', 'validation_path': '../data/sample.csv', 'test_path': '../data/sample.csv', 'batch_size': 2, 'forecast_history': 2, 'forecast_length': 1, 'train_end': 4500, 'valid_start': 4501, 'valid_end': 6000, 'test_start': 6001, 'target_col': ['Lane 1 Flow (Veh/5 Minutes)'], 'relevant_cols': ['Lane 1 Flow (Veh/5 Minutes)', 'day_of_week'], 'scaler': 'StandardScaler', 'interpolate': False}, 'early_stopping': {'patience': 2}, 'training_params': {'criterion': 'MSE', 'optimizer': 'Adam', 'optim_params': {}, 'lr': 0.01, 'epochs': 10, 'batch_size': 2}, 'GCS': False, 'sweep': True, 'wandb': {'project': 'sample-transformer', 'name': None, 'tags': None}, 'forward_params': {'t': {}}, 'takes_target': True, 'metrics': ['MSE'], 'inference_params': {'datetime_start': '2016-02-24', 'hours_to_forecast': 150, 'test_csv_path': '../data/sample.csv', 'decoder_params': {'decoder_function': 'simple_decode', 'unsqueeze_dim': 1}, 'dataset_params': {'file_path': '../data/sample.csv', 'forecast_history': 2, 'forecast_length': 1, 'relevant_cols': ['Lane 1 Flow (Veh/5 Minutes)', 'day_of_week'], 'target_col': ['Lane 1 Flow (Veh/5 Minutes)'], 'scaling': 'StandardScaler', 'interpolate_param': False}}}
2021-07-06 19:14:07,235 INFO    Thread-22 :99618 [wandb_init.py:init():415] wandb.init() called when a run is still active
2021-07-06 19:14:07,243 INFO    Thread-22 :99618 [wandb_watch.py:watch():39] Watching
2021-07-06 19:41:21,372 INFO    Thread-22 :99618 [wandb_config.py:__setitem__():141] config set save_path = model_save/06_July_202107_41PM_model.pth - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x2b1734579150>>
2021-07-06 19:41:21,599 INFO    Thread-22 :99618 [wandb_run.py:_config_callback():749] config_cb save_path model_save/06_July_202107_41PM_model.pth None
2021-07-06 19:41:24,268 INFO    Thread-22 :99618 [wandb_run.py:finish():1167] finishing run joelrorseth/sample-transformer/fsm7yosh

This output doesn't appear to mention anything about the positional argument error that was logged to stdout, am I looking in the right place? It's probably worth noting that several wandb runs occurred, logging similar output / errors, before the entire thing actually stopped execution.

[isaacmg]

Yeah Wandb doesn't do a very good stack trace. I have a video where I explain how to get the full error message I'll link to it

[isaacmg]

Sorry just realized I didn't link to it https://youtu.be/ZcHO-k3dkX0?t=3699 starts right around here.

[strakehyr]

Hi, is there any solution to this? I am having the same issue.