VariableElimination NaN values

[jonasvit]

Hello! I have built a BayesianNetwork model using pgmpy library. It all works well if my evaluation dataset does not contain any NaN values. If I have any NaN in any parameter of the model, it gives an error of

IndexError: only integers, slices (':'), ellipsis ('...'), numpy.newaxis ('None') and integer or boolean arrays are valid indices.

The evaluation technique that I use (code snippet):

from pgmpy.inference import VariableElimination infer = VariableElimination(model) infer.query(['Fraud'], evidence={ 'Event': df['Event'].iloc[nrows], 'Profitability': df['Event'].iloc[nrows]

Does anyone know how can I make VariableElimination work including NaN values? I tried to google some solutions but I didn't find any information about handling missing values with VariableElimination. Thank you!

[ankurankan]

@jonasvit Does NaN represent missing values in your dataset? If that is the case, you can just leave it out from the evidence argument and the algorithm will marginalize (i.e. treat it to be possible in any state with probability depending on the training dataset) the missing variable automatically.

[jonasvit]

@ankurankan It actually is the case. I could exclude it from the evidence argument but I am running this code for a large dataset, thus missing values can be in one or another evidence column, so it wouldn't work for a large dataset.

[ankurankan]

@jonasvit Since you will have to do a query call for each row of your dataset (as the evidence argument only accepts single state names),
you can write a simple check before the call which modifies the evidence argument according to the observed values.

[jonasvit]

@ankurankan is there any other way to evaluate one node in the network based on other nodes information just as query does? I need to perform mass scoring every time. Also, I was thinking about modifying the code as you described, I hoped there was a better way that wouldn't be so computationally expensive.