Reduce memory usage!

[georgemilosh]

There are not so many machines that can handle all the data in memory. We need to find a way to deal with this issue. There are many materials online which show how it can be achieved in tensorflow (pipelines etc). The problem is that I still get out of memory issue (the process simply gets killed).

[georgemilosh]

• Using tensorflow pipelines which should take care of our custom stratified k-fold-cross-validation that doesn't mix different years
• When computing $A(t)$ we only need a small portion of the globe, so no need to load all the fields for that purpose
• Balancing folds should be possible just based on this $A(t)$ but perhaps we don't need to actually shuffle the data. We just need to tell tensorflow the map to the year labels in the pipelines
• Consider tensorflow pipelines which train from the disk rather than RAM

[georgemilosh]

I should note that there are extra memory leaks when trying current tensorflow version (that is no longer tensorflow-gpu and that is installed with pip rather than conda)

[AlessandroLovo]

Check for easy improvement when loading data, especially if we don't want to use the whole dataset. In particular inside the Plasim_Field object

[georgemilosh]

At the moment I create a separate dataset with cdo that's the subset, and everything works much faster. But I feel this is ad hoc. Alternatively we could always work with smaller datasets that are to be concatenated in Learn2_new.py if needed, but also a bit annoying

[AlessandroLovo]

Yeah, indeed. I was planning to do line by line evaluation of the code monitoring the RAM to see exactly when we load the data into memory and if we can do something about it, but I cannot guarantee I'll have time for that

[AlessandroLovo]

I was just doing some runs and I noticed that now with 1000 years of data training a CNN uses 1.3TB of virtual memory... Did you change anything? If yes is not going in the right direction 😆

[AlessandroLovo]

Found the reason: before the network had ~200.000 trainable parameters... now it has ~8 millions...

[AlessandroLovo]

And that is because the MaxPool layers have disappeared

[georgemilosh]

Yes, sorry forgot to re-implement them. As modified the way create_model works. I was working with strides instead.

[georgemilosh]

Although virtual memory normally doesn't matter.

[AlessandroLovo]

I will re-implement the MaxPool for backward compatibility

[georgemilosh]

My view is that we will need to implement tensorflow datasets. Operations such as shuffling (balancing) and train/validation have to be done virtually by permuting only indices. The dataset has to somehow know which portions of the full dataset it has to provide for the next(batch). In pytorch it is quite easy to control how data is extracted. I don't like that we have to copy paste data in memory. Normalization is another step, but it could be achieved by a layer implementation rather than doing it explicitly like we do?