v0.5.2

New feature

The main feature of this patch release is that AccumBN can now be used as drop-in replacement for any BatchNormalization layer, even for pretrained networks. Old weights are sufficiently transferred and documentations have been updated to include how to do this.

import tensorflow as tf
from gradient_accumulator import GradientAccumulateModel
from gradient_accumulator.layers import AccumBatchNormalization
from gradient_accumulator.utils import replace_batchnorm_layers

accum_steps = 4

# replace BN layer with AccumBatchNormalization
model = tf.keras.applications.MobileNetV2(input_shape(28, 28, 3))
model = replace_batchnorm_layers(model, accum_steps=accum_steps)

# add gradient accumulation to existing model
model = GradientAccumulateModel(accum_steps=accum_steps, inputs=model.input, outputs=model.output)

What's Changed

• Docs: Support tf 2.2-2.12 by @andreped in #100
• Allow poorer approximation for older tf versions in model test by @andreped in #101
• Fixed typo in setup.cfg by @andreped in #104
• Ignore .pyc [no ci] by @andreped in #106
• Delete redundant .pyc file [no ci] by @andreped in #107
• Added Applications to README by @andreped in #109
• Fixed whl installation in test CI by @andreped in #110
• Added method to replace BN layers by @andreped in #112

Full Changelog: v0.5.1...v0.5.2

v0.5.1

Announcement

This patch release adds support for all tf versions 2.2-2.12 and Python 3.6-3.11. The model wrapper should work as intended for all combinations, whereas the optimizer is only compatible with tf>=2.8 and with poorer performance for tf>=2.10.

What's Changed

• v0.5.0 zenodo + cite by @andreped in #89
• Added opt distribute unit test + added model distribute test to CI by @andreped in #91
• Further refined the bug report template by @andreped in #97
• Fixed dynamic optimizer wrapper inheritance + support tf >= 2.8 by @andreped in #95
• Fixed tensorflow-datasets protobuf issue by @andreped in #98
• Added model wrapper test for tf<2.8 + refactored tests by @andreped in #99

Full Changelog: v0.5.0...v0.5.1

v0.5.0

New feature!

• Multi-GPU support has now been added! Support has been added for both optimizer and model wrappers.
• Note that only SGD works with the model wrapper, due to challenges controlling the optimizer state during gradient accumulatiom

What's Changed

• DOI badge + bibtex update in README by @andreped in #77
• Fixed DOI in README by @andreped in #78
• batch size vs mini-batch size by @mhoibo in #79
• Docs: README + redirect + tf <=2.10 + python <= 3.11 by @andreped in #80
• Added multi-gpu support (model wrapper) by @tno123 in #82
• Multi-GPU works + macOS-11 CI + Docs update by @andreped in #83
• added parameter count test by @tno123 in #84
• Added linting + author update + v0.5.0 by @andreped in #86
• Reduced sphinx version + fixed import by @andreped in #87
• Docs: added urllib3==1.26.15 to docs req by @andreped in #88

New Contributors

• @mhoibo made their first contribution in #79
• @tno123 made their first contribution in #82

Full Changelog: v0.4.2...v0.5.0

v0.4.2

What's Changed

• Add seemless support for mixed precision in AccumBatchNormalization by @andreped in #66
• Cleanup and refactored unit tests by @andreped in #67
• CI test dispatch + README improvements by @andreped in #69
• Minor fix for Accum BN in 3D [skip ci] by @dbouget in #70
• Added mixed precision AccumBN CI test by @andreped in #74
• Fixed AccumBN to work ND by @andreped in #75
• README technique order update by @andreped in #76

Full Changelog: v0.4.1...v0.4.2

v0.4.1

What's Changed

• Added issue templates by @andreped in #59
• Fixed bug in AccumBatchNormalizer - identical results to Keras BN by @andreped in #61
• Docs: Added AccumBN example + docs README + minor fixes by @andreped in #62
• bump v0.4.1 by @andreped in #63

New API

You can now use gradient accumulation with the AccumBatchNormalization layer:

from gradient_accumulator import GradientAccumulateModel, AccumBatchNormalization
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# define model and add accum BN layer
model = Sequential()
model.add(Dense(32, activation="relu"))
model.add(AccumBatchNormalization(accum_steps=8))
model.add(Dense(10))

# add gradient accumulation to the rest of the model
model = GradientAccumulateModel(accum_steps=8, inputs=model.input, outputs=model.output)

More information about remarks and usage can be found at gradientaccumulator.readthedocs.io

Full Changelog: v0.4.0...v0.4.1

v0.4.0

What's Changed

• Added custom AccumBatchNormalization layer with gradient accumulation support.
• Added more unit tests -> code coverage = 99%
• Made proper documentations which is hosted at gradientaccumulator.readthedocs.io/
• Reduced runtime on several unit tests to make CI jobs faster
• Added fix for protobuf for tfds in CIs
• Reworked README - moved most stuff to the Documentations + added CI section w/badges
• Header image by @jpdefrutos in #51

New Contributors

• @jpdefrutos made their first contribution in #51

New API feature

from gradient_accumulator import AccumBatchNormalization

layer = AccumBatchNormalization(accum_steps=4)

Can be used as a regular Keras BatchNormalization layer, but with reduced functionality.

Full Changelog: v0.3.2...v0.4.0

v0.3.2

What's changed

• Fix for GradientAccumulateOptimizer to support tf >= 2.10 by dynamically inheriting from (legacy) Optimizer related to #37
• Deprecated and removed tensorflow-addons (now only tensorflow is required) related to #40
• Fix for macOS builds failing due to poor dependency versioning related to #39
• Added support for Python 3.11-3.12, see #43
• Added support for TensorFlow 2.2 again (after tf-addons removal), see #44
• Fixed tensorflor-datasets versioning in unit tests to work across all relevant setups, see #41
• Added custom AccumBatchNormalization layer demonstrating similar results to regular keras' BN, see here
• Adding notebook for HuggingFaceTF models by @0syrys in #34
• Improved code coverage from 46% to 74%

Full Changelog: v0.3.1...v0.3.2

New Contributors

• @0syrys made their first contribution in #34

Full Changelog: v0.3.1...v0.3.2

How to install?

pip install gradient-accumulator==0.3.2

New API feature

Custom Batch Normalization layer

from gradient_accumulator.layers import AccumBatchNormalization

model = Sequential()
model.add(AccumBatchNormalization())

v0.3.1

What's changed

• Simplified imports - can now directly import accumulators without middle step
• Renamed GAModelWrapper -> GradientAccumulateModel
• Renamed GAOptimizerWrapper -> GradientAccumulateOptimizer
• Updated README and all CIs accordingly
• Deprecated tensorflow==2.2, due to tensorflow-addons incompatiblity. Now tf >= 2.3 supported.

Full Changelog: v0.3.0...v0.3.1

How to install?

pip install gradient-accumulator==0.3.1

New API!

Model wrapper:

from gradient_accumulator import GradientAccumulateModel

model = Model(...)
model = GradientAccumulateModel(accum_steps=4, inputs=model.input, outputs=model.output)

Optimizer wrapper:

from gradient_accumulator import GradientAccumulateModel

opt = tf.keras.optimizers.SGD(1e-2)
opt = GradientAccumulateOptimizer(accum_steps=4, optimizer=opt)

v0.3.0

What's changed

• Added experimental Optimizer wrapper solution through GAOptimizerWrapper by @andreped in #28
• Added experimental multi-GPU support through GAModelWrapperV2 by @andreped in #28
• Added reduction method to Optimizer wrapper and set it to MEAN by default by @andreped in #29
• Added fix related to model subclassing, see #23
• Added support for and verified that macOS M1 silicon works
• Added GAOptimizerWrapper CI unit test

How to install?

pip install gradient-accumulator==0.3.0

How to use?

Method	Usage
GAModelWrapper	model = GAModelWrapper(accum_steps=4, inputs=model.input, outputs=model.output)
GAOptimizerWrapper	opt = GAOptimizerWrapper(accum_steps=4, optimizer=tf.keras.optimizers.Adam(1e-3))

Full Changelog: v0.2.2...v0.3.0

v0.2.2

This is a minor patch release.

What's changed:

• Added support for tensorflow-metal, enabling GA on macOS with GPUs
• Bug fix related to undefined gradients: #24

Full Changelog: v0.2.1...v0.2.2