ItCodinTime/neat-optimizer

NEAT Optimizer

NEAT (Noise-Enhanced Adaptive Training) is a novel optimization algorithm for deep learning that combines adaptive learning rates with controlled noise injection to improve convergence and generalization.

Overview

The NEAT optimizer enhances traditional adaptive optimization methods by intelligently injecting noise into the gradient updates. This approach helps:

• Escape local minima more effectively
• Improve generalization performance
• Achieve faster and more stable convergence
• Reduce overfitting on training data

Installation

From PyPI (recommended)

pip install neat-optimizer

From Source

git clone https://github.com/yourusername/neat-optimizer.git
cd neat-optimizer
pip install -e .

Quick Start

import tensorflow as tf
from neat_optimizer import NEATOptimizer

# Create your model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

# Use NEAT optimizer
optimizer = NEATOptimizer(
    learning_rate=0.001,
    noise_scale=0.01,
    beta_1=0.9,
    beta_2=0.999
)

# Compile and train
model.compile(
    optimizer=optimizer,
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

model.fit(x_train, y_train, epochs=10, validation_data=(x_val, y_val))

Key Features

• Adaptive Learning Rates: Automatically adjusts learning rates per parameter
• Noise Injection: Controlled stochastic perturbations for better exploration
• TensorFlow Integration: Drop-in replacement for standard TensorFlow optimizers
• Hyperparameter Flexibility: Customizable noise schedules and adaptation rates

Parameters

• learning_rate (float, default=0.001): Initial learning rate
• noise_scale (float, default=0.01): Scale of noise injection
• beta_1 (float, default=0.9): Exponential decay rate for first moment estimates
• beta_2 (float, default=0.999): Exponential decay rate for second moment estimates
• epsilon (float, default=1e-7): Small constant for numerical stability
• noise_decay (float, default=0.99): Decay rate for noise scale over time

Requirements

• Python >= 3.7
• TensorFlow >= 2.4.0
• NumPy >= 1.19.0

Citation

If you use NEAT optimizer in your research, please cite:

@software{neat_optimizer,
  title={NEAT: Noise-Enhanced Adaptive Training Optimizer},
  author={Your Name},
  year={2025},
  url={https://github.com/yourusername/neat-optimizer}
}

References

• Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
• Neelakantan, A., et al. (2015). Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.06807.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Support

For issues, questions, or feature requests, please open an issue on GitHub.