Hyperparameter

ENGLISH | 中文文档

Hyperparameter, Make configurable AI applications. Build for Python/Rust hackers.

Hyperparameter is a versatile library designed to streamline the management and control of hyperparameters in machine learning algorithms and system development. Tailored for AI researchers and Machine Learning Systems (MLSYS) developers, Hyperparameter offers a unified solution with a focus on ease of use in Python, high-performance access in Rust and C++, and a set of macros for seamless hyperparameter management.

Key Features

For Python Users

• Pythonic Syntax: Define hyperparameters using keyword argument syntax;

• Intuitive Scoping: Control parameter scope through with statement;

• Configuration File: Easy to load parameters from config files;

For Rust and C++ Users

• High-Performance Backend: Hyperparameter is implemented in Rust, providing a robust and high-performance backend for hyperparameter management. Access hyperparameters in Rust and C++ with minimal overhead, making it ideal for ML and system developers who prioritize performance.

• Macro-Based Parameter Management: Hyperparameter provides a set of macros for both Rust and C++ users. These macros mimic Python's with statements and adhere to language-specific scoping rules.

• Compile-Time Hashing: Both Rust and C++ interfaces utilize compile-time hashing of hyperparameter names, reducing runtime hash computation overhead.

Quick Start

Installation

pip install hyperparameter

Python

from hyperparameter import auto_param, param_scope

@auto_param("foo")
def foo(x=1, y="a"):
    return f"x={x}, y={y}"

foo()  # x=1, y='a'

with param_scope(**{"foo.x": 2}):
    foo()  # x=2, y='a'

Rust

fn foo() -> i32 {
    with_params! {
        get x = foo.x or 1i32; // Read hyperparameter with default value

        println!("x={}", x);
    }
}

fn main() {
    foo(); // x=1

    with_params! {
        set foo.x = 2i32; // Set hyperparameter

        foo(); // x=2
    }

    foo(); // x=1
}

C++

ASSERT(1 == GET_PARAM(a.b, 1), "get undefined param");
{
  auto guard = WITH_PARAMS(a, 1,        //
                            a.b, 2.0,    //
                            a.b.c, true, //
                            a.b.c.d, "str");
  ASSERT(1 == GET_PARAM(a, 0), "get int value");
  ASSERT(1 == GET_PARAM(a, 0), "get int value");
}

Detailed Usage Examples

Support for Default Values

Python

x = param_scope.foo.x | "default value"

Rust

get x = foo.x or "default value";

Scope Control of Parameter Values

Python

with param_scope() as ps: # 1st scope start
    ps.foo.x=1
    with param_scope() as ps2: # 2nd scope start
        ps.foo.y=2
    # 2nd scope end
# 1st scope end

Rust

with_params!{ // 1st scope start
    set foo.x=1;

    with_params!{ //2nd scope start
        set foo.y=2

        ...
    } // 2nd scope end
} // 1st scope end

Thread Isolation/Thread Safety

Python

@auto_param("foo")
def foo(x=1): # Print hyperparameter foo.x
    print(f"foo.x={x}")

with param_scope() as ps:
    ps.foo.x=2 # Modify foo.x in the current thread
    
    foo() # foo.x=2
    threading.Thread(target=foo).start() # foo.x=1, new thread's hyperparameter value is not affected by the main thread

Rust

fn foo() { // Print hyperparameter foo.x
    with_params!{
        get x = foo.x or 1;

        println!("foo.x={}", x);
    }
}

fn main() {
    with_params!{
        set foo.x = 2; // Modify foo.x in the current thread
        
        foo(); // foo.x=2
        thread::spawn(foo); // foo.x=1, new thread's hyperparameter value is not affected by the main thread
    }
}

Command Line Application

In command line applications, it's common to define hyperparameters using command line arguments (e.g., -D, --define) and control hyperparameters on the command line. Here's an example in Python and Rust:

Python

# example.py
from hyperparameter import param_scope, auto_param

@auto_param("example")
def main(a=0, b=1):
    print(f"example.a={a}, example.b={b}")

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser()

    parser.add_argument("-D", "--define", nargs="*", default=[], action="extend")
    args = parser.parse_args()

    with param_scope(*args.define):
        main()

Rust

// example.rs
use hyperparameter::*;
use hyperparameter_derive::Parser;

fn main() {
    #[derive(Parser, Debug)]
    struct DeriveArgs {
        #[arg(short = 'D', long)]
        define: Vec<String>,
    }

    let args = DeriveArgs::parse();

    with_params! {
        params ParamScope::from(&args.define);

        foo()
    }
}

fn foo() {
    with_params! {
        get a = example.a or 0;
        get b = example.b or 1;
        
        println!("example.a={}, example.b={}",a ,b);
    }
}

More Examples

parameter tunning for researchers

This example demonstrates how to use hyperparameter in research projects, and make experiments reproducible.

experiment tracing for data scientists

This example showcases experiment management with hyperparameter and result tracing with mlflow.tracing.