Nonlinear Optimization

Ivan Perez Avellaneda, Ph.D.

April 26th, 2025

Line Search Methods

Inexact Search

\[ x_{k+1} = x_{k} - \gamma \nabla f(x_{k}) \]

Proposition: The solution to the following problem: \[ \min_{h} f(x) + \nabla f(x)\cdot h + \cdots, \quad \text{s.t } ||h|| \leq 1 \] is given by \( h = \nabla f(x)/||\nabla f(x)|| \).

Proof: Using the Taylor approximation, we solve the following problem: \[ \min_{h} f(x) + \nabla f(x)\cdot h, \quad \text{s.t } ||h|| \leq 1 \] Use \(\nabla f(x)\cdot h = ||\nabla f(x)||\ ||h||\cos(\theta)\) where \(\theta\) is the angle between the vectors \(\nabla f(x)\) and \(h\). The minimum is achieved when \(\cos(\theta) = -1\), i.e. \(h = -k\nabla f(x) \) with \(k \geq 0\). Take \(k = 1/||\nabla f(x)||\) then \( h = \nabla f(x)/||\nabla f(x)|| \) to satisfy the constraint.

∎

Ivan Perez Avellaneda — Fig.1 - Reachable set overestimation by the Mixed-Monotonicity, Gradient Descent and Input-Output Mixed-Monotonicity approaches.

Here is some code:

example.py Python

import math
      from typing import List, Dict, Optional
      
      # This is a sample Python class demonstrating various features
      @dataclass
      class DataProcessor:
          name: str
          values: List[float]
          options: Dict[str, bool] = field(default_factory=dict)
          
          def process_data(self) -> Optional[float]:
              """
              Process the values list and return statistical information
              based on the options provided.
              """
              if not self.values:
                  return None
                  
              results = []
              
              if self.options.get("calculate_mean", False):
                  mean = sum(self.values) / len(self.values)
                  results.append(mean)
                  
              if self.options.get("calculate_std_dev", False):
                  mean = sum(self.values) / len(self.values)
                  variance = sum((x - mean) ** 2 for x in self.values) / len(self.values)
                  std_dev = math.sqrt(variance)
                  results.append(std_dev)
                  
              return sum(results) if results else None
              
      # Example usage
      if __name__ == "__main__":
          processor = DataProcessor(
              name="Temperature Analysis",
              values=[21.5, 22.1, 23.4, 20.8, 21.9],
              options={"calculate_mean": True, "calculate_std_dev": True}
          )
          result = processor.process_data()
          print(f"Processing result: {result:.2f}")