Support Lexicographic optimization in Blendsearch #972

skzhang1 · 2023-04-01T23:19:11Z

Why are these changes needed?

We plan to support lexicographic optimization in BlendSearch. This pull request is primarily intended to track the implementation progress. We encourage everyone to provide feedback in this pull request regarding both implementation suggestions and application scenarios.

TODO:

Implement a easier ECI calculation method.
Force the global search algorithm to MO-HPO in lexicographic optimization.
Add tests and docs.

Related issue number

N/A

Checks

I've used pre-commit to lint the changes in this PR, or I've made sure lint with flake8 output is two 0s.
I've included any doc changes needed for https://microsoft.github.io/FLAML/. See https://microsoft.github.io/FLAML/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

sonichi · 2023-06-12T18:18:22Z

Is it better to create a new subclass than modifying the original BlendSearch? It's an open question.

skzhang1 · 2023-06-13T13:46:27Z

Is it better to create a new subclass than modifying the original BlendSearch? It's an open question.

I think we do not need to do it for now. Because LexiFlow is also not independent of flow2 in the current version.

sonichi

This PR involves many big changes. There are many changes I don't understand. Could you summarize the changes first? Please also justify why a particular change should be done in that place. We do not want to add unnecessary changes or break the original functionality.
Later, I'll also request adding comments for all the non-obvious changes. Let's get a high-level understanding first.

sonichi · 2023-06-16T14:57:39Z

flaml/tune/searcher/blendsearch.py

-            metric += self.lagrange
+            if self.lexico_objectives:
+                self._metric_constraints = None
+                logger.info("Do not support providing metric_constraints in lexicographic optimization for now.")


Raise an error? Metric constraints should be provided via a target.

sonichi · 2023-06-17T02:20:23Z

flaml/tune/tune.py

+            assert "lexico_algorithm" in lexico_objectives and lexico_objectives["lexico_algorithm"] in [
+                "CFO",
+                "BlendSearch",
+            ], 'When performing lexicographic optimization, "lexico_algorithm" should be provided in lexicographic objectives (CFO or BlendSearch).'
+            SearchAlgorithm = CFO if lexico_objectives["lexico_algorithm"] == "CFO" else BlendSearch


This is strange. Redundant with search_alg.

I think it is simpler for users to specify algorithm in lexico_objectives. Because search_alg in tune represents algorithm instance rather than string. We have also not tried to use tune by passing instance of CFO before.

You can support str type search_alg in tune to make it easier. But don't create a new argument which is confusing.
AutoML uses tune by passing instance of CFO.

sonichi · 2023-06-17T02:29:49Z

flaml/tune/searcher/suggestion.py

@@ -739,3 +921,100 @@ def resolve_value(domain: Domain) -> ot.distributions.BaseDistribution:
        values = {"/".join(path): resolve_value(domain) for path, domain in domain_vars}

        return values
+
+
+class LexiGlobalSearch(OptunaSearch):


why adding this class? None of the method looks necessary. The update of fbest etc. can happen in SearchThread.

sonichi · 2023-06-17T03:20:19Z

flaml/tune/searcher/blendsearch.py

+            if obj_1 > obj_2:
+                return True
+            else:
+                return False
+


sonichi · 2023-06-17T03:21:34Z

flaml/tune/searcher/blendsearch.py

+        n = len(arr)
+        for i in range(n):
+            for j in range(0, n - i - 1):
+                if self._lexico_(arr[j], arr[j + 1]):
+                    arr[j], arr[j + 1] = arr[j + 1], arr[j]


there should be a more standard way for sorting in python

sonichi · 2023-06-17T03:23:58Z

flaml/tune/searcher/blendsearch.py

@@ -748,6 +910,10 @@ def suggest(self, trial_id: str) -> Optional[Dict]:
            self._subspace[trial_id] = space
        else:  # use init config
            if self._candidate_start_points is not None and self._points_to_evaluate:
+                if self.lexico_objectives:
+                    raise NotImplementedError(
+                        "It doesn't support providing points_to_evaluate in lexicographic optimization for now."


revise English.

sonichi · 2023-06-17T03:43:29Z

flaml/tune/searcher/blendsearch.py

+                        self._result[config_signature][k_metric] = {
+                            self._metric: np.inf * -1 if k_mode == "max" else np.inf
+                        }
+                        self._result[config_signature]["time_total_s"] = 1


move out of the loop

sonichi · 2023-06-17T14:02:31Z

flaml/tune/searcher/flow2.py

@@ -312,6 +314,24 @@ def denormalize(self, config):
        """denormalize each dimension in config from [0,1]."""
        return denormalize(config, self._space, self.best_config, self.incumbent, self._random)

+    def _get_lexico_bound(self, metric, mode):


The same function appears in multiple places, which is not a good design.

sonichi · 2023-06-17T14:04:32Z

flaml/tune/searcher/search_thread.py

+                )
+            else:
+                self.speed = 0
+        elif self._search_alg._histories:


don't access private fields

sonichi · 2023-06-17T14:07:27Z

test/tune/test_lexiflow.py

Do you have a test for BlendSearch

resolved. I added a simple test function for now

qingyun-wu · 2023-06-17T18:19:59Z

test/tune/test_lexiflow.py

@@ -103,6 +103,7 @@ def evaluate_function(configuration):

    lexico_objectives = {}
    lexico_objectives["metrics"] = ["error_rate", "flops"]
+    lexico_objectives["algorithm"] = ["CFO"]


Add a test for BlendSearch as well.

qingyun-wu · 2023-06-17T18:26:31Z

flaml/tune/searcher/suggestion.py

+                this functionality.
+
+    Tune automatically converts search spaces to Optuna's format:
+


Why such a significant change on docstr?

qingyun-wu · 2023-06-17T18:27:49Z

flaml/tune/searcher/suggestion.py

        points_to_evaluate: Optional[List[Dict]] = None,
        sampler: Optional["BaseSampler"] = None,
        seed: Optional[int] = None,
        evaluated_rewards: Optional[List] = None,
    ):
        assert ot is not None, "Optuna must be installed! Run `pip install optuna`."
-        super(OptunaSearch, self).__init__(metric=metric, mode=mode, max_concurrent=None, use_early_stopped_trials=None)
+        super(OptunaSearch, self).__init__(metric=metric, mode=mode)


Why removing max_concurrent=None, use_early_stopped_trials=None?

qingyun-wu · 2023-06-17T18:32:02Z

flaml/tune/searcher/suggestion.py

@@ -500,7 +676,7 @@ def _setup_study(self, mode: str):
                for point in self._points_to_evaluate:
                    self._ot_study.enqueue_trial(point)

-    def set_search_properties(self, metric: Optional[str], mode: Optional[str], config: Dict) -> bool:
+    def set_search_properties(self, metric: Optional[str], mode: Optional[str], config: Dict, **spec) -> bool:


how spec is used in this func?

qingyun-wu · 2023-06-17T18:33:58Z

flaml/tune/searcher/suggestion.py

@@ -553,21 +729,8 @@ def suggest(self, trial_id: str) -> Optional[Dict]:
            raise RuntimeError(
                UNDEFINED_METRIC_MODE.format(cls=self.__class__.__name__, metric=self._metric, mode=self._mode)
            )
-
-        if isinstance(self._space, list):


why removing this case?

qingyun-wu · 2023-06-17T18:35:09Z

flaml/tune/searcher/suggestion.py

+            # Optuna doesn't support incremental results
+            # for multi-objective optimization
+            return
+        if trial_id in self._completed_trials:


Why/when this happen?

qingyun-wu · 2023-06-17T18:35:28Z

flaml/tune/searcher/suggestion.py

        metric = result[self.metric]
        step = result[TRAINING_ITERATION]
        ot_trial = self._ot_trials[trial_id]
        ot_trial.report(metric, step)

    def on_trial_complete(self, trial_id: str, result: Optional[Dict] = None, error: bool = False):
+        if trial_id in self._completed_trials:


same comment with the question in on_trial_result

qingyun-wu · 2023-06-17T18:36:41Z

flaml/tune/searcher/suggestion.py

@@ -641,34 +834,23 @@ def add_evaluated_point(
        self._ot_study.add_trial(trial)

    def save(self, checkpoint_path: str):


why this change is needed?

qingyun-wu · 2023-06-17T18:36:49Z

flaml/tune/searcher/suggestion.py

-            self._points_to_evaluate,
-            self._evaluated_rewards,
-        )
+        save_object = self.__dict__.copy()
        with open(checkpoint_path, "wb") as outputFile:
            pickle.dump(save_object, outputFile)

    def restore(self, checkpoint_path: str):


why this change is needed?

qingyun-wu · 2023-06-27T03:12:17Z

flaml/tune/searcher/blendsearch.py

@@ -26,7 +28,7 @@
 from .flow2 import FLOW2
 from ..space import add_cost_to_space, indexof, normalize, define_by_run_func
 from ..result import TIME_TOTAL_S
-
+from ..utils import get_lexico_bound


get_lexico_bound could be included in this file as a staticmethod of BlendSearch

first draft

fb0909d

skzhang1 requested review from sonichi and qingyun-wu April 1, 2023 23:19

skzhang1 self-assigned this Apr 1, 2023

skzhang1 changed the title ~~Support Lexicographic optimization in Blendsearch~~ [Draft] Support Lexicographic optimization in Blendsearch Apr 7, 2023

skzhang1 marked this pull request as draft April 7, 2023 01:25

skzhang1 changed the title ~~[Draft] Support Lexicographic optimization in Blendsearch~~ Support Lexicographic optimization in Blendsearch Apr 7, 2023

skzhang1 removed request for sonichi and qingyun-wu April 7, 2023 01:25

update

982fe03

update

ac258ad

“skzhang1” added 7 commits June 14, 2023 14:40

update

f1de480

update

321ef3e

update

142bf16

update

cfbf7d3

update

3f220cd

update

588de22

update

6158bac

skzhang1 requested review from sonichi, qingyun-wu and feiran-jia June 16, 2023 09:13

sonichi reviewed Jun 17, 2023

View reviewed changes

qingyun-wu reviewed Jun 17, 2023

View reviewed changes

“skzhang1” added 5 commits June 21, 2023 22:06

1. remove redundant get_lexibound 2. use origin optuna searcher

36d616d

set init blendsearch

f070d80

fix lexiglobal sampler error

1ba9b90

follow chi's old suggestions

df9440e

remove lexicoglobalsearch

5969d16

clean up

bb87ea5

qingyun-wu reviewed Jun 27, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Lexicographic optimization in Blendsearch #972

Support Lexicographic optimization in Blendsearch #972

skzhang1 commented Apr 1, 2023 •

edited

sonichi commented Jun 12, 2023

skzhang1 commented Jun 13, 2023

sonichi left a comment

sonichi Jun 16, 2023

skzhang1 Jun 26, 2023

sonichi Jun 17, 2023

skzhang1 Jun 26, 2023 •

edited

sonichi Jun 26, 2023

sonichi Jun 17, 2023

skzhang1 Jun 26, 2023

sonichi Jun 17, 2023

skzhang1 Jun 26, 2023

sonichi Jun 17, 2023

skzhang1 Jun 26, 2023

sonichi Jun 17, 2023

skzhang1 Jun 26, 2023

sonichi Jun 17, 2023

skzhang1 Jun 26, 2023

sonichi Jun 17, 2023

skzhang1 Jun 26, 2023

sonichi Jun 17, 2023

skzhang1 Jun 26, 2023

sonichi Jun 17, 2023

skzhang1 Jun 26, 2023

qingyun-wu Jun 17, 2023

qingyun-wu Jun 17, 2023

qingyun-wu Jun 17, 2023

qingyun-wu Jun 17, 2023

qingyun-wu Jun 17, 2023

qingyun-wu Jun 17, 2023

qingyun-wu Jun 17, 2023

qingyun-wu Jun 17, 2023

qingyun-wu Jun 17, 2023

qingyun-wu Jun 27, 2023

		this functionality.

		Tune automatically converts search spaces to Optuna's format:

		@@ -641,34 +834,23 @@ def add_evaluated_point(
		self._ot_study.add_trial(trial)

		def save(self, checkpoint_path: str):

Support Lexicographic optimization in Blendsearch #972

Are you sure you want to change the base?

Support Lexicographic optimization in Blendsearch #972

Conversation

skzhang1 commented Apr 1, 2023 • edited

Why are these changes needed?

Related issue number

Checks

sonichi commented Jun 12, 2023

skzhang1 commented Jun 13, 2023

sonichi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skzhang1 Jun 26, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skzhang1 commented Apr 1, 2023 •

edited

skzhang1 Jun 26, 2023 •

edited