Sustainability POC Add Optimization #3515
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@romilbhardwaj Ready for review. Please let me know if you have any questions. |
There was a problem hiding this comment.
Thanks @dmatch01! Took a quick look, will continue testing/reading.
sky/optimizer.py
Outdated
| @@ -894,13 +1014,18 @@ def sort_key(row, accelerator_spot_list=accelerator_spot_list): | |||
|
|
|||
| rows = sorted(rows, key=sort_key) | |||
| else: | |||
| rows = sorted(rows, key=lambda row: float(row.cost_str)) | |||
| rows = sorted( | |||
There was a problem hiding this comment.
When there are clouds with Unknown footprint, the optimizer selects that cloud. Is this behavior expected? Should we instead pick the lowest across all non-unknown values?
There was a problem hiding this comment.
Got it. I was able to recreate with a local k8s. I'm on it!
sky/clouds/cloud.py
Outdated
| region: Optional[str], | ||
| zone: Optional[str]) -> float: | ||
| """Returns the hourly carbon cost for an instance type.""" | ||
| raise NotImplementedError |
There was a problem hiding this comment.
Instead of NotImplementedError, should we return -1 as the default base implementation? I had some other clouds enabled (cudo, paperspace), and had to disable them otherwise I'd get this error:
Traceback (most recent call last):
File "/Users/romilb/tools/anaconda3/bin/sky", line 8, in <module>
sys.exit(cli())
File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 354, in _record
return f(*args, **kwargs)
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 828, in invoke
return super().invoke(ctx)
File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 375, in _record
return f(*args, **kwargs)
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 1143, in launch
_launch_with_confirm(task,
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 592, in _launch_with_confirm
dag = sky.optimize(dag, objective_target=optimize_cost_target_enum)
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/optimizer.py", line 154, in optimize
unused_best_plan = Optimizer._optimize_dag(
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/optimizer.py", line 1138, in _optimize_dag
Optimizer._estimate_nodes_cost_or_time(local_topo_order,
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/optimizer.py", line 302, in _estimate_nodes_cost_or_time
_fill_in_launchable_resources(
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/optimizer.py", line 1395, in _fill_in_launchable_resources
_make_launchables_for_valid_region_zones(
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/optimizer.py", line 1304, in _make_launchables_for_valid_region_zones
ci_cost = launchable_res_cp.get_carbon_cost(execution_time)
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/resources.py", line 1007, in get_carbon_cost
carbs = self.cloud.instance_type_to_hourly_carbon_cost(
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/clouds/cloud.py", line 247, in instance_type_to_hourly_carbon_cost
raise NotImplementedError
dmatch01
force-pushed
the
sustainability_poc_add_optimization
branch
2 times, most recently
from
b92793f
to
82a31d0
Compare
|
@romilbhardwaj Made fixes, ready for review. |
There was a problem hiding this comment.
Thanks @dmatch01! Left some comments, otherwise looks mostly good to me!
Comment on lines
+158
to
+159
| carbon_footprint_objective=objective_target == | ||
| OptimizeObjectiveTarget.CARBON_FOOTPRINT, |
There was a problem hiding this comment.
What should be the behavior when the task requests a cloud that does not have any carbon information? E.g. this command currently fails:
sky launch --optimize-cost-target=carbon_footprint --cloud lambda
Traceback (most recent call last):
File "/Users/romilb/tools/anaconda3/bin/sky", line 8, in <module>
sys.exit(cli())
File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 354, in _record
return f(*args, **kwargs)
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 828, in invoke
return super().invoke(ctx)
File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 375, in _record
return f(*args, **kwargs)
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 1143, in launch
_launch_with_confirm(task,
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 592, in _launch_with_confirm
dag = sky.optimize(dag, objective_target=optimize_cost_target_enum)
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/optimizer.py", line 155, in optimize
unused_best_plan = Optimizer._optimize_dag(
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/optimizer.py", line 1181, in _optimize_dag
local_best_plan, best_total_objective = Optimizer._optimize_by_dp(
File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/optimizer.py", line 481, in _optimize_by_dp
best_resources = dp_point_backs[node][best_resources]
KeyError: DummyResources
If carbon optimization fails (e.g., no cloud in the search space has carbon information), is it possible to fall back to $ optimization?
There was a problem hiding this comment.
no worries if this is complex to implement, we can do it in future versions. As a stop gap, would be nice to print out a cleaner error/warning.
There was a problem hiding this comment.
Will take a look at a fix to address failure.
with rebase. Update examples/sustainability/README.md Fix to handle carbon footprint optimization without a supported carbon supported cloud in the optimization plan. Signed-off-by: dmatch01 <darroyo@us.ibm.com> Co-authored-by: Romil Bhardwaj <romil.bhardwaj@gmail.com>
dmatch01
force-pushed
the
sustainability_poc_add_optimization
branch
from
c91aba2
to
fbd01f0
Compare
|
@romilbhardwaj Made a fix to handle special use case of carbon footprint optimization with only unsupported clouds. The fix detects this condition and chooses cost price to optimize in place of carbon footprint. The output still respects the user request to optimize based on carbon footprint and produces information and summary related to unknown carbon footprints for each cloud. See example below: |
romilbhardwaj
approved these changes
May 29, 2024
There was a problem hiding this comment.
Thanks @dmatch01! This is good to be merged. Thanks for your contributions!
|
Thanks @romilbhardwaj! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial implementation of sustainability support. Includes cost plus carbon footprint optimization. To enable cost plus carbon footprint optimization launch skypilot with the command line option
--optimize-cost-target=carbon_footprint.Tested (run the relevant ones):
bash format.shpytest tests/test_smoke.pypytest tests/test_smoke.py::test_fill_in_the_nameconda deactivate; bash -i tests/backward_compatibility_tests.sh