Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/came #5491

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open

Feature/came #5491

wants to merge 21 commits into from

Conversation

chongqichuizi875
Copy link
Contributor

@chongqichuizi875 chongqichuizi875 commented Mar 22, 2024 •
edited

📌 Checklist before creating the PR

  • I have created an issue for this PR for traceability
  • The title follows the standard format: [doc/gemini/tensor/...]: A concise description
  • I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

e.g. fixed #1234, closed #1234, resolved #1234

📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

💥 Checklist before requesting a review

  • I have linked my PR to an issue (instruction)
  • My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
  • I have performed a self-review of my code
  • I have added thorough tests.
  • I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

  • 🌝 Yes, I do.
  • 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

@chongqichuizi875 chongqichuizi875 requested a review from a team as a code owner March 22, 2024 10:41
@chongqichuizi875
Copy link
Contributor Author

CAME optimizer under TP (row and col distribute) and zero.
Self-test passed.

chongqichuizi875 and others added 14 commits April 8, 2024 16:06
@chongqichuizi875
reform code, Modify some expressions and add comments
291db12
@Edenzzzz
[feat] Add distributed lamb; minor fixes in DeviceMesh (hpcaitech#5476)
67476a9 
* init: add dist lamb; add debiasing for lamb

* dist lamb tester mostly done

* all tests passed

* add comments

* all tests passed. Removed debugging statements

* moved setup_distributed inside plugin. Added dist layout caching

* organize better

---------

Co-authored-by: Edenzzzz <wtan45@wisc.edu>
@chongqichuizi875
minor fix of test came
0727821
@chongqichuizi875
minor fix of test came
e4606df
@chongqichuizi875
minor fix of dist_came and test
58ec531
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
e7707f0 
for more information, see https://pre-commit.ci
@Edenzzzz
[hotfix] Improve tester precision by removing ZeRO on vanilla lamb (h…
447f69e 
…pcaitech#5576)

Co-authored-by: Edenzzzz <wtan45@wisc.edu>
@chongqichuizi875
minor fix of dist_came and test
924984b
@chongqichuizi875
Merge branch 'feature/came' of github.com:chongqichuizi875/ColossalAI…
7febec7 
… into feature/came
@chongqichuizi875
Merge branch 'feature/dist-optim' of https://github.com/hpcaitech/Col…
ded354b 
…ossalAI into feature/came
@chongqichuizi875
rebase dist-optim
bff8c54
@chongqichuizi875
rebase dist-optim
3ea5034
@chongqichuizi875
fix remaining comments
ec44099
@chongqichuizi875
add test dist came using booster api
93c4a88
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@chongqichuizi875 @Edenzzzz