Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add file chunk uploading for Azure Blob Storage Integration #1440

Open
bdstout opened this issue Feb 29, 2024 · 1 comment
Open

Add file chunk uploading for Azure Blob Storage Integration #1440

bdstout opened this issue Feb 29, 2024 · 1 comment

Comments

@bdstout
Copy link

bdstout commented Feb 29, 2024

Is your feature request related to a problem? Please describe.
Uploading large files (>50MB) to blob storage fails.

Describe the solution you'd like
Use file chunking in the Azure client.py:

block_list=[]
chunk_size=1024*1024*4

with open(local_html_file_path, "rb") as data:
    while True:
        read_data = data.read(chunk_size)

        if not read_data:
            break # done

        blk_id = str(uuid.uuid4())
        blob_handle.stage_block(block_id=blk_id,data=read_data) 
        block_list.append(BlobBlock(block_id=blk_id))

blob_handle.commit_block_list(block_list)

I have tested the above by updating the client.py locally. The code above needs review and some love (removing the while true would be first on the list). <3

Describe alternatives you've considered
The only alternative I've considered is performing the upload outside of elementary.

Additional context
Nope.

Would you be willing to contribute this feature?
I would, if I had time. I'll leave it in your capable hands.

@haritamar
Copy link
Collaborator

Hi @bdstout ,
Thanks for opening this issue and apologies for the late reply!
This makes a lot of sense, however due to a significant backlog I don't believe the Elementary team will be able to get to it in the near future, so I'm adding the "Open to Contribution" label.

If you are willing to add as a contribution I will love to provide further guidance.

Thanks,
Itamar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants