Download fails when ATL06 data larger than around 500 MB

[arindan]

Hi everyone,

I am Arindan from India. I have been exploring the ALT06 data through icepyx for the past few weeks. Its has been great going so far, but when my download query goes beyond ~500 MB, the download fails with an error message ChunkedEncodingError

ChunkedEncodingError: ('Connection broken: IncompleteRead(6138 bytes read, 4102 more expected)', IncompleteRead(6138 bytes read, 4102 more expected))

Below I have attached a screen shot of my recent query which got failed even the data was <500 MB.

First I thought it would because of my personal machine which cannot download that much data in one go. Later I tried to run the Jupyter book in a HPC environment, but still it shows the same error message.

Any help or suggestion would be super useful to solve my problem.
Thank you.

Best,
Arindan

[JessicaS11]

Hello @arindan! Thanks for reaching out. We've had a few other reports of data downloads being incomplete, hanging, etc. and are working with the data center (NSIDC) to figure them out. To help us, would it be possible for you to share the entire error traceback (i.e. that whole long message that prints out with the error)? This looks like it may be another issue that's not technically happening within icepyx (but clearly still needs to be addressed). Thanks!

[arindan]

Dear Jessica,

Thank you for the comment. Below I have copied the complete error message, if can find something which is not working in my own machine.

Best,
Arindan

`Total number of data order requests is 1 for 16 granules.
Data request 1 of 1 is submitting to NSIDC
order ID: 5000003725469
Initial status of your order request at NSIDC is: processing
Your order status is still processing at NSIDC. Please continue waiting... this may take a few moments.
Your order is: complete
NSIDC returned these messages
['Granule 232441186 contained no data within the spatial and/or temporal '
'subset constraints to be processed']
Beginning download of zipped output...

---

IncompleteRead Traceback (most recent call last)
File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/urllib3/response.py:443, in HTTPResponse._error_catcher(self)
442 try:
--> 443 yield
445 except SocketTimeout:
446 # FIXME: Ideally we'd like to include the url in the ReadTimeoutError but
447 # there is yet no clean way to get at it from this context.

File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/urllib3/response.py:818, in HTTPResponse.read_chunked(self, amt, decode_content)
817 break
--> 818 chunk = self._handle_chunk(amt)
819 decoded = self._decode(
820 chunk, decode_content=decode_content, flush_decoder=False
821 )

File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/urllib3/response.py:762, in HTTPResponse._handle_chunk(self, amt)
761 elif amt < self.chunk_left:
--> 762 value = self._fp._safe_read(amt)
763 self.chunk_left = self.chunk_left - amt

File ~/anaconda3/envs/icesat2/lib/python3.10/http/client.py:632, in HTTPResponse._safe_read(self, amt)
631 if len(data) < amt:
--> 632 raise IncompleteRead(data, amt-len(data))
633 return data

IncompleteRead: IncompleteRead(6138 bytes read, 4102 more expected)

During handling of the above exception, another exception occurred:

ProtocolError Traceback (most recent call last)
File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/requests/models.py:816, in Response.iter_content..generate()
815 try:
--> 816 yield from self.raw.stream(chunk_size, decode_content=True)
817 except ProtocolError as e:

File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/urllib3/response.py:623, in HTTPResponse.stream(self, amt, decode_content)
622 if self.chunked and self.supports_chunked_reads():
--> 623 for line in self.read_chunked(amt, decode_content=decode_content):
624 yield line

File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/urllib3/response.py:803, in HTTPResponse.read_chunked(self, amt, decode_content)
798 raise BodyNotHttplibCompatible(
799 "Body should be http.client.HTTPResponse like. "
800 "It should have have an fp attribute which returns raw chunks."
801 )
--> 803 with self._error_catcher():
804 # Don't bother reading the body of a HEAD request.
805 if self._original_response and is_response_to_head(self._original_response):

File ~/anaconda3/envs/icesat2/lib/python3.10/contextlib.py:153, in _GeneratorContextManager.exit(self, typ, value, traceback)
152 try:
--> 153 self.gen.throw(typ, value, traceback)
154 except StopIteration as exc:
155 # Suppress StopIteration unless it's the same exception that
156 # was passed to throw(). This prevents a StopIteration
157 # raised inside the "with" statement from being suppressed.

File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/urllib3/response.py:460, in HTTPResponse._error_catcher(self)
458 except (HTTPException, SocketError) as e:
459 # This includes IncompleteRead.
--> 460 raise ProtocolError("Connection broken: %r" % e, e)
462 # If no exception is thrown, we should avoid cleaning up
463 # unnecessarily.

ProtocolError: ('Connection broken: IncompleteRead(6138 bytes read, 4102 more expected)', IncompleteRead(6138 bytes read, 4102 more expected))

During handling of the above exception, another exception occurred:

ChunkedEncodingError Traceback (most recent call last)
Cell In [9], line 9
7 region.earthdata_login(earthdata_uid, email)
8 region.order_granules(email=False)
----> 9 region.download_granules(data_home)

File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/icepyx/core/query.py:957, in Query.download_granules(self, path, verbose, subset, restart, **kwargs)
951 if (
952 not hasattr(self._granules, "orderIDs")
953 or len(self._granules.orderIDs) == 0
954 ):
955 self.order_granules(verbose=verbose, subset=subset, **kwargs)
--> 957 self._granules.download(verbose, path, session=self._session, restart=restart)

File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/icepyx/core/granules.py:532, in Granules.download(self, verbose, path, session, restart)
529 print("Beginning download of zipped output...")
531 try:
--> 532 zip_response = session.get(downloadURL)
533 # Raise bad request: Loop will stop for bad response code.
534 zip_response.raise_for_status()

File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/requests/sessions.py:600, in Session.get(self, url, **kwargs)
592 r"""Sends a GET request. Returns :class:Response object.
593
594 :param url: URL for the new :class:Request object.
595 :param **kwargs: Optional arguments that request takes.
596 :rtype: requests.Response
597 """
599 kwargs.setdefault("allow_redirects", True)
--> 600 return self.request("GET", url, **kwargs)

File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/requests/sessions.py:587, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
582 send_kwargs = {
583 "timeout": timeout,
584 "allow_redirects": allow_redirects,
585 }
586 send_kwargs.update(settings)
--> 587 resp = self.send(prep, **send_kwargs)
589 return resp

File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/requests/sessions.py:745, in Session.send(self, request, **kwargs)
742 pass
744 if not stream:
--> 745 r.content
747 return r

File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/requests/models.py:899, in Response.content(self)
897 self._content = None
898 else:
--> 899 self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
901 self._content_consumed = True
902 # don't need to release the connection; that's been handled by urllib3
903 # since we exhausted the data.

File ~/anaconda3/envs/icesat2/lib/python3.10/site-packages/requests/models.py:818, in Response.iter_content..generate()
816 yield from self.raw.stream(chunk_size, decode_content=True)
817 except ProtocolError as e:
--> 818 raise ChunkedEncodingError(e)
819 except DecodeError as e:
820 raise ContentDecodingError(e)

ChunkedEncodingError: ('Connection broken: IncompleteRead(6138 bytes read, 4102 more expected)', IncompleteRead(6138 bytes read, 4102 more expected))`