Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SharePoint extractor broken on private links #9904

Open
10 of 11 tasks
thuvasooriya opened this issue May 10, 2024 · 0 comments
Open
10 of 11 tasks

SharePoint extractor broken on private links #9904

thuvasooriya opened this issue May 10, 2024 · 0 comments
Labels
account-needed Account details are needed to test/fix this can-share-account Someone is willing to provide account details for development site-bug Issue with a specific website triage Untriaged issue

Comments

@thuvasooriya
Copy link

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

  • I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

Region

No response

Provide a description that is worded well enough to be understood

I've tried downloading a lecture video which was view only and is accessible through my university account. but the resulted file was only a few kb and unplayable. I was able to download the file using generic extractor and a slightly modified videomanifest url (it works when &enableCdn=1 param is removed from the request url otherwise yt-dlp returns DRM protected error).

SharePoint extractor seems to work fine when adding -S proto:dash flags ( thanks to @seproDev )

Provide verbose output that clearly demonstrates the problem

  • Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
  • If using API, add 'verbose': True to YoutubeDL params instead
  • Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['https://uniofmora-my.sharepoint.com/personal/kasunh_uom_lk/_layouts/15/stream.aspx?id=/personal/kasunh_uom_lk/Documents/EN2074%20-%2021%20batch/Recording%20-%20Lec%2010.mp4&nav=eyJyZWZlcnJhbEluZm8iOnsicmVmZXJyYWxBcHAiOiJPbmVEcml2ZUZvckJ1c2luZXNzIiwicmVmZXJyYWxBcHBQbGF0Zm9ybSI6IldlYiIsInJlZmVycmFsTW9kZSI6InZpZXciLCJyZWZlcnJhbFZpZXciOiJNeUZpbGVzTGlua0NvcHkifX0&ga=1', '-vU']
[debug] User config "/Users/tony/.config/yt-dlp/config": ['-o', '~/Downloads/vids/%(playlist|)s/%(playlist_index|)s_%(title)s.%(ext)s', '--restrict-filenames', '--user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36', '--cookies-from-browser', 'safari', '--downloader', 'aria2c', '--concurrent-fragment', '10']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2024.05.08.232715 from yt-dlp/yt-dlp-nightly-builds [6b54cccdc] (darwin_exe)
[debug] Python 3.12.1 (CPython arm64 64bit) - macOS-14.4.1-arm64-arm-64bit (OpenSSL 3.0.11 19 Sep 2023)
[debug] exe versions: ffmpeg 6.1.1 (setts), ffprobe 6.1.1
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.31.0, sqlite3-3.43.1, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Trying secondary cookie location
[debug] skipping 4 bytes (unknown page header field): b'\x00\x00\x00\x00'
[Cookies] Loading cookie      0/     6[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown page header field): b'\x00\x00\x00\x00'
[Cookies] Loading cookie      0/     2[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown page header field): b'\x00\x00\x00\x00'
[Cookies] Loading cookie      0/     1[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown page header field): b'\x00\x00\x00\x00'
[Cookies] Loading cookie      0/     2[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown page header field): b'\x00\x00\x00\x00'
[Cookies] Loading cookie      0/     3[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown page header field): b'\x00\x00\x00\x00'
[Cookies] Loading cookie      0/     4[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown page header field): b'\x00\x00\x00\x00'
[Cookies] Loading cookie      0/     3[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 1): b'\x00\x00\x00\x00'
[debug] skipping 4 bytes (unknown record field 2): b'\x00\x00\x00\x00'
[debug] skipping 8 bytes (unknown record field 3): b'\x00\x00\x00\x00\x00\x00\x00\x00'
[debug] skipping 87 bytes (footer): b'\x00\x01\xe3\x8f\x07\x17 \x05\x00\x00\x00Kbplist00\xd1\x01\x02_\x10\x18NSHTTPCookieAcceptPolicy\x10\x02\x08\x0b&\x00\x00\x00\x00\x00\x00\x01\x01\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00('
Extracted 21 cookies from safari
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Loaded 1810 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2024.05.08.232715 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2024.05.08.232715 from yt-dlp/yt-dlp-nightly-builds)
[SharePoint] Extracting URL: https://uniofmora-my.sharepoint.com/personal/kasunh_uom_lk/_layouts/15/stream.aspx?id=/personal/kasunh_uom_lk/Documents/EN2074%20-%2021%20batch/Recording%20-%20Lec%2010.mp4&nav=eyJyZWZlcnJhbEluZm8iOnsicmVmZXJyYWxBcHAiOiJPbmVEcml2ZUZvckJ1c2luZXNzIiwicmVmZXJyYWxBcHBQbGF0Zm9ybSI6IldlYiIsInJlZmVycmFsTW9kZSI6InZpZXciLCJyZWZlcnJhbFZpZXciOiJNeUZpbGVzTGlua0NvcHkifX0&ga=1
[SharePoint] /personal/kasunh_uom_lk/Documents/EN2074 - 21 batch/Recording - Lec 10.mp4: Downloading webpage
[SharePoint] 01DB26FLJ5J6AFN32IMJHK47XZM4IGMIS4: Downloading MPD manifest
[SharePoint] 01DB26FLJ5J6AFN32IMJHK47XZM4IGMIS4: Downloading m3u8 information
[SharePoint] 01DB26FLJ5J6AFN32IMJHK47XZM4IGMIS4: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, size, br, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 01DB26FLJ5J6AFN32IMJHK47XZM4IGMIS4: Downloading 1 format(s): source
[debug] Invoking aria2c downloader on "https://uniofmora-my.sharepoint.com/personal/kasunh_uom_lk/_layouts/15/download.aspx?UniqueId=56804f3d-48ef-4e62-ae7e-f9671066225c&Translate=false&tempauth=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvdW5pb2Ztb3JhLW15LnNoYXJlcG9pbnQuY29tQGFhYzBjNTY0LTZjNWUtNGIwNS04ZGMzLTQwODA4N2Y3N2Y3NiIsImNhY2hla2V5IjoiMGguZnxtZW1iZXJzaGlwfDEwMDMyMDAyMjkyZGRiOWZAbGl2ZS5jb20iLCJjaWQiOiJvU2RnSzQ3UUFEQlFuYlVzSkYvQ2FBPT0iLCJlbmRwb2ludHVybCI6ImFOd095OGRuc3pwVmlncXgwdVF5bjBCcTRlY0ZrV0NQakFhcDloZmlHK2M9IiwiZW5kcG9pbnR1cmxMZW5ndGgiOiIxNDYiLCJleHAiOiIxNzE1MzcwNTEyIiwiaXBhZGRyIjoiMjQwNzpjMDA6NjAwNzphYzVjOjQ0YWQ6MzViYjoyNTMzOjMwMGEiLCJpc2xvb3BiYWNrIjoiVHJ1ZSIsImlzcyI6IjAwMDAwMDAzLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMCIsImlzdXNlciI6InRydWUiLCJuYW1laWQiOiIwIy5mfG1lbWJlcnNoaXB8dGh1dmFyYWdhbnMuMjFAdW9tLmxrIiwibmJmIjoiMTcxNTM2NjkxMiIsIm5paSI6Im1pY3Jvc29mdC5zaGFyZXBvaW50Iiwic2lkIjoiMjMyNDQ0MjUtZjZjMy00MjQ5LWI2MDMtYjNhMjQ4NmFmZDRkIiwic2lnbmluX3N0YXRlIjoiW1wia21zaVwiXSIsInNpdGVpZCI6IlltWXpZVEpoT0RVdFpEQXlPUzAwTm1VekxUaGtOV010T0dZeVl6UXpNV1ZpT1RsbCIsInR0IjoiMCIsInZlciI6Imhhc2hlZHByb29mdG9rZW4ifQ.c4Ssl4gVmLiUF2IdJg_KhAWjGxoynWNmsAHnvT7yi6E"
[download] Destination: /Users/tony/Downloads/vids//_Recording_-_Lec_10.mp4
[download] Writing temporary cookies file to "/var/folders/2j/pvqqyvg55vdb3xz5tmq6gc2h0000gn/T/tmp8c5u5zxc.cookies"
[debug] aria2c command line: aria2c -c --no-conf --console-log-level=warn --summary-interval=0 --download-result=hide --http-accept-gzip=true --file-allocation=none -x16 -j16 -s16 --min-split-size 1M --load-cookies=/var/folders/2j/pvqqyvg55vdb3xz5tmq6gc2h0000gn/T/tmp8c5u5zxc.cookies --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-us,en;q=0.5' --header 'Sec-Fetch-Mode: navigate' --check-certificate=true --remote-time=true --show-console-readout=true --dir /Users/tony/Downloads/vids/ --out ./_Recording_-_Lec_10.mp4.part --auto-file-renaming=false -- 'https://uniofmora-my.sharepoint.com/personal/kasunh_uom_lk/_layouts/15/download.aspx?UniqueId=56804f3d-48ef-4e62-ae7e-f9671066225c&Translate=false&tempauth=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvdW5pb2Ztb3JhLW15LnNoYXJlcG9pbnQuY29tQGFhYzBjNTY0LTZjNWUtNGIwNS04ZGMzLTQwODA4N2Y3N2Y3NiIsImNhY2hla2V5IjoiMGguZnxtZW1iZXJzaGlwfDEwMDMyMDAyMjkyZGRiOWZAbGl2ZS5jb20iLCJjaWQiOiJvU2RnSzQ3UUFEQlFuYlVzSkYvQ2FBPT0iLCJlbmRwb2ludHVybCI6ImFOd095OGRuc3pwVmlncXgwdVF5bjBCcTRlY0ZrV0NQakFhcDloZmlHK2M9IiwiZW5kcG9pbnR1cmxMZW5ndGgiOiIxNDYiLCJleHAiOiIxNzE1MzcwNTEyIiwiaXBhZGRyIjoiMjQwNzpjMDA6NjAwNzphYzVjOjQ0YWQ6MzViYjoyNTMzOjMwMGEiLCJpc2xvb3BiYWNrIjoiVHJ1ZSIsImlzcyI6IjAwMDAwMDAzLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMCIsImlzdXNlciI6InRydWUiLCJuYW1laWQiOiIwIy5mfG1lbWJlcnNoaXB8dGh1dmFyYWdhbnMuMjFAdW9tLmxrIiwibmJmIjoiMTcxNTM2NjkxMiIsIm5paSI6Im1pY3Jvc29mdC5zaGFyZXBvaW50Iiwic2lkIjoiMjMyNDQ0MjUtZjZjMy00MjQ5LWI2MDMtYjNhMjQ4NmFmZDRkIiwic2lnbmluX3N0YXRlIjoiW1wia21zaVwiXSIsInNpdGVpZCI6IlltWXpZVEpoT0RVdFpEQXlPUzAwTm1VekxUaGtOV010T0dZeVl6UXpNV1ZpT1RsbCIsInR0IjoiMCIsInZlciI6Imhhc2hlZHByb29mdG9rZW4ifQ.c4Ssl4gVmLiUF2IdJg_KhAWjGxoynWNmsAHnvT7yi6E'
[download] 100% of  208.83KiB in 00:00:01 at 115.33KiB/s
@thuvasooriya thuvasooriya added site-bug Issue with a specific website triage Untriaged issue labels May 10, 2024
@bashonly bashonly added account-needed Account details are needed to test/fix this can-share-account Someone is willing to provide account details for development labels May 10, 2024
@bashonly bashonly changed the title SharePoint extractor broken on password protected links SharePoint extractor broken on private links May 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
account-needed Account details are needed to test/fix this can-share-account Someone is willing to provide account details for development site-bug Issue with a specific website triage Untriaged issue
Projects
None yet
Development

No branches or pull requests

2 participants