Unable to catch error encountered when blocked by Yahoo for making too many requests

[Liam-OShea]

Hey Luke,

When making too many requests to Yahoo, Yahoo blocks us for a short period of time based on the app ID we have registered on their developer portal.

I am looking to catch this error so that I can handle it properly and send an informative error message to the front end of the application. However I cannot catch the error using try catch or .then.catch syntax, and when the error occurs the program crashes. The error is occuring in YahooFantasy.mjs and occurs when parsing the result from Yahoo.

The issue seems to be occurring when trying to use JSON.parse to parse the "Request denied" response from Yahoo (possibly line 287 in YahooFantasy.mjs?) but I am not sure why this error is uncatchable.

Here are some examples of how I am trying to handle the error:
yf.players.leagues(sandlot, { start: 0 }, ['stats', 'ownership', 'percent_owned', 'draft_analysis']).then((result) => { console.log(result) }).catch((error) => { console.log(error) // Never reached })

try { const result = yf.players.leagues(sandlot, { start: 0 }, ['stats', 'ownership', 'percent_owned', 'draft_analysis']) } catch (e) { console.log(e) // Never reached }

[whatadewitt]

That's a helpful error! Hah!

Should be a fairly error to catch. I can try to take a look tonight at this.

[whatadewitt]

Had a few minutes before calls this morning.

The error is clearly data = JSON.parse(data); on 287, where data isn't JSON. I'm not sure why that isn't being caught, but again, once I have some REAL time I'm confident this can be solved!

[ptoninato]

I've been trying to get around this error too. Does anyone know what a) the rate limit is? and b) what to do when the limit is reached?

I tried writing logic such that it attempted to refresh the token when it hit the limit, but it still kept giving me the request denied message.

[Liam-OShea]

@ptoninato Here are my notes from a while ago very roughly determining the rate limit. Its very rough because I was making 120 calls at a time and did this twice, with failure happening on the second batch of calls. You could test this more exactly by making individual calls until you hit the limit and recording how many calls you made.

• Determine rate limit
  • I hit the rate limit after trying to retrieve 2812 players twice. We receive 25 players a call, and we batched calls in groups of 8.
  • 3000/200*8 = 120 calls
  • Failure between 120 and 240 calls. (Probably 200)

I did some tests and determined the API is blocked at an app registration level (you are identified and blocked based on the API keys you get from Yahoo after registering the app, rather than specific user's being blocked based on their access or refresh token)

To get around this you could register multiple times to get extra sets of credentials, and switch between the credentials whenever you are blocked. This is against Yahoo ToS however.

[whatadewitt]

Sorry, it's been a tough month-plus for me here and I've been largely avoiding the machine at night.

I did put a request in with my contact at Yahoo! to get a more definitive answer about the actual rates, but it's been radio silent. I will ping him again here now that football season is underway.

I just pushed a potential fix to a branch called "over-limit-fix" -- can you test it out and let me know if that helps? If it does, I will merge it into the main branch this week.

[Liam-OShea]

It sounds ominous when you call it "The Machine"!

I will give it a test later tonight and get back to you. Hope your connect comes back from AWOL soon

[Liam-OShea]

Hey Luke,

I ran a test using over-limit-fix and am receiving the same result as before (no change). Let me know if I can help you test this again in the future. I can also send you my code to make bulk parallel requests to the players endpoint (easy to hit API limit this way) if you feel that would be helpful.

[whatadewitt]

Damn... If you want to send it to me I will take a peek!

[Liam-OShea]

Here you go! This will collect all available players in a league. You may need to run it 2-3 times to hit the request limit.

async function getPlayersFromLeagueParallel(league_key) {
  try {
    const subresources = ['stats', 'ownership', 'percent_owned', 'draft_analysis'];
    const batchSize = 300
    let startIndex = 0
    let playersRemain = true
    let allPlayers = []

    if (batchSize % 25 !== 0) throw new Error("Batch size must be multiple of 25")

    while (playersRemain) {
      let promises = []
      for (i = startIndex; i <= startIndex + batchSize - 25; i += 25) {
        promises.push(yf.players.leagues(league_key, { start: i }, subresources)) // Your module used here
      }

      await Promise.all(promises).then((values, err) => {
        values.forEach(val => {
          if (val[0].players.length === 0) {
            playersRemain = false
          } else {
            allPlayers = allPlayers.concat(val[0].players)
          }
        })
      })
      startIndex += batchSize

    }

    return allPlayers
  } catch (e) {
    console.log(e)
  }
}

[whatadewitt]

Thanks, I'll check this out.

In the meantime, I got a response yesterday!

"As far as rate limits, nothing published about it; we are kind of loose about app-based rate limits and instead enforce some general abuse-oriented rate limits. It's on the order of a couple thousand requests per hour, but don't have it handy now."

So if that's something different than what you're seeing we could be running into a different issue...

[Liam-OShea]

Interesting! Perhaps I am blocked not by making too many requests in general, but by making too many request simultaneously through my batching method.

[coolsoftwaretyler]

Hey @Liam-OShea - what was the time period for your 200 calls estimation? 1 second? 60 seconds? Something else?

[Liam-OShea]

Probably a couple seconds, I would make a request and as soon as I received the result make the next request. There may be different limits for situations when you are making calls over a greater interval of time

…

On Mon., Jul. 18, 2022, 12:21 p.m. Tyler Scott Williams, < ***@***.***> wrote: Hey @Liam-OShea <https://github.com/Liam-OShea> - what was the time period for your 200 calls estimation? 1 second? 60 seconds? Something else? — Reply to this email directly, view it on GitHub <#81 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG5E7BC6E3PVY3LE5SDZX33VUVZALANCNFSM5BEKBIQA> . You are receiving this because you were mentioned.Message ID: ***@***.***>