I miss session and limiter from yfinance

[ibart]

Hi there, I am kinda of a noob. I just switch to yahooquery from yfinance but I miss 2 functions:

• a way to handle a cache sessions
• a way to cycle without flooding Yahoo with requests. Right now I added a delay but would be nice having something else...

Plus: I am having trouble with mpl.plot; I am receiving Expect data.index as DatetimeIndex

I don't understand how to solve it, using Panda and the .set_index. Any suggestion? Thanks.

^ plus resolved:

                        hist1yData = hist1yData.reset_index()
                        hist1yData = hist1yData.set_index(['date'])
                        hist1yData.index = pd.DatetimeIndex(hist1yData.index)

[ms82494]

Yahooquery allows you to supply a custom session object, just like yfinance.

yqConn = yq.Ticker(tickers, session=mysessionobject).

@ValueRaider explained it in an old Issue thread. Look at this: #151 (comment) and this: https://github.com/ranaroussi/yfinance#smarter-scraping

[ms82494]

I think your charting problem is due to the dataframe having a MultiIndex. The error message tells you to turn that into a DateTime index. So take the symbol out of the index.

[ibart]

Not really... a brainless copy and paste is not working.

class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
    pass

session = CachedLimiterSession(
    limiter=Limiter(RequestRate(2, Duration.SECOND*5)),  # max 2 requests per 5 seconds
    bucket_class=MemoryQueueBucket,
    backend=SQLiteCache("yq.cache"),
)

yq = Ticker(tickername, session=session)

[ms82494]

You need to import the necessary libraries. And the session needs to be initialized with a Yahoo cookie and you need to supply a crumb.

import yahooquery as yq
import requests as rq
from datetime import datetime, timedelta
from requests_cache import CacheMixin, SQLiteCache
from requests_ratelimiter import LimiterMixin, MemoryQueueBucket
from pyrate_limiter import Duration, RequestRate, Limiter
yqinit = yq.Ticker('SPY', user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36")
class CachedLimiterSession(CacheMixin, LimiterMixin, rq.Session):
    def initialize(self):
        self.headers = yqinit.session.headers
        self.cookies = yqinit.session.cookies

s = CachedLimiterSession(
    limiter=Limiter(RequestRate(2, Duration.SECOND*4)),  # max 2 requests per 5 seconds
    expire_after=timedelta(days=7),
    bucket_class=MemoryQueueBucket,
    backend=SQLiteCache("yfinance.cache"),
)

s.initialize()

tickerlist = ['AMZN','GOOG','MSFT']
yqConn = yq.Ticker(tickerlist, session=s)
yqConn.crumb = yqinit.crumb
yqConn.price

This works for me.

[ibart]

thanks, i understand the idea behind but I miss the crumb part.

{'RACE.MI': 'Invalid Crumb'}

Checking the console while browsing Yahoo I see the crumb token but I don't understand how to integrate on my script.

BTW, yfinance was working without this crumb thing and it was awesome! :p

EDIT >

Ok, I got it but I am getting 404 error... the reason why i switched from yfinance to yahooquery. It's a cul-de-sac!

[ms82494]

The following is my understanding, so no warranties: YQ and YF both create a session for you, and both retrieve a crumb. The Yahoo API requires it. You don't get data out of Yahoo Finance without a crumb. The two third-party APIs take care of this and hide the complexity from you. But if you want to replace yahooquery's plain-vanilla session with a rate-limiting/caching one, then it's up to you to make sure that the Ticker object still presents a valid session cookie and a valid crumb to Yahoo. Cookies are properties of a Session object, which is attached to the Ticker object, and the crumb is a direct property of the Ticker object. The important thing is that the two are linked. A crumb is only valid for a given session, and vice versa.

That all said, what my code tries to do is to create a valid yahooquery Ticker object, yqinit, including cookie and crumb. And then I graft the headers and cookies of that Ticker object's session object onto a juiced CachedLimiterSession using its initialize method. Then I create a second Ticker object, yqConn, which is the one I'd be looking to use for many data retrieval requests. Lastly, I copy the crumb property of the yqinit Ticker object to the yqConn Ticker object.

I am sorry if it doesn't work for you. It (mostly) works for me, but currently everything is unsettled about the Yahoo APIs. I've just slightly changed the code above to pull the last step (copying the crumb over to the new Ticker object) into a separate line. It seems to be more robust.

If you get 404 errors, maybe check whether your code works with a plain-vanilla Ticker object.