Replies: 2 comments
-
🤖 Hello, To add a recency filter to your LlamaIndex code, you can modify the from datetime import timedelta
import pandas as pd
import numpy as np
from llama_index.indices.postprocessor import BaseNodePostprocessor
from llama_index.types import NodeWithScore, QueryBundle
from typing import List, Optional
class FixedRecencyPostprocessor(BaseNodePostprocessor):
"""Recency post-processor."""
service_context: ServiceContext
top_k: int = 1
date_key: str = "date"
recency_months: int = 3 # set the recency filter to 3 months
@classmethod
def class_name(cls) -> str:
return "FixedRecencyPostprocessor"
def postprocess_nodes(
self,
nodes: List[NodeWithScore],
query_bundle: Optional[QueryBundle] = None,
) -> List[NodeWithScore]:
"""Postprocess nodes."""
if query_bundle is None:
raise ValueError("Missing query bundle in extra info.")
# sort nodes by date
node_dates = pd.to_datetime(
[node.node.metadata[self.date_key] for node in nodes]
)
sorted_node_idxs = np.flip(node_dates.argsort())
sorted_nodes = [nodes[idx] for idx in sorted_node_idxs]
# filter nodes based on recency
current_date = pd.Timestamp.now()
recency_filter = timedelta(days=self.recency_months * 30)
filtered_nodes = [node for node in sorted_nodes if current_date - pd.to_datetime(node.node.metadata[self.date_key]) <= recency_filter]
return filtered_nodes[: self.top_k] In this code, we've added a new attribute You can adjust the Please note that this code assumes that the 'date' metadata of nodes is in a format that can be converted to a pandas datetime object. If the 'date' metadata is in a different format, you may need to adjust the code accordingly. I hope this helps! Let me know if you have any other questions. Sources
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot. |
Beta Was this translation helpful? Give feedback.
-
all the imports, you mentioned are not present in the library
you are hallucinating the correct imports were;
|
Beta Was this translation helpful? Give feedback.
-
Question Validation
Question
hi, I need some assistant in creating a recency filter, which picks up nodes based date in metadata
All nodes in the index have date in metadata, "date" is corresponding to date of the document (news article).
The code
does sort the nodes by date order from top-k matches. But, I want a additional filter such that it nodes to query based on the recency of stories within 3 or 6 months of the current date (and exclude older stories even if they are matching 'Growth' in their content), using metadata 'date' of nodes.
How can I do that?
Beta Was this translation helpful? Give feedback.
All reactions