Enhancement: Citation support for custom plugins

[bstar]

Contact Details

bob@dinicolas.com

What features would you like to see added?

At the moment it does not look like citations are supported for custom langchain tools/plugins. It appears that (handleText handler) support is limited to askOpenAI and Bing responses.

More details

I'd like to potentially offer a PR for this, but some guidance on where to start would be helpful.

I've confirmed my plugin endpoints are returning the citations in response.data.choices.messages.content.citations and summaries have appropriate citation injections ([doc1], [doc2], etc). It looks like the messageSchema needs changing, but I've not been able to interrogate exactly where the citations are getting dropped (askGPTPlugins.js?). I think once the schema is updated and citation data is persisted it should be relatively straight forward to parse and display in the UI.

Which components are impacted by your request?

Endpoints, Plugins

Pictures

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Hi @bstar thanks for looking into this. the citation injection helper function was designed exclusively for bing since it would consistently return a response object with numbered citations, but not actually format any of its messages with markdown hyperlinks. As far as I know, no other AI API does this, so that is why it's not supported.

Can you give me more context? is your plugin a custom langchain tool? Any logic to format citations are best left within the plugin logic. Adding handleText as an operation for every plugin call would not benefit the endpoint much. Since you have a need for it, and you formatted it to match the function, you could probably just import the helper function to your plugin.

I would need to look at your code to help further

[RobertDiNicolasCoke]

Thanks @danny-avila. I was able to confirm this morning that the data issue I was having was an issue with my plugin, so I'm now able to pass along the citation data and it is properly surfacing in the final response- so my concern that the citation data was getting stripped out was not correct.

Regarding surfacing the citations in the UI, I am open to eventually submitting that as a PR. My main concern is implementing the citation UI components (with cooresponding document viewers) with as little upstream friction as possible.

I can chime in when I have something I think is generic enough to benefit this project. Thanks for your time.

[bstar]

I forgot to mention that this is related to custom Langchain Tools.

[bstar]

I wanted to follow up to this discussion... citations for langchain tools are already supported via this project's markdown implementation (and citations being in the data source) so there's nothing to do here. My specific scenario required a more custom approach, but it's not relevant to the upstream code.

[danny-avila]

thanks for following up!