Screen reader announcement for chat response is too verbose

[bpasero]

Have a response that contains references to code, e.g. :

Start screen reader and click on the response. Notice how the full URLs are being read out aloud which are very long.

Verbatim text:

The [`IPositionList`](command:_github.copilot.openSymbolFromReferences?%5B%7B%22%24mid%22%3A1%2C%22fsPath%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FGame.ts%22%2C%22external%22%3A%22file%3A%2F%2F%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FGame.ts%22%2C%22path%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FGame.ts%22%2C%22scheme%22%3A%22file%22%7D%2C%7B%22line%22%3A14%2C%22character%22%3A4%7D%5D "Game.ts") interface you've defined in TypeScript extends from a base interface [`Base.IList`](command:_github.copilot.openSymbolFromReferences?%5B%7B%22%24mid%22%3A1%2C%22path%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2Flib%2FBase.ts%22%2C%22scheme%22%3A%22file%22%7D%2C%7B%22line%22%3A0%2C%22character%22%3A0%7D%5D "lib/Base.ts"). This means that [`IPositionList`](command:_github.copilot.openSymbolFromReferences?%5B%7B%22%24mid%22%3A1%2C%22fsPath%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FGame.ts%22%2C%22external%22%3A%22file%3A%2F%2F%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FGame.ts%22%2C%22path%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FGame.ts%22%2C%22scheme%22%3A%22file%22%7D%2C%7B%22line%22%3A14%2C%22character%22%3A4%7D%5D "Game.ts") inherits all the properties and methods of [`Base.IList`](command:_github.copilot.openSymbolFromReferences?%5B%7B%22%24mid%22%3A1%2C%22path%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2Flib%2FBase.ts%22%2C%22scheme%22%3A%22file%22%7D%2C%7B%22line%22%3A0%2C%22character%22%3A0%7D%5D "lib/Base.ts") and adds some more specific ones. 

Here are the additional properties and methods in [`IPositionList`](command:_github.copilot.openSymbolFromReferences?%5B%7B%22%24mid%22%3A1%2C%22fsPath%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FGame.ts%22%2C%22external%22%3A%22file%3A%2F%2F%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FGame.ts%22%2C%22path%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FGame.ts%22%2C%22scheme%22%3A%22file%22%7D%2C%7B%22line%22%3A14%2C%22character%22%3A4%7D%5D "Game.ts"):

- `data: Position;` This is a property of type [`Position`](command:_github.copilot.openSymbolFromReferences?%5B%7B%22%24mid%22%3A1%2C%22fsPath%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FPosition.ts%22%2C%22external%22%3A%22file%3A%2F%2F%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FPosition.ts%22%2C%22path%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FPosition.ts%22%2C%22scheme%22%3A%22file%22%7D%2C%7B%22line%22%3A46%2C%22character%22%3A4%7D%5D "Position.ts"). It's not clear from the provided code what [`Position`](command:_github.copilot.openSymbolFromReferences?%5B%7B%22%24mid%22%3A1%2C%22fsPath%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FPosition.ts%22%2C%22external%22%3A%22file%3A%2F%2F%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FPosition.ts%22%2C%22path%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FPosition.ts%22%2C%22scheme%22%3A%22file%22%7D%2C%7B%22line%22%3A46%2C%22character%22%3A4%7D%5D "Position.ts") is, but it's likely a custom type or interface defined elsewhere in your code.
- `push(pos: Position);` This is a method that takes a [`Position`](command:_github.copilot.openSymbolFromReferences?%5B%7B%22%24mid%22%3A1%2C%22fsPath%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FPosition.ts%22%2C%22external%22%3A%22file%3A%2F%2F%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FPosition.ts%22%2C%22path%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FPosition.ts%22%2C%22scheme%22%3A%22file%22%7D%2C%7B%22line%22%3A46%2C%22character%22%3A4%7D%5D "Position.ts") object as an argument. The implementation of this method is not provided in the interface, it's expected to be provided in any class that implements this interface.
- `pop(): Position;` This is a method that doesn't take any arguments and returns a [`Position`](command:_github.copilot.openSymbolFromReferences?%5B%7B%22%24mid%22%3A1%2C%22fsPath%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FPosition.ts%22%2C%22external%22%3A%22file%3A%2F%2F%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FPosition.ts%22%2C%22path%22%3A%22%2FUsers%2Fbpasero%2FDocuments%2Fticino-playground%2FPosition.ts%22%2C%22scheme%22%3A%22file%22%7D%2C%7B%22line%22%3A46%2C%22character%22%3A4%7D%5D "Position.ts") 

I think this stems from the fact that we just take the response as string here:

vscode/src/vs/workbench/contrib/chat/browser/chatAccessibilityService.ts

Line 35 in ffb2581

const responseContent = typeof response === 'string' ? response : response?.response.asString();

[bpasero]

Btw I have a similar challenge for the text-to-speech integration where I am thinking about converting the markdown string to a plain text string using renderStringAsPlaintext:

vscode/src/vs/base/browser/markdownRenderer.ts

Line 489 in ffb2581

export function renderStringAsPlaintext(string: IMarkdownString | string) {

But I did notice that this would remove some text, such as ordered list.

[bpasero]

@alexdima made a good point in our standup today: why are we not leveraging the HTML screen reader support for chat items but use aria.alert?

vscode/src/vs/workbench/contrib/chat/browser/chatAccessibilityService.ts

Line 41 in ffb2581

status(responseContent + errorDetails);

[meganrogge]

The HTML screen reader support requires users to enter different modes to navigate - browse/focus mode. Feedback from screen reader users suggested this is not user friendly.

Users will examine content in detail in the chat accessible view, not here. Providing an overview of the response with aria alert is desired. This is similar to how sighted users see the response streaming in. Screen reader users press escape if they don't wish to hear the content.

[meganrogge]

I agree it's not good that it's reading the full URLs. A similar problem is here #210665 and my solution to that will be to remove those links and provide an alternate way of setting any keybindings which are undefined.

[rzhao271]

Potential verification found: I asked for the location of some code, and after Copilot finished typing out the response, it announced the entire response, including all the link text and full command IDs.

[rzhao271]

LGTM now. I might've forgotten to update something.

[meganrogge]

A screenshot of the text that was announced would help to know if this was fixed or not @rzhao271

[rzhao271]

nvm. The symbols are good but the URLs still expand

[meganrogge]

unclear how this could be happening as I'm using renderStringAsPlaintext

vscode/src/vs/workbench/contrib/chat/browser/chatAccessibilityService.ts

Line 42 in 1221742

status(renderStringAsPlaintext(responseContent) + errorDetails);

[meganrogge]

figured it out

[sandy081]

@meganrogge It is still reading the complete path for me.

[meganrogge]

@rzhao271 and @sandy081 the complete path is expected. what's not expected is these chars 5B%7B%22%24m

[meganrogge]

going to see if I can repro on windows