structure-aware chunking of code and markdown

[pchalasani]

The chunker in CodeParser is not structure-aware. We should use something based on treesitter that produces an AST of code structure (even markdown). Especially markdown is a useful case because it can be an intermediate stage in chunking pdf docs (i.e. pdf -> markdown with headers -> structure-aware chunks)

[Mohannadcse]

is there a specific use case?

[pchalasani]

Structure-aware chunking in general is good to have. E.g. in a markdown doc, it's good to avoid having a logically coherent section broken up, as long as chunk size limits and overlap params are respected.

[Mohannadcse]

I can work on this issue if it's not assigned yet