Would you accept pull-requests that greatly change the architecture ?

[edo-vi]

Hi @lukas-blecher,
I'm interested in contributing to this project as I really think our community deserves a good OS latex OCR.
Would you accept a future pull request with changes to the architecture or do you think it'll probably be just a separate fork? Because maybe we could have more gains working on that instead of simply find better hyperparameters. (Obv it's just hypothetical for now).

Thanks and great work!

[lukas-blecher]

Hello Edoardo,
Awesome to hear! I'm open to any improvements. Right now I believe one of the main bottlenecks is data related. But a better architecture might also help a lot. See #162 for an alternative model implementation.

Do you have resources to train and develop such a model?

[edo-vi]

Hi Lukas,
thanks for the link. So in the end you didn't follow through with that different implementation?
Like almost everybody I'm resource bound so I don't think I have the resources for that, but I will look into it. If it can achieve greater results with high probability there are ways to find the resources.
I'm reviewing the literature and probably there at least some tweaks we can make to the architecture, although I agree maybe data can also be improved. For example, I've found that it really struggles with 'vertical' equations like column vectors with many expressions and arrays etc. Regarding the architecture, did you follow some particular paper with custom modifications or did you conceive it from scratch?

Also, against what dataset did you measure its performance? Only the custom dataset or something more standard like im2latex-100k? You referenced it but in the performance section you don't specify the test dataset.

Cheers

[lukas-blecher]

In the end I didn't merge the PR, but for other reasons. I experimented with the ConvNext architecture and found it was able to achieve similar performance while the training is actually more stable. Sadly, as of right now, I'm not allowed to share the results. I hope I can get permission for this at some point.

Regarding the vertical equations, I found that there was a bug in the preprocessing pipeline, which I fixed now, plus more data was helping massively.

I never implemented a dataloader for im2latex-100k. You're right that would be a good test set to compare to other published techniques. The performance was measured on my own test set.

I started the project shortly after the ViT paper was published and wanted to give it a go, so the modifications I implemented were just some ideas I had. I was very interested (probably too much) in optimising the model for different image sizes since the equations come in all shapes and forms. I don't really know how much it influences the performance.
As you can see, I never really treated the project as a research project, so there is ground work left to cover.

[edo-vi]

Thank you for making that clear.
It seems to me the project can be improved upon a great deal with a more systematic approach, both in terms of practical results and in 'scientific' terms, if you like. As I said I'm very interested in working on it and, in the near future, I'll have quite a bit of free time.

The optimization for different sizes sounds quite good and, maybe, if we manage to set up a more standardized benchmark we will be able to understand its influence on the performance. I may look into that, it'll be useful anyway.

After a brief review of the literature I got that math OCR has been studied quite a lot but I still have some ideas that have not been tried and that I intend to test on my fork. I will get back to you (don't know when tho).

One more point: if I'm not mistaken you are a student/researcher, is that right? Again hypothetically, would you be interested in collaborating on a high-level research project based on this work? I have read from other discussions that you do not have written a paper based on this, but are you planning to? Are you allowed? I'm not sure what's the situation here.

Cheers,
Edoardo