Use a trie to speed up index construction

[lapp0]

Replaces #507

Fixes #795

Awaiting #904 before I continue working on this

In regex.py Outlines compiles an index of legal tokens for each state of the FSM (state_scan_tokens)

On the main branch we use this naive approach

• for each state S_n in the FSM
  • for each token in vocabulary
    • simulate FSM traversal character by character starting with S_n, if successful, add the token to to S_n

Calling _walk_fsm once for every num_tokens * num_states_per_token is inefficient and the current bottleneck in index construction.

We can improve this by using a Trie (implementation details to come)

[rlouf]

Could you add a high-level description of what the PR does so the PR is self-contained? Is there an issue we could link to this PR? Also there is no need to add "WIP" to the title, this is what "Draft PR" means :)

[lapp0]

I believe I've found a bug in regex.py's reduced_vocabulary()

For the token 188 in the gpt2 tokenizer ('\x00'), token_tuple_np is empty (array([''], dtype='<U2')), however it isn't added to empty_token_ids.

Edit: appears it's being addressed in #904