Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Log Logits #616

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Log Logits #616

wants to merge 12 commits into from

Conversation

lapp0
Copy link
Contributor

@lapp0 lapp0 commented Feb 6, 2024 •
edited

Fixes #614

  • Documents debugging via `enable_logits_logging()``
  • adds logging call to sequence_generator
  • defines OutlinesModel to allow type checking guarantee that models have __call__ method and a tokenizer attribute

Enable via

import outlines.logging
outlines.logging.enable_logits_logging()

Example of use: #612 (comment)

TODO:

  • implement
  • document (Rendered)
  • tests to ensure expensive debug log code isn't run unless explicitly enabled

@lapp0 lapp0 mentioned this pull request Feb 6, 2024
Closed
@lapp0 lapp0 marked this pull request as ready for review February 6, 2024 17:01
@rlouf rlouf mentioned this pull request Feb 7, 2024
Merged
@lapp0
Copy link
Contributor Author

lapp0 commented Feb 9, 2024 •
edited

Beam search example logs (samplers.md). Indicates beam work is working properly in that it's selecting the tokens which maximize sequence weight, and pruning the worst sequences.

The model itself seems to want to continue generating for some reason.

Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France']

Logs:

--------Batch Logit Details--------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Selected: '\n' for batch_item=0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
        Top Raw Tokens: '\n': 0.236, 'The': 0.129, 'It': 0.086, 'This': 0.064, 'That': 0.054, 'I': 0.040, '': 0.030, 'Well': 0.025, EOS: 0.025                                                                                                                                                                                                                                                                                                                                                                                                                            
        Top Guided Tokens: '\n': 0.236, 'The': 0.129, 'It': 0.086, 'This': 0.064, 'That': 0.054, 'I': 0.040, '': 0.030, 'Well': 0.025, EOS: 0.025                                                                                                                                                                                                                                                                                                                                                                                                                         
Generation: ['', 'What', 'is', '', '2', '+', '2', '?']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Selected: 'The' for batch_item=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
        Top Raw Tokens: '\n': 0.236, 'The': 0.129, 'It': 0.086, 'This': 0.064, 'That': 0.054, 'I': 0.040, '': 0.030, 'Well': 0.025, EOS: 0.025                                                                                                                                                                                                                                                                                                                                                                                                                            
        Top Guided Tokens: '\n': 0.236, 'The': 0.129, 'It': 0.086, 'This': 0.064, 'That': 0.054, 'I': 0.040, '': 0.030, 'Well': 0.025, EOS: 0.025                                                                                                                                                                                                                                                                                                                                                                                                                         
Generation: ['', 'What', 'is', '', '2', '+', '2', '?']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Selected: 'It' for batch_item=2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
        Top Raw Tokens: '\n': 0.236, 'The': 0.129, 'It': 0.086, 'This': 0.064, 'That': 0.054, 'I': 0.040, '': 0.030, 'Well': 0.025, EOS: 0.025                                                                                                                                                                                                                                                                                                                                                                                                                            
        Top Guided Tokens: '\n': 0.236, 'The': 0.129, 'It': 0.086, 'This': 0.064, 'That': 0.054, 'I': 0.040, '': 0.030, 'Well': 0.025, EOS: 0.025                                                                                                                                                                                                                                                                                                                                                                                                                         
Generation: ['', 'What', 'is', '', '2', '+', '2', '?']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Selected: 'This' for batch_item=3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
        Top Raw Tokens: '\n': 0.236, 'The': 0.129, 'It': 0.086, 'This': 0.064, 'That': 0.054, 'I': 0.040, '': 0.030, 'Well': 0.025, EOS: 0.025                                                                                                                                                                                                                                                                                                                                                                                                                            
        Top Guided Tokens: '\n': 0.236, 'The': 0.129, 'It': 0.086, 'This': 0.064, 'That': 0.054, 'I': 0.040, '': 0.030, 'Well': 0.025, EOS: 0.025                                                                                                                                                                                                                                                                                                                                                                                                                         
Generation: ['', 'What', 'is', '', '2', '+', '2', '?']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Selected: 'That' for batch_item=4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
        Top Raw Tokens: '\n': 0.236, 'The': 0.129, 'It': 0.086, 'This': 0.064, 'That': 0.054, 'I': 0.040, '': 0.030, 'Well': 0.025, EOS: 0.025                                                                                                                                                                                                                                                                                                                                                                                                                            
        Top Guided Tokens: '\n': 0.236, 'The': 0.129, 'It': 0.086, 'This': 0.064, 'That': 0.054, 'I': 0.040, '': 0.030, 'Well': 0.025, EOS: 0.025                                                                                                                                                                                                                                                                                                                                                                                                                         
--------Batch Logit Details--------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
        Selected: '\n' for batch_item=0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
        Top Raw Tokens: '\n': 0.970, 'The': 0.005, '2': 0.003, 'A': 0.003, 'I': 0.002, 'It': 0.001, 'An': 0.001, 'This': 0.001, EOS: 0.001                                                                                                                                                                                                                                                                                                                                                                                                                                
        Top Guided Tokens: '\n': 0.970, 'The': 0.005, '2': 0.003, 'A': 0.003, 'I': 0.002, 'It': 0.001, 'An': 0.001, 'This': 0.001, EOS: 0.001                                                                                                                                                                                                                                                                                                                                                                                                                             
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'The']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
        Selected: 'answer' for batch_item=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
        Top Raw Tokens: 'answer': 0.874, 'question': 0.028, 'correct': 0.024, 'most': 0.006, 'obvious': 0.005, 'sum': 0.005, 'simple': 0.004, 'common': 0.004, EOS: 0.004                                                                                                                                                                                                                                                                                                                                                                                                 
        Top Guided Tokens: 'answer': 0.874, 'question': 0.028, 'correct': 0.024, 'most': 0.006, 'obvious': 0.005, 'sum': 0.005, 'simple': 0.004, 'common': 0.004, EOS: 0.004                                                                                                                                                                                                                                                                                                                                                                                              
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'It']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
        Selected: 'question' for batch_item=2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
        Top Raw Tokens: "'": 0.323, '_': 0.304, 'is': 0.172, 'seems': 0.062, 'depends': 0.054, 'sounds': 0.022, 'may': 0.020, 'might': 0.014, EOS: 0.014                                                                                                                                                                                                                                                                                                                                                                                                                  
        Top Guided Tokens: "'": 0.323, '_': 0.304, 'is': 0.172, 'seems': 0.062, 'depends': 0.054, 'sounds': 0.022, 'may': 0.020, 'might': 0.014, EOS: 0.014                                                                                                                                                                                                                                                                                                                                                                                                               
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'This']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
        Selected: "'" for batch_item=3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Top Raw Tokens: 'question': 0.447, 'is': 0.400, 'may': 0.045, 'might': 0.026, 'seems': 0.023, 'simple': 0.015, 'seemingly': 0.008, 'answer': 0.003, EOS: 0.003                                                                                                                                                                                                                                                                                                                                                                                                    
        Top Guided Tokens: 'question': 0.447, 'is': 0.400, 'may': 0.045, 'might': 0.026, 'seems': 0.023, 'simple': 0.015, 'seemingly': 0.008, 'answer': 0.003, EOS: 0.003                                                                                                                                                                                                                                                                                                                                                                                                 
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'That']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
        Selected: '_' for batch_item=4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Top Raw Tokens: '_': 0.254, "'": 0.248, 'is': 0.175, 'question': 0.146, 'seems': 0.040, 'may': 0.039, 'might': 0.028, 'depends': 0.013, EOS: 0.013                                                                                                                                                                                                                                                                                                                                                                                                                
        Top Guided Tokens: '_': 0.254, "'": 0.248, 'is': 0.175, 'question': 0.146, 'seems': 0.040, 'may': 0.039, 'might': 0.028, 'depends': 0.013, EOS: 0.013                                                                                                                                                                                                                                                                                                                                                                                                             
--------Batch Logit Details--------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
        Selected: 'is' for batch_item=0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
        Top Raw Tokens: 'This': 0.163, '2': 0.155, 'The': 0.143, 'It': 0.061, 'That': 0.052, 'I': 0.040, '4': 0.026, 'A': 0.025, EOS: 0.025                                                                                                                                                                                                                                                                                                                                                                                                                               
        Top Guided Tokens: 'This': 0.163, '2': 0.155, 'The': 0.143, 'It': 0.061, 'That': 0.052, 'I': 0.040, '4': 0.026, 'A': 0.025, EOS: 0.025                                                                                                                                                                                                                                                                                                                                                                                                                            
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'The', 'answer']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
        Selected: 'This' for batch_item=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
        Top Raw Tokens: 'is': 0.680, 'to': 0.122, ',': 0.040, 'seems': 0.035, 'might': 0.030, 'may': 0.026, 'depends': 0.007, ':': 0.006, EOS: 0.006                                                                                                                                                                                                                                                                                                                                                                                                                      
        Top Guided Tokens: 'is': 0.680, 'to': 0.122, ',': 0.040, 'seems': 0.035, 'might': 0.030, 'may': 0.026, 'depends': 0.007, ':': 0.006, EOS: 0.006                                                                                                                                                                                                                                                                                                                                                                                                                   
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'This', 'question']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
        Selected: '2' for batch_item=2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Top Raw Tokens: 'may': 0.219, 'is': 0.199, 'seems': 0.189, 'might': 0.149, ',': 0.056, 'has': 0.056, 'can': 0.016, 'sounds': 0.016, EOS: 0.016                                                                                                                                                                                                                                                                                                                                                                                                                    
        Top Guided Tokens: 'may': 0.219, 'is': 0.199, 'seems': 0.189, 'might': 0.149, ',': 0.056, 'has': 0.056, 'can': 0.016, 'sounds': 0.016, EOS: 0.016                                                                                                                                                                                                                                                                                                                                                                                                                 
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'It', "'"]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
        Selected: 'The' for batch_item=3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
        Top Raw Tokens: 's': 1.000, 'll': 0.000, 'd': 0.000, 'a': 0.000, 's': 0.000, 'ss': 0.000, 'S': 0.000, 'a': 0.000, EOS: 0.000                                                                                                                                                                                                                                                                                                                                                                                                                                      
        Top Guided Tokens: 's': 1.000, 'll': 0.000, 'd': 0.000, 'a': 0.000, 's': 0.000, 'ss': 0.000, 'S': 0.000, 'a': 0.000, EOS: 0.000                                                                                                                                                                                                                                                                                                                                                                                                                                   
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'It', '_']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
        Selected: 's' for batch_item=4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Top Raw Tokens: 's': 1.000, 'll': 0.000, 'd': 0.000, 'a': 0.000, 's': 0.000, '_': 0.000, 'is': 0.000, 'a': 0.000, EOS: 0.000                                                                                                                                                                                                                                                                                                                                                                                                                                      
        Top Guided Tokens: 's': 1.000, 'll': 0.000, 'd': 0.000, 'a': 0.000, 's': 0.000, '_': 0.000, 'is': 0.000, 'a': 0.000, EOS: 0.000                                                                                                                                                                                                                                                                                                                                                                                                                                   
--------Batch Logit Details--------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'The', 'answer', 'is']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
        Selected: '' for batch_item=0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
        Top Raw Tokens: '': 0.529, 'simple': 0.155, 'four': 0.110, 'easy': 0.024, 'straightforward': 0.018, 'a': 0.017, 'obvious': 0.014, 'obviously': 0.011, EOS: 0.011                                                                                                                                                                                                                                                                                                                                                                                                  
        Top Guided Tokens: '': 0.529, 'simple': 0.155, 'four': 0.110, 'easy': 0.024, 'straightforward': 0.018, 'a': 0.017, 'obvious': 0.014, 'obviously': 0.011, EOS: 0.011                                                                                                                                                                                                                                                                                                                                                                                               
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'This']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
        Selected: '+' for batch_item=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Top Raw Tokens: 'question': 0.547, 'is': 0.358, 'may': 0.028, 'might': 0.016, 'seems': 0.015, 'simple': 0.015, 'seemingly': 0.007, 'answer': 0.001, EOS: 0.001                                                                                                                                                                                                                                                                                                                                                                                                    
        Top Guided Tokens: 'question': 0.547, 'is': 0.358, 'may': 0.028, 'might': 0.016, 'seems': 0.015, 'simple': 0.015, 'seemingly': 0.007, 'answer': 0.001, EOS: 0.001                                                                                                                                                                                                                                                                                                                                                                                                 
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
        Selected: 'answer' for batch_item=2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
        Top Raw Tokens: '+': 0.935, '+': 0.059, 'plus': 0.003, '.': 0.001, 'ap': 0.001, 'and': 0.000, 'is': 0.000, 'solutions': 0.000, EOS: 0.000                                                                                                                                                                                                                                                                                                                                                                                                                         
        Top Guided Tokens: '+': 0.935, '+': 0.059, 'plus': 0.003, '.': 0.001, 'ap': 0.001, 'and': 0.000, 'is': 0.000, 'solutions': 0.000, EOS: 0.000                                                                                                                                                                                                                                                                                                                                                                                                                      
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
        Selected: 'question' for batch_item=3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
        Top Raw Tokens: 'answer': 0.905, 'sum': 0.028, 'correct': 0.018, 'question': 0.012, 'most': 0.007, 'simple': 0.003, 'number': 0.002, 'mathematical': 0.002, EOS: 0.002                                                                                                                                                                                                                                                                                                                                                                                            
        Top Guided Tokens: 'answer': 0.905, 'sum': 0.028, 'correct': 0.018, 'question': 0.012, 'most': 0.007, 'simple': 0.003, 'number': 0.002, 'mathematical': 0.002, EOS: 0.002                                                                                                                                                                                                                                                                                                                                                                                         
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'It', "'", 's']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Selected: 'a' for batch_item=4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Top Raw Tokens: 'a': 0.667, 'an': 0.134, '': 0.052, 'four': 0.025, 'easy': 0.022, 'not': 0.017, 'simple': 0.013, 'the': 0.011, EOS: 0.011                                                                                                                                                                                                                                                                                                                                                                                                                         
        Top Guided Tokens: 'a': 0.667, 'an': 0.134, '': 0.052, 'four': 0.025, 'easy': 0.022, 'not': 0.017, 'simple': 0.013, 'the': 0.011, EOS: 0.011                                                                                                                                                                                                                                                                                                                                                                                                                      
--------Batch Logit Details--------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'The', 'answer', 'is', '']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
        Selected: '4' for batch_item=0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Top Raw Tokens: '4': 1.000, '2': 0.000, '5': 0.000, '1': 0.000, 'four': 0.000, '3': 0.000, 'Four': 0.000, '6': 0.000, EOS: 0.000                                                                                                                                                                                                                                                                                                                                                                                                                                  
        Top Guided Tokens: '4': 1.000, '2': 0.000, '5': 0.000, '1': 0.000, 'four': 0.000, '3': 0.000, 'Four': 0.000, '6': 0.000, EOS: 0.000                                                                                                                                                                                                                                                                                                                                                                                                                               
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
        Selected: '2' for batch_item=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Top Raw Tokens: '2': 1.000, '': 0.000, '3': 0.000, '1': 0.000, 'is': 0.000, 'two': 0.000, 'two': 0.000, 'Two': 0.000, EOS: 0.000                                                                                                                                                                                                                                                                                                                                                                                                                                  
        Top Guided Tokens: '2': 1.000, '': 0.000, '3': 0.000, '1': 0.000, 'is': 0.000, 'two': 0.000, 'two': 0.000, 'Two': 0.000, EOS: 0.000                                                                                                                                                                                                                                                                                                                                                                                                                               
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
        Selected: 'to' for batch_item=2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
        Top Raw Tokens: 'to': 0.613, 'is': 0.336, ',': 0.017, 'might': 0.006, 'may': 0.006, 'seems': 0.005, 'depends': 0.005, ':': 0.004, EOS: 0.004                                                                                                                                                                                                                                                                                                                                                                                                                      
        Top Guided Tokens: 'to': 0.613, 'is': 0.336, ',': 0.017, 'might': 0.006, 'may': 0.006, 'seems': 0.005, 'depends': 0.005, ':': 0.004, EOS: 0.004                                                                                                                                                                                                                                                                                                                                                                                                                   
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'This', 'question']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        Selected: 'simple' for batch_item=3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
        Top Raw Tokens: 'may': 0.309, 'might': 0.242, 'seems': 0.232, 'is': 0.112, ',': 0.035, 'has': 0.019, 'appears': 0.013, 'sounds': 0.007, EOS: 0.007
        Top Guided Tokens: 'may': 0.309, 'might': 0.242, 'seems': 0.232, 'is': 0.112, ',': 0.035, 'has': 0.019, 'appears': 0.013, 'sounds': 0.007, EOS: 0.007
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'It', "'", 's', 'a']
        Selected: 'is' for batch_item=4                                                                                                   
        Top Raw Tokens: 'simple': 0.604, 'question': 0.215, 'basic': 0.037, 'pretty': 0.021, 'very': 0.019, 'fundamental': 0.016, 'common': 0.007, 'number': 0.007, EOS: 0.007
        Top Guided Tokens: 'simple': 0.604, 'question': 0.215, 'basic': 0.037, 'pretty': 0.021, 'very': 0.019, 'fundamental': 0.016, 'common': 0.007, 'number': 0.007, EOS: 0.007
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'The', 'answer', 'is', '', '4']
        Selected: 'is' for batch_item=0                                                                                                   
        Top Raw Tokens: '.': 0.606, ',': 0.314, '!': 0.037, '(': 0.005, 'if': 0.005, 'but': 0.004, '_': 0.003, '\n': 0.003, EOS: 0.003
        Top Guided Tokens: '.': 0.606, ',': 0.314, '!': 0.037, '(': 0.005, 'if': 0.005, 'but': 0.004, '_': 0.003, '\n': 0.003, EOS: 0.003
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2']
        Selected: '.' for batch_item=1                                                                                                    
        Top Raw Tokens: 'is': 0.776, '=': 0.073, '=': 0.068, 'equals': 0.064, 'can': 0.012, ',': 0.001, 'means': 0.001, 'has': 0.001, EOS: 0.001
        Top Guided Tokens: 'is': 0.776, '=': 0.073, '=': 0.068, 'equals': 0.064, 'can': 0.012, ',': 0.001, 'means': 0.001, 'has': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to']
        Selected: ',' for batch_item=2                                                                                                    
        Top Raw Tokens: 'this': 0.538, 'the': 0.194, '': 0.189, 'that': 0.051, '"': 0.016, '_': 0.005, 'what': 0.005, "'": 0.001, EOS: 0.001
        Top Guided Tokens: 'this': 0.538, 'the': 0.194, '': 0.189, 'that': 0.051, '"': 0.016, '_': 0.005, 'what': 0.005, "'": 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'It', "'", 's', 'a', 'simple']
        Selected: 'this' for batch_item=3                                                                                                 
        Top Raw Tokens: 'question': 0.758, 'enough': 0.055, 'mathematical': 0.047, 'ar': 0.038, 'math': 0.036, 'sum': 0.013, 'equation': 0.012, 'addition': 0.010, EOS: 0.010
        Top Guided Tokens: 'question': 0.758, 'enough': 0.055, 'mathematical': 0.047, 'ar': 0.038, 'math': 0.036, 'sum': 0.013, 'equation': 0.012, 'addition': 0.010, EOS: 0.010
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'is']
        Selected: 'question' for batch_item=4                                                                                             
        Top Raw Tokens: '': 0.805, 'simple': 0.116, 'four': 0.017, 'straightforward': 0.016, ':': 0.009, 'easy': 0.005, 'quite': 0.004, 'a': 0.003, EOS: 0.003
        Top Guided Tokens: '': 0.805, 'simple': 0.116, 'four': 0.017, 'straightforward': 0.016, ':': 0.009, 'easy': 0.005, 'quite': 0.004, 'a': 0.003, EOS: 0.003
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is']
        Selected: 'the' for batch_item=0                                                                                                  
        Top Raw Tokens: 'the': 0.580, 'a': 0.237, 'equal': 0.125, 'an': 0.037, '': 0.008, 'four': 0.004, 'one': 0.001, 'equivalent': 0.001, EOS: 0.001
        Top Guided Tokens: 'the': 0.580, 'a': 0.237, 'equal': 0.125, 'an': 0.037, '': 0.008, 'four': 0.004, 'one': 0.001, 'equivalent': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'The', 'answer', 'is', '', '4', '.']
        Selected: 'question' for batch_item=1                                                                                             
        Top Raw Tokens: 'But': 0.201, '\n': 0.111, 'What': 0.098, 'Or': 0.068, 'This': 0.050, 'That': 0.048, 'Wait': 0.037, 'No': 0.032, EOS: 0.032
        Top Guided Tokens: 'But': 0.201, '\n': 0.111, 'What': 0.098, 'Or': 0.068, 'This': 0.050, 'That': 0.048, 'Wait': 0.037, 'No': 0.032, EOS: 0.032
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'The', 'answer', 'is', '', '4', ',']
        Selected: 'a' for batch_item=2                                                                                                    
        Top Raw Tokens: 'right': 0.390, 'but': 0.227, 'or': 0.050, 'isn': 0.040, 'of': 0.033, 'and': 0.026, 'unless': 0.025, 'not': 0.023, EOS: 0.023
        Top Guided Tokens: 'right': 0.390, 'but': 0.227, 'or': 0.050, 'isn': 0.040, 'of': 0.033, 'and': 0.026, 'unless': 0.025, 'not': 0.023, EOS: 0.023
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this']
        Selected: 'right' for batch_item=3                                                                                                
        Top Raw Tokens: 'question': 0.820, 'simple': 0.087, 'seemingly': 0.019, 'common': 0.012, 'classic': 0.012, 'basic': 0.009, 'fundamental': 0.006, 'age': 0.005, EOS: 0.005
        Top Guided Tokens: 'question': 0.820, 'simple': 0.087, 'seemingly': 0.019, 'common': 0.012, 'classic': 0.012, 'basic': 0.009, 'fundamental': 0.006, 'age': 0.005, EOS: 0.005
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'It', "'", 's', 'a', 'simple', 'question']
        Selected: 'But' for batch_item=4                                                                                                  
        Top Raw Tokens: ',': 0.508, 'with': 0.241, 'that': 0.100, '.': 0.050, 'and': 0.032, 'but': 0.021, 'for': 0.008, 'to': 0.008, EOS: 0.008
        Top Guided Tokens: ',': 0.508, 'with': 0.241, 'that': 0.100, '.': 0.050, 'and': 0.032, 'but': 0.021, 'for': 0.008, 'to': 0.008, EOS: 0.008
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the']
        Selected: 'sum' for batch_item=0                                                                                                  
        Top Raw Tokens: 'sum': 0.542, 'mathematical': 0.408, 'ar': 0.010, 'simple': 0.010, 'basic': 0.010, 'addition': 0.006, 'most': 0.003, 'simpl': 0.002, EOS: 0.002
        Top Guided Tokens: 'sum': 0.542, 'mathematical': 0.408, 'ar': 0.010, 'simple': 0.010, 'basic': 0.010, 'addition': 0.006, 'most': 0.003, 'simpl': 0.002, EOS: 0.002
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question']
        Selected: 'is' for batch_item=1                                                                                                   
        Top Raw Tokens: 'is': 0.860, 'may': 0.039, 'seems': 0.034, 'might': 0.024, 'depends': 0.021, ',': 0.013, 'appears': 0.005, 'would': 0.001, EOS: 0.001
        Top Guided Tokens: 'is': 0.860, 'may': 0.039, 'seems': 0.034, 'might': 0.024, 'depends': 0.021, ',': 0.013, 'appears': 0.005, 'would': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'a']
        Selected: 'mathematical' for batch_item=2                                                                                         
        Top Raw Tokens: 'mathematical': 0.422, 'simple': 0.314, 'basic': 0.228, 'fundamental': 0.018, 'common': 0.009, 'mat': 0.003, 'very': 0.001, 'sum': 0.001, EOS: 0.001
        Top Guided Tokens: 'mathematical': 0.422, 'simple': 0.314, 'basic': 0.228, 'fundamental': 0.018, 'common': 0.009, 'mat': 0.003, 'very': 0.001, 'sum': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'The', 'answer', 'is', '', '4', ',', 'right']
        Selected: '?' for batch_item=3                                                                                                    
        Top Raw Tokens: '?': 0.997, '?!': 0.001, ',': 0.000, '.': 0.000, 'or': 0.000, '!': 0.000, '?.': 0.000, '(': 0.000, EOS: 0.000
        Top Guided Tokens: '?': 0.997, '?!': 0.001, ',': 0.000, '.': 0.000, 'or': 0.000, '!': 0.000, '?.': 0.000, '(': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'The', 'answer', 'is', '', '4', '.', 'But']
        Selected: 'mathematical' for batch_item=4                                                                                         
        Top Raw Tokens: 'what': 0.331, 'why': 0.087, 'how': 0.065, 'if': 0.063, ',': 0.048, 'it': 0.036, 'is': 0.031, 'that': 0.031, EOS: 0.031
        Top Guided Tokens: 'what': 0.331, 'why': 0.087, 'how': 0.065, 'if': 0.063, ',': 0.048, 'it': 0.036, 'is': 0.031, 'that': 0.031, EOS: 0.031
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum']
        Selected: 'of' for batch_item=0                                                                                                   
        Top Raw Tokens: 'of': 0.998, 'or': 0.001, 'm': 0.001, 'that': 0.000, ',': 0.000, 'obtained': 0.000, 'when': 0.000, 'the': 0.000, EOS: 0.000
        Top Guided Tokens: 'of': 0.998, 'or': 0.001, 'm': 0.001, 'that': 0.000, ',': 0.000, 'obtained': 0.000, 'when': 0.000, 'the': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is']
        Selected: '' for batch_item=1                                                                                                     
        Top Raw Tokens: '': 0.647, 'simple': 0.114, 'four': 0.045, 'straightforward': 0.040, 'a': 0.033, 'not': 0.018, 'quite': 0.015, 'univers': 0.013, EOS: 0.013
        Top Guided Tokens: '': 0.647, 'simple': 0.114, 'four': 0.045, 'straightforward': 0.040, 'a': 0.033, 'not': 0.018, 'quite': 0.015, 'univers': 0.013, EOS: 0.013
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'mathematical']
        Selected: 'expression' for batch_item=2                                                                                           
        Top Raw Tokens: 'expression': 0.549, 'equation': 0.192, 'sum': 0.134, 'problem': 0.043, 'operation': 0.029, 'representation': 0.016, 'symbol': 0.010, 'addition': 0.009, EOS: 0.009
        Top Guided Tokens: 'expression': 0.549, 'equation': 0.192, 'sum': 0.134, 'problem': 0.043, 'operation': 0.029, 'representation': 0.016, 'symbol': 0.010, 'addition': 0.009, EOS: 0.009
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'The', 'answer', 'is', '', '4', ',', 'right', '?']
        Selected: 'expression' for batch_item=3                                                                                           
        Top Raw Tokens: 'Well': 0.342, 'Wr': 0.095, 'But': 0.074, 'Not': 0.064, 'No': 0.048, 'Actually': 0.047, '\n': 0.042, 'That': 0.038, EOS: 0.038
        Top Guided Tokens: 'Well': 0.342, 'Wr': 0.095, 'But': 0.074, 'Not': 0.064, 'No': 0.048, 'Actually': 0.047, '\n': 0.042, 'That': 0.038, EOS: 0.038
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'a', 'mathematical']
        Selected: 'Well' for batch_item=4                                                                                                 
        Top Raw Tokens: 'expression': 0.698, 'equation': 0.226, 'problem': 0.031, 'operation': 0.027, 'question': 0.008, 'sum': 0.003, 'addition': 0.002, 'statement': 0.001, EOS: 0.001
        Top Guided Tokens: 'expression': 0.698, 'equation': 0.226, 'problem': 0.031, 'operation': 0.027, 'question': 0.008, 'sum': 0.003, 'addition': 0.002, 'statement': 0.001, EOS: 0.001
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of']
        Selected: 'two' for batch_item=0                                                                                                  
        Top Raw Tokens: 'two': 0.609, 'the': 0.330, '': 0.057, 'any': 0.002, 'adding': 0.001, 'Two': 0.000, 'numbers': 0.000, 'a': 0.000, EOS: 0.000
        Top Guided Tokens: 'two': 0.609, 'the': 0.330, '': 0.057, 'any': 0.002, 'adding': 0.001, 'Two': 0.000, 'numbers': 0.000, 'a': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '']
        Selected: '4' for batch_item=1                                                                                                    
        Top Raw Tokens: '4': 1.000, '2': 0.000, '5': 0.000, 'four': 0.000, '1': 0.000, '3': 0.000, 'Four': 0.000, 'four': 0.000, EOS: 0.000
        Top Guided Tokens: '4': 1.000, '2': 0.000, '5': 0.000, 'four': 0.000, '1': 0.000, '3': 0.000, 'Four': 0.000, 'four': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'mathematical', 'expression']
        Selected: 'the' for batch_item=2                                                                                                  
        Top Raw Tokens: 'for': 0.797, 'representing': 0.112, 'that': 0.054, 'of': 0.008, 'used': 0.005, 'which': 0.005, 'meaning': 0.003, 'den': 0.002, EOS: 0.002
        Top Guided Tokens: 'for': 0.797, 'representing': 0.112, 'that': 0.054, 'of': 0.008, 'used': 0.005, 'which': 0.005, 'meaning': 0.003, 'den': 0.002, EOS: 0.002
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'a', 'mathematical', 'expression']
        Selected: 'for' for batch_item=3                                                                                                  
        Top Raw Tokens: 'that': 0.464, 'representing': 0.164, ',': 0.079, 'which': 0.070, '.': 0.043, 'where': 0.034, 'with': 0.031, 'meaning': 0.030, EOS: 0.030
        Top Guided Tokens: 'that': 0.464, 'representing': 0.164, ',': 0.079, 'which': 0.070, '.': 0.043, 'where': 0.034, 'with': 0.031, 'meaning': 0.030, EOS: 0.030
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'The', 'answer', 'is', '', '4', ',', 'right', '?', 'Well']
        Selected: ',' for batch_item=4                                                                                                    
        Top Raw Tokens: ',': 0.937, 'not': 0.020, '_': 0.008, 'actually': 0.004, 'yes': 0.004, '...': 0.004, 'that': 0.003, 'it': 0.003, EOS: 0.003
        Top Guided Tokens: ',': 0.937, 'not': 0.020, '_': 0.008, 'actually': 0.004, 'yes': 0.004, '...': 0.004, 'that': 0.003, 'it': 0.003, EOS: 0.003
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two']
        Selected: '.' for batch_item=0                                                                                                    
        Top Raw Tokens: 'numbers': 0.817, 'and': 0.090, 'positive': 0.029, 'tw': 0.011, 'quantities': 0.010, 'inte': 0.010, '': 0.007, 'ones': 0.004, EOS: 0.004
        Top Guided Tokens: 'numbers': 0.817, 'and': 0.090, 'positive': 0.029, 'tw': 0.011, 'quantities': 0.010, 'inte': 0.010, '': 0.007, 'ones': 0.004, EOS: 0.004
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '', '4']
        Selected: 'numbers' for batch_item=1                                                                                              
        Top Raw Tokens: '.': 0.931, ',': 0.067, '!': 0.001, 'if': 0.000, '-': 0.000, '(': 0.000, ';': 0.000, '_': 0.000, EOS: 0.000
        Top Guided Tokens: '.': 0.931, ',': 0.067, '!': 0.001, 'if': 0.000, '-': 0.000, '(': 0.000, ';': 0.000, '_': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the']
        Selected: 'numbers' for batch_item=2                                                                                              
        Top Raw Tokens: 'numbers': 0.961, 'two': 0.018, 'inte': 0.007, 'number': 0.005, 'first': 0.002, 'dig': 0.002, 'numer': 0.001, 'quantities': 0.001, EOS: 0.001
        Top Guided Tokens: 'numbers': 0.961, 'two': 0.018, 'inte': 0.007, 'number': 0.005, 'first': 0.002, 'dig': 0.002, 'numer': 0.001, 'quantities': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'mathematical', 'expression', 'for']
        Selected: 'the' for batch_item=3                                                                                                  
        Top Raw Tokens: 'the': 0.420, 'adding': 0.403, 'a': 0.087, 'addition': 0.032, 'finding': 0.027, 'representing': 0.014, 'an': 0.006, 'asking': 0.002, EOS: 0.002
        Top Guided Tokens: 'the': 0.420, 'adding': 0.403, 'a': 0.087, 'addition': 0.032, 'finding': 0.027, 'representing': 0.014, 'an': 0.006, 'asking': 0.002, EOS: 0.002
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', 'The', 'answer', 'is', '', '4', ',', 'right', '?', 'Well', ',']
        Selected: 'adding' for batch_item=4                                                                                               
        Top Raw Tokens: 'not': 0.350, 'that': 0.126, 'it': 0.122, 'in': 0.052, 'maybe': 0.048, 'sort': 0.040, 'yes': 0.032, 'if': 0.030, EOS: 0.030
        Top Guided Tokens: 'not': 0.350, 'that': 0.126, 'it': 0.122, 'in': 0.052, 'maybe': 0.048, 'sort': 0.040, 'yes': 0.032, 'if': 0.030, EOS: 0.030
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '', '4', '.']
        Selected: ',' for batch_item=0                                                                                                    
        Top Raw Tokens: 'This': 0.386, 'However': 0.254, '\n': 0.164, 'But': 0.055, 'The': 0.040, 'In': 0.023, '': 0.018, 'While': 0.015, EOS: 0.015
        Top Guided Tokens: 'This': 0.386, 'However': 0.254, '\n': 0.164, 'But': 0.055, 'The': 0.040, 'In': 0.023, '': 0.018, 'While': 0.015, EOS: 0.015
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers']
        Selected: '' for batch_item=1                                                                                                     
        Top Raw Tokens: ',': 0.854, ':': 0.099, '.': 0.025, '': 0.009, '-': 0.003, '_': 0.001, ';': 0.001, 'called': 0.001, EOS: 0.001
        Top Guided Tokens: ',': 0.854, ':': 0.099, '.': 0.025, '': 0.009, '-': 0.003, '_': 0.001, ';': 0.001, 'called': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers']
        Selected: 'This' for batch_item=2                                                                                                 
        Top Raw Tokens: '': 0.920, 'two': 0.079, ',': 0.001, 'Two': 0.000, 'twenty': 0.000, "'": 0.000, '"': 0.000, 'in': 0.000, EOS: 0.000
        Top Guided Tokens: '': 0.920, 'two': 0.079, ',': 0.001, 'Two': 0.000, 'twenty': 0.000, "'": 0.000, '"': 0.000, 'in': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'mathematical', 'expression', 'for', 'the'] 
        Selected: 'However' for batch_item=3                                                                                              
        Top Raw Tokens: 'sum': 0.805, 'addition': 0.103, 'problem': 0.026, 'ar': 0.016, 'operation': 0.012, 'basic': 0.012, 'simple': 0.009, 'concept': 0.002, EOS: 0.002
        Top Guided Tokens: 'sum': 0.805, 'addition': 0.103, 'problem': 0.026, 'ar': 0.016, 'operation': 0.012, 'basic': 0.012, 'simple': 0.009, 'concept': 0.002, EOS: 0.002
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'mathematical', 'expression', 'for', 'adding']
        Selected: 'sum' for batch_item=4                                                                                                  
        Top Raw Tokens: 'the': 0.743, 'two': 0.156, '': 0.069, 'together': 0.031, 'any': 0.000, 'to': 0.000, 'Two': 0.000, 'a': 0.000, EOS: 0.000
        Top Guided Tokens: 'the': 0.743, 'two': 0.156, '': 0.069, 'together': 0.031, 'any': 0.000, 'to': 0.000, 'Two': 0.000, 'a': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',']
        Selected: '' for batch_item=0                                                                                                     
        Top Raw Tokens: '': 0.871, 'each': 0.040, 'which': 0.034, 'two': 0.010, 'where': 0.010, 'specifically': 0.007, "'": 0.005, '"': 0.005, EOS: 0.005
        Top Guided Tokens: '': 0.871, 'each': 0.040, 'which': 0.034, 'two': 0.010, 'where': 0.010, 'specifically': 0.007, "'": 0.005, '"': 0.005, EOS: 0.005
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '']
        Selected: '2' for batch_item=1                                                                                                    
        Top Raw Tokens: '2': 1.000, '1': 0.000, 'two': 0.000, 'two': 0.000, 'Two': 0.000, 'Two': 0.000, '3': 0.000, '__': 0.000, EOS: 0.000
        Top Guided Tokens: '2': 1.000, '1': 0.000, 'two': 0.000, 'two': 0.000, 'Two': 0.000, 'Two': 0.000, '3': 0.000, '__': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '', '4', '.', 'This']
        Selected: 'is' for batch_item=2                                                                                                   
        Top Raw Tokens: 'is': 0.764, 'may': 0.098, 'simple': 0.032, 'question': 0.025, 'answer': 0.022, 'might': 0.012, 'basic': 0.010, 'statement': 0.009, EOS: 0.009
        Top Guided Tokens: 'is': 0.764, 'may': 0.098, 'simple': 0.032, 'question': 0.025, 'answer': 0.022, 'might': 0.012, 'basic': 0.010, 'statement': 0.009, EOS: 0.009
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '', '4', '.', 'However']
        Selected: ',' for batch_item=3                                                                                                    
        Top Raw Tokens: ',': 1.000, 'in': 0.000, 'it': 0.000, 'this': 0.000, 'the': 0.000, 'if': 0.000, 'there': 0.000, 'I': 0.000, EOS: 0.000
        Top Guided Tokens: ',': 1.000, 'in': 0.000, 'it': 0.000, 'this': 0.000, 'the': 0.000, 'if': 0.000, 'there': 0.000, 'I': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'mathematical', 'expression', 'for', 'the', 'sum']
        Selected: 'of' for batch_item=4                                                                                                   
        Top Raw Tokens: 'of': 0.999, 'm': 0.001, 'or': 0.000, 'that': 0.000, ',': 0.000, 'between': 0.000, 'two': 0.000, 'med': 0.000, EOS: 0.000
        Top Guided Tokens: 'of': 0.999, 'm': 0.001, 'or': 0.000, 'that': 0.000, ',': 0.000, 'between': 0.000, 'two': 0.000, 'med': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '']
        Selected: '2' for batch_item=0                                                                                                    
        Top Raw Tokens: '2': 1.000, '1': 0.000, 'two': 0.000, 'each': 0.000, 'which': 0.000, 'two': 0.000, 'Two': 0.000, 'the': 0.000, EOS: 0.000
        Top Guided Tokens: '2': 1.000, '1': 0.000, 'two': 0.000, 'each': 0.000, 'which': 0.000, 'two': 0.000, 'Two': 0.000, 'the': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2']
        Selected: 'and' for batch_item=1                                                                                                  
        Top Raw Tokens: 'and': 1.000, ',': 0.000, '(': 0.000, 'added': 0.000, 'AND': 0.000, '&': 0.000, 'and': 0.000, '': 0.000, EOS: 0.000
        Top Guided Tokens: 'and': 1.000, ',': 0.000, '(': 0.000, 'added': 0.000, 'AND': 0.000, '&': 0.000, 'and': 0.000, '': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '', '4', '.', 'This', 'is']
        Selected: 'a' for batch_item=2                                                                                                    
        Top Raw Tokens: 'a': 0.927, 'because': 0.027, 'the': 0.023, 'not': 0.007, 'an': 0.004, 'one': 0.002, 'based': 0.002, 'true': 0.001, EOS: 0.001
        Top Guided Tokens: 'a': 0.927, 'because': 0.027, 'the': 0.023, 'not': 0.007, 'an': 0.004, 'one': 0.002, 'based': 0.002, 'true': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '', '4', '.', 'However', ',']
        Selected: 'two' for batch_item=3                                                                                                  
        Top Raw Tokens: 'the': 0.245, 'it': 0.180, 'in': 0.165, 'if': 0.138, 'this': 0.127, 'there': 0.043, 'when': 0.038, 'depending': 0.013, EOS: 0.013
        Top Guided Tokens: 'the': 0.245, 'it': 0.180, 'in': 0.165, 'if': 0.138, 'this': 0.127, 'there': 0.043, 'when': 0.038, 'depending': 0.013, EOS: 0.013
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'mathematical', 'expression', 'for', 'the', 'sum', 'of']
        Selected: 'the' for batch_item=4                                                                                                  
        Top Raw Tokens: 'two': 0.566, 'the': 0.424, '': 0.009, 'adding': 0.000, 'Two': 0.000, 'any': 0.000, 'numbers': 0.000, 'two': 0.000, EOS: 0.000
        Top Guided Tokens: 'two': 0.566, 'the': 0.424, '': 0.009, 'adding': 0.000, 'Two': 0.000, 'any': 0.000, 'numbers': 0.000, 'two': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
--------Batch Logit Details--------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2']                                                                                                                                                                                                                                                                                                                                                                                                                               
        Selected: '' for batch_item=0                                                                                                     
        Top Raw Tokens: 'and': 0.645, ',': 0.340, 'added': 0.010, 'being': 0.002, '.': 0.000, 'repeated': 0.000, '(': 0.000, 'twice': 0.000, EOS: 0.000
        Top Guided Tokens: 'and': 0.645, ',': 0.340, 'added': 0.010, 'being': 0.002, '.': 0.000, 'repeated': 0.000, '(': 0.000, 'twice': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and']
        Selected: 'and' for batch_item=1                                                                                                  
        Top Raw Tokens: '': 1.000, 'the': 0.000, 'another': 0.000, ' ': 0.000, 'two': 0.000, '2': 0.000, ',': 0.000, 'and': 0.000, EOS: 0.000
        Top Guided Tokens: '': 1.000, 'the': 0.000, 'another': 0.000, ' ': 0.000, 'two': 0.000, '2': 0.000, ',': 0.000, 'and': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '', '4', '.', 'This', 'is', 'a']
        Selected: ',' for batch_item=2                                                                                                    
        Top Raw Tokens: 'basic': 0.759, 'fundamental': 0.138, 'simple': 0.097, 'well': 0.001, 'common': 0.001, 'straightforward': 0.001, 'fact': 0.001, 'mathematical': 0.001, EOS: 0.001
        Top Guided Tokens: 'basic': 0.759, 'fundamental': 0.138, 'simple': 0.097, 'well': 0.001, 'common': 0.001, 'straightforward': 0.001, 'fact': 0.001, 'mathematical': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'mathematical', 'expression', 'for', 'the', 'sum', 'of', 'two']
        Selected: 'basic' for batch_item=3                                                                                                
        Top Raw Tokens: 'numbers': 0.534, 'and': 0.321, 'tw': 0.046, '': 0.041, 'ones': 0.014, 'quantities': 0.014, 'consecutive': 0.006, 'inte': 0.006, EOS: 0.006
        Top Guided Tokens: 'numbers': 0.534, 'and': 0.321, 'tw': 0.046, '': 0.041, 'ones': 0.014, 'quantities': 0.014, 'consecutive': 0.006, 'inte': 0.006, EOS: 0.006
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'mathematical', 'expression', 'for', 'the', 'sum', 'of', 'the']
        Selected: 'numbers' for batch_item=4                                                                                              
        Top Raw Tokens: 'numbers': 0.969, 'number': 0.012, 'two': 0.009, 'inte': 0.005, 'dig': 0.001, 'numer': 0.001, 'quantities': 0.001, 'quantity': 0.001, EOS: 0.001
        Top Guided Tokens: 'numbers': 0.969, 'number': 0.012, 'two': 0.009, 'inte': 0.005, 'dig': 0.001, 'numer': 0.001, 'quantities': 0.001, 'quantity': 0.001, EOS: 0.001
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '']
        Selected: '2' for batch_item=0                                                                                                    
        Top Raw Tokens: '2': 1.000, '1': 0.000, '3': 0.000, '_': 0.000, '_': 0.000, 'two': 0.000, 'Two': 0.000, 'Two': 0.000, EOS: 0.000
        Top Guided Tokens: '2': 1.000, '1': 0.000, '3': 0.000, '_': 0.000, '_': 0.000, 'two': 0.000, 'Two': 0.000, 'Two': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and']
        Selected: '' for batch_item=1                                                                                                     
        Top Raw Tokens: '': 1.000, 'another': 0.000, 'two': 0.000, 'also': 0.000, 'the': 0.000, 'again': 0.000, 'a': 0.000, 'itself': 0.000, EOS: 0.000
        Top Guided Tokens: '': 1.000, 'another': 0.000, 'two': 0.000, 'also': 0.000, 'the': 0.000, 'again': 0.000, 'a': 0.000, 'itself': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', ',']
        Selected: 'ar' for batch_item=2                                                                                                   
        Top Raw Tokens: 'added': 0.523, 'when': 0.291, 'and': 0.066, 'which': 0.047, 'each': 0.021, 'that': 0.014, 'combined': 0.014, 'if': 0.003, EOS: 0.003
        Top Guided Tokens: 'added': 0.523, 'when': 0.291, 'and': 0.066, 'which': 0.047, 'each': 0.021, 'that': 0.014, 'combined': 0.014, 'if': 0.003, EOS: 0.003
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '', '4', '.', 'This', 'is', 'a', 'basic']
        Selected: 'added' for batch_item=3                                                                                                
        Top Raw Tokens: 'ar': 0.857, 'mathematical': 0.123, 'addition': 0.007, 'math': 0.004, 'and': 0.003, 'fact': 0.002, 'concept': 0.001, 'principle': 0.001, EOS: 0.001
        Top Guided Tokens: 'ar': 0.857, 'mathematical': 0.123, 'addition': 0.007, 'math': 0.004, 'and': 0.003, 'fact': 0.002, 'concept': 0.001, 'principle': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'mathematical', 'expression', 'for', 'the', 'sum', 'of', 'the', 'numbers']
        Selected: '' for batch_item=4                                                                                                     
        Top Raw Tokens: '': 0.869, 'two': 0.128, ',': 0.001, "'": 0.000, '"': 0.000, 'Two': 0.000, 'twenty': 0.000, '_': 0.000, EOS: 0.000
        Top Guided Tokens: '': 0.869, 'two': 0.128, ',': 0.001, "'": 0.000, '"': 0.000, 'Two': 0.000, 'twenty': 0.000, '_': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2']
        Selected: '.' for batch_item=0                                                                                                    
        Top Raw Tokens: '.': 0.934, ',': 0.062, ':': 0.003, 'which': 0.000, ';': 0.000, 'and': 0.000, '(': 0.000, 'respectively': 0.000, EOS: 0.000
        Top Guided Tokens: '.': 0.934, ',': 0.062, ':': 0.003, 'which': 0.000, ';': 0.000, 'and': 0.000, '(': 0.000, 'respectively': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '']
        Selected: '2' for batch_item=1                                                                                                    
        Top Raw Tokens: '2': 1.000, '1': 0.000, '3': 0.000, 'two': 0.000, '_': 0.000, '_': 0.000, 'Two': 0.000, 'zwe': 0.000, EOS: 0.000
        Top Guided Tokens: '2': 1.000, '1': 0.000, '3': 0.000, 'two': 0.000, '_': 0.000, '_': 0.000, 'Two': 0.000, 'zwe': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar']
        Selected: 'ith' for batch_item=2                                                                                                  
        Top Raw Tokens: 'ith': 1.000, 'ITH': 0.000, 'it': 0.000, 'th': 0.000, 'thm': 0.000, 'che': 0.000, 'itu': 0.000, 'math': 0.000, EOS: 0.000
        Top Guided Tokens: 'ith': 1.000, 'ITH': 0.000, 'it': 0.000, 'th': 0.000, 'thm': 0.000, 'che': 0.000, 'itu': 0.000, 'math': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', ',', 'added']
        Selected: 'together' for batch_item=3                                                                                             
        Top Raw Tokens: 'together': 0.998, 'to': 0.001, 'twice': 0.000, 'consecut': 0.000, 'two': 0.000, 'success': 0.000, 'sequ': 0.000, '': 0.000, EOS: 0.000
        Top Guided Tokens: 'together': 0.998, 'to': 0.001, 'twice': 0.000, 'consecut': 0.000, 'two': 0.000, 'success': 0.000, 'sequ': 0.000, '': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'mathematical', 'expression', 'for', 'the', 'sum', 'of', 'the', 'numbers', '']
        Selected: '2' for batch_item=4                                                                                                    
        Top Raw Tokens: '2': 1.000, '1': 0.000, 'two': 0.000, 'two': 0.000, 'Two': 0.000, 'Two': 0.000, '__': 0.000, '___': 0.000, EOS: 0.000
        Top Guided Tokens: '2': 1.000, '1': 0.000, 'two': 0.000, 'two': 0.000, 'Two': 0.000, 'Two': 0.000, '__': 0.000, '___': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.']
        Selected: '.' for batch_item=0                                                                                                    
        Top Raw Tokens: 'The': 0.445, 'In': 0.170, 'It': 0.124, 'So': 0.057, '\n': 0.051, 'Therefore': 0.045, 'This': 0.033, '': 0.016, EOS: 0.016
        Top Guided Tokens: 'The': 0.445, 'In': 0.170, 'It': 0.124, 'So': 0.057, '\n': 0.051, 'Therefore': 0.045, 'This': 0.033, '': 0.016, EOS: 0.016
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2']
        Selected: 'The' for batch_item=1                                                                                                  
        Top Raw Tokens: '.': 0.989, ',': 0.009, ':': 0.001, 'respectively': 0.000, '(': 0.000, ';': 0.000, 'which': 0.000, '.': 0.000, EOS: 0.000
        Top Guided Tokens: '.': 0.989, ',': 0.009, ':': 0.001, 'respectively': 0.000, '(': 0.000, ';': 0.000, 'which': 0.000, '.': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith']
        Selected: 'metic' for batch_item=2                                                                                                
        Top Raw Tokens: 'metic': 0.998, 'met': 0.002, 'matic': 0.000, 'm': 0.000, 'metric': 0.000, 'met': 0.000, 'mic': 0.000, 'ometric': 0.000, EOS: 0.000
        Top Guided Tokens: 'metic': 0.998, 'met': 0.002, 'matic': 0.000, 'm': 0.000, 'metric': 0.000, 'met': 0.000, 'mic': 0.000, 'ometric': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', ',', 'added', 'together']
        Selected: '.' for batch_item=3                                                                                                    
        Top Raw Tokens: '.': 0.891, ',': 0.059, 'twice': 0.017, ':': 0.012, 'to': 0.007, 'which': 0.005, 'two': 0.002, 'resulting': 0.002, EOS: 0.002
        Top Guided Tokens: '.': 0.891, ',': 0.059, 'twice': 0.017, ':': 0.012, 'to': 0.007, 'which': 0.005, 'two': 0.002, 'resulting': 0.002, EOS: 0.002
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'mathematical', 'expression', 'for', 'the', 'sum', 'of', 'the', 'numbers', '', '2']
        Selected: 'In' for batch_item=4                                                                                                   
        Top Raw Tokens: 'and': 1.000, '(': 0.000, ',': 0.000, 'AND': 0.000, 'added': 0.000, 'twice': 0.000, '&': 0.000, 'and': 0.000, EOS: 0.000
        Top Guided Tokens: 'and': 1.000, '(': 0.000, ',': 0.000, 'AND': 0.000, 'added': 0.000, 'twice': 0.000, '&': 0.000, 'and': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.']
        Selected: 'answer' for batch_item=0                                                                                               
        Top Raw Tokens: 'The': 0.373, 'When': 0.198, 'So': 0.108, 'In': 0.105, 'Therefore': 0.065, '\n': 0.035, '': 0.019, 'It': 0.017, EOS: 0.017
        Top Guided Tokens: 'The': 0.373, 'When': 0.198, 'So': 0.108, 'In': 0.105, 'Therefore': 0.065, '\n': 0.035, '': 0.019, 'It': 0.017, EOS: 0.017
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The']
        Selected: 'The' for batch_item=1                                                                                                  
        Top Raw Tokens: 'answer': 0.760, 'result': 0.143, 'mathematical': 0.029, 'solution': 0.014, 'correct': 0.012, 'basic': 0.008, 'numbers': 0.003, 'sum': 0.003, EOS: 0.003
        Top Guided Tokens: 'answer': 0.760, 'result': 0.143, 'mathematical': 0.029, 'solution': 0.014, 'correct': 0.012, 'basic': 0.008, 'numbers': 0.003, 'sum': 0.003, EOS: 0.003
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic']
        Selected: 'When' for batch_item=2                                                                                                 
        Top Raw Tokens: 'problem': 0.319, 'fact': 0.219, 'equation': 0.142, 'operation': 0.075, 'principle': 0.069, 'question': 0.052, 'concept': 0.051, 'calculation': 0.024, EOS: 0.024
        Top Guided Tokens: 'problem': 0.319, 'fact': 0.219, 'equation': 0.142, 'operation': 0.075, 'principle': 0.069, 'question': 0.052, 'concept': 0.051, 'calculation': 0.024, EOS: 0.024
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', ',', 'added', 'together', '.']
        Selected: 'The' for batch_item=3                                                                                                  
        Top Raw Tokens: 'The': 0.509, 'In': 0.180, 'So': 0.086, '\n': 0.069, '': 0.038, 'Therefore': 0.030, 'It': 0.023, 'This': 0.017, EOS: 0.017
        Top Guided Tokens: 'The': 0.509, 'In': 0.180, 'So': 0.086, '\n': 0.069, '': 0.038, 'Therefore': 0.030, 'It': 0.023, 'This': 0.017, EOS: 0.017
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'In']
        Selected: 'problem' for batch_item=4                                                                                              
        Top Raw Tokens: 'mathematical': 0.351, 'mathemat': 0.184, 'basic': 0.128, 'ar': 0.092, 'other': 0.071, 'simple': 0.059, 'simpler': 0.024, 'standard': 0.020, EOS: 0.020
        Top Guided Tokens: 'mathematical': 0.351, 'mathemat': 0.184, 'basic': 0.128, 'ar': 0.092, 'other': 0.071, 'simple': 0.059, 'simpler': 0.024, 'standard': 0.020, EOS: 0.020
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer']
        Selected: 'answer' for batch_item=0                                                                                               
        Top Raw Tokens: 'to': 0.559, 'is': 0.435, ',': 0.005, 'depends': 0.000, 'we': 0.000, 'can': 0.000, '': 0.000, 'you': 0.000, EOS: 0.000
        Top Guided Tokens: 'to': 0.559, 'is': 0.435, ',': 0.005, 'depends': 0.000, 'we': 0.000, 'can': 0.000, '': 0.000, 'you': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The']
        Selected: 'to' for batch_item=1                                                                                                   
        Top Raw Tokens: 'answer': 0.676, 'result': 0.228, 'mathematical': 0.020, 'correct': 0.014, 'solution': 0.010, 'basic': 0.008, 'sum': 0.006, 'standard': 0.005, EOS: 0.005
        Top Guided Tokens: 'answer': 0.676, 'result': 0.228, 'mathematical': 0.020, 'correct': 0.014, 'solution': 0.010, 'basic': 0.008, 'sum': 0.006, 'standard': 0.005, EOS: 0.005
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When']
        Selected: 'is' for batch_item=2                                                                                                   
        Top Raw Tokens: 'you': 0.791, 'we': 0.184, 'added': 0.006, 'adding': 0.006, 'two': 0.004, 'these': 0.002, 'combining': 0.001, 'combined': 0.001, EOS: 0.001
        Top Guided Tokens: 'you': 0.791, 'we': 0.184, 'added': 0.006, 'adding': 0.006, 'two': 0.004, 'these': 0.002, 'combining': 0.001, 'combined': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', ',', 'added', 'together', '.', 'The']
        Selected: 'you' for batch_item=3                                                                                                  
        Top Raw Tokens: 'answer': 0.656, 'result': 0.275, 'mathematical': 0.017, 'solution': 0.013, 'correct': 0.005, 'concept': 0.003, 'basic': 0.003, 'process': 0.002, EOS: 0.002
        Top Guided Tokens: 'answer': 0.656, 'result': 0.275, 'mathematical': 0.017, 'solution': 0.013, 'correct': 0.005, 'concept': 0.003, 'basic': 0.003, 'process': 0.002, EOS: 0.002
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', 'The', 'answer', 'to', 'this', 'question', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic', 'problem']
        Selected: 'result' for batch_item=4                                                                                               
        Top Raw Tokens: 'that': 0.400, 'and': 0.286, ',': 0.243, 'where': 0.028, 'with': 0.024, '.': 0.007, 'in': 0.003, 'which': 0.002, EOS: 0.002
        Top Guided Tokens: 'that': 0.400, 'and': 0.286, ',': 0.243, 'where': 0.028, 'with': 0.024, '.': 0.007, 'in': 0.003, 'which': 0.002, EOS: 0.002
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer']
        Selected: '' for batch_item=0                                                                                                     
        Top Raw Tokens: 'to': 0.657, 'is': 0.334, ',': 0.007, 'depends': 0.001, 'we': 0.000, 'would': 0.000, 'you': 0.000, 'can': 0.000, EOS: 0.000
        Top Guided Tokens: 'to': 0.657, 'is': 0.334, ',': 0.007, 'depends': 0.001, 'we': 0.000, 'would': 0.000, 'you': 0.000, 'can': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'to']
        Selected: 'to' for batch_item=1                                                                                                   
        Top Raw Tokens: 'this': 0.592, 'the': 0.330, '': 0.074, 'that': 0.003, '"': 0.000, 'any': 0.000, 'a': 0.000, 'adding': 0.000, EOS: 0.000
        Top Guided Tokens: 'this': 0.592, 'the': 0.330, '': 0.074, 'that': 0.003, '"': 0.000, 'any': 0.000, 'a': 0.000, 'adding': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is']
        Selected: 'add' for batch_item=2                                                                                                  
        Top Raw Tokens: '': 0.989, 'the': 0.002, 'typically': 0.002, 'usually': 0.001, ':': 0.001, 'commonly': 0.001, 'four': 0.001, 'also': 0.001, EOS: 0.001
        Top Guided Tokens: '': 0.989, 'the': 0.002, 'typically': 0.002, 'usually': 0.001, ':': 0.001, 'commonly': 0.001, 'four': 0.001, 'also': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you']
        Selected: 'this' for batch_item=3                                                                                                 
        Top Raw Tokens: 'add': 0.922, 'perform': 0.057, 'combine': 0.010, 'put': 0.008, 'do': 0.001, 'calculate': 0.000, '"': 0.000, 'bring': 0.000, EOS: 0.000
        Top Guided Tokens: 'add': 0.922, 'perform': 0.057, 'combine': 0.010, 'put': 0.008, 'do': 0.001, 'calculate': 0.000, '"': 0.000, 'bring': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'result']
        Selected: 'is' for batch_item=4                                                                                                   
        Top Raw Tokens: 'of': 0.859, 'is': 0.127, ',': 0.006, 'or': 0.003, 'when': 0.003, 'obtained': 0.000, 'ant': 0.000, 'would': 0.000, EOS: 0.000
        Top Guided Tokens: 'of': 0.859, 'is': 0.127, ',': 0.006, 'or': 0.003, 'when': 0.003, 'obtained': 0.000, 'ant': 0.000, 'would': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '']
        Selected: '4' for batch_item=0                                                                                                    
        Top Raw Tokens: '4': 1.000, '5': 0.000, 'four': 0.000, 'four': 0.000, 'Four': 0.000, '__': 0.000, '1': 0.000, 'fourth': 0.000, EOS: 0.000
        Top Guided Tokens: '4': 1.000, '5': 0.000, 'four': 0.000, 'four': 0.000, 'Four': 0.000, '__': 0.000, '1': 0.000, 'fourth': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'to']
        Selected: 'these' for batch_item=1                                                                                                
        Top Raw Tokens: 'this': 0.475, 'the': 0.413, '': 0.107, 'that': 0.004, '"': 0.000, 'what': 0.000, 'adding': 0.000, 'their': 0.000, EOS: 0.000
        Top Guided Tokens: 'this': 0.475, 'the': 0.413, '': 0.107, 'that': 0.004, '"': 0.000, 'what': 0.000, 'adding': 0.000, 'their': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add']
        Selected: '' for batch_item=2                                                                                                     
        Top Raw Tokens: 'these': 0.716, '': 0.104, 'them': 0.102, 'those': 0.034, 'the': 0.024, 'two': 0.019, 'both': 0.001, 'together': 0.000, EOS: 0.000
        Top Guided Tokens: 'these': 0.716, '': 0.104, 'them': 0.102, 'those': 0.034, 'the': 0.024, 'two': 0.019, 'both': 0.001, 'together': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'to', 'this']
        Selected: 'this' for batch_item=3                                                                                                 
        Top Raw Tokens: 'simple': 0.447, 'mathematical': 0.221, 'basic': 0.167, 'ar': 0.066, 'equation': 0.037, 'common': 0.029, 'fundamental': 0.019, 'math': 0.003, EOS: 0.003
        Top Guided Tokens: 'simple': 0.447, 'mathematical': 0.221, 'basic': 0.167, 'ar': 0.066, 'equation': 0.037, 'common': 0.029, 'fundamental': 0.019, 'math': 0.003, EOS: 0.003
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is']
        Selected: 'the' for batch_item=4                                                                                                  
        Top Raw Tokens: '': 0.972, 'also': 0.004, 'the': 0.004, 'obtained': 0.002, 'typically': 0.002, 'usually': 0.002, ':': 0.002, 'therefore': 0.001, EOS: 0.001
        Top Guided Tokens: '': 0.972, 'also': 0.004, 'the': 0.004, 'obtained': 0.002, 'typically': 0.002, 'usually': 0.002, ':': 0.002, 'therefore': 0.001, EOS: 0.001
answer = generator("What is 2+2?")--------Batch Logit Details--------                                                                     
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4']
        Selected: '.' for batch_item=0                                                                                                    
        Top Raw Tokens: '.': 0.988, ',': 0.006, ':': 0.004, '(': 0.001, 'when': 0.000, 'because': 0.000, ';': 0.000, '!': 0.000, EOS: 0.000
        Top Guided Tokens: '.': 0.988, ',': 0.006, ':': 0.004, '(': 0.001, 'when': 0.000, 'because': 0.000, ';': 0.000, '!': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these']
        Selected: 'two' for batch_item=1                                                                                                  
        Top Raw Tokens: 'two': 0.948, 'numbers': 0.051, 'together': 0.001, '': 0.000, 'quantities': 0.000, 'two': 0.000, 'values': 0.000, 'number': 0.000, EOS: 0.000
        Top Guided Tokens: 'two': 0.948, 'numbers': 0.051, 'together': 0.001, '': 0.000, 'quantities': 0.000, 'two': 0.000, 'values': 0.000, 'number': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '']
        Selected: '4' for batch_item=2                                                                                                    
        Top Raw Tokens: '4': 1.000, 'four': 0.000, '5': 0.000, '1': 0.000, 'four': 0.000, 'Four': 0.000, 'fourth': 0.000, '__': 0.000, EOS: 0.000
        Top Guided Tokens: '4': 1.000, 'four': 0.000, '5': 0.000, '1': 0.000, 'four': 0.000, 'Four': 0.000, 'fourth': 0.000, '__': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'to', 'this']
        Selected: 'mathematical' for batch_item=3                                                                                         
        Top Raw Tokens: 'simple': 0.398, 'mathematical': 0.288, 'basic': 0.115, 'equation': 0.071, 'ar': 0.065, 'common': 0.025, 'fundamental': 0.013, 'question': 0.005, EOS: 0.005
        Top Guided Tokens: 'simple': 0.398, 'mathematical': 0.288, 'basic': 0.115, 'equation': 0.071, 'ar': 0.065, 'common': 0.025, 'fundamental': 0.013, 'question': 0.005, EOS: 0.005
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'to', 'the']
        Selected: 'simple' for batch_item=4                                                                                               
        Top Raw Tokens: 'mathematical': 0.501, 'equation': 0.317, 'expression': 0.032, 'addition': 0.031, 'simple': 0.028, 'question': 0.024, 'ar': 0.022, 'sum': 0.020, EOS: 0.020
        Top Guided Tokens: 'mathematical': 0.501, 'equation': 0.317, 'expression': 0.032, 'addition': 0.031, 'simple': 0.028, 'question': 0.024, 'ar': 0.022, 'sum': 0.020, EOS: 0.020
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.']
        Selected: 'numbers' for batch_item=0                                                                                              
        Top Raw Tokens: 'This': 0.545, '\n': 0.322, 'In': 0.052, 'It': 0.030, 'The': 0.011, '': 0.010, 'Here': 0.006, EOS: 0.006
        Top Guided Tokens: 'This': 0.545, '\n': 0.322, 'In': 0.052, 'It': 0.030, 'The': 0.011, '': 0.010, 'Here': 0.006, EOS: 0.006
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two']
        Selected: 'This' for batch_item=1                                                                                                 
        Top Raw Tokens: 'numbers': 0.995, 'quantities': 0.002, 'equal': 0.001, 'together': 0.000, 'identical': 0.000, '': 0.000, 'values': 0.000, 'dig': 0.000, EOS: 0.000
        Top Guided Tokens: 'numbers': 0.995, 'quantities': 0.002, 'equal': 0.001, 'together': 0.000, 'identical': 0.000, '': 0.000, 'values': 0.000, 'dig': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4']
        Selected: '.' for batch_item=2                                                                                                    
        Top Raw Tokens: '.': 0.975, ',': 0.015, ':': 0.006, 'when': 0.001, '(': 0.001, 'because': 0.001, ';': 0.000, '!': 0.000, EOS: 0.000
        Top Guided Tokens: '.': 0.975, ',': 0.015, ':': 0.006, 'when': 0.001, '(': 0.001, 'because': 0.001, ';': 0.000, '!': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'to', 'the', 'mathematical']
        Selected: '\n' for batch_item=3                                                                                                   
        Top Raw Tokens: 'expression': 0.693, 'equation': 0.274, 'problem': 0.016, 'operation': 0.010, 'question': 0.005, 'calculation': 0.001, 'statement': 0.000, 'addition': 0.000, EOS: 0.000
        Top Guided Tokens: 'expression': 0.693, 'equation': 0.274, 'problem': 0.016, 'operation': 0.010, 'question': 0.005, 'calculation': 0.001, 'statement': 0.000, 'addition': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'to', 'this', 'simple']
        Selected: 'expression' for batch_item=4                                                                                           
        Top Raw Tokens: 'ar': 0.656, 'mathematical': 0.240, 'addition': 0.054, 'equation': 0.026, 'math': 0.016, 'mathemat': 0.003, 'question': 0.002, 'calculation': 0.001, EOS: 0.001
        Top Guided Tokens: 'ar': 0.656, 'mathematical': 0.240, 'addition': 0.054, 'equation': 0.026, 'math': 0.016, 'mathemat': 0.003, 'question': 0.002, 'calculation': 0.001, EOS: 0.001
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers']
        Selected: 'together' for batch_item=0                                                                                             
        Top Raw Tokens: 'together': 0.980, ',': 0.020, 'you': 0.000, 'using': 0.000, 'according': 0.000, 'up': 0.000, 'the': 0.000, 'with': 0.000, EOS: 0.000
        Top Guided Tokens: 'together': 0.980, ',': 0.020, 'you': 0.000, 'using': 0.000, 'according': 0.000, 'up': 0.000, 'the': 0.000, 'with': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This']
        Selected: 'is' for batch_item=1                                                                                                   
        Top Raw Tokens: 'is': 0.652, 'simple': 0.103, 'basic': 0.098, 'question': 0.039, 'fundamental': 0.031, 'equation': 0.025, 'mathematical': 0.023, 'ar': 0.007, EOS: 0.007
        Top Guided Tokens: 'is': 0.652, 'simple': 0.103, 'basic': 0.098, 'question': 0.039, 'fundamental': 0.031, 'equation': 0.025, 'mathematical': 0.023, 'ar': 0.007, EOS: 0.007
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.']
        Selected: '\n' for batch_item=2                                                                                                   
        Top Raw Tokens: 'This': 0.549, '\n': 0.263, 'It': 0.059, 'In': 0.055, 'The': 0.023, '': 0.013, 'Here': 0.006, 'However': 0.005, EOS: 0.005
        Top Guided Tokens: 'This': 0.549, '\n': 0.263, 'It': 0.059, 'In': 0.055, 'The': 0.023, '': 0.013, 'Here': 0.006, 'However': 0.005, EOS: 0.005
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n']
        Selected: 'This' for batch_item=3                                                                                                 
        Top Raw Tokens: '\n': 0.998, '``': 0.001, '2': 0.000, 'This': 0.000, 'It': 0.000, '\\': 0.000, 'In': 0.000, 'Here': 0.000, EOS: 0.000
        Top Guided Tokens: '\n': 0.998, '``': 0.001, '2': 0.000, 'This': 0.000, 'It': 0.000, '\\': 0.000, 'In': 0.000, 'Here': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'to', 'the', 'mathematical', 'expression']
        Selected: '\n' for batch_item=4                                                                                                   
        Top Raw Tokens: '': 0.874, '"': 0.092, ',': 0.013, "'": 0.009, 'is': 0.007, '_': 0.002, '_': 0.001, '`': 0.001, EOS: 0.001
        Top Guided Tokens: '': 0.874, '"': 0.092, ',': 0.013, "'": 0.009, 'is': 0.007, '_': 0.002, '_': 0.001, '`': 0.001, EOS: 0.001
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together']
        Selected: ',' for batch_item=0                                                                                                    
        Top Raw Tokens: ',': 0.981, 'the': 0.011, 'you': 0.008, 'using': 0.000, 'their': 0.000, ':': 0.000, 'they': 0.000, 'according': 0.000, EOS: 0.000
        Top Guided Tokens: ',': 0.981, 'the': 0.011, 'you': 0.008, 'using': 0.000, 'their': 0.000, ':': 0.000, 'they': 0.000, 'according': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This', 'is']
        Selected: 'a' for batch_item=1                                                                                                    
        Top Raw Tokens: 'a': 0.979, 'one': 0.009, 'an': 0.006, 'basic': 0.002, 'the': 0.001, 'known': 0.001, 'considered': 0.000, 'called': 0.000, EOS: 0.000
        Top Guided Tokens: 'a': 0.979, 'one': 0.009, 'an': 0.006, 'basic': 0.002, 'the': 0.001, 'known': 0.001, 'considered': 0.000, 'called': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n']
        Selected: 'What' for batch_item=2                                                                                                 
        Top Raw Tokens: 'What': 0.522, 'Can': 0.111, 'This': 0.072, 'Is': 0.057, 'Why': 0.045, 'How': 0.028, 'In': 0.021, 'Here': 0.021, EOS: 0.021
        Top Guided Tokens: 'What': 0.522, 'Can': 0.111, 'This': 0.072, 'Is': 0.057, 'Why': 0.045, 'How': 0.028, 'In': 0.021, 'Here': 0.021, EOS: 0.021
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This']
        Selected: 'is' for batch_item=3                                                                                                   
        Top Raw Tokens: 'is': 0.607, 'simple': 0.119, 'basic': 0.087, 'question': 0.062, 'equation': 0.039, 'mathematical': 0.030, 'fundamental': 0.022, 'ar': 0.009, EOS: 0.009
        Top Guided Tokens: 'is': 0.607, 'simple': 0.119, 'basic': 0.087, 'question': 0.062, 'equation': 0.039, 'mathematical': 0.030, 'fundamental': 0.022, 'ar': 0.009, EOS: 0.009
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n']
        Selected: '\n' for batch_item=4                                                                                                   
        Top Raw Tokens: '\n': 0.998, '``': 0.001, '2': 0.000, 'This': 0.000, 'It': 0.000, '\\': 0.000, 'The': 0.000, 'In': 0.000, EOS: 0.000
        Top Guided Tokens: '\n': 0.998, '``': 0.001, '2': 0.000, 'This': 0.000, 'It': 0.000, '\\': 0.000, 'The': 0.000, 'In': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',']
        Selected: 'the' for batch_item=0                                                                                                  
        Top Raw Tokens: 'the': 0.774, 'you': 0.211, 'their': 0.013, 'they': 0.001, '': 0.000, 'it': 0.000, 'your': 0.000, 'according': 0.000, EOS: 0.000
        Top Guided Tokens: 'the': 0.774, 'you': 0.211, 'their': 0.013, 'they': 0.001, '': 0.000, 'it': 0.000, 'your': 0.000, 'according': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This', 'is', 'a']
        Selected: 'basic' for batch_item=1                                                                                                
        Top Raw Tokens: 'basic': 0.649, 'fundamental': 0.277, 'simple': 0.071, 'very': 0.002, 'common': 0.001, 'well': 0.000, 'mathematical': 0.000, 'straightforward': 0.000, EOS: 0.000
        Top Guided Tokens: 'basic': 0.649, 'fundamental': 0.277, 'simple': 0.071, 'very': 0.002, 'common': 0.001, 'well': 0.000, 'mathematical': 0.000, 'straightforward': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What']
        Selected: 'is' for batch_item=2                                                                                                   
        Top Raw Tokens: 'is': 0.958, 'are': 0.036, 'does': 0.003, "'": 0.001, 'do': 0.000, 'operations': 0.000, 'operation': 0.000, 'happens': 0.000, EOS: 0.000
        Top Guided Tokens: 'is': 0.958, 'are': 0.036, 'does': 0.003, "'": 0.001, 'do': 0.000, 'operations': 0.000, 'operation': 0.000, 'happens': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This', 'is']
        Selected: 'a' for batch_item=3                                                                                                    
        Top Raw Tokens: 'a': 0.974, 'one': 0.011, 'an': 0.007, 'the': 0.002, 'basic': 0.002, 'known': 0.001, 'considered': 0.000, 'called': 0.000, EOS: 0.000
        Top Guided Tokens: 'a': 0.974, 'one': 0.011, 'an': 0.007, 'the': 0.002, 'basic': 0.002, 'known': 0.001, 'considered': 0.000, 'called': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n']
        Selected: 'you' for batch_item=4                                                                                                  
        Top Raw Tokens: 'What': 0.500, 'Can': 0.101, 'This': 0.076, 'Is': 0.053, 'Why': 0.050, 'How': 0.025, 'However': 0.023, 'In': 0.022, EOS: 0.022
        Top Guided Tokens: 'What': 0.500, 'Can': 0.101, 'This': 0.076, 'Is': 0.053, 'Why': 0.050, 'How': 0.025, 'However': 0.023, 'In': 0.022, EOS: 0.022
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the']
        Selected: 'result' for batch_item=0                                                                                               
        Top Raw Tokens: 'result': 0.642, 'answer': 0.242, 'sum': 0.067, 'total': 0.043, 'solution': 0.003, 'resulting': 0.002, 'correct': 0.001, 'mathematical': 0.000, EOS: 0.000
        Top Guided Tokens: 'result': 0.642, 'answer': 0.242, 'sum': 0.067, 'total': 0.043, 'solution': 0.003, 'resulting': 0.002, 'correct': 0.001, 'mathematical': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This', 'is', 'a', 'basic']
        Selected: 'ar' for batch_item=1                                                                                                   
        Top Raw Tokens: 'ar': 0.901, 'mathematical': 0.078, 'addition': 0.009, 'math': 0.005, 'concept': 0.002, 'and': 0.001, 'example': 0.001, 'principle': 0.001, EOS: 0.001
        Top Guided Tokens: 'ar': 0.901, 'mathematical': 0.078, 'addition': 0.009, 'math': 0.005, 'concept': 0.002, 'and': 0.001, 'example': 0.001, 'principle': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is']
        Selected: 'get' for batch_item=2                                                                                                  
        Top Raw Tokens: 'the': 0.731, '': 0.220, 'a': 0.021, 'sub': 0.011, 'multi': 0.008, 'an': 0.001, 'addition': 0.001, 'zero': 0.001, EOS: 0.001
        Top Guided Tokens: 'the': 0.731, '': 0.220, 'a': 0.021, 'sub': 0.011, 'multi': 0.008, 'an': 0.001, 'addition': 0.001, 'zero': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This', 'is', 'a']
        Selected: 'the' for batch_item=3                                                                                                  
        Top Raw Tokens: 'basic': 0.656, 'fundamental': 0.253, 'simple': 0.086, 'common': 0.002, 'very': 0.001, 'well': 0.001, 'straightforward': 0.000, 'mathematical': 0.000, EOS: 0.000
        Top Guided Tokens: 'basic': 0.656, 'fundamental': 0.253, 'simple': 0.086, 'common': 0.002, 'very': 0.001, 'well': 0.001, 'straightforward': 0.000, 'mathematical': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'you']
        Selected: 'answer' for batch_item=4                                                                                               
        Top Raw Tokens: 'get': 0.987, 'find': 0.005, 'arrive': 0.003, 'will': 0.002, 'obtain': 0.002, 'have': 0.001, "'": 0.000, 'reach': 0.000, EOS: 0.000
        Top Guided Tokens: 'get': 0.987, 'find': 0.005, 'arrive': 0.003, 'will': 0.002, 'obtain': 0.002, 'have': 0.001, "'": 0.000, 'reach': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result']
        Selected: 'is' for batch_item=0                                                                                                   
        Top Raw Tokens: 'is': 0.998, 'will': 0.001, ',': 0.000, 'or': 0.000, 'you': 0.000, '(': 0.000, 'of': 0.000, 'would': 0.000, EOS: 0.000
        Top Guided Tokens: 'is': 0.998, 'will': 0.001, ',': 0.000, 'or': 0.000, 'you': 0.000, '(': 0.000, 'of': 0.000, 'would': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar']
        Selected: 'ith' for batch_item=1                                                                                                  
        Top Raw Tokens: 'ith': 1.000, 'ITH': 0.000, 'it': 0.000, 'th': 0.000, 'thm': 0.000, 'itu': 0.000, 'che': 0.000, 'the': 0.000, EOS: 0.000
        Top Guided Tokens: 'ith': 1.000, 'ITH': 0.000, 'it': 0.000, 'th': 0.000, 'thm': 0.000, 'itu': 0.000, 'che': 0.000, 'the': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'you', 'get']
        Selected: 'is' for batch_item=2                                                                                                   
        Top Raw Tokens: 'the': 0.484, 'a': 0.346, '': 0.169, 'an': 0.000, 'four': 0.000, 'their': 0.000, 'another': 0.000, 'your': 0.000, EOS: 0.000
        Top Guided Tokens: 'the': 0.484, 'a': 0.346, '': 0.169, 'an': 0.000, 'four': 0.000, 'their': 0.000, 'another': 0.000, 'your': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the']
        Selected: 'capital' for batch_item=3                                                                                              
        Top Raw Tokens: 'capital': 0.611, 'square': 0.174, 'difference': 0.149, 'meaning': 0.010, 'value': 0.007, 'definition': 0.006, 'formula': 0.006, 'Py': 0.003, EOS: 0.003
        Top Guided Tokens: 'capital': 0.611, 'square': 0.174, 'difference': 0.149, 'meaning': 0.010, 'value': 0.007, 'definition': 0.006, 'formula': 0.006, 'Py': 0.003, EOS: 0.003
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'answer']
        Selected: 'the' for batch_item=4                                                                                                  
        Top Raw Tokens: 'is': 0.994, 'you': 0.004, 'will': 0.001, ',': 0.000, 'would': 0.000, 'to': 0.000, 'or': 0.000, 'that': 0.000, EOS: 0.000
        Top Guided Tokens: 'is': 0.994, 'you': 0.004, 'will': 0.001, ',': 0.000, 'would': 0.000, 'to': 0.000, 'or': 0.000, 'that': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is']
        Selected: '' for batch_item=0                                                                                                     
        Top Raw Tokens: '': 0.971, 'the': 0.018, 'also': 0.004, 'a': 0.003, 'called': 0.001, 'four': 0.001, 'another': 0.000, 'obtained': 0.000, EOS: 0.000
        Top Guided Tokens: '': 0.971, 'the': 0.018, 'also': 0.004, 'a': 0.003, 'called': 0.001, 'four': 0.001, 'another': 0.000, 'obtained': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith']
        Selected: 'metic' for batch_item=1                                                                                                
        Top Raw Tokens: 'metic': 0.998, 'met': 0.002, 'matic': 0.000, 'm': 0.000, 'metric': 0.000, 'met': 0.000, 'mic': 0.000, 'ometric': 0.000, EOS: 0.000
        Top Guided Tokens: 'metic': 0.998, 'met': 0.002, 'matic': 0.000, 'm': 0.000, 'metric': 0.000, 'met': 0.000, 'mic': 0.000, 'ometric': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'answer', 'is']
        Selected: '' for batch_item=2                                                                                                     
        Top Raw Tokens: '': 0.997, 'the': 0.001, 'also': 0.001, 'four': 0.001, ':': 0.000, 'FO': 0.000, 'obtained': 0.000, 'reached': 0.000, EOS: 0.000
        Top Guided Tokens: '': 0.997, 'the': 0.001, 'also': 0.001, 'four': 0.001, ':': 0.000, 'FO': 0.000, 'obtained': 0.000, 'reached': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital']
        Selected: 'city' for batch_item=3                                                                                                 
        Top Raw Tokens: 'city': 0.952, 'of': 0.048, 'city': 0.000, 'City': 0.000, 'cities': 0.000, 'letter': 0.000, 'City': 0.000, 'ization': 0.000, EOS: 0.000
        Top Guided Tokens: 'city': 0.952, 'of': 0.048, 'city': 0.000, 'City': 0.000, 'cities': 0.000, 'letter': 0.000, 'City': 0.000, 'ization': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'you', 'get', 'the']
        Selected: 'answer' for batch_item=4                                                                                               
        Top Raw Tokens: 'answer': 0.871, 'result': 0.114, 'sum': 0.009, 'solution': 0.001, 'total': 0.001, 'number': 0.001, 'final': 0.001, 'answer': 0.000, EOS: 0.000
        Top Guided Tokens: 'answer': 0.871, 'result': 0.114, 'sum': 0.009, 'solution': 0.001, 'total': 0.001, 'number': 0.001, 'final': 0.001, 'answer': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '']
        Selected: '4' for batch_item=0                                                                                                    
        Top Raw Tokens: '4': 1.000, 'four': 0.000, 'Four': 0.000, 'four': 0.000, '5': 0.000, 'fourth': 0.000, '__': 0.000, 'FO': 0.000, EOS: 0.000
        Top Guided Tokens: '4': 1.000, 'four': 0.000, 'Four': 0.000, 'four': 0.000, '5': 0.000, 'fourth': 0.000, '__': 0.000, 'FO': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic']
        Selected: '4' for batch_item=1                                                                                                    
        Top Raw Tokens: 'problem': 0.252, 'question': 0.185, 'concept': 0.177, 'equation': 0.159, 'operation': 0.070, 'principle': 0.067, 'fact': 0.052, 'formula': 0.011, EOS: 0.011
        Top Guided Tokens: 'problem': 0.252, 'question': 0.185, 'concept': 0.177, 'equation': 0.159, 'operation': 0.070, 'principle': 0.067, 'fact': 0.052, 'formula': 0.011, EOS: 0.011
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'answer', 'is', '']
        Selected: 'of' for batch_item=2                                                                                                   
        Top Raw Tokens: '4': 1.000, 'four': 0.000, '5': 0.000, 'Four': 0.000, 'four': 0.000, '__': 0.000, 'fourth': 0.000, '_': 0.000, EOS: 0.000
        Top Guided Tokens: '4': 1.000, 'four': 0.000, '5': 0.000, 'Four': 0.000, 'four': 0.000, '__': 0.000, 'fourth': 0.000, '_': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city']
        Selected: 'problem' for batch_item=3                                                                                              
        Top Raw Tokens: 'of': 1.000, 'of': 0.000, 'Of': 0.000, 'or': 0.000, 'and': 0.000, 'in': 0.000, 'if': 0.000, 'for': 0.000, EOS: 0.000
        Top Guided Tokens: 'of': 1.000, 'of': 0.000, 'Of': 0.000, 'or': 0.000, 'and': 0.000, 'in': 0.000, 'if': 0.000, 'for': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'you', 'get', 'the', 'answer']
        Selected: 'question' for batch_item=4                                                                                             
        Top Raw Tokens: '': 0.569, ',': 0.280, 'of': 0.135, ':': 0.009, 'which': 0.006, 'that': 0.000, '"': 0.000, '-': 0.000, EOS: 0.000
        Top Guided Tokens: '': 0.569, ',': 0.280, 'of': 0.135, ':': 0.009, 'which': 0.006, 'that': 0.000, '"': 0.000, '-': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4']
        Selected: '.' for batch_item=0                                                                                                    
        Top Raw Tokens: '.': 0.993, ':': 0.004, ',': 0.002, '(': 0.001, 'or': 0.000, '\n': 0.000, ';': 0.000, '!': 0.000, EOS: 0.000
        Top Guided Tokens: '.': 0.993, ':': 0.004, ',': 0.002, '(': 0.001, 'or': 0.000, '\n': 0.000, ';': 0.000, '!': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'answer', 'is', '', '4']
        Selected: '.' for batch_item=1                                                                                                    
        Top Raw Tokens: '.': 0.997, ':': 0.002, ',': 0.001, '(': 0.000, 'or': 0.000, '!': 0.000, '\n': 0.000, 'because': 0.000, EOS: 0.000 
        Top Guided Tokens: '.': 0.997, ':': 0.002, ',': 0.001, '(': 0.000, 'or': 0.000, '!': 0.000, '\n': 0.000, 'because': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of']
        Selected: 'France' for batch_item=2                                                                                               
        Top Raw Tokens: 'France': 0.938, 'Australia': 0.027, 'Canada': 0.010, 'Japan': 0.004, 'New': 0.004, 'Italy': 0.004, 'England': 0.003, 'the': 0.002, EOS: 0.002
        Top Guided Tokens: 'France': 0.938, 'Australia': 0.027, 'Canada': 0.010, 'Japan': 0.004, 'New': 0.004, 'Italy': 0.004, 'England': 0.003, 'the': 0.002, EOS: 0.002
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic', 'problem']
        Selected: 'that' for batch_item=3                                                                                                 
        Top Raw Tokens: 'that': 0.405, 'and': 0.345, ',': 0.169, '.': 0.040, 'often': 0.009, 'in': 0.008, 'commonly': 0.007, 'which': 0.005, EOS: 0.005
        Top Guided Tokens: 'that': 0.405, 'and': 0.345, ',': 0.169, '.': 0.040, 'often': 0.009, 'in': 0.008, 'commonly': 0.007, 'which': 0.005, EOS: 0.005
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic', 'question']
        Selected: 'and' for batch_item=4                                                                                                  
        Top Raw Tokens: 'and': 0.517, 'that': 0.253, ',': 0.183, '.': 0.019, 'with': 0.013, 'which': 0.004, 'often': 0.003, 'in': 0.002, EOS: 0.002
        Top Guided Tokens: 'and': 0.517, 'that': 0.253, ',': 0.183, '.': 0.019, 'with': 0.013, 'which': 0.004, 'often': 0.003, 'in': 0.002, EOS: 0.002
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.']
        Selected: 'This' for batch_item=0                                                                                                 
        Top Raw Tokens: 'This': 0.292, 'In': 0.194, 'So': 0.157, '\n': 0.124, 'Therefore': 0.107, 'The': 0.057, 'It': 0.014, 'Thus': 0.014, EOS: 0.014
        Top Guided Tokens: 'This': 0.292, 'In': 0.194, 'So': 0.157, '\n': 0.124, 'Therefore': 0.107, 'The': 0.057, 'It': 0.014, 'Thus': 0.014, EOS: 0.014
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'answer', 'is', '', '4', '.']
        Selected: '?' for batch_item=1                                                                                                    
        Top Raw Tokens: 'This': 0.336, 'So': 0.182, 'In': 0.131, '\n': 0.118, 'Therefore': 0.114, 'The': 0.051, 'Thus': 0.015, 'It': 0.015, EOS: 0.015
        Top Guided Tokens: 'This': 0.336, 'So': 0.182, 'In': 0.131, '\n': 0.118, 'Therefore': 0.114, 'The': 0.051, 'Thus': 0.015, 'It': 0.015, EOS: 0.015
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France']
        Selected: 'In' for batch_item=2                                                                                                   
        Top Raw Tokens: '?': 1.000, 'and': 0.000, '?': 0.000, '__': 0.000, '??': 0.000, '?"': 0.000, ',': 0.000, '?)': 0.000, EOS: 0.000
        Top Guided Tokens: '?': 1.000, 'and': 0.000, '?': 0.000, '__': 0.000, '??': 0.000, '?"': 0.000, ',': 0.000, '?)': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic', 'problem', 'that']
        Selected: 'So' for batch_item=3                                                                                                   
        Top Raw Tokens: 'most': 0.716, 'is': 0.093, 'can': 0.088, 'has': 0.026, 'everyone': 0.020, 'almost': 0.016, 'involves': 0.011, 'does': 0.004, EOS: 0.004
        Top Guided Tokens: 'most': 0.716, 'is': 0.093, 'can': 0.088, 'has': 0.026, 'everyone': 0.020, 'almost': 0.016, 'involves': 0.011, 'does': 0.004, EOS: 0.004
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic', 'question', 'and']
        Selected: 'This' for batch_item=4                                                                                                 
        Top Raw Tokens: 'the': 0.942, 'is': 0.038, 'it': 0.004, 'concept': 0.002, 'one': 0.002, 'its': 0.001, 'an': 0.001, 'answer': 0.001, EOS: 0.001
        Top Guided Tokens: 'the': 0.942, 'is': 0.038, 'it': 0.004, 'concept': 0.002, 'one': 0.002, 'its': 0.001, 'an': 0.001, 'answer': 0.001, EOS: 0.001
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This']
        Selected: '\n' for batch_item=0                                                                                                   
        Top Raw Tokens: 'is': 0.573, 'simple': 0.133, 'basic': 0.115, 'fundamental': 0.074, 'mathematical': 0.034, 'equation': 0.028, 'question': 0.012, 'relationship': 0.008, EOS: 0.008
        Top Guided Tokens: 'is': 0.573, 'simple': 0.133, 'basic': 0.115, 'fundamental': 0.074, 'mathematical': 0.034, 'equation': 0.028, 'question': 0.012, 'relationship': 0.008, EOS: 0.008
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France', '?']
        Selected: 'mathematical' for batch_item=1                                                                                         
        Top Raw Tokens: '\n': 1.000, 'What': 0.000, 'Paris': 0.000, '': 0.000, 'The': 0.000, 'Is': 0.000, '\\': 0.000, '\r': 0.000, EOS: 0.000
        Top Guided Tokens: '\n': 1.000, 'What': 0.000, 'Paris': 0.000, '': 0.000, 'The': 0.000, 'Is': 0.000, '\\': 0.000, '\r': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In']
        Selected: 'is' for batch_item=2                                                                                                   
        Top Raw Tokens: 'mathematical': 0.885, 'mathemat': 0.035, 'other': 0.030, 'basic': 0.015, 'simple': 0.010, 'simpler': 0.009, 'algebra': 0.004, 'the': 0.003, EOS: 0.003
        Top Guided Tokens: 'mathematical': 0.885, 'mathemat': 0.035, 'other': 0.030, 'basic': 0.015, 'simple': 0.010, 'simpler': 0.009, 'algebra': 0.004, 'the': 0.003, EOS: 0.003
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'So']
        Selected: ',' for batch_item=3                                                                                                    
        Top Raw Tokens: ',': 0.753, 'the': 0.230, '': 0.014, 'in': 0.001, 'to': 0.001, 'you': 0.000, 'simply': 0.000, 'we': 0.000, EOS: 0.000
        Top Guided Tokens: ',': 0.753, 'the': 0.230, '': 0.014, 'in': 0.001, 'to': 0.001, 'you': 0.000, 'simply': 0.000, 'we': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'answer', 'is', '', '4', '.', 'This']
        Selected: 'is' for batch_item=4                                                                                                   
        Top Raw Tokens: 'is': 0.568, 'simple': 0.134, 'basic': 0.124, 'fundamental': 0.072, 'mathematical': 0.030, 'equation': 0.025, 'question': 0.021, 'concept': 0.006, EOS: 0.006
        Top Guided Tokens: 'is': 0.568, 'simple': 0.134, 'basic': 0.124, 'fundamental': 0.072, 'mathematical': 0.030, 'equation': 0.025, 'question': 0.021, 'concept': 0.006, EOS: 0.006
--------Batch Logit Details--------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France', '?', '\n']                                                                                                                                                                                                                                                                                           
        Selected: '\n' for batch_item=0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
        Top Raw Tokens: '\n': 1.000, 'The': 0.000, 'Par': 0.000, 'Paris': 0.000, 'An': 0.000, 'PAR': 0.000, 'I': 0.000, 'A': 0.000, EOS: 0.000
        Top Guided Tokens: '\n': 1.000, 'The': 0.000, 'Par': 0.000, 'Paris': 0.000, 'An': 0.000, 'PAR': 0.000, 'I': 0.000, 'A': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical']                                                                                                                                                                                                                                                                             
        Selected: 'a' for batch_item=1                                                                                                    
        Top Raw Tokens: 'terms': 0.587, 'notation': 0.347, 'termin': 0.018, 'language': 0.015, 'expressions': 0.014, 'sh': 0.007, 'equations': 0.005, 'expression': 0.004, EOS: 0.004
        Top Guided Tokens: 'terms': 0.587, 'notation': 0.347, 'termin': 0.018, 'language': 0.015, 'expressions': 0.014, 'sh': 0.007, 'equations': 0.005, 'expression': 0.004, EOS: 0.004
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This', 'is']
        Selected: 'terms' for batch_item=2                                                                                                
        Top Raw Tokens: 'a': 0.987, 'known': 0.003, 'the': 0.002, 'an': 0.002, 'one': 0.002, 'called': 0.001, 'true': 0.000, 'basic': 0.000, EOS: 0.000
        Top Guided Tokens: 'a': 0.987, 'known': 0.003, 'the': 0.002, 'an': 0.002, 'one': 0.002, 'called': 0.001, 'true': 0.000, 'basic': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'So', ',']
        Selected: 'a' for batch_item=3                                                                                                    
        Top Raw Tokens: 'the': 0.617, '': 0.374, 'to': 0.007, 'in': 0.001, 'you': 0.001, 'we': 0.000, 'when': 0.000, 'if': 0.000, EOS: 0.000
        Top Guided Tokens: 'the': 0.617, '': 0.374, 'to': 0.007, 'in': 0.001, 'you': 0.001, 'we': 0.000, 'when': 0.000, 'if': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'answer', 'is', '', '4', '.', 'This', 'is']
        Selected: 'the' for batch_item=4                                                                                                  
        Top Raw Tokens: 'a': 0.989, 'one': 0.003, 'the': 0.002, 'known': 0.002, 'an': 0.002, 'called': 0.000, 'basic': 0.000, 'considered': 0.000, EOS: 0.000
        Top Guided Tokens: 'a': 0.989, 'one': 0.003, 'the': 0.002, 'known': 0.002, 'an': 0.002, 'called': 0.000, 'basic': 0.000, 'considered': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France', '?', '\n', '\n']
        Selected: 'The' for batch_item=0                                                                                                  
        Top Raw Tokens: 'The': 0.973, 'Par': 0.027, 'The': 0.000, 'Paris': 0.000, 'PAR': 0.000, 'I': 0.000, 'There': 0.000, 'This': 0.000, EOS: 0.000
        Top Guided Tokens: 'The': 0.973, 'Par': 0.027, 'The': 0.000, 'Paris': 0.000, 'PAR': 0.000, 'I': 0.000, 'There': 0.000, 'This': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This', 'is', 'a']                                                                                                                                                                                                                                                                                
        Selected: 'basic' for batch_item=1                                                                                                
        Top Raw Tokens: 'basic': 0.617, 'fundamental': 0.292, 'simple': 0.088, 'very': 0.001, 'common': 0.001, 'straightforward': 0.000, 'well': 0.000, 'found': 0.000, EOS: 0.000
        Top Guided Tokens: 'basic': 0.617, 'fundamental': 0.292, 'simple': 0.088, 'very': 0.001, 'common': 0.001, 'straightforward': 0.000, 'well': 0.000, 'found': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms']                                                                                                                                                                                                                                                                    
        Selected: ',' for batch_item=2                                                                                                    
        Top Raw Tokens: ',': 0.997, ':': 0.003, 'it': 0.000, '': 0.000, 'we': 0.000, 'this': 0.000, '"': 0.000, ';': 0.000, EOS: 0.000
        Top Guided Tokens: ',': 0.997, ':': 0.003, 'it': 0.000, '': 0.000, 'we': 0.000, 'this': 0.000, '"': 0.000, ';': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'answer', 'is', '', '4', '.', 'This', 'is', 'a']                                                                                                                                                                                                                                                                                
        Selected: 'answer' for batch_item=3                                                                                               
        Top Raw Tokens: 'basic': 0.647, 'fundamental': 0.276, 'simple': 0.074, 'very': 0.001, 'common': 0.001, 'well': 0.000, 'straightforward': 0.000, 'found': 0.000, EOS: 0.000
        Top Guided Tokens: 'basic': 0.647, 'fundamental': 0.276, 'simple': 0.074, 'very': 0.001, 'common': 0.001, 'well': 0.000, 'straightforward': 0.000, 'found': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'So', ',', 'the']                                                                                                                                                                                                                                                                                 
        Selected: 'fundamental' for batch_item=4                                                                                          
        Top Raw Tokens: 'answer': 0.976, 'solution': 0.011, 'equation': 0.008, 'expression': 0.002, 'mathematical': 0.001, 'statement': 0.001, 'correct': 0.000, 'answer': 0.000, EOS: 0.000
        Top Guided Tokens: 'answer': 0.976, 'solution': 0.011, 'equation': 0.008, 'expression': 0.002, 'mathematical': 0.001, 'statement': 0.001, 'correct': 0.000, 'answer': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France', '?', '\n', '\n', 'The']                                                                                                                                                                                                                                                                              
        Selected: 'capital' for batch_item=0                                                                                              
        Top Raw Tokens: 'capital': 1.000, 'Capital': 0.000, 'capit': 0.000, 'city': 0.000, 'cap': 0.000, 'cap': 0.000, 'French': 0.000, 'correct': 0.000, EOS: 0.000
        Top Guided Tokens: 'capital': 1.000, 'Capital': 0.000, 'capit': 0.000, 'city': 0.000, 'cap': 0.000, 'cap': 0.000, 'French': 0.000, 'correct': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This', 'is', 'a', 'basic']                                                                                                                                                                                                                                                                       
        Selected: 'ar' for batch_item=1                                                                                                   
        Top Raw Tokens: 'ar': 0.873, 'mathematical': 0.103, 'addition': 0.008, 'math': 0.005, 'concept': 0.003, 'example': 0.002, 'and': 0.002, 'principle': 0.001, EOS: 0.001
        Top Guided Tokens: 'ar': 0.873, 'mathematical': 0.103, 'addition': 0.008, 'math': 0.005, 'concept': 0.003, 'example': 0.002, 'and': 0.002, 'principle': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',']                                                                                                                                                                                                                                                               
        Selected: 'to' for batch_item=2                                                                                                   
        Top Raw Tokens: '': 0.390, 'the': 0.299, 'you': 0.133, 'this': 0.092, 'it': 0.041, 'we': 0.041, 'if': 0.002, '"': 0.001, EOS: 0.001
        Top Guided Tokens: '': 0.390, 'the': 0.299, 'you': 0.133, 'this': 0.092, 'it': 0.041, 'we': 0.041, 'if': 0.002, '"': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'So', ',', 'the', 'answer']                                                                                                                                                                                                                                                                       
        Selected: '' for batch_item=3                                                                                                     
        Top Raw Tokens: 'to': 0.989, 'is': 0.011, '': 0.000, 'for': 0.000, 'of': 0.000, 'the': 0.000, 'would': 0.000, 'you': 0.000, EOS: 0.000
        Top Guided Tokens: 'to': 0.989, 'is': 0.011, '': 0.000, 'for': 0.000, 'of': 0.000, 'the': 0.000, 'would': 0.000, 'you': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This', 'is', 'a', 'fundamental']                                                                                                                                                                                                                                                                 
        Selected: 'the' for batch_item=4                                                                                                  
        Top Raw Tokens: 'ar': 0.434, 'concept': 0.272, 'mathematical': 0.258, 'principle': 0.017, 'math': 0.005, 'question': 0.003, 'and': 0.003, 'equation': 0.002, EOS: 0.002
        Top Guided Tokens: 'ar': 0.434, 'concept': 0.272, 'mathematical': 0.258, 'principle': 0.017, 'math': 0.005, 'question': 0.003, 'and': 0.003, 'equation': 0.002, EOS: 0.002
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France', '?', '\n', '\n', 'The', 'capital']                                                                                                                                                                                                                                                                   
        Selected: 'city' for batch_item=0                                                                                                 
        Top Raw Tokens: 'city': 1.000, 'city': 0.000, 'of': 0.000, 'City': 0.000, 'cities': 0.000, 'country': 0.000, 'City': 0.000, 'ciudad': 0.000, EOS: 0.000
        Top Guided Tokens: 'city': 1.000, 'city': 0.000, 'of': 0.000, 'City': 0.000, 'cities': 0.000, 'country': 0.000, 'City': 0.000, 'ciudad': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar']                                                                                                                                                                                                                                                                 
        Selected: 'ith' for batch_item=1                                                                                                  
        Top Raw Tokens: 'ith': 1.000, 'ITH': 0.000, 'it': 0.000, 'th': 0.000, 'thm': 0.000, 'itu': 0.000, 'che': 0.000, 'math': 0.000, EOS: 0.000
        Top Guided Tokens: 'ith': 1.000, 'ITH': 0.000, 'it': 0.000, 'th': 0.000, 'thm': 0.000, 'itu': 0.000, 'che': 0.000, 'math': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'So', ',', 'the', 'answer', 'to']                                                                                                                                                                                                                                                                 
        Selected: 'the' for batch_item=2                                                                                                  
        Top Raw Tokens: 'the': 0.856, '"': 0.121, '': 0.012, 'what': 0.008, 'this': 0.001, "'": 0.001, '_': 0.000, 'What': 0.000, EOS: 0.000
        Top Guided Tokens: 'the': 0.856, '"': 0.121, '': 0.012, 'what': 0.008, 'this': 0.001, "'": 0.001, '_': 0.000, 'What': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', '']                                                                                                                                                                                                                                                           
        Selected: '2' for batch_item=3                                                                                                    
        Top Raw Tokens: '2': 1.000, '4': 0.000, '\n': 0.000, '1': 0.000, 'you': 0.000, 'the': 0.000, '3': 0.000, '"': 0.000, EOS: 0.000
        Top Guided Tokens: '2': 1.000, '4': 0.000, '\n': 0.000, '1': 0.000, 'you': 0.000, 'the': 0.000, '3': 0.000, '"': 0.000, EOS: 0.000 
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', 'the']                                                                                                                                                                                                                                                        
        Selected: 'equation' for batch_item=4                                                                                             
        Top Raw Tokens: 'equation': 0.795, 'expression': 0.093, 'sum': 0.057, 'problem': 0.016, 'operation': 0.011, 'addition': 0.008, 'formula': 0.007, 'solution': 0.002, EOS: 0.002
        Top Guided Tokens: 'equation': 0.795, 'expression': 0.093, 'sum': 0.057, 'problem': 0.016, 'operation': 0.011, 'addition': 0.008, 'formula': 0.007, 'solution': 0.002, EOS: 0.002
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France', '?', '\n', '\n', 'The', 'capital', 'city']                                                                                                                                                                                                                                                           
        Selected: 'of' for batch_item=0                                                                                                   
        Top Raw Tokens: 'of': 1.000, 'of': 0.000, 'Of': 0.000, 'if': 0.000, 'France': 0.000, 'Paris': 0.000, 'Of': 0.000, 'to': 0.000, EOS: 0.000
        Top Guided Tokens: 'of': 1.000, 'of': 0.000, 'Of': 0.000, 'if': 0.000, 'France': 0.000, 'Paris': 0.000, 'Of': 0.000, 'to': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith']                                                                                                                                                                                                                                                          
        Selected: 'metic' for batch_item=1                                                                                                
        Top Raw Tokens: 'metic': 0.999, 'met': 0.001, 'matic': 0.000, 'metric': 0.000, 'met': 0.000, 'm': 0.000, 'mic': 0.000, 'ometric': 0.000, EOS: 0.000
        Top Guided Tokens: 'metic': 0.999, 'met': 0.001, 'matic': 0.000, 'metric': 0.000, 'met': 0.000, 'm': 0.000, 'mic': 0.000, 'ometric': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'So', ',', 'the', 'answer', 'to', 'the']                                                                                                                                                                                                                                                          
        Selected: 'question' for batch_item=2                                                                                             
        Top Raw Tokens: 'question': 0.923, 'equation': 0.038, 'mathematical': 0.022, 'expression': 0.009, 'simple': 0.005, 'math': 0.001, 'basic': 0.000, 'ar': 0.000, EOS: 0.000
        Top Guided Tokens: 'question': 0.923, 'equation': 0.038, 'mathematical': 0.022, 'expression': 0.009, 'simple': 0.005, 'math': 0.001, 'basic': 0.000, 'ar': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', '', '2']                                                                                                                                                                                                                                                      
        Selected: '+' for batch_item=3                                                                                                    
        Top Raw Tokens: '+': 0.935, '+': 0.054, 'plus': 0.007, 'is': 0.003, 'and': 0.000, '(': 0.000, '+(': 0.000, 'added': 0.000, EOS: 0.000
        Top Guided Tokens: '+': 0.935, '+': 0.054, 'plus': 0.007, 'is': 0.003, 'and': 0.000, '(': 0.000, '+(': 0.000, 'added': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', 'the', 'equation']                                                                                                                                                                                                                                            
        Selected: 'for' for batch_item=4                                                                                                  
        Top Raw Tokens: 'for': 0.718, '': 0.185, 'is': 0.026, 'that': 0.019, 'representing': 0.016, 'would': 0.015, '"': 0.007, 'to': 0.005, EOS: 0.005
        Top Guided Tokens: 'for': 0.718, '': 0.185, 'is': 0.026, 'that': 0.019, 'representing': 0.016, 'would': 0.015, '"': 0.007, 'to': 0.005, EOS: 0.005
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France', '?', '\n', '\n', 'The', 'capital', 'city', 'of']                                                                                                                                                                                                                                                     
        Selected: 'France' for batch_item=0                                                                                               
        Top Raw Tokens: 'France': 1.000, 'Fr': 0.000, 'Francia': 0.000, '____': 0.000, 'Paris': 0.000, 'Frances': 0.000, 'the': 0.000, 'a': 0.000, EOS: 0.000
        Top Guided Tokens: 'France': 1.000, 'Fr': 0.000, 'Francia': 0.000, '____': 0.000, 'Paris': 0.000, 'Frances': 0.000, 'the': 0.000, 'a': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic']                                                                                                                                                                                                                                                 
        Selected: '"' for batch_item=1                                                                                                    
        Top Raw Tokens: 'concept': 0.330, 'problem': 0.238, 'equation': 0.109, 'operation': 0.109, 'principle': 0.078, 'question': 0.062, 'fact': 0.040, 'calculation': 0.012, EOS: 0.012
        Top Guided Tokens: 'concept': 0.330, 'problem': 0.238, 'equation': 0.109, 'operation': 0.109, 'principle': 0.078, 'question': 0.062, 'fact': 0.040, 'calculation': 0.012, EOS: 0.012
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'So', ',', 'the', 'answer', 'to', 'the', 'question']                                                                                                                                                                                                                                              
        Selected: '2' for batch_item=2                                                                                                    
        Top Raw Tokens: '"': 0.956, ',': 0.019, "'": 0.013, '': 0.008, 'is': 0.003, '_': 0.001, 'What': 0.000, 'what': 0.000, EOS: 0.000
        Top Guided Tokens: '"': 0.956, ',': 0.019, "'": 0.013, '': 0.008, 'is': 0.003, '_': 0.001, 'What': 0.000, 'what': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', '', '2', '+']                                                                                                                                                                                                                                                 
        Selected: 'concept' for batch_item=3                                                                                              
        Top Raw Tokens: '2': 1.000, '': 0.000, '3': 0.000, 'two': 0.000, '1': 0.000, 'Two': 0.000, 'Two': 0.000, 'two': 0.000, EOS: 0.000
        Top Guided Tokens: '2': 1.000, '': 0.000, '3': 0.000, 'two': 0.000, '1': 0.000, 'Two': 0.000, 'Two': 0.000, 'two': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', 'the', 'equation', 'for']                                                                                                                                                                                                                                     
        Selected: 'problem' for batch_item=4                                                                                              
        Top Raw Tokens: '': 0.827, 'finding': 0.098, 'solving': 0.026, 'this': 0.016, 'the': 0.015, 'determining': 0.007, 'adding': 0.006, 'calcul': 0.001, EOS: 0.001
        Top Guided Tokens: '': 0.827, 'finding': 0.098, 'solving': 0.026, 'this': 0.016, 'the': 0.015, 'determining': 0.007, 'adding': 0.006, 'calcul': 0.001, EOS: 0.001
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France', '?', '\n', '\n', 'The', 'capital', 'city', 'of', 'France']                                                                                                                                                                                                                                           
        Selected: 'is' for batch_item=0                                                                                                   
        Top Raw Tokens: 'is': 1.000, 'is': 0.000, 'isn': 0.000, 'Is': 0.000, 'was': 0.000, 'in': 0.000, 'has': 0.000, 'are': 0.000, EOS: 0.000
        Top Guided Tokens: 'is': 1.000, 'is': 0.000, 'isn': 0.000, 'Is': 0.000, 'was': 0.000, 'in': 0.000, 'has': 0.000, 'are': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'So', ',', 'the', 'answer', 'to', 'the', 'question', '"']                                                                                                                                                                                                                                         
        Selected: 'What' for batch_item=1                                                                                                 
        Top Raw Tokens: 'What': 0.948, 'what': 0.052, '2': 0.000, 'What': 0.000, 'what': 0.000, 'WH': 0.000, 'Wh': 0.000, 'How': 0.000, EOS: 0.000
        Top Guided Tokens: 'What': 0.948, 'what': 0.052, '2': 0.000, 'What': 0.000, 'what': 0.000, 'WH': 0.000, 'Wh': 0.000, 'How': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', '', '2', '+', '2']                                                                                                                                                                                                                                            
        Selected: 'that' for batch_item=2                                                                                                 
        Top Raw Tokens: 'is': 0.460, 'can': 0.394, 'equals': 0.046, '=': 0.044, '=': 0.030, 'represents': 0.018, 'means': 0.004, 'refers': 0.001, EOS: 0.001
        Top Guided Tokens: 'is': 0.460, 'can': 0.394, 'equals': 0.046, '=': 0.044, '=': 0.030, 'represents': 0.018, 'means': 0.004, 'refers': 0.001, EOS: 0.001
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic', 'concept']                                                                                                                                                                                                                                      
        Selected: 'is' for batch_item=3                                                                                                   
        Top Raw Tokens: 'that': 0.851, 'in': 0.041, ',': 0.036, 'and': 0.035, 'learned': 0.018, 'known': 0.009, '.': 0.003, 'taught': 0.002, EOS: 0.002
        Top Guided Tokens: 'that': 0.851, 'in': 0.041, ',': 0.036, 'and': 0.035, 'learned': 0.018, 'known': 0.009, '.': 0.003, 'taught': 0.002, EOS: 0.002
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic', 'problem']                                                                                                                                                                                                                                      
        Selected: 'can' for batch_item=4                                                                                                  
        Top Raw Tokens: 'that': 0.478, 'and': 0.314, ',': 0.163, 'with': 0.023, 'often': 0.005, 'in': 0.004, 'commonly': 0.004, '.': 0.004, EOS: 0.004
        Top Guided Tokens: 'that': 0.478, 'and': 0.314, ',': 0.163, 'with': 0.023, 'often': 0.005, 'in': 0.004, 'commonly': 0.004, '.': 0.004, EOS: 0.004
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France', '?', '\n', '\n', 'The', 'capital', 'city', 'of', 'France', 'is']                                                                                                                                                                                                                                     
        Selected: 'Paris' for batch_item=0                                                                                                
        Top Raw Tokens: 'Paris': 1.000, 'called': 0.000, 'not': 0.000, 'actually': 0.000, 'named': 0.000, 'known': 0.000, 'Par': 0.000, '"': 0.000, EOS: 0.000
        Top Guided Tokens: 'Paris': 1.000, 'called': 0.000, 'not': 0.000, 'actually': 0.000, 'named': 0.000, 'known': 0.000, 'Par': 0.000, '"': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'So', ',', 'the', 'answer', 'to', 'the', 'question', '"', 'What']                                                                                                                                                                                                                                 
        Selected: 'is' for batch_item=1                                                                                                   
        Top Raw Tokens: 'is': 1.000, 'Is': 0.000, 'is': 0.000, "'": 0.000, 'are': 0.000, 'it': 0.000, 'does': 0.000, '': 0.000, EOS: 0.000 
        Top Guided Tokens: 'is': 1.000, 'Is': 0.000, 'is': 0.000, "'": 0.000, 'are': 0.000, 'it': 0.000, 'does': 0.000, '': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic', 'concept', 'that']                                                                                                                                                                                                                              
        Selected: 'is' for batch_item=2                                                                                                   
        Top Raw Tokens: 'is': 0.676, 'most': 0.206, 'forms': 0.069, 'can': 0.010, 'has': 0.008, 'applies': 0.006, 'everyone': 0.005, 'holds': 0.004, EOS: 0.004
        Top Guided Tokens: 'is': 0.676, 'most': 0.206, 'forms': 0.069, 'can': 0.010, 'has': 0.008, 'applies': 0.006, 'everyone': 0.005, 'holds': 0.004, EOS: 0.004
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', '', '2', '+', '2', 'is']                                                                                                                                                                                                                                      
        Selected: 'be' for batch_item=3                                                                                                   
        Top Raw Tokens: 'an': 0.558, 'expressed': 0.094, 'read': 0.073, 'represented': 0.071, 'called': 0.040, 'written': 0.038, 'equal': 0.028, 'the': 0.017, EOS: 0.017
        Top Guided Tokens: 'an': 0.558, 'expressed': 0.094, 'read': 0.073, 'represented': 0.071, 'called': 0.040, 'written': 0.038, 'equal': 0.028, 'the': 0.017, EOS: 0.017
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', '', '2', '+', '2', 'can']                                                                                                                                                                                                                                     
        Selected: 'an' for batch_item=4                                                                                                   
        Top Raw Tokens: 'be': 0.994, 'also': 0.006, 'simply': 0.000, 'represented': 0.000, 'be': 0.000, 'represent': 0.000, 'been': 0.000, 'expressed': 0.000, EOS: 0.000
        Top Guided Tokens: 'be': 0.994, 'also': 0.006, 'simply': 0.000, 'represented': 0.000, 'be': 0.000, 'represent': 0.000, 'been': 0.000, 'expressed': 0.000, EOS: 0.000
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France', '?', '\n', '\n', 'The', 'capital', 'city', 'of', 'France', 'is', 'Paris']                                                                                                                                                                                                                            
        Selected: '.' for batch_item=0                                                                                                    
        Top Raw Tokens: '.': 1.000, ',': 0.000, '(': 0.000, '!': 0.000, '\n': 0.000, ';': 0.000, 'which': 0.000, ':': 0.000, EOS: 0.000
        Top Guided Tokens: '.': 1.000, ',': 0.000, '(': 0.000, '!': 0.000, '\n': 0.000, ';': 0.000, 'which': 0.000, ':': 0.000, EOS: 0.000 
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'So', ',', 'the', 'answer', 'to', 'the', 'question', '"', 'What', 'is']                                                                                                                                                                                                                           
        Selected: '' for batch_item=1                                                                                                     
        Top Raw Tokens: '': 1.000, 'the': 0.000, 'two': 0.000, '2': 0.000, ' ': 0.000, '\\(': 0.000, '$': 0.000, 'Two': 0.000, EOS: 0.000
        Top Guided Tokens: '': 1.000, 'the': 0.000, 'two': 0.000, '2': 0.000, ' ': 0.000, '\\(': 0.000, '$': 0.000, 'Two': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic', 'concept', 'that', 'is']                                                                                                                                                                                                                        
        Selected: 'represented' for batch_item=2                                                                                          
        Top Raw Tokens: 'commonly': 0.252, 'taught': 0.133, 'learned': 0.122, 'fundamental': 0.117, 'widely': 0.061, 'easy': 0.054, 'found': 0.051, 'often': 0.048, EOS: 0.048
        Top Guided Tokens: 'commonly': 0.252, 'taught': 0.133, 'learned': 0.122, 'fundamental': 0.117, 'widely': 0.061, 'easy': 0.054, 'found': 0.051, 'often': 0.048, EOS: 0.048
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', '', '2', '+', '2', 'can', 'be']                                                                                                                                                                                                                               
        Selected: 'commonly' for batch_item=3                                                                                             
        Top Raw Tokens: 'represented': 0.522, 'written': 0.281, 'expressed': 0.194, 'described': 0.001, 'denoted': 0.000, 'read': 0.000, 'defined': 0.000, 'represent': 0.000, EOS: 0.000
        Top Guided Tokens: 'represented': 0.522, 'written': 0.281, 'expressed': 0.194, 'described': 0.001, 'denoted': 0.000, 'read': 0.000, 'defined': 0.000, 'represent': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', '', '2', '+', '2', 'is', 'an']                                                                                                                                                                                                                                
        Selected: 'written' for batch_item=4                                                                                              
        Top Raw Tokens: 'ar': 0.397, 'example': 0.357, 'addition': 0.165, 'equation': 0.035, 'expression': 0.015, 'equality': 0.014, 'equal': 0.005, 'elementary': 0.004, EOS: 0.004
        Top Guided Tokens: 'ar': 0.397, 'example': 0.357, 'addition': 0.165, 'equation': 0.035, 'expression': 0.015, 'equality': 0.014, 'equal': 0.005, 'elementary': 0.004, EOS: 0.004
--------Batch Logit Details--------                                                                                                       
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'the', 'numbers', '', '2', 'and', '', '2', '.', 'The', 'answer', 'is', '', '4', '.', '\n', '\n', 'What', 'is', 'the', 'capital', 'city', 'of', 'France', '?', '\n', '\n', 'The', 'capital', 'city', 'of', 'France', 'is', 'Paris', '.']                                                                                                                                                                                                                       
        Selected: '\n' for batch_item=0                                                                                                   
        Top Raw Tokens: '\n': 0.551, 'Paris': 0.348, 'It': 0.092, 'This': 0.008, 'The': 0.000, 'France': 0.000, 'Its': 0.000, 'Par': 0.000, EOS: 0.000
        Top Guided Tokens: '\n': 0.551, 'Paris': 0.348, 'It': 0.092, 'This': 0.008, 'The': 0.000, 'France': 0.000, 'Its': 0.000, 'Par': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'So', ',', 'the', 'answer', 'to', 'the', 'question', '"', 'What', 'is', '']                                                                                                                                                                                                                       
        Selected: 'Paris' for batch_item=1                                                                                                
        Top Raw Tokens: '2': 1.000, '1': 0.000, '3': 0.000, 'Two': 0.000, 'two': 0.000, 'two': 0.000, 'Two': 0.000, '_': 0.000, EOS: 0.000
        Top Guided Tokens: '2': 1.000, '1': 0.000, '3': 0.000, 'Two': 0.000, 'two': 0.000, 'two': 0.000, 'Two': 0.000, '_': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', '', '2', '+', '2', 'can', 'be', 'represented']                                                                                                                                                                                                                
        Selected: '2' for batch_item=2                                                                                                    
        Top Raw Tokens: 'as': 0.854, 'by': 0.116, 'using': 0.027, 'in': 0.002, 'with': 0.001, 'symbol': 0.000, 'through': 0.000, 'algebra': 0.000, EOS: 0.000
        Top Guided Tokens: 'as': 0.854, 'by': 0.116, 'using': 0.027, 'in': 0.002, 'with': 0.001, 'symbol': 0.000, 'through': 0.000, 'algebra': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'This', 'is', 'a', 'basic', 'ar', 'ith', 'metic', 'concept', 'that', 'is', 'commonly']                                                                                                                                                                                                            
        Selected: 'It' for batch_item=3                                                                                                   
        Top Raw Tokens: 'taught': 0.774, 'understood': 0.085, 'learned': 0.072, 'known': 0.042, 'used': 0.022, 'referred': 0.003, 'introduced': 0.001, 'accepted': 0.000, EOS: 0.000
        Top Guided Tokens: 'taught': 0.774, 'understood': 0.085, 'learned': 0.072, 'known': 0.042, 'used': 0.022, 'referred': 0.003, 'introduced': 0.001, 'accepted': 0.000, EOS: 0.000
Generation: ['', 'What', 'is', '', '2', '+', '2', '?', '\n', '\n', '2', '+', '2', 'is', 'the', 'sum', 'of', 'two', 'numbers', ',', '', '2', 'and', '', '2', '.', 'When', 'you', 'add', 'these', 'two', 'numbers', 'together', ',', 'the', 'result', 'is', '', '4', '.', 'In', 'mathematical', 'terms', ',', '', '2', '+', '2', 'can', 'be', 'written']                                                                                                                                                                                                                    
        Selected: 'as' for batch_item=4                                                                                                   
        Top Raw Tokens: 'as': 0.995, 'using': 0.003, 'in': 0.002, 'with': 0.000, 'and': 0.000, 'out': 0.000, 'an': 0.000, 'hor': 0.000, EOS: 0.000
        Top Guided Tokens: 'as': 0.995, 'using': 0.003, 'in': 0.002, 'with': 0.000, 'and': 0.000, 'out': 0.000, 'an': 0.000, 'hor': 0.000, EOS: 0.000
--------Batch Logit Details--------

@lapp0 lapp0 mentioned this pull request Feb 9, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Allow Debug Logging of Logits
1 participant
@lapp0