Phi-3-mini-4k-instruct : Failing to stop at <|end|> on generating the answer.

[mustangs0786]

from mlx_lm import load, generate


model, tokenizer = load(path_or_hf_repo = phi3_path,#,tokenizer_config=tokenizer_config
                  # adapter_path = adapt_path,
                        tokenizer_config={"trust_remote_code": True})

messages = [
        {"role": "user", "content": 'Where is delhi, explain it only 3 points.'},
    ]
prompt = tokenizer.apply_chat_template(
            messages , tokenize=False, add_generation_prompt=True
        )

response = generate(model, tokenizer, prompt=prompt, verbose=True, temp=0.0, max_tokens=700)

########################## Output ##########

==========
Prompt: <s><|user|>
Where is delhi, explain it only 3 points.<|end|>
<|assistant|>

1. Capital city: Delhi is the capital city of India, serving as the political, cultural, and economic center of the country.

2. Historical significance: Delhi has a rich history dating back over 2,000 years, with several important empires and dynasties having ruled the city, including the Mauryans, Guptas, and Mughals. The city is home to numerous historical landmarks, such as the Red Fort, Qutub Minar, and Humayun's Tomb.

3. Modern metropolis: Today, Delhi is a bustling metropolis, with a population of over 30 million people. It is known for its diverse culture, vibrant street life, and numerous educational institutions, including the Indian Institute of Technology (IIT) and Jawaharlal Nehru University (JNU). The city is also a major hub for business, commerce, and transportation, with the Indira Gandhi International Airport being one of the busiest airports in India.<|end|><|assistant|> 1. Capital of India: Delhi is the capital city of India, serving as the seat of the government and the location of the President's official residence, Rashtrapati Bhavan.

2. Cultural melting pot: Delhi is a city with a rich and diverse cultural heritage, influenced by various religions, languages, and traditions. It is home to numerous historical monuments, museums, and art galleries, such as the National Museum, India Gate, and Lotus Temple.

3. Economic powerhouse: Delhi is one of the largest and fastest-growing metropolitan areas in India, with a thriving economy driven by sectors such as information technology, telecommunications, and manufacturing. The city is also a major center for education, with numerous prestigious universities and research institutions, including the Indian Institute of Technology (IIT) and Jawaharlal Nehru University (JNU).<|end|><|assistant|> 1. Political center: Delhi is the capital of India, housing the central government's administrative buildings, including the Parliament House, Rashtrapati Bhavan, and the Supreme Court of India.

2. Cultural and historical significance: Delhi is a city with a rich history and diverse cultural heritage, dating back over 2,000 years. It is home to numerous UNESCO World Heritage Sites, such as the Qutub Minar, Humayun's Tomb, and the Red Fort, which showcase the city's historical and architectural significance.

3. Economic and educational hub: Delhi is a major economic and educational center in India, with a thriving service sector, including information technology, telecommunications, and finance. The city is also home to numerous prestigious educational institutions, such as the Indian Institute of Technology (IIT), Jawaharlal Nehru University (JNU), and the University of Delhi.<|end|><|assistant|> 1. Capital city: Delhi is the capital of India, serving as the political, administrative, and cultural center of the country. It houses the central government's institutions, including the Parliament, the President's residence (Rashtrapati

[mustangs0786]

i dont know how text is getting cross lines. please anyhelp thanks

[awni]

Could you share the model you are using (e.g. phi3_path)?

[mustangs0786]

convert(hf_path = "microsoft/Phi-3-mini-4k-instruct",
mlx_path = "phir3_model_f",q_bits = 8,q_group_size=32,
quantize=True)

phi3_path = "/Train_custom_LLM/LLMX/phir3_model_f/"

Thanks for quick reply and editing my question..

[JosefAlbers]

It seems this isn't a bug. The original Phi-3-mini-4k-instruct model uses "<|endoftext|>" (not <|end|>) as its end-of-sequence (EOS) token. You can verify this in the model's tokenizer.config file.

[awni]

I updated the MLX Community Phi-3 models to use the correct eos token.

Also you can always specify the eos token in the load function like so:

load("model_name", tokenizer_config={"eos_token": "<|eos|>"})