AIMon Rely is a state-of-the-art system consisting of multiple models for detecting LLM quality issues during offline evaluations and continuous production monitoring. We offer hallucination metrics that is fast, reliable and cost-effective. We also support additional metrics such as completeness, conciseness and toxicity.
Read our blog post for more details.
✨ Join our community on Slack or reach out to us at info@aimon.ai to get your API key.
These are the quality metrics that are currently available via the API. Some of them are in progress and will be available in a future release.
| Metric | Status |
|---|---|
| Model Hallucination (Passage and Sentence Level) | ✓ |
| Completeness | ✓ |
| Conciseness | ✓ |
| Toxicity | ✓ |
| Semantic Similarity | ⌛ |
| Sentiment | ⌛ |
| Coherence | ⌛ |
| Sensitive Data (PII/PHI/PCI) | ⌛ |
Follow these steps to use the product:
- Step 1: Get access to the beta product by joining the wait list on our website or by requesting it on Slack or sending an email to info@aimon.ai
- Step 2: Install the AIMon SDK by running
pip install aimonin your terminal. - Step 3: Refer to the sample notebook for an example of how to instrument an LLM application using our SDK.
Steps to use the API:
- Step 1: Get your API key by requesting it on our Slack or sending an email to info@aimon.ai
- Step 2: You can try the API using either of these methods
- [OPTION 1] Try the simple langchain summarization application that is augmented with AIMon Rely to detect
hallucinations at the sentence level.
- Step 1: Run
pip install -r examples/requirements.txt && pip install aimon - Step 2: Run
streamlit run examples/langchain_summarization_app.py
- Step 1: Run
- [OPTION 2] Download the Postman collection specified below to access the API
- Model Hallucination (Passage and Sentence Level): Postman Collection
- [OPTION 1] Try the simple langchain summarization application that is augmented with AIMon Rely to detect
hallucinations at the sentence level.
You can play with a Sandbox that is available on our website.
To demonstrate the effectiveness of our system, we benchmarked it against popular industry benchmarks for the hallucination detection task. The table below shows our results.
A few key takeaways:
✅ AIMon Rely is 10x cheaper than GPT-4 Turbo.
✅ AIMon Rely is 4x faster than GPT-4 Turbo.
✅ AIMon Rely provides the convenience of a fully hosted API that includes baked-in explainability.
✅ Support for a context length of up to 32,000 tokens (with plans to further expand this in the near future).
Overall, AIMon Rely is 10 times cheaper, 4 times faster and close to or even better than GPT-4 on the benchmarks making it a suitable choice for both offline and online detection of hallucinations.
There is a lack of industry standard benchmark datasets for these metrics. We will be publishing an evaluation dataset soon. Stay Tuned! ⌛
Please reach out to info@aimon.ai for pricing details related to the product and the API.
- We are working on additional metrics as detailed in the table above.
- In addition, we are working on something awesome to make the offline evaluation and continuous model quality monitoring experience more seamless.
Join our Slack for the latest updates and discussions on generative AI reliability.