Merge pull request #38 from zhudotexe/v1-dev

kani v1

.github/workflows/pytest.yml

@@ -54,7 +54,7 @@ jobs:
         uses: actions/cache@v3
         with:
           path: ~/.cache/huggingface
-          key: ${{ runner.os }}-llamav2
+          key: ${{ runner.os }}-llamav2-cpp
 
       - name: Run LLaMA tests
         run: pytest -m llama

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2023 Andrew Zhu
+Copyright (c) 2023-present Andrew Zhu
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -18,6 +18,13 @@
   <a href="https://discord.gg/eTepTNDxYT">
     <img alt="Discord" src="https://img.shields.io/discord/1150902904773935214?color=5865F2&label=discord&logo=discord&logoColor=white">
   </a>
+  <br/>
+  <a href="examples/4_engines_zoo.py">
+    <img alt="Model zoo" src="https://img.shields.io/badge/examples-model_zoo-blue">
+  </a>
+  <a href="examples/5_advanced_retrieval.py">
+    <img alt="Retrieval example" src="https://img.shields.io/badge/examples-retrieval-blue">
+  </a>
 </p>
 
 # kani (カニ)
@@ -32,10 +39,26 @@ developers alike.
 kani comes with support for the following models out of the box, with a model-agnostic framework to add support for many
 more:
 
+**Hosted Models**
+
 - OpenAI Models (GPT-3.5-turbo, GPT-4, GPT-4-turbo)
 - Anthropic Models (Claude, Claude Instant)
-- LLaMA v2 (via Hugging Face or ctransformers) & fine-tunes
-- Vicuna v1.3 (via Hugging Face) & fine-tunes
+
+**Open Source Models**
+
+kani supports every chat model available on Hugging Face through `transformers` or `llama.cpp`!
+
+In particular, we have reference implementations for the following base models, and their fine-tunes:
+
+- [Command R](https://huggingface.co/CohereForAI/c4ai-command-r-v01)
+  and [Command R+](https://huggingface.co/CohereForAI/c4ai-command-r-plus)
+- [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)
+  and [Mixtral-8x7B](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) (uses LLaMA prompting format)
+- [Gemma](https://huggingface.co/collections/google/gemma-release-65d5efbccdbb8c4202ec078b) (all sizes)
+- [LLaMA 2](https://huggingface.co/meta-llama) (all sizes)
+- [Vicuna v1.3](https://huggingface.co/lmsys/vicuna-7b-v1.3)
+
+Check out the [Model Zoo](examples/4_engines_zoo.py) to see how to use each of these models in your application!
 
 **Interested in contributing? Check out our
 [guide](https://kani.readthedocs.io/en/latest/community/contributing.html).**
@@ -49,7 +72,7 @@ more:
 - **Lightweight and high-level** - kani implements common boilerplate to interface with language models without forcing
   you to use opinionated prompt frameworks or complex library-specific tooling.
 - **Model agnostic** - kani provides a simple interface to implement: token counting and completion generation.
-  Implement these two, and kani can run with any language model.
+  kani lets developers switch which language model runs on the backend without major code refactors.
 - **Automatic chat memory management** - Allow chat sessions to flow without worrying about managing the number of
   tokens in the history - kani takes care of it.
 - **Function calling with model feedback and retry** - Give models access to functions in just one line of code.
@@ -79,6 +102,7 @@ Then, let's use kani to create a simple chatbot using ChatGPT as a backend.
 
 ```python
 # import the library
+import asyncio
 from kani import Kani, chat_in_terminal
 from kani.engines.openai import OpenAIEngine
 
@@ -94,9 +118,17 @@ engine = OpenAIEngine(api_key, model="gpt-3.5-turbo")
 # system_prompt="You are..." here.
 ai = Kani(engine)
 
-# kani comes with a utility to interact with a kani through your terminal! Check out 
-# the docs for how to use kani programmatically.
+# kani comes with a utility to interact with a kani through your terminal...
 chat_in_terminal(ai)
+
+
+# or you can use kani programmatically in an async function!
+async def main():
+    resp = await ai.chat_round("What is the airspeed velocity of an unladen swallow?")
+    print(resp.text)
+
+
+asyncio.run(main())
 ```
 
 kani makes the time to set up a working chat model short, while offering the programmer deep customizability over
@@ -112,8 +144,9 @@ decorator.
 
 ```python
 # import the library
+import asyncio
 from typing import Annotated
-from kani import AIParam, Kani, ai_function, chat_in_terminal
+from kani import AIParam, Kani, ai_function, chat_in_terminal, ChatRole
 from kani.engines.openai import OpenAIEngine
 
 # set up the engine as above
@@ -136,13 +169,47 @@ class MyKani(Kani):
 
 
 ai = MyKani(engine)
+
+# the terminal utility allows you to test function calls...
 chat_in_terminal(ai)
+
+
+# and you can track multiple rounds programmatically.
+async def main():
+    async for msg in ai.full_round("What's the weather in Tokyo?"):
+        print(msg.role, msg.text)
+
+
+asyncio.run(main())
 ```
 
 kani guarantees that function calls are valid by the time they reach your methods while allowing you to focus on
 writing code. For more information, check
 out [the function calling docs](https://kani.readthedocs.io/en/latest/function_calling.html).
 
+## Streaming
+
+kani supports streaming responses from the underlying language model token-by-token, even in the presence of function
+calls. Streaming is designed to be a drop-in superset of the ``chat_round`` and ``full_round`` methods, allowing you to
+gradually refactor your code without ever leaving it in a broken state.
+
+```python
+async def stream_chat():
+    stream = ai.chat_round_stream("What does kani mean?")
+    async for token in stream:
+        print(token, end="")
+    print()
+    msg = await stream.message()  # or `await stream`
+
+
+async def stream_with_function_calling():
+    async for stream in ai.full_round_stream("What's the weather in Tokyo?"):
+        async for token in stream:
+            print(token, end="")
+        print()
+        msg = await stream.message()
+```
+
 ## Why kani?
 
 Existing frameworks for language models like LangChain and simpleaichat are opinionated and/or heavyweight - they edit
@@ -159,7 +226,6 @@ kani is to LangChain as Flask (or FastAPI) is to Django.
 kani is appropriate for everyone from academic researchers to industry professionals to hobbyists to use without
 worrying about under-the-hood hacks.
 
-
 ## Docs
 
 To learn more about how
@@ -178,21 +244,6 @@ https://github.com/zhudotexe/kani/actions/workflows/pytest.yml?query=branch%3Ama
 
 Simply click on the latest build to see LLaMA's output!
 
-## Kani in the News
-
-Kani will appear at the NLP Open Source Software workshop at EMNLP 2023!
-
-We are really excited and grateful to see people talking about Kani online. We are also trending on Papers With Code,
-GitHub, and OSS Insight. Check out some recent articles and videos below!
-
-- [Researchers from the University of Pennsylvania Introduce Kani: A Lightweight, Flexible, and Model-Agnostic Open-Source AI Framework for Building Language Model Applications](https://www.marktechpost.com/2023/09/18/researchers-from-the-university-of-pennsylvania-introduce-kani-a-lightweight-flexible-and-model-agnostic-open-source-ai-framework-for-building-language-model-applications/)
-- [Unlocking AI Potential: Unveiling Kani, the Groundbreaking Open-Source Framework Revolutionizing Large Language Model Applications](https://www.cjco.com.au/article/news/unlocking-ai-potential-unveiling-kani-the-groundbreaking-open-source-framework-revolutionizing-large-language-model-applications/)
-- [Kani: A Lightweight and Customizable Framework for Language Model Applications](https://ts2.space/en/kani-a-lightweight-and-customizable-framework-for-language-model-applications/)
-- [Introducing Kani (Sanskrit Word): A Game-Changing Open-Source AI Framework for Language Models](https://www.linkedin.com/pulse/introducing-kani-sanskrit-word-game-changing/)
-    - *Kani was originally named after the Japanese word for crab and coincidentally means "knowledge" in Sanskrit.*
-- [kani: lightweight LLM framework (Japanese)](https://note.com/hamachi_jp/n/n342becc4f345)
-- [Top Trending LLM Projects of the Week: Dive into the Future of Tech! 🚀](https://www.youtube.com/watch?v=qoGKzmnhAnA)
-
 ## Who we are
 
 <img alt="University of Pennsylvania Logo" src="docs/_static/penn-logo.jpg" width="300">
@@ -220,13 +271,25 @@ University of Pennsylvania. We're all members of
 If you use Kani, please cite us as:
 
 ```
-@misc{zhu2023kani,
-      title={Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applications}, 
-      author={Andrew Zhu and Liam Dugan and Alyssa Hwang and Chris Callison-Burch},
-      year={2023},
-      eprint={2309.05542},
-      archivePrefix={arXiv},
-      primaryClass={cs.SE}
+@inproceedings{zhu-etal-2023-kani,
+    title = "Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applications",
+    author = "Zhu, Andrew  and
+      Dugan, Liam  and
+      Hwang, Alyssa  and
+      Callison-Burch, Chris",
+    editor = "Tan, Liling  and
+      Milajevs, Dmitrijs  and
+      Chauhan, Geeticka  and
+      Gwinnup, Jeremy  and
+      Rippeth, Elijah",
+    booktitle = "Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)",
+    month = dec,
+    year = "2023",
+    address = "Singapore",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2023.nlposs-1.8",
+    doi = "10.18653/v1/2023.nlposs-1.8",
+    pages = "65--77",
 }
 ```
 

citation.cff

@@ -15,7 +15,7 @@ authors:
   orcid: "https://orcid.org/0000-0001-8196-1943"
 title: "kani"
 date-released: 2023-09-11
-url: "https://github.com/zhudotexe/kani/tree/main"
+url: "https://github.com/zhudotexe/kani"
 preferred-citation:
   type: generic
   authors:
@@ -32,10 +32,13 @@ preferred-citation:
     given-names: "Chris"
     orcid: "https://orcid.org/0000-0001-8196-1943"
   year: 2023
-  month: 9
-  day: 11
+  month: 12
   title: "Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applications"
-  eprint: 2309.05542
-  archive-prefix: "arXiv"
-  primary-class: "cs.SE"
-  url: "https://arxiv.org/abs/2309.05542"
+  collection-title: "Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)"
+  collection-type: proceedings
+  location: "Singapore"
+  doi: "10.18653/v1/2023.nlposs-1.8"
+  url: "https://aclanthology.org/2023.nlposs-1.8"
+  publisher: "Association for Computational Linguistics"
+  start: 65
+  end: 77

docs/api_reference.rst

@@ -48,6 +48,27 @@ AI Function
 .. autoclass:: kani.AIParam
     :members:
 
+Streaming
+---------
+.. autoclass:: kani.streaming.StreamManager
+    :members:
+
+    .. automethod:: __await__
+
+    .. automethod:: __aiter__
+
+Prompting
+---------
+.. automodule:: kani.prompts
+
+.. autoclass:: kani.PromptPipeline
+    :members:
+
+    .. automethod:: __call__
+
+.. autoclass:: kani.prompts.PipelineStep
+    :members:
+
 Internals
 ---------
 .. autoclass:: kani.FunctionCallResult
@@ -66,6 +87,8 @@ Utilities
 
 .. autofunction:: kani.chat_in_terminal_async
 
+.. autofunction:: kani.print_stream
+
 Message Formatters
 ^^^^^^^^^^^^^^^^^^
 .. automodule:: kani.utils.message_formatters

docs/community/contributing.rst

@@ -11,8 +11,6 @@ You can contribute in many ways: helping us develop the library,
 
 (Funding agencies: the core devs are PhD students - you can support us that way too! |:wink:| Send us an email.)
 
-.. todo: cite us
-
 Contributing Code
 -----------------
 When contributing code and opening PRs, there are a couple things to keep in mind. kani is built on the principles

docs/conf.py

@@ -14,7 +14,7 @@
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
 
 project = "kani"
-copyright = "2023, Andrew Zhu, Liam Dugan, and Alyssa Hwang"
+copyright = "2023-present, Andrew Zhu, Liam Dugan, and Alyssa Hwang"
 author = "Andrew Zhu, Liam Dugan, and Alyssa Hwang"
 
 # -- General configuration ---------------------------------------------------
@@ -51,8 +51,10 @@
 nitpick_ignore_regex = [
     (r"py:class", r"aiohttp\..*"),  # aiohttp intersphinx is borked
     (r"py:class", r"torch\..*"),  # idk if torch has intersphinx
-    (r"py:class", r"kani\.engines\.openai\.models\..*"),  # internal models are just pydantic models of the oai api
+    (r"py:class", r"openai\..*"),  # openai does not use sphinx for docs
     (r"py:class", r"anthropic\..*"),  # anthropic does not use sphinx for docs
+    (r"py:class", r"asyncio\.\w+\..*"),  # asyncio submodule intersphinx is borked
+    (r"py:class", r"kani\..*\.[\w_0-9]*T"),  # ignore generics and other typevars
 ]
 
 # sphinx.ext.autodoc

docs/engine_reference.rst

@@ -21,9 +21,6 @@ OpenAI
 ------
 .. autoclass:: kani.engines.openai.OpenAIEngine
 
-.. autoclass:: kani.engines.openai.client.OpenAIClient
-    :members:
-
 Anthropic
 ---------
 .. autoclass:: kani.engines.anthropic.AnthropicEngine
@@ -37,13 +34,13 @@ Hugging Face
 .. autoclass:: kani.engines.huggingface.llama2.LlamaEngine
     :members:
 
-.. autoclass:: kani.engines.huggingface.vicuna.VicunaEngine
+.. autoclass:: kani.engines.huggingface.cohere.CommandREngine
     :members:
 
-CTransformers
--------------
-.. autoclass:: kani.engines.ctransformers.CTransformersEngine
+.. autoclass:: kani.engines.huggingface.vicuna.VicunaEngine
     :members:
 
-.. autoclass:: kani.engines.ctransformers.llama2.LlamaCTransformersEngine
+llama.cpp
+---------
+.. autoclass:: kani.engines.llamacpp.LlamaCppEngine
     :members:

docs/engines.rst

@@ -18,4 +18,4 @@ In this section, we'll discuss how to implement your own engine to use any langu
     engines/implementing
     engines/http
     engines/huggingface
-    engines/ctransformers
+    engines/llamacpp

docs/engines/http.rst

@@ -1,14 +1,18 @@
 HTTP Client
 ===========
+
+.. danger::
+    The aiohttp-based BaseClient has been deprecated in v1.0.0 and will be removed in a future version.
+    We recommend using `httpx <https://www.python-httpx.org/>`_ to make HTTP requests instead.
+
+    We have removed the top-level library dependency on ``aiohttp`` - existing code using the BaseClient will require
+    manual installation of the ``aiohttp`` package.
+
 If your language model backend exposes an HTTP API, you can create a subclass of :class:`.BaseClient` to interface with
 it. Your engine should then create an instance of the new HTTP client and call it to make predictions.
 
 Minimally, to use the HTTP client, your subclass should set the ``SERVICE_BASE`` class variable.
 
-.. seealso::
-
-    The source code of the :class:`.OpenAIClient`, which uses the HTTP client.
-
 .. autoclass:: kani.engines.httpclient.BaseClient
     :noindex:
     :members: