Skip to content

Releases: xenova/transformers.js

2.17.2

29 May 14:36
Compare
Choose a tag to compare

🚀 What's new?

  • Add support for MobileViTv2 in #721

    import { pipeline } from '@xenova/transformers';
    
    // Create an image classification pipeline
    const classifier = await pipeline('image-classification', 'Xenova/mobilevitv2-1.0-imagenet1k-256', {
        quantized: false,
    });
    
    // Classify an image
    const url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/tiger.jpg';
    const output = await classifier(url);
    // [{ label: 'tiger, Panthera tigris', score: 0.6491137742996216 }]

    See here for the full list of supported models.

  • Add support for FastViT in #749

    import { pipeline } from '@xenova/transformers';
    
    // Create an image classification pipeline
    const classifier = await pipeline('image-classification', 'Xenova/fastvit_t12.apple_in1k', {
      quantized: false
    });
    
    // Classify an image
    const url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/tiger.jpg';
    const output = await classifier(url, { topk: 5 });
    // [
    //   { label: 'tiger, Panthera tigris', score: 0.6649345755577087 },
    //   { label: 'tiger cat', score: 0.12454754114151001 },
    //   { label: 'lynx, catamount', score: 0.0010689536575227976 },
    //   { label: 'dhole, Cuon alpinus', score: 0.0010422508930787444 },
    //   { label: 'silky terrier, Sydney silky', score: 0.0009548701345920563 }
    // ]

    See here for the full list of supported models.

  • Optimize FFT in #766

  • Auto rotate image by @KTibow in #737

  • Support reading data from blob URI by @hans00 in #645

  • Add sequence post processor in #771

  • Add model file name by @NawarA in #594

  • Update pipelines.js to allow for token_embeddings as well by @NikhilVerma in #770

  • Remove old import from stream/web for ReadableStream in #752

  • Update tokenizer playground by @xenova in #717

  • Use ungated version of mistral tokenizer by @xenova in #718

  • docs: update vanilla-js.md by @eltociear in #738

  • Fix CI by in #768

  • Update Next.js demos to 14.2.3 in #772

🤗 New contributors

Full Changelog: 2.17.1...2.17.2

2.17.1

18 Apr 15:46
Compare
Choose a tag to compare

What's new?

  • Add ignore_merges option to BPE tokenizers in #716

Full Changelog: 2.17.0...2.17.1

2.17.0

11 Apr 00:08
Compare
Choose a tag to compare

What's new?

💬 Improved text-generation pipeline for conversational models

This version adds support for passing an array of chat messages (with "role" and "content" properties) to the text-generation pipeline (PR). Check out the list of supported models here.

Example: Chat with Xenova/Qwen1.5-0.5B-Chat.

import { pipeline } from '@xenova/transformers';

// Create text-generation pipeline
const generator = await pipeline('text-generation', 'Xenova/Qwen1.5-0.5B-Chat');

// Define the list of messages
const messages = [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'Tell me a funny joke.' }
]

// Generate text
const output = await generator(messages, {
    max_new_tokens: 128,
    do_sample: false,
})
console.log(output[0].generated_text);
// [
//   { role: 'system', content: 'You are a helpful assistant.' },
//   { role: 'user', content: 'Tell me a funny joke.' },
//   { role: 'assistant', content: "Sure, here's one:\n\nWhy was the math book sad?\n\nBecause it had too many problems.\n\nI hope you found that joke amusing! Do you have any other questions or topics you'd like to discuss?" },
// ]

We also added the return_full_text parameter, which means if you set return_full_text=false, only the newly-generated tokens will be returned (only applicable if passing the raw text prompt to the pipeline).

🔢 Binary embedding quantization support

Transformers.js v2.17 adds two new parameters to the feature-extraction pipeline ("quantize" and "precision"), enabling you to generate binary embeddings. These can be used with certain embedding models to shrink the size of the document embeddings for retrieval. This results in reductions in index size/memory usage (for storage) and improvements in retrieval speed. Surprisingly, you can still achieve up to ~95% of the original performance, but at 32x storage savings and up to 32x retrieval speeds! 🤯 Thanks to @jonathanpv for this addition in #691!

import { pipeline } from '@xenova/transformers';

// Create feature-extraction pipeline
const extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');

// Compute binary embeddings
const output = await extractor('This is a simple test.', { pooling: 'mean', quantize: true, precision: 'binary' });
// Tensor {
//   type: 'int8',
//   data: Int8Array [49, 108, 24, ...],
//   dims: [1, 48]
// }

As you can see, this produces a 32x smaller output tensor (a 4x reduction in data type with Float32Array → Int8Array, as well as an 8x reduction in dimensionality from 384 → 48). For more information, check out this PR in sentence-transformers, which inspired this update!

🛠️ Misc. improvements

🤗 New contributors

Full Changelog: 2.16.1...2.17.0

2.16.1

20 Mar 15:11
Compare
Choose a tag to compare

What's new?

  • Add support for the image-feature-extraction pipeline in #650.

    Example: Perform image feature extraction with Xenova/vit-base-patch16-224-in21k.

    const image_feature_extractor = await pipeline('image-feature-extraction', 'Xenova/vit-base-patch16-224-in21k');
    const url = 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png';
    const features = await image_feature_extractor(url);
    // Tensor {
    //   dims: [ 1, 197, 768 ],
    //   type: 'float32',
    //   data: Float32Array(151296) [ ... ],
    //   size: 151296
    // }

    Example: Compute image embeddings with Xenova/clip-vit-base-patch32.

    const image_feature_extractor = await pipeline('image-feature-extraction', 'Xenova/clip-vit-base-patch32');
    const url = 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png';
    const features = await image_feature_extractor(url);
    // Tensor {
    //   dims: [ 1, 512 ],
    //   type: 'float32',
    //   data: Float32Array(512) [ ... ],
    //   size: 512
    // }
  • Fix channel format when padding non-square images for certain models in #655. This means you can now perform super-resolution for non-square images with APISR models:

    Example: Upscale an image with Xenova/4x_APISR_GRL_GAN_generator-onnx.

    import { pipeline } from '@xenova/transformers';
    
    // Create image-to-image pipeline
    const upscaler = await pipeline('image-to-image', 'Xenova/4x_APISR_GRL_GAN_generator-onnx', {
        quantized: false,
    });
    
    // Upscale an image
    const url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/anime.png';
    const output = await upscaler(url);
    // RawImage {
    //   data: Uint8Array(16588800) [ ... ],
    //   width: 2560,
    //   height: 1920,
    //   channels: 3
    // }
    
    // (Optional) Save the upscaled image
    output.save('upscaled.png');
    See example output

    Input image:
    image

    Output image:
    image

  • Update tokenizer apply_chat_template functionality in #647. This PR added functionality to support the new C4AI Command-R tokenizer.

    See example tool usage
    import { AutoTokenizer } from "@xenova/transformers";
    
    const tokenizer = await AutoTokenizer.from_pretrained("Xenova/c4ai-command-r-v01-tokenizer")
    
    // define conversation input:
    const conversation = [
      { role: "user", content: "Whats the biggest penguin in the world?" }
    ]
    // Define tools available for the model to use:
    const tools = [
      {
        name: "internet_search",
        description: "Returns a list of relevant document snippets for a textual query retrieved from the internet",
        parameter_definitions: {
          query: {
            description: "Query to search the internet with",
            type: "str",
            required: true
          }
        }
      },
      {
        name: "directly_answer",
        description: "Calls a standard (un-augmented) AI chatbot to generate a response given the conversation history",
        parameter_definitions: {}
      }
    ]
    
    
    // render the tool use prompt as a string:
    const tool_use_prompt = tokenizer.apply_chat_template(
      conversation,
      {
        chat_template: "tool_use",
        tokenize: false,
        add_generation_prompt: true,
        tools,
      }
    )
    console.log(tool_use_prompt)
    See example RAG usage
    import { AutoTokenizer } from "@xenova/transformers";
    
    const tokenizer = await AutoTokenizer.from_pretrained("Xenova/c4ai-command-r-v01-tokenizer")
    
    // define conversation input:
    const conversation = [
      { role: "user", content: "Whats the biggest penguin in the world?" }
    ]
    // define documents to ground on:
    const documents = [
      { title: "Tall penguins", text: "Emperor penguins are the tallest growing up to 122 cm in height." },
      { title: "Penguin habitats", text: "Emperor penguins only live in Antarctica." }
    ]
    
    // render the RAG prompt as a string:
    const grounded_generation_prompt = tokenizer.apply_chat_template(
      conversation,
      {
        chat_template: "rag",
        tokenize: false,
        add_generation_prompt: true,
    
        documents,
        citation_mode: "accurate", // or "fast"
      }
    )
    console.log(grounded_generation_prompt);
  • Add support for EfficientNet in #639.

    Example: Classify images with chriamue/bird-species-classifier

    import { pipeline } from '@xenova/transformers';
    
    // Create image classification pipeline
    const classifier = await pipeline('image-classification', 'chriamue/bird-species-classifier', {
        quantized: false,      // Quantized model doesn't work
        revision: 'refs/pr/1', // Needed until the model author merges the PR
    });
    
    // Classify an image
    const url = 'https://upload.wikimedia.org/wikipedia/commons/7/73/Short_tailed_Albatross1.jpg';
    const output = await classifier(url);
    console.log(output)
    // [{ label: 'ALBATROSS', score: 0.9999023079872131 }]

Full Changelog: 2.16.0...2.16.1

2.16.0

07 Mar 14:52
Compare
Choose a tag to compare

What's new?

💬 StableLM text-generation models

This version adds support for the StableLM family of text-generation models (up to 1.6B params), developed by Stability AI. Huge thanks to @D4ve-R for this contribution in #616! See here for the full list of supported models.

Example: Text generation with Xenova/stablelm-2-zephyr-1_6b.

import { pipeline } from '@xenova/transformers';

// Create text generation pipeline
const generator = await pipeline('text-generation', 'Xenova/stablelm-2-zephyr-1_6b');

// Define the prompt and list of messages
const prompt = "Tell me a funny joke."
const messages = [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": prompt },
]

// Apply chat template
const inputs = generator.tokenizer.apply_chat_template(messages, {
    tokenize: false,
    add_generation_prompt: true,
});

// Generate text
const output = await generator(inputs, { max_new_tokens: 20 });
console.log(output[0].generated_text);
// "<|system|>\nYou are a helpful assistant.\n<|user|>\nTell me a funny joke.\n<|assistant|>\nHere's a joke for you:\n\nWhy don't scientists trust atoms?\n\nBecause they make up everything!"

Note: these models may be too large to run in your browser at the moment, so for now, we recommend using them in Node.js. Stay tuned for updates on this!

🔉 Speaker verification and diarization models

Example: Speaker verification w/ Xenova/wavlm-base-plus-sv.

import { AutoProcessor, AutoModel, read_audio, cos_sim } from '@xenova/transformers';

// Load processor and model
const processor = await AutoProcessor.from_pretrained('Xenova/wavlm-base-plus-sv');
const model = await AutoModel.from_pretrained('Xenova/wavlm-base-plus-sv');

// Helper function to compute speaker embedding from audio URL
async function compute_embedding(url) {
    const audio = await read_audio(url, 16000);
    const inputs = await processor(audio);
    const { embeddings } = await model(inputs);
    return embeddings.data;
}

// Generate speaker embeddings
const BASE_URL = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/sv_speaker';
const speaker_1_1 = await compute_embedding(`${BASE_URL}-1_1.wav`);
const speaker_1_2 = await compute_embedding(`${BASE_URL}-1_2.wav`);
const speaker_2_1 = await compute_embedding(`${BASE_URL}-2_1.wav`);
const speaker_2_2 = await compute_embedding(`${BASE_URL}-2_2.wav`);

// Compute similarity scores
console.log(cos_sim(speaker_1_1, speaker_1_2)); // 0.959439158881247 (Both are speaker 1)
console.log(cos_sim(speaker_1_2, speaker_2_1)); // 0.618130172602329 (Different speakers)
console.log(cos_sim(speaker_2_1, speaker_2_2)); // 0.962999814169370 (Both are speaker 2)

Example: Perform speaker diarization with Xenova/wavlm-base-plus-sd.

import { AutoProcessor, AutoModelForAudioFrameClassification, read_audio } from '@xenova/transformers';

// Read and preprocess audio
const processor = await AutoProcessor.from_pretrained('Xenova/wavlm-base-plus-sd');
const url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav';
const audio = await read_audio(url, 16000);
const inputs = await processor(audio);

// Run model with inputs
const model = await AutoModelForAudioFrameClassification.from_pretrained('Xenova/wavlm-base-plus-sd');
const { logits } = await model(inputs);
// {
//   logits: Tensor {
//     dims: [ 1, 549, 2 ],  // [batch_size, num_frames, num_speakers]
//     type: 'float32',
//     data: Float32Array(1098) [-3.5301010608673096, ...],
//     size: 1098
//   }
// }

const labels = logits[0].sigmoid().tolist().map(
    frames => frames.map(speaker => speaker > 0.5 ? 1 : 0)
);
console.log(labels); // labels is a one-hot array of shape (num_frames, num_speakers)
// [
//     [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0],
//     [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0],
//     [0, 0], [0, 1], [0, 1], [0, 1], [0, 1], [0, 1],
//     ...
// ]

These additions were made possible thanks to the following PRs:

  • Add support for WavLMForXVector by @D4ve-R in #603
  • Add support for WavLMForAudioFrameClassification and Wav2Vec2ForAudioFrameClassification by @D4ve-R in #611
  • Add support for UniSpeech and UniSpeechSat models in #624

📝 Improved chat templating operation coverage

With this release, we're pleased to announce that Transformers.js is now able to parse every single valid chat template that is currently on the Hugging Face Hub! 🤯 As of 2024/03/05, this is around ~12k conversational models (of which there were ~250 unique templates). Of course, future models may introduce more complex chat templates, and we'll continue to add support for them!

For example, transformers.js can now generate the prompt for highly complex function-calling models (e.g., fireworks-ai/firefunction-v1):

See code
import { AutoTokenizer } from '@xenova/transformers';

const tokenizer = await AutoTokenizer.from_pretrained('fireworks-ai/firefunction-v1')

const function_spec = [
    {
        name: 'get_stock_price',
        description: 'Get the current stock price',
        parameters: {
            type: 'object',
            properties: {
                symbol: {
                    type: 'string',
                    description: 'The stock symbol, e.g. AAPL, GOOG'
                }
            },
            required: ['symbol']
        }
    },
    {
        name: 'check_word_anagram',
        description: 'Check if two words are anagrams of each other',
        parameters: {
            type: 'object',
            properties: {
                word1: {
                    type: 'string',
                    description: 'The first word'
                },
                word2: {
                    type: 'string',
                    description: 'The second word'
                }
            },
            required: ['word1', 'word2']
        }
    }
]

const messages = [
    { role: 'functions', content: JSON.stringify(function_spec, null, 4) },
    { role: 'system', content: 'You are a helpful assistant with access to functions. Use them if required.' },
    { role: 'user', content: 'Hi, can you tell me the current stock price of AAPL?' }
]

const inputs = tokenizer.apply_chat_template(messages, { tokenize: false });
console.log(inputs);
// <s>SYSTEM: You are a helpful assistant ...

🎨 New example applications and demos

🛠️ Misc. improvements

  • Add support for the starcoder2 architecture in #622. Note: we haven't yet added transformers.js-compatible versions of the 3B and 7B models.
  • Check for existence of onnx_env.wasm before updating wasmPaths in #621

🤗 New contributors

Full Changelog: 2.15.1...2.16.0

2.15.1

21 Feb 14:47
Compare
Choose a tag to compare

What's new?

  • Add Background Removal demo in #576 (online demo).

    background-removal

  • Add support for owlv2 models in #579

    Example: Zero-shot object detection w/ Xenova/owlv2-base-patch16-ensemble.

    import { pipeline } from '@xenova/transformers';
    
    const detector = await pipeline('zero-shot-object-detection', 'Xenova/owlv2-base-patch16-ensemble');
    
    const url = 'http://images.cocodataset.org/val2017/000000039769.jpg';
    const candidate_labels = ['a photo of a cat', 'a photo of a dog'];
    const output = await detector(url, candidate_labels);
    console.log(output);
    // [
    //   { score: 0.7400985360145569, label: 'a photo of a cat', box: { xmin: 0, ymin: 50, xmax: 323, ymax: 485 } },
    //   { score: 0.6315087080001831, label: 'a photo of a cat', box: { xmin: 333, ymin: 23, xmax: 658, ymax: 378 } }
    // ]

    image

  • Add support for Adaptive Retrieval w/ Matryoshka Embeddings (nomic-ai/nomic-embed-text-v1.5) in #587 and #588 (online demo).

    adaptive-retrieval

  • Add support for Gemma Tokenizer in #597 and #598

Full Changelog: 2.15.0...2.15.1

2.15.0

06 Feb 14:03
Compare
Choose a tag to compare

What's new?

🤖 Qwen1.5 Chat models (0.5B and 1.8B)

Yesterday, the Qwen team (Alibaba Group) released the Qwen1.5 series of chat models. As part of the release, they published several sub-2B-parameter models, including Qwen/Qwen1.5-0.5B-Chat and Qwen/Qwen1.5-1.8B-Chat, which both demonstrate strong performance despite their small sizes. The best part? They can run in the browser with Transformers.js (PR)! 🚀 See here for the full list of supported models.

demo-2x

Example: Text generation with Xenova/Qwen1.5-0.5B-Chat.

import { pipeline } from '@xenova/transformers';

// Create text-generation pipeline
const generator = await pipeline('text-generation', 'Xenova/Qwen1.5-0.5B-Chat');

// Define the prompt and list of messages
const prompt = "Give me a short introduction to large language model."
const messages = [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": prompt }
]

// Apply chat template
const text = generator.tokenizer.apply_chat_template(messages, {
    tokenize: false,
    add_generation_prompt: true,
});

// Generate text
const output = await generator(text, {
    max_new_tokens: 128,
    do_sample: false,
});
console.log(output[0].generated_text);
// 'A large language model is a type of artificial intelligence system that can generate text based on the input provided by users, such as books, articles, or websites. It uses advanced algorithms and techniques to learn from vast amounts of data and improve its performance over time through machine learning and natural language processing (NLP). Large language models have become increasingly popular in recent years due to their ability to handle complex tasks such as generating human-like text quickly and accurately. They have also been used in various fields such as customer service chatbots, virtual assistants, and search engines for information retrieval purposes.'

🧍 MODNet for Portrait Image Matting

Next, we added support for MODNet, a small (but powerful) portrait image matting model (PR). Thanks to @cyio for the suggestion!

animation

Example: Perform portrait image matting with Xenova/modnet.

import { AutoModel, AutoProcessor, RawImage } from '@xenova/transformers';

// Load model and processor
const model = await AutoModel.from_pretrained('Xenova/modnet', { quantized: false });
const processor = await AutoProcessor.from_pretrained('Xenova/modnet');

// Load image from URL
const url = 'https://images.pexels.com/photos/5965592/pexels-photo-5965592.jpeg?auto=compress&cs=tinysrgb&w=1024';
const image = await RawImage.fromURL(url);

// Pre-process image
const { pixel_values } = await processor(image);

// Predict alpha matte
const { output } = await model({ input: pixel_values });

// Save output mask
const mask = await RawImage.fromTensor(output[0].mul(255).to('uint8')).resize(image.width, image.height);
mask.save('mask.png');
Input image Output mask
image/png image/png

🧠 New text embedding models

We also added support for several new text embedding models, including:

Check out the links for example usage.

🛠️ Other improvements

  • Fix example links in documentation (#550).
  • Improve unknown model warnings (#554).
  • Update jsdoc-to-markdown dev dependency (#574).

Full Changelog: 2.14.2...2.15.0

2.14.2

29 Jan 12:54
Compare
Choose a tag to compare

What's new?

Full Changelog: 2.14.1...2.14.2

2.14.1

25 Jan 14:16
Compare
Choose a tag to compare

What's new?

  • Add support for Depth Anything (#534). See here for the list of available models.

    Example: Depth estimation with Xenova/depth-anything-small-hf.

    import { pipeline } from '@xenova/transformers';
    
    // Create depth-estimation pipeline
    const depth_estimator = await pipeline('depth-estimation', 'Xenova/depth-anything-small-hf');
    
    // Predict depth map for the given image
    const url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/bread_small.png';
    const output = await depth_estimator(url);
    // {
    //   predicted_depth: Tensor {
    //     dims: [350, 518],
    //     type: 'float32',
    //     data: Float32Array(181300) [...],
    //     size: 181300
    //   },
    //   depth: RawImage {
    //     data: Uint8Array(271360) [...],
    //     width: 640,
    //     height: 424,
    //     channels: 1
    //   }
    // }

    You can visualize the output with:

    output.depth.save('depth.png');
    Input image Visualized output
    image image

    Online demo: https://huggingface.co/spaces/Xenova/depth-anything-web

    Example video:

    depth-anything-demo-final.mp4
  • Fix typo in tokenizers.js (#518)

  • Return empty tokens array if text is empty after normalization (#535)

Full Changelog: 2.14.0...2.14.1

2.14.0

10 Jan 17:50
Compare
Choose a tag to compare

What's new?

🚀 Segment Anything Model (SAM)

The Segment Anything Model (SAM) can be used to generate segmentation masks for objects in a scene, given an input image and input points. See here for the full list of pre-converted models. Support for this model was added in #510.

demo

Demo + source code: https://huggingface.co/spaces/Xenova/segment-anything-web

Example: Perform mask generation w/ Xenova/slimsam-77-uniform.

import { SamModel, AutoProcessor, RawImage } from '@xenova/transformers';

const model = await SamModel.from_pretrained('Xenova/slimsam-77-uniform');
const processor = await AutoProcessor.from_pretrained('Xenova/slimsam-77-uniform');

const img_url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/corgi.jpg';
const raw_image = await RawImage.read(img_url);
const input_points = [[[340, 250]]] // 2D localization of a window

const inputs = await processor(raw_image, input_points);
const outputs = await model(inputs);

const masks = await processor.post_process_masks(outputs.pred_masks, inputs.original_sizes, inputs.reshaped_input_sizes);
console.log(masks);
// [
//   Tensor {
//     dims: [ 1, 3, 410, 614 ],
//     type: 'bool',
//     data: Uint8Array(755220) [ ... ],
//     size: 755220
//   }
// ]
const scores = outputs.iou_scores;
console.log(scores);
// Tensor {
//   dims: [ 1, 1, 3 ],
//   type: 'float32',
//   data: Float32Array(3) [
//     0.8350210189819336,
//     0.9786665439605713,
//     0.8379436731338501
//   ],
//   size: 3
// }

You can then visualize the 3 predicted masks with:

const image = RawImage.fromTensor(masks[0][0].mul(255));
image.save('mask.png');
Input image Visualized output
corgi mask

Next, select the channel with the highest IoU score, which in this case is the second (green) channel. Intersecting this with the original image gives us an isolated version of the subject:

Selected Mask Intersected
mask corgi-masked

🛠️ Improvements

  • Add support for processing non-square images w/ ConvNextFeatureExtractor in #503
  • Encode revision in remote URL by #507

Full Changelog: 2.13.4...2.14.0