Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn when errors occurred during batch operations #61

Open
dandv opened this issue May 25, 2023 · 0 comments
Open

Warn when errors occurred during batch operations #61

dandv opened this issue May 25, 2023 · 0 comments

Comments

@dandv
Copy link
Contributor

dandv commented May 25, 2023

I had forgotten to include the OpenAI API key in the headers, and noticed that batch imports appeared to succeed, but no objects were committed to the database.

Turned out that the response to batcher.do() included a result field with the error buried inside. I expected the batcher to throw.

import assert from 'assert';

import weaviate from 'weaviate-ts-client';
import fs from 'fs';
import csv from 'csv-parser';

const client = weaviate.client({
  scheme: 'http',
  host: 'localhost:8080',
  // no OpenAI API key
});

const classDefinition = {
  class: 'JeopardyQuestion',
  description: 'A Jeopardy! question',
  vectorizer: 'text2vec-openai',
};

try {
  await client.schema.classDeleter().withClassName('JeopardyQuestion').do();
} catch { }
finally {
  await client.schema.classCreator().withClass(classDefinition).do();
}

const qs = [
  { question: 'What color?', answer: 'Blue' },
  { question: 'Where?', answer: 'In the sky' },
]

let batcher = client.batch.objectsBatcher();

for (const q of qs)
  batcher = batcher.withObject({
    class: 'JeopardyQuestion',
    properties: q,
  });

const response = await batcher.do();
console.log(response.map(r => r.result.errors.error));  // <-- all objects OK, but nothing written to the DB
batcher = client.batch.objectsBatcher();

const result = await client.graphql.aggregate().withClassName('JeopardyQuestion').withFields('meta { count }').do();
assert.deepEqual(result.data['Aggregate']['JeopardyQuestion'], [{ meta: { count: 2 } }]);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant