New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Features: Batching and graphile-worker upgrade to 0.16.0-rc.1
#524
base: main
Are you sure you want to change the base?
Conversation
🦋 Changeset detectedLatest commit: f130fbd The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
@@ -51,10 +60,11 @@ const AddJobResultsSchema = z.array(GraphileJobSchema); | |||
|
|||
export type ZodTasks<TConsumerSchema extends MessageCatalogSchema> = { | |||
[K in keyof TConsumerSchema]: { | |||
queueName?: string | ((payload: z.infer<TConsumerSchema[K]>) => string); | |||
queueName?: string | ((payload: z.infer<TConsumerSchema[K]>, jobKey?: string) => string); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This helps decrease payload size for batch jobs
})), | ||
runs: job.runs.map((r) => { | ||
// For compatibility with old Job Runs where payload is only available on the related Event Record | ||
const payload = r.payload !== null ? JSON.parse(r.payload) : r.event.payload; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll be able to simplify this in a lot of places once we migrate existing data and make JobRun.payload non-optional.
apps/webapp/app/services/sources/webhookDeliveryBatcher.server.ts
Outdated
Show resolved
Hide resolved
"events.invokeDispatcher": { | ||
priority: 0, // smaller number = higher priority | ||
maxAttempts: 6, | ||
handler: async (payload, job) => { | ||
const service = new InvokeDispatcherService(); | ||
|
||
await service.call(payload.id, payload.eventRecordId); | ||
await service.call(payload.id, [payload.eventRecordId]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Takes an array now, but only batches if the dispatcher has batching enabled.
id, | ||
name: execution.event.name, | ||
context: parsedContext[i], | ||
timestamp: execution.event.timestamp, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cheating here - timestamp comes from the first event. Timestamps would either be another thing to attach to JobRun, get for each execution, or remove.
Alternatively, we could create many-to-many relations between JobRun and EventRecord. This would be a big change.
Closes #396
Closes #348
✅ Checklist
Testing
Migration
docker-compose.yml
Batching
trigger.dev/references/job-catalog/src/events.ts
Lines 156 to 211 in 2afa199
Changelog
v0.16.0-rc.1
add_batch_job
functionDATABASE_READ_REPLICA_URL
env varScreenshots
💯