Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MySQL -> ES, 自定义的 ES output,执行时 Execute 函数接受到 Msgs 数量为1 #254

Open
Stanley3 opened this issue Jan 4, 2020 · 1 comment

Comments

@Stanley3
Copy link

Stanley3 commented Jan 4, 2020

MySQL Input 的配置

[input]
    type = "mysql"
    mode = "batch"

[input.config]
    nr-scanner = 10
    table-scan-batch = 10000
    batch-per-second-limit = 10
    max-full-dump-count = 10000

ES Output 的配置

[output]
    type="custom-elasticsearch"

[output.config]
    ignore-bad-request = true

Scheduler 的配置

[scheduler]
    type = "batch-table-scheduler"

[scheduler.config]
    nr-worker = 10
    batch-size = 1000
    queue-size = 1024
    sliding-window-size = 10240
    nr-retries = 3
    retry-sleep = "2s"

按照这个配置,在执行到 func (output *customESOutput) Execute(msgs []*core.Msg) error 函数时,msgs 数量总是为1,导致 ES 的 bulk request 形同虚设。
请问,应该如何调整配置,解决这个问题。

@Ryan-Git
Copy link
Collaborator

Ryan-Git commented Jan 4, 2020

贴下监控?配置没看出啥问题

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants