Grafana 1.7.1 and Elasticsearch 7.9.0 Models and Job button not working

[fadjar340]

Dear @vsergeyev ,

I just run grafana-loudml-app in grafana 1.7.1 and using Elasticsearch 7.9.0.
The loudml server is on another server using same subnet, the installation of the grafana-loudml-app is very easy but I have something weird that the button on Add Model and Schedule Job not working.

I can create the temp-model using curl from grafana server to loudml server and it succeded, so the temp-model created, then I tried to edit the model but no luck.
Is there any specific requirement to make it work?

I use grafana-loudml-app v 1.7.1, BTW

Regards,
Fadjar340

[fadjar340]

Also I tried using git clone,
when I ran yarn dev --watch, I got this:

And this condition likely hung but the process still running untuk I'm posting this.

Please give some clues to solve this problem...

Regards,
Fadjar340

[fadjar340]

Now I know the problems:
I found some problem when using git clone and build from source:

1. for problem above with the @babel/preset-env, I solved it with

yarn add @babel/preset-env --dev

Then the issue was fixed
2. There some code that need to rectified,
In /src/panel/GraphPanel.tsx

// @ts-nocheck
import React from 'react';
import { GraphWithLegend, Chart } from '@grafana/ui';
import { GraphWithLegend2 } from './GraphWithLegend2';
import { PanelProps } from '@grafana/data';
import { Options } from './types';
import { GraphPanelController, CreateBaselineButton, MLModelController } from './GraphPanelController';
import { LegendDisplayMode } from '@grafana/ui/src/components/Legend/Legend';

interface GraphPanelProps extends PanelProps<Options> {}

It should be like below:

// @ts-nocheck
import React from 'react';
import { GraphWithLegend, Chart } from '@grafana/ui';
import { GraphWithLegend2 } from './GraphWithLegend2';
import { PanelProps } from '@grafana/data';
import { Options } from './types';
import { GraphPanelController, CreateBaselineButton, MLModelController } from './GraphPanelController';
import { LegendDisplayMode } from '@grafana/ui';

interface GraphPanelProps extends PanelProps<Options> {}

3. In the /dist/datasource/module.js

   key: "addModel",
    value: function addModel() {
      this.model = Object.assign(Object.assign(Object.assign({}, types_1.DEFAULT_MODEL), {
        features: [Object.assign({}, types_1.DEFAULT_FEATURE)]
      }));
      app_events_1["default"].emit('show-modal', {
        src: '/public/plugins/loudml-grafana-app/datasource/partials/add_model.html',
        modalClass: 'confirm-modal',
        model: this
      });
    }
  }, {
    key: "editModel",
    value: function editModel(name) {
      var model = this.$scope.ctrl.modelsList.find(function (el) {
        return el.settings.name === name;
      });
      this.model = model.settings; // appEvents.emit(CoreEvents.showModal, {

      app_events_1["default"].emit('show-modal', {
        src: '/public/plugins/loudml-grafana-app/datasource/partials/add_model.html',
        modalClass: 'confirm-modal',
        model: this
      });
    }

Because you use static URL like

 src: '/public/plugins/loudml-grafana-app/datasource/partials/add_model.html',

Then when I use nginx as reverse proxy then the pagecan not shown because not found, and if I change the URL to the proper one, will look like below:

It will be the same as the Schedule job and other function that use static URL.

Perhaps you can fixed this in the source, and I'll try to rebuild...

Regards,
Fadjar340

[fadjar340]

Now I can show the page...
I change the code in src/datasource/config_ctrl.ts as below:

 key: "addModel",
    value: function addModel() {
      this.model = Object.assign(Object.assign(Object.assign({}, types_1.DEFAULT_MODEL), {
        features: [Object.assign({}, types_1.DEFAULT_FEATURE)]
      }));
      app_events_1["default"].emit('show-modal', {
        src: 'public/plugins/loudml-grafana-app/datasource/partials/add_model.html',
        modalClass: 'confirm-modal',
        model: this
      });
    }
  }, {
    key: "editModel",
    value: function editModel(name) {
      var model = this.$scope.ctrl.modelsList.find(function (el) {
        return el.settings.name === name;
      });
      this.model = model.settings; // appEvents.emit(CoreEvents.showModal, {

      app_events_1["default"].emit('show-modal', {
        src: 'public/plugins/loudml-grafana-app/datasource/partials/add_model.html',
        modalClass: 'confirm-modal',
        model: this
      });
    }

I remove in 4 lines and run yarn dev again.
Then viola...
The page shown now...

[fadjar340]

Next problem...
Maybe you forgot the output bucket in here:

Because in the dev console, I saw that output bucket parameter is undefined.

Or there are config about the output bucket that could be set to false?

[fadjar340]

I can train the model from loudml machine using loudml command...
then the datasource like below:

[fadjar340]

Still trying to make this work..

After I follow your youtube, I found that if elasticsearch as datasource, there is a bug about the model name, as follow

The apps read the index pattern that have * (asterisk) in the index pattern, then it creates the model name using * (asterisk) and then loudml rejected the naming of the model.
Perhaps you need to sanitize the special character from the index pattern in elasticsearch data source. Just use the value from the bucket, instead of the query result from elasticsearch datasource.
At this point, I can't go further to the code...

Regards,
Fadjar340

[fadjar340]

Ohh my...

After searching and googling also understand the code...
I have succeded to save the model..

In src/panel/GrapPanelController.tst at line 436 i change the replace command to replace all unnecesary character

  this.getDatasource(source.datasource)
      .then(result => {
        this.datasource = result;
        // TODO: find a way to pass all this.datasource connection params to Loud ML server
        // This will allow to auto create bucket to store ML Model training results

        // this.ds.loudml.createAndGetBucket(
        //   this.datasource.database,
        //   source.policy,
        //   source.measurement,
        //   this.datasource
        // ).then(result => {
        //     const bucket = result;
        const bucket = this.props.panelOptions.datasourceOptions.input_bucket;
        const output_bucket = this.props.panelOptions.datasourceOptions.output_bucket;
        const measurement = extract_model_measurement(source);
        const fill = extract_model_fill(source);
        const match_all = extract_model_tags_map(source);
        const name = [
          extract_model_database(this.datasource),
          measurement,
          extract_model_select(source, fields[0]), // will use 1st metric to name model - fields[0]
          extract_model_tags(source),
          extract_model_time_format(source),
        ]
          .join('_')
          // .replace(/\./g, '_');
          .replace(/[.*+?^${}()|[\]\\]/g, '_');

        // Group By Value – [{params: ["5m"], type: "time"}, {params: ["linear"], type: "fill"}]
        // Let parse a "5m" time from it
        const time = extract_model_time(source);
        const model = {
          ...DEFAULT_MODEL,
          max_evals: 10,
          name: name,
          interval: this.normalizeInterval(time),
          span: this.normalizeSpan(time),
          default_bucket: bucket, //bucket.name - if we will use createAndGetBucket()
          bucket_interval: time,
          features: fields.map(field => ({
            name: extract_model_select(source, field),
            measurement: measurement,
            field: extract_model_feature(source, field),
            metric: extract_model_func(source, field), // aggregator, avg/mean
            io: 'io',
            default: fill,
            match_all: match_all,
          })),
	};

Below the result

Then the model saved and start to training....

[vsergeyev]

Dear @fadjar340

You have done a lot of fixes and your comments and details are of a big value!

Output bucket field is in LoudML Panel. Please try if it set properly.
(in datasource it is old field for output bucket, it not used)

Your patches are great, please try to create a git pull request.
From my side I will be happy to include you into repository authors list.

Stay cool!

V.

[fadjar340]

Please send me the password to commit my changes...