real time update mechanism for the chart

[yezhengli-Mr9]

For #641, try to figure out real time update mechanism for the chart suitable for

chart-fx/chartfx-chart/src/main/java/io/fair_acc/chartfx/renderer/spi/financial/CandleStickRenderer.java

Line 105 in ed86318

protected void render(GraphicsContext gc, DataSet ds, DataSetNode styleNode) {

(1) real time candlestick Chart with lines and points together

for example,
(1a) incremental update?
(1b) need to integrate with real-time streaming data.
(2) Do not feel these financial examples are (updated) real time since they do not update the similar to

chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/chart/Histogram2DimSample.java

Lines 138 to 144 in ed86318

	timer.scheduleAtFixedRate(new TimerTask() {
	@Override
	public void run() {
	fillData();
	chart.invalidate();
	}
	}, Histogram2DimSample.UPDATE_DELAY, Histogram2DimSample.UPDATE_PERIOD);

likely because these "real-time" financial examples are not handling streaming(-like) data but calling loadTestData(...) instead

chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/financial/AbstractBasicFinancialApplication.java

Lines 207 to 215 in ed86318

	} else {
	ohlcvDataSet = new OhlcvDataSet(resource);
	indiSet = new DefaultDataSet("MA(24)");
	try {
	loadTestData(resource, ohlcvDataSet, indiSet);
	} catch (IOException e) {
	throw new IllegalArgumentException(e.getMessage(), e);
	}
	}

Need real-time update of daily minute-level financial charts.

[yezhengli-Mr9]

By the way, to accomplish #641, I was able to conduct substantial modification on several classes relevant to

chart-fx/chartfx-dataset/src/main/java/io/fair_acc/dataset/spi/financial/OhlcvDataSet.java

Line 19 in ed86318

public class OhlcvDataSet extends AbstractDataSet<OhlcvDataSet> implements Iterable<IOhlcvItem>, IOhlcvItemAware, AttributeModelAware {

which seems enough to resolve issue (1) and hopefully can resolve (2) as well in the near future.

Understand real-time update mechanism seems one of the very few (two~three) bottlenecks remaining to fully finish #641

[raven2cz]

Class FinancialRealtimeCandlestickSample is example for realtime ohlcv stream processing with simple execution platform.

new SimpleOhlcvDailyParser().getContinuousOHLCV(data);
// look to this method
convertTsRowStream() 
// and
DefaultOHLCV

Next important classes for you for realtime processing of ohlcv data.

SimpleOhlcvReplayDataSet
SCIDByNio.createTickDataReplayStream()
// creates ohlcv tick data processing provider
TickOhlcvDataProvider
de.gsi.financial.samples.service.SimpleOhlcvReplayDataSet#fillTestData
de.gsi.financial.samples.service.SimpleOhlcvReplayDataSet#tick

About timeframe consolidation, you can look here

OhlcvTimeframeConsolidation
// there are next examples of consolidations
RangeBarsIncrementalOhlcvConsolidation
VolumeIncrementalOhlcvConsolidation

Could you provide a hand-drawn example of the chart you're aiming for? I'm having trouble grasping the specifics of your goal from the description. Thanks.

[yezhengli-Mr9]

Dear @raven2cz ,
Thanks~ First goal is simple: real-time minute level candlesticks for intra day -- but for each candlestick want to put more points (moving average, some local extreme values).

Feel "real-time“/handle with real-time streaming API is the only issue left for this first goal -- rest are resolved (I mean add extra points, etc.)
Following should help if indeed called in loadTestdata(...) and I will track:

chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/financial/service/consolidate/IncrementalOhlcvConsolidation.java

Line 7 in ed86318

public interface IncrementalOhlcvConsolidation {

chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/financial/AbstractBasicFinancialApplication.java

Lines 207 to 215 in ed86318

	} else {
	ohlcvDataSet = new OhlcvDataSet(resource);
	indiSet = new DefaultDataSet("MA(24)");
	try {
	loadTestData(resource, ohlcvDataSet, indiSet);
	} catch (IOException e) {
	throw new IllegalArgumentException(e.getMessage(), e);
	}
	}

as well as

chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/financial/service/SimpleOhlcvReplayDataSet.java

Line 202 in ed86318

protected Runnable getDataUpdateTask() {

Meanwhile, #378 may help.

[yezhengli-Mr9]

Can screenshot a current realized chart later today.

[yezhengli-Mr9]

how to put code to get chart updated while setData(...) updates data timely already?

While consuming stream API in extended version of FinancialRealtimeCandlestickSample,can definitely update data similar as

chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/financial/AbstractBasicFinancialApplication.java

Line 319 in ed86318

dataSet.setData(ohlcv);

chart-fx/chartfx-dataset/src/main/java/io/fair_acc/dataset/spi/financial/OhlcvDataSet.java

Lines 40 to 42 in ed86318

	public void setData(IOhlcv ohlcv) {
	this.ohlcv = ohlcv;
	}

in ways similar to #378

chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/chart/PaddedAutoGrowAxisSample.java

Lines 40 to 62 in ed86318

	CachedDaemonThreadFactory.getInstance().newThread(() -> {
	while (true) {
	ds.reset();
	try {
	TimeUnit.MILLISECONDS.sleep(100);
	FXUtils.runAndWait(() -> {
	yAxis.getAutoRange().set(0, 10);
	});
	TimeUnit.MILLISECONDS.sleep(100);
	} catch (Exception e) {
	e.printStackTrace();
	}
	for (int i = 0; i < 500; i++) {
	ds.add(i, 40 * Math.sin(i * 0.1) + 100 * Math.sin(i * 0.02), 0, 0);
	try {
	TimeUnit.MILLISECONDS.sleep(40);
	} catch (InterruptedException e) {
	e.printStackTrace();
	}
	}
	}
	})
	.start();

or

chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/chart/Histogram2DimSample.java

Lines 138 to 144 in ed86318

	timer.scheduleAtFixedRate(new TimerTask() {
	@Override
	public void run() {
	fillData();
	chart.invalidate();
	}
	}, Histogram2DimSample.UPDATE_DELAY, Histogram2DimSample.UPDATE_PERIOD);

chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/chart/Histogram2DimSample.java

Lines 39 to 59 in ed86318

	private void fillData() {
	counter++;
	final double angle = Math.PI / 4;
	for (int i = 0; i < Histogram2DimSample.UPDATE_N_SAMPLES; i++) {
	final double x0 = rnd.nextGaussian() * 0.5 + 5.0;
	final double y0 = rnd.nextGaussian() * 0.5 + 5.0;
	final double x = rnd.nextGaussian() * 1.5;
	final double y = rnd.nextGaussian() * 0.5;
	final double x1 = x * Math.sin(angle) + y * Math.cos(angle);
	final double x2 = x * Math.cos(angle) - y * Math.sin(angle);
	histogram1.fill(x0, y0);
	histogram2.fill(x1 + 14.0, x2 + 20.0);
	}
	if (counter % 500 == 0) {
	// reset distribution every now and then
	counter = 0;
	histogram1.reset();
	histogram2.reset();
	}
	}

However, do not know how to put code (likely between certain lines in the following) to get chart updated? While tried to add following between lines:

           CachedDaemonThreadFactory.getInstance().newThread(() -> {
            try {
                getStreamChartDataSetUpdate(symbolSignal, 1, null, 1, "Minute", true,
                        ohlcvDataSet,indiSet,
                        title, chart);
            }catch(IOException | ParseException e){
                e.printStackTrace();
            }
        }).start();
/**
  * getStreamChartDataSetUpdate(...)
  */
    // ...
             dataSet.setData(ohlcvOutput);// with new data
    //       Renderer renderer = new CandleStickWithMALocalMaxMinRenderer(true);
    //       renderer.getDatasets().addAll(ds);
    //       chart.getRenderers().clear();
    //       chart.getRenderers().add(renderer);

chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/financial/FinancialRealtimeCandlestickSample.java

Lines 84 to 152 in ed86318

	final Chart chart = getDefaultFinancialTestChart(theme);
	final AbstractFinancialRenderer<?> renderer = (AbstractFinancialRenderer<?>) chart.getRenderers().get(0);
	chart.setTitle(title);
	// prepare top financial toolbar with replay support
	ToolBar testVariableToolBar = getTestToolBar(chart, renderer, true);
	// prepare financial y-value indicator
	Axis yAxis = chart.getAxes().get(1);
	if (ohlcvDataSet instanceof SimpleOhlcvReplayDataSet) {
	SimpleOhlcvReplayDataSet replayDataSet = ((SimpleOhlcvReplayDataSet) ohlcvDataSet);
	// close prices visualization
	final YWatchValueIndicator closeIndicator = new YWatchValueIndicator(yAxis);
	closeIndicator.setPreventOcclusion(true);
	closeIndicator.setId("price");
	closeIndicator.setLineVisible(false);
	closeIndicator.setEditable(false);
	chart.getPlugins().add(closeIndicator);
	replayDataSet.addOhlcvChangeListener(ohlcvItem -> FXUtils.runFX(() -> closeIndicator.setMarkerValue(ohlcvItem.getClose())));
	// define context
	AttributeModel context = new AttributeModel()
	.setAttribute(ORDERS, new OrderContainer())
	.setAttribute(POSITIONS, new PositionContainer());
	// position/order visualization
	String asset = replayDataSet.getResource(); // just example, it is more complex in real platform
	PositionFinancialDataSet positionFinancialDataSet = new PositionFinancialDataSet(
	asset, ohlcvDataSet, context);
	// example of addition complex extension-point to renderer
	if (renderer instanceof RendererPaintAfterEPAware) {
	((RendererPaintAfterEPAware) renderer).addPaintAfterEp(new PositionFinancialRendererPaintAfterEP(positionFinancialDataSet, (XYChart) chart));
	}
	// execution platform (has to be last added to dataset)
	BacktestExecutionPlatform executionPlatform = new BacktestExecutionPlatform();
	executionPlatform.setContext(context);
	executionPlatform.addExecutionPlatformListener(positionFinancialDataSet); // order notification listens position dataset
	// basic handling of orders simple example (MKT only)
	BasicOrderExecutionService orderExecutionService = new BasicOrderExecutionService(context, executionPlatform);
	// create custom trade plan
	MktOrderListTradePlan tradePlan = new MktOrderListTradePlan(context, asset, orderExecutionService, orders);
	// connection OHLC/V listeners
	replayDataSet.addOhlcvChangeListener(tradePlan);
	replayDataSet.addOhlcvChangeListener(executionPlatform); // execution platform listens replay data (has to be last!)
	}
	// manual levels
	chart.getPlugins().add(new YWatchValueIndicator(yAxis, 4727.5));
	chart.getPlugins().add(new YWatchValueIndicator(yAxis, 4715.0));
	// simple S/R ranges
	chart.getPlugins().add(createRsLevel(yAxis, 4710, 4711, "Daily Support"));
	chart.getPlugins().add(createRsLevel(yAxis, 4731, 4733, "Daily Resistance"));
	// apply all changes by addons and extensions
	theme.applyPseudoClasses(chart);
	VBox root = new VBox();
	VBox.setVgrow(chart, Priority.SOMETIMES);
	root.getChildren().addAll(testVariableToolBar, chart);
	return root;
	}

[ennerf]

@yezhengli-Mr9 Can you provide more information about your application and what you are trying to achieve? That will make it easier to recommend something.

• Where is the data coming from?
• How latency sensitive is the application? Do you have specific targets?
• Are you planning on doing actual trading from that application?

You should also learn about the JavaFX threading model. Architecture.md has a small summary about the individual phases. In general you can only update the chart from the JavaFX thread. Datasets have some built-in support for concurrent updates, but for low-latency applications with small updates it may make sense to do something more custom.

[yezhengli-Mr9]

Dear @ennerf ,
thanks

• API data feed, for example, similar to TradeStation API.
  

  chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/financial/service/SimpleOhlcvDailyParser.java

  Lines 79 to 81 in ed86318

  	private interface TradeStationRowParser {
  	OHLCVItem parse(TSConvertSettings ref, Calendar cal, String r);
  	}

• just updated every one minute. For short term, just need real-time (every one minute) candlestick update.
  At most to visualize some volume delta in the near future.
  No need for any complicated stuff (no need for any button, etc.) But X-axis (time) with adaptive values are good.
• just need to visualize real-time (per minute) GUI via chart-fx. Other non-GUI issue, can solve by java backend codes.

[ennerf]

Ok, that sounds fairly simple. Does it not work if you just add a new datapoint to the dataset? Changing the data should automatically trigger a chart update.

[yezhengli-Mr9]

Thanks, yes, just simple version.

Changing the data should automatically trigger a chart update.

Ok, that is the (main) thing I am checking from sample codes -- change the data, chart gets "updated automatically".

[yezhengli-Mr9]

Try to combine @raven2cz 's hints on SimpleOhlcvReplayDataSet, de.gsi.financial.samples.service.SimpleOhlcvReplayDataSet#fillTestData and simple setData(newData), still no luck, attempts including:

1. add following

replayDataSet.addOhlcvChangeListener(ohlcvItem -> FXUtils.runFX(() -> {
                Interval<Calendar> timeRangeInt =null,ttInt=null;
                Calendar replayFromCal=null;
                try{
                    timeRangeInt = CalendarUtils.createByDateTimeInterval(timeRange);
                    ttInt = CalendarUtils.createByTimeInterval(tt);
                    replayFromCal = CalendarUtils.createByDateTime(replayFrom);
                }catch(ParseException pe){
                    pe.printStackTrace();
                }

                replayDataSet.fillTestData(period,
                        timeRangeInt,
                        ttInt, replayFromCal, consolidationAddons);
                closeIndicator.setMarkerValue(ohlcvItem.getClose());
            }));

instead of

chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/financial/FinancialRealtimeCandlestickSample.java

Line 103 in ed86318

replayDataSet.addOhlcvChangeListener(ohlcvItem -> FXUtils.runFX(() -> closeIndicator.setMarkerValue(ohlcvItem.getClose())));

2.

try {
                getStreamChartDataSetUpdate(symbolSignal, 1, null, 1, "Minute", true,
                        ohlcvDataSet,indiSet,
                        title, chart);
            }catch(IOException | ParseException e){
                e.printStackTrace();
            }

where within getStreamChartDataSetUpdate(...) there are

BufferedReader responseReader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
                while ((line = responseReader.readLine()) != null) {

                    if (TokenResponse.aboutToExpire(tokenResponse, lagSeconds)) {
                        tokenResponse = Credential.getAccessToken();
                        continue;
                    }
                    if (!line.isEmpty()) {
                        // Assuming you have a JSONObject class for parsing JSON
                        Bar b = Bar.readBarFromJsonString(line, symbol);
                        System.out.println("[getStreamBarchartQuotes]" + b);
                        if (b == null) {
                            System.out.println("barList.size()\t" + barList.size() + ":\n" + barList.get(barList.size() - 1));
                            continue;
                        }
                        if (barList.size() == 0) {
                            barList.add(b);
                            // update the chart
                        } else {
                            long diff = Util.calculateTimeDiffSecUTC(b.getTimeStamp(), barList.get(barList.size() - 1).getTimeStamp());
                            if (diff > 30) {
                                barList.add(b);
                                /*
                                 * here is part to ohlcvDataSet.setData(...) new data
                                 */
                                Thread.sleep(20000); // try to sleep 20 seconds to provide chart enough time to update/refresh
                                // update the chart
                            } else {
                                barList.set(barList.size() - 1, b);
                            }
                        }
                    }
                }
            } catch (Exception e) {
                e.printStackTrace();
                System.out.println(connection.getResponseCode());
                System.out.println(connection.getErrorStream());
                System.out.println(connection.getInputStream());

            }

Will try more later today, tomorrow or later this week.

• should try more hints from @raven2cz ;
• reduce many unnecessary part of FinancialRealtimeCandlestickSample. Only necessary part so far -- to see update chart, when adding one more point, or providing the chart new dataset by setData(...) .
  

  chart-fx/chartfx-samples/src/main/java/io/fair_acc/sample/financial/FinancialRealtimeCandlestickSample.java

  Line 49 in ed86318

  public class FinancialRealtimeCandlestickSample extends AbstractBasicFinancialApplication {

• Feel have to better understsand Architecture.md and https://www.javatpoint.com/javafx-stacked-bar-chart, https://docs.oracle.com/javafx/2/charts/chart-overview.htm#CJAHHJCB

[ennerf]

ChartFX originally started off as an extension of the built-in Oracle charts, but over time it has become quite different. The content gets redrawn when a flag indicates that the corresponding parts are dirty.

It looks like OhlcvDataSet::setData does not do update notifications to trigger a redraw. You can manually trigger one using dataSet.fireInvalidated(ChartBits.DataSetData); after setting the data.

[yezhengli-Mr9]

Thanks very much, I noticed fireInvalidated(ChartBits.DataSetDataAdded); or fireInvalidated(ChartBits.DataSetData); earlier. Let me debug this part with more attention.

[ennerf]

The docs contain a section on the event system

[yezhengli-Mr9]

Resolved.