Dashboard solution - Trouble with grafana update speed

We are gathering data through I2C protocol from the sensors at 10hz and storing it on a PostgreSQL database on the raspberry pi provided with each machine, visualization is supposed to happen on a Grafana dashboard hosted locally on the pi and visualized on portable devices through wifi. Unfortunately, I can’t seem to surpass the 1hz refresh mark, there’s also trouble in visualizing the dashboard from my smartphone. I am not sure if it is right to persevere with Grafana as we should probably build some kind of realtime visualization tool.

I once believed Grafana was the best tool for the job as we could’ve sent alerts to doctors through a variety of sources, and that has to stay (push alerts are, by the way redundant, as there are actual hardware beepers already in place in the electronics to be released soon). .

Here’s a work in progress image of the dashboard. These metrics Volume, Pressure, and Flowrate are the ones that are usually shown on ICU monitors, last 30s are shown in the screenshot.

This is just a work in progress, we will also need other metrics, some of them derived from the ones above, some other more technical and related to machine functioning. General patient data (name, age, time in ICU) must be shown as well, there is going to be a bunch of ventilators so doctors need to know which one’s which. There’s a need to provide a dialog box to set some thresholds for alerts, maybe a wizard to detect Plateau pressure,

I’m now working on a wireframe mockup for the interface as well, I’ll be clearer in the next hours.

NOTICE: the data points shown in the image above were generated through a python script.

EDIT: tons of edits

Hi Simone,

I will forward this to some Grafana experts in our group. In the meantime, I am anticipating what they will ask: it would be good for them to be able to reproduce your problem.

  • What is the hardware setup? Exact model of RPi, Linux distro, mobile device: model, OS, version.
  • Could you push your code and config files, both for the RPi and for the mobile platform to the git repo, maybe even providing a dummy function to emulate the delay in the response from the sensors and supply dummy data so people can reproduce your issue?
  • Could you be more specific about the problem? Is it about throughput (1 data point per second) or about refresh rate (as perceived visually), or both?

Thanks,

Javier

Thanks Javier,

I will go through your points:

  • I am using a Raspberry Pi 3 Model B V 1.2, Raspbian GNU/Linux 10 (buster) mobile device is a motoG 7 with Android 9

  • I am about to push everything you need including the dummy python script that generates data points and inserts them in the PostgreSQL, in the next hour or so.

  • Throughput is great, we get 10 data points per seconds at each refresh, but I can’t seem to visualize anything on any mobile browser, even in kiosk mode.

Thanks again,
Simone

EDIT: messed up last point, nothing really crashed

Hi Simone,

As Javier already mentioned, we have other colleagues who are experts in this field.

Still, just a couple of suggestions, if you need to go a slightly lower level of development to achieve your requirements, these are three Python-based tools that can do real-time data visualisation (and you can always serve them through Python of course):

The first one (Bokeh) I haven’t tried but it might be very close to what you actually need.

If you need coders, I happily volunteer.

Cheers,
Dimitris

Thanks Dimitris,
both me and Francesco already thought about matplotlib and bokeh as a last resort solution. The thing is that Grafana is very neat. The layout received a go ahead from our MD consultant and the built-in alerts work amazingly good. There’s also some plug-ins that could really help visualization under critical conditions, usability is key and we can work on the run with Grafana just by drag and dropping. This stuff has to be used by tired professionals who work 12+ hours shift and are generally trained to use very specific products.

I don’t want to lose time by overcomitting to this software so let’s hear what the experts have to say.

UPDATE. doctor says 1hz refresh rate is acceptable, as long as we sample faster.

I’ve pushed some simulation code and the .json Grafana setup

Hi,

I think that in time-critical application we had to follow a ‘minimalistic’ approach to achieve desired performance goals, thus evaluating some lower level libraries for UI in production and using Grafana only for rapid prototyping. But since this project should be completed quickly and you would use Grafana in production, we had to look for possible optimizations in the whole process.

I made some thought about possible bottlenecks:

  • data preparation (business logic)
  • data transfer (web server & communication channel)
  • data visualization (UI)

By googling (‘grafana performance tuning’) I found some references on Grafana optimization, such as

https://www.neteye-blog.com/2017/02/how-to-tune-your-grafana-dashboards/

The subject seems popular, so I suggest to spend some time on it (and maybe using such techniques would lower time and bandwidth).

In general, it would be essential to know whether delays are due to data transfer or UI drawing (eventually to DB query performances) by making some data analysis: I will tought about it.

Best wishes
Cinzia

1 Like

Hi, I forwarded the thread to a Grafana expert at CERN. He says that, if 10Hz refresh rate is the goal, Grafana is not an appropriate tool. He says Grafana reloads the page from the Web server and rebuilds the whole graph at each refresh.

Hi everyone,

Cinzia, welcome to the project.

I just checked the page from Chrome Developer tools, although database must definitely be optimized I don’t see it being a bottleneck.

image

As you are saying the problem lies in the fact that Grafana tries to visualize 7+KB of data at each refresh. I will check out the workaround described in this article, if that does not work, we will have to visualize through lower level libraries, either python or javascript:

Thank you,
Simone

Hi Simone,

have you enable HTTP compression in Grafana configuration ?

Reference: https://grafana.com/docs/grafana/latest/installation/configuration/

[…] ### enable_gzip
Set this option to true to enable HTTP compression, this can improve transfer speed and bandwidth utilization. It is recommended that most users set it to true . By default it is set to false for compatibility reasons. […]

Maybe it can hep…
Cinzia

Thanks Cinzia,

Does not seem to be working. I will start trying either the InfluxDB workaround or Loki, which is a plug-in for Grafana.

I think we should start making a contingency plan with some low level libraries as you were saying, at this point we should weight pros and cons of either a Javascript approach or a Django. Note that security of data and system stability is key, what is your opinion about this?

Thanks

Hi,

at the moment I would suggest to test canvasj dynamic charts:


I would use a bottom-up approach, at first by making a rough dynamic diagram that would meet basic speed requirements and then proceed with optimizations and aesthetic requirements, i.e:

  • Query optimization
  • High speed data output (binary data)
  • Grafana UI look-and-feel

How many data per second you have to show ?
Cinzia

Hi Simone,

I prepared a (very basic!) proof of concept of a canvasjs-based UI with three area charts plotting inline phantom data:

https://jsfiddle.net/clicciardello/rbcfxdn7/

It seems that charts refresh on mobile devices is as fluid as on PC.

Tomorrow I will add a dynamic JSON example relying on canvasjs demo services (had to fix chart syncing).

Bye.
Cinzia

Hello everyone, I’ve just pushed a preliminary UI block diagram, it has to be verified by the electronics/automation team and receive a go ahead from the doctors, so It is not definitive at all. I will now work with the doctors on screens for the dashboard, the setting pages and the wizard. We also need to define an alarm hierarchy by taking a look at the relevant standards.

Interface-01-01-01-01

We might be able to use Grafana as a visualization tool by modifying some plugins and using InfluxDB. I’ll keep you posted.