Real time Grafana count DAG scheduled every minute instead of every 5 minutes. That might be too much you tell me.
We restructure the Grafana dashboard as in the screenshots below.
We have new panels that shows how many patients are stored in the different zones. That's just counting on the corresponding database tables. The user can have an idea on where the patients are and how many they are.
In the second part we have the pipelines history and then the real time count of processed patients. The difference compared to the previous version is that these panels only show the count of patients which are currently being processed so that the user can have an overview of the percentage of the process.
Before and after every step we reset the count of the patients processed in the real time Grafana table. We can therefore remove the headache of trying to reduce the number if some patients are added or removed as the count is only related to patients being processed at that time.
We use the same logic for the pre-check/de-id step as well. That will be slower in terms of reporting but it will be consistent with the other steps monitoring. Moreover this step is usually quite fast and therefore monitoring it in real time might not be really necessary.
The endpoint /Get_status is still working as before, therefore it is always possible to extract an "instant" result about the processing status.
