addd partial counts to SPHN Connector Overview Dashboard
Adds "real_time_counts" to the sphn connector overview dashboard in Grafana.
- new pipeline run_grafana_count (scheduled to run every five minutes). DAG is not activate as a default.
- added start & stop tasks to different steps:
PreCheck&De-ID
,Integration
,validation
andstatistics
- data is saved in new table in the postgres database:
grafana_real_time_report
- after the 'main' tasks have finished for integration / validation a last update from the
pipeline_history
is made to extract the total amount of patients processed - feature not added for the landing zone Graphic since the update of the landing_zone table is faster than the periodic update of the
grafana_real_time_report
table
New visual:
The number of patients will "move" through the different zones as they progress in the Connector.
Short summary:
- update Airflow version to
2.9.0
- adds new DAG:
run_grafana_count
- updates
sphn connector overview Dashboard
- adds new module
grafana_counts.py
- new table in postgres:
grafana_real_time_report
Let me know what you think. @nicola.stoira @philip.krauss
Edited by Maximilian Pickl