Skip to content

Draft: addd partial counts to SPHN Connector Overview Dashboard

Maximilian Pickl requested to merge sphnc-291 into main

Adds "real_time_counts" to the sphn connector overview dashboard in Grafana.

  • new pipeline run_grafana_count (scheduled to run every five minutes). DAG is not activate as a default.
  • added start & stop tasks to different steps: PreCheck&De-ID, Integration, validation and statistics
  • data is saved in new table in the postgres database: grafana_real_time_report
  • after the 'main' tasks have finished for integration / validation a last update from the pipeline_history is made to extract the total amount of patients processed
  • feature not added for the landing zone Graphic since the update of the landing_zone table is faster than the periodic update of the grafana_real_time_report table

Old visual:partial_counts

New visual: The number of patients will "move" through the different zones as they progress in the Connector. image

Short summary:

  • update Airflow version to 2.9.0
  • adds new DAG: run_grafana_count
  • updates sphn connector overview Dashboard
  • adds new module grafana_counts.py
  • new table in postgres: grafana_real_time_report

Let me know what you think. @nicola.stoira @philip.krauss

Edited by Maximilian Pickl

Merge request reports