Automated research data pipeline development
Umbizo built an automated pipeline to transfer and analyse electronic health record data securely, replacing manual processes that raised error risk and weakened reproducibility.
The challenge
Healthcare researchers struggle to transfer and analyse EHR data securely. Manual processes raise the risk of errors and weaken reproducibility, slowing research and making outputs harder to trust.
What we set out to do
- Automate data transfer from EHR to electronic data capture (EDC)
- Meet HIPAA data security requirements
- Build reproducible, adaptable workflows
- Integrate public datasets such as the MIMIC critical care database and UK Biobank
- Standardise and automate reporting
Our approach
Pipeline architecture
Built API-based data extraction and processing, automated cleaning protocols, integrated statistical analysis and modelling, and generated reports using R Markdown and Jupyter.
Technical implementation
Developed an EHR data lake architecture with security protocols, automated quantitative analysis, added support for machine learning, MLP, and LLM methods, and provided plug-in support for public dataset tools.
Workflow optimisation
Implemented error flagging, audit trails, a reproducibility framework, and automated documentation.
Results and impact
The pipeline delivered a 90% reduction in time spent on manual processing, 100% reproducible analyses, zero security breaches, and 75% time savings in reporting. It also provided complete audit trails, stronger data security, automated quality control, and standardised outputs.
“The team's technical ability and communication were excellent, and the automated pipeline has revolutionised our research workflow.”
Timeline
- Iterative pipeline development: 12 weeks
- Data integration and beta testing: 6 weeks
- Ongoing maintenance and support
Looking forward
Expanding the pipeline to a wider range of machine learning models and platform options, and developing additional public dataset tools.
