TheUKDave
Lead-level Pipeline Engineer at AstraZeneca, Cambridge
1 Apr 2022 - Present
Working to build out AstraZeneca's internal pipeline management platform, handling data pipelines from sequencing instruments and mass spectrometers, through to output of QC validated reports, utilising many AWS products.
I was primarily repsonsible for adding new modalities for proteomics data to the platform, and parallelising multiple pipelines to greatly reduce the pipeline duration.
I was also invovled in the update to version 3 of the platform, which uses a microservices architecture using Conductor, and Nextflow.
Responsibilities & Achievements
- Developed a detailed and extensible task coordination system to manage the steps of any pipeline
- Specified and developed a new version of the JSON protocol used to communicate between the server and workers
- Implemented brand new pipelines from the ground up to handle targeted and global proteomics data
- Parallelised multiple pipelines to greatly increase performance
- Initial 9 month contract extended for multiple time to a total of 3 years
Technologies & Techniques Used
- Python
- CI/CD
- AWS
- Postgres
- DevOps
- Django
- Javascript
- HTML5
- CSS3