SlideShare uma empresa Scribd logo
1 de 25
IGA Workflow
Alessandro Gervaso   Vittorio Zamboni
Introduction
IGA Workflow is a web-based tool created
using the Django Framework.
It is intended as a management tool for the IGA
wet-lab.

We use it to track the lifecycle of biological
samples, from vial to file.
Overview
● Laboratory management, from sample to
  flowcell
● Bioinformatic analyses
● Pipelines management
● Technology
● Other applications and future developments
Biology for dummies
Technology - Basics
Database server
● Postgresql
● Redis
Workers
● Celery
Web server
● nginx + uWSGI
Overview - Lab
● SAMPLE: The basic unit in the lab

● LIBRARY: a treated sample with an attached
  chemical TAG.

● POOL: a set of libraries, ready to be placed
  on a flowcell's lane.
Overview - Lab
Overview - Lab
The main challenge was to replace notebooks
with a tool that allows to:
● insert samples, libraries, pools, and edit
   them;
● create lanes and runs and the configuration
   files for the physical sequencer;
● collect the sequencer results and map them
   in a easy way
Almost done!
Overview - Lab
We started using the basic Django admin
                     BUT

● the page loading was slow
● due to the admin's nature we lacked
  flexibility
● we were forcing the lab people procedures
● management was cumbersome
Overview - Analyses
After the physical sequencing the raw data
(basecalls) must be converted in FASTQ files.

The FASTQ files are FASTA files with some
embedded quality stats.

They are the starting point for almost every
genomic analysis.
Overview - Analyses
To optimize time and resources we use a
cluster of Celery workers.

● we track the software packages used
● we track their parameters
● we create a set of useful stats
Technology
Additions:

● Informational celery tasks (list directories
  content, copy files between devices with
  dbus plugin and UDisks instead of ugly
  hacks, ...)
Overview - Pipelines
FASTQ files     alignments and assemblies

Each analysis use different software in
sequence or in parallel.
Using hundreds of samples, the analyses can't
be handled manually.
Pipelines
The results of each pipelines (like previous
analyses) must be tracked.

Since CLI based software is not user-friendly,
we develop a graphical pipeline builder.

Users are able to choose and combine different
softwares to perform their own analyses.
Pipeline
● Workers have different queues in order to
  satisfy different tasks

● Worker's tasks talk each other with Redis to
  avoid inconsistencies and to improve
  performances
Pipelines
Different queues (stored in Redis dbs):

  ○   available
  ○   queued
  ○   active
  ○   completed
  ○   error
Pipelines
Video demonstration of a working pipeline with
use of different kind of steps.
(1 video: 1 minute)
Under development
● simple interface that allow customers to:
  ○ insert their samples directly
  ○ watch the results of their pipeline in a genome
      browser - also made with Django (see below)!

● barcodes

● genome browser (like GMOD GBrowse, but
  with the greatness of Python instead of the
  confusion of Perl)
Genome browser
An application that allows browsing a genome's
annotations (like genes, or where reads are
aligned).

Actually, the best web genome browser is
GMOD Gbrowse.
Genome browser
The challenge is to develop a genome browser
that has a set of basic features and could
accept plugins for particular type of data - like
GMOD Gbrowse.

In addition, it must be quick and easy to
manage - NOT like GMOD Gbrowse.
Genome browser
Video demonstration of the genome browser.
(2 videos: 1 minute + 1 minute)
Acknowledgments
● The wet-lab     Teams that developed:
  ladies at IGA   ● Django
                  ● JQuery
                  ● nginx
● WEBdeBS         ● uWSGI
                  ● Celery
                  ● Redis
                  ● pip and virtualenv
                  ● PostgreSQL
                  ● All the open source projects involved

Mais conteúdo relacionado

Mais procurados

Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...Develcz
 
Bgoug 2019.11 test your pl sql - not your patience
Bgoug 2019.11   test your pl sql - not your patienceBgoug 2019.11   test your pl sql - not your patience
Bgoug 2019.11 test your pl sql - not your patienceJacek Gebal
 
Grunt - The JavaScript Task Runner
Grunt - The JavaScript Task RunnerGrunt - The JavaScript Task Runner
Grunt - The JavaScript Task RunnerMohammed Arif
 
Killer R10K Workflow - PuppetConf 2014
Killer R10K Workflow - PuppetConf 2014Killer R10K Workflow - PuppetConf 2014
Killer R10K Workflow - PuppetConf 2014Puppet
 
Stop Sucking at Building Stuff!
Stop Sucking at Building Stuff!Stop Sucking at Building Stuff!
Stop Sucking at Building Stuff!Puppet
 
Sandstorm.io and OpenDocument
Sandstorm.io and OpenDocumentSandstorm.io and OpenDocument
Sandstorm.io and OpenDocumentAudrey Tang
 

Mais procurados (10)

Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
 
Bgoug 2019.11 test your pl sql - not your patience
Bgoug 2019.11   test your pl sql - not your patienceBgoug 2019.11   test your pl sql - not your patience
Bgoug 2019.11 test your pl sql - not your patience
 
Grunt - The JavaScript Task Runner
Grunt - The JavaScript Task RunnerGrunt - The JavaScript Task Runner
Grunt - The JavaScript Task Runner
 
Killer R10K Workflow - PuppetConf 2014
Killer R10K Workflow - PuppetConf 2014Killer R10K Workflow - PuppetConf 2014
Killer R10K Workflow - PuppetConf 2014
 
Stop Sucking at Building Stuff!
Stop Sucking at Building Stuff!Stop Sucking at Building Stuff!
Stop Sucking at Building Stuff!
 
What grunt?
What grunt?What grunt?
What grunt?
 
OpenCms Days 2012 - Developing OpenCms with Gradle
OpenCms Days 2012 - Developing OpenCms with GradleOpenCms Days 2012 - Developing OpenCms with Gradle
OpenCms Days 2012 - Developing OpenCms with Gradle
 
Groovy and noteworthy
Groovy and noteworthyGroovy and noteworthy
Groovy and noteworthy
 
Sandstorm.io and OpenDocument
Sandstorm.io and OpenDocumentSandstorm.io and OpenDocument
Sandstorm.io and OpenDocument
 
React Django Presentation
React Django PresentationReact Django Presentation
React Django Presentation
 

Semelhante a Iga workflow

Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...Sauce Labs
 
DevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and ProjectsDevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and ProjectsFedir RYKHTIK
 
SciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programmingSciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programmingSamuel Lampa
 
ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest presentation in MadridJUG (Madrid December 2017)ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest presentation in MadridJUG (Madrid December 2017)ElasTest Project
 
Reproducibility in artificial intelligence
Reproducibility in artificial intelligenceReproducibility in artificial intelligence
Reproducibility in artificial intelligenceCarlos Toxtli
 
2016_04_04_CNI_Spring_Meeting_Microservices
2016_04_04_CNI_Spring_Meeting_Microservices2016_04_04_CNI_Spring_Meeting_Microservices
2016_04_04_CNI_Spring_Meeting_MicroservicesJason Varghese
 
Software Delivery in 2016 - A Continuous Delivery Approach
Software Delivery in 2016 - A Continuous Delivery ApproachSoftware Delivery in 2016 - A Continuous Delivery Approach
Software Delivery in 2016 - A Continuous Delivery ApproachGiovanni Toraldo
 
Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Jenkinsconf Presentation - Advance jenkins management with multiple projects.Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Jenkinsconf Presentation - Advance jenkins management with multiple projects.Ohad Basan
 
TDD with Python and App Engine
TDD with Python and App EngineTDD with Python and App Engine
TDD with Python and App EngineRicardo Bánffy
 
ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest presentation in Panel Sistemas company (Madrid December 2017)ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest presentation in Panel Sistemas company (Madrid December 2017)ElasTest Project
 
Introduzione a junit + integrazione con archibus
Introduzione a junit + integrazione con archibusIntroduzione a junit + integrazione con archibus
Introduzione a junit + integrazione con archibusDavide Fella
 
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie..."How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...Fwdays
 
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08Борис Зора
 
OpenProdoc 0.8 Load Tests
OpenProdoc 0.8 Load TestsOpenProdoc 0.8 Load Tests
OpenProdoc 0.8 Load Testsjhierrot
 

Semelhante a Iga workflow (20)

Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
 
DevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and ProjectsDevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and Projects
 
SciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programmingSciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programming
 
ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest presentation in MadridJUG (Madrid December 2017)ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest presentation in MadridJUG (Madrid December 2017)
 
Reproducibility in artificial intelligence
Reproducibility in artificial intelligenceReproducibility in artificial intelligence
Reproducibility in artificial intelligence
 
2016_04_04_CNI_Spring_Meeting_Microservices
2016_04_04_CNI_Spring_Meeting_Microservices2016_04_04_CNI_Spring_Meeting_Microservices
2016_04_04_CNI_Spring_Meeting_Microservices
 
Software Delivery in 2016 - A Continuous Delivery Approach
Software Delivery in 2016 - A Continuous Delivery ApproachSoftware Delivery in 2016 - A Continuous Delivery Approach
Software Delivery in 2016 - A Continuous Delivery Approach
 
ElasTest Webinar
ElasTest WebinarElasTest Webinar
ElasTest Webinar
 
Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Jenkinsconf Presentation - Advance jenkins management with multiple projects.Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Jenkinsconf Presentation - Advance jenkins management with multiple projects.
 
TDD with Python and App Engine
TDD with Python and App EngineTDD with Python and App Engine
TDD with Python and App Engine
 
ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest presentation in Panel Sistemas company (Madrid December 2017)ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest presentation in Panel Sistemas company (Madrid December 2017)
 
Go at Skroutz
Go at SkroutzGo at Skroutz
Go at Skroutz
 
Introduzione a junit + integrazione con archibus
Introduzione a junit + integrazione con archibusIntroduzione a junit + integrazione con archibus
Introduzione a junit + integrazione con archibus
 
Cypress Testing.pptx
Cypress Testing.pptxCypress Testing.pptx
Cypress Testing.pptx
 
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie..."How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
 
Integration testing - A&BP CC
Integration testing - A&BP CCIntegration testing - A&BP CC
Integration testing - A&BP CC
 
Mono Repo
Mono RepoMono Repo
Mono Repo
 
Cypress for Testing
Cypress for TestingCypress for Testing
Cypress for Testing
 
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
 
OpenProdoc 0.8 Load Tests
OpenProdoc 0.8 Load TestsOpenProdoc 0.8 Load Tests
OpenProdoc 0.8 Load Tests
 

Mais de WEBdeBS

Legal Pirlo
Legal PirloLegal Pirlo
Legal PirloWEBdeBS
 
Nodejsconf 2012
Nodejsconf 2012Nodejsconf 2012
Nodejsconf 2012WEBdeBS
 
Mockup, wireframe e visual: una breve introduzione
Mockup, wireframe e visual: una breve introduzioneMockup, wireframe e visual: una breve introduzione
Mockup, wireframe e visual: una breve introduzioneWEBdeBS
 
Djangoday lt 20120420
Djangoday lt 20120420Djangoday lt 20120420
Djangoday lt 20120420WEBdeBS
 
Unbit djangoday 20120419
Unbit djangoday 20120419Unbit djangoday 20120419
Unbit djangoday 20120419WEBdeBS
 
Django mongodb -djangoday_
Django mongodb -djangoday_Django mongodb -djangoday_
Django mongodb -djangoday_WEBdeBS
 
Virtualenv
VirtualenvVirtualenv
VirtualenvWEBdeBS
 
Django e il Rap Elia Contini
Django e il Rap Elia ContiniDjango e il Rap Elia Contini
Django e il Rap Elia ContiniWEBdeBS
 
PepperTweet - Project presentation Startup Weekend Brescia
PepperTweet - Project presentation Startup Weekend BresciaPepperTweet - Project presentation Startup Weekend Brescia
PepperTweet - Project presentation Startup Weekend BresciaWEBdeBS
 
Peppertweet - Presentazione Startup Weekend Brescia
Peppertweet - Presentazione Startup Weekend BresciaPeppertweet - Presentazione Startup Weekend Brescia
Peppertweet - Presentazione Startup Weekend BresciaWEBdeBS
 
NoSql Day - Chiusura
NoSql Day - ChiusuraNoSql Day - Chiusura
NoSql Day - ChiusuraWEBdeBS
 
NoSql Day - Apertura
NoSql Day - AperturaNoSql Day - Apertura
NoSql Day - AperturaWEBdeBS
 

Mais de WEBdeBS (14)

Legal Pirlo
Legal PirloLegal Pirlo
Legal Pirlo
 
Nodejsconf 2012
Nodejsconf 2012Nodejsconf 2012
Nodejsconf 2012
 
Mockup, wireframe e visual: una breve introduzione
Mockup, wireframe e visual: una breve introduzioneMockup, wireframe e visual: una breve introduzione
Mockup, wireframe e visual: una breve introduzione
 
Djangoday lt 20120420
Djangoday lt 20120420Djangoday lt 20120420
Djangoday lt 20120420
 
Unbit djangoday 20120419
Unbit djangoday 20120419Unbit djangoday 20120419
Unbit djangoday 20120419
 
Geodjango
GeodjangoGeodjango
Geodjango
 
Django mongodb -djangoday_
Django mongodb -djangoday_Django mongodb -djangoday_
Django mongodb -djangoday_
 
Virtualenv
VirtualenvVirtualenv
Virtualenv
 
Fagungis
FagungisFagungis
Fagungis
 
Django e il Rap Elia Contini
Django e il Rap Elia ContiniDjango e il Rap Elia Contini
Django e il Rap Elia Contini
 
PepperTweet - Project presentation Startup Weekend Brescia
PepperTweet - Project presentation Startup Weekend BresciaPepperTweet - Project presentation Startup Weekend Brescia
PepperTweet - Project presentation Startup Weekend Brescia
 
Peppertweet - Presentazione Startup Weekend Brescia
Peppertweet - Presentazione Startup Weekend BresciaPeppertweet - Presentazione Startup Weekend Brescia
Peppertweet - Presentazione Startup Weekend Brescia
 
NoSql Day - Chiusura
NoSql Day - ChiusuraNoSql Day - Chiusura
NoSql Day - Chiusura
 
NoSql Day - Apertura
NoSql Day - AperturaNoSql Day - Apertura
NoSql Day - Apertura
 

Último

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slidevu2urc
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 

Último (20)

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 

Iga workflow

  • 2. Introduction IGA Workflow is a web-based tool created using the Django Framework. It is intended as a management tool for the IGA wet-lab. We use it to track the lifecycle of biological samples, from vial to file.
  • 3. Overview ● Laboratory management, from sample to flowcell ● Bioinformatic analyses ● Pipelines management ● Technology ● Other applications and future developments
  • 5. Technology - Basics Database server ● Postgresql ● Redis Workers ● Celery Web server ● nginx + uWSGI
  • 6. Overview - Lab ● SAMPLE: The basic unit in the lab ● LIBRARY: a treated sample with an attached chemical TAG. ● POOL: a set of libraries, ready to be placed on a flowcell's lane.
  • 8. Overview - Lab The main challenge was to replace notebooks with a tool that allows to: ● insert samples, libraries, pools, and edit them; ● create lanes and runs and the configuration files for the physical sequencer; ● collect the sequencer results and map them in a easy way Almost done!
  • 9. Overview - Lab We started using the basic Django admin BUT ● the page loading was slow ● due to the admin's nature we lacked flexibility ● we were forcing the lab people procedures ● management was cumbersome
  • 10.
  • 11.
  • 12. Overview - Analyses After the physical sequencing the raw data (basecalls) must be converted in FASTQ files. The FASTQ files are FASTA files with some embedded quality stats. They are the starting point for almost every genomic analysis.
  • 13. Overview - Analyses To optimize time and resources we use a cluster of Celery workers. ● we track the software packages used ● we track their parameters ● we create a set of useful stats
  • 14.
  • 15. Technology Additions: ● Informational celery tasks (list directories content, copy files between devices with dbus plugin and UDisks instead of ugly hacks, ...)
  • 16. Overview - Pipelines FASTQ files alignments and assemblies Each analysis use different software in sequence or in parallel. Using hundreds of samples, the analyses can't be handled manually.
  • 17. Pipelines The results of each pipelines (like previous analyses) must be tracked. Since CLI based software is not user-friendly, we develop a graphical pipeline builder. Users are able to choose and combine different softwares to perform their own analyses.
  • 18. Pipeline ● Workers have different queues in order to satisfy different tasks ● Worker's tasks talk each other with Redis to avoid inconsistencies and to improve performances
  • 19. Pipelines Different queues (stored in Redis dbs): ○ available ○ queued ○ active ○ completed ○ error
  • 20. Pipelines Video demonstration of a working pipeline with use of different kind of steps. (1 video: 1 minute)
  • 21. Under development ● simple interface that allow customers to: ○ insert their samples directly ○ watch the results of their pipeline in a genome browser - also made with Django (see below)! ● barcodes ● genome browser (like GMOD GBrowse, but with the greatness of Python instead of the confusion of Perl)
  • 22. Genome browser An application that allows browsing a genome's annotations (like genes, or where reads are aligned). Actually, the best web genome browser is GMOD Gbrowse.
  • 23. Genome browser The challenge is to develop a genome browser that has a set of basic features and could accept plugins for particular type of data - like GMOD Gbrowse. In addition, it must be quick and easy to manage - NOT like GMOD Gbrowse.
  • 24. Genome browser Video demonstration of the genome browser. (2 videos: 1 minute + 1 minute)
  • 25. Acknowledgments ● The wet-lab Teams that developed: ladies at IGA ● Django ● JQuery ● nginx ● WEBdeBS ● uWSGI ● Celery ● Redis ● pip and virtualenv ● PostgreSQL ● All the open source projects involved