How difficult is to reproduce an experiment, even when the input results and the outputs are available online? In this work we quantify it.

1. Beyond the PDF 2
Demo and Supporting
Material
Daniel Garijo Verdejo
Ontology Engineering Group. Laboratorio de Inteligencia Artificial
Departamento de Inteligencia Artificial
Facultad de Informática
Universidad Politécnica de Madrid

2. The TB-Drugome
Have you ever heard about the TB-Drugome?
http://funsite.sdsc.edu/drugome/TB/
•Workflow presented on the previous Beyond the PDF Workshop
•Constructed to detect whether any approved drug could be a candidate to cure
the TB.
•All materials used for the experiment are presented in the website.
•Can it be reproduced?? How much would it cost??
2

3. The Original Workflow
Main Functionality
3

4. The Reproduced Workflow
LINK to execution materials
4

5. Differences with reproduced results?
Connections- Connection-
Drug Name Connections solvedStructures Drug Name Connections SolvedStructures
Tretinoin 257 46
Alitretinoin 98 14
Levothyroxine 173 36
Levothyroxine 63 14
Methotrexate 156 32
Methotrexate 48 10
4-Hydroxytamoxifen 115 25
Estradiol 38 10 Estradiol 98 20
Rifampin 34 6 Amantadine 79 1
Rifampin 78 13
4-Hydroxytamoxifen 33 10 Raloxifene 75 18
Amantadine 32 0 Propofol 54 5
Indinavir 51 14
Raloxifene 28 10 Penicillamine 44 10
Propofol 24 3 Daunorubicin 44 12
Indinavir 23 2 Triclosan 42 5
Ritonavir 22 7 Darunavir 40 15
Darunavir 22 5 Desoxycorticosterone 39 12
Lopinavir 22 4 Diethylstilbestrol 39 7
Penicillamine 20 5 Amprenavir 38 14
Nelfinavir 20 3 Tadalafil 36 7
Pemetrexed 35 17
Original Lopinavir 35 10
Saquinavir 34 13
Indomethacin 34 8
Bepridil 33 14
Testosterone 32 14
2
Bexarotene 32 10
Reproduced Imatinib 32 12
Ritonavir 32 10

6. Measuring the reproducibility effort
Tasks Time (days)
Familiarization with workflow and running software 20
SMAP steps 4
SMAP result sorter steps 1
Merger steps 0.5
Get significant results 0.5
FATCAT URL checker 1
FATCAT step 0.5
Remove significant pairs 0.5
Create clip files 1
Create ideal ligands 1
Ideal ligand checker 1
Autodock Vina 2
Data visualization steps 2
TOTAL 35 days
6

7. Linked Data Publication
Workflow executed with the Wings workflow engine
Provenance and specification published according the PROV standard and the
Open Provenance Model. All data is published under the Linked Data Principles
automatically.
The workflow is part of a corpus (LINK) which has been exposed as Linked
Data, along with its provenance and specifications.
LINK: Querying the LD workflow templates
LINK: Exposing workflows as Wikipages automatically
7

8. Beyond the PDF 2
Demo and Supporting
Material
Daniel Garijo Verdejo
Ontology Engineering Group. Laboratorio de Inteligencia Artificial
Departamento de Inteligencia Artificial
Facultad de Informática
Universidad Politécnica de Madrid