EINFÜHRUNGSVERANSTALTUNG CSP 2018 QUEST-KRITERIEN

Berliner Institut für Gesundheitsforschung13. September 2018

Die folgenden Folien sind als Anregung zu verstehen,

sie ersetzen nicht die AKTIVE Auseinandersetzung mit den

QUEST-Kriterien im Hinblick auf Ihre spezifische

Forschungsfrage.Bei Fragen wenden Sie sich gerne an Dr. Miriam Kip (miriam.kip@bihealth.de oder

quest@bihealth.de).

QUEST Tool box: (https://www.bihealth.org/en/quest-center/mission-approaches/englische-

uebersetzung/the-quest-toolbox/)

mailto:quest@bihealth.de

MERIT-Project

Miriam Kip

Axel Pries

Development of attributes of a robust and innovative research/ Merkmale einer robusten und innovativenForschung (MERIT)

QUEST-criteria:open questions:

• priority setting• strategies of scientific rigor• transparency and dissemination or results• participation

• intramural funding schemes, e.g. CSP, Validation fund• introduction to the doctoral/dissertation program at Charité

PRIORITY SETTINGS

Warum?

• Vorhandene Evidenz darlegen• Wissenslücken identifizieren• Bisherige Studienqualität kritisch bewerten• Identifikation von Faktoren, die die Effektivität einer Maßnahme

beeinflussen• Ableiten wichtiger Informationen hinsichtlich Design neuer

Studien• “evidence-based trial design”

- Reduce waste in future research- Reduce risk for humans and animals- Reduce risk of unnecessary enrollment of humans

Wie? (a very short introduction)

• Clinical Interventions - PICOS• Population• Intervention• Control/Comparators• Outcome• Study design

• preclinical• (a) treatment/intervention• (b) disease or condition of interest• (c) animal species/cell population studied• (d) outcome measures

MeshtermsVolltextsuchegeneral expressions

Boolsche Operanden

Filter

Wo?

• PubMed, Embase• Clinicaltrials.gov, Metaregister der WHO etc..• Eigene Vorstudien (dabei Daten nachvollziehbar darstellen)

• BMJ 2011;343:d5928 doi: 10.1136/bmj.d5928• Hooijmans et al. BMC Medical Research Methodology 2014, 14:43

http://www.biomedcentral.com/1471-2288/14/43• http://syrf.org.uk/library/• https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf• https://www.cochrane.de/de/ressourcen• https://www.radboudumc.nl/en/research/radboud-technology-

centers/animal-research-facility/systematic-review-center-for-laboratory-animal-experimentation

• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3265183/pdf/LA-11-087.pdf• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3104815/pdf/LA-09-117.pdf

Ressourcen (Auswahl)

http://www.biomedcentral.com/1471-2288/14/43http://syrf.org.uk/library/https://www.york.ac.uk/media/crd/Systematic_Reviews.pdfhttps://www.cochrane.de/de/ressourcenhttps://www.radboudumc.nl/en/research/radboud-technology-centers/animal-research-facility/systematic-review-center-for-laboratory-animal-experimentationhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC3265183/pdf/LA-11-087.pdfhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC3104815/pdf/LA-09-117.pdf

STRATEGIES OF SCIENTIFIC RIGOR (I)

Highly effective treatmentsin animal stroke models

PLoS Biol. 2010;8(3):e1000344.

http://www.ncbi.nlm.nih.gov/pubmed/20361022

Phase III studiesshow no effect

Minnerup et al.Exp.Transl.Stroke Med.2014;6:2

PossibleSolutions

Reduce Bias!Use blinding, randomization,in/exclusion criteria.Report results according to guidelines (e.g. ARRIVE, CONSORT, PRISMA, etc.)Pre-register

Use statistics sensibly!Go beyond the bar plot! Show individual data points and distributions.

Think biological significance, think effect size.Consult a statistician.

Bias influences effect size

Alzheimer's disease models models

Blinded conduct of experiment

Blinded assessment of outcome

Blinded assessment of outcome

Stroke models (NXY-095)

Impr

ovem

ent i

n be

havi

oura

l out

com

e (S

tand

ardi

sed

Effe

ct S

ize)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Yes

No

Red

uctio

n in

infa

rct s

ize

Red

uctio

n in

infa

rct s

ize

> 30 studies > 500 animals

Toolsand Resources

Statistical Consulting (CRU and Biostats)QUEST toolbox (includes tools for Figure creation)Experimental Design Assistant for animal experimentsCourses at the QUEST Center and Promotionskolleg

STRATEGIES OF SCIENTIFIC RIGOR (II)

U N I V E R S I T Ä T S M E D I Z I N B E R L I N

Clinical Scientist

Prof. Dr. Geraldine RauchDr. Jochen Kruppa

Institute of Biometry and Clinical Epidemiology

geraldine.rauch@charite.de

Auswahl von Endpunkten

Endpunkte: Diejenigen Merkmale in klinischen Studien, anhand derer der Erfolg der Studie gemessen werden soll. Man unterscheidet dabei zwischen primären und sekundären Endpunkten.

Primäre Endpunkt: Erfasst dabei das Hauptziel der Studie. Dieses wird am Ende der Studie mit einem statistischen Test überprüft.

Sekundären Endpunkte: Erfassen dabei weitere Aspekte der Studie. Die Auswertung erfolgt rein deskriptiv.

Bei der Auswahl von Endpunkten sollte Folgendes beachtet werden: Angemessen für die medizinische Fragestellung Möglichst objektiv erfassbar Möglichst hohes Skalenniveau (vgl. spätere Folien)

Begriffe Zielgröße / Endpunkt: das Merkmal, das als Ergebnis einer Untersuchung interessiert, z.B. eine unter dem Einfluss der Therapie sich verändernder Laborwert oder ein Krankheitssymptom

Einflussgrößen: alle Merkmale die im funktionellen Zusammenhang zur Zielgröße stehen, z.B. bestimmte Behandlungsformen, Therapiemaßnahmen

Störgrößen/Confounder: Einflussgrößen, deren Untersuchung nicht Ziel der Studie ist, z.B. die unerwünschte Abhängigkeit vom Alter oder Geschlecht. Störgrößen sollten entweder eliminiert oder in der Analyse der Zielgrößen berücksichtigt werden. Störgrößen sind jedoch nicht immer alle erfassbar.

Harte und weiche Endpunkte

• Harte Endpunkte– lassen sich direkt erheben– Beispiele:

• Überlebenszeit• Tumoransprechen

• Weiche Endpunkte– lassen sich nur indirekt erheben– Beispiele:

• Lebensqualität• Schmerzempfinden

Studiendesign

nicht kontrolliert/einarmig

kontrolliert/mehrarmig

nicht randomisiert randomisiert

offen verblindet

monozentrisch multizentrisch

einfach

doppelt

Nicht für jede Fragestellung ist jedes Design möglich!

Goldstandard für Studien zum Wirksamkeitsnachweis

StudientypenEs gibt viele verschiedene StudientypenKlassifizierung nach unterschiedlichen Kriterien möglich:

• Fragestellung (z.B. Therapiestudie, Diagnosestudie, Prognosestudie),

• Blickrichtung (prospektiv, retrospektiv), • “Aktivität” des Forschers

– Beobachtungsstudien (Querschnittsstudien, Kohortenstudien, Fall-Kontroll-Studien)

– Interventionsstudien (nichtrandomisierte Studien, randomisierte Studie)

• Bei Arzneimittelstudien: „Phase“ der Erprobung (Phase I, II, III, IV)• usw.

Hier kein umfassender Überblick möglich!

Quelle: Dtsch Arztebl 2009; 106(15)

Literaturempfehlung

• Schumacher M, Schulgen G (2008, 3. Auflage): Methodik klinischer Studien, Springer.

• Sachs L (1993, 7. Auflage): Statistische Methoden: Planung und Auswertung, Springer.

• Weiß C (2010, 5. Auflage): Basiswissen Medizinische Statistik, Springer.• Gonick L (1993): The Cartoon Guide to Statsitics, HarperCollins Publisher

STRATEGIES OF SCIENTIFIC RIGOR (III)

Tracey L. Weissgerber, PhDTwitter: @T_Weissgerber

Reveal, Don’t Conceal: Transforming Data Visualization to Improve Transparency

Data presentation is the foundation of our collective scientific knowledge…

Figures are especially important.They often show data for key findings.

Many different data distributions can lead to the same bar graph…

Symmetric Outlier Bimodal Unequal n

Test p valueT-test: equal var. 0.035 0.074 0.033 0.051T-test: Unequal var. 0.035 0.076 0.033 0.035Wilcoxon 0.056 0.10 0.173 0.067

30

20

10

0

Weissgerber et al., PLOS Biology 2015

Why you shouldn’t use a bar graph even if your data are normally distributed

Sedentary ExerciseTrained

Sedentary ExerciseTrained

Hea

rt ra

te (b

eats

/min

)

Bar graph(mean ± SE)

Bar graphwith points

Univariatescatterplot

Range ofObserved

Values

Zone ofIrrelevance

Zone of Invisibility

0

Bar graphs1. Don’t allow you to critically evaluate continuous data2. Arbitrarily assign importance to bar height, instead of showing

how the difference between means compares to the variabilityWeissgerber et al., JBC 2017

Graphics for:- Cross sectional studies

- Experimental studies with independent groups

Dotplot Boxplot with points

Boxplot Violin plot(with or withoutpoints)

Bar graph

Outcome variable

Continuous Continuous Continuous Continuous Counts & proportions

Sample size Small Medium Large Medium to Large

Any

Data distribution

Any Any Do not use for bimodal data

Any N/A

Free violin plot tool: https://interactive-graphics.shinyapps.io/violin/

https://interactive-graphics.shinyapps.io/violin/

Free Tools for Interactive Graphics

Dot, box or violin plot: http://statistika.mfub.bg.ac.rs/interactive-dotplot/

Interactive line graph:http://statistika.mfub.bg.ac.rs/interactive-graph/

Additional resources: Twitter @T_Weissgerber

Group 1

Group 2

Group 3

Condition 1 Condition 2 Condition 30

5

10

15

20

http://statistika.mfub.bg.ac.rs/interactive-dotplot/http://statistika.mfub.bg.ac.rs/interactive-graph/

Why we need to report more than “Data were analyzed by t-tests and ANOVA”

• Meta-research studies show that statistical errors are common. These include:– Failing to specify what test was used – Using suboptimal or inappropriate tests– Incorrect p-values

• T-tests and ANOVA are the most common analysis techniques in many basic biomedical science fields

Why we need to report more than “Data were analyzed by t-tests and ANOVA”

Systematic review: Many physiology papers are missing information needed to determine what type

of ANOVA was performed

Essential Details Papers with Missing Information

Number of factors 17%

Names of factors 54%

Post-hoc tests 27%

Between vs. within-subjects factors for repeated measures ANOVA

63%

Papers rarely contain information needed to verify the test result

Essential Details Papers with Missing Information

T-tests(n = 163)

ANOVA(n = 225)

Test statistic 96% 95%

Degrees of freedom * 7% 97%

Exact p-value 69% 78%

* Exact sample size is also acceptable for t-tests

This information is essential to identify bias & correct errors

1. Confirm that the correct test was used2. Confirm test results

– Errors in reported p-value are common; may alter conclusions in 1/8 papers1

3. Assess bias: Were observations excluded without explanation?– Among papers with animal models of cancer & stroke2

• 7-8% excluded animals without explanation• 2/3 didn’t have enough information to assess

1Nuitjen et al., Behav Res Methods 20152Holman et al., PLOS Biol 2016

Solutions• Report exactly what test you used

Test ReportingT-tests Unpaired vs. paired, equal vs. unequal variance

ANOVA Number & names of factors, between vs. within subjects factors, post-hoc tests, any interaction terms included in the ANOVA

More complex tests Detail needed to reproduce analysis

These two tests…

ANOVAwithout repeated measures

Repeated MeasuresANOVA

…see the data differently

Compares3 independent groups

(n = 30, 10/group)

Compares10 pairs

of related observations

(n = 10)

…test differenthypotheses

Null hypothesis: Mean T1 = Mean T2 = Mean T3

Null hypothesis: Mean T1 = Mean T2 = Mean T3

when population means are related

…use information differently

More unexplained variability Less unexplained variability – we can account for the effect of “subject”.

Sums of squaresConditions (between groups)

Residual (within groups)

Total

240

2604

2844

Sums of squaresConditions (between groups)

Residual (within groups)SubjectsError

Total

240

26042195

409

2844

…give different results

overall p = 0.304 overall p = 0.016

Why does this matter? An example…

?

0

20

40

60

0 1 2 3T1 T2 T30

20

40

60

T1 T2 T3

Solutions• Report exactly what test you used• Improve clarity by describing simple tests in

figure & table legends• Report test statistic, degrees of freedom,

exact p-value• Use Statcheck: http://statcheck.io• Deposit code: Make your analyses

reproducibleClear reporting allows you to identify &

correct errors prior to publication!

http://statcheck.io

STAKEHOLDER ENGAGEMENT

Patient Engagement in biomedical research: From research subjects to partners in research

Engaging patients as partners and not just subjects in research can improve research! They hold important experience-based expertise from living with diseases.

Miravittles M et al. (2013), Respiratory Medicine 107, 1977-1985

Patient Engagementin biomedical research:Key areas for patient engagement

Geissler J et al. (2017), Therapeutic Innovation & Regulatory Science 51(5), 612-619.

Research phases

Key areas

Key areas

Patient Engagementin biomedical research:Important resources

Best Practice Examples Possible gatekeeper to identify patients……although so far no

institution appointed.

Other stakeholders whose involvement might benefit research:• Political actors involved in decision-making about medical

products, e.g. G-BA, IQWiG, BfArM, PEI?• Industry actors from the pharmaceutical and biotech world?• Clinicians using the research outcomes?• Training institutions teaching the findings?• …

Patient Engagementin biomedical research:Who else should be involved?

• Who should represent (and why) particular stakeholder groups?• Do representatives need particular training?• How should engagement activities be structured to ensure a level

playing field?• What conflicts of interest exist and how can they be managed?• Is there a need for a coordinating institution for patient

engagement?• …

Patient Engagementin biomedical research:Further open questions

EInführungsveranstaltung CSP 2018 �QUEST-KriterienDie folgenden Folien sind als Anregung zu verstehen,�sie ersetzen nicht die AKTIVE Auseinandersetzung mit den QUEST-Kriterien im Hinblick auf Ihre spezifische Forschungsfrage.�Bei Fragen wenden Sie sich gerne an Dr. Miriam Kip (miriam.kip@bihealth.de oder quest@bihealth.de).�QUEST Tool box: (https://www.bihealth.org/en/quest-center/mission-approaches/englische-uebersetzung/the-quest-toolbox/)��MERIT-Project�PRIORITY SETTINGSWarum?Wie? (a very short introduction) Wo?Ressourcen (Auswahl)�STRATEGIES OF SCIENTIFIC RIGOR (I)Highly effective treatments�in animal stroke modelsPhase III studies�show no effectPossible�SolutionsBias influences effect sizeTools�and Resources�STRATeGIES OF SCIENTIFIC RIGOR (II)Foliennummer 16Auswahl von EndpunktenBegriffe Harte und weiche EndpunkteStudiendesignStudientypenFoliennummer 22LiteraturempfehlungSTRATEGIES OF SCIENTIFIC RIGOR (III)��Foliennummer 25Data presentation is the foundation of our collective scientific knowledge…Many different data distributions can lead to the same bar graph…Why you shouldn’t use a bar graph even if your data are normally distributedGraphics for:�- Cross sectional studies �- Experimental studies with independent groupsFree Tools for Interactive GraphicsWhy we need to report more than “Data were analyzed by t-tests and ANOVA”Why we need to report more than “Data were analyzed by t-tests and ANOVA”Papers rarely contain information needed to verify the test resultThis information is essential to identify bias & correct errorsSolutionsWhy does this matter? An example…Solutions�STAKEHOLDER EngagementPatient Engagement �in biomedical research: �From research subjects to partners in researchPatient Engagement�in biomedical research:�Key areas for patient engagementPatient Engagement�in biomedical research:�Important resourcesPatient Engagement�in biomedical research:�Who else should be involved?Patient Engagement�in biomedical research:�Further open questions