Item 1: clear description of the goal of research, study objective(s), study design and study population

The research goal, study objectives, study design, and study and target populations must be clearly described so that the editors of journals and readers can judge the internal and external validity (generalisability) of the study.

Being explicit about the goal of research is a prerequisite for good science regardless of the scientific discipline. For such clarification, a threefold classification of the research goal may be used: (1) to describe; (2) to predict, which is equivalent to identifying ‘who’ is at greater risk of experiencing the outcome; or (3) to draw a causal inference, which attempts to explain ‘why’ the outcome occurs (eg, investigating causal effects).5 19

The study objective refers to the rationale behind the study and points to the specific scientific question being addressed. For example, the objective of the heated water-based exercise (HEx) trial, a randomised controlled trial (RCT), was to evaluate the effect of heated water-based exercise training on 24-hour ambulatory blood pressure (BP) levels among patients with resistant hypertension.20 The study objective is usually provided in the introduction after the rationale has been established.

The study design refers to the type of the study, which is explained in the Methods section.21 Examples of common study designs include RCTs and observational studies such as cohort, case–control or cross-sectional studies.22 The study design should be described in detail. In particular, the randomisation procedure in RCTs, follow-up time for cohort studies, control selection for case–control designs and sampling procedure for cross-sectional studies should be adequately explained.6 7 As a general principle, the study design must be explained sufficiently so that another investigator would be able to repeat the study exactly.

The study population refers to the source population from which data are collected, whereas the target population refers to the population to whom we are going to generalise the study results; the relationship between these two populations may be characterised using inclusion and exclusion criteria and is crucial for assessing generalisability. Returning to the HEx trial, the study population was restricted to persons whose ages were between 40 and 65 years with resistant hypertension for more than 5 years.20 For both trials and observational studies, it is important to know what proportion of the source population is studied and what proportion of the intended data set is used in the analysis data set. For example, the source population may include all patients admitted to a hospital with a certain condition over a certain period of time. However, the analysis data set may only be 50% of this, for various reasons such as patients refusing consent, measurements not taken and patients dropping out. In the HEx trial, for instance, they screened 125 patients with hypertension to find 32 who met the inclusion criteria with resistant hypertension. This has some bearing on the generalisability of the study and to whom heated water-based exercise training can be given and how likely it is to be relevant to practitioners.

Item 2: clear description of outcomes, exposures/treatments and covariates, and their measurement methods

All variables considered for statistical analysis should be stated clearly in the paper, including outcomes, exposures/treatments, predictors and potential confounders/mediators/effect-measure modifiers (see Box 1). The measurement method and timing of measurement for each of these variables should also be specified. If the goal of the research is to draw a causal inference (explain ‘why’ the outcome occurs) via observational studies, authors should present their causal assumptions in a causal diagram.23–25 To exemplify this concept, in a cohort study evaluating the effect of physical activity on functional performance and knee pain in patients with osteoarthritis,26 physical activity (exposure) was measured using the Physical Activity Scale for the Elderly, and functional performance and self-reported knee pain (outcomes) were measured by the Timed 20-Metre Walk Test and the Western Ontario and McMaster Universities Osteoarthritis Index, respectively. Depressive symptoms were considered a potential confounder and measured using the Center for Epidemiologic Studies Depression Scale. All variables were measured at baseline and in three annual follow-up visits, and a causal method along with a causal diagram representing the study population was used to estimate the effect of interest.26

Item 3: validity of the study design

The design should be valid and match the research question without introducing bias in the study results. For example, an editor should be able to assess whether the controls in a case–control study were adequately representative of the source population of the cases. Alternatively, in a clinical trial, it should be clear whether there was one (or more) control groups, and if so, whether patients were randomised to treatment or control, and if so, whether the randomisation method and allocation concealment were appropriate.

Item 4: clear statement and justification of sample size

The manuscript should have a section clearly justifying the sample size.27 When a sample size calculation is warranted, the sample size section should be described in enough detail to allow replication of the estimate, along with a clear rationale (supported by references) on choice of values used in the calculation, the outcome for which the calculation is based on, including the minimum clinically important effect size.28 29 For example, typical sample size calculations aim to ensure that the study contains a sufficiently large precision for estimates of occurrence measures (eg, risk) or association measures (eg, risk ratio),30 31 or that there is an adequate power to detect genuine effects (eg, true differences) if they exist (statistical tests). Attrition/loss to follow-up/non-response and design effects (eg, due to clustering) should be taken into consideration. Guidance for sample size calculation for prediction model development and validation has been described previously.32–34

Item 5: clear declaration of design violations and acceptability of the design violations

Design violations frequently occur in research. Non-response in surveys, censoring (loss to follow-up or competing risks) in prospective studies35 and non-compliance with the study treatments in RCTs are examples and should be declared explicitly in the paper.36 37 Given the validity of the design, the acceptability of violations should be assessed. For example, was an observed non-response/censoring proportion too high? What were the reasons for data loss, and is this level acceptable to achieve the scientific goals of the study?

Item 6: consistency between the paper and its previously published protocol

The reviewer should identify inconsistencies with any published protocol (and where relevant, registry information) regarding important features of the study, including sample size, primary/secondary/exploratory outcomes and statistical methods.

Item 7: correct and complete description of statistical methods

A separate part in the Methods section of the manuscript should be devoted to the description of the statistical procedures. Both descriptive and analytical statistical methods should be sufficiently described so that the methods can be assessed by a statistical reviewer to judge their suitability and completeness in addressing the study objectives.

Item 8: valid statistical methods used and assumptions outlined

The validity of statistical analyses relies on some assumptions. For example, the independent t-test for the comparison of two means requires three assumptions: independence of the observations, normality and homogeneity of variance.38 As another example, all expected values for a χ² test must be more than 1, and at most 20% of the expected values can be less than 5. These statistical assumptions should be judged as a matter of context or assessed using appropriate methods such as a normal probability plot for checking the normality assumption.39 In this regard, an alternative statistical test should be applied if some assumptions are clearly violated. It should be noted that some statistical tests are robust against mild-to-moderate violations of some assumptions. For the t-test, lack of normality and lack of homogeneity of variance do not necessarily invalidate the t-test, whereas lack of independence of the outcome variables will imply the results are invalid.40 It has been demonstrated that the independent t-test can be valid but suboptimal for the ordinal scaled data (eg, a variable with values 0, 1, 2, 3) even with a sample size of 20.41

An important but often ignored aspect in practice is that ratio estimates such as the estimated odds ratio (OR), risk ratio and rate ratio are biased away from the null value. This bias is amplified with sparse data known as sparse-data bias.42 A sign of sparse data is an unrealistically large ratio estimate or confidence limit which is simply an artefact of sparse data. For example, an OR >10 for a non-communicable disease should be considered as a warning sign for sparse-data bias. In the extreme, an empty cell leads to an absurd OR estimate of infinity, known as separation.43 Special statistical methods such as penalisation or Bayesian methods must be applied to decrease the sparse-data bias.43 44 Some other important considerations in statistical analysis are (1) accounting for correlation in the analysis of correlated data (eg, variables with repeated measurements in longitudinal studies,45 cluster randomised trials46 and complex surveys47); (2) accounting for matching in the analysis of matched case–control and cohort data48–50; (3) considering ordering of several groups in the analysis; (4) considering censoring in the analysis of survival data; (5) adjusting for baseline values of the outcome in the analysis of randomised clinical trials28; (6) correct calculation and interpretation of the population attributable fraction51 52; (7) adjusting for overfitting using shrinkage or penalisation methods when developing a prediction model53 54; and (8) assessment of similarity and consistency assumptions in network meta-analysis.55

Item 9: appropriate assessment of treatment effect or interaction between treatment and another covariate

Appropriate statistical tests should be used for the assessment of treatment effects and potential interactions. Assessment of overlapping treatment group-specific confidence intervals (CIs can be misleading.56–58 Thus, the comparison of the CIs of the treatment groups should not be used as an alternative to the statistical test of treatment effect. Moreover, comparing p values for the treatment effect at each level of the covariate (eg, men and women) should not be used as an alternative for an interaction test between the treatment and covariate. For example, in the case of observing p value <0.05 in men and p value >0.05 in women, one might incorrectly conclude that gender was an effect modifier.59 Similarly, we cannot conclude no effect modification if the CIs of the subgroups are overlapping.60

Item 10: correct use of correlation and associational statistical testing

The misuse of correlation and associational statistical testing is not uncommon. As an example, correlation should not be used for assessing the agreement between two methods in methods-comparison studies.61 To see why, two measures of X and Y are perfectly correlated but in poor agreement if X is twice Y. Similarly, we cannot infer that the two methods agree well because the p value is large enough using the statistical testing of the means such as paired t-test. In fact, a high variance of differences indicates poor agreement but also increases the chance that the paired t-test will result in a large p value, and thus the methods will appear to agree.1

Item 11: appropriate handling of continuous predictors

Reviewers should be wary of studies that have dichotomised or categorised continuous variables–this should be generally avoided.62 Bias, inefficiency and residual confounding may also result from dichotomising/categorising a continuous variable and using it as a categorical variable in a model. Continuous variables should be retained as continuous and their functional form be examined, as a linearity assumption may not be correct. Approaches for handling continuous predictors include fractional polynomials or regression splines.62–65

Item 12: CIs do not include impossible values

A valid CI should exclude impossible values. For instance, a simple Wald CI for a proportion ( ) is not valid when p is close to 0 or 1, and may yield negative values outside the possible range for a proportion (0≤p≤1).66 To remedy such conditions, the Wilson score or Agresti-Coull interval can be applied.6

Item 13: appropriate comparison of baseline characteristics between the study arms in randomised trials

In a randomised clinical trial, any baseline characteristic difference between groups should be due to chance (or unreported bias). Reviewers should look out for any statistical testing at the baseline as reporting p values does not make sense.67 The decision on which baseline characteristics (prognostic factors) are included in any adjustment should be prespecified in the protocol and should be based on the subject-matter knowledge, not on p values. The differences between groups in baseline characteristics should be identified by their size and discussed in terms of potential implications for the interpretation of the results.

Item 14: correct assessment and adjustment of confounding

An important goal of health research is drawing a causal inference. Here, the interest is in the causal effect of an exposure on the outcome. The major source of bias threatening causality studies, including observational studies as well as randomised studies (with small-to-moderate sample size), is confounding.68–71 Confounding can be controlled in the design phase (eg, through restriction or matching) or analysis phase (eg, using regression models, standardisation or propensity score methods).72–74 Selection of confounders should be based on a priori causal knowledge, often represented in causal diagrams,23 75–77 not p values (eg, using stepwise approaches). Automated statistical procedures, such as stepwise regression, do not discriminate between confounders and other covariates like mediators or colliders which should not be adjusted for in the analysis. Moreover, stepwise regression is only based on the association between confounders and outcome, and disregards the association between the confounders and exposure. Thus, stepwise procedures should not be used for confounder selection. In practice, many confounders (and exposures and outcomes)78 79 are time-varying, and the so-called ‘causal methods’ should be applied for the appropriate adjustment of time-varying confounders.80 81 Similarly, in studies evaluating the prognostic effect of a new variable, adjustment for existing prognostic factors should be routinely performed, and variable selection of the existing factors is not generally needed.53

Item 15: avoiding model extrapolation not supported by data

The goal of interest in many health studies is predicting an outcome from one or more explanatory variables using a regression model. The model is valid only within the range of observed data on the explanatory variables, and we cannot make prediction for people outside the range. This is known as model extrapolation.82 Suppose we have found a linear relation between body mass index (BMI) and BP based on the following equation in a cohort study:

Now the intercept, A, cannot be interpreted because it corresponds to the expected BP value of a person with BMI of zero! The remedy is centring BMI and including the centred variable (BMI−average BMI) in the model so that the new intercept refers to the expected BP value of a person with the average BMI in the population.

As another example, suppose the following linear relation holds in an RCT:

where TRT denotes treatment (1: intervention, 0: placebo) and TRT*BMI is the product term (interaction term) between treatment and BMI. In this model, the parameter B cannot be interpreted on its own because it is the mean difference in BP between two treatment groups for a person with BMI of zero. Again, the solution is centring BMI and including centred BMI and the product term between TRT and centred BMI in the model so that (coefficient of TRT in the new model) refers to the mean difference in BP of a person with the average BMI in the population.

Item 16: adequate handling of missing data

The methods used for handling missing data should be described and justified in relation to stated assumptions about the missing data (missing completely at random, missing at random and missing not at random), and sensitivity analyses must be done if appropriate. Missing data83 can introduce selection bias and should be handled using appropriate methods such as multiple imputation84 and inverse probability weighting.85 Naïve methods such as complete-case analysis, single imputation using the mean of the observed data, last observation carried forward and the missing indicator method are statistically invalid in general and they can lead to serious bias.86

Item 17: adequate and correct description of the data

The mean and standard deviation (SD) provide a satisfactory summary of data for continuous variables that have a reasonably symmetric distribution. The standard error (SE) is not a sound choice to be used in place of SD.87 A useful memory aid is to use SD to Describe data and SE to Estimate parameters.88 Besides, ‘mean±SD’ is not suitable because it implies the range in which 68% of data are within (not a relevant concept we are looking for), and ‘mean (SD)’ should be reported instead.1 In case of having highly skewed quantitative data, median and interquartile range (IQR) are more informative summary statistics for description. It should be noted that the mean/SD ratio of <2 for positive variables is a sign of skewness.89 Categorical data should be summarised as number and percentage.90 For cohort data, a summary of follow-up time such as median and IQR should be reported.

Item 18: descriptive results provided as occurrence measures with CIs and analytical results provided as association measures and CIs along with p values

The point estimates of the occurrence measures, for instance, prevalence, risk and incidence rate with 95% CIs, should be reported for descriptive objectives.90 Alternatively, the point estimates of the association measures, for instance, OR, risk ratio and rate ratio with 95% CIs along with p values, should be reported for analytical objectives as part of the Results section.91 92

Item 19: CIs provided for the contrast between groups rather than for each group

For analytical studies like RCTs, the 95% CIs should be given for the contrast between groups rather than for each group.6 For the BP example mentioned above,20 the authors reported the mean of BP with 95% CI in each group, but they should have given the mean difference in 24-hour ambulatory BP levels between groups with 95% CI as the aim of the trial was to compare treatment with control.

Item 20: avoiding selective reporting of analyses and p-hacking

All statistical analyses performed should be reported regardless of the results. P-hacking, playing with data to produce the desired p value (upwards as well as downwards), must be avoided.93–95 This is probably difficult to assess as a reader/reviewer, but usually one would be clued in if there are many more analyses than those stated in the objectives or only statistically significant comparisons are presented when a larger pool of variables were identified in the methods.

Item 21: appropriate and consistent numerical precisions for effect sizes, test statistics and p values, and reporting the p values rather than their range

P values should be reported directly with one or two significant figures even if they are greater than 0.05, for example, p value=0.09 or p value=0.28. One should not focus on ‘statistical significance’ or dichotomise p values (eg, p<0.05)96–98 or express them as ‘0.000’ or ‘NS’. Nonetheless, spurious precision, too many decimals, in numerical presentation should be avoided.99 100 For example, typically p values less than 0.001 can be written as <0.001 without harm, and it does not make sense to present percentages with more than one decimal when the sample size is much less than 100.

Item 22: providing sufficient numerical results that could be included in a subsequent meta-analysis

Meta-analyses of randomised trials and observational studies provide high levels of evidence in health research. Providing numerical results in individual studies contributing to subsequent meta-analysis is of special importance. Follow-up score and change score from the baseline are two possible approaches that can be applied to estimate treatment effect in RCTs.101 While the follow-up score meta-analysis requires after-intervention mean and SD in two groups of intervention and control, the mean and SD of differences from the baseline are prerequisite for performing change-score meta-analysis. However, authors often only report mean and SD before and after intervention. The mean of the difference in each group can be calculated from the difference of the means, but calculating the SD of differences needs a guessed group-specific correlation between baseline and follow-up scores besides before-intervention and after-intervention SDs.

Item 23: acceptable presentation of the figures and tables

Tables and figures are effective data presentation that should be properly managed.102–105 Figures should be selected based on the type of variable(s) and appropriately scaled. The error bar graph as an illustration can be used for displaying the mean and CI. It is inappropriate to give a bar chart with an SE bar superimposed instead (the so-called ‘dynamite plunger plot’105). Tables should be able to stand on their own and include sufficient details such as labels, units and values.

Item 24: interpreting the results based on association measures and 95% CIs along with p values and correctly interpreting large p values as indecisive results, not evidence of absence of an effect

The study results should be interpreted in light of the point estimate of appropriate association measures such as mean difference and 95% CI as well as precise p values. When testing a null hypothesis of no treatment effect, the p value is the probability the statistical association would be as extreme as observed or more extreme, assuming that null hypothesis and all assumptions used for the test are correct. P values for non-null effect sizes can also be computed. The point estimate is the effect size most compatible with the data in that it has p value=1.00, while the 95% CI shows the range of effect sizes reasonably compatible with the data in the sense of having p value >0.05.97 We should judge the clinical importance and statistical reliability of the results by examining both of the 95% CIs as well as looking at precise p values, not just whether a p value crosses a threshold or not.28 106 It is incorrect to interpret p value >0.05 as showing no treatment effect; instead, it represents an ambiguous outcome.107 108 It is not evidence that the effect is unimportant (‘absence of evidence is not evidence of absence’); inferring unimportance requires that every effect size inside the CI be considered unimportant.97

Item 25: using CIs rather than post hoc power analysis for interpreting the results of studies

Conceptually, it is not valid to interpret power as if it pertains to the observed study results.109–111 Rather, power should be treated as part of the study rationale and design before actual conduct begins, for example, as in sample size calculations. Power does not correctly account for the observations that follow; for example, a study could have high power and observe a high p value, yet still favour the alternative hypothesis over the null hypothesis.111 The precision of results should be gauged using CIs.

Item 26: correctly interpreting occurrence or association measures

It will be crucial to interpret occurrence and association measures correctly. ORs commonly provide examples of misinterpretation: if the event is rare, they can approximate risk ratios, but they are not conceptually the same and will differ considerably if the event is common.112 113 In a study with a risk of 60% in an exposed group and 40% in an unexposed group, the error in interpreting the OR (2.25) as a risk ratio (1.5) is considerable. Prevalence in cross-sectional studies is another example, which sometimes has been incorrectly called ‘risk’.

Item 27: distinguishing causation from association and correlation

We should be cautious about the correct use of technical terms such as effect, association and correlation. Association, meaning no independence, does not imply causation (and effect). Causal effect estimation requires measurement of exposure before outcome (temporality) as well as confounding adjustment. The correlation refers to a monotonic association between two variables. Therefore, no correlation does not imply no association.

Item 28: results of prespecified analyses are distinguished from the results of exploratory analyses in the interpretation

The results obtained from the prespecified (a priori) analyses that have been already designed and mentioned in a protocol are much more reliable than the results obtained after data dredging (data-derived or post hoc analysis).

Item 29: appropriate discussion of the study methodological limitations

The methodological limitations of the study design and analysis should be discussed. Ideally, the probabilistic bias analysis, in which a probability distribution is assumed for the bias parameters and bias is probabilistically accounted for using Monte Carlo sensitivity or Bayesian analysis, should be performed for adjustment of uncontrolled confounding (eg, due to an unmeasured confounder), selection bias (eg, through missing outcome data) and measurement bias (eg, subsequent to measurement error in the exposure).114–116 The authors should at least qualitatively discuss the main sources of bias and their impact on the study results.117 118

Item 30: drawing only conclusions supported by the statistical analysis and no generalisation of the results to subjects outside the target population

The study interpretation must be based not only on the results but also in the light of the study population as well as any limitation in the design and analysis.82 For example, if the study has been done in women, it cannot necessarily be generalised to a population of men and women.