Study design

The number of conditions a study has corresponds to the number of levels the independent variable has, for example 1 condition = 1 level, 2 conditions = 2 levels, and so on. This is different from the number of independent variables a study has. If a study has multiple independent variables, each will have its own number of levels. Statistical tests can be categorized as to whether they can handle 1, 2, or 2+ levels.

If you have 2 or more levels, then the type of grouping will need to be considered: are they between-subjects* or repeated-measures**. A between-subject design means that each participant undergoes one condition whereas a repeated-measures design has each participant undergo all conditions. A third type exists called matched groups, where different subjects undergo different conditions but are matched to each other in some important way, and may be treated as a repeated-measure design when selecting for a statistical test

*Independent design = between-subjects = between-groups; not to be confused with the variable. **Dependent design = repeated-measures = within-subjects = within-groups; also not to be confused with the variable.

Variables

The characteristics of your variables, both independent (predictor) and dependent (descriptor), will help you choose which statistical test to use. Specifically, the number of each, their type, and the level of measurement, will help you narrow your choices to select the appropriate test.

Data Distribution (Parametric vs Nonparametric)

This is the third factor to keep in mind for test selection and only applicable for NON-nominal dependent variables. Parametric tests make the assumption that the population the data is sourced from has a normal or Gaussian distribution. These are powerful tests because of their data distribution assumption, with the downside of only being able to be used in select cases. Nonparametric tests do not assume a normal distribution and therefore can be used in a wide range of cases, thought they are less likely to find significance in the results.

One-Sample T-Test

Dependent Variables: 1, continuous [ratio and interval]

Independent Variables: 0 variables

Design: 1 group, 1 level

Parametric: Yes

# One-sample t-test
t.test(x, #numeric vector of data 
       mu = 0, #True value of the mean. Defaults to 0
       alternative = "two.sided", # ("two.sided", "less", "greater") depending on which you want to use. Defaults to "two.sided
       conf.level = 0.95 #1-alpha. Defaults to 0.95
)

The t.test() function allows for multiple kinds of t-tests. To use this function to perform a one sample t-test, you will need a numeric vector of the data you want to run the data on, the mean you want to compare it to, your confidence level (1 - alpha), and your alternative hypothesis. The t-test, like other kinds of t-tests, will compare the means by calculating the t-statistic, p-value and confidence intervals of your data’s mean.

t: The calculated t-statistic, uses formula : (mean(x) - mu)/(sd(x)*sqrt(n_samples))

df: Degrees of Freedom. Calculated by: n_samples-1

p: The p-value of the t-test as determined by the t-statistic and df on the t-distribution table

(conf.level) percent confidence interval: calculated interval where the true mean(x) is with conf.level % certainty.

sample estimates, mean of x: The calculated mean of x

Accepting or rejecting the null hypothesis is a matter of determining if mean(x) is outside the percent confidence interval range, if it is then the p-value will determine the significance of the results.

#Example
set.seed(10)
x0 <- rnorm(100, 0, 1) #rnorm(n_samples, true_mean, standard_deviation)

ttestx0 <- t.test(x0, mu = 0, alternative = "two.sided", conf.level = 0.95) #actual running of the t-test
#t.test(x0) yeilds the same results bc I used default values for mu, alternative, and conf.level

tlessx0 <- t.test(x0, mu = 0, alternative = "less", conf.level = 0.90)

knitr::knit_print(ttestx0) #display t-test output via knitr

## 
##  One Sample t-test
## 
## data:  x0
## t = -1.4507, df = 99, p-value = 0.15
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  -0.32331056  0.05021268
## sample estimates:
##  mean of x 
## -0.1365489

In t.test_x0, we are running a two-tailed t-test on a vector of 100 doubles generated with rnorm() where the intended mean is 0. Since this is a two-tailed test, the alternate hypothesis is mean(x0) != 0 while the null hypothesis is mean(x0) = 0

Since mu=0 and 0 is within the range (-.323, 0.050), we have failed to reject the null hypothesis with this set of data

Unpaired T-Test

Dependent Variables: 1, continuous [ratio and interval]

Independent Variables: 1

Design: 2 groups, 2 levels

Parametric: Yes

# One-sample t-test
t.test(x, #numeric vector of data 1
       y, #numeric vector of data 2
       alternative = "two.sided", # ("two.sided", "less", "greater") depending on which you want to use. Defaults to "two.sided
       conf.level = 0.95 #1-alpha. Defaults to 0.95
)

The unpaired t-test functions similarly to the one-sample t-test except instead of mu, it uses dataset y. Variance is assumed to be about equal between dataset x and y.

#Unpaired t-test example
set.seed(10)
x <- rnorm(100, 0, 1)
y <- rnorm(100, 3, 1)

unpaired <- t.test(x, y)

knitr::knit_print(unpaired)

## 
##  Welch Two Sample t-test
## 
## data:  x and y
## t = -22.51, df = 197.83, p-value < 2.2e-16
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -3.308051 -2.775122
## sample estimates:
##  mean of x  mean of y 
## -0.1365489  2.9050374

Output for an unpaired two sample t-test is similar to the output for the one-sample. Mu is typically 0 (though one could change it with ‘mu = n’ in the function call) and the test uses the confidence interval for (mean(x)-mean(y)), which is (-3.308, -2.775). Since mu does not exist within the confidence interval, the null hypothesis can be rejected.

Paired T-Test

Dependent Variables: 1, continuous [ratio and interval]

Independent Variables: 1

Design: 1 group, 2 levels

Parametric: Yes

# One-sample t-test
t.test(x, #numeric vector of data 1
       y, #numeric vector of data 2
       paired = TRUE, #defaults to FALSE
       alternative = "two.sided", # ("two.sided", "less", "greater") depending on which you want to use. Defaults to "two.sided
       conf.level = 0.95 #1-alpha. Defaults to 0.95
)

The paired t-test works for matched and repeated-measures designs. Like the unpaired t-test, it has a second data vector and does not need ‘mu.’ When running a paired t-test make sure that you specify that paired = TRUE when calling the function and that x and y have the same length.

More information about paired t-tests here

#Unpaired t-test example
set.seed(10)
x <- rnorm(100, 0, 13)
y <- rnorm(100, 3, 1)

unpaired <- t.test(x, y, paired = TRUE)

knitr::knit_print(unpaired)

## 
##  Paired t-test
## 
## data:  x and y
## t = -3.7956, df = 99, p-value = 0.0002539
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -7.126787 -2.233560
## sample estimates:
## mean of the differences 
##               -4.680174

Similar to the unpaired t-test, the paired t-test looks at the differences between x and y. Unlike the unpaired t-test, the statistic creates and runs its test on a third set of data created from the differences of x and y. For an x and y of length n, r would create a third data set of new = ((x1-y1), (x2-y2),...(xn-1-yn-1), (xn-yn)) and run its testing with mean(new) and sd(new)

Chi-Squared test

Dependent Variables: 1, categorical

Independent Variables: 1

Design: 2 groups, 2 + levels

Parametric: No

More Information on Chi-Squared tests

#How to Run a chi-squared test
chisq.test(x, # numeric vector, matrix, or factor
           y # numeric vector; ignored if x is a matrix; factor of the same length as x if x is a factor
)

Wilcoxon-Mann Whitney test

Dependent Variables: 1, (ordinal, interval, or ratio)

Independent Variables: 1

Design: 2 groups, 2 levels

Parametric: No

More information here

wilcox.test(x,#Numeric vector of data
            y #optional numeric vector of data)

Wilcoxon Signed Ranks test

Dependent Variables: 1, (ordinal, interval, or ratio)

Independent Variables: 1

Design: 1 groups, 2 levels

Parametric: No

More information here

wilcox.test(x,#Numeric vector of data
            y, #numeric vector of data from 2nd level- same length
            paired = TRUE)

One-Way ANOVA

Dependent Variables: 1, continuous

Independent Variables: 1

Design: 2+ groups, 2+ levels

Parametric: Yes

More information here

summary(aov(formula, #formula specifying the model
            data #data frame containing the variables in the formula))

Regressions

Factor Analysis

Applications of Statistical Power

One of the applications of statistical power, besides as a measure of Type II error probability, is to allow you to run a power analysis. A power analysis involves using the relationship between Effect Size, Sample Size, Significance, and Statistical Power when you have three of the four parts in order to find the value of the fourth. This is typically used to find the effect size of a study, as that is often the hardest to estimate, or to find a sample size that corresponds with a desired effect size when designing a study. Further reading on statistical power and power analysis

Effect Size

The effect size is a measurement of the magnitude of experimental effects. There are a number of ways to calculate effect size including but not limited to Cohen’s D, Pearson’s R, and Odds Ratio. If you would like to read about what BU’s SPH has to say about it, here is the SPH module link

Citations: Dorey FJ. Statistics in brief: Statistical power: what is it and when should it be used?. Clin Orthop Relat Res. 2011;469(2):619-620. doi:10.1007/s11999-010-1435-0