0
$\begingroup$

I have a matched sample created from a larger nonrandom sample using an ex post facto design. I've been told that you should never generalize to a population from nonrandom samples.

I've read that you can sometimes (if the nonrandom sample is large enough) create a representative sample by means of a sampling frame, or through calibration weighting. However, since I have no information on the population distribution, and so there's nothing to calibrate to.

Are there other ways to generalize findings if the sample differs meaningfully from the population?

$\endgroup$

1 Answer 1

1
$\begingroup$

An effect from a nonrandom sample only fails to generalize to a population when there is effect modification by the covariates that cause selection into the nonprobability sample. This is likely the case, however, so it's right of you to be cautious in generalizing your findings. When you don't have the raw population data, it can be challenging to perform the calibration or target weight estimation.

If you know the moments and some other features of the population you want to generalize to, then you can use that information to estimate weights using optimization. These features might include the means and standard deviations of covariates, their medians or quantiles, or correlations among covariates in the population. This can be found out through papers that have attempted to describe the population of interest. You can use optimization to estimate weights that satisfy the constraints that represent the differences between the characteristics of the weighted sample and those of the target population. This method has not been extensively used but a version of it is described by Josey et al. (2020). Another version of this is available in the R package optweight using the optweight.svy() function. It is currently an experimental method.

If you truly know nothing about the population you want to generalize to, then there is nothing you can do. You just have to explain that the estimated effect represents a causal effect, but perhaps not the causal effect of interest. There is a recent tradition of focusing on estimating causal effects for unspecified target populations; for example, work by Li and others on ATO weights focuses on estimating a treatment effect for treated and control units most similar to each other rather than for members of a specific population.

$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.