Validity is the cornerstone of psychometrics. Simply put, validity is the extent to which a scale measures what is proposes to measure (a construct). Most texts identify 3 types of validity.
Content validity is the extent to which an empirical measurement reflects a specific domain of content. Here, a research must assemble the fully elaborated content of a domain, athleticism for instance, and bring together all relevant content, such as strength, endurance, agility, coordination, strategic understanding of the game and its rules, experience in the sport, enthusiasm for the sport, etc. Then the research must strategically sample from these areas, items that can be brought together to assess athleticism. Keep in mind that some areas may be more important or complex than others, knowing that some areas will be over sampled.
Criterion validity captures the extent to which some test predicts
an outcome or phenomenon outside of the test and can be divided into 2
types according to time:
Concurrent (present) Criterion validity focuses on processes
currently occurring outside of the test. For instance, people who
do well on a test of athleticism are stronger than people who do poorly
on athleticism.
Predictive (future) Criterion validity focuses on the relationship
between the test and related future outcome. For instance, the SAT
test is supposed to give people an idea of future college performance.
Construct validity takes into account the theoretical context of a measure, the bigger picture, or the extent to which a test is related to other measures consistent with the phenomenon being measured. This kind of validity is especially useful when the thing being measured is very fuzzy or theoretical, and cannot be attached to a particular content (as described above) or outcome (as described above). In this case of athleticism (which is not a good example here because athleticism has so much content and outcome attached to it), people who score high at our test of athleticism are more likely to be on a sports team.
Finally, one may bring these types of validity together with research design to begin assembling support and evidence for validity in a handy matrix where each validity / design cell can be filled with a relationship that represents support (or lack there of) for your study in the context of what is known about your test / measurement.
Let us say that an athleticism test is known to be related to gender
in the U.S., (women score higher than men because laundry, cooking, and
cleaning count as athletic events). And let us further say that the
relationship between gender and athleticism is a theoretical aspect of
our research where we say that high levels of estrogen are a fundamental
aspect of success in any team experience. One could then administer
this test in another country (new population) and upon finding a similar
gender difference, conclude that there is a form of construct validity
for this test of athleticism in this new population (another country).
