How To Improve Reliability Psychology

Why You Absolutely Must Understand The Basics Of Psychological Measurement

Posted October 29, 2017

• Find a therapist near me

Those of us in the business of psychological measurement use the terms reliability and validity a lot. You’ve probably seen those terms on the Psychology Today website and elsewhere. You might have some sense of what it means for a psychological test to be reliable or valid. You have probably assumed that a good test must be both reliable and valid .

But what are reliability and validity exactly, how do we assess reliability and validity, and why are these properties of psychological tests so crucially important? In this and a following blog post, I hope to answer these questions in a totally non-technical way, avoiding statistical language as much as humanly possible. If I succeed, you will see why understanding measurement reliability and validity is so important for judging the usefulness of an IQ or personality test. Many psychological “quizzes” on the Web have absolutely no evidence of reliability or validity, so you should not take them seriously. Even the claims about the reliability or validity of professionally-developed tests are sometimes overstated. Your understanding of reliability and validity from reading this blog post may help you to recognize when this happens and to use caution before accepting results based on overstated claims.

How Can Validity And Reliability Be Improved

How can validity be improved?

The validity of the research findings are influenced by a range of different factors including choice of sample, researcher bias and design of the research tools. The table below compares the factors influencing validity within qualitative and quantitative research contexts :

Validity should be viewed as a continuum, at is possible to improve the validity of the findings within a study, however 100% validity can never be achieved. A wide range of different forms of validity have been identified, which is beyond the scope of this Guide to explore in depth .

The chosen methodology needs to be appropriate for the research questions being investigated and this will then impact on your choice of research methods. The design of the instruments used for data collection is critical in ensuring a high level of validity. For example it is important to be aware of the potential for researcher bias to impact on the design of the instruments. It is necessary to consider how effective the instruments will be in collecting data which answers the research questions and is representative of the sample.

How can reliability be improved?

In qualitative research, reliability can be evaluated through:

In quantitative research, the level of reliability can evaluated be through:

Types Of Validity In Psychological Testing And Research:

FACE VALIDITY: A type of measurement validity in which an indicator makes sense as a measure of a construct in the judgment of others, especially in the scientific community. This is a term used to characterize test materials that appear to measure what the tests author desires to measure.

CONTENT VALIDITY: Measurement validity that requires a measure to represent all the aspects of the conceptual definitions of a construct. It is estimated by evaluating the relevance of test items, where each item must be a sampling of information the test purports to measure.

CRITERION VALIDITY: Measurement validity that relies on some independent, outside verification.

CONCURRENT VALIDITY: measurement validity that relies on the preexisting and already accepted measure to verify the indicator of a construct. It indicates the process of validating a new test by correlating it with some present source of information.

PREDICTIVE VALIDITY: Measurement validity that relies on the occurrence of a future event or behavior that is logically consistent to verify the indicator of a construct. It is the extent to which the test is efficient in forecasting and differentiating behaviour in a specified area under actual living conditions

FACTORIAL VALIDITY: This method utilizes factor analysis techniques. A test has high factorial validity if it is a measure of one functional unity to the exclusion of other elements.

Recommended Reading: Is Chemistry Or Physics Harder

Assessing The Validity Of Test

Assessing the Validity of Test

There there are two main categories of validity used to assess the validity of test : Content and criterion.

What is face validity in research?

Face validity is simply whether the test appears to measure what it claims to. This is the least sophisticated measure of validity.

Tests wherein the purpose is clear, even to naïve respondents, are said to have high face validity. Accordingly, tests wherein the purpose is unclear have low face validity .

A direct measurement of face validity is obtained by asking people to rate the validity of a test as it appears to them. This rater could use a likert scale to assess face validity. For example:

It is important to select suitable people to rate a test . For example, individuals who actually take the test would be well placed to judge its face validity.

Also people who work with the test could offer their opinion . Finally, the researcher could use members of the general public with an interest in the test .

The face validity of a test can be considered a robust construct only if a reasonable level of agreement exists among raters.

It should be noted that the term face validity should be avoided when the rating is done by “expert” as content validity is more appropriate.

What is construct validity in research?

How Is Reliability Measured

In order for a diagnosis to be considered reliable, it should remain relatively constant over time, assuming the symptoms have not changed. This can be established using test-retest, meaning the same patient will be diagnosed twice, a number of weeks apart. If their symptoms are highly changeable, it may be impossible to make a reliable diagnosis. Furthermore, an individual should receive the same diagnosis when re-diagnosed by another practitioner, assuming of course that they are using the same version of the same classification system!

In practice, psychiatrists make their diagnoses, having gathered information about their patients through the use of unstructured, clinical interviews meaning patients may provide differing descriptions to different practitioners dependent upon many factors. Given that psychiatrists base their diagnosis upon the subjective interpretation of what a patient has said, it is understandable why the process may lead to unreliable labeling.

Recommended Reading: What Are Probes In Biology

Additional Information And Declarations

The authors declare there are no competing interests.

Granville J. Matheson conceived and designed the experiments, analyzed the data, contributed reagents/materials/analysis tools, prepared figures and/or tables, authored or reviewed drafts of the paper, approved the final draft.

Data Availability

The following information was supplied regarding data availability:

Data is available at GitHub .

What Is Reliability And Its Importance

Reliability importance is a measure of how much impact each component has on the overall reliability of the system. One simple way to demonstrate reliability importance is to look at a series system. In general, the least reliable component in a series system has the greatest effect on the reliability.

You May Like: What Is Groynes In Geography

Making A Level Psychology Easier

Assessing and improving reliabilityAssessing and improving reliability of observers

• Training observers in the observation techniques being used and making sure everyone agrees with them.
• Ensuring behaviour categories are correctly and objectively operationalised. This means that the behaviour being observed can only be that behaviour. For example, aggressive behaviour is subjective and not operationalised, but pushing is objective and operationalised.

Assessing and improving reliability of psychology testsAssessing and improving internal validityAssessing and improving external validity

• Population validity refers to the extent to which the results can be generalised to groups of people other than the sample of participants used. Much psychological research uses university students as participants, e.g. Asch , and it is difficult to say for sure that the results can be generalised to anyone other than university students.
• Ecological validity refers to the extent to which the task used in a research study is representative of real life. Research into eyewitness testimony, for example, has generally lacked ecological validity as participants viewed incidents on video screens rather than in real life.

Assessing and improving validity of psychology tests

PsychTeacher – the number one site for A Level Psychology

What Is Reliability In Psychology And Why Is It Important

The Indeed Editorial Team comprises a diverse and talented team of writers, researchers and subject matter experts equipped with Indeed’s data and insights to deliver useful tips to help guide your career journey.

Researchers use many methods to assess and improve the reliability of their work, and they consistently re-evaluate their processes to ensure effectiveness. Reliability in psychology helps researchers conduct tests and studies in a consistent fashion. If you want to ensure that the results of your research studies and psychological testing are more dependable, you may want to learn more about reliability in psychology. In this article, we discuss what reliability in psychology is, why it’s important, methods researchers use to assess the reliability of tests and studies and tips for improving reliability in your own work.

Related:22 Different Types of Psychology

Also Check: What Is Toposheet In Geography

Effects Of Practice And Learning

Such effects will depend upon the content of the test, length of the interval, and upon the examinees experiences during the interval. For example, if some months have elapsed between two administrations of an educational achievement test, different people may have had different amounts and qualities of instruction during the period.

Relationship Between Reliability And Validity

Both measurement reliability and validity need to be in sync and coherent for the research to be accurate, authentic and empirical. It is important to note that a reliable measure may or may not be valid. But, valid measure is essentially/necessarily reliable. Therefore, reliability is necessary for validity. Because an unreliable measure will certainly be invalid. If people receive different scores on the same test everytime they take it, such a test is not likely to predict anything. Ergo, even if a test is a reliable measure it is not mandatory that it will be a valid measure too.

Don’t Miss: How To Make Math Symbols With Keyboard

I Then Ask The Following Questions And Elicit The Following Answers From The Students:

Did you achieve inter-rater reliability?

No, there was too much variability in scores.

What are the implications of poor inter-rater reliability?

The care a patient receives is dependent on who judged them and not on their actual functioning. One rater might have recommended inpatient hospitalization whereas another might have seen psychotherapy as unnecessary.

How could I have achieved better inter-rater reliability?

Training the employees on how to use the Global Assessment of Functioning Scale could have enhanced reliability.

Once inter-rater reliability is achieved is it maintained over the course of time?

No, rater drift can occur. Rater drift is when the raters return to their previous tendency of rating.

How can we prevent rater drift and ensure inter-rater reliability?

Using Evidence To Evaluate The Reliability Of The Dsm

Now we have some idea about how psychologists talk about reliability and diagnosis, lets see what research evidence there is on this topic:

Chop up the studies in the following worksheet and sort them into 2 piles according to whether you think they are about reliability or validity of diagnosis. Then resort the pile of studies that you think are about reliability into whether they suggest diagnosis can be made reliably or not:

Validity and Reliability Studies for sorting activity: R and V study squares

We will come back to the validity pile later! As you consider each of the studies on reliability, think about possible GRAVE points that you could make about these studies.You can now use these studies to answer the following question as the question requires you to ASSESS, this requires you to make a judgment about reliability, its an 8 marker and so needs to follow ATCHOOBC.

Assessment Tasks:

You should also be able to answer questions such as:

3. Explain ONE issue regarding the reliability of diagnosis using classifications systems such as the DSM4TR or DSM5

Also Check: What Is Fk In Physics

The Importance Of Establishing Reliability

Establishing reliability in psychological testing is crucial. This is because, without it, people’s conditions may not be accurately diagnosed and, as a result, they will not be provided with the appropriate treatment.

The timing of the test can also affect its reliability, particularly when implementing the test-retest method. If the researchers don’t wait long enough between tests, then the participants may remember information from the first test that can bias their answers to the second. Conversely, if the time between tests is too long, the participants’ situations may have changed to the extent that it can bias the results.

For instance, if the subject being tested is depression and its effects, some participants may have begun treating their condition with medication in between the first and second tests. Such a treatment can skew the results of the second test if the participants report a reduction in symptoms leading to the alleviation of their depression. If the symptoms aren’t there anymore to study, then the results of the test have been compromised.

What Are Some Ways To Improve Validity

References

American Educational Research Association, American Psychological Association, &

National Council on Measurement in Education. . Standards for educational and psychological testing. Washington, DC: Authors.

Cozby, P.C. . Measurement Concepts. Methods in Behavioral Research .

California: Mayfield Publishing Company.

Cronbach, L. J. . Test validation. In R. L. Thorndike . Educational

Measurement. Washington, D. C.: American Council on Education.

Moskal, B.M., & Leydens, J.A. . Scoring rubric development: Validity and

reliability. Practical Assessment, Research & Evaluation, 7. .

The Center for the Enhancement of Teaching. How to improve test reliability and

validity: Implications for grading. .

Don’t Miss: What Is The Opposite Of Biological

Example : Reliability Can Be Approximated For Applied Studies From A Test

AZ10419369 is a radiotracer for the serotonin 1B receptor, which is widely expressed in the brain and an important target in depression . In the test-retest study published with this ligand , using the frontal cortex as an example, it showed what is considered a high mean BPND , a favourable absolute variability and a good coefficient of variation . However in the sample measured, the ICC was very low: 0.32. It was concluded that can be explained bya low between-subject variance. Thus, despite the low ICC values, it cannot be excluded that the test-retest reliability is also high in these regions .

We can calculate the reliability of the new study using the results of the test-retest study by assuming the same measurement error between the studies. In this way we obtain a reliability of this tracer and for this particular correlation equal to 0.93. From this analysis, the outcome of the study can be considered reliable in terms of measurement error, and that individuals can be easily distinguished from one another by their outcome measures. This conclusion also holds after taking into account the use of partial volume effect correction in this study .

Demonstration of the outcomes of Example 1.

How Reliable Is Dsm 5

Mad in America are a not-or-profit organisation calling for profound change to the current drug-based paradigm of care which they say has failed our society. On their website, Rachel Cooper explains that the DSM III required much higher Kappa scores than the DSM 5. She explains that in the DSM 5 field trials, , figures which previously would have been deemed poor or unacceptable were now seen as good. Robert Spitzer the chairman of the DSM III task force chose 0.7 as the threshold for good agreement and some of the most common disorders seen in adults achieved values of 0.8 and over in DSM III however, the DSM5 task force suggested that values as high as 0.8 would be miraculous and note that values of 0.4-0.6 are realistic but values of 0.2-0.4 are acceptable. This has led to concerns about the reliability of DSM5.

With regard to research and clinical practice, does it really matter whether a diagnosis is reliable?

Before you go any further make sure you have answered the following questions.

Read Also: How To Find Ksp Chemistry

Validity In Quantitative Research

VALIDITY refers to as truth fidelity or authenticity of any psychological instrument. In psychological research, it is referred to as MEASUREMENT VALIDITY.

MEASUREMENT VALIDITY: it explains how well an empirical indicator and the conceptual definition of the construct that the indicator is supposed to measure fit together. It refers to how well the conceptual and operational definitions mesh with each other. The better the fit, the greater the measurement of validity.

Validity is part of a dynamic process that grows by accumulating evidence over time, and without it, all measures become meaningless.