How to make your data valid and reliable
Over the millennia, there have been two truly international languages: music and mathematics. We will talk about the second one here. But, keep the first in mind – music is a sustaining and balancing outlet for everyone as they live and work.
“All my boss cares about are the numbers,” some mutter. Sometimes true. But without data, your knowledge about business operations, what is working well and what is not working is left to judgment. And judgment is easily influenced by bias.
So, even if you hire someone to manage data, you still need basic knowledge to understand what they present to you, to ask relevant questions and to speak to stakeholders and potential investors.
Financial data is relatively straightforward. There are revenues, expenses, profits and losses. Although nearly every year, there is a story showing that financial data can be manipulated to look better than it actually is.
Quality or outcome data is more complex because there can be many different measures. Two factors to keep in mind in choosing quality and outcome data are: Validity and Reliability.
- Validity refers to whether the data measures what it is designed to measure.
- Reliability refers to the stability or consistency of the measure.
Two examples will be the best way to explain these concepts.
Customer satisfaction: Customer satisfaction is an important measure for any business. A valid measure of this would be customers’ responses to a rating scale: On a scale of 1-5 with 5 being very satisfied, how satisfied were you with the product/service?
At the other extreme, knowing the age or gender of customers would not be a valid measure of satisfaction. It may be important data for other reasons but it is not a measure of satisfaction.
But is the valid question reliable? Would it yield consistent results across various situations or persons? How does one know that the question will be interpreted the same by different people? One person’s sense of being “satisfied” may be different from another person’s. If the question has to be asked in a different language, the concept of “satisfied” may be understood differently amid other languages or cultures.
For this reason, a more reliable assessment of customer satisfaction would include multiple questions. How promptly was your request/purchase/problem handled? How responsive was the representative to your questions? How likely would you be to recommend this product/service to a friend?
Education: Improving learning in community members is a common goal for many products or services. Because learning is a complex concept, it is challenging to develop valid and reliable measures. Having a clear sense of the target of the product or service will help to develop such measures.
Testing a group of people before and after the intervention is a valid way of measuring whether learning occurred. There is, however, short term and long term learning. One could perform well on the test immediately after the intervention but then forget what one learned weeks or months later. So, a more valid measure might be to add a re-test 6 months after the intervention to see if learning was retained.
Test reliability will vary depending on the subject matter. If one is teaching a group of people a set of information (for example, how to use a spreadsheet to track expenses), it will be relatively easy to develop a reliable test. One can develop problems that the participant needs to solve using data, a spreadsheet and the spreadsheet tools.
If, however, one has a more behavioral skill to teach — such as anger management in parents – it will be more difficult to develop a reliable measure. Asking parents if they got angry with their children is not a reliable measure because one cannot presume honest answers in all parents. Self-report by children about their parents’ behavior has the same problem.
School reports of children who seem more withdrawn or angry at school may be an indication of how parents are behaving at home. This measure has both validity and reliability problems. There are many reasons why children behave the way they do in school.
Having others observe parents and children may be more reliable but people behave differently when they are observed than when they are alone. Also, how do you know that each observer will evaluate the parent’s behavior the same way?
In this case, multiple measures will lead to more reliable data. Long term research also will help to identify which measures might be best for behavioral goals such as parenting skills.
Think about what is practical: In addition to validity and reliability you must consider what is a practical way to collect data. A 50 item questionnaire or 60 minute interview will provide a wealth of data about customer satisfaction or parenting skills. But only if you have enough people to participate in this. A 10 item survey or a 5 minute (or even shorter) interview is likely to gather more data even if the data is incomplete.
Lastly, don’t try to reinvent the wheel. There may be research or past experiences with measurement tools that you can use to evaluate your product or service.
Some questions to think about:
- What concepts are important to measure in your business?
- How do you know that your data is valid? reliable?
What could help you develop more practical data measures?