A Critical Review of Compensation Data from Surveys and Websites: Do you Know the Quality of the Data you are Using?

“I’m updating the organization’s compensation plan and need reliable market data. Will I need to purchase a survey or find information on the Internet to get quality compensation data?” No! You do not need to purchase an expensive survey or join an online web service to get reputable compensation data. As an MSEC member, you have access to the most reliable source of compensation survey data in the Arizona, Colorado, Utah, and Wyoming markets. MSEC has been collecting survey data since 1947!

HR professionals have access to many more sources of compensation data today than ever before. Today you can find compensation information through surveys produced by associations or work-related websites such as Glassdoor, Indeed, LinkedIn, PayScale.com, etc. However, it is essential to know the source of the compensation data before using them in your compensation plan.
Following are some critical factors an employer should consider when evaluating the quality and reliability of compensation data from surveys and websites.

1) Methodology
a. Who is reporting the compensation data?
MSEC collects compensation data from human resource or finance professionals in member organizations, not data self-reported by individuals. Many of the work-related websites noted above provide data self-reported by individuals vs. employer-reported data.

b.How were the data collected?
MSEC collects compensation data through online questionnaire, Excel spreadsheet, paper copy–whatever is most convenient for the participant. It is important to understand how the data are collected through other compensation sources you use.

c. What was the validation process?
MSEC has a three-step validation process to ensure the publication of accurate data for our members. First, we compare an organization’s data to its previous survey submission. Next, we compare all data by job from low to high. Finally, we compare the data to the previous survey report. It is important to understand how other sources validate the data submitted or if they even validate data!

d. What data are collected and displayed?
MSEC collects individual base pay, formal ranges, incentive pay, exemption status, as well as number of organizations reporting and number of incumbents by job. All data collected through MSEC’s surveys are displayed in the published report along with the effective date of the data. What data are the other sources collecting and publishing? Does the resource disclose the date of data collection or just date of publication?

e. What should you know about survey methodology?
If I am updating my organization’s compensation plan, I need to know the survey methodology, as I may be questioned by senior leadership for detail. I should be prepared to explain the statistical data and know how the organization positions itself relative to the market data. In other words, if the organization wants to pay above market, will I age the data and by what factor? Do I also know the organization’s labor market(s) we compete in and the corresponding data lines in the survey?

2) Job Matches
a. Does the survey or website contain job descriptions or job titles only?
MSEC collects salary data based on job descriptions. Chances are other surveys or websites that are collecting data based solely on job titles will have erroneous job matching–meaning bad data.

b. Survey data should be used based on job content, not job title. What percent of the job description should be matched for a valid match?
MSEC defines that participants can report an employee who spends 70 percent or more of their time in the described function. What criteria are used to match jobs with website sources or association surveys?

c. What if a job in my organization matches more than one job in a survey?
There may be jobs that are blends of two or more jobs found in the survey. If the job in your organization has duties in more than one functional area, it is matched to two survey jobs. Once the match is approved and salary survey data are pulled, the weighted averages count equally and an aggregate average is calculated.

3) Participant Base
a. How large is the participation and how consistent is it from the previous survey?
MSEC collects data from organizations throughout Arizona, Colorado, Utah, and Wyoming. Sixty-five percent of participants in the Benchmark Compensation Surveys contribute data two years in a row. What is the consistency of the data reported in website sources or association surveys?

b. What types of industries are included?
MSEC collects data across a wide array of industries such as manufacturing, government, nonprofit, health care, financial, retail/wholesale, ski, etc. What types of industries are reporting data in website sources or association surveys?

c. What is the employment size of organizations included?
MSEC collects data from employers of all sizes. What is the employment sizes of the organizations represented in website sources or association surveys?

4) Customized Reports/Extracts
a. Does the website resource or survey publisher offer customized reports of the data?
MSEC publishes data by industry type, employment size, and geographic location on MSEC’s website. MSEC offers Custom Survey Analysis (CSA) by peer group or custom demographic cuts from our surveys. And, members who participate by the survey deadline are eligible for a free CSA!

Final thoughts–It is critical to know the source of the compensation data being reported (employer or individual), and if the data are validated when using compensation information from association surveys or website sources. MSEC’s survey data meet both of these two most important criteria. MSEC publishes over 30 compensation and benefit surveys each year with the data being reported by MSEC’s member organizations. MSEC’s compensation data are extensively analyzed and validated; 80 percent of participating employers are contacted to validate their input. Many organizations want to utilize as many survey sources as possible. While that is considered a good practice, focus on survey quality over quantity.

Survey data are an important tool when updating a compensation plan, but they are not the only tool. Reliable survey data are meant to provide an indicator of external competitiveness. They must be analyzed and balanced with other elements of a successful plan design. These elements include an articulated compensation philosophy based on the organization’s business strategy and objectives, consideration of compensable factors that survey data don’t provide, and consistent pay practices.

Contact MSEC’s Surveys Department at surveys@msec.org or Compensation Consulting Services at compensation@msec.org if you have any questions on survey resources or consulting services available to MSEC members.

Thank you for your support and willingness to participate in MSEC’s Surveys. Your participation does matter!