In our second and previous installment of our summer blog series contrasting "expert” versus ”robot” approaches to conducting a pay-equity study, we described how the more traditional, consultant-led approaches (i.e., expert) differ from newer, automated approaches (i.e., robot) in the initial phases of a typical study: project kickoff and data cleaning. In our third installment, we examine how these approaches to pay equity differ in the planning and analysis phases.
Planning
In the planning phase of a pay-equity study, critical decisions about two important aspects must be made: (1) how to group employees for the analysis and (2) which legitimate, pay-related factors should explain differences in employee pay within those groups (and, in turn, be included in the study).
How to group employees represents the most important decision in a pay equity study. Under a Title VII framework, employees should be placed into similarly situated employee groups (SSEGs), and these groups should be analyzed separately. A given SSEG should contain employees who perform similar work duties with similar levels of responsibility requiring similar skills and qualifications. One consideration for how to group employees may involve the size of the groups: Larger groups enable more robust statistical analyses, whereas smaller groups may require the use of alternative, less flexible analytic methods.
Also important is the decision of which legitimate pay factors to include in the pay equity study. Pay-related factors (e.g., time in job, prior experience, job performance) should be job-related and consistent with the organization’s compensation philosophy and practices. In practice, these factors should explain why, for instance, two employees in the same SSEG are compensated differently. Some factors may be relevant to all SSEGs, whereas other factors may only apply to a subset of SSEGs.
Representatives from the compensation team, human resources, and legal can provide important context and feedback on these topics. This team can discuss potential employee groupings that maximize statistical coverage (i.e., groups large enough for statistical analysis) while maintaining the integrity of the SSEGs. They can also advise on which pay-related factors should be included in the study. The final analysis plan will clearly define the SSEGs and specify the pay-related factors to include in the study.
An expert will have experience guiding these discussions and eliciting critical information to help make decisions about SSEGs and pay factors. In addition, an expert will have the flexibility to consider multiple employee grouping frameworks (or options for defining SSEGs) and different statistical models (i.e., different pay-related factors for different SSEGs). Experts will have the knowledge to understand nuanced situations that may require the inclusion of advanced quantitative factors, such as quadratic or interaction terms, or structural factors like department or job family. An expert can also help navigate the complexities of planning a Title VII study while helping an organization’s compensation team realign its compensation philosophy. That is, experts can work with the compensation team to differentiate among legitimate pay factors that presently explain differences in pay, factors that the organization would ultimately like to explain differences in pay, and factors that the organization no longer wishes to explain differences in pay.
Because one goal of robot approaches to conducting pay-equity studies is often to minimize or eliminate the role of the expert from the process, robot approaches may only offer general guidelines or one-size-fits-all methods for grouping employees and selecting legitimate pay factors to include in the study. In fact, some robot approaches may simply offer users a set of fields (that happen to exist in an organization’s HRIS) as options for legitimate pay factors to include in the study without offering appropriate guidance for which to include and under what circumstances. The user interfaces of some robot approaches may actually discourage analysis planning and encourage users to pick and choose at whim which pay factors to include for different groups.
When a robot approach is used to conduct a pay-equity study, decisions on how to group employees may be driven less by the Title VII definition of an SSEG and more by the technology’s capability. For example, many robot approaches require that SSEGs be sufficiently large— so large that employees with very different skills or levels of responsibility are grouped together—so that they meet sample-size requirements (i.e., have enough employees) to conduct a regression analysis (a statistical analysis commonly conducted in pay-equity studies). Putting the issue of inappropriately grouping dissimilar employees together aside, analyzing SSEGs that are too large can prevent the detection of real pay disparities that may exist in smaller pockets within the broader SSEG. Even controlling for different structural variables in the analysis like job level, job family, or division is insufficient to circumvent this issue.
Because robot approaches may lack the capability to include advanced quantitative factors (e.g., quadratic and interaction terms), these factors may never be addressed or considered in the planning phase—even when they may be critical to include in the study. Without them, models may be misspecified and the results may not be meaningful.
One of the downsides of relying on the robot approach for a pay-equity study is that there are many instances where experts are really necessary to help organizations guide critical, complex decisions about how an analysis should proceed. And, frequently, there are changes to an organization’s workforce, hiring practices, compensation practices, etc., that may necessitate critical adjustments to an analysis plan. Organizations should not assume that it’s safe to leverage the previous advice of an expert (or past analysis plans) with a robot approach in the future, without additional guidance from an expert. We caution users of robot approaches that the simplicity they offer can mask the complexity involved in carefully planning an effective pay-equity study.
Analysis
After finalizing the analysis plan and processing the data, the next step in conducting the pay-equity study involves executing the analysis. While regression allows one to control for pay-related factors such as tenure and performance before evaluating race and gender differences in pay, regression is not the only analytic method available and should not be prioritized over the appropriate grouping of employees. Regression requires groups of a certain size with that threshold increasing as the number of control factors increases. Certainly, there are employees who should be placed in SSEGs too small for a regression (e.g., executives, senior leaders). In those situations, other statistical and non-statistical methods are perfectly appropriate.
An expert can use different statistical methods depending on the size of the SSEG, which is important to ensure that overly broad groups are not formed to fit only one type of statistical method. An expert can interpret model diagnostics (e.g., R-squared), the appropriateness of including different pay-related factors (e.g., Are they predictive of pay? Do they predict pay in the expected direction?), and the incremental effects of including additional pay factors to a model (e.g., Does adding job performance help explain pay?).
An expert can also adjust models for different SSEGs based on the regression results. If a regression is going to be used to generate salary adjustments, these model indicators become especially important. Adjustments generated from a poor model or based on factors not predictive of pay or that predict pay in unexpected ways (e.g., higher performance relates to lower pay) could result in adjustments inconsistent with the organization’s pay practices and could serve to exacerbate group differences.
A robot may or may not generate information on the model fit and the appropriateness of including certain pay factors. Even if a robot provides information on model fit, the appropriateness of including certain pay factors, or some guidance around interpreting warnings or issues, it’s unlikely that a robot can share information on or investigate issues further and provide remedial guidance. At worst, users may receive no such warnings about issues from a robot approach and may proceed under the false assumption that the results from the robot’s approach are sound and valid.
Additionally, a robot may have limited or no capability for utilizing analytic methods other than regression, such as exact tests and non-statistical comparisons that are suited to small groups. So, portions of an organization’s population may either go unanalyzed or be inappropriately grouped with dissimilar employees to ensure that they are covered.
Although a robot approach may make the act of executing a pay-equity analysis simple and allow a user to reanalyze compensation efficiently at different in points in time, it’s unclear that users can be guaranteed that the models are correctly specified and appropriate, and that they will actually help organizations make sound decisions later on when it comes to potentially making pay adjustments.
In our next installment in the series, we will cover investigating the results and making pay adjustments.
