By Maretha Prinsloo on August 15, 2019
This article on facilitated and interpreted assessment methodologies forms the third of a four-part series on cognitive assessment techniques aimed at selection, placement and development of people in the educational and work contexts. The first part entailed a discussion of simulation exercises, the second a review of Conventional Psychometrics and the fourth part is a comparative summary of these various approaches.
Here the focus will be on Structured interviews, 360-degree evaluations as well as data scraping and artificial intelligence (AI) solutions.
Facilitated or Interpreted Methodologies
3.1 Structured interviews
Although there is ample evidence from meta-studies that unstructured interviews are no less effective than structured interviews in predicting work performance, the use of structured interviews is an established and widespread practice in selection and placement, especially with regards to the placement of people in key roles in organisations.
As in the case of assessment centre observations, the validity of structured interview results depends on a number of factors. The insight, skill and objectivity of the interviewer and the verbal skill, honesty, accuracy and objectivity of the interviewee are, for example, crucial factors in determining the metric properties of interview results.
Several structured interview techniques are available for the assessment of a person’s current and potential job suitability and career progress. The results of these are often assumed as indicative of the cognitive functioning of candidates. A well-known example of a structured interview technique is Gillian Stamp’s Career Path Appreciation (CPA) which is based on the Stratified Systems Theory of Elliott Jaques. According to the SST, various levels of work complexity can be identified based on the time span involved in the implementation of decisions. The CPA interview reflects the principle that both work and the human capacity to master that work, are hierarchically stratified, where higher levels entail greater levels of work complexity.
Stamp’s structured CPA interview technique consists of three sections focusing on phrases, symbols and work history. The interviewee has to select phrase cards which best describe their approach to work and then elaborate on those choices. In addition, a card sorting task using geometric symbols requires the test candidate to discover a sorting rule. This card sorting task informs the stylistic preference of the candidate. In the work history part of the interview, the candidate has to provide a chronological description of past work assignments, their personal experience of the associated level of challenge involved and the time span of these tasks.
A candidate’s CPA test result to some extent depends on the interviewer’s understanding and classification of the candidate’s explanation of their career preferences and progress to date, as well as their future ambitions. The CPA yields an indication of a person’s current- plus a prediction of their future work capacity. The latter is referred to as “mode”. The mode is determined according to empirically derived age-related progression curves which indicate levels of work progress. Stamp’s work laid the foundation for the development of a large number of relatively similar structured interview techniques.
Benefits of structured interviews aimed at evaluating cognitive competencies are that they inform person-job matching for selection and placement decisions, leadership identification and development as well as organisational development initiatives.
Structured interviews are also characterised by certain shortcomings. The subjective perceptions of interviewers and thus the challenge of inter-rater reliability, for example, remains a serious challenge. Further factors that may derail the predictive validity of structured interviews include the test candidate’s verbal skills, warmth and personality orientation. Important too is the degree of rapport between the interviewer and the interviewee. Since the test candidate is expected to report on their own performance, instead of actually performing a task, the individual’s hindsight of their own functioning may include the justification of past performance, possible overgeneralisation or exaggeration of previous work-related achievements. Cultural factors, ambition, honesty and self-image are all factors which may also skew the results.
A serious weakness of structured interviews such as the CPA is that these techniques often capitalise on the current position and the career history of the individual, which may well skew the outcome of the results. By basing a finding regarding a person’s ideal work environment on their current position, can almost be seen as a circular argument. In addition, while many people would prefer to talk about their work rather than to do an assessment, some personality types, such as introverts, and/or unassuming individuals, may be underestimated in interview situations. Skilled and wise interviewers, as well as clear scoring criteria, are therefore required in the case of structured interviews.
Regardless of possible criticism, structured interviews such as the CPA, which are linked to the requirements of work environments as specified by the SST model, are widely and effectively applied for job- and organisational structuring as well as for people-job matching, succession and remuneration purposes.
3.2 360-degree evaluations
Human Resources decisions regarding the placement, promotion, succession, team compilation, development and remuneration of people, can significantly benefit from valid performance appraisal data. Unfortunately, the latter is normally subjective in nature and typically of poor quality. The use of 360-degree competency-based questionnaires, also referred to as multi-rater or multi-source feedback techniques are thus often deployed in order to introduce some degree of objectivity in this regard; to gather various opinions and to provide anonymous feedback to role players on how others perceive their performance.
The raters involved may include the candidates themselves as well as their managers, peers, subordinates and other stakeholders. The competencies according to which performance is measured, are usually operationalised in terms of observable behaviours, and contextualised to reflect the strategic aims and core competence of the organisation as well as its culture. The evaluations are mostly online, require 20 – 30 minutes to complete, are largely qualitative and involve the identification and rating of work-related strengths and weaknesses. The results usually inform performance feedback and development plans or training initiatives.
Given the subjective nature of these interpersonal evaluations, the results are often biased and fail to meet the metric criteria of validity and reliability. 360-Degree procedures inherently also emphasise employee shortcomings which may cause demotivation and sap morale. Cumbersome data collection processes may even be involved. In addition, the use of 360-degree feedback may encourage employees to manage their image as opposed to concentrating on their own value add within the organisation. Halo effects may prevail in that those who make a favourable social impression, or who come across as extraverted, are normally regarded as more intelligent than those who communicate less, are task-focused and pursue long terms goals as opposed to immediately observable results. Whether 360-degree performance evaluations contribute to employee performance, however, remains a controversial issue.
The use of this assessment methodology has, nevertheless, been around since the 1950s and remains widely implemented in most large organisations. It seems that the use of 360-degree evaluations is, however, far from ideal in capturing the cognitive skills and potential of the candidate involved.
3.3 Data scraping and Artificial Intelligence (AI)
The use of Artificial Intelligence (AI) and machine learning techniques have recently opened up new avenues in people assessment. Within the assessment sphere, scraping, or the extraction of data from websites, has become very popular. Automated scraping of web pages is enabled by techniques such as parsing for text extraction; the harvesting of cloud-based platforms; and text pattern matching.
Given the availability of the social media profiles of most people, social media data analysis has become the most viable source of information to HR practitioners and recruiters. The techniques used to analyse personal data, mostly focus on the profiling of personality. For this purpose, the big five personality characteristics, or OCEAN framework (Openness, Conscientiousness, Extraversion, Agreeableness and Neuroticism) is mostly relied and reported on. Cognitive characteristics too are inferred from social media activities.
The information that is electronically gathered on personality and cognitive attributes can be used for diverse purposes including targeted marketing, commercial competition, political and market manipulation, as well as for HR purposes such as recruitment and the matching of people and job profiles to inform placements. Through job and resume analyses aimed at optimising job fit, for example, these techniques largely reduce the time and effort to recruit, place, develop and retain employees. Artificial intelligence (AI) and machine learning technology combined with competency analyses are thus used to significantly refine talent management practices.
Several benefits can be identified for the use of AI and data scraping techniques within the recruitment domain. For one, it seems that data scraping can potentially generate more accurate results than self-report psychometric personality tests, although this only seems to apply for screening purposes. Providers of data scraping software have also shown the superiority of their technique compared to 360-degree evaluations in that just 10 “likes” on social media can appraise a person’s profile better than his colleagues. The techniques can also be implemented quickly and effectively and thus hold significant financial and logistical benefits.
AI solutions for people assessment are, however, also characterised by certain weaknesses. Potential job applicants who are not active on social media and/or who have limited digital footprints may be excluded from employment opportunities. Especially in the case of high stakes employees and leadership roles, a more substantial assessment approach is required. Malicious scraping to steal information or use it for illegal purposes has also surfaced, as was the case with the now infamous Cambridge Analytica where personal data was leaked to politicians and marketeers to inform political manipulation strategies.
Many applicant tracking and data scraping systems are currently available. Investment in their further improvement may even render human involvement in recruitment redundant, which will undoubtedly impact on the quality of the process involve, not to mention the outcomes. In the meantime, the goal remains to fully automate high volume recruitment.