Tag Archives: CPP

CPP Career Group Analytics: Action research and Artificial Intelligence (AI) analyses

By Christoff Prinsloo & Maretha Prinsloo on December 24, 2019

© 腾龙 郭/ adobe.stock.com

 

Introduction

The goal of this study differs from that reported in part 2, which focused on the information processing tendencies of high level CPP performers or high capability groups across career fields. This study involved the use of AI techniques to identify the unique processing tendencies of various career groups, regardless of the levels of CPP performance involved. Both cognitive styles as well as Information Processing Competencies (IPCs) were considered to differentiate between groups. AI findings were also compared to typical career trends which emerged from qualitative action research by means of personal feedback on CPP results (and in some cases alternative values and personality test results as well) to approximately 5,000 individuals.

Given the challenges involved in statistically analyzing psychometric data, especially those of averaging effects, AI and action research are both well-positioned to reflect more realistic findings than is the case with “blind” number crunching.

For current purposes, an Artificial Intelligence (AI) product, namely Microsoft Power BI’s Key Influencer Visualization was applied to the CPP results of a sample of 24,600 candidates. Normalized-Standardized T-scores were used. The goal was to identify dimensions that best discriminate between various career groups by identifying “key influencers”. These differentiators indicate the likelihood of a specific career group to be characterized by certain processing tendencies at specific score ranges.

The analysis sample spanned several career fields, genders, age groups, geographical areas, educational levels and CPP results as indicated below:

 

Table 1: The Career groups used in this analysis

 

Figure 1: The gender representation of the sample

 

Figure 2: The sample’s educational level and educational field

Figure 3: The geographical areas represented

 

Findings

The AI program used in this study was aimed at analyzing the differences between groups to identify the cognitive dimensions that best differentiate a specific career group from all the others.  Whereas rank ordered scores (in the case of the Cognitive Styles) and Standardized T-scores (in the case of the IPCs) were used the first part of this blog series, rank ordered Normalized-Standardized T-scores (NST-scores) were used in part 2 and in this final part of the blog series.

The likelihood that a processing score (a specific factor) predicts the career group indicated, is also provided. For example, for older Accountants there is a likelihood of 4.19 for a factor value beyond the listed threshold value to belong to the career group of older Accountants. In other words, any candidate is 4.19 times more likely to belong to the older Accountants target group if they have an Analytical score above 28, which is the threshold value.

Another example, looking at the thinking styles of younger Accountants, shows the Intuitive Style ranking as the most powerful differentiator with a likelihood of 4.21. Of all the career groups, respondents with an Intuitive style score above 77 are thus 4.21 more likely to belong to the group of younger Accountants than of respondents with lower Intuitive Style scores. The bar chart below shows the distribution of respondents in the career groups for various levels of the specific factor.

For younger Accountants, the blue bar shows that 3.53 % of the entire sample belongs to the career group younger Accountants, whereas on average, 0.69% of the sample would have belonged to younger Accountants. That ratio (3.53/0.69) roughly indicates the 4.21 factor.

The styles and IPCs that best differentiate a specific career group from the others are not necessarily the group’s highest score dimensions, but may be the opposite or alternatively, reflect a specific score range.

Examples of the way in which the best differentiators per career group are indicated appear in Figure 4.

 

Figure 4: Best cognitive Style and IPC influencers of the older and younger Accountant groups

 

The results obtained through this AI study are reflected in Table 2 and Table 3:

Table 2: Key Cognitive Style differentiators of various career groups

 

Table 3: Key Information Processing Competencies (IPCs) of various Career groups

 

An interpretation of the findings of this AI study

Accountants

The key influencer for older Accountants, or the stylistic preference that best differentiates them from other career groups is that of the:

  • Analytical style (T62 above 28 at a factor of 4.19)

And the key Information Processing Competency (IPC) influencers for this career group are those of:

  • Metacognitive Awareness and Alertness (T34 above 22 – at infinity)
  • Spontaneous comparison of elements (T09, above 80 at a factor of 5.98)

Younger Accountants (24 – 44 years of age) are best differentiated from other career groups by the stylistic preference of the:

  • Intuitive (T65 above 77 at 4.21)
  • Memory (T66 above 69 at 2.37)

And the key IPC influencers those of:

  • Extracting core elements (T23 above 80 at 41,68)
  • Speed (T56 above 75 at 9.77)
  • Learning (T44 above 72 at 8.51)

This may imply that long term involvement in an accounting career encourages the use of a high degree of Metacognitive awareness and an Analytical approach as characterized by detailed, precision and rule-based tendencies.

In the case of younger Accountants who, more so than other career groups tend to show an Intuitive stylistic approach (which to some extent overlaps with a Learning style), there is a reliance on the Memory of their knowledge base to meet the detail requirements of their jobs. Over time their Intuitive approach may therefore make way for a more detailed, factual approach to problem solving and strategizing. However, the exceptional speed, flexibility and learning orientation of younger Accountants, combined with the skill of identifying and extracting core elements, ensure highly effective cognitive functioning that basically covers most of the processing bases.

Action research aimed at identifying the cognitive characteristics of younger Accountants has indicated that most show a Tactical Strategy with potential for Parallel Processing SST complexity level. More than 70% of younger Accountants show particularly well-developed Logical-Analytical skills and preferences. The Intuitive approach of younger accountants come as a surprise, but less so given the Memory component which supports both an intuitive and an analytical application.

 

Actuaries

The key stylistic differentiators seem those of the:

  • Holistic (T64 above 63 at 6.39)
  • Memory (T66 above 66 at 5.75)
  • Analytical (T62 above 64 at 5.67)
  • Low Impulsive / Reactive (T68 less than 35 at 5.59)
  • Low Random / Trial-and-error (T69 less than 32 at 5.31) approaches.

In terms of their IPCs they are best differentiated from other career groups in terms of:

  • Metacognitive strategies for Logical reasoning (T30 above 73 at 13.46)
  • General processing approach (T04 above 67 at 7.56)
  • The use of Hypotheses (T42 above 68 at 7.07)
  • Metacognitive Strategies (T37 above 60 at 6.30)
  • Metacognitive Strategies for Exploration (T06 above 64 at 6.24)
  • The tendency to Extract Core Elements (T23 above 67 at 6.18).

As in the previous studies, Actuaries again showed a highly balanced cognitive approach in that they seem metacognitively aware, do not resort to random and impulsive approaches and accommodate both detailed and holistic information. Action research of the cognitive tendencies of Actuaries has indicated that this career group tends to obtain higher scores than most other career groups on almost all the processing scores – particularly those of Rule orientation and Categorization – which is not typical of comparable professional fields. They thus apply their thinking in a most rigorous, detailed and structured manner in a way that exceeds any of the other career groups.

 

Finance (other) Group

In this somewhat diverse career group which include Bankers, Economists, Financial advisors, Bookkeepers, etcetera, the relatively weak AI identified stylistic influencers included:

  • Impulsive (T68 below 52 at 1.65)
  • Learning (T72 above 48 at 1.58)
  • Holistic (T64 above 50 at 1.57).

The IPC influencers were:

  • Coherence (T16 above 40 at 1.67)
  • Exploration (T05 above 30 at 1.64)
  • Card Movements (T00 67-81 at 1.64)

Given the diversity of this group, averaging effects have resulted in somewhat unremarkable results. The group as a whole do, however, seem to be characterized by somewhat uneconomical exploration processes which in turn tend to lower the effectiveness of their more complex processes.

Previous research studies of the preferences and capabilities of financial people other than Accountants, included empirical and qualitative explorations. These studies were conducted in various sectors within the Financial Industry, such as Banks and Insurance firms, as well as across geographical regions and specific jobs. Interesting results, for example, include that Economists at a Central Bank, who generally show a Tactical Strategy SST level or higher, and who have postgraduate or multiple degrees, showed either a surprisingly high Random (which is an ineffective approach) or a Holistic cognitive style (which is an effective big picture approach within both practical and theoretical contexts).

Action research also indicated that almost all Economists seem to show an ideas orientation which differs from the more factual and logical approaches which is generally characteristic of Accountants. Credit Managers from Commercial versus Corporate Banking divisions showed very clear differences in cognitive approach. In one such study, 79% of Credit managers from the Corporate division of a bank showed a marked “left-brain” preference through the application of Logical, Analytical and Reflective styles in dealing with issues of corporate viability and financial structuring. Credit managers from the Commercial division of the same bank, who mostly deal with individuals and small businesses, showed a typical “right-brain” approach though a reliance on Metaphoric, Integrative and Random approaches. The Commercial Credit Managers also showed a significantly higher need for structure than the Corporate Credit managers did. Insurers from various regions seem to work at different levels of complexity (SST): Insurance Executives from South African, the United Kingdom and one Asian insurer, seemed to work at similar levels of complexity which significantly differed from that of their Australian counterparts.

Those involved in IT banking systems from South Africa and the UK also showed relatively similar cognitive profiles which clearly differed from that of the rest of SADC African banks in terms of levels of complexity and stylistic preferences. Whereas the former group prefer a Logical and Integrative approach to problem solving, the latter SADC group applied a more operational approach characterized by Explorative and Reflective cognitive styles.

 

Administrators

The key stylistic influencer seems to be that of:

  • Low scores on Logical reasoning (T67 below 21 at 6.44)

whereas the key IPC influencers include a:

  • Low degree of Card movements (T00 below 22 at 4.54)
  • Inadequate Exploration (T05 below 28 at 1.77)

The Administrators generally show under-developed cognitive skills, particularly where it comes to Exploration processes which are crucial to identify the relevant information which more complex processing relies on. Analytical skills training with administrators showed them responsive to the repeated practice and internalization of the metacognitive criteria aimed at optimizing Exploration skills, including: “Is this important?”, “Is this clear?” and “Do I need more information?”. Their cognitive effectiveness also tends to improve by developing the habit of making lists and representing information in simple structures or graphs.

 

Engineers

In the case of older Engineers, the stylistic differentiators were:

  • Explorative (T61 above 26 at 8.55)
  • Integrative (T70 above 46 at 2.25)
  • Holistic (T64 above 45 at 2.17)

The IPC influencers were:

  • Exploration (T05 above 24 at infinity)
  • Detail (T17 above 75 at 4.48)
  • Precision (T11 above 45 at 2.60)

It thus seems that older Engineers, more so than most other career groups except IT developers, invest the bulk of their energy in the detailed exploration of unfamiliar situations. Their Exploration processes (as measured by “explorative tendencies” or T05 as opposed to T43 which measures “effective exploration processes” only), may not necessarily be economical or carefully directed. They therefore tend to look for information from various perspectives and may over-explore at times. While exploring as widely and in such depth as they tend to, which reflects a detail orientation, they also manage to continually integrate relevant information to understand the situation and formulate a solution.

Compared to other career groups, younger Engineers also showed a rigorous detailed and integrative approach as indicated by their preferred cognitive styles of:

  • Analytical (T62 above 64 at 2.66)
  • Logical (T67 above 61 at 2.50)
  • Integrative (T70 above 63 at 2.46)

In terms of their IPCs, the younger Engineers could be recognized by their higher scores for:

  • Metacognitive strategies for Logical Reasoning (T30 above 69 at 2.90)
  • Metacognitive Exploration Strategies (T06 above 69 at 2.76)
  • Analytical Processes (T40 above 58 at 2.43)

Given their strong metacognitive awareness and the degree of self-monitoring of their thinking processes, young Engineers thus deal effectively and economically with new and unfamiliar information. Both the older and younger engineers do, however, show a technical and rigorous approach which to some extent differentiates them from other career groups. Action research through personal feedback, aimed at identifying the cognitive characteristics of Engineers, has indicated that this career group, more so than most others, are cognitively diverse and rely on a wide range of cognitive preferences and capabilities.

 

IT Developers, Programmers and Data Analysts

This group seem best differentiated by their cognitive style preferences for:

  • Learning (T72 above 46 at 2.09)
  • Reflective (T74 above 50 at 1.99)
  • Logical (T67 above 50 at 1.94)

Their most characteristic IPCs included:

  • Exploration (T05 above 29 at 4.82)
  • Low Coherence (T16 below 17 at 4.27)
  • Precision (T11 above 45 at 2.32).

Here the averaging effects of group diversity somewhat obscured the strong visual mode of processing and well-developed learning and analytical orientation of programmers as found through qualitative and action research investigations. The low Coherence scores, which is measured by the CPP according to various aspects of conceptualization, using auditive modes of processing, do however reflect the visual orientation of this group which has repeatedly been confirmed in case studies and personal feedback.

Action research has also indicated IT developers and programmers as the most Learning oriented of all the career groups. This group generally consists of younger people who work in fast-changing environments which tends to attract those who are cognitively adaptable and agile. Qualitative action research also indicated that besides Quick insight Learning (T15), this groups is also characterized by an Intuitive (T65 and T54) approach and their personality characteristics include openness, curiosity, a need for challenge and self-confidence regarding their intellectual prowess.

 

Lawyers and Attorneys

Those from the Legal fraternity can be expected to differ from other groups’ stylistic preferences in terms of:

  • Random (T69 below 63 at 1.48)
  • Metaphorically inclined (T75 above 65 at 1.47).

In terms of their IPCs the key influencers were:

  • A Focus on External Clues (T1 above 22 at 11.60)
  • Self-Monitoring (T36 below 66 at 2.05)
  • Task Orientation (T35 between 32 and 63 at 1.97)

Based on action research involving personal feedback to people from this career category, they generally seem to show complexity preferences for Tactical Strategy work – in some cases with “potential” for Parallel Processing work complexity. They also, almost invariably, showed the fairly unique combination of a factual, Logical with an ideas-oriented, Metaphoric cognitive styles. Other than alternative career groups such as the Accountants, Programmers and Engineers who are analytically inclined, Lawyers thus seem to be more interested in the world of ideas, they often capitalize on metaphors and analogies when explaining concepts, tend to use abstract language and apply a metacognitively rigorous approach to problem solving. The latter findings based on informal action research are thus partly verified by the AI results on this sample of legal experts.

 

The Social Sciences Career Group

For this group which includes HR practitioners, Social workers and Teachers the processing style differentiators revolved around average to above average:

  • Random (T68, 50 – 71 at 1.48)
  • Low Integrative (T70 below 50 at 1.47)
  • Low Logical (T67 below 50 at 1.47)
  • Low Reflective (T74 below 50 at 1.43)

The best IPC differentiators seemed that of:

  • Speed (T56 below 70 at 2.53)
  • Low Metacognitive monitoring of Analytical Processes (T12 below 52 at 1.74)

Compared to most other professionally trained career groups, those from Social Sciences seem somewhat less effective in terms of their cognitive competence in that this group shows relatively low metacognitive awareness, as well as low integrative and logical skill. Action research as well as quantitative research amongst social sciences groups have indicated a widely diverse group, though. In terms of the levels of complexity these individuals seem to display capabilities ranging from Pure Operational to Pure Strategic SST Levels. Their stylistic preferences and information processing competencies also widely differed. It would be misleading to generalize specific processing findings for this group.

 

Marketing and Sales

In the case of these career groups the sample showed the key cognitive style influencers to be that of:

  • Memory (T66 below 73 at 3.89)
  • Holistic (T64 above 38 at 1.59)
  • Metaphoric (T75 above 35 at 1.57)

IPC influencers were those of:

  • Low on the Degree of Detail Focused on (T17 below 20 at 6.40)
  • High Memory (T31 below 74 at 3.19).

This particular combination means that although this career group relies on detail, this detail is mostly based on previously acquired knowledge and experience (as opposed to the tendency to pull situations apart in order to understand the building blocks and their interrelationships) as opposed to fresh analyses of unfamiliar information. Within an unfamiliar problem-solving situation, they may thus fall back on a more general problem-solving approach.

Again, action research on the cognitive characteristics of people in media, journalism, marketing and advertising showed a clear ideas orientation and tendency to capitalize on verbal conceptualization (as reflected by the Metaphoric style tendency). Those involved in the formulation of creative advertising strategies, often seem to intentionally apply a particularly Random and Impulsive approach. More characteristic of this Marketing and Sales career group than their cognition, however, is their worldview as indicated by the Value Orientations (VO) assessment.

On the VO, those in Marketing roles often show an “Orange” orientation characterized by a focus on the creation of value; skill in manipulating perceptions; a belief in their capability to create their own future; personal resilience; as well as a creative strategic orientation. This worldview has been found to prevail across regions and is as prevalent in Africa as it is in Europe, Asia or the Americas. Within Sales the Red value system as measured by the VO was the most common preferences. The latter is associated with high energy as well as a degree of forcefulness and a need for achievement and proving oneself.

 

Medical Doctors

This career group was best differentiated from other career groups based on the following stylistic and processing characteristics. In terms of cognitive style:

  • Reflective (T74 above 43 at 1.68)
  • Low Impulsivity (T69 below 55 at 1.64)
  • Analytical (T62 above 44 at 1.62)
  • Quick Insight (T73 above 42 at 1.60)

In terms of their IPCs, differentiators included:

  • Spontaneous comparison (T09 above 33 at 2.20)
  • Strategizing (T37 above 42 at 1.78)
  • General approach (T04 above 45 at 1.74)
  • Detail (T17 above 44 at 1.70).

A general observation is that those in healthcare and medicine are often academically astute.  They also tend to capitalize on knowledge and experience, tend to be factual, realistic and practically inclined and, probably given the risks involved, are relatively conservative in their approach – therefore the tendency to spontaneously compare elements and to be Reflective as opposed to Impulsive regardless of their characteristic Quick Insight. The majority show awareness of the integrated nature of their subject matter by apply systems thinking. However, action research results suggest that over time, career-related fatigue often seems to result in a more linear and structured approach in solving problems, as opposed to the systemic view required for medical careers.

 

Politicians

Caution should be taken to generalize the results of this group of Politicians as the sample largely consisted of managers, director generals and others in executive roles in the public sector in South Africa. Key stylistic differentiators included:

  • Low Effectiveness in Explorative (T61 below 27 at 3.77)
  • Low Logical Reasoning (T67 below 31 at 3.50)
  • Low Reflective (T74 below 33 at 2.94)
  • High Random (T69 above 66 at 2.81).

Key IPC differentiators included:

  • Few and ineffective Card Movements (T00 below 27 at 4.30)
  • Inadequate Strategies to deal with Complexity (T25 below 28 at 4.20)
  • Low tendency to look for Logical Evidence (T29 below 29 at 3.66)
  • Undeveloped Analytical Skills (T40 below 32 at 3.16).

In terms of their results regarding cognitive complexity and ideal SST environment, the majority of these individuals showed Pure Operational and Diagnostic current levels of SST work complexity and a small proportion showed potential for Tactical Strategy work.

 

Conclusions

In constructing an evidence base for the CPP, which transcends the shortcomings of conventional Psychometrics, while also pursuing the further understanding of cognitive competence across groups, the contribution of qualitative and quantitative research techniques was also evaluated.

Based on approximately 50 formal empirical studies aimed at the validation of the CPP (as reported on in the CPP Technical Manual); countless smaller empirical  case studies aimed at quantifying and clarifying group-specific cognitive characteristics; qualitative research conducted via personal feedback to approximately 5,000 individuals where alternative test results were mostly available; and big data AI investigations, it seems that the most viable research approach in dealing with psychological data is one characterized by a focus on smaller, homogeneous and contextualized samples, where subjective insight and experience is capitalized on. Such case studies seem to produce clear and intuitively appealing results – more so than big data or even AI analyses do. It is, however, useful to combine big data AI analyses with a qualitative case-study approach to best understand the complexity and dynamics involved in cognitive functioning.

From the studies reported on in these blogs, it seems that the most effective and complex thinkers across career groups apply relatively similar cognitive approaches in that they capitalize on thinking skills “high up” in the holonic structure of the information processing model (IPM) – processes which are of an encompassing and inclusive nature. High level performers across career groups seem metacognitively aware and generally apply a Logical, Integrative and Learning orientation while capitalizing on the full spectrum of cognitive competencies.

However, it is also of interest to differentiate between the cognitive characteristics of various career groups. These differentiators are obscured by normal statistical practice which is subject to averaging effects. Artificial Intelligence (AI) techniques come to the rescue in this regard in that it reveals typical career group-specific cognitive indicators as discussed above. The AI findings seem to verify conclusions based on qualitative action research using a case study approach.

 

A summary of the findings

Cognitive characteristics of top performers on the CPP:

Styles:

  • Integrative (T70)
  • Logical (T67)
  • Holistic (T64)
  • Learning (T72)

Information Processing competencies (IPCs):

  • Learning (T44, T15)
  • Metacognition (T37)

 

Table 4 summarizes all the key features associated with the career groups analyzed in the three blogs constituting this series.

Table 4: Career specific cognitive competencies

CPP Career Group Analytics: Career related processing preferences

By Christoff Prinsloo & Maretha Prinsloo on December 18, 2019

© 腾龙 郭/ adobe.stock.com

 

Introduction

Cognadev’s CPP database which currently consists of approximately 400 000 sets of cognitive results, offers interesting insights into the intellectual functioning of various groups within the work environment, globally. The results span the cognitive preferences and complexity capabilities of candidates including their cognitive styles of thinking, information processing competencies (IPCs), units of information used and their learning potential. As such the insights gained from the data underline tendencies in cognitive functioning that are generally expected from certain age, ethnic, gender, educational level, educational field, employment category and other groupings. New insights are also offered with regards to the cognitive approaches of various career groups across regions. In this document, examples of the findings pivoting around a few interesting areas are reported.

A number of investigations on cognition have, over the years, focused on whether some commonly held beliefs about organisational roles are supported by CPP results. The aim therefore was to test whether the strengths and weaknesses traditionally associated with specific career paths held true, and whether new insights could be gained. This three-part blog series reports the results arising from an investigation of cognitive differences between various career categories such as engineers, marketers, accountants and administrative personnel, by means of three visualisation/analytics approaches:

  • In part 1 of this blog series, the processing preferences of various career groups were visualised using two-dimensional scatter plots to investigate common expectations of their cognitive functioning.
  • In part 2, the cognitive styles and information processing tendencies of top CPP performers across various career categories were analysed and visualised by means of tree structure representations to investigate the impact of career specific processing requirements and practice on cognitive functioning versus the impact of general cognitive competence on cognitive functioning.
  • In part 3, we report the results of an application of an Artificial Intelligence technique used to identify the cognitive style and information processing competencies as measured by the CPP, which best differentiate between various career groups. The findings were compared to results emerging from case study-based action research using empirical or quantitative as well as qualitative investigations.

 

Analytics Sample Information

For the purposes of this study, the following sample was selected as a target group for exploratory investigation.

  • Level-of-work: Only SST Level 3, alternatively referred to as Tactical Strategy (TS) or Alternative Paths (AP) level of work
  • Age at assessment: 35-54
  • Level of Education: Multiple Degrees

This yielded a sample size of 13,100 candidates, spread across a multitude of functional areas, disciplines of qualifications and sectoral involvement. Some examples of the many analyses that were conducted, are shown here.

 

Functional Area Analytics

Using the CPP-based Standardised T-scores, the aim of the visualisations below was to determine whether different functional areas consistently exhibited different information processing competency (IPC) profiles and differently ranked cognitive style preferences; and to what extent they mirrored expected cognitive characteristics. Results based on selected combinations of 56 IPC and 14 style scores were used. An example of the Metaphoric versus Analytical style scatter plot for various functional areas is shown below to illustrate cognitive processing characteristics of the various groups. The results represented in Figure 1 mirror our core understanding of typical job processing characteristics, with Engineering/Technical preferring an Analytical Thinking Style. Likewise, it is no surprise to see Customer Service and Teachers/Lecturers/Social scientists groups opting for a Metaphorical, verbally oriented, approach.

Figure 1: Functional groups: Metaphoric versus Analytical styles

 

The second example of the visualisations focusing on Employment Sectors depicts a two-dimensional graph or scatter plot regarding their information processing competency (IPC) scores.  The results shown in Figure 2 exhibits a plot of Judgement versus Analysis, showing IT dominating Judgement scores and again Legal, Pharmaceutical and Mining presenting top Analytical scores. The Retail and Food and Beverage as well as Consulting industries achieved lower average scores in terms of these two processing scores.

Figure 2: Sector of employment: Information processing competencies – Analytical versus Judgement IPC scores

 

Discipline of Education Analytics

The next sub-sample came in lieu of discipline of qualification. So, although an employee might have ended up in a managerial functional area, and in Retail, he/she might have had a scientific undergraduate qualification. This analysis was therefore done to investigate whether an employee’s fundamental academic training shaped their thinking, as opposed to the jobs they now fill.

Figure 3 shows an example of the scatter plot visualisations done. It again contrasts the Analytical and Metaphorical thinking styles, using discipline of qualification as the core dimension. The results indicate the clear Metaphorical thinking style for those from Marketing and Sales disciplines, while Military trained groups showed a clear Analytical thinking style preference. The long list of similar good analytical performers are the usual suspect of science and engineering. These results seem intuitively appealing.

Figure 3: Discipline of qualification: Metaphoric versus Analytical styles

 

Our final graph depicted here in Figure 4 shows Judgement versus Story telling Information Processing Competencies (IPCs). Not surprisingly, Marketers have found their niche, while Military personnel still know how make the tough calls objectively whilst the rest seem to cluster together and therefore show equally developed skills in this regard.

Figure 4: Discipline of qualification – Information processing skills of Judgement versus Story telling

 

The above findings to some extent exhibit the depth of insights that can be gained from Cognadev’s CPP database through the use of a reputable business intelligence platform. A number of interesting insights were revealed by the entire exercise. Most of these confirmed our understanding of the job market as it operates now, but the many different analyses also revealed a few surprises.

Intellectual Capital Management: Practical guidelines for an Intellectual Capital Management solution

By Christoff Prinsloo & Maretha Prinsloo on September 17, 2019

©metamorworks / adobe.stock.com

 

In this third blog of the 4-part series, we outline our practical guidelines for helping choose an assessment solution.

In order to manage talent, one needs to understand both the work-related requirements and the psychological factors involved. The assessment of these aspects offers a solid foundation for the entire value chain of people- or talent management. We recommend an assessment solution which is guided by the following principles:

  • Cognitive functioning forms an integral and important part of work performance. It refers to a dynamic, adaptive and multi-dimensional factor incorporating intellectual, motivational and consciousness factors, all of which need to be contextualised. A holistic approach is thus required for its measurement and management in the work environment.
  • Besides the psychological factors, cognitive functioning at work is also affected by a wide range of variables such as previously acquired knowledge and skills; physical health and well-being; nutritional factors; past and future exposure and learning opportunities; socio-economic background and circumstances; educational and cultural factors; relationships, love and acceptance by significant others; and spiritual factors related to a personal sense of purpose, to mention but a few.
  • The work and home environments in which a person operates are therefore crucial as cognitive functioning cannot be separated from its context or seen in isolation.
  • To leverage the concept of Intelligence in the work environment involves more than merely “how intelligent” a person is. Each person is perfectly suited to a particular kind of work. Appropriate and sufficient information is thus required to understand, position and develop cognitive functioning.
  • Conventional Psychometric test methodologies are often flawed and rely on limited techniques. The idea is to move away from such inadequate and cross-culturally loaded test practices and to cease assessing educationally acquired skills as is the case with various intelligence test methodologies.
  • More robust methodologies are required, based upon validated theoretical models and automated simulation techniques which operationalise, externalise and track thinking processes. Given the complexities involved in mental functioning, the analysis of candidate responses by means of algorithmic expert systems, AI or machine learning and fuzzy logic, will contribute to the richness in interpretations of assessment results.
  • The ultimate goal of an assessment battery remains that of discovering and mapping the unique territory of an individual mind to honour and position their distinctive repertoire of talents in a way which will facilitate the full realisation of their potential.

 

The implementation of these principles, as part of an Intellectual Capital Management solution, may involve the following action steps and products, or assessment methodologies:

1. The training and accreditation of HR practitioners in terms of the underlying theory, measurement and utilisation of the constructs which underlie competence at work. For this purpose, Cognadev provides in-depth e-Learning courses on cognition, motivational drive and levels of consciousness, otherwise referred to as valuing systems. The models and methodologies covered by these courses are at present not yet addressed by university courses.

2. Facilitated discussions between HR and line functions are required to contextualise the entire talent management approach. Here the focus should be on the nature and prospects of the industry, the organisational value- proposition, and the core competence of the organisation. The core organisational competencies will inform the job-specific competency requirements to which candidate profiles can be compared. For this purpose, functional job families also need to be identified.

3. This is followed by a job-analysis to determine the Stratified Systems Theory (SST) levels of work complexity of specific job families, as well as the identification of 10 to 12 job-related competency requirements of those roles or job families. The competency definitions need to reflect the correct SST level of work of a position or role, to inform job specs and facilitate the appointment of suitable role players. This is done by means of the Contextualised Competency Mapping (CCM) tool of Cognadev.

4. An organisational audit by means of a volume assessment system further contributes towards an understanding of the talent within the organisation. In addition, a mass recruitment exercise aimed at creating a virtual talent pool, may be useful. Together these mass assessment initiatives will resolve the typical succession problems related to crowding and vacuum. The creation of virtual talent pools will become more critical as the new world of work, characterised by a-typical organisational structures around project-based undertakings, emerges. Cognadev provides a volume assessment tool, called Cliquidity, which holistically assesses candidates and possesses the required functionalities to enable organisational audits and the creation of virtual talent pools.

5. In terms of the assessment of people, cognition, motivation as well as values and culture, need to be addressed as these three factors form a crucial foundation of any intellectual capital management solution:

  • The culture of the organisation is best determined by assessing the executive as well as representative groups of employees from various regions or functional units. Understanding the organisational culture requirements will optimise selection, placement, team compilation, leadership, developmental, and succession solutions within the organisation. For this purpose, Cognadev provides the Value Orientation (VO) assessment tool which is largely based on the Spiral Dynamics (SD) model.
  • Seeing that the proposed Intellectual Capital Management approach mainly rests on levels of work complexity, the cognitive preferences and capabilities of candidates need to be assessed. For the cognitive assessment of existing staff, Cognadev provides the Cognitive Process Profile (CPP) while the assessment of school and university leavers can be undertaken by means of the Learning Orientation Index (LOI). For purposes of mass recruitment and organisational audits, a low-cost volume assessment, the Cliquidity Adaptive Reasoning Assessment (CARA) can be used.
  • Motivation too, is a critical prerequisite of work performance. Constructs related to drive and energy; factors that could energise a person or drain their energy, self-insight, energy themes, defence mechanisms, life scripts, dynamic personality patterns, EQ and motivational patterns, amongst others, can be measured using a non-transparent assessment tool, called the Motivational Profile (MP).

6. Assessment results need to be reported upon by integrating the cognitive, values and motivational profiles of role players with the competency requirements of their work. If possible, not only the psychological results, but information on a candidate’s knowledge and experience as well as performance ratings should be covered by the integrated reports. Seeing that it can be a tiresome and time-consuming job to compile hand-written reports, Cognadev has developed an automated report generator, namely the Integrated Competency Report (ICR). These integrated person-job matching competency reports are also very useful for feedback purposes and for future performance discussions between assessment candidates and their managers.

7. Individualised feedback to test candidates contributes to their personal development as it enhances self-insight, informs their career and development decisions and optimises interpersonal functioning. Feedback can be done through the provision of written reports, or more ideally, through personal discussions with HR practitioners and/or managers. At executive levels the involvement of a psychologist or an executive coach may be required. Group feedback can also be provided and create an opportunity for team development.

8. Performance evaluations in terms of job-related competencies further contribute to Intellectual Capital Management. 360 Degree online performance questionnaires, completed by the individuals themselves, their peers, managers and subordinates, may also contribute to employee performance, self-insight and engagement. Performance discussions aimed at realistic and honest feedback on the person’s strengths and development areas are aimed at culminating in a developmental plan. The CCM job analysis tool offers a 360 Degree competency evaluation questionnaire.

9. The above-mentioned competency-based approach to talent and intellectual capital management, will enable the integration and alignment of all HR functions, including:

  • recruitment
  • selection and placement or talent acquisition and retention
  • succession planning and promotion
  • performance management
  • individual and team development
  • team compilation
  • career guidance and development
  • job structuring
  • organisational development (OD) as well as
  • remuneration and compensation.

10. The entire approach should ideally be accompanied by a ROI evaluation and reporting.

The validity of the CPP report

By Maretha Prinsloo on August 23, 2019

 

There are several factors that could render a CPP result invalid. This blog outlines some of those factors that could potentially impact on a candidate’s CPP results report.

 

The factors contributing to invalid CPP results:

There are a number of factors that could render CPP results invalid. Acceptable test conditions with regards to noise, temperature, lighting and privacy, are prerequisites for obtaining valid assessment results. Factors such as incorrect administration; inadequate language proficiency (a grade 5 mother tongue language proficiency is required for a valid CPP result); computer illiteracy; emotional factors including anxiety, demotivation, preoccupation and depression; health-related factors; disabilities and medication amongst other factors, may all affect the validity of a CPP assessment result.

The most common factors contributing to invalid CPP results are, however, anxiety, demotivation and inadequate language proficiency.

In some cases, invalid reports will either not be scored by Cognadev or will be flagged as “Validity questionable”. It is, however, the task of the CPP practitioner involved, to clarify whether a report is valid or not.

 

The most common causes of invalid reports:

Practitioners can determine the validity of CPP reports by checking for the most common cause of invalid reports, namely performance anxiety. They can also contact a Cognadev consultant for an expert opinion.

Anxiety or preoccupation, both of which may affect concentration, are usually indicated by significantly lower scores on the “Pragmatic” and “Judgement” dimensions as compared to other processing scores in the CPP report. These two scores indicate the candidate’s task focus.  Relatively low scores on “Pragmatic” and “Judgement”, in combination with a “Trial-and-error” and/or “Reactive” style, may indicate performance anxiety or preoccupation during the time of the assessment.  Please note that these two styles alone do not indicate an invalid CPP result but may actually reflect a tendency to go about problem-solving in an unplanned, reckless or superficial manner in unfamiliar contexts. In other words, “Trial-and-error” and/or “Reactive” styles may be a valid reflection of a person’s cognitive approach.

Candidates from disadvantaged educational backgrounds may show undeveloped cognitive capability. They are likely to obtain relatively low scores on the “Analytical”, “Logical” and “Judgement” processing competencies of the CPP. In other words, they are unlikely to independently analyse issues by pulling them apart and critically reason about further possibilities. Instead, there may be a tendency to memorise, rely on intuition and capitalise on previous experience. Their “Learning“and “Memory” scores are thus often significantly higher than all the other processing scores. In the case of educationally disadvantaged candidates, it should be pointed out that analytical skills can be acquired relatively easily via cognitive training aimed at the internalisation of certain metacognitive criteria. Cognadev can advise HR practitioners in this regard.

Although CPP reports characterised by significantly lower scores on “Logical reasoning” and “Verbal Conceptualisation” only, as compared to the rest of the processing profile, are not necessarily invalid, they indicate a personality- or culturally-based resistance to transformational thinking.  It thus shows a preference for the familiar and a tendency not to apply critical thinking or to reconceptualise issues. This tendency may also affect the candidate’s performance in everyday life and work. Such a profile may at times indicate a degree of demotivation.

 

The CPP has not been devised to diagnose neurological problems caused by factors such as trauma, long-term stress, substance abuse and/or psychiatric conditions. Alternative evidence of a person’s cognitive functioning may be required in the case of possibly invalid CPP reports. This can be obtained through structured interviews, performance appraisals, assessment centre evaluations or other techniques.

Invalid CPP reports nevertheless offer valuable information that can be interpreted qualitatively.

 

To read more on the reassessment of the Cognitive Process Profile, follow the link provided here: https://www.cognadev.com/reassessment-of-the-cognitive-process-profile.

Facilitated and/or Interpreted Assessments

By Maretha Prinsloo on August 15, 2019

Untitled design (15)

This article on facilitated and interpreted assessment methodologies forms the third of a four-part series on cognitive assessment techniques aimed at selection, placement and development of people in the educational and work contexts. The first part entailed a discussion of simulation exercises, the second a review of Conventional Psychometrics and the fourth part is a comparative summary of these various approaches.

Here the focus will be on Structured interviews, 360-degree evaluations as well as data scraping and artificial intelligence (AI) solutions.

Facilitated or Interpreted Methodologies

 

3.1 Structured interviews

Although there is ample evidence from meta-studies that unstructured interviews are no less effective than structured interviews in predicting work performance, the use of structured interviews is an established and widespread practice in selection and placement, especially with regards to the placement of people in key roles in organisations.

As in the case of assessment centre observations, the validity of structured interview results depends on a number of factors. The insight, skill and objectivity of the interviewer and the verbal skill, honesty, accuracy and objectivity of the interviewee are, for example, crucial factors in determining the metric properties of interview results.

Several structured interview techniques are available for the assessment of a person’s current and potential job suitability and career progress. The results of these are often assumed as indicative of the cognitive functioning of candidates. A well-known example of a structured interview technique is Gillian Stamp’s Career Path Appreciation (CPA) which is based on the Stratified Systems Theory of Elliott Jaques. According to the SST, various levels of work complexity can be identified based on the time span involved in the implementation of decisions. The CPA interview reflects the principle that both work and the human capacity to master that work, are hierarchically stratified, where higher levels entail greater levels of work complexity.

Stamp’s structured CPA interview technique consists of three sections focusing on phrases, symbols and work history. The interviewee has to select phrase cards which best describe their approach to work and then elaborate on those choices. In addition, a card sorting task using geometric symbols requires the test candidate to discover a sorting rule. This card sorting task informs the stylistic preference of the candidate. In the work history part of the interview, the candidate has to provide a chronological description of past work assignments, their personal experience of the associated level of challenge involved and the time span of these tasks.

A candidate’s CPA test result to some extent depends on the interviewer’s understanding and classification of the candidate’s explanation of their career preferences and progress to date, as well as their future ambitions. The CPA yields an indication of a person’s current- plus a prediction of their future work capacity. The latter is referred to as “mode”. The mode is determined according to empirically derived age-related progression curves which indicate levels of work progress. Stamp’s work laid the foundation for the development of a large number of relatively similar structured interview techniques.

Benefits of structured interviews aimed at evaluating cognitive competencies are that they inform person-job matching for selection and placement decisions, leadership identification and development as well as organisational development initiatives.

Structured interviews are also characterised by certain shortcomings. The subjective perceptions of interviewers and thus the challenge of inter-rater reliability, for example, remains a serious challenge. Further factors that may derail the predictive validity of structured interviews include the test candidate’s verbal skills, warmth and personality orientation. Important too is the degree of rapport between the interviewer and the interviewee. Since the test candidate is expected to report on their own performance, instead of actually performing a task, the individual’s hindsight of their own functioning may include the justification of past performance, possible overgeneralisation or exaggeration of previous work-related achievements. Cultural factors, ambition, honesty and self-image are all factors which may also skew the results.

A serious weakness of structured interviews such as the CPA is that these techniques often capitalise on the current position and the career history of the individual, which may well skew the outcome of the results. By basing a finding regarding a person’s ideal work environment on their current position, can almost be seen as a circular argument. In addition, while many people would prefer to talk about their work rather than to do an assessment, some personality types, such as introverts, and/or unassuming individuals, may be underestimated in interview situations. Skilled and wise interviewers, as well as clear scoring criteria, are therefore required in the case of structured interviews.

Regardless of possible criticism, structured interviews such as the CPA, which are linked to the requirements of work environments as specified by the SST model, are widely and effectively applied for job- and organisational structuring as well as for people-job matching, succession and remuneration purposes.

3.2 360-degree evaluations

Human Resources decisions regarding the placement, promotion, succession, team compilation, development and remuneration of people, can significantly benefit from valid performance appraisal data. Unfortunately, the latter is normally subjective in nature and typically of poor quality. The use of 360-degree competency-based questionnaires, also referred to as multi-rater or multi-source feedback techniques are thus often deployed in order to introduce some degree of objectivity in this regard; to gather various opinions and to provide anonymous feedback to role players on how others perceive their performance.

The raters involved may include the candidates themselves as well as their managers, peers, subordinates and other stakeholders. The competencies according to which performance is measured, are usually operationalised in terms of observable behaviours, and contextualised to reflect the strategic aims and core competence of the organisation as well as its culture. The evaluations are mostly online, require 20 – 30 minutes to complete, are largely qualitative and involve the identification and rating of work-related strengths and weaknesses. The results usually inform performance feedback and development plans or training initiatives.

Given the subjective nature of these interpersonal evaluations, the results are often biased and fail to meet the metric criteria of validity and reliability. 360-Degree procedures inherently also emphasise employee shortcomings which may cause demotivation and sap morale. Cumbersome data collection processes may even be involved. In addition, the use of 360-degree feedback may encourage employees to manage their image as opposed to concentrating on their own value add within the organisation. Halo effects may prevail in that those who make a favourable social impression, or who come across as extraverted, are normally regarded as more intelligent than those who communicate less, are task-focused and pursue long terms goals as opposed to immediately observable results. Whether 360-degree performance evaluations contribute to employee performance, however, remains a controversial issue.

The use of this assessment methodology has, nevertheless, been around since the 1950s and remains widely implemented in most large organisations. It seems that the use of 360-degree evaluations is, however, far from ideal in capturing the cognitive skills and potential of the candidate involved.

3.3 Data scraping and Artificial Intelligence (AI)

The use of Artificial Intelligence (AI) and machine learning techniques have recently opened up new avenues in people assessment. Within the assessment sphere, scraping, or the extraction of data from websites, has become very popular. Automated scraping of web pages is enabled by techniques such as parsing for text extraction; the harvesting of cloud-based platforms; and text pattern matching.

Given the availability of the social media profiles of most people, social media data analysis has become the most viable source of information to HR practitioners and recruiters. The techniques used to analyse personal data, mostly focus on the profiling of personality. For this purpose, the big five personality characteristics, or OCEAN framework (Openness, Conscientiousness, Extraversion, Agreeableness and Neuroticism) is mostly relied and reported on. Cognitive characteristics too are inferred from social media activities.

The information that is electronically gathered on personality and cognitive attributes can be used for diverse purposes including targeted marketing, commercial competition, political and market manipulation, as well as for HR purposes such as recruitment and the matching of people and job profiles to inform placements. Through job and resume analyses aimed at optimising job fit, for example, these techniques largely reduce the time and effort to recruit, place, develop and retain employees. Artificial intelligence (AI) and machine learning technology combined with competency analyses are thus used to significantly refine talent management practices.

Several benefits can be identified for the use of AI and data scraping techniques within the recruitment domain. For one, it seems that data scraping can potentially generate more accurate results than self-report psychometric personality tests, although this only seems to apply for screening purposes. Providers of data scraping software have also shown the superiority of their technique compared to 360-degree evaluations in that just 10 “likes” on social media can appraise a person’s profile better than his colleagues. The techniques can also be implemented quickly and effectively and thus hold significant financial and logistical benefits.

AI solutions for people assessment are, however, also characterised by certain weaknesses. Potential job applicants who are not active on social media and/or who have limited digital footprints may be excluded from employment opportunities. Especially in the case of high stakes employees and leadership roles, a more substantial assessment approach is required. Malicious scraping to steal information or use it for illegal purposes has also surfaced, as was the case with the now infamous Cambridge Analytica where personal data was leaked to politicians and marketeers to inform political manipulation strategies.

Many applicant tracking and data scraping systems are currently available. Investment in their further improvement may even render human involvement in recruitment redundant, which will undoubtedly impact on the quality of the process involve, not to mention the outcomes. In the meantime, the goal remains to fully automate high volume recruitment.

Simulation Exercises

By Maretha Prinsloo on August 15, 2019

Untitled design (15)

Introduction

Based on decades of research findings regarding the predictive validity of intelligence tests in the work environment, most Human Resource (HR) practitioners regard intellectual functioning as the best psychometric indicator of work performance. Intellectual functioning largely refers to a person’s cognitive preferences and capabilities in terms of learning, problem solving, understanding, conceptualisation, decision making and responding. Organisations thus tend to opt for the use of cognitive assessments for the selection and placement of employees.

In this series of four articles, of which this is the first, various methodological approaches to the assessment of intellectual functioning are reviewed and summarised. The articles also touch on the specific theoretical assumptions which underlie the assessment methodologies of the various schools of thought within intelligence research.

The most common approaches capitalised on in this regard are the Differential, Information Processing, Developmental, Contextualist and the Neurosciences paradigms. Whereas the most commonly applied test methodology, namely “ability testing” which is associated with the Differential approach in Psychology, focuses on domain-specific knowledge and the application of logical-analytical skills; the Information Processing approach tracks dynamic thinking processes; the Developmental approach measures the acquisition of age-related intellectual and behavioural skill; the Contextualist position analyses context- and culture specific cognitive competencies; and the Neurosciences approaches focus on brain activity.

Differential psychology assumes that “ability” reflects domain specific skills which results from a combination of hereditary and education factors. It is referred to as the “what” of intelligence. The Information Processing approach, on the other hand, concentrates on the “how” of thinking processes; and tends to externalise and track cognitive “preferences and capabilities” through means such as algorithmically driven expert systems. It is thus subject-dependent and largely transcends domain specific content. The Contextualist paradigm, which underlies the assessment of cognitive competencies, emphasise the “where” and “when” of intelligence within specific cultural- and environmental settings; and the Neurosciences approach capitalises on advanced measurement techniques including Transcranial Magnetic Stimulation (TMS), Electroencephalography (EEG), Average Evoked Potential (AEP), Functional Magnetic Resonance Imaging (fMRI), Diffusion Tensor Imaging MRI (DTI-MRI), and other techniques to study brain processes related to attention, epilepsy, and the like. The Neurosciences approaches are not addressed in this series.

The various assessment methodologies used for the assessment of the intellectual functioning of people, include:

Given the sophisticated and detailed nature of thinking process simulations, this methodology will first be addressed here, followed by a critical comparison of simulations with alternative cognitive assessment methodologies. The overall emphasis of this series of brief articles, will thus be on the simulation of thinking processes as a cognitive assessment methodology.

1. Simulation Exercises

Simulation exercises involve dealing with real life cognitive challenges. It may or may not include content specific challenges. Most of these techniques can, however, be regarded as competency-based. Given the fact that they replicate the processing requirements of actual work requirements, their metric properties of construct- and predictive validity in particular, tend to overshadow that of most other approaches to cognitive assessment.

Typical simulations include in-basket exercises, role plays and group exercises, all of which are mostly referred to as assessment center methodologies, as well as knowledge and skill-based games and situational judgement tests which assess technical skills and job-related decision-making capabilities. In addition, there are also simulation games which require the application of detailed, operationalised thinking skills, but which are largely devoid of domain-specific content and thus do not focus on previously acquired knowledge and skills. The latter simulations are best suited to assess cognitive processing tendencies and learning potential across groups and cultural contexts.

The following types of simulation exercises will now briefly be discussed: thinking process simulations, assessment centers, situational judgement tests (SJT) and gaming.

1.1 Thinking process simulations

The assessment methodology which can be described as thinking process simulations, reflects the Information Processing paradigm in Intelligence research. This assessment approach does not rely on job specific content as do most in-basket and other assessment center techniques aimed at measuring managerial or job-related skills. Instead, thinking process simulations involve unfamiliar tasks which require the application of specific information processing competencies. In other words, other than simulations which measure specific knowledge, this approach is largely subject-dependent and content-independent. The theoretical model involved, namely the Information Processing Model (IPM) forms the basis of the specific assessment techniques which represent process simulations, namely the Cognitive Process Profile (CPP) and the Learning Orientation Index (LOI) of Cognadev.

Although supervised, the CPP and LOI are largely self-administered assessments. The standardised delivery and automated scoring of these assessment tools, are aimed at producing consistent and comparable result. Extensive and in-depth reports are generated automatically. Subjective interpretations of a candidate’s performance, therefore do not apply.

The CPP and LOI represent a cognitive assessment methodology aimed at operationalising, externalising and tracking a test candidate’s thinking processes according to thousands of measurement points and feedback loops. These two assessments measure a person’s real cognitive responses to an unfamiliar assessment environment where the person has to make sense of, and meaningfully interpret, both structured and fuzzy information.

In the case of the CPP and the LOI, a test candidate can project their own preferred level of cognitive complexity onto the task (which is indicated as their preferred “unit of information”); apply a preferred stylistic approach (such as an Intuitive, Logical, Random, Metaphoric, Learning or any of the 15 cognitive styles measured) and create meaning in any way, as there are strictly no right and wrong answers in these assessment – especially the CPP. The person can also work at their own pace as time does not affect power or capability scores. This is an important consideration in cognitive test construction as speed and power are separate constructs when it comes to intellectual functioning. The undifferentiated measurement of speed and power in intelligence research also holds implications for the adverse impact of an assessment.

Furthermore, the content of the CPP and LOI is unfamiliar and not knowledge-based and therefore to some extent independent of previous educational and work exposure. Because the CPP and LOI tasks are of an equally unfamiliar nature to all test takers, without presenting the information in a decontextualised or dis-embedded manner, the possibility of group bias of the assessments, is reduced. Conventional ability testing, on the other hand, tends to capitalise on specific content or knowledge domains (such as spatial, verbal, non-verbal item content) while presenting the item content in a dis-embedded manner.

The CPP and LOI both measure cognitive “capability and preference” and predict the way in which a person is likely to perform in the work environment. These assessments do not claim to measure “ability” as alleged by the providers of IQ testing. Other than ability tests, the CPP and LOI also indicate detailed developmental guidelines that can be used for the further development of thinking processes.

The above thinking process simulations for cognitive assessment, and the available techniques namely the CPP and LOI, offer various benefits to test users, including:

  • Other than alternative test techniques such as IQ tests, assessment centers, STJs, gaming, data scraping and questionnaires, the thinking process simulations namely the CPP and LOI, are based on a sound theoretical foundation, the Information Processing Model (IPM). Besides the CPP and LOI, no other assessment methodology to date seems based on a self-contained theoretical model with construct validity.
  • The processing simulations involved in these assessments are also not interpreted by the candidate or others, but involve real problem solving performance. As in the case of questionnaires and structured interviews, the CPP and LOI thus do not require self-reporting, which introduces measurement error and the justification of past personal performance.
  • In addition, the task requirements of the CPP and LOI are not as transparent as that of questionnaires, and the results can therefore not be manipulated by the test taker.
  • The problem of subjective rater interpretation, as often is the case with assessment center and structured interview methodologies, is also resolved by the standardised and automated nature of the CPP and LOI assessments and reports, where the results are objectively and algorithmically calculated in terms of thousands of measurement points.
  • The CPP and LOI furthermore offer a fundamental solution to the limited, timed and cross-culturally loaded nature of typical IQ and ability tests.
  • The CPP and LOI do not only focus on already developed knowledge and skills as in the case of IQ tests and assessment centers but predict learning potential and the acquisition of information processing competence in the future, capitalising on the domain-free content of unfamiliar tasks.
  • Thinking process simulations as capitalised on in the case of the CPP and LOI, thus incorporate a sound theoretical basis for the measurement of learning potential by tracking and analysing learning curves as well as processing tendencies in terms of 16 criteria or characteristics of cognitive functioning, to identify strengths, weaknesses and metacognitive awareness, all of which can be addressed developmentally.
  • Unlike some structured interviews the CPP and LOI do not reduce the complex concept of learning potential and cognitive modifiability to assumptions regarding age-based prediction of potential.
  • The cost involved in time intensive assessment center and structured interview techniques, is in the case of the CPP and LOI reduced by automated and online assessment and automated reporting.
  • In the case of the CPP and LOI, the focus is on the practical utility of the results for developmental, selection and placement requirements.
  • One of the key advantages of using the CPP and LOI as opposed to alternative methodologies of cognitive assessment, is the lack of adverse impact and cross-cultural bias. The CPP and LOI rely on several design features to ensure valid assessment across groups, including:
    • allowing test candidates to apply any of 15 different stylistic approaches to accommodate for personal and cultural preferences in problem solving approach;
    • not capitalising on right-or-wrong answers, instead focusing on meaningful conceptualisation, which is scored in terms of certain processing criteria;
    • not applying time limitations, in that cognitive speed and power are measured separately;
    • the activation of auditory, visual and kinesthetic modes of processing to accommodate for individual and group differences in processing approach;
    • the avoidance of decontextualised and disembedded item content to cater for test candidates from contextual language backgrounds;
    • the use of test-train-test techniques to gradually introduce unfamiliar task requirements;
    • providing interactive feedback on performance, to track learning curves;
    • utilizing unfamiliar task content to create equal opportunities for candidates from different educational and socio-economic backgrounds;
    • requiring only low level (grade 5 mother tongue) language proficiency for those who are not linguistically skilled;
    • not measuring grammar, spelling or sentence construction skills which are largely educationally developed.
  • The CPP and LOI have been researched in-depth and the results of validity, reliability and adverse impact studies are summarized in the research manuals.
  • The CPP results of adults in the work context are commonly used for purposes of career guidance, selection, placement, development and coaching, succession, identification and development of leadership potential, as well as organisational development. The LOI, aimed at the 16 – 30 age range, is used for purposes of career guidance, bursary allocation, fast tracking, developmental and selection purposes.

1.2 Assessment center methodologies

Assessment center methodologies represent the Contextualist approach to psychological research in that the focus is on the measurement of the competencies required for effective performance within specific knowledge and skill domains.

The use of a variety of customised assessment centers is becoming common practice within organisations aiming to determine technical skills, behavioural tendencies, managerial skill and leadership potential.

Assessment center evaluations largely focus on behavioural and/or conceptual performance within domain specific areas. A variety of techniques are involved including in-basket exercises, leaderless groups, interactive group exercises, skill-specific games and case studies as well as questionnaires. Candidates who are evaluated by means of these exercises are often, but not necessarily, observed live and in real time by raters. Responses can also be evaluated by manual scoring of open-ended questions or by automated means. These techniques can briefly be described as follows:

  • Group exercises mostly involve a small group of young professionals or managers who are required to perform a pre-defined task which involves collaboration, decision making and leadership, while being observed by raters in terms of certain performance criteria. These exercises may be fairly time consuming.
  • Virtual stylised simulations, in the form of video games that are built around specific business skills, are often used for the screening of young professional candidates. These assessments are popular in large organisations where the aim is to create talent pools.
  • Both online and directly observed in-basket exercises, capitalising on real life managerial challenges, are often performed to determine managerial and/or other skills. The scores of candidates with previous exposure to managerial requirements may therefore be elevated.
  • Role plays are often used to determine behavioural skills associated with sales or leadership performance. It may also form part of an interview.
  • Questionnaires in the form of assignments that are often completed in the candidate’s own time may be used to determine the managerial insight, decision making skill and procedural approach of candidates. Here, the validity of the results may be derailed in cases where test candidates obtain guidance and advice from others.

The benefits of assessment centre methodologies include adaptability to a variety of applications, including real and online games, interviews and questionnaires aimed at measuring different competencies and skill sets. In addition, assessment centres reflect real life work requirements and capabilities, which increase the predictive and face validity involved. Given the subjective nature of rater impressions, inter-rater reliability may pose challenges, though. To alleviate this problem the assessment criteria need to be clearly operationalised and specified in detail. Another challenge is related to the fact that the performance measured by assessment centres is largely affected by previous experience and therefore not suitable for the prediction of learning potential. The use of assessment centre methodology can also be expensive and time consuming. It is, however, widely used and relied on for purposes of selection, placement, succession and the development of people in the work environment.

1.3 Situational Judgement Tests (SJTs) and gaming for screening purposes

Situational Judgement Test or Inventories, abbreviated as STJs or STIs, have been around since the mid-1900’s. These assessments capitalise on realistic workplace scenario’s for recruitment, screening and selection purposes.

The construction of STJs rely on job analyses and the opinions of job experts as most of these assessments are tailor made in terms of particular work requirements. The test content of STJs can be presented through a variety of modalities, including video, audio and printed materials. The test items normally describe work challenges where certain responses need to be selected or prioritised. The goal is to evaluate the appropriateness of a person’s responses or judgements in certain work-related situations. Behavioural tendencies are also inferred as a basis for predicting a person’s role suitability. The test content often directly reflects role-related operational tasks and decisions. STJs are usually not timed.

There are a number of benefits to using STJ assessments. First and foremost, they reflect specific role requirements as the scenario’s specified in the assessment closely overlap with work-related tasks. This normally contributes to the predictive validity of assessment results. STJs can be used to assess a variety of competency constructs, using different techniques, and are relatively easy to develop, customise, administer and score. Online STJs are most appropriate for high volume screening purposes.

The use of STJs is, however, also criticised for certain shortcomings. For one, the value-add of an STJ entirely depends on the quality of the items or the specific test. These assessments are also most appropriate for selecting candidates for operational roles as opposed to strategic or creative roles. In addition, the scoring of STJs remain problematic given the absence of objective criteria for determining the best possible answers. Job experts may, for example, differ as to the most appropriate responses to a situation. In these instances, a consensual scoring approach is often used. The latter may, however, not necessarily appreciate the potential value add of unusual, creative, intuitive or complex logical approaches, though.

Adverse impact effects often characterise STJs given their experience-based, visual, cultural and socio-economic bias. In terms of the metric properties of STJs, it seems that they may lack what is referred to as content validity in that the work samples used as items mostly fail to represent the entire required knowledge and skills base involved. In addition, their content-specificity makes it difficult to investigate their metric properties – the test-retest reliability in particular.

STJ tests, games and simulation exercises overlap and represent assessment centre methodologies.

Gamification which often involves scenario-based items has become a popular screening technique in the recruitment of job candidates. Not only competencies and decision-making skills, but also conventional psychometric constructs related to personality and intellectual functioning are inferred from these techniques. The often quick and easy to use games are usually delivered on mobile and aimed at younger generations. Not only do these techniques access a wider audience for both candidates and employers, but the data can easily be filtered and matched to the competency requirements of work to improve placement decisions. Candidates who have technological skills and experience may well achieve better scores on gamified assessments, without any assurance that the skills measured would necessarily transfer to work-related performance.

Reassessment of the Cognitive Process Profile

By Maretha Prinsloo on August 15, 2019

 

The validity of CPP results

The CPP capitalises on a person’s cognitive responses to new and unfamiliar information. A candidate’s first CPP results are thus usually the most valid, unless:

  • The candidate’s performance has been affected by emotional factors including extreme performance anxiety, stress, preoccupation and/or demotivation (note that a manageable degree of performance anxiety may even improve concentration and will thus not affect the validity of the report);
  • Physical factors related to excessive fatigue, medication, disability and/or pain, for example, have played a role;
  • The assessment took place under unfavourable assessment conditions, which may include noise, extreme temperatures, technological problems and/or other disturbances; and
  • A considerable period of time has elapsed since the previous CPP assessment, during which time the candidate may have developed further cognitive skills.

In the absence of the factors listed above, and in instances where the first CPP can be regarded as valid, a reassessment should be postponed by at least 5 years or more if possible. An exact time frame for valid reassessment is, however, difficult to specify.

At times it is, however, useful to re-administer the CPP to determine the impact that developmental initiatives, work exposure, maturity, changes in attitude, self-confidence and interests may have had on a candidate’s cognitive functioning. It may also be useful to reassess those whose existing CPP reports are of questionable validity.

 

What if there are several sets of CPP results available?

When several sets of CPP results are available for one candidate, qualitative interpretation by a skilled practitioner is required. Cognadev consultants can assist in this regard. Cognadev consultants normally also link the most valid set of results to the client’s account. Accredited practitioners may, however, request access to all the various sets of results of a particular candidate.

 

The potential impact of CPP reassessment

Certain processing scores are more easily affected by reassessment than others. The “Learning”, “Speed”, “Judgement” and “Memory” scores of a second or third set of CPP results are often somewhat elevated – but not for all candidates. Other dimensions are, however, more resistant to change. These include the “Potential level of work” indication, as well as the “Units of information”, “Complexity” and “Integration” scores.

It seems that candidates who prefer familiarity are more likely to obtain higher scores when reassessed, whereas candidates who tend to seek cognitive challenge and who achieved strategic profiles with a first CPP assessment, may not find a repeated exposure to the task as engaging. The latter candidates may, therefore, not apply themselves as rigorously as they initially did. This may result in somewhat lower reassessment scores for such candidates. Their first set of CPP results, thus, remain the most valid.

 

Statistical evaluation of CPP reassessment

Even though reassessment may affect the validity of the assessment results, the CPP test-retest reliability studies on a homogeneous sample of n = 87 and heterogeneous samples of n = 2724 and n = 475, have indicated test-retest Gower similarity indices of around 0.7 to 0.9 for the cognitive styles, processing competencies and level of work results. These reliability studies are reported on in more depth in the CPP Technical Manual. To read up more on the evidence-based research on the CPP, please take a look at Cognadev’s Technical Report Series.

Strategies to develop analytical and strategic thinking

By Maretha Prinsloo on June 26, 2019

 

This blog follows on the principles of organisational and individual learning entry which focused on the different ways in which organisations can extend the traditional concept of learning by implementing principles of “learning organisations”(See the link to navigate to this blog included here: https://www.cognadev.com/principles-of-organisational-and-individual-learning). In this previous entry, it was found that organisations can enhance learning by focusing on the development of employees’ cognitive thinking processes. 

In order to influence the cognitive development of employees within an organisation, it is necessary to start by assessing the current and potential cognitive processing capabilities and preferences of individuals and their teams. Two of the assessment techniques available for this purpose are the Cognitive Process Profile (CPP) and the Learning Orientation Index (LOI) as provided by Cognadev. Both these tools offer automated simulation exercises which operationalise, externalise and track thinking processes according to thousands of measurement points. The results are algorithmically interpreted by expert systems and comprehensive reports on an individual’s cognitive functioning and metacognitive awareness are generated. Metacognition refers to an awareness of one’s own thinking processes as well as the use of certain metacognitive guidelines or criteria to direct and evaluate these thinking processes.

The cognitive requirements of the organisational work environment may also be assessed by means of the Contextualised Competency Mapping (CCM) tool, which allows for person-job and team-job matching to guide learning and development initiatives. The CCM tool is also provided by Cognadev.

All of Cognadev’s cognitive assessments and training initiatives are based on a theoretical model of thinking processes. More so, the cognitive developmental initiative itself is guided by the application of a particular methodology anchored in metacognitive awareness.

 

The theoretical foundation for the Cognitive Process Profile (CPP) and the Learning Orientation Index (LOI):

The Information Processing Model (IPM) is the theoretical model on which both CPP and LOI assessments are based. It can be represented as follows, whereby the various cognitive processes are organised holonically, and guided from a metacognitive perspective.

 

The Information Processing Model (IPM):Metacognitive Criteria Illustration

 

The following metacognitive criteria guide the application of all processing skills:

Metacognitive Criteria Illustration

 

Taking account of the above, the development of both analytical and strategic thinking skills will briefly be explained here.

Additionally, once the strengths and development areas, stylistic preferences and the complexity capabilities of candidates have been assessed, suitable cognitive development programmes can be planned and implemented. These should ideally involve facilitated group exercises which capitalise on real, job-related cognitive challenges.

 

The importance of metacognition in developing cognitive skills:

Analytical thinking and Systems / Strategic thinking skills courses should ideally entail two to three days of intensive facilitation. This should also be followed up in the longer term in collaboration with peers and mentors from the work environment. Besides, as these two thinking skills programmes are of a purely cognitive focus, a broader perspective to develop emotional and consciousness factors in addition to these cognitive skills should also be offered. This will accommodate for the highly integrated nature of human functioning. Most participants are able to embrace these broader principles with relative ease, whereas the development of advanced cognitive skills requires long term practice aimed at the internalisation of metacognitive criteria.

 

1. The development of analytical thinking skills:

Analytical skills are particularly useful for dealing with the challenges of operational environments. Due to the dynamics and fuzziness of practical situations, everyday problem-solving challenges are often vastly more complex than the available theoretical models. Analytical problem-solving skills are thus required. The good news is that these can be developed with relative ease.

Specifically, analytical skills development entails the activation of metacognitive guidelines to deal with domain specific, job-related, problem-solving challenges. A methodology aimed at transferring and internalising specific metacognitive criteria is recommended.

The learning outcomes of analytical skills development normally includes the following capabilities, including the ability to:

  • Explore situations in terms of the metacognitive criteria of relevance and clarity;
  • Establish an appropriate level of detail versus generality at which to deal with a task;
  • Work with the required detail and precision;
  • Compare various task-related aspects spontaneously;
  • Identify and apply rules systematically;
  • Differentiate between elements;
  • Identify relationships and link related aspects;
  • Order and structure information coherently to make sense;
  • Integrate new and discrepant information into existing frameworks;
  • Restructure maps and models to accommodate emerging requirements;
  • Contextualise own solutions, models or ideas appropriately; and
  • Show metacognitive awareness of all of the above cognitive processes.

 

Analytical skills training thus involves the assessment of each candidate’s current cognitive preferences and capabilities. Their metacognitive awareness is of particular interest. Once these profiles have been investigated, strengths and development areas in terms of metacognitive awareness can be identified at the individual and team level. A metacognitive “voice”, which best reflects a person’s strength, can then be allocated to that individual member. For example, one person may take on the role of “The voice of relevance”, another “The voice of coherence”, or “The voice of clarity” (refer to the IPM).

 

Furthermore, analytical skills training sessions should be facilitated by a trained professional who understands the dynamics involved in cognitive functioning. Exercises in these sessions centre around dealing with real job-related challenges. Typically, during this facilitated process, those holding certain metacognitive “voices” have the authority to interrupt the group’s problem-solving process at any time, in order to create an awareness of a neglected metacognitive criterion. The facilitator’s role will, therefore, include to capitalise on such insights as well as to reinforce the internalisation of specific metacognitive criteria or “voices”.

The full training process is thus aimed at the “automation” or internalisation of metacognitive criteria by participants to guide their thinking processes in future. Once all group members have practised their metacognitive strengths, the roles are reversed so that they take on their underdeveloped metacognitive “voices”. This facilitated workshop should be followed up with applicable projects and include the involvement of mentors or peers to further consolidate the metacognitive skills acquisition of each person.

Whereas analytical thinking involves a focus on the constituent parts of systems, strategic / systems thinking processes reverse the relationships between the parts and the whole. In other words, in the case of systems thinking, the parts can only be understood in terms of the dynamics of the whole.

 

2. Building strategic / systems thinking capabilities:

Systems thinking remains a critical prerequisite for the strategic viability of organisations. This skill is unfortunately seldom taught within educational and training contexts, where the focus is mostly on the transfer of knowledge and skills. Unless systems / strategic thinking awareness is cultivated within an organisation, the principles of learning organisations will not be adopted by a critical mass of people. This may result in the gradual deterioration of learning values and ideals into mere ‘window dressing’.

According to a systems approach, the world is regarded as an integrated, dynamic whole, or a series of nested sub-systems forming a holon. In agreement with Ken Wilber (2007), who popularised Arthur Koestler’s concept of holons, the entire universe is organised such that subsequent systems levels include and transcend their predecessors.

The development of systems thinking skills to optimise strategic thinking is thus aimed at the acquisition of integrative, intuitive and holistic cognitive capabilities. As in the case of analytical thinking training, pre-assessment of delegates’ cognitive capacity and preferences will guide the specific design of a systems thinking development initiative.

A Systems / Strategic thinking course normally involves a focus on real organisational challenges in terms of four phases of analysis and conceptualisation, namely:

  • An in-depth investigation of the strategic challenges of the organisation to generate ideas, externalise positions and polarise perspectives. The complex information generated is then analysed according to a typical, matrix-like technique (Kepner-Tregeo, 1997), using unique and appropriate evaluation criteria. At the end of the first phase, most participants will fully understand the strategic challenges faced. These challenges can then be defined and metaphors selected to represent their underlying dynamics.
  • The next phase involves a dynamic analysis of the root causes of these strategic challenges, and the identification of leverage points and catalysts of change. Here the definition of strategic challenge is analysed in terms of the four most appropriate archetypes as proposed by Peter Senge (1990).
  • The following step involves creative strategy formulation, whereby unusual techniques are used to stimulate creativity. Initially, this is often met with resistance from rationally trained executives and managers. However, in retrospect, these creative exercises are often regarded as the most liberating and enlightening aspect of the course. Various intuitive techniques are practised – most of which have been proposed by the intuitive Sue Mehrtens (2002).
  • Consolidation of the entire process takes place by evaluating the extent to which the formulated creative strategy – aimed at dealing with the organisation’s strategic challenges – can be implemented and contextualised. Factors such as organisational structure, processes, power dynamics and measurement options to track intangible progress, are considered.

 

Thus, to summarise, in the case of analytical thinking, elements are isolated, the nature of interrelationships are investigated, a detailed and precise approach followed, single variables are modified, and a linear or structured approach applied. Systems thinking, on the other hand, involves unifying interrelationships, investigating the effects of interaction, considering groups of variables, and is best suited to dealing with vague, fuzzy, complex, and dynamic information. Systems thinking focuses on purposes and objectives and considers the impact of the duration of time.

Nevertheless, the analytical approach is often criticised for its somewhat reductionist and factual nature, and the systems approach, characterised by holistic thinking, thereby emphasise the different sides of the proverbial coin. Both are equally valuable components of individual and organisational learning and performance.  Ultimately, all learning is rooted in culture and values – constructs that are best described by the construct of consciousness.  Organisational and individual learning initiatives should, therefore, be approached in an integrative manner, by simultaneously taking into consideration the impact of cognitive preferences and the valuing systems or cultural memes rooted in consciousness.

 

An in-depth description of the above mentioned cognitive training processes can be found in http://integralleadershipreview.com/15984-15984/.

 

 

 

Bibliography:

Beck, DE & Cowan, CC (2005). The Spiral Dynamics: mastering values, leadership and change. Wiley-Blackwell.

Kepner, CH & Tregeo, BB, (1997). The New Rational Manager. Princeton Research Press.

Mehrtens, Sue E (2002). Intuition overview. Unpublished article.

Mindell, Arnold (2010). Process Mind: A User’s Guide to Connecting with the Mind of God. Quest Books. Theosophic Publishing House, Wheaton IL.

Prinsloo, M. & Prinsloo, R. (2018). The assessment and development of Analytical and Systems thinking skills in the work environment. Integral Leadership Review, November. http://integralleadershipreview.com/15984-15984/

Prinsloo, M & Barrett P (2013). Cognition: Theory, measurement, implications. The Integral Leadership Review, June. http://integralleadershipreview.com/9270-cognition-theory-measurement-implications/

Prinsloo, M (2012). Consciousness models in action: Comparisons. The Integral Leadership Review, June http://integralleadershipreview.com/7166-consciousness-models-in-action-comparisons/

Senge, Peter (1990). The Fifth Discipline (2nd Edition). Cornerstone Import.

Wilber, Ken (2007). The Integral Vision. Shambhala Publications: Boston.

 

Understanding Human Consciousness: Theory and Application

Maretha Prinsloo
October 08, 2018

The study of consciousness attracts the attention of psychologists, philosophers and scientists. It is, however, mostly dealt with in a descriptive and speculative manner, without explaining the nature of the subjective experience and the dynamics involved.
For the full article click the icon below.
Equifinal profiling