SHAP Importance and the RF Probability
Two outputs, the z-score, and what the category radar actually measures — a note for the modeler
1 Purpose of this note
This is a short, self-contained companion to the model’s methods appendix. It clarifies three things that come up when the random-forest output is turned into the Atlas capability radar:
- The two outputs are distinct objects. The predicted competitiveness score is a single probability; the SHAP values are a per-feature attribution vector. They are linked by an exact identity — the SHAP values decompose the probability — but they are not the same quantity, and one cannot be read as the other.
- The year-specific z-score removes magnitude. Standardizing each feature’s SHAP by its own spread is a legitimate feature-selection step, but it discards exactly the information (“which capability matters more”) that a quantitative “need” axis would need.
- Summing the retained z-scores by category reduces to a count. Because every retained feature lands at roughly the same standardized value, a category total is, in practice, a headcount of selected products — not a magnitude ranking.
None of this says the model is wrong. The point is narrower and, we think, uncontroversial: a principled feature-selection output is being repurposed downstream as a quantitative importance axis, and the standardize → absolute-value → average → sum pipeline strips out the magnitude, sign, and country-specificity that a “need-vs-have” reading depends on.
2 Two outputs: the probability and its SHAP decomposition
The methods appendix lists two of the three model outputs as (1) the predicted competitiveness score and (2) the features most predictive of competitiveness. These are related but formally different objects.
2.1 The predicted competitiveness score
For a country-year with feature vector \(x\), the random forest of \(T\) trees returns a probability that the country is competitive (\(\text{RCA} > 1\)):
\[ \hat p(x) \;=\; \widehat{P}\big(\text{RCA} > 1 \mid x\big) \;=\; \frac{1}{T}\sum_{t=1}^{T} h_t(x), \]
where \(h_t(x)\in[0,1]\) is tree \(t\)’s class-1 vote. This scalar \(\hat p(x)\in[0,1]\) is the predicted competitiveness score. It is one number per country-year.
2.2 The SHAP decomposition of that score
SHAP does not produce a second, independent prediction. It explains the same \(\hat p(x)\) by splitting it into per-feature contributions that satisfy an exact additive identity (local accuracy):
\[ \hat p(x) \;=\; \phi_0 \;+\; \sum_{j=1}^{M} \phi_j(x), \]
where \(\phi_0\) is the baseline (the model’s average output) and \(\phi_j(x)\) is the SHAP value of feature \(j\) for this specific country-year — the signed amount by which feature \(j\) pushed the prediction above or below the baseline. Equivalently:
\[ \sum_{j=1}^{M} \phi_j(x) \;=\; \hat p(x) - \phi_0 . \]
So both statements are true at once, and there is no contradiction:
- They are distinct outputs — \(\hat p(x)\) is a single scalar prediction; \(\{\phi_j(x)\}_{j=1}^{M}\) is a length-\(M\) vector of signed attributions.
- The vector decomposes the scalar — by construction the \(\phi_j\) sum to \(\hat p(x) - \phi_0\).
The predicted competitiveness score is therefore not “the sum of the SHAP weights.” The SHAP values sum to the prediction minus the baseline, for one observation, and they are signed. The importance numbers used in the radar are a further, lossy summary of these \(\phi_j\), described next.
2.3 From per-observation SHAP to a category “need” score
The capability radar is built by summarizing the \(\phi_j\) across observations and then across features, in four steps:
Global importance of a feature — average the absolute SHAP over the \(N\) country-years: \[ I_j \;=\; \frac{1}{N}\sum_{i=1}^{N} \big|\phi_j(x_i)\big| \qquad(\text{mean}|\text{SHAP}|). \]
Year-specific standardization — within each year, z-score each feature’s SHAP across countries and average the absolute z across countries and years: \[ z_{ij} \;=\; \frac{\phi_{ij} - \bar\phi_{j}}{\sigma_{j}}, \qquad \bar z_j \;=\; \frac{1}{N}\sum_{i=1}^{N}\big|z_{ij}\big|. \]
Cohen’s-\(d\) selection — keep features clearing a moderate-effect bar: \[ \text{retain } j \iff \bar z_j \ge 0.5 . \]
Category score — sum the retained standardized importances within each SHAP category \(c\): \[ S_c \;=\; \sum_{\substack{j \in c \\ \bar z_j \ge 0.5}} \bar z_j . \]
Step 4 is the number plotted on each radar axis. The rest of this note is about what steps 2–4 do to that number.
3 What the z-score does to the numbers
The z-score does two things to each feature’s SHAP: it centers it (subtract the mean) and — the important one — divides it by its own standard deviation. That division is what changes the meaning: it rescales every feature to the same spread, which removes the magnitude.
Concretely, imagine two features in one year, across a handful of countries:
| Feature | raw \(|\text{SHAP}|\) across countries | raw mean\(|\text{SHAP}|\) |
|---|---|---|
| Feature A (swings predictions a lot) | 0.40, 0.20, 0.08, 0.02, 0.01 | ≈ 0.14 |
| Feature B (barely matters) | 0.04, 0.02, 0.008, 0.002, 0.001 | ≈ 0.014 |
On the raw scale, A is ~10× more important than B — a clear ranking.
Now z-score each feature (divide by its own SD). Both distributions get squeezed to spread 1, so:
| after z-score, mean\(|z|\) | |
|---|---|
| Feature A | ≈ 0.8 |
| Feature B | ≈ 0.8 |
They come out nearly identical. The z-score divided out the very thing that made A more important than B. A feature that genuinely swings predictions and one that barely moves them end up looking equally “important,” because each was rescaled by its own size.
This is not a bug in the standardization — it is what standardization is for. The appendix’s stated rationale is sound: z-scoring puts features on a comparable scale across technologies with different class balance, so a threshold can be applied uniformly. The consequence, though, is that once features clear the bar, their standardized importances are nearly interchangeable — which is what makes the category sum behave like a count (Section 4).
4 Feature selection vs. repurposing
4.1 The accurate version
It wasn’t a manipulation aimed at the classification. The z-score + Cohen’s-\(d\) threshold is a legitimate, standard feature-selection step: its job was to answer “which capabilities clear a moderate-effect bar?” and produce a clean list of important features per tech. For that purpose it’s fine — even principled.
The problem is repurposing. That selection-oriented output — standardized, absolute-valued, averaged, thresholded — then gets used downstream (in the Atlas) as a quantitative “need” axis and summed by category. It’s a feature-selection artifact being stretched into a role it wasn’t built for: a magnitude ranking. And in that role, it doesn’t carry the information the role needs.
4.2 What’s actually lost vs. retained
| Step | What it keeps | What it discards |
|---|---|---|
| z-score (\(\div\) own SD) | relative position within a feature | magnitude — which capability matters more |
| \(\lvert\cdot\rvert\) absolute value | strength | direction — helps vs. hurts |
| mean over countries | a common per-tech profile | country-specificity |
| Cohen’s-\(d > 0.5\) cut | the set of moderate+ features (meaningful!) | the below-threshold tail |
| sum by category | a per-cluster total | reduces to a count (magnitude already gone) |
So what survives is real and useful: the set of capabilities the model found at least moderately important for each tech. What’s lost — for a need-vs-have tool — is the ordering among them (z-score), the sign (absolute value), and the country dimension (averaging); and then the category sum turns the survivors into a headcount.
4.3 Why the category sum reduces to a count
After steps 2–3, every retained feature sits at roughly the same standardized value — call it \(\bar z_j \approx \kappa\) (empirically \(\kappa \approx 0.6\), since a spread-1 distribution thresholded at 0.5 concentrates its survivors in the $$0.5–0.8 band). Then the category score is approximately
\[ S_c \;=\; \sum_{\substack{j\in c \\ \bar z_j \ge 0.5}} \bar z_j \;\approx\; \kappa \cdot n_c , \]
where \(n_c\) is the number of retained features in category \(c\). Empirically this near-proportionality is essentially exact: across the shipped data, the category “need” score and the category product count correlate at \(r \approx 0.999\). The axis reads as if it ranks capability importance, but mostly it counts selected products.
4.4 The one-sentence takeaway
Rather than “they manipulated SHAP and lost information,” the fair statement is: a principled feature-selection output was repurposed as a quantitative importance axis, and the standardization / absolute-value / averaging / sum strip out exactly the magnitude, direction, and country-specificity that a “need-vs-have” gap analysis depends on. So the axis reads as if it ranks capability importance, but mostly it counts selected products.
5 Where this leaves the model, and a suggested fix
To be clear about scope: the random forest, the SHAP explanation, and the Cohen’s-\(d\) selection are all sound and appropriately documented. The issue is confined to one downstream transform — summing the standardized, thresholded importances by category to form the radar’s “need” axis — which is not part of the published method and which, for the reasons above, behaves like a product count rather than a magnitude ranking.
If the goal of the category axis is genuinely to say “how much does this capability cluster matter for competitiveness in this technology,” the most direct fix is to build it from the quantity that still carries magnitude:
- Aggregate the raw \(\text{mean}|\text{SHAP}|\) (\(I_j\) from Section 2) by category, rather than the z-scored \(\bar z_j\). Within a single technology the SHAP values are already on a common unit (one pooled forest), so no per-feature standardization is needed to compare them — and the raw values preserve the 10:1 kind of contrast the z-score erases.
- Keep the Cohen’s-\(d\) list as the selection step (which features enter), but let the magnitude come from the raw importances (how much each selected feature matters).
- If a country-specific “need” is wanted, avoid averaging \(\phi_j\) across countries before plotting — the per-country \(|\phi_{ij}|\) already localizes importance.
That keeps everything principled about the current pipeline while restoring the three things the “need-vs-have” reading requires: magnitude, and (optionally) sign and country-specificity.