Which Pages Should a Content Team Fix First?

Introduction & Problem

A content team maintaining thousands of pages cannot review all of them on any regular cadence. Someone has to decide which pages get a reviewer's limited time this cycle, and right now that decision usually runs on recency or gut feel, not on the signals that actually distinguish a page worth fixing from one that's fine as-is.

This project treats that decision as a ranking problem: given a page's search visibility, click-through rate, position, and freshness at the time of review, how should pages be ordered so the top of the queue is where the reviewer's time pays off most? We are not trying to explain why Google ranks a page where it does, and we make no claim about the search algorithm itself — only about which observable, pre-decision signals are associated with a page needing attention.

Data

Two FlyRank releases were used, at different stages of the project. All data below is pseudonymized at the client and content level; no client names, domains, URLs, or raw queries appear anywhere in this paper or its supporting notebooks.

Starter release — used for the model that ships

A 30,000-row cross-sectional pull (content_refresh_anonymized.csv), one row per pseudonymized page across 32 pseudonymized clients, 44 columns. The label used throughout — is_declining_label, derived from a provided trend_direction field — has a base rate of 0.542. This is a single snapshot, not a time series; that limitation is carried through the whole analysis rather than glossed over.

Warehouse release — a methodological dry run, not the shipped model

A genuine fact table (fact_content_daily_performance) hosted on Hugging Face, March 2026 partition: 9.84 million page-day rows across 55 clients and 331,437 content items. Here we built a proper future-window setup — features from the first half of March, label from the second half — to prove a cleaner label design is buildable, and to run a deliberate leakage test (see Methodology). The shipped model does not use this release directly; it uses the starter release's proxy label instead, for continuity with the baseline built earlier in the project. We name that gap directly rather than blur the two releases together.

Excluded from every feature set

• content_id / client_id — used only to group the train/test split, never as a model input.
• trend_direction / trend_pct — the source of the label itself; including them as features would leak the answer.
• Any raw target-window metric, and all raw client identifiers, domains, URLs, or search queries.

Methodology

Label

is_declining_label = (trend_direction == "down"), base rate 0.542. This is a proxy for "this page may need attention," not a causal or independently verified outcome — a page can carry this label and still be performing acceptably by other criteria.

Baseline: a transparent rule, no training

Before modeling anything, we audited two signals for whether they actually moved with the label. CTR against search position was confirmed: the share of low-CTR pages climbs from 0.79 near the top of the results to 0.96 by position 50+. Staleness was mixed: decline rate rises through the 91–180 day freshness window, but reverses in the 181+ day bucket — a bucket with only 174 pages, too small to trust on its own. The baseline rule combines both signals with visibility and reachable position:

Model

A logistic regression (class-balanced) trained on 20 numeric and 8 categorical features — log-scaled traffic and impressions, position, CTR, engagement and scroll rate, content age, days since last update, word count, and tiered versions of several of these — checked against a random forest of similar feature access and against the baseline rule, all three scored on the identical held-out split.

Validation design

We split by client, not by row: GroupShuffleSplit, 75/25, so no client's pages appear in both train and test. This matters because pages from the same client share portfolio-level quirks a model can memorize instead of generalizing from. Training set: 22,885 rows across 24 clients. Test set: 7,115 rows across 8 clients never seen in training, base rate 0.517.

Leakage checks

1. A forbidden-feature scan on the final feature set found nothing label-adjacent.
2. Removing the single strongest feature (log_impressions_90d) and retraining changed average precision by less than a point (0.607 → 0.600) — no one feature is silently carrying the label.
3. Re-running the same model on a naive random split (not grouped by client) reported precision@50 = 0.88 — 12 points higher than the honest, grouped-split number of 0.76. That gap is the cost of letting a client's pages leak across train and test; we report the grouped number everywhere in this paper.
4. On the warehouse release, planting a deliberate leak column equal to the label pushed ROC AUC from an honest 0.587 to a suspicious 1.000 — confirming our leakage check catches an obvious leak before it ships.

Results

All three methods below are scored on the exact same 7,115 held-out pages, from 8 clients that never appeared in training. Base rate on this test set: 0.517.

Reading the top 50 pages the model flags, 38 of them (76%) are genuinely declining by our proxy label — up from 21 of 50 (42%) for the rule it replaces, on the identical 50-page slot, identical clients, identical split. A random forest with access to the same features did not beat either the logistic regression or, on precision@50, even the simple rule — we tried more complexity and rejected it rather than ship it because it looked more sophisticated.

We also ran the shipped model on a naive random split, the kind of validation that's easy to reach for by default. It reported precision@50 = 0.88, a full 12 points above the honest, client-grouped result. We report only the grouped number in this paper; the gap itself is one of the more useful findings here, and one worth watching for in any future model built on this kind of client-partitioned data.

Limitations & Honest Framing

• Proxy label, not ground truth. The label used throughout is derived from a provided trend field, not an independently verified outcome. A page can carry the "declining" label and still be performing fine by other measures.
• About 1 in 4 top-50 flags will be wrong. Precision@50 = 0.76 means roughly a quarter of the top of the queue is not actually declining by our proxy. This is a triage aid, not a verdict on any individual page.
• One snapshot, one portfolio. The model trains and tests on a single 30,000-row cross-sectional pull from 32 clients. It has not been tested across other verticals, other time windows, or a genuine future-window label — our warehouse work shows that harder setup is buildable, but it is not what shipped here.
• Reason codes describe, they don't prove. The queue's reason codes are rule-based explanations layered on a probabilistic score — a plausible reason a page scored high, not proof that reason caused the decline.
• Freshness and age features are snapshot-frozen and would need to be recomputed on every scoring run in a live system, not treated as static.
• The staleness signal itself is mixed, not confirmed — it rises through 91–180 days, then reverses in a 181+ day bucket too small (174 pages) to trust on its own.
• We do not claim anything about Google's ranking algorithm, about causal refresh impact, or about any specific client, page, domain, or query. Every claim in this paper uses observed, directional, or decision-support language.

Ranked Recommendations

The held-out predictions feed a five-code priority queue. Sorted purely by model score, the queue surfaces near-dead pages first — 0% CTR, almost no traffic, technically "confidently declining" but with nothing left to recover. Those correctly route to monitor-only. The actionable queue filters those out and sorts what's left by opportunity value, then model confidence — this is the order a reviewer should actually work.

The 91-day freshness threshold behind STALE_UPDATE_DUE and AGING_REACHABLE is not arbitrary — it triangulates our own signal audit (the 91–180 day freshness tier has the highest observed decline rate, 0.611) against FlyRank's own published research, using its more stable 31–90 day freshness-refresh window rather than a headline number built on a single-page denominator (see Limitations).

Intended use

A ranked review queue for a human content reviewer or strategist to work top-down. Not for unattended automation: never auto-publish, auto-delete or deindex, make client-facing guarantees, or bulk-edit from this queue without review — especially on high-stakes or YMYL content. A human still needs to check content accuracy, brand voice, and context the model can't see, such as seasonality, discontinued products, or legal and compliance constraints.

Reproducibility

Every number in this paper traces back to a committed notebook or a committed JSON metrics file in the project repository — nothing here is a one-off calculation that can't be re-run.

• work/notebooks/capstone.ipynb — the notebook this paper is drawn from
• w04_baseline_score.ipynb — baseline rule and signal audit
• w05_model.ipynb — model training, comparison, held-out split
• w06_validation_audit.ipynb — random-vs-grouped split audit, leakage checks
• w07_action_playbook.ipynb — the final ranked queue
• w05_model_metrics.json and w07_playbook_summary.json — committed metrics receipts backing every number above

Split: GroupShuffleSplit by client, 75/25, random_state=42. Model: LogisticRegression(class_weight="balanced", max_iter=1000).