scripts/research.py:
- New --differentiators CLI flag accepts a comma-separated list of brand
USPs (e.g. "women-owned, 24/7 service, no hidden fees"). Flows through
to research output as research.differentiators for the writing agent
to enforce verbatim in body content + AI Summary Nugget.
- New extract_missing_spokes() walks the top 3 ranking competitors'
internal-link anchors, filters out generic navigation (Home, Contact,
Privacy, FAQ, Login, social media, etc.) and broken-markdown image-
link leakage ([](href) nesting), and outputs a ranked
missing_spokes list. The list is the client's build-order priority
for filling the topical-silo gap.
- Compact + brief outputs both surface differentiators and missing_spokes.
scripts/lib/massive.py:
- _parse_markdown now returns a links list ([{text, url}]) from standard
[text](url) syntax. Skips image links, hash anchors, mailto/tel/javascript.
scripts/lib/dataforseo.py:
- New _extract_links static method walks page_content.main_topic and
secondary_topic, pulling anchor_text + url from primary_content[].urls.
- content_parse output now includes links field matching MassiveClient.
SKILL.md (v1.9.0 -> v1.9.1):
- Execution Protocol step 2 brief template gains Brand Differentiators /
USPs field. New paragraph instructs the agent to STOP and ASK the user
for differentiators if not provided up front.
- Section 12 Hub & Spoke Internal Linking gets a new Missing Spoke
Detection subsection requiring every generated page to append a
"## Recommended Spoke Pages" block built from missing_spokes data.
- Section 14 checklist expands 48 -> 51 points:
#49 Decision Fit (heading structure maps to buyer stage)
#50 Brand Identity (differentiators verbatim in chunks + nugget)
#51 Topical Silo (Recommended Spoke Pages block appended)
Passing threshold raised to 42/51.
references/quality-checklist.md: new v1.9.1 section detailing the three
new checks. Top-section reference updated to 51-point.
README.md, CHANGELOG.md, CLAUDE.md: version bumped, release-notes block
added, capability list updated. Historical version blocks restored to
their version-of-the-time checklist sizes (28, 34, 38, 41, 45, 48)
after over-greedy replace_all in prior commits.
Tests:
- 16 new tests in tests/test_research_v191.py covering --differentiators
parsing, domain normalization, generic-anchor filtering (including
nested-image-link leakage regression test), missing-spokes extraction
(same-domain filter, top-N respect, empty-input safety), markdown
link parsing in MassiveClient, and topic-tree link extraction in
DataForSEOClient.
- All 6 test files green.
Live smoke-tested against airport parking JFK with both flags:
- differentiators populated in compact output
- missing_spokes returned 12 semantic anchors after filtering
(SpotHero for Business, Reserve your spot, Parking details by lot,
EV charging stations, Learn about the JFK AirTrain, etc.)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Massive (render.joinmassive.com) as the primary competitor content
parser when MASSIVE_API_TOKEN is configured. Returns clean rendered
markdown including JS-loaded content, which DataForSEO's
content_parsing/live endpoint has always missed.
Architecture:
- scripts/lib/massive.py wraps the /browser endpoint and outputs the
same shape as DataForSEOClient._extract_content (title, word_count,
headings, plain_text_size) so it's a drop-in replacement.
- scripts/research.py branches on Massive availability per URL. If
Massive errors or returns empty for any single URL, that URL falls
back to DataForSEO -- a partial Massive outage cannot break a run.
- env.py and .env.example pick up MASSIVE_API_TOKEN. New
creds["has_massive"] flag.
Scope intentionally narrow:
- SERP organic, PAA, and keyword data continue to come from DataForSEO.
- Massive's /search endpoint as of v1.9.0 only returns "also-searched"
query suggestions, not organic results. Tested directly to confirm
before scoping the integration.
Observability:
- research output now carries a content_parsers field summarizing
which parser handled each URL (e.g. {"massive": 4,
"dataforseo-fallback": 1}).
- Per-URL stderr log lines tag the parser: "Parsing content
(1/5, via massive)" or "via dataforseo".
Tests:
- 8 new unit tests in tests/test_massive.py covering the markdown
parser, the shape contract with DataForSEOClient, and client
construction.
- All 5 test files pass.
Live smoke-tested against airport parking JFK:
- With token: content_parsers = {"massive": 5}, real word counts on
all 5 competitors (Massive sees full body content; e.g. SpotHero
1132 words vs DataForSEO's previous 245).
- Without token: content_parsers = {"dataforseo": 4}, graceful
fallback to existing behavior.
No real token committed -- .env.example uses an empty placeholder.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
