Most "is this backlink any good?" advice stops at rel="nofollow". I went through what rel actually says on editorial and directory pages last week, so this time I wanted the other stuff: the checks that can quietly cancel a link even when the anchor tag looks perfect.
So this morning I pointed a small Node script at 48 pages on 10 sites I run:
- 22 product pages on my launch sites and directory (Aura++, EarlyHunt, IndieHunt, MakerHunt, SideHunt, Uno Directory)
- 22 pages on my four small press sites (14 published articles, 8 project pages)
- 4 of the Markdown copies those press sites publish next to each article
These are the kind of pages people submit to, or pay for, to get a link. They're mine, so I can poke at them all I want and tell you what's broken without pointing at anyone else's site.
Checked Monday 9 Nov 2026, between 10:54 and 11:14 IST. Four full runs, two with Googlebot's user agent string and two with a Chrome one. Raw HTML only, no JavaScript executed.
What the script checks
For each page, plus the URL it is supposed to link to:
- Status and redirects, hop by hop, so a 301 to a 302 to a 200 shows up as exactly that.
-
noindex, in
<meta name="robots">,<meta name="googlebot">and theX-Robots-Tagheader. The header is the one people forget, because View Source never shows it. - Page-level nofollow, in meta or header. That one cancels every link on the page, even an anchor with no rel at all.
- robots.txt for Googlebot. Google's "Block Search indexing with noindex" doc (last updated 2025-12-10) says: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file." A blocked page can't even tell Google it's noindexed.
-
Canonical: missing, pointing at another URL, two different ones, sent as a
Linkheader, or sitting outside<head>. - Sitemap: is the page in the site's own sitemap. It's a cheap orphan check, not proof that anything links to the page.
-
The link itself: is there a real
<a href>to your domain in the raw HTML, or only inside script data. Is it inside<noscript>or a hidden element, does it go through a redirect URL, and does your URL still load.
Point 7 matters because of one line in Google's "SEO link best practices" doc (last updated 2025-12-10): "Generally, Google can only crawl your link if it's an <a> HTML element (also known as anchor element) with an href attribute."
The boring part, which is good news
Across all 48 pages:
- 0 noindex (meta or header)
- 0 page-level nofollow
- 0 robots.txt blocks (all 10 sites only disallow things like /admin, /dashboard and /api)
- 0 redirects, 0 non-200 pages
- 0 canonicals pointing at a different URL
- 0 pages missing from their sitemap
- 0 links that only existed in JavaScript data, 0 inside
<noscript>
I half expected at least one stray noindex, so that was nice to see.
Finding 1: 27 pages send the canonical in the body
26 pages had the canonical <link> after </head> on every run: all 4 Aura++ project pages and all 22 press-site pages. One or two EarlyHunt and SideHunt pages flipped between runs. One SideHunt page, curled three times in a row with the same user agent, came back body, head, head. The total per run was 28, 26, 27 and 27.
The cause is Next.js streaming metadata. The Next.js generateMetadata docs say that when it resolves, "the resulting metadata tags are appended to the <body> tag", and that this is "interpreted correctly by bots that execute JavaScript and inspect the full DOM (e.g. Googlebot)". So for Google, this is probably fine once the page is rendered.
Not every crawler renders, though. Google's own canonical doc (last updated 2026-07-10) says the rel="canonical" link element "is only accepted if it appears in the <head> section of the HTML", and raw HTML is all a non-rendering crawler gets. So I curled one press page with 8 user agents. facebookexternalhit and bingbot got the canonical in <head>, which fits Next.js keeping a built-in list of "HTML-limited" bots it waits for. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, AhrefsBot and plain curl all got it in the body.
I wrote up how this happens, and the htmlLimitedBots setting that fixes it, in a canonical in your Next.js root layout lands on every page. The lesson here is simpler: when you check a placement, check it with more than one user agent.
Finding 2: on 30 pages the product link sits inside a hidden div
This one looked scary. On 30 pages the link to the product was inside an element with the hidden attribute. My first thought was that a template was hiding links.
It isn't. When part of a React page streams in late, React sends that chunk inside <div hidden id="S:2"> near the end of the HTML, and a small inline script moves it into place. I went back over all 30 and the hidden wrapper was always one of those S:n boundaries (21 of them were S:2). None were display:none.
Google renders JavaScript, so this should be fine for Google. A parser that skips hidden elements and doesn't run JavaScript would skip the link. I don't know which crawlers actually do that, so I won't pretend it's a disaster. It does mean "the link is in the HTML" isn't the whole answer, and any checker that flags hidden links needs a second look at what is hidden.
Finding 3: the Markdown copies have no canonical
The press sites publish a .md version of each article for AI agents, and those URLs are in the sitemap. All 4 I checked returned 200, text/markdown, X-Robots-Tag: index, follow, and no canonical. The file has a "Canonical:" line in its text, but crawlers don't treat text as a canonical. Google accepts canonicals as an HTTP header, so the fix is one response header on each .md URL:
Link: <https://example.com/publications/some-article>; rel="canonical"
Finding 4: rel, briefly
7 pages had a followed, visible link to the product. 28 had nofollow on every link to it. 9 had a followable link, but only inside one of those streamed hidden divs. On the press sites the Markdown copy prints the plan, and 10 of 14 articles say Plan: nofollow, which matches their links. So it's not a bug, just a reminder to read the HTML instead of the sales page.
Finding 5: some of the linked sites are gone
Of the 44 product URLs these pages link to, 1 returned 404, 1 returned 402 and 2 didn't respond to my requests at all. The listing page is fine. The thing it points at isn't. That's the listing site's cleanup job, not yours, but it's a good reminder to re-check your own link targets now and then.
The script
Node 18+, no dependencies. Give it a text file with one <page-url> <your-url> pair per line (use - instead of a URL for page-only checks):
node placement-check.mjs pairs.txt
UA="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/130.0 Safari/537.36" node placement-check.mjs pairs.txt
#!/usr/bin/env node
// placement-check.mjs: checks the things beyond rel="nofollow" that quietly stop a backlink from counting.
// Usage: node placement-check.mjs pairs.txt (UA="..." to change the user agent)
// pairs.txt: one "<page-url> <your-url>" per line (# for comments, "-" = page checks only)
// Node 18+, no dependencies. Reads raw HTML only; it does not run JavaScript.
import { readFileSync } from "node:fs";
// Default: Googlebot's user agent string, because some sites (Next.js streaming metadata, for one)
// send crawlers different HTML than browsers. Run again with UA="Mozilla/5.0" to compare.
const UA = process.env.UA || "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)";
const lines = readFileSync(process.argv[2] || "pairs.txt", "utf8").split("\n")
.map(s => s.trim()).filter(s => s && !s.startsWith("#"));
const attr = (tag, name) => (tag.match(new RegExp(`\\s${name}\\s*=\\s*("([^"]*)"|'([^']*)'|([^\\s>]+))`, "i")) || [])
.slice(2).find(v => v !== undefined);
const host = u => { try { return new URL(u).hostname.replace(/^www\./, ""); } catch { return ""; } };
const norm = u => { try { const x = new URL(u); return (x.origin + x.pathname).replace(/\/$/, ""); } catch { return u; } };
const decode = s => s.replace(/&/g, "&");
// 1. follow redirects by hand so we can see every hop
async function get(url) {
const hops = [];
for (let i = 0; i < 6; i++) {
const r = await fetch(url, { redirect: "manual", headers: { "user-agent": UA } });
hops.push(r.status);
const loc = r.headers.get("location");
if (r.status >= 300 && r.status < 400 && loc) { url = new URL(loc, url).href; continue; }
return { res: r, url, hops, body: await r.text() };
}
throw new Error("too many redirects");
}
// 2. robots.txt: longest matching Allow/Disallow in the Googlebot group, else the * group
const robotsCache = {};
async function robotsBlocks(url) {
const u = new URL(url);
if (!(u.origin in robotsCache)) {
let txt = "";
try { const r = await fetch(u.origin + "/robots.txt", { headers: { "user-agent": UA } }); if (r.ok) txt = await r.text(); } catch {}
const groups = []; let cur = null, lastWasUA = false;
for (const raw of txt.split("\n")) {
const [k, ...v] = raw.replace(/#.*/, "").split(":"); const key = k.trim().toLowerCase(), val = v.join(":").trim();
if (key === "user-agent") { if (!lastWasUA) groups.push(cur = { agents: [], rules: [] }); cur.agents.push(val.toLowerCase()); lastWasUA = true; }
else if ((key === "allow" || key === "disallow") && cur) { cur.rules.push([key, val]); lastWasUA = false; }
else lastWasUA = false;
}
const g = groups.filter(x => x.agents.includes("googlebot"));
robotsCache[u.origin] = (g.length ? g : groups.filter(x => x.agents.includes("*"))).flatMap(x => x.rules);
}
const path = u.pathname + u.search; let best = ["allow", ""];
for (const [k, v] of robotsCache[u.origin]) {
if (!v) continue;
const re = new RegExp("^" + v.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*").replace(/\\\$$/, "$"));
if (re.test(path) && (v.length > best[1].length || (v.length === best[1].length && k === "allow"))) best = [k, v];
}
return best[0] === "disallow" ? best[1] : null;
}
// 3. walk the tags so we know if a link sits inside <noscript> or a hidden parent
const VOID = new Set("area base br col embed hr img input link meta source track wbr".split(" "));
function anchors(html) {
const out = [], stack = []; const re = /<(\/?)([a-zA-Z][\w-]*)([^>]*)>/g; let m;
while ((m = re.exec(html))) {
const [, close, name0, rest] = m, name = name0.toLowerCase();
if (!close && (name === "script" || name === "style")) { const end = html.toLowerCase().indexOf(`</${name}`, re.lastIndex); re.lastIndex = end < 0 ? html.length : end; continue; }
if (close) { const i = stack.map(s => s.name).lastIndexOf(name); if (i >= 0) stack.length = i; continue; }
const style = (attr(m[0], "style") || "").replace(/\s/g, "").toLowerCase();
const bare = rest.replace(/"[^"]*"|'[^']*'/g, ""); // attribute names only, so class="md:hidden" doesn't count
const hidden = /(^|\s)hidden(\s|=|\/|$)/i.test(bare) || /display:none|visibility:hidden/.test(style);
if (name === "a") out.push({ href: decode(attr(m[0], "href") || ""), rel: (attr(m[0], "rel") || "").toLowerCase(),
noscript: stack.some(s => s.name === "noscript"), hidden: hidden || stack.some(s => s.hidden) });
if (!VOID.has(name) && !rest.trim().endsWith("/")) stack.push({ name, hidden });
}
return out;
}
const sitemapCache = {};
async function inSitemap(url) {
const o = new URL(url).origin;
if (!(o in sitemapCache)) {
const locs = new Set(); const queue = [o + "/sitemap.xml"]; let n = 0;
while (queue.length && n++ < 10) {
try { const x = await (await fetch(queue.shift(), { headers: { "user-agent": UA } })).text();
for (const [, l] of x.matchAll(/<loc>\s*([^<\s]+)\s*<\/loc>/g)) (/<sitemapindex/i.test(x) ? queue.push(l) : locs.add(norm(decode(l))));
} catch {}
}
sitemapCache[o] = locs;
}
return sitemapCache[o].has(norm(url));
}
const tally = {}; const bump = k => (tally[k] = (tally[k] || 0) + 1);
for (const line of lines) {
const [page, target] = line.split(/\s+/); const flags = [];
let r; try { r = await get(page); } catch (e) { console.log(`\n${page}\n FETCH ERROR ${e.message}`); bump("fetch error"); continue; }
const { res, url, hops, body } = r, ctype = res.headers.get("content-type") || "";
if (hops.length > 1) flags.push(`redirects: ${hops.join(" -> ")} -> ${url}`);
if (res.status !== 200) flags.push(`final status ${res.status}`);
const xr = (res.headers.get("x-robots-tag") || "").toLowerCase();
if (/noindex|none/.test(xr)) flags.push(`X-Robots-Tag: ${xr}`);
if (/nofollow/.test(xr)) flags.push(`X-Robots-Tag nofollow (every link on the page): ${xr}`);
const isHtml = ctype.includes("html");
const head = body.split(/<\/head>/i)[0];
for (const t of isHtml ? body.match(/<meta[^>]+>/gi) || [] : []) {
const name = (attr(t, "name") || "").toLowerCase();
if (name !== "robots" && name !== "googlebot") continue;
const c = (attr(t, "content") || "").toLowerCase();
if (/noindex|none/.test(c)) flags.push(`meta ${name}: ${c}`);
if (/nofollow/.test(c)) flags.push(`meta ${name} nofollow (every link on the page): ${c}`);
}
const canon = isHtml ? (body.match(/<link[^>]+>/gi) || []).filter(t => /\brel\s*=\s*["']?canonical/i.test(t)).map(t => attr(t, "href")) : [];
const hdrCanon = ((res.headers.get("link") || "").match(/<([^>]+)>\s*;\s*rel="?canonical/i) || [])[1];
if (hdrCanon) canon.push(hdrCanon);
if (canon.length > 1 && new Set(canon.map(c => norm(new URL(c, url).href))).size > 1) flags.push(`${canon.length} different canonicals: ${canon.join(" , ")}`);
else if (canon[0] && norm(new URL(canon[0], url).href) !== norm(url)) flags.push(`canonical points elsewhere: ${canon[0]}`);
if (isHtml && canon.length && !(head.match(/<link[^>]+>/gi) || []).some(t => /canonical/i.test(t)) && !hdrCanon) flags.push("canonical is outside <head> in the raw HTML");
if (!canon.length) flags.push(isHtml ? "no canonical" : `no canonical (non-HTML: ${ctype.split(";")[0]}, no Link header)`);
const blocked = await robotsBlocks(url); if (blocked) flags.push(`robots.txt Disallow: ${blocked}`);
if (!(await inSitemap(url))) flags.push("not in the site's sitemap");
let linkSummary = "page checks only";
if (target && target !== "-") {
const th = host(target);
const all = isHtml ? anchors(body) : [];
const mine = all.filter(a => host(new URL(a.href, url).href) === th);
// e.g. /out?url=https://you.com : a param whose whole value is your URL
const viaRedirect = all.filter(a => { try { const x = new URL(a.href, url); return host(x.href) !== th && [...x.searchParams.values()].some(v => /^https?:/.test(v) && host(v) === th); } catch { return false; } });
const followed = mine.filter(a => !/nofollow|sponsored|ugc/.test(a.rel) && !a.noscript && !a.hidden);
const inScripts = (body.match(/<script[\s\S]*?<\/script>/gi) || []).some(s => s.includes(th));
linkSummary = `${mine.length} <a> to ${th} (${followed.length} followed & visible)` + (mine.length ? `, rel: ${[...new Set(mine.map(a => a.rel || "(none)"))].join(" | ")}` : "");
if (!mine.length && inScripts) flags.push(`link to ${th} only inside <script> data (JS-rendered, not in raw HTML)`);
else if (!mine.length) flags.push(`no link to ${th} at all`);
if (mine.length && !followed.length) flags.push(`every link to ${th} is nofollow/sponsored/ugc, hidden or in <noscript>`);
if (mine.some(a => a.noscript)) flags.push(`a link to ${th} sits inside <noscript>`);
if (mine.some(a => a.hidden)) flags.push(`a link to ${th} sits in a hidden element`);
if (viaRedirect.length) flags.push(`${viaRedirect.length} link(s) go through a redirect URL instead of straight to ${th}`);
try { const t = await get(target); if (t.res.status >= 400) flags.push(`your URL returns ${t.res.status}`); } catch (e) { flags.push(`your URL failed to load: ${e.message}`); }
}
console.log(`\n${page}\n HTTP ${hops.join(">")} | ${ctype.split(";")[0]} | ${linkSummary}`);
for (const f of flags) { console.log(" FLAG " + f); bump(f.replace(/https?:\/\/\S+/g, "").replace(/(to|straight to) [\w.-]+\.[a-z]+/g, "$1 your URL").replace(/: .*|\(.*?\)/g, "").replace(/^\d+ /, "").trim()); }
if (!flags.length) console.log(" ok");
bump(flags.length ? "pages with any flag" : "clean pages");
}
console.log(`\n== summary (${lines.length} pages, UA: ${UA})`); for (const [k, v] of Object.entries(tally).sort((a, b) => b[1] - a[1])) console.log(`${String(v).padStart(3)} ${k}`);
Before the real run I tested it on 15 local pages: one clean, and 14 each broken in one way (noindex meta, noindex header, page-level nofollow, canonical elsewhere, canonical in body, robots.txt block, JS-only link, noscript, display:none, redirect link, rel=nofollow, 301 to 302, 404, dead target). Every flag fired. It's regex over raw HTML, so treat it as a first pass and open DevTools on anything it flags.
If you only have two minutes, these cover most of it:
curl -sI "$PAGE" | grep -i -E "^(HTTP/|location:|x-robots-tag:|link:.*canonical)"
curl -s -A "GPTBot/1.2" "$PAGE" | grep -o -i -E '<meta[^>]+robots[^>]*>|<link[^>]+canonical[^>]*>|</head>'
curl -s "$PAGE" | grep -o -E '<a [^>]*href="https?://(www\.)?yourdomain\.com[^>]*>'
If </head> prints before the canonical (or the robots meta), that tag is in the body for that user agent. On my own press page with GPTBot's user agent, both were.
What I'd check before paying for a link
- Open the live page. Check the status,
X-Robots-Tag, and meta robots for noindex and nofollow. - Check robots.txt doesn't block it.
- Check the canonical points at the page itself and sits in
<head>, with more than one user agent. - Check your link is a real
<a href>in the raw HTML, and what is wrapping it. - Check the page is in the site's sitemap and linked from somewhere a crawler will find.
Lede Press is a press site I run, and it's in this audit, warts and all. It's one of the sites that sends the canonical in the body to GPTBot, ClaudeBot and plain curl. It publishes editorial backlinks listed in public: one article about your product with one link in the text, for $29, or free if your site links back. The site says the link is dofollow on those two plans (there's also a free nofollow plan). Run the script on the live page and check it yourself. No promises about rankings.
Which of these do you actually check before you pay for a link? And has a hidden-div or body-canonical flag ever turned out to be a real problem for you?
Top comments (0)