https://netpeaksoftware.com/spider – looks useful to crawl big sites
Had Emma create some example links to showcase her work.
They are good but problem seems to be that they don’t get indexed
Each link is on a strong and reputable site
But they all seem to suffer from the same index budget / crawl budget
Google has already indexed a lot of their content
and without a compelling reason
Can’t find a reason to index or include
These extra pages
Its difficult to know how to address this issue, I keep researching, hoping to find an approach that works.