On occasion, the MirrorWeb crawlers may experience difficulty capturing certain web sites or assets. This is usually due to security that is in place on the hosting web server.
All of our crawls are done using the following user agent string - if this is whitelisted by the web hosts then the crawl will be much more replete and comprehensive
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 +https://www.mirrorweb.com
or a variation thereof (customer depending). If your crawler uses a custom user agent, you will have been informed as part of your onboarding.