This is an administrator tool with a Moodle cron task, that reaches into your Moodle site via curl (effectively from outside Moodle) and scrapes each page, parses it, and follows links. Because it comes in from outside it needs to authenticate and has a dependancy on the moodle-auth_basic plugin. It is recommended that you setup a dedicated 'robot' user that has readonly access to all the site pages you wish to crawl.
