To Webdir [ robots.txt ]

Transform essentially just adds on robots.txt and returns all the directories located within that file. Useful when you have a large amount of websites and would like to analyse what they are stripping away from search engines ( such as /administrator/ for joomla sites)

Display Name: To Webdir [ robots.txt ]
Description: Looks for robots.txt and returns Webdirs
Author: Andrew MacPherson
Input Entity Type: Website

*Next*

Command: /path/to/php (eg. it could be c:php5inphp.exe or /usr/bin/php)
Parameters:robots.php
Working Directory: Directory you placed the attached script in as well as the php lib ( http://www.paterva.com/forum/index.php/topic,95.0.html )

*Finish and save*

You can now select any website and run the transform against that entity.

Note: This transform runs from your pc.

Included is the .mtg graph as well as a .zip containing the php script (feel free to edit this