Item31: Remote crawlers require more reliable publishing mechanism

Currently, the crawler publishes results by writing an XML file onto the nfs-mount, and signalling the front-end that a new XML file is ready for publishing by tickling a certain url.

This is fine for the situation where all crawler nodes were in the datacenter, with a dedicated backbone link. It is less optimal for remote crawler node's.

Proposed solution: fix the 'publish by posting xml via http' feature.

-- KoenMartens - 09 Apr 2008