How to process logs and debug information generated by applications running on spot instances

view story

http://serverfault.com – I am using Amazon Spot Instances to crawl a lot data. Most of them run until Amazon terminates them once the current price exceeds our max bid. I need to monitor and mainly archive the logs generated in those spot instances. Those logs are very important for debugging and analytics. We have application logs, system logs such as syslog, secure log. Below are the options I could think of: use Chkwa/Flume. Not listing Facebook's scribe here because I think the project is dead. There is a rare possibility to lose few logs with this approach. Attach an EBS volume to those spot instance. But then (HowTos)