I have a stream of data being delivered to my Amazon S3 bucket in the form of tab-delimited text. I would like to scan as it comes in to keep a very small subset of the rows from that match certain criteria. Given the following constraints:
- Each file is 5-50GB in size.
- I'm receiving just over 1 billion rows of data per day.
- I am to use Amazon AWS services.
- I need to do it as cheaply as possible.
All of the data only needs to be read once then can be deleted. How do I most efficiently perform both a range filter on one field, and a keyword filter on another, keeping the entire row that matches a given query for ingest into another system?
I can think of a number of solutions and none of them are cheap. Originally I was thinking of loading into a Redshift cluster but this seems like it may be overkill on the basis of both complexity. Lambda seems like an obvious choice except for the fact that it doesn't support loading that much data into memory. I suppose I could see how well an auto-scaling group of workers performs, but that doesn't seem like the most efficient way. Ideally I'd like to perform spatial queries on locations represented as x,y coordinates but I can simplify that to a series of bounding box queries if necessary. All suggestions appreciated.