I will put an example.

I have a 1 TB CSV file, I need to conserve the raw file and sanitize then.

After the sanitization near 10% of entries has changed.

Both files need to be stored for years.

So, what's the best way to handle the two files?

I need to avoid duplicate the files to save 0.9 TB of disk usage, so the best way I find it's to store only the 10% of sanitized data when somebody wants to read an entry that not exists in the sanitized file the software layer try to find it in the raw file.

So, i can save 0.9 TB of disk space but I need a software layer to handle the requests for the entries.

The problem is, I don't have found a software that does it for me, so I need to write them.

However, Linux BtrFS filesystem has a deduplication resource, I can have the raw file and the sanitized file and allow the filesystem to handle the deduplication. But in a test, an estimate can save only 0.5 TB because need space for metadata.

I'm a little lost here:

How can I apply my requirement to save disk space without writing the software?

Or perhaps, the question should be: What methodology I need to apply to save disk space in file duplication and operation?

If someone likes to recommend a book, I'm open to reading it.

  • Why not to keep both files compressed? May 30, 2017 at 19:07
  • Hi, because I need to keep the entries available for online requests. If I compress one or other file each request can take, perhaps, 1 or 2 seconds. It's too much for this case. May 31, 2017 at 15:44


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.