7

How do I perform a streaming character conversion?

view full story
linux-howto

http://stackoverflow.com – I have data stored on disk in files that are far too big to store in main memory. I want to stream this data from the disk into a data processing pipeline via iconv, like this: zcat myfile | iconv -f L1 -t UTF-8 | # rest of the pipeline goes here Unfortunately, I'm seeing iconv buffer the entire file in memory until it's exhausted before outputting any data. This means that I'm using up all of my main memory on a blocking operation in a pipeline whose memory footprint is otherwise minimal. I've tried calling iconv like this: stdbuf -o 0 iconv -f L1 -t UTF-8 But it looks like iconv is m (HowTos)