gzipped CSV files

[psaffrey-biomodal]

Hi there,

Does anybody have a recipe or pattern for working efficiently with gzipped CSV files? I'm working in bioinformatics, where these are extremely common! I could unzip with a JS library like pako but then I would probably have to do unzip and then parse (two steps), and I'd rather do this in one go, perhaps in a stream, for speed.

Cheers,

Peter

[jheer]

Hi! Arquero does not currently include support for gzipped CSV files. So unfortunately you will need to decompress using a separate library, then pass the resulting CSV text to Arquero. I don't have much experience with JS gzip decoders, so I don't have much more to add.

[jheer]

Thanks! I have a PR (#384) in the works that adds support for loading data from input streams, and includes decompression support.

[psaffrey-biomodal]

That's amazing, thanks so much for giving some time to this.

In an ideal world, this feature would work with the pipe operator. Here's a code fragment that uses papaparse with the NODE_STREAM_INPUT option to read unzipped data in a stream:

import fs from 'fs'
import zlib from 'zlib'
import papaparse from 'papaparse'

function parseGzippedCsv(filePath) {
    const data = [];

    const parser = papaparse.parse(papaparse.NODE_STREAM_INPUT)

    parser.on('data', (chunk) => {
        data.push(chunk);
        }
    );

    parser.on('end', () => {
        console.log(data)
        }
    );

    fs.createReadStream(filePath)
      .pipe(zlib.createGunzip())
      .pipe(parser);
}

parseGzippedCsv('/path/to/gzipped/csv')

[jheer]

The top level parseXXX methods don't work that way, but they all accept a stream (which you can modify or further chain prior to input). There is one other important difference: we use the Web Streams API, not the Node Streams API. The Web Streams API is a newer, cross-platform standard, allowing us to provide the same functionality across Node and the browser with one implementation.

[psaffrey-biomodal]

That sounds great - yes, using the cross-platform standard is important. I found it really confusing to try to reconcile what is supported in Node to what is supported in the browser.

As a developer, I would really appreciate some examples of using the new stream functionality to help me integrate it. Once I understand how it works, I'd also be happy to contribute some documentation myself to help others.

[jheer]

Of course! The PR isn't 100% done yet, documentation and release is forthcoming...