KungFQ: A Simple and Powerful Approach to Compress fastq Files

Nowadays storing data derived from deep sequencing experiments has become pivotal and standard compression algorithms do not exploit in a satisfying manner their structure. A number of reference-based compression algorithms have been developed but they are less adequate when approaching new species...

Full description

Saved in:
Bibliographic Details
Published in:IEEE/ACM transactions on computational biology and bioinformatics Vol. 9; no. 6; pp. 1837 - 1842
Main Authors: Grassi, E., Gregorio, F. D., Molineris, I.
Format: Journal Article
Language:English
Published: United States IEEE 01.11.2012
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithms
algorithms for data and knowledge management
Bioinformatics
Biology and genetics
Compression algorithms
Data compression
Data Compression - methods
Databases, Genetic
Decoding
Encoding
Genomics
Genomics - methods
High-Throughput Nucleotide Sequencing
Internet
Software
ISSN:1545-5963, 1557-9964, 1557-9964
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Nowadays storing data derived from deep sequencing experiments has become pivotal and standard compression algorithms do not exploit in a satisfying manner their structure. A number of reference-based compression algorithms have been developed but they are less adequate when approaching new species without fully sequenced genomes or nongenomic data. We developed a tool that takes advantages of fastq characteristics and encodes them in a binary format optimized in order to be further compressed with standard tools (such as gzip or lzma). The algorithm is straightforward and does not need any external reference file, it scans the fastq only once and has a constant memory requirement. Moreover, we added the possibility to perform lossy compression, losing some of the original information (IDs and/or qualities) but resulting in smaller files; it is also possible to define a quality cutoff under which corresponding base calls are converted to N. We achieve 2.82 to 7.77 compression ratios on various fastq files without losing information and 5.37 to 8.77 losing IDs, which are often not used in common analysis pipelines. In this paper, we compare the algorithm performance with known tools, usually obtaining higher compression levels.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:1545-5963
1557-9964
1557-9964
DOI:10.1109/TCBB.2012.123