Sparse Indexing: Large Scale, Inline Deduplication Using Sampling and Locality

Keywords

Computer science, Data deduplication, Bottleneck, Search engine indexing, Backup

References

• Bigtable: a distributed storage system for structured dataFay W. Chang, Jay B. Dean, Sanjay Ghemawat — 2006
• Evaluation of Efficient Archival Storage TechniquesLawrence L. You, Christos Karamanolis — 2004
• Avoiding the Disk Bottleneck in the Data Domain Deduplication File SystemBenjamin Zhu, Li Kai, Hugo Patterson — 2008
• Microsoft Exchange ServerChern., 黄强 — 1996
• Alternatives for Detecting Redundancy in Storage Systems DataCalicrates Policroniades, Ian Pratt — 2004
• Jumbo Store: Providing Efficient Incremental Upload and Versioning for a Utility Rendering ServiceKave Eshghi, Mark Lillibridge, Lawrence Wilcock — 2007
• An evaluation of buffer management strategies for relational database systemsHong‐Tai Chou, David J. DeWitt — 1986
• Finding Similar Files in a Large File SystemUdi Manber — 1994
• Bigtable: A Distributed Storage System for Structured DataFay W. Chang, Jay B. Dean, Sanjay Ghemawat — 2008
• Copy detection mechanisms for digital documentsSergey Brin, James Davis, Héctor García-Molina — 1995
• TAPER: tiered approach for eliminating redundancy in replica synchronizationNavendu Jain, Mike Dahlin, Renu Tewari — 2005
• A low-bandwidth network file systemAthicha Muthitacharoen, Benjie Chen, David Mazières — 2001
• A universal algorithm for sequential data compressionJ. Ziv, A. Lempel — 1977
• Finding similar files in large document repositoriesGeorge Forman, Kave Eshghi, Stephane Chiocchetti — 2005
• Space/time trade-offs in hash coding with allowable errorsBurton H. Bloom — 1970
• On the resemblance and containment of documentsArndt Bröder — 1997
• Opportunistic Use of Content Addressable Storage for Distributed File SystemsNiraj H. Tolia, Michael A. Kozuch, Mahadev Satyanarayanan — 2003
• Compactly encoding unstructured inputs with differential compressionMiklós Ajtai, Randal Burns, Ronald Fagin — 2002
• Fast, Inexpensive Content-Addressed Storage in FoundationSean Rhea, Russ Cox, A. A. Pesterev — 2008
• Deep Store: an archival storage system architectureLawrence L. You, K.T. Pollack, Darrell D. E. Long — 2005

Cited by

• Tradeoffs in Scalable Data Routing for Deduplication ClustersWei Dong, Fred Douglis, Kai Li — 2011
• Characteristics of backup workloads in production systemsGrant Wallace, Fred Douglis, Hangwei Qian — 2012
• Low-Cost Data Deduplication for Virtual Machine Backup in Cloud StorageWei Zhang, Tao Yang, Gautham Narayanasamy — 2013
• Statistical characterization of storage system workloads for data deduplication and load placement in heterogeneous storage environmentsNohhyun Park — 2013
• Improving restore speed for backup systems that use inline chunk-based deduplicationMark Lillibridge, Kave Eshghi, Deepavali Bhagwat — 2013
• CAFTL: A Content-Aware Flash Translation Layer Enhancing the Lifespan of Flash Memory based Solid State DrivesFeng Chen, Tian Luo, Xiaodong Zhang — 2011
• Using multi-threads to hide deduplication I/O latency with low synchronization overheadRui Zhu, Leihua Qin, Jingli Zhou — 2013
• Optimization Techniques To Record DeduplicationA. — 2012
• iDedup: latency-aware, inline data deduplication for primary storageKiran Srinivasan, Tim Bisson, Garth R. Goodson — 2012
• Building a High-performance Deduplication SystemFanglu Guo, Petros Efstathopoulos — 2011
• Leveraging content properties to optimize distributed storage systems. (Exploitation du contenu pour l'optimisation du stockage distribué)Konstantinos Kloudas — 2013
• Research on a new peer to cloud and peer model and a deduplication storage systemZhe Sun — 2011
• Incremental parallel and distributed systemsPramod Bhatotia — 2015
• Courbes remplissant l'espace et leur application en traitement d'images. (Spacer-filling curves and their application in image processing)Cu Nguyen Giap — 2013
• Memory efficient sanitization of a deduplicated storage systemFabiano C. Botelho, Philip Shilane, Nitin Garg — 2013
• Design Tradeoffs for Data Deduplication Performance in Backup WorkloadsMin Fu, Dan Feng, Yu Hua — 2015
• File recipe compression in data deduplication systemsDirk Meister, André Brinkmann, Tim Süß — 2013
• Primary Data Deduplication - Large Scale Study and System DesignAhmed El-Shimi, Ran Kalach, Ankit Kumar — 2012
• Improving caches in consolidated environmentsRicardo Koller — 2012
• ProSy: A similarity based inline deduplication system for primary storageXin Du, W K Hu, Qiang Wang — 2015

Related papers

• Tradeoffs in Scalable Data Routing for Deduplication ClustersWei Dong, Fred Douglis, Kai Li — 2011
• HYDRAstor: A Scalable Secondary StorageCezary Dubnicki, Leszek Gryz, Lukasz Heldt — 2009
• SiLo: A Similarity-Locality based Near-Exact Deduplication Scheme with Low RAM Overhead and High ThroughputWen Xia, Hong Jiang, Dan Feng — 2011
• Characteristics of backup workloads in production systemsGrant Wallace, Fred Douglis, Hangwei Qian — 2012
• dedupv1: Improving deduplication throughput using solid state drives (SSD)Dirk Meister, André Brinkmann — 2010
• Fast, Inexpensive Content-Addressed Storage in FoundationSean Rhea, Russ Cox, A. A. Pesterev — 2008
• Redundancy elimination within large collections of filesPurushottam Kulkarni, Fred Douglis, Jason LaVoie — 2004
• Space/time trade-offs in hash coding with allowable errorsBurton H. Bloom — 1970
• A study of practical deduplicationDutch T. Meyer, William J. Bolosky — 2011
• A low-bandwidth network file systemAthicha Muthitacharoen, Benjie Chen, David Mazières — 2001