python 在内存中处理tar.bz2文件

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

如果tar.bz2文件是通过网络进行下载，那么可以直接在内存进行解压后读取文件内容，不用将文件缓存到本地然后解压再进行读取，可以节省IO。

处理经过gzip压缩的tar文件的方法见：https://stackoverflow.com/questions/15352668/download-and-decompress-gzipped-file-in-memory

处理bz2压缩的tar文件的方法：https://stackoverflow.com/questions/46291529/how-to-decompress-tar-bz2-in-memory-with-python

import requests
import tarfile
from io import BytesIO

url = "www.google.com"   # the url you get tar.bz2 file from, need to change according to your application.

filename = "res_test.csv"  # the filename in your tar.bz2 file.

def decompress_tar_bz2_from_net(url, filename):

    """

    decompress the tar.bz2 format file in memory, instead of buffer it on disk

    and then decompress.

    :param url:

    :param filename:

    :return:

    """

    fileobj = BytesIO(requests.get(url).content)

    contents = tarfile.open(fileobj=fileobj).extractfile(filename).read()

    return contents

更多使用方法见：https://github.com/buxizhizhoum/tool_scripts/blob/master/app/bin/decompress_in_memory.py