List Archive

Thread

Thread Index

Message

From: Thomas Klausner <tk%giga.or.at@localhost>
To: Markus Pöpping <markus%precipoint.de@localhost>
Subject: Re: Access to a Zip archive from multiple threads
Date: Fri, 17 Feb 2017 15:38:30 +0100

Hi Markus!

> I am trying to write code that handles zip archives with lots of
> entries, in the range of 100k to 1 million. It's read-only access. So
> far, it works pretty well.
> Now, the big issue is, I would like to have parallel access from
> multiple threads to the same archive. Obviously, zip_fopen cannot handle
> this, unless I implement waiting locks.
> One possible solution would be to open several instances with zip_open,
> but that solution obviously costs too much time and resources: A few
> seconds for each zip_open and >60 MB heap space for the directory.
> The other one I perceive is to get a new handle to the archive with
> fopen(), and somehow ask LibZip for the stream offset and call the
> necessary decompression functions by hand. Apparently, such operations
> are not really desirable, as I see *struct zip_dirent* and others are
> completely private.
> 
> Is there a possible workaround that doesn't rely too much on private
> implementation details, or is there any feature planned that does
> facilitate this sort of operation ?

I'm sorry, that's currently not supported, nor easily added.

All data runs through zip_sources, and make these thread-safe would be
hard (they could e.g. have FILE * inside).

Perhaps it's a possible workaround to read the data into a local
buffer and then pass the zip archive on to the next thread?

Cheers,
 Thomas

Made by MHonArc.