Cross-platform zips
-
So on a linux machine I'm running
zip -r components.zip ./components/*
The resulting zip is 45 MB. I can download it on another linux machine and unzip it just fine. Windows claims it's corrupt using the native utility. I need to figure out why this is happening and determine if it can be resolved without installing any other unzip utility; I know for a fact 7zip can handle it, but if I'm going to hard sell the 7zip solution I need to understand what's wrong with it and if it's fixable.
]$ zip --version Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license. This is Zip 3.0 (July 5th 2008), by Info-ZIP. Currently maintained by E. Gordon. Please send bug reports to the authors using the web page at www.info-zip.org; see README for details. Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip, as of above date; see http://www.info-zip.org/ for other sites. Compiled with gcc 4.4.4 20100518 (Red Hat 4.4.4-4) for Unix (Linux ELF) on May 24 2010.
-
what's wrong with it
on a linux machine
Ok.... I'll be serious.... :P
<something>- Is there another implementation of .zip for linux?
- What about reversing the process. Zip on windows and try to unzip on linux.
-
Have you tried without compressing it? Or are the files so big that compressing it is a requirement?
-
From the man page:
Large Archives and Zip64. zip automatically uses the Zip64 extensions when files larger than 4 GB are added to an archive, an archive containing Zip64 entries is updated (if the resulting ar‐ chive still needs Zip64), the size of the archive will exceed 4 GB, or when the number of entries in the archive will exceed about 64K. Zip64 is also used for archives streamed from standard input as the size of such archives are not known in advance, but the option -fz- can be used to force zip to create PKZIP 2 compatible archives
I would give the -fz- option a try. How many files are in ./components?
-
Have you tried without compressing it?
There's a lot of small files I want to transport cleanly via the network, so an archive of some sort seems appropriate. Windows can handle zip natively.
How many files are in ./components?
9135
I would give the -fz- option a try
Alright, giving that a shot.
-
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Huh. With something that old it should be more compatible, amirite?
-
There's a lot of small files I want to transport cleanly via the network, so an archive of some sort seems appropriate. Windows can handle zip natively.
You can still zip it without compressing it.
zip -Z store -r whatever.zip whatever/*
If it's the compression method which is upsetting Windows, then storing the files instead should work.
-
I almost suggested this as well, definitely worth a shot if -fz- doesn't do it.
-
Update: this has to do with something in the node modules, because when I disable the npm install, the zip opens fine.
So now I gotta figure out why npm is broken on my sandbox server. Le sigh.
-
That doesn't make any sense.
What happens with npm enabled but using the options upthread?
-
I'm trying to ensure I repro the problem first, because I discovered it on the prod build server and I'm testing fixes in my sandbox.
My guess is symlinks?
-
My guess is symlinks?
I think you have to explicitly tell
zip
to store symlinks as symlinks, otherwise it just stores the referenced file.
-
It's not the npm install.
The build script? I guess?
-
Symlinks or hardlinks are things I would suspect as well.
-
WTF server team.
My sandbox is performing literally identical steps on a literally identical repository and not replicating the issue. Must be a configuration issue. So we'll test in prod I guess. yippie.
-
All the best testing happens in Production.
-
All the best testing happens in Production.
QFT. Literally did this yesterday, because apparently changes were made in Prod that didn't go through Dev or Test.
Since I don't have time to do a schema compare by hand (because for some reason quite a bit of nonsense is happening in Prod that doesn't exist on Test or Dev, that shouldn't actually get replicated back to either), we just splashed the script on the prod server.Luckily it's a repeatable self-healing defensive script, so when they re-run it today it will just shrug its shoulders with a "Good, everything's done!" message.
-
test in prod
WHY THE HELL IS A CHECKOUT TAKING NINETY MINUTES.
God dammit, server team!
-
-
-
zip -fz- -r whatever.zip whatever/*
edit: That.
-
$ zip -fz- -r components1.zip ./components/*
zip error: Invalid command arguments (no such option: -)
-
Huh. Does your version support
-fz-
? Is it listed inman zip
?
-
Probably just the extra
-
at the end of-fz-
-
I have a -f and a -z but no -fz-
-
Boo.
-
Missed my window. Nobody's getting anything done the rest of the day.
Fucking hell.
-
Does your man page mention Zip64 at all? I don't see -fz- listed in the option flags list, only in the paragraph about Zip64 towards the top.
-
$ man zip | grep Zip64
No results
-
Huh. If your
zip
doesn't support Zip64 then that probably isn't the issue. This is Windows 2k8?
-
yeah, also 7 shows the symptom
-
What does
file
have to say about the zip file?
-
$ file components.zip
components.zip: Zip archive data, at least v1.0 to extract
-
Grand total of not much at all. I'm used to getting more details nowadays. Or that zip file really isn't using any special options.
-
-
BeyondCompare says the version of the zip utility is identical between the sandbox and the prod server:
$ zip --version Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license. This is Zip 3.0 (July 5th 2008), by Info-ZIP. Currently maintained by E. Gordon. Please send bug reports to the authors using the web page at www.info-zip.org; see README for details. Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip, as of above date; see http://www.info-zip.org/ for other sites. Compiled with gcc 4.4.4 20100518 (Red Hat 4.4.4-4) for Unix (Linux ELF) on May 24 2010. Zip special compilation options: USE_EF_UT_TIME (store Universal Time) SYMLINK_SUPPORT (symbolic links supported) LARGE_FILE_SUPPORT (can read and write large files on file system) ZIP64_SUPPORT (use Zip64 to store large files in archives) UNICODE_SUPPORT (store and read UTF-8 Unicode paths) STORE_UNIX_UIDs_GIDs (store UID/GID sizes/values using new extra field) UIDGID_NOT_16BIT (old Unix 16-bit UID/GID extra field not used) [encryption, version 2.91 of 05 Jan 2007] (modified for Zip 3) Encryption notice: The encryption code of this program is not copyrighted and is put in the public domain. It was originally written in Europe and, to the best of our knowledge, can be freely distributed in both source and object forms from any country, including the USA under License Exception TSU of the U.S. Export Administration Regulations (section 740.13(e)) of 6 June 2002. Zip environment options: ZIP: [none] ZIPOPT: [none]
What else could affect it? Maybe the virus scan utility that I typically turn off on the sandbox?
-
-
BeyondCompare says the version of the zip utility is identical between the sandbox and the prod server:
hmm.... what about the resulting zipfiles it generates? are they binary identical or not?
-
-
ZIP64_SUPPORT (use Zip64 to store large files in archives)
Strange that it's been compiled with Zip64 support but the
man
pages don't mention it.
Shouldn't be an issue as it still doesn't work when set to store not compress.
-
Obviously not, since I can open some of them :) Let me see...
-
Strange that it's been compiled with Zip64 support but the man pages don't mention it.
Could I have grepped for the wrong string?
-
Could I have grepped for the wrong string?
I don't think so - I tried the same command and it worked here.
-
Let me see...
So I can't seem to find two that have identical contents to check their hex comparison. And looking into them with beyondcompare, one of them contains a shitton more in the grunt node-modules folder than the other. However, both are doing an npm install of the same code in the same folder.
So... node version differences? The working one was installed with 0.10.36, while the non-working one was installed via 0.12.4....
-
I can't see how the node version could impact the file created by the
zip
utility, but you never know.Does the MD5 remain the same after it's been sent?
-
The working one was installed with 0.10.36, while the non-working one was installed via 0.12.4....
hmm.... it's possible...
but i'm not sure how just the contents of node_modules/ inside the zip could cause the issue....
-
TBH I'm not sure how to calculate the MD5 sum on Windows
-
Powershell 4 has it:
Get-FileHash $FILENAME -Algorithm MD5
If you're on an earlier version, check the top answer here:
-
-
Uhh....
but I assume I did something wrong there770dc4a9b3e27f361f218924628e8328
doesn't look the same as11-47-58-14-40-B1-06-6B-41-6C-29-53-DB-D3-6F-A8
...I did, see below