Temporary paths
-
I used to use NTFS but that usually made Linux crap all over itself and me.
That's why I use UDF - works without problems on both Linux and Windows (unlike exFAT).FAT32 supports volumes up to 2TebiBytes
I think you can get it up to 16TiB with 4kB clusters.
-
yes, but non default cluster size support is not guaranteed.
-
TebiBytes
You stop that! Everyone who isn't a pedantic dickweed, or German[1], knows what it means.
[1] but I repeat myself, as far as this goes.
-
did you see footnote #2?
-
Of course I did. If everyone knows what it means, you don't need to specify you're disambiguating. I could've made that more clearerer.
-
then someone spams %TEMP% with symbolic links to some important system file -- say a registry hive, or some executable needed for the system to run. Whoops!
Yeah, the least-effort way is rather fragile. It also breaks for multiple cmd scripts launched in parallel, because cmd.exe seeds its random number generator from a global timer with 1-second resolution.
The more robust choice is to have the script generate a unique temp directory early on, and stuff all its temp files into that. This is safe because (a) mkdir is atomic on every Windows filesystem and (b) mkdir just fails if you ask it to clobber an existing dir or file, so even if somebody is playing silly tricks with symlinks you don't end up overwriting important things. It also makes cleanup easy.
call :mktmpdir || goto :eof :: do all the things, then rmdir /s /q "%tmpdir%" goto :eof :mktmpdir setlocal enabledelayedexpansion for /L %%I in (1,1,1000) do @( set tmpdir=%TEMP%\!RANDOM!-!RANDOM!-!RANDOM! mkdir "!tmpdir!" 2>nul && goto madetmpdir ) endlocal & set tmpdir= verify error 2>nul goto :eof :madetmpdir endlocal & set tmpdir=%tmpdir% goto :eof
-
Interesting -- why does Windows expect developers to get this right on their own, then, considering the security implications of getting this wrong?
Someone somewhere decided to write a helper function once for some Unix. They didn't bother on Windows. Neither platform needs it. It's just a convenience.
@tarunik said:Yeah --
mkdtemp()
isn't all that hot (it gives you back a name). I suspect the problem has something to do with permissions, though, considering that neithermkdir()
norCreateDirectory()
will clobber an existing thing...
The big problem is that the result is never a handle that you can use unambiguously to say “use this directory, whatever it is called”. This means that there are a number of race conditions with (relatively-highly privileged — at least “same user account”) other processes, such as by renaming the directory and putting another one in place. This means that temporary directories are always slightly more vulnerable to problems than you'd really like, even though they're a really nice solution to many other difficulties.You don't have the problem with files because you can rely on the file having identity independent of its name on Unix or apply appropriate locking during opening on Windows (each of which has its own set of caveats, but whatever…)
-
relatively-highly privileged — at least “same user account”
You have vulnerabilities there even with files (on unix, not sure on windows). There is no mandatory file locking in unix.
-
2 => i hate the use of the Tebi prefix, but i felt it important to be unambiguous this time
Kinda off-topic, but what should one use to be unambiguous while meaning the non-*bibytes? I propose "*debytes", for decimal-bytes. So that would be "tedebytes".
I do kind of wonder why anyone would ever speak of "tedebytes" without actually intending to sound ambiguous though.
-
"(n2^x bytes)" or "(n10^x bytes)" for the first event, and than switch back to regular SI prefixes.
-
There are other ways to fix that (e.g., having the file be unlinked from the FS and then checking that the number of remaining links is just 1 — the currently open file descriptor — which you can do without reference to the file other than your FD thanks to
fdstat()
) but they're a little exotic. It is certainly possible to ensure that the file that is there is the file that you think it ought to be.And some Unix FSes support mandatory locking (pretty much the local ones, plus a half-assed attempt in NFS). Hardly anyone uses it because it's a massive mistake.
-
Isn't the link count you get only the filesystem link count, not number of times open? I might be wrong, and there might be two fields - don't have the man page to hand...
-
I CBA to check right now; got a plane to catch…
-
. Hardly anyone uses it because it's a massive mistake.
NFS or the locking?
cause i use NFS in my home network as the most conveniant way to mount my NAS on my linux boxes.
SSHFS has issues dealing with large amounts of data and the layer of encryption is unnecessary because it's entirely in my network and i am the only human allowed in there.
CIFS also has issues, albeit less than SSHFS, but the real no-go was that it's support for POSIX permissions is halfassed at best and i hate seeing that execute bit get turned on for stuff like PNGs and MKVs
FTP was rejected outright for a filesytem mount option. yes it "works" but it's stupid.
so I went with NFS.
-
Pretty sure he just meant locking. In my experience NFS is fine, until you use advanced filesystem features like anything that is not a flat file or a symlink. Or your connection goes down (depending on client version and mount options).
-
Or your connection goes down (depending on client version and mount options).
yeah. that was a problem.... until i put the NAS on a UPS and have it be the master monitor. so now it broadcasts when it's shutting down allowing the linux boxes to shut down cleanly. It's also why i only mount /home on the NAS now....
-
-
So that would be "tedebytes".I do kind of wonder why anyone would ever speak of "tedebytes" without actually intending to sound ambiguous though.
The entire coding priesthood needs to get over its obsession with hijacking SI prefixes to mean something else. Kilo, mega, giga and tera are for 103n and kibi, mebi, gibi and tebi are for 210n and that's just the way it is.
The general public doesn't give a shit about the difference. Hell, the general public is flat out distinguishing a million from a billion.
People who object to the *bi prefixes because they "sound stupid" are just being stupid. Stop it.
-
renaming the directory and putting another one in place
That's largely solved by using temporary names that come out of a half decent random number generator, because those names are not predictable in advance.
Of course the random number generator built into cmd.exe doesn't count as half decent but meh. It's a cmd script. You can't expect it to be bulletproof when the interpreter that runs it is constructed from baling wire and old chewing gum.
-
tedebytes
http://www.hip-hop.com.pl/zdjecia/Tede.jpg
Filed under: obscure Polish joke, but I had to make it; thank me later
-
-
it is, however not completely supported on every filesystem. In particular the network filesystems have spotty if any support for it.
of course if you need to generate secure temporary files and your implementation puts them on a remote disk than your program or your computer setup are TRWTF.
(i mention computer setup because /tmp is the canonical place for temporary files (unless you can be reasonably sure they'll fit in /dev/shm in which case you should put them there for speed (if you need it. otherwise back to /tmp for you(nesting another level because i (can)))) and it is possible for your sysadmin to have /tmp be an NFS or other network mount point that may or may not support mandatory locking)
-
-
Ah - point...
-
-
i mention computer setup because /tmp is the canonical place for temporary files (unless you can be reasonably sure they'll fit in /dev/shm )
/tmp is effectively /dev/shm in Solaris anyway - not sure if that's the case for other UNIX based OSes.
-
really? so what happens when i need to generate a 50GB temporary file and try to drop it in /tmp? do i really error out because of out of disk space (because you only have 8GB of RAM and tmpfs refuses to grow to more than 50% of ram)
:frystare.mpg:
-
so what happens when i need to generate a 50GB temporary file and try to drop it in /tmp?
Bad things unless you have over 50GB of swap.
From experience, you get shouted at by the UNIX support team who remind you again that /tmp is special in Solaris and to not use it like you are. Especially that one time you completely fill it up.Having checked Google again, they're not exactly the same as it seems /dev/shm is on RAM but /tmp on Solaris is on swap which is only partially RAM (and the rest is swap devices).
-
still think that's a bad design, but then so is needing to generate 50GB of temporary file.
-
/tmp is effectively /dev/shm in Solaris anyway - not sure if that's the case for other UNIX based OSes.
I think I’ve seen some [spoiler]GNU/[/spoiler]Linux distributions with a tmpfs mounted over
/tmp
, which kind of defeats the purpose of/tmp
... Current distributions don’t seem to do that anymore, though.I also didn’t know people were using
/dev/shm/
to store temporary files which should be kept in memory; for me, this directory has always been an implementation detail of the the POSIX shared memory feature.
-
Hadn't even heard of /dev/shm before today but I don't really work with Linux.
-
I also didn’t know people were using /dev/shm/ to store temporary files which should be kept in memory; for me, this directory has always been an implementation detail of the the POSIX shared memory feature.
i use it as a place to store files that i need to pass between parts of some complicated batch files that do processing on them. I possibly could have used named pipes for the data, but i hadn't learned about them when i started the scripts and someone came along after me and before i learned about the pipes and hung a couple of monitor scripts off the temporary files so i can't change them into named pipes without rewriting his PERL golf'd scripts....
/me starts growling softly and showing some vulpine fang [spoiler]See current avatar if that confuses you. ;-)[/spoiler]
-
Bad things unless you have over 50GB of swap.From experience, you get shouted at by the UNIX support team who remind you again that /tmp is special in Solaris and to not use it like you are. Especially that one time you completely fill it up.
Having checked Google again, they're not exactly the same as it seems /dev/shm is on RAM but /tmp on Solaris is on swap which is only partially RAM (and the rest is swap devices).
Isn't this what /var/tmp is for? Then again -- building LibreOffice from source is a great way to test the size of your /var...
-
building LibreOffice from source is a great way to test the size of your /var
I see the makings of a good pick up line here.
-
I see the makings of a good pick up line here.
No, it was a hard-learned lesson about partitioning...
-
No, it was a hard-learned lesson about partitioning...
Enough of your kinky sex stories.
-
Isn't this what /var/tmp is for?
Quite, hence being shouted at.
We (ab)use /tmp because a) it's faster and b) it's larger.
-
c) it's a shorter path
oh come on! you know it's true!
-
But / is an even shorter path!
-
which also requiers superuser rights.... i'm not saying it's not abused but just using /tmp is shorter than having to sprinkle everything with calls to sudo...
-
rm -rf /*
?
-
well you can leave off the * and it'll still work
although you possibly also want to add --no-preserve-root to allow GNU compliant versions to actually execute that command (they will refuse without that switch)
-
That was sort of my point, yes ;)
-
rm -rf /*
?http://what.thedailywtf.com/t/phew-i-cancelled-the-sudo-rm-rfv-no-preserve-root-just-in-time/3992
-
rm -rf metric
-
Especially that one time you completely fill it up.
Shouldn't have tempted fate, really....
[code]Memory: 64G real, 1281M free, 115G swap in use, 149M swap free[/code]
Oops.
-
.... frack that's a big swap space!
what you use it all for?
-
-
what you use it all for?
Filling up with temporary files.
That box hosts multiple test environments and half of our pre-Prod environment and some other bits.
Between 40GB and 60GB used is where it sits most of the time.
-