Cloning drives


  • ♿ (Parody)

    I bought the wife and kids SSDs for their laptops this year for Christmas. I already had an Apricorn laptop kit that came with a USB connector and software, EZ Gig IV.

    I looked into the software, and it claimed to handle copying a larger drive to a smaller drive, even if there was more data on the big drive (you could tell it to ignore, e.g., Documents or Videos or something). Well, cool, my wife's laptop (only a few months old) had a 1TB drive, but there was less than 200GB on there, and I was planning on getting her a 750GB SSD. The kids both got 500GB SSDs (one HD was already 500GB, the other was 320, so no problems there).

    OK, plug in the drive, insert the CD and fire up the cloning software!

    ERROR: THE TARGET DRIVE IS SMALLER THAN THE SOURCE DRIVE

    Uh...yeah, but it's a tiny amount of data! You said you handled this! Ugh...OK, thinks I, I can outsmart this software! I'll create some giant "fake" files in Videos and tell it to exclude those!

    But...it turns out that it can't find the directory. I seem to be a victim of MS's CADT approach to user home directory layouts. The software works with XP up to 7. Wife's computer is on 10 (INB4 idiot!). Ugh...OK, well, the drive came with a license key for Acronis!

    Download...download...install....run...Select source...Select destination....

    You must reboot before this copy can happen. If you don't reboot immediately the copy won't happen!

    Yikes! Worse than Windows Update! But OK, it's what I'm doing, so go ahead...I expected that it would set one of those "Do this on next restart" tasks and happen probably before I could even login.

    Except...the login page came up as normal. :wtf: Hmm...maybe it's that fucking fast restart bug! DISABLE. OK, try this again...nope, same deal. Well, fuck.

    So, I start looking around...Father in law recommended Miray HDClone. Ah, cool! Download...install....and only then I notice that to actually clone a working harddrive you need the paid version (something like 16 euros). Shit, gotta be a better way than ruining some dollars by converting to funny money...

    Went back to Macrium Reflect (which I had started downloading, but stopped because it was huge and kept stalling). Went much faster and it actually cloned as expected.

    Kids' drives worked no problem with the EZ Gig IV thing.


  • FoxDev


  • Discourse touched me in a no-no place

    @boomzilla said in Cloning drives:

    claimed to handle copying a larger drive to a smaller drive, even if there was more data on the big drive

    dd if=/dev/sda of=/dev/sdb bs=64K conv=noerror,sync # who cares what's at the end of /dev/sda!
    

  • Java Dev

    @boomzilla When I did this, several years back, the path I followed was reducing the partition size from inside windows. However the guide I used was in a print magazine so there are no links I could give you.

    The path involved defragmenting the drive, using the disk management tool to shrink the drive (can only go down to 51% of original size due to the page file), restarting the system to get the page file and some other stuff moved, then resizing the original drive again. Rinse/repeat until small enough.

    Once the source partition is small enough any partition copy tool can do the job.

    I think there were some windows subsystems that needed to get temporarily disabled as well to get their unmovable files moved, but I can't recall which ones.


  • Notification Spam Recipient

    @PleegWat said in Cloning drives:

    think there were some windows subsystems that needed to get temporarily disabled as well to get their unmovable files moved, but I can't recall which ones.

    Solved by booting into another OS temporarily to do the clone.



  • @PJH
    I thought the fastest way to clone drives was dd if=/dev/null of=/dev/* 🚎


  • Java Dev

    @Tsaukpaetra Well obviously you can't do the clone while it's running. The problem was in the partition resize which (at least in win7) wasn't capable of moving certain system files, so you had to disable the subsystem that owned them or trick it into moving them.


  • Grade A Premium Asshole

    @PJH said in Cloning drives:

    dd if=/dev/sda of=/dev/sdb bs=64K conv=noerror,sync # who cares what's at the end of /dev/sda!
    

    Pretty sure that won't work:

    1. You're copying the partition table from a larger drive to a smaller drive. Whether it's a GPT (which comes with its own problems, as you've just copied the partition's GUID) or MBR disk, you're declaring that the smaller disk has a too-large partition.
    2. You're copying a too-large NTFS fs onto a smaller disk. Dunno if the modern CHKDISK-equivalent is smart enough to just resize the FS, or if it'll shit the bed or what.

    But maybe I :whoosh: ?


  • Notification Spam Recipient

    @PleegWat said in Cloning drives:

    @Tsaukpaetra Well obviously you can't do the clone while it's running. The problem was in the partition resize which (at least in win7) wasn't capable of moving certain system files, so you had to disable the subsystem that owned them or trick it into moving them.

    ... I meant, sure that's true about the MFT, but there's no reason to assume that files that aren't open by a process aren't able to be moved. And to my knowledge, most Linux distros do not open files willy nilly on NTFS volumes unless you're intentionally doing so...

    Still, yes, back then resizing NTFS was a pain, because the tooling didn't move the MFT all that well, and certain other issues that required a chkdsk to fix.

    @bugmenot said in Cloning drives:

    if the modern CHKDISK-equivalent is smart enough to just resize the FS, or if it'll shit the bed or what.

    Well, so long as there weren't actual files in the missing area, it would be fine for a little while. Until it tried to write to that area...


  • Java Dev

    @Tsaukpaetra said in Cloning drives:

    MFT

    That was probably it. The article I used as guide very strongly recommended using only official windows-native tools (which ran on the windows install being moved) to resize NTFS partitions. It could move most files, even open files, but IIRC not hiberfil.sys and pagefile.sys so those had to be disabled. The MFT may well have been the critical one in the middle of the disk that couldn't be moved, but ended up moving itself at the next reboot.



  • Bootable CD/USB of gparted and g4l for all your partition mangling and copying needs.


  • Notification Spam Recipient

    @billhead said in Cloning drives:

    Bootable CD/USB of gparted and g4l for all your partition mangling and copying needs.

    Or, if you can get iPXE booted, chain https://tsaukpaetra.com/iPXE/ and you can load my (somewhat old) copy of it from the interwebs!


  • Discourse touched me in a no-no place

    @bugmenot said in Cloning drives:

    But maybe I :whoosh: ?

    Maybe.



  • EaseUS has a bunch of free software that is easy to use and very useful, IMO. They're my go-to programs for anything partition-related in Windows, including managing non-Windows partitions.



  • @Tsaukpaetra said in Cloning drives:

    @billhead said in Cloning drives:

    Bootable CD/USB of gparted and g4l for all your partition mangling and copying needs.

    Or, if you can get iPXE booted, chain https://tsaukpaetra.com/iPXE/ and you can load my (somewhat old) copy of it from the interwebs!

    0_1483443013168_upload-6e5f209f-bf7d-4c76-ad9d-678b1306ebe4



  • @boomzilla Standard Windows drive cloning process for me:

    1. Use MyDefrag's "System disk monthly" script to defrag the source drive. If I'm going to be cloning to an SSD, the "System disk daily" script is faster and equally good. The monthly script does more work to get stuff positioned optimally for a spinny disk, but both will push almost everything as close to the beginning of the drive as it will go.

    2. Boot Knoppix.

    3. Use dd to copy the boot code and partition table from the old drive to the new one. This means that Windows will end up seeing the same disk ID it started with so it doesn't get all stroppy when I eventually boot the clone.

    4. Use parted to fix the partition table on the new drive so that the Windows partition actually fits on it.

    5. Use ntfsresize on the source drive to shrink the filesystem to the size of the target partition. This goes very fast and doesn't leave the drive all fragmented to fuck, because most of the work has already been done by MyDefrag.

    6. Use ntfsclone to copy the filesystem from the source drive to the target drive.

    100% reliable and all free.



  • @PleegWat said in Cloning drives:

    (can only go down to 51% of original size due to the page file)

    It's pretty easy and generally worthwhile to disable paging and hibernation, reboot, then del /a:h C:\pagefile.sys & del /a:h C:\hiberfil.sys before doing the defrag.



  • @Tsaukpaetra said in Cloning drives:

    the tooling didn't move the MFT all that well

    MyDefrag's System Disk Monthly script moves it to 30% of the way into the volume as a matter of course.



  • @Tsaukpaetra said in Cloning drives:

    so long as there weren't actual files in the missing area, it would be fine for a little while.

    Nup. NTFS has a sentinel sector at the very end of the volume, and if that's not there it will refuse to mount.



  • @PleegWat said in Cloning drives:

    The article I used as guide very strongly recommended using only official windows-native tools (which ran on the windows install being moved) to resize NTFS partitions.

    Clearly written by a Windows weenie. ntfsresize has been reliable since well before ntfs-3g was.

    The only real gotchas I'm aware of with using Linux-based partition management and cloning tools are these:

    1. The Windows boot sequence fails if the disk signature of the disk you're booting from (dword at offset 0x1B8 in the MBR) is different from that of the disk on which Windows was originally installed. If you ever clone a disk and then make the mistake of booting Windows while both the original and the clone are connected to the machine, Windows will see two disks with the same signature and silently resolve that by randomizing one of them. That disk will not subsequently boot successfully until its signature is restored to the original value.

    You can copy the disk signature from $src to $tgt using dd if=$src of=$tgt bs=1 count=4 skip=440 seek=440 conv=notrunc, but it's easier just to copy the whole MBR using dd if=$src of=$tgt bs=512 count=1 conv=notrunc and then fix the partition table afterwards. Note that $src and $tgt should be the names of whole-disk devices or device images, not partitions or partition images.

    1. If you clone an NTFS boot partition and your cloning process means that the target partition begins at a different offset from the source (perhaps because you're doing a P2V on a business-critical legacy Windows XP installation and you don't want the massive misalignment inefficiency you'll get from keeping XP's default starting LBA of 0x3F) you have to fix the NTFS Hidden Sectors field (dword at offset 0x1C inside the NTFS boot sector) to match the partition's new starting LBA, or the boot just hangs.

    Easiest way to write that field from the command line is to convert the dword value you want to a little-endian hex byte string by hand, then write it out using xxd and dd. For example, for a starting LBA of 2048 = 0x00000400, you'd use echo 00 04 00 00 | xxd -p -r | dd of=$tgt bs=1 count=4 seek=28 conv=notrunc. In this case $tgt should be the name of a partition device or image, not a whole-disk one.



  • @PleegWat said in Cloning drives:

    The MFT may well have been the critical one in the middle of the disk that couldn't be moved, but ended up moving itself at the next reboot.

    The MFT never moves all by itself. pagefile.sys and hiberfil.sys will quietly recreate themselves if they're missing.

    Before I found out about MyDefrag, I would generally use ntfsresize without a preliminary defrag step. This often takes a long time if the source disk is heavily fragmented, and if that's the case the resized volume will be fragmented much worse, but I've never seen it create filesystem errors (at least, none that the chkdsk it schedules after doing its work have ever found). And of course having booted into Knoppix or Trinity Rescue Kit or whatever Linux live-CD tool, mounting the volume and deleting pagefile.sys and hiberfil.sys before resizing it doesn't need any kind of special dance.


  • Java Dev

    @flabdablet

    shrugs

    I'm working off what I remember of a process I did >4 years ago. I'm gonna stop arguing.


  • ♿ (Parody)

    @flabdablet said in Cloning drives:

    100% reliable and all free.

    Wow. That looks like a ton of work and exactly what I was trying to avoid. Macrium Reflect "just worked" for me without having to do any of that stuff (the other tool did, too, for the same sized drives).

    There was a step where it was laying out the partitions and told me there wasn't enough room (duh!). So I put the max size for the main partition where it fit and left enough room for the 35MB restore partition or whatever that last little bit was and then it copied everything over no problem.



  • @boomzilla said in Cloning drives:

    Macrium Reflect "just worked" for me without having to do any of that stuff

    I'm probably a bit peculiar but I don't mind doing it all by hand. I've done it so often that I no longer have to look anything up, the amount of time I spend typing stuff is minuscule compared to the time it takes to do the actual cloning anyway, and I like that I end up with a nicely defragged clone - especially if the target is an SSD. Defragging SSDs involves loads of writes for not much performance gain, so it's generally a bad tradeoff; but if your Windows installation starts out with most files in one fragment, you benefit from the SSD's extreme sequential read speed as well as its lack of seek delay.

    Also not needing to fuck around downloading this tool and that tool and the other tool only to have most of them fail mysteriously is nice. Provided only that the target partition really is big enough to accommodate the used space on the source, my standard process always works.


  • ♿ (Parody)

    @flabdablet said in Cloning drives:

    I've done it so often that I no longer have to look anything up, the amount of time I spend typing stuff is minuscule compared to the time it takes to do the actual cloning anyway, and I like that I end up with a nicely defragged clone - especially if the target is an SSD.

    This was the first time I've done anything like this and I don't expect to do it in the near future. The drives had been defragged previously by their users.



  • @boomzilla said in Cloning drives:

    This was the first time I've done anything like this and I don't expect to do it in the near future.

    I take it you're not looking after 120 school workstations then :-)



  • I can't believe all of those crappy 90's cloning programs still exist. Who buys (or even tries) that crap?

    The proper procedure:

    1. Download CloneZilla and burn to CD
    2. In CloneZilla, run ntfsresize --size ###G /dev/sdaX
    3. Resize the partition using fdisk (make it slightly larger than the number above just in case of rounding issues)
    4. Run CloneZilla (ocs-live) to clone the disk
    5. If the boot sector isn't restored, use a Windows CD to repair the boot sector.

    I've done this twice on my main Windows 7 desktop to move to SSD and then a larger SSD later, and numerous disk upgrades for other people. Works every time. CloneZilla even has an option to delete pagefile.sys and hiberfil.sys before cloning.

    The only thing that would be better is that CloneZilla could do the ntfsresize steps as part of the nice menus that it has.


  • Discourse touched me in a no-no place

    Why not just buy an SSD larger than the HDD it is replacing? Costs a bit, but very nice since you don't need to spend ages figuring things out…


  • ♿ (Parody)

    @dkf said in Cloning drives:

    Why not just buy an SSD larger than the HDD it is replacing? Costs a bit, but very nice since you don't need to spend ages figuring things out…

    Well, based on what I'd seen, it shouldn't have been a problem at all. And it shaved off about $90 or so from the price when I was also buying two other drives. It seemed like a decent compromise. And it was, even though I had to spend a bit more time and frustration on Christmas and the following day getting it all to work.



  • @quijibo said in Cloning drives:

    Download CloneZilla and burn to CD

    Still deeply in love with my Zalman virtual CD drive.



  • @flabdablet said in Cloning drives:

    Still deeply in love with my Zalman virtual CD drive.

    Seriously yes. (I put a 500G SSD in there)


  • Notification Spam Recipient

    @flabdablet said in Cloning drives:

    @quijibo said in Cloning drives:

    Download CloneZilla and burn to CD

    Still deeply in love with my Zalman virtual CD drive.

    Yeah. Those are fscking amazing (If the machine supports USB-CD boot). I slapped a 2Tb drive in mine, and haven't really needed anything else since (in fact, my net-boot setup has been gathering dust since I haven't set up UEFI netbooting in dnsmasq yet).

    Mines branded IODD though, but I'm sure it's the same thing...



  • @flabdablet said in Cloning drives:

    Still deeply in love with my Zalman virtual CD drive.

    I didn't know about that. That's a good idea.

    What I did was follow a guide to install Clonezilla on an 2TB external hard disk with a second data partition. This way I can plug it into any machine and back up Windows up to the data partition. That's often my first step before messing with anything complicated.


  • Grade A Premium Asshole

    @Tsaukpaetra said in Cloning drives:

    @billhead said in Cloning drives:

    Bootable CD/USB of gparted and g4l for all your partition mangling and copying needs.

    Or, if you can get iPXE booted, chain https://tsaukpaetra.com/iPXE/ and you can load my (somewhat old) copy of it from the interwebs!

    And then infect my computer with MLP desktop porn? No thanks. :P


  • Grade A Premium Asshole

    @djls45 said in Cloning drives:

    EaseUS has a bunch of free software that is easy to use and very useful, IMO. They're my go-to programs for anything partition-related in Windows, including managing non-Windows partitions.

    Agreed. It is what I use for SSD upgrades also.


  • Notification Spam Recipient

    @Polygeekery said in Cloning drives:

    @Tsaukpaetra said in Cloning drives:

    @billhead said in Cloning drives:

    Bootable CD/USB of gparted and g4l for all your partition mangling and copying needs.

    Or, if you can get iPXE booted, chain https://tsaukpaetra.com/iPXE/ and you can load my (somewhat old) copy of it from the interwebs!

    And then infect my computer with MLP desktop porn? No thanks. :P

    Well it's an http server, you can inspect the files yourself!


  • :belt_onion:

    @flabdablet said in Cloning drives:

    Clearly written by a Windows weenie.

    Or someone who wants things to work and doesn't want to invest two months of his life in researching and figuring out a working SOP to do it in a non-native environment.

    1. The Windows boot sequence fails if the disk signature of the disk you're booting from (dword at offset 0x1B8 in the MBR) is different from that of the disk on which Windows was originally installed.

    Only if both are connected at the same time (as you later note); don't do that. Of course it's intended to be unique at boot.

    As far as references to disk signatures in the Registry, I'd be surprised if those are maintained after you do a sysprep generalize. (If you're not doing a sysprep generalize, you don't know how to clone the OS properly and should not be administering Windows systems.)

    Easiest way to write that field from the command line is to convert the dword value you want to a little-endian hex byte string by hand, then write it out using xxd and dd. For example, for a starting LBA of 2048 = 0x00000400, you'd use echo 00 04 00 00 | xxd -p -r | dd of=$tgt bs=1 count=4 seek=28 conv=notrunc. In this case $tgt should be the name of a partition device or image, not a whole-disk one.

    HOLY SHIT, THESE INSTRUCTIONS MAKE ME WANT TO DIE. DO YOU NOT SEE HOW INSANE THE WORDS ARE THAT YOU ARE WRITING?

    @flabdablet said in Cloning drives:

    I take it you're not looking after 120 school workstations then :-)

    Once you get to that scale (I administered 300 school workstations at my last job, six years ago), you should be able to shell out the money for Symantec Ghost or something of that nature. One click and you're cloning an image to an entire lab, unattended.

    Complexity for the sake of complexity is bullshit.



  • @flabdablet said in Cloning drives:

    @quijibo said in Cloning drives:

    Download CloneZilla and burn to CD

    Still deeply in love with my Zalman virtual CD drive.

    I just have CloneZilla on a USB key.

    You guys like to have complicated setup 🙄



  • @heterodox said in Cloning drives:

    Complexity for the sake of complexity is bullshit.

    I agree.

    However, a general preference for digging into into stuff and getting jobs done piecewise using simple straightforward tools, rather than relying on other people's black-box solutions, has on balance saved me vast amounts of time due to knowing exactly what to fix when shit breaks. Because there's a good chance I actually broke it that way myself at some point while learning about it.

    @heterodox said in Cloning drives:

    THESE INSTRUCTIONS MAKE ME WANT TO DIE. DO YOU NOT SEE HOW INSANE THE WORDS ARE THAT YOU ARE WRITING?

    Not at all insane. Just derived from a different fundamental attitude from yours. Which I think is perfectly OK; there's plenty of room for a variety of attitudes and motivations in this industry.

    If you want standard jobs done quickly and competently, employ somebody skilled at wielding standard tools; your expressed preferences seem to indicate that that would be you. If you want curly problems investigated and their root causes found and fixed, employ somebody like me. It's all good.



  • @heterodox said in Cloning drives:

    Only if both are connected at the same time

    Really, no. If you partition and format a disk so it gets a new MBR and consequently a new disk signature, then you clone the Windows system partition onto that from some other disk, then try to boot the clone, that boot will fail even if the clone is the only disk connected.

    I know this from personal experience.


  • :belt_onion:

    @flabdablet said in Cloning drives:

    If you want standard jobs done quickly and competently, employ somebody skilled at wielding standard tools; your expressed preferences seem to indicate that that would be you. If you want curly problems investigated and their root causes found and fixed, employ somebody like me. It's all good.

    I'm not sure that's the dichotomy, but it's close. I do see the value in wanting to know what's inside the black box; I wouldn't be in the programs I'm in if I didn't. But IMHO, that should not be your job in 99% of cases; it should be a hobby. It's what you do in your off-time. I may have misunderstood your posts above to imply that you go through that ridiculous set of steps in looking after all of your workstations, which seems downright negligent to me.



  • @TimeBandit said in Cloning drives:

    You guys like to have complicated setup

    No, I just like not needing to worry about whether the ISO to USB converter I used last month will still work for this ISO I've just downloaded now.

    With the Zalman enclosure, there's no conversion step. Any CD or DVD image that comes as an ISO can just be saved straight to the drive when it's in HD mode; put it in optical disk mode, select any of your downloaded ISOs using its inbuilt jog switch and display, and the target machine sees it as an optical drive with that disc in place.

    Since acquiring this toy I have yet to encounter a machine with a BIOS too braindead to allow booting from it.

    By way of contrast, I used to waste quite a lot of time trying to get assorted uncooperative ISO images converted to USB images that would actually boot successfully. I have yet to find an ISO to bootable-USB conversion tool with 100% coverage of the ISO images I've needed to use.



  • @heterodox said in Cloning drives:

    I may have misunderstood your posts above to imply that you go through that ridiculous set of steps in looking after all of your workstations, which seems downright negligent to me.

    I'll use those steps for one-off cloning of customer's machines as the need arises. At the school, they've naturally all been scripted.


  • :belt_onion:

    @flabdablet said in Cloning drives:

    Since acquiring this toy I have yet to encounter a machine with a BIOS too braindead to allow booting from it.

    Hmmm. Do you know offhand if it has a boot manager to work with UEFI/no CSM? That'd be pretty banging.

    I'd say having a specialized device like that is kind of the opposite of complicated setup, really. Using the right tools for the right job and all of that.



  • @flabdablet said in Cloning drives:

    I have yet to find an ISO to bootable-USB conversion tool with 100% coverage of the ISO images I've needed to use.

    My experience with CloneZilla's ISO is 100% success with

    dd if=/home/user/CloneZilla_some_version_number.iso of=/dev/sdc bs=4M
    

    Maybe I am just lucky 🤷



  • @heterodox said in Cloning drives:

    If you're not doing a sysprep generalize, you don't know how to clone the OS properly and should not be administering Windows systems.

    In fact cloned Winboxen work just fine without sysprep generalize (as long as they've been cloned from compatible hardware).

    Conventional wisdom has it that the machine ID regeneration that sysprep generalize (or ghostwalker or NewSID) performs is necessary for vaguely defined reasons having something to do with security, but in fact that's not so; there is no networked attack you can mount against a fleet of completely identical clones that won't work equally well against a fleet of "properly" sysprepped ones.

    There's also no issue with cloned machine IDs when you join them to a domain; domain SIDs are generated by the DC and are completely unrelated to the local Windows SID.

    None of which is to deny that sysprep generalize is indeed MS-recommended best practice. But unless you have a specific reason for claiming that choosing to skip it must inevitably be a mark of incompetence, I don't believe that that is the case.



  • @heterodox said in Cloning drives:

    IMHO, that should not be your job in 99% of cases; it should be a hobby.

    I've always been fortunate enough to be in a position where the lines between those remain quite blurred; I like getting paid to play with the toys I'd be playing with anyway and for better or worse I seem to have made something of a career of it.


  • :belt_onion:

    @flabdablet said in Cloning drives:

    There is no networked attack you can mount against a fleet of completely identical clones that won't work equally well against a fleet of "properly" sysprepped ones.

    I'm aware of that. Kind of a strawman, isn't it? But I have seen problems with licensing and activation if you don't generalize (especially when using a KMS server), it's nice to be able to have the old machine up while bringing up the cloned machine (you won't have duplicate computer names or IP addresses), and most importantly, as you point out, it allows you to port the image over to machines with different HALs and hardware. (Less important as we continue full-speed into the realm of virtualization, but we're not all there yet.)

    Generalizing (see what I did there?) to total incompetence is admittedly 🚎, but IMO you need a specific reason not to follow best practice rather than the other way around. I don't have a list of every internal cleanup task that sysprep generalize performs to make sure an image will work better on a cloned machine, but I know things do work better, and that's good enough for me.



  • @heterodox said in Cloning drives:

    I have seen problems with licensing and activation if you don't generalize (especially when using a KMS server)

    Only an issue for the first 25 boxes attached to any given KMS server AFAIK. After that it doesn't care.



  • @TimeBandit said in Cloning drives:

    Maybe I am just lucky

    There are quite a few of those "universal" ISO images about, that do just boot if you block-for-block clone them to USB media. They're in the minority though. None of the Windows Setup ISOs work that way, though many Linux live images do.


Log in to reply