Author Topic: [Solved]How to format an external 1TB USB hard disk for backup, thanks.  (Read 2317 times)

Offline joejac

  • Full Member
  • ***
  • Posts: 168
Hello!,

I would like to copy my VMs to a 1TB WD Elements External USB HD . Unfortunately NTFS do not allow me to copy them, the entire VMs directory is 170GB and only the Win 7 VM is 84GB.

I know Linux can handle this, so does anybody can please provide a step by step procedure to reformat the entire HD for PCLinux and be able to use the full capacity of the HD (1TB) or a link to a tutorial?

I found this but I am unsure:
http://community.wdc.com/t5/Other-Externals/WD-elements-1TB-Desktop-formatting-for-Linux-ext3/td-p/61240

Thanks a lot in advance
Best regards
joejac
« Last Edit: June 03, 2012, 03:11:48 PM by joejac »

Offline bicol_willem

  • PCLinuxOS Tester
  • Hero Member
  • *******
  • Posts: 2376
Re: How to format an external 1TB USB hard disk for backup, thanks.
« Reply #1 on: March 27, 2012, 09:45:24 PM »
Think there will be many views on this.
I use FAT32 partitions (3). Mainreason is that, instead of when ext is used, also Windows can read and share the files AND last but not least, it forecomes at times strange (especially to Linux newbe's) permission problems when files are shared with other users or other machines.

Offline Old-Polack

  • Administrator
  • Super Villain
  • *****
  • Posts: 11688
  • ----IOFLU----
Re: How to format an external 1TB USB hard disk for backup, thanks.
« Reply #2 on: March 27, 2012, 10:19:13 PM »
Hello!,

I would like to copy my VMs to a 1TB WD Elements External USB HD . Unfortunately NTFS do not allow me to copy them, the entire VMs directory is 170GB and only the Win 7 VM is 84GB.

I know Linux can handle this, so does anybody can please provide a step by step procedure to reformat the entire HD for PCLinux and be able to use the full capacity of the HD (1TB) or a link to a tutorial?

I found this but I am unsure:
http://community.wdc.com/t5/Other-Externals/WD-elements-1TB-Desktop-formatting-for-Linux-ext3/td-p/61240

Thanks a lot in advance
Best regards
joejac


I broke my 1 TB drive into 17 partitions, all formatted ext3 or ext4. I have full operating systems on some, and data partitions for the rest. Big advantage is a filesystem check doesn't take forever, and if one partition gets totally borked, the rest of the drive is still usable, and the maximum loss is only one partition's data, not the whole drive's data.

In the link you provided, one poster showed how to eliminate the reserved space. That's idiotic, as that space is needed to allow deleting expendable data should the drives other 95% capacity be exceeded. Without that reserve, even root cannot delete a single file, and the drive has been effectively bricked. In that case the only remedy is a complete reformat with a 100% data loss. Not a bright move.

With that said, partitioning and formatting that drive is exactly the same as any internal drive, and can be done with the Linux partitioning/formatting tool(s) of your choice. I prefer the command line fdisk for partitioning and mkfs for formatting. Both are extremely easy to use, and each does only one thing but does it very well. It's very easy to stay focused on the job at hand that way.

If you prefer GUI tools, PCC --> Local disks --> Manage disk partitions will do just fine.
« Last Edit: March 27, 2012, 10:20:44 PM by Old-Polack »
Old-Polack

Of what use be there for joy, if not for the sharing thereof?



Lest we forget...

Offline Xenaflux

  • PCLinuxOS Tester
  • Hero Member
  • *******
  • Posts: 3829
Re: How to format an external 1TB USB hard disk for backup, thanks.
« Reply #3 on: March 28, 2012, 07:47:04 AM »
Quote
I broke my 1 TB drive into 17 partitions

OP, just for curiosity , I would like to know how you did that, as
when I used Gparted LiveCD, the maximum I could get to was 14.
If I tried 15, I got an error message.
Tried it twice.
Once with the 1T which is in my Desktop and once with my 1T backup.
Both are Western Digital drives

Your thoughts will be appreciated.

Xena
The great thing in this world is not so much where we stand,
as in what direction we are moving.
                                                    (Oliver Wendell Holmes )

Offline Just17

  • PCLinuxOS Tester
  • Super Villain
  • *******
  • Posts: 11051
  • MLUs Forever!
Re: How to format an external 1TB USB hard disk for backup, thanks.
« Reply #4 on: March 28, 2012, 09:47:49 AM »
Quote
I broke my 1 TB drive into 17 partitions

OP, just for curiosity , I would like to know how you did that, as
when I used Gparted LiveCD, the maximum I could get to was 14.
If I tried 15, I got an error message.
Tried it twice.
Once with the 1T which is in my Desktop and once with my 1T backup.
Both are Western Digital drives

Your thoughts will be appreciated.

Xena

Was that recently?

Quite a while ago there was a limit on the number of partitions on one drive ....  15 IIRC .....  which was much lower than what was allowed prior to that.
But that situation has been 'fixed' for a long time .......  unless some regression has crept in some place, which seems doubtful .....
MLUs rule the roost!

Linux XPS 3.4.48-pclos1.bfs  64 bit
Intel Core2 Quad CPU Q9450 @ 2.66GHz
4 GB RAM
MCP51 High Def Audio
GeForce GTX 550 Ti
PHILIPS  ‎DVD+-RW DVD8701
‎Logitech ‎BT Mini-Receiver
Afatech DTT

Offline joejac

  • Full Member
  • ***
  • Posts: 168
Re: How to format an external 1TB USB hard disk for backup, thanks.
« Reply #5 on: March 28, 2012, 11:29:06 AM »
Thanks to all for your kind responses.

Hi bicol_willem, I understand the convenience of the NTFS, but I have not found the way to copy a large file without those file splitting programs, that is very time consuming and probably risky. Do you have a reliable and fast method for copying big files?

I was guessing that if my Host is PCLinuxOS I can copy to it a VM and if I need to run a VM inside a Windows VM I can share the directory of the Host, where I copied the VM from the USB Drive, and the file would be available for the guest Windows VM. Please correct me if I am wrong.

Since I come from many, many years on Windows I prefer the GUI solution, I do not want to make mistakes via the command line.

Hello Old-Polack, thanks for the explanation of the reserved space. I can not imagine formatting a HD with 17 partitions and worst than that then organizing the data and OSs, what a headache. VirtualBox is wonderful for this. I have never had a crash in a HD in 34 years working with computers, and I have used even an old computer with more than 12 years old HD.

Thanks and regards
joejac

Offline Old-Polack

  • Administrator
  • Super Villain
  • *****
  • Posts: 11688
  • ----IOFLU----
Re: How to format an external 1TB USB hard disk for backup, thanks.
« Reply #6 on: March 28, 2012, 11:36:12 AM »
Quote
I broke my 1 TB drive into 17 partitions

OP, just for curiosity , I would like to know how you did that, as
when I used Gparted LiveCD, the maximum I could get to was 14.
If I tried 15, I got an error message.
Tried it twice.
Once with the 1T which is in my Desktop and once with my 1T backup.
Both are Western Digital drives

Your thoughts will be appreciated.

Xena

There used to be a 15 partition limit on SCSI drives and a 63 partition limit on IDE drives. Like most limits they were programed limits not physical. When the kernels IDE tree was abandoned, and all drives were seen as SCSI drives that meant all drives had a 15 partition limit, which seemed rather silly to some of us, as an old 20 GB IDE drive could have 63 partitions while a new 1TB SATA drive only 15. Also IDE drives with more than 15 partitions suddenly seemed to lose those partitions past the first 15.

This derives from the major and minor numbers associated with the devices on a Linux system. All SCSI hard drives have the major number 8. The first SCSI drive has the major number 8 and the minor numbers 0-15 assigned to it. The drive itself is major 8 minor 0, or M-8 m-0, which to the kernel is the definition of /dev/sda. That leaves the remaining minor numbers available to describe 15 partitions. The second SCSI drive will again have the major number 8 but minor numbers from 16 - 31. Each additional drive will have another 16 number block of minor numbers that define the drive letter as well as the partition numbers. One must remember that this system was derived when hard drives were still measured in MB, so 15 partitions seemed rather large and possibly excessive.

IDE hard drives had the major number 3 but the minor numbers were assigned in blocks of 64; ie M-3 m-0 defined /dev/hda and the remaining minor numbers set the limit for the number of partitions (63). M-3 m-64 defined /dev/hdb,  and so on.

Sometime in the last two years, or so, the kernel code was rewritten to raise the partition limit, and some of the partitioning tools were rewritten to take advantage of this, as well as change their code to properly align partitions on the new 4 KB sector hard drives now being produced. Partitions beyond the previously defined limit of 15 are added with the major number 259 and minor numbers starting at 0, as seen here;

[root@fatman ~]# ls -l /dev |grep sda
brw-rw----  1 root   disk      8,   0 Mar 28 03:33 sda
brw-rw----  1 root   disk      8,   1 Mar 28 03:33 sda1
brw-rw----  1 root   disk      8,  10 Mar 28 03:33 sda10
brw-rw----  1 root   disk      8,  11 Mar 28 03:33 sda11
brw-rw----  1 root   disk      8,  12 Mar 28 03:33 sda12
brw-rw----  1 root   disk      8,  13 Mar 28 03:33 sda13
brw-rw----  1 root   disk      8,  14 Mar 28 03:33 sda14
brw-rw----  1 root   disk      8,  15 Mar 28 03:33 sda15
brw-rw----  1 root   disk    259,   0 Mar 28 03:33 sda16
brw-rw----  1 root   disk    259,   1 Mar 28 03:33 sda17
brw-rw----  1 root   disk      8,   2 Mar 28 03:33 sda2
brw-rw----  1 root   disk      8,   3 Mar 28 03:33 sda3
brw-rw----  1 root   disk      8,   4 Mar 28 03:33 sda4
brw-rw----  1 root   disk      8,   5 Mar 28 03:33 sda5
brw-rw----  1 root   disk      8,   6 Mar 28 03:33 sda6
brw-rw----  1 root   disk      8,   7 Mar 28 03:33 sda7
brw-rw----  1 root   disk      8,   8 Mar 28 03:33 sda8
brw-rw----  1 root   disk      8,   9 Mar 28 03:33 sda9
 
In my case I used Linux fdisk to do the partitioning. One can tell the original partitioning was done with the older version of fdisk, because the first partition starts at sector 63.

[root@fatman ~]# fdisk -l /dev/sda
Code: [Select]
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0000d21c

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              63      626534      313236   83  Linux
/dev/sda2          626535    16820054     8096760   82  Linux swap / Solaris
/dev/sda3        16820055   114495254    48837600   83  Linux
/dev/sda4       114495255  1953520064   919512405    5  Extended
/dev/sda5       114495318   329332499   107418591   83  Linux
/dev/sda6       329332563   534434354   102550896   83  Linux
/dev/sda7       534434418   596943269    31254426   83  Linux
/dev/sda8       596943333   659452184    31254426   83  Linux
/dev/sda9       659452248   721961099    31254426   83  Linux
/dev/sda10      721961163   784470014    31254426   83  Linux
/dev/sda11      784470078   994198589   104864256   83  Linux
/dev/sda12      994198653  1623368249   314584798+  83  Linux
/dev/sda13     1623368313  1685893229    31262458+  83  Linux
/dev/sda14     1685893293  1749366044    31736376   83  Linux
/dev/sda15     1749366108  1816485614    33559753+  83  Linux
/dev/sda16     1816487663  1889887982    36700160   83  Linux
/dev/sda17     1889892080  1953520064    31813992+  83  Linux

With the new version of fdisk the first partition would start at sector 2048. However, the new version recognizes and can work with the older partitioning scheme if it already exists, and allows for the creation of the additional partitions past the previous 15 limit, so I added those two partitions when the new fdisk version became available.

While the new fdisk will work with the existing partitions as they are, it will not recreate them in the older manner should you delete a partition and then try to recreate it with the same start and end sectors. For that one would need the older version of fdisk which I still have available on some older liveCDs, but they, in turn, cannot see the last two partitions.

For this reason I will be relocating everything on the 1 TB drive to my new 2 TB drive, using rsync, then repartition the 1 TB drive entirely with the new version fdisk, then rsync everything back from the 2 TB drive to the new partitions on the 1 TB drive.
Old-Polack

Of what use be there for joy, if not for the sharing thereof?



Lest we forget...

Offline pags

  • Hero Member
  • *****
  • Posts: 2602
  • Keep it clean.
Re: How to format an external 1TB USB hard disk for backup, thanks.
« Reply #7 on: March 28, 2012, 11:37:43 AM »
What kind of error are you getting with NTFS (not that I'm opposed to re-formatting an external drive as ext3 or ext4...in fact it would be my first recommendation if you don't need to use it on Windows).

On this laptop, it dual boots with Windows 7:
Code: [Select]
[jpaglia@paglia-e6500 ~]$ df
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             173G  146G   19G  89% /
none                  4.0G   23M  4.0G   1% /tmp
/dev/sda2              50G   34G   17G  67% /mnt/windows
[jpaglia@paglia-e6500 ~]$ mount
/dev/sda5 on / type ext4 (rw,user_xattr)
none on /proc type proc (rw)
none on /dev/pts type devpts (rw)
none on /tmp type tmpfs (rw)
/dev/sda2 on /mnt/windows type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
gvfs-fuse-daemon on /home/jpaglia/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=jpaglia)
[jpaglia@paglia-e6500 ~]$ ls /mnt/windows/ -lhS
total 14G
-rwxrwxrwx 1 root root 8.0G Mar 23 14:10 pagefile.sys*
-rwxrwxrwx 1 root root 6.0G Mar 23 14:10 hiberfil.sys*
drwxrwxrwx 1 root root  16K Mar 23 14:11 Windows/
drwxrwxrwx 1 root root 8.0K Mar  5 22:29 807fef20ed760aa79d/
drwxrwxrwx 1 root root 8.0K Feb 20 11:41 Program Files (x86)/
drwxrwxrwx 1 root root 4.0K Feb 20 11:48 ProgramData/
drwxrwxrwx 1 root root 4.0K Mar 14 11:50 Program Files/
drwxrwxrwx 1 root root 4.0K Dec 20 10:49 $Recycle.Bin/
drwxrwxrwx 1 root root 4.0K Mar 14 11:50 System Volume Information/
drwxrwxrwx 1 root root 4.0K Jan 18 14:07 Users/
-rwxrwxrwx 1 root root 3.3K Dec 20 11:18 bootsqm.dat*
-rwxrwxrwx 1 root root  820 Dec 23 14:08 NcAdmin.log*
lrwxrwxrwx 2 root root   60 Jul 14  2009 Documents and Settings -> /mnt/windows/Users/
drwxrwxrwx 1 root root    0 Dec 20 11:27 dell/
drwxrwxrwx 1 root root    0 Dec 23 14:08 Network Connect 6.1.0/
drwxrwxrwx 1 root root    0 Dec 20 11:37 NVIDIA/
drwxrwxrwx 1 root root    0 Jul 13  2009 PerfLogs/
drwxrwxrwx 1 root root    0 Dec 12 15:40 Recovery/
[jpaglia@paglia-e6500 ~]$

The only large (>1GB) files on it now are pagefile.sys (8.0GB) and hiberfil.sys (6.0GB), but I did have a copy of a VirtualBox .vdi file copied to it, at one point (+30GB), but deleted it to make space when running Windows <shudder!>.

Offline bicol_willem

  • PCLinuxOS Tester
  • Hero Member
  • *******
  • Posts: 2376
Re: How to format an external 1TB USB hard disk for backup, thanks.
« Reply #8 on: March 28, 2012, 04:52:16 PM »
Thanks to all for your kind responses.

Hi bicol_willem, I understand the convenience of the NTFS, but I have not found the way to copy a large file without those file splitting programs, that is very time consuming and probably risky. Do you have a reliable and fast method for copying big files?

I was guessing that if my Host is PCLinuxOS I can copy to it a VM and if I need to run a VM inside a Windows VM I can share the directory of the Host, where I copied the VM from the USB Drive, and the file would be available for the guest Windows VM. Please correct me if I am wrong.

Since I come from many, many years on Windows I prefer the GUI solution, I do not want to make mistakes via the command line.

Hello Old-Polack, thanks for the explanation of the reserved space. I can not imagine formatting a HD with 17 partitions and worst than that then organizing the data and OSs, what a headache. VirtualBox is wonderful for this. I have never had a crash in a HD in 34 years working with computers, and I have used even an old computer with more than 12 years old HD.

Thanks and regards
joejac

I am not using NTFS .... I use "good ole FAT32 (LOL)
About copying large files ....... Now you said something.  I have been "bitten" several times over the years with something like that. Copying stopped exactly by 4 Gb .... (don't know it that is the case with you as well). There is a workaround for that but ..... for my life, I can't recall right now.
Maybe someone can bring this on again?

Offline djohnston

  • PCLinuxOS Tester
  • Hero Member
  • *******
  • Posts: 6224
  • I don't do Windows
Re: How to format an external 1TB USB hard disk for backup, thanks.
« Reply #9 on: March 28, 2012, 05:07:10 PM »
There is a workaround for that but ..... for my life, I can't recall right now.
Maybe someone can bring this on again?

Only one I know of is the newer exFAT system, and it's proprietary.
Bare metal                           VBox
AMD Athlon 7750 Dual-Core    Single core
4GiB RAM                              1GiB RAM
nVidia GeForce FX 5200          64MB video
LXDE 32bit                            KDE 64bit

Registered Linux User #416378

Offline bicol_willem

  • PCLinuxOS Tester
  • Hero Member
  • *******
  • Posts: 2376
Re: How to format an external 1TB USB hard disk for backup, thanks.
« Reply #10 on: March 28, 2012, 05:18:30 PM »
There is a workaround for that but ..... for my life, I can't recall right now.
Maybe someone can bring this on again?

Only one I know of is the newer exFAT system, and it's proprietary.


No, that is not the one, LOL.
It is really possible to copy large files a simple way with the stuff on hands.  As I said, I was bitten a few times over the years by this "inconvenience" as I call it.  I.e. a larger .VDI file of about 7 Gb would simply stop copying at exactly 4 Gb. I wasn't confronted with it lately but it did happen in the past and might happen tomorrow again. I just can't recall the work around ...... (sigh)

Offline djohnston

  • PCLinuxOS Tester
  • Hero Member
  • *******
  • Posts: 6224
  • I don't do Windows
Re: How to format an external 1TB USB hard disk for backup, thanks.
« Reply #11 on: March 28, 2012, 05:23:46 PM »
No, that is not the one, LOL.
It is really possible to copy large files a simple way with the stuff on hands.  As I said, I was bitten a few times over the years by this "inconvenience" as I call it.  I.e. a larger .VDI file of about 7 Gb would simply stop copying at exactly 4 Gb. I wasn't confronted with it lately but it did happen in the past and might happen tomorrow again. I just can't recall the work around ...... (sigh)

Splitting the file to be copied? That's the most commonly recommended method.
Bare metal                           VBox
AMD Athlon 7750 Dual-Core    Single core
4GiB RAM                              1GiB RAM
nVidia GeForce FX 5200          64MB video
LXDE 32bit                            KDE 64bit

Registered Linux User #416378

Offline Xenaflux

  • PCLinuxOS Tester
  • Hero Member
  • *******
  • Posts: 3829
Re: How to format an external 1TB USB hard disk for backup, thanks.
« Reply #12 on: March 28, 2012, 06:13:13 PM »
@ Old Polack
Thanks a zillion for the explanation.
I had some talks about it with some other guys , long ago,
but none of us could figure it out.

@ Just18
>>>Was that recently?<<<
in just under a year ago for the backup and about 18 months ago for the desktop dirve

Again, thanks .
Xena
The great thing in this world is not so much where we stand,
as in what direction we are moving.
                                                    (Oliver Wendell Holmes )