Re: Split from: MyLiveCD Guide

[AS]

I believe it is the /union fs limit.

iso9660 file size limit, see 2/4 GiB limit section: http://en.wikipedia.org/wiki/ISO_9660

At the risk of this developing into a philosophical discussion, the ISO 9660 file size limit refers to the maximum single file size  that can be written to an optical disk, this is similar to the file size limitation of FAT32, this should not be confused with the image file size which is being written to hard disk, in the case of Linux mostly with file system ext2 ext3 ext4 which have 16GB limit, although the image file has the extension .iso, it is not being written to an ISO file system but to the file system of your writing media, in this case your HDD.
The image file with the the .iso extension is actually an archive of many files and directories, when it is decompressed to an optical media it deflates and re-creates the original file and directory structure.
please see:
http://en.wikipedia.org/wiki/Comparison_of_file_systems
From what I have been able to find out, the 4 or 4.3 GB limit is not due to ext file system, and under ext2-3-4 you can have an image file or any file whether it ends with .iso, .avi, .mkv  or any extension up to 16GB and up to 2TB in some special circumstances.
I have not been able to find documentation of the /union fs or the file size limit,
If you know of a link, please post it.
Cheers
 

Hi hasmak,

Your point is correct, so it's mine because I've written file size limit. Instead should be noted that actually the ISO contains some files necessary to boot (the equivalent of the /boot directory ...) and a single file liveCD.sqfs which account for the 99% of the ISO image and that contain the compressed/archived image of the installation. From a practical point ov view this means that the mylivecd ISO limit is approx. 4 Gb.

unionfs (or afs, or gvfs) is a virtual filesystem that allow to overlap two or more filesystems, meaning that the directory structure of one filesystem overlap the directory structure of a second filesystem, this allow to mount a read-only filesystem (from liveCD.sqfs, loop mounted) to a read-write filesystem (from initrd), to form a single read-write filesystem.

AS

[hasmak]

AS
Thanks for the explanation, it is the size of the .sqfs file that cause the error when the .iso is being written.
Do you know if there is a way to get mylivecd to calculate (even roughly) the projected size of the .sqfs file size, compare that with free space and inform the user??
That would really help the users and avoid a lot of lost time and effort.
Cheers

[melodie]

AS
Thanks for the explanation, it is the size of the .sqfs file that cause the error when the .iso is being written.
Do you know if there is a way to get mylivecd to calculate (even roughly) the projected size of the .sqfs file size, compare that with free space and inform the user??
That would really help the users and avoid a lot of lost time and effort.
Cheers

Hi,

What makes it difficult to say is that some file are already very much compressed, and some aren't. For example, often if we do a remaster from a 2500 MB or under, we used to get almost 700 MB but not quite. The PCLinuxOS Education I remastered is almost 700 (686 MB if I remember well) for an install which uses a bit less than 2300 MB. So the installed size of less than 2300 should have produced a smaller iso, but this was not the case. The reason is that there are many sound files in it (games for children with sound and music).

In addition, with the options of compression, default xz very much compressed, --xz less compressed and --gzip very much less compressed but faster... I don't quite see how you could know before.

If the user has personal data such as music, videos, all large files and or very much compressed files, the most easy procedure could be to manage to get the future iso lower : first, to place them all in one directory that he will exclude (with the options of mylivecd command line), and second, to do a good cleaning in the system : use Bleachbit after having configured it correctly, and Bleachbit-root as well, remove the logs files, or at least the old ones, most of the /usr/share/doc files, except the ones you think you might need, and the directory /usr/share/doc/HTML, if you want to keep it (links to the PCLinuxOS website in Firefox, usually)...

But saying before how much will be left... Less than half, hopefully ? :p

[AS]

Agreed with Melodie explanation. It's not possible to calculate the compression results, until you have already done it.

Roughly you can say the compressed image will result in a 35% of the original size, and that is all, so you can do that yourself.
When you are going to backup a system that use between 10 and 13 Gb you know (in advance) that you are in the mydeathcd (TM) danger zone! 

AS

[hasmak]

Melodie, As, Neal
I was not thinking of my self, I was thinking of the average user who spend hours trying to remaster without success and then spends more hours in forums trying to figure out what went wrong. It would be helpful for the user to have a clear message of what to do and what went wrong if there is an error.
You all probably know more about how mylivecd works, that leads to a couple of questions;
the first is who is the developer or developers
the second question is theoretical; Is it possible (in theory) to split the sqfs file into 1GB chunks (same idea as vob files in DVD) and mount the multiple chunks at boot time in separate volumes or merged volume using the same approach of unionfs?
if this is possible, then you should be able to create larger images for DVD9 or even larger for bootable external HDD
If it can be done it would be interesting to try
Cheers

[melodie]

When you are going to backup a system that use between 10 and 13 Gb you know (in advance) that you are in the mydeathcd (TM) danger zone! 

AS

LOL !

It is much better to make a big tarball, with bunzip, tar or xz, of all the personal datas, then to copy the archive to another support, external drive or big pendrive, or even to copy it without archieving it, directly from one machine to another when possible, through the local network. This way only the system has to be remastered.

Melodie, As, Neal
I was not thinking of my self, I was thinking of the average user who spend hours trying to remaster without success and then spends more hours in forums trying to figure out what went wrong. It would be helpful for the user to have a clear message of what to do and what went wrong if there is an error.

More we practice, more we learn. More we make errors more we practice. Sometimes, more we make errors more we learn, for some of us it is not always true unfortunately.

Well I remaster a lot for others, but the one day, recently when I needed to remaster for myself it was a different scheme, and AS helped me while loling at me at same time !

[Neal ManBear]

Melodie, As, Neal
I was not thinking of my self, I was thinking of the average user who spend hours trying to remaster without success and then spends more hours in forums trying to figure out what went wrong. It would be helpful for the user to have a clear message of what to do and what went wrong if there is an error.
You all probably know more about how mylivecd works, that leads to a couple of questions;
the first is who is the developer or developers

Texstar is the author. And I believe etjr assists him in maintaining it.     

the second question is theoretical; Is it possible (in theory) to split the sqfs file into 1GB chunks (same idea as vob files in DVD) and mount the multiple chunks at boot time in separate volumes or merged volume using the same approach of unionfs?
if this is possible, then you should be able to create larger images for DVD9 or even larger for bootable external HDD
If it can be done it would be interesting to try
Cheers

So far as I know, you can not split sqfs. Doing so would make it unbootable.     

[hasmak]

So far as I know, you can not split sqfs. Doing so would make it unbootable.     

Neal
there is more than one way to skin a cat 
I'll think about it and let you know if I come up with a new idea
cheers

[Neal ManBear]

So far as I know, you can not split sqfs. Doing so would make it unbootable.     

Neal
there is more than one way to skin a cat 

Of course there is. But there is only one way to not get scratched and bitten --- don't even try to skin the cat.    

I'll think about it and let you know if I come up with a new idea
cheers

In any Linux system, there is always more than one way to do something. However, altering some parts of the system from their default behavior is inherently dangerous to the system. Go forward with care.     

[melodie]

Hi,

I have heard about tools allowing to chunk big files and reconstitute them later, but I don't think it's worth the time you will dedicate to it for the use we have of the distro.

On the opposite, you can if you wish, make one iso for the personal datas, and one for the system. For the system, with the mylivecd script, and for the datas, with the dd command. There you could make as many as you wish: one for the Sci-Fi movies, one for the Western movies, one for the Desktop work, one for the files related to your bills...

Code:

dd if=/home/user/Documents/Voyages_pictures of=my_voyages_pictures.iso

To decompress, reverse the command.

Wouldn't that look KISS ?

[hasmak]

Hi,

I have heard about tools allowing to chunk big files and reconstitute them later, but I don't think it's worth the time you will dedicate to it for the use we have of the distro.

Melodie
There are many reason for remastering, most people will remaster their installation as a sort of customized backup, for those, it should be easy for each person to decide what to do with their data files (copy  to external media, copy to another partition and unmount it ...etc)
Other people may have other reasons, like creating live specialized distros for domains like education, medicine, engineering ..etc or backing up an enterprise central system, such installations are inherently large and it would be very usefull for such people (and others) to break the 4BG barier
My own reasons are even different
First, I don't like to be told that "it can't be done". In computing, almost every thing can be done, you will never know until you try, and even if the tools are not here today, someone will make the tools tomorrow (probably the person who was told it can't be done, and decided to do it )
Second, I like to keep my brain engaged, I am 67 years old (a dinosaur) If I don't use it, it will die from disuse atrophy 
 

Wouldn't that look KISS ?

Some of us don't know how to KISS 

Cheers

[melodie]

First, I don't like to be told that "it can't be done". In computing, almost every thing can be done, you will never know until you try, and even if the tools are not here today, someone will make the tools tomorrow (probably the person who was told it can't be done, and decided to do it )
Second, I like to keep my brain engaged, I am 67 years old (a dinosaur) If I don't use it, it will die from disuse atrophy 

Precisely, the method you talk about has not been used by anyone I saw talking about computing in the forums where I go, for a long long time. It looks like prehistoric method.

If you have a big hard drive to backup and restore, what about giving a try to Clonezilla ? It's a fantastic tool. It allows you to backup diverse methods, one being from a pendrive you can tunnel you backup through ssh. You can backup all or partition per partition...

You talk about chunks, there are your chunks !

I mean, if you chunk from the source, instead of cutting a big file into pieces, it seems that the actual tools are fit for that. And also, if you want to create a very big remaster, you might give it a try even if the limit size advised is of 4 Gb. I think I red from some members that they are used to remaster whatever size fits their needs. (They were coming to rant about the fact they were not succeeding anymore... )

You will need a large amount of free space for the temporary files...

I hope this answer suits your will to train.

[hasmak]

Precisely, the method you talk about has not been used by anyone I saw talking about computing in the forums where I go, for a long long time. It looks like prehistoric method.

Thank you for your comments

[melodie]

Precisely, the method you talk about has not been used by anyone I saw talking about computing in the forums where I go, for a long long time. It looks like prehistoric method.

Thank you for your comments

Hi,

Please, don't take it bad. I think I understand your point of view, at least partly. I just wanted to present you other possibilities for a similar target. (More used actually also... )

Regards,
Mélodie

[hasmak]

Melodie
I am old enough and experienced enough to backup my systems whether by remastering, data back up, disk image, sector by sector using any of the dozens of tools available.
This discussion was never about me backing my system. It is about tossing ideas, and opening possibilities for a simplified error free method of remastering a Linux system for the average beginner user (if you look around, some of them don't even know what to do with the ISO image).
If you think this is worth while, then by all means we can discuss how to do that, if it is not worth while for you, then there is no point of discussing it any further.
Best regards