Extending mylivecd 4 GB size limit

[AS]

It is nearly done!  

I was actually successful in building ISO images larger than 4 GB:
The GUI still need some work to be finished ... long TODO list, although the prototype is actually working ...  





mylivecd:
The ISO size 4GB limit come from ISO9660 filesystem:
http://en.wikipedia.org/wiki/ISO_9660#The_2.2F4_GiB_file_size_limit

to overcome the ISO9660 limitations, it is necessary to change filesystem type, using UDF:
http://en.wikipedia.org/wiki/Universal_Disc_Format

... so adding support for UDF in mylivecd ... and some other tweak ... finally we can implement support for larger ISO images.

The basic tests I have done show that the resulting ISO image is properly built, and can successfully boot in VirtualBox.
Of course, extended testing will be needed before the official release ... but the path should be straight.

Building and testing remaster is time consuming especially for large ISO images, so I have a few questions:

1) mediacheck/implantisomd5
One thing that now look like broken (when using ISO > 4GB) is mediacheck, it was already removed previously because of other (non-)issues, so I assume we can easily do without, comments about are welcome.

2) ISO9660 vs. UDF, for ISO images lesser than 4 GB
That's my concern, as I understood from the documentation I have found, UDF is a complete replacement for ISO9660, therefore I assume it can be used for smaller images like those used on CDs.

If the filesystem changes is a problem, I think we can however implement some conditional build, using the traditional ISO 9660 for smaller images and using UDF for larger images.

Comments about are greatly desired.  

3) GUI
As you can see from the above screen-shot, the GUI offer some chance to select the filesystems that must be be included in the remaster, this is a feature that should replace the current mylivecd option --nodir.

I need also to explain that the GUI is designed to be purely a front-end to mylivecd, not an alternative program, doing so will assure back-compatibility and the possibility to use the CLI interface, granting the same results using both programs.

I want to highlight the fact that selecting the filesystem is semantically different from the --nodir option, because the latter require as argument an "expression".

Therefore I'm probably going to add a further option to mylivecd, say --excl which will require as argument a "directory" and not an "expression".

It is a small difference, but it is still a difference, --nodir require "^/home/user/something", while --excl will require "/home/user/something".

Please, comment about, this is the most critical point related to the GUI.

Also, please don't ask implementation details right now, very soon the enhanced mylivecd will be available for testing, as I consider the changes a prerequisite before further development of the GUI.

A BIG Thanks to Just18, who kindly shared with me some of the issues, in the end pointing my attention to the right direction.

Now you know one of the reason for my poll, please vote:
mylivecd: How would you use a remaster image larger than 4 GB ?

Thanks for reading, now you can have a coffee!  

AS

[pinoc]

Hi AS,

though unfortunately I had no time for testing this yet, I'm convinced you did an outstanding job on this new invaluable project! I think I can speak for many if I say that we are very happy and lucky to have you in our community. A big THANK YOU once more for joining in here and the countless high-quality help you provide not only here but in general throughout the forum, it is very much appreciated and we have learned a lot!

Thanks again for spawning off this great new exciting project!
all the best!
-p.

[parnote]

Wow, AS! That looks awesome!

parnote

[Just17]

Congrats AS .....  glad you got the problem located 

There are several aspects that will need serious testing, as you well know, but here are a couple of thoughts ...

the extended size of the large loop image is - if I read correctly - deliberately misrepresented in some situations.
whether this will have an affect on some users or not I have no idea

if burned to optical media, will copying of the large file from the media to a HDD be correct?

initially at least, I would like to see the traditional method used for smaller ISOs .....  maybe I am being too conservative but 'if it ain't broke then don't fix it' 

also there may be some problems with hardware compatibility between the new media and older CD ROM drives.

using  --nodir ^/home/user/something presently causes each file in the directory to be listed for exclusion.
if there was a means to omit the directory itself then there should be no need for all its contents to be listed for exclusion. I don't know if this is possible 
but if   --excl  can do this then it would be a big advance

at present there is no means of excluding a directory (or mount point) and then including some part of what has been excluded.
as an example, I could have a directory (Videos) with lots of videos in different directories, but might like to include one of those sub directories.
the logical thing to my mind would be to exclude all contents of the Videos directory, and then include the one I wanted.
the alternative is to list all the excluded directories, which is the present way, and to not list the one I want to keep.
I guess some thought needs to go into all that and no doubt there will be limitations on what can be done.

Congrats again on great progress

regards

[AS]

Congrats AS .....  glad you got the problem located  

 

There are several aspects that will need serious testing, as you well know, but here are a couple of thoughts ...

Thank you, I greatly appreciate your help.  

the extended size of the large loop image is - if I read correctly - deliberately misrepresented in some situations.
whether this will have an affect on some users or not I have no idea

You read it correctly, but this should not be a real issue, let me show the practical aspects:

# ls -l livecd.iso
-rw-r--r-- 1 root root 7625451520 Jan 28 22:55 livecd.iso

# mount -t iso9660 livecd.iso /iso
# ls -l /iso
total 3237822
dr-xr-xr-x 3 root root       2048 Jan 28 22:20 boot/
dr-xr-xr-x 2 root root       8192 Jan 28 22:52 isolinux/
-r--r--r-- 1 root root 3315519488 Jan 28 22:52 livecd.sqfs
# umount /iso
#
# mount -t udf livecd.iso /iso
# ls -l /iso
total 7432122
dr-xr-xr-x 3 root root         84 Jan 28 22:20 boot/
dr-xr-xr-x 2 root root       2560 Jan 28 22:52 isolinux/
-r--r--r-- 1 root root 7610486784 Jan 28 22:52 livecd.sqfs
# umount /iso
#
# md5sum livecd.iso
5287e3329e4cac2549e0f56cdc6acdee  livecd.iso
#

if burned to optical media, will copying of the large file from the media to a HDD be correct?

EDIT: I misunderstood the question, the answer however is that if the media is mounted correctly (UDF) there will be no problem.

I would say that, if any, it would be a separate issue, the job of mylivecd is to prepare an ISO image, anyway k3b look fine, although I have not performed any writing test so far:

initially at least, I would like to see the traditional method used for smaller ISOs .....  maybe I am being too conservative but 'if it ain't broke then don't fix it'  

also there may be some problems with hardware compatibility between the new media and older CD ROM drives.

while I fully agree about 'if it ain't broke then don't fix it', some changes need to be considered, also some extended testing should solve the doubts.

using  --nodir ^/home/user/something presently causes each file in the directory to be listed for exclusion.
if there was a means to omit the directory itself then there should be no need for all its contents to be listed for exclusion. I don't know if this is possible  
but if   --excl  can do this then it would be a big advance

I was thinking at this problem, a possible solutions is a further option: --excl-content (self explanatory)

at present there is no means of excluding a directory (or mount point) and then including some part of what has been excluded.
as an example, I could have a directory (Videos) with lots of videos in different directories, but might like to include one of those sub directories.
the logical thing to my mind would be to exclude all contents of the Videos directory, and then include the one I wanted.
the alternative is to list all the excluded directories, which is the present way, and to not list the one I want to keep.
I guess some thought needs to go into all that and no doubt there will be limitations on what can be done.

In a far future I hope to substitute the mounted filesystem you see at top left with a full filesystem browser like used on several burning programs to select files and directory, obviously this will require a lot of underlying work.

Consider that the GUI was primarily intended as an aid to those users that have more difficult to use the CLI, but that probably do have simple setup, the goal is ambitious, and I strongly prefer to implement new features a few at a time, otherwise I'm sure it would become an endless project.  

Congrats again on great progress

regards

Thank you, I greatly appreciated your comments.  

AS

[AS]

A side note about the udf advantage when using DVDs:

The maximum size we can use on a DVD increase from 4.3GB (4 GiB) to 4.7GB (4.3GiB).
Made a test, using the following image:

# ls -l livecd.iso
-rw-r--r-- 1 root root 4650483712 Jan 30 00:18 livecd.iso

Burned using K3b, md5sum verified, successfully booted and working as expected, the followings are the "mediainfo" from K3b:



So far, so good! 

AS

[melodie]

This is very beautiful!!

For mediacheck : I was never confident in this feature in "other distros" providing it. I prefer md5sum and sha1sum.

[AS]

This is very beautiful!!

For mediacheck : I was never confident in this feature in "other distros" providing it. I prefer md5sum and sha1sum.

Further test, on a newer but similar ISO, removed the package "mediacheck" and installed the package "isomd5sum":
NOTE: there is a conflict between the above two packages, never install both.

# ls -l livecd.iso
-rw-r--r-- 1 root root 4649965568 Jan 30 03:06 livecd.iso
# md5sum livecd.iso
71a620350d96aa58ba916854e1bc4b0a  livecd.iso
# implantisomd5 livecd.iso
Inserting md5sum into iso image...
md5 = 587bc8fcce2e105d2474a1dbd1f04940
Inserting fragment md5sums into iso image...
fragmd5 = 5d5b23e87bf4638f247b587b7d59846214788ca2d683541fbb58876e5fcb
frags = 20
Setting supported flag to 0
# checkisomd5 --verbose livecd.iso
livecd.iso:   587bc8fcce2e105d2474a1dbd1f04940
Fragment sums: 5d5b23e87bf4638f247b587b7d59846214788ca2d683541fbb58876e5fcb
Fragment count: 20
Press [Esc] to abort check.
Checking: 100.0%

The media check is complete, the result is: PASS.

It is OK to use this media.

The result from using the package isomd5sum looks good ... but ... md5sum is broken ? 

[Just17]

The result from using the package isomd5sum looks good ... but ... md5sum is broken ?

Was there not a discussion on this topic some time ago? ... 

[AS]

The result from using the package isomd5sum looks good ... but ... md5sum is broken ?

Was there not a discussion on this topic some time ago? ... 

this one ? (at the end, a conflict between the two packages was found)
http://www.pclinuxos.com/forum/index.php/topic,97871.0.html

[Just17]

The result from using the package isomd5sum looks good ... but ... md5sum is broken ?

Was there not a discussion on this topic some time ago? ... 

this one ? (at the end, a conflict between the two packages was found)
http://www.pclinuxos.com/forum/index.php/topic,97871.0.html

Probably what I had in mind  ......

[AS]

There is a tool in repo "iat-0.1.3" that perform ISO9660 analysis, the updated version 0.1.7 add support for UDF.

http://iat.berlios.de/index.php?page=Features
http://developer.berlios.de/project/showfiles.php?group_id=6784

Code: [Select]

# iat --debug -i as.iso
Get info from as.iso
 100% [:=====================:]
Iso9660 Analyzer Tool v0.1.7
Pregap  : (0)
Block   : (2048)
Size    : (4402710528) bytes
TYPE            : (1)
ID              : CD001
VERSION         : (1)
SYSTEM ID       : LINUX
VOLUME ID       : livecd 2012013103:51
VOLUME SPACE    : 2149761
VOLUME SET      : 1
VOLUME NUMBER   : 1
LOGICAL BLOCK   : 2048
PATH TABLE      : 50
PATH TABLE L    : 320
TYPE TABLE L    : 0
PATH TABLE M    : 322
TYPE TABLE M    : 0
ROOT DIR        : 34
VOLUME SET ID   :
PUBLISHER       : http://pclinuxos.com/
PREPARED        : mylivecd 0.9.4
APPLICATION     : PCLinuxOS release (PCLinuxOS) for LiveCD
COPYRIGHT       :
ABSTRACT        :
CREATED         : 2012013104084300
MODIFICATED     : 2012013104084300
EXPIRE          : 0000000000000000
DATA            : 2012013104084300
FILE VERSION    : 1
APPLICATION USE :                                                                                                                                                                
Here the updated packages:

EDIT: download links removed because the package is now available on PCLinuxOS repos.