(SOLVED) Flash Drive observation

[Ferdes Fides]

Sent an email request for product info to
Kingston Tech support for cache or disk buffer
size for a Kingston DT101G2 8GB and 16GB
flash drive.   Hope there's a big enough
difference to explain something about
those flash drive's performance then.

I'll post back when they do.

FF

[Just18]

Sent an email request for product info to
Kingston Tech support for cache or disk buffer
size for a Kingston DT101G2 8GB and 16GB
flash drive.   Hope there's a big enough
difference to explain something about
those flash drive's performance then.

I'll post back when they do.

FF

Hopefully they will have the info for you 

[Ferdes Fides]

Hopefully they will have the info for you 

Well they emailed back there is no difference, actually there
is no cache or buffer at all they said.   

Just restored from an archive and made the RAM cache
bigger for private browsing.   Seems to be a little
better overall.

FF

[AS]

Hopefully they will have the info for you 

Well they emailed back there is no difference, actually there
is no cache or buffer at all they said.   

Just restored from an archive and made the RAM cache
bigger for private browsing.   Seems to be a little
better overall.

FF

Make sense, as disk buffer/cache was introduced to smooth the difference of speed between processing operations and mechanical disk operations, and cache act as a fast but temporary storage.

May be you have used different filesystem on the two USB stick ?  unlikely, but let me ask about :-)

[Ferdes Fides]

May be you have used different filesystem on the two USB stick ?  unlikely, but let me ask about :-)

Both are ext4.   The bootup started to slow a little and the Firefox
screens were stalling a bit, things that should be in RAM working.
On a hunch some upgrade caused a 1000 fragments I restored
from an unfragmented archive, I know most people will say Linux
doesn't have fragments, but a hunch nonetheless.   

Then just increased the RAM cache on Firefox from 18MB to 100MB
to see what happens.  Probably too big.   Disk cache is zero.

I'll get to use it a little more over the weekend.

Thanks for the response.

FF

[AS]

May be you have used different filesystem on the two USB stick ?  unlikely, but let me ask about :-)

Both are ext4.   The bootup started to slow a little and the Firefox
screens were stalling a bit, things that should be in RAM working.
On a hunch some upgrade caused a 1000 fragments I restored
from an unfragmented archive, I know most people will say Linux
doesn't have fragments, but a hunch nonetheless.   

Then just increased the RAM cache on Firefox from 18MB to 100MB
to see what happens.  Probably too big.   Disk cache is zero.

I'll get to use it a little more over the weekend.

Thanks for the response.

FF

Of course Linux filesystem do fragments, it also implements strategies to reduce fragmentation, but definitely do fragments.
That said, fragments should not be a performance issue when using flash memory, not like on conventional disks.

I would try to change journal behavior, data=writeback as option in /etc/fstab or even to temporarily disable the journal using the option nointegrity.

Also, the default commit time is 5 secs, (max time before to write pending blocks), you could try to increase the time to 20 secs
adding commit=20 as option in /etc/fstab.

You can find further info about mount options using man mount.

Also, I'm assuming you have performed a plain installation on the USB stick and it's not a LiveUSB with persistence.

[Ferdes Fides]

Also, I'm assuming you have performed a plain installation on the USB stick and it's not a LiveUSB with persistence.

Yes it's a plain installation.   I'll look into "man mount" etc.  over the weekend.
One thing I forgot to consider is hardware limitations on the overall machine.
For example, one or two html pages per page screen is fine, but 5 or 10 pages on
one page screen does overtax it a bit, and I think that's one thing it coudn't do very well, as
well as having many tabs open searching, etc..   Have to go a little easier on this machine
perhaps.  

Thanks for the info.

FF

[djohnston]

Online defragmentation

While delayed allocation, extents and multiblock allocation help to reduce the fragmentation, with usage filesystems can still fragment. For example: You write three files in a directory and continually on the disk. Some day you need to update the file of the middle, but the updated file has grown a bit, so there's not enough room for it. You have no option but fragment the excess of data to another place of the disk, which will cause a seek, or allocate the updated file continually in another place, far from the other two files, resulting in seeks if an application needs to read all the files on a directory (say, a file manager doing thumbnails on a directory full of images). Besides, the filesystem can only care about certain types of fragmentation, it can't know, for example, that it must keep all the boot-related files contiguous, because it doesn't know which files are boot-related. To solve this issue, Ext4 will support online fragmentation, and there's a e4defrag tool which can defragment individual files or the whole filesystem.

[Ferdes Fides]

To solve this issue, Ext4 will support online fragmentation, and there's a e4defrag tool which can defragment individual files or the whole filesystem.

Hi,

And thanks for the info.   That's not in the repo, good it's not probably, we'd
probably have defragging every night.   Contrary to textbook theory I think
fragments affect performance on all media, including HD, flash drives, USB
drives, etc..

If one takes a distro, upgrades it monthly, and after one year backups and restores
that drive from an unfragmented archive, I would expect from my experience only
a performance increase over 33 %.    Just the drive performance, especially with
larger files and larger programs.   That's what is happening here.   

IMHO

Have a good one.

FF

[AS]

Online defragmentation

While delayed allocation, extents and multiblock allocation help to reduce the fragmentation, with usage filesystems can still fragment. For example: You write three files in a directory and continually on the disk. Some day you need to update the file of the middle, but the updated file has grown a bit, so there's not enough room for it. You have no option but fragment the excess of data to another place of the disk, which will cause a seek, or allocate the updated file continually in another place, far from the other two files, resulting in seeks if an application needs to read all the files on a directory (say, a file manager doing thumbnails on a directory full of images). Besides, the filesystem can only care about certain types of fragmentation, it can't know, for example, that it must keep all the boot-related files contiguous, because it doesn't know which files are boot-related. To solve this issue, Ext4 will support online fragmentation, and there's a e4defrag tool which can defragment individual files or the whole filesystem.

I was aware of this tool and the related system call, but as far as I remember from a few months ago when I looked at this argument, it's still a 'beta functionality' not really usable on 'production' systems.

AS

[AS]

If one takes a distro, upgrades it monthly, and after one year backups and restores
that drive from an unfragmented archive, I would expect from my experience only
a performance increase over 33 %.    Just the drive performance, especially with
larger files and larger programs.   That's what is happening here.   

+1

I have verified this for all fs including ext2 and ext3, only recently I started to use ext4 therefore my experience is limited on this specific fs version.

AS

[Just18]

If one takes a distro, upgrades it monthly, and after one year backups and restores
that drive from an unfragmented archive, I would expect from my experience only
a performance increase over 33 %.    Just the drive performance, especially with
larger files and larger programs.   That's what is happening here.   

+1

I have verified this for all fs including ext2 and ext3, only recently I started to use ext4 therefore my experience is limited on this specific fs version.

AS

Simplest method I have used is to rsync the partition to another, format the original, and rsync back again ....  all should be defragmented .....  or appeared so to me  .....

[Ferdes Fides]

+1

I have verified this for all fs including ext2 and ext3, only recently I started to use ext4 therefore my experience is limited on this specific fs version.

AS

Thanks for the +1.   My first PCL I put on a flash drive with xfs.   They (xfs) had some
lengthy changelogs so I switched everything to ext4.   Google chose ext4 over xfs
also last year for it's systems.

[Ferdes Fides]

Simplest method I have used is to rsync the partition to another, format the original, and rsync back again ....  all should be defragmented .....  or appeared so to me  .....

If rsync has a verbose option use it next time, you might be able to
see it working on each file in each directory, and take the time also.
My FSArchiver does so.   Plus you can keep several month's archives
compressed in minimum space.

FF

[djohnston]

There is a way to check for fragmentation. This won't defragment anything, just give you a summary.

su -
e2fsck -fvE fragcheck /dev/sdx0

Substitute your partition for sdx0.


  176193 inodes used (0.96%)
     126 non-contiguous files (0.1%)
      24 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 0/0/0
         Extent depth histogram: 175625/55
25891803 blocks used (35.35%)
       0 bad blocks
       7 large files

  163034 regular files
   12646 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
     503 symbolic links (503 fast symbolic links)
       0 sockets
--------
  176183 files