(SOLVED) Computer becomes unresponsive when copying files to memory stick

[veronicathecow]

SOLUTION From tschommer solved this problem.
In BIOS disable  Legacy USB support.

Thanks to everyone for their help.

Hi, I am using an Asus M4A78LT-M LE AMD 760G Socket AM3 3GHz quad core with Integrated ATI Radeon 3000 2 Gb RAM
When I try to copy a 20gb file from my sata HDD to an empty kingston class 4 micro SDHC 32 GB in a kingston usb reader the processor usage goes up near the 100% mark and the computer become unresponsive. Originally I tried with it the default formatting (fat32?) but it failed. So I formatted to ext3 and it was still causing 100% processor usage.
I then rebooted into Puppy 5.2.8 and it seems to be copying okay and the processor is sitting at about 3-4% and the machine runs as it should.
Am using KDE all up to date.
Any thoughts appreciated.

[AS]

Hi, I am using an Asus M4A78LT-M LE AMD 760G Socket AM3 3GHz quad core with Integrated ATI Radeon 3000 2 Gb RAM
When I try to copy a 20gb file from my sata HDD to an empty kingston class 4 micro SDHC 32 GB in a kingston usb reader the processor usage goes up near the 100% mark and the computer become unresponsive. Originally I tried with it the default formatting (fat32?) but it failed. So I formatted to ext3 and it was still causing 100% processor usage.
I then rebooted into Puppy 5.2.8 and it seems to be copying okay and the processor is sitting at about 3-4% and the machine runs as it should.
Am using KDE all up to date.
Any thoughts appreciated.

The correct and expected behavior is about 3%/4% CPU usage, the failure of fat32 is because of max file size limits 2/4 Gb of fat32.
Launch ksysguard (System Monitor) while copying, and look for which process is actually eating the CPU... could be a filemanager issue ... or something else...

AS

[menotu]

In addition I would also do it after a fresh reboot so you have the maximum amount of "services" available.

Is the SATA HD internal or external?

[veronicathecow]

Hi AS, very strange goings on. When copying sysguard shows X taking about 40% but only 10% when not copying.
However looking at system load it shows 2-3 processors running at 100% when copying
qps shows X is using  about 50% but if I speed up the refresh rate to 100ms to shows it using 200, sometimes 300%
The file copies fine in Puppy linux 5.2.8
I then tried to add extra files with PCLinuxos Dolphin and this seems to corrupt these files as when I try to plug in the flash drive mounting either takes a very long time or it will not mount at all.
I deleted these extra files in puppy and now the flash drives opens normally.

Hi menotu, sata is internal.

Makes me vary wary about using PCLinuxOS at the moment. Nothing worse than files corrupting.
Any help appreciated.

[AS]

Hi AS, very strange goings on. When copying sysguard shows X taking about 40% but only 10% when not copying.
However looking at system load it shows 2-3 processors running at 100% when copying
qps shows X is using  about 50% but if I speed up the refresh rate to 100ms to shows it using 200, sometimes 300%
The file copies fine in Puppy linux 5.2.8
I then tried to add extra files with PCLinuxos Dolphin and this seems to corrupt these files as when I try to plug in the flash drive mounting either takes a very long time or it will not mount at all.
I deleted these extra files in puppy and now the flash drives opens normally.

Hi menotu, sata is internal.

Makes me vary wary about using PCLinuxOS at the moment. Nothing worse than files corrupting.
Any help appreciated.

.... look like some issue around kernel/drivers ... what kernel are you running ? And what kernel is using puppy ?

[T6]

you mention a ati hd 3000

do you have the right driver loaded?  you could need a ati proprietary driver

what kernel do you have?

on konsole write uname -r to see that info

[veronicathecow]

Hi AS and T6, my kernel is 2.6.38.7-pclos1.bfs  and Puppy 5.2.8 uses 2.6.33.2
Using ATI driver with catalyst (Ver 11.5) control center. 8.85-110419a-118394c-ati which correctly recognizes as a Radeon 3000
Additionally when copying the 20gb from usb to sata my machine remains responsive but is using approx 40% of processing power at an average of 1.8GHz as shown on system monitor, system load. Process table does not show anything that is using more than a few percent.
qps shows X taking about 12% and dolphin 10% and other bits that fill upto the 40% that the system is using. Think I will stick with qps from now on.

[johan162]

With 99% certainty you are experiencing the effects of THP, Transparent Huge Pages, or rather the defragementation process. The good news is that there is a simply solution (see below)

Background (short and simplified)

The problem occurse in all recent kernels that has THP enabled (which most do).

Very simplistic the problem with unresponsive desktop when copying to a slow media (like your standard USB stick) is that the memory buffers will quickly fill up since the memory write to the stick can't keep up (cheap USB sticks only has an effective write rate of 1-3MB/s). This means that lot of buffers are allocated (common configuration is too allow up to 20% of the memory to be used in such write buffers).
When another process, say your browser of the desktop. tries to request more memory the kernel MM tries to find a  a new memory page. When THP is eabled the kernel tries to allocate a "huge page" and if that is not possible it tries to compact the used pages to (hopefully) make enouhg consecutive room for a huge page.

The problem now is that in the process of compaction it must do an fsync on the dirty buffers to have them committed to the USB stick (which was there destination). The fsync operation will now take a long time due to the slowness of the USB stick. As a side effect fsync will block a lot of other work, such as desktop or browsers.

So until all buffers are written to the USB stick (which could easily take several minutes for large files) the desktop is in effect unusable. Once the file is written the desktop will become "unlocked" and things canproceed as usual.

Solution

The way around this for desktop users is to disable the compaction for huge pages. This is done by giving the command (as root)

Code: [Select]

$> echo madvise > /sys/kernel/mm/transparent_hugepage/defrag
This will still leave compaction enabled but it will very rarely be performed. To completely turn it off give the command

Code: [Select]

$> echo never > /sys/kernel/mm/transparent_hugepage/defrag
Comment

There are some discussions around this on the kernel mailing list. The problem here is that THP is good thing (and so is the compaction) in many circumstances but copying large files on to slow devices is not one of them.

Hopefully this will see some resolution in fothcoming kernels since at the end of the day this behaviour is not really acceptable since copying large files onto a USB stick on a desktop is not exactly uncommon.

There is also a bug report on the kernel for this behaviour (see https://bugzilla.kernel.org/show_bug.cgi?id=34132)

There was also an article discussing a related issue in lwn.net not that long ago but I don't have a reference at hand now.

[veronicathecow]

Hi johan162, many thanks for replying to this thread.
I tried "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag" but if anything it seems to make it worse! The desktop almost completely froze.
However
"echo never > /sys/kernel/mm/transparent_hugepage/defrag" sorted the problem completely. Desktop was fully responsive, playing an Iplayer vid and flipping between tabs and different programs.

Many thanks for that, when copying large files to USB my machine was becmoing unresponsive for upto 15 minutes.
Cheers. Tony

[veronicathecow]

I posted this a year ago and marked as solved someone kindly gave me some code to run that solves the problem.
Trouble is I keep forgetting to run it and the system goes slow on me.
I tried making a file with echo never > /sys/kernel/mm/transparent_hugepage/defrag in it and making it executable.
I then added this script to the autostart in Configure your desktoop, system administration, startup and shutdown, script file
When I rebooted the desktop came up but no apps would load and the other autostart items like firefox failed to load. Also  a small round icon (run or execute?) came up beside the cursor.
In the end I booted into carolina puppy and deleted the file I had created.
Any help much appreciated
Thanks Tony

[µT6]

if it is still the same system, i would verify the hard disk with fsck to discard problems there

also what kernel are you still running?  same old 2.6.38.7 ? uname -r in konsole 

you shuold be running at least 2.6.38.8-pclos3.a64 or pae or bfs

what video driver are you running now?  is your video card still supported by amd driver?

[Just17]

I posted this a year ago and marked as solved someone kindly gave me some code to run that solves the problem.
Trouble is I keep forgetting to run it and the system goes slow on me.
I tried making a file with echo never > /sys/kernel/mm/transparent_hugepage/defrag in it and making it executable.
I then added this script to the autostart in Configure your desktoop, system administration, startup and shutdown, script file
When I rebooted the desktop came up but no apps would load and the other autostart items like firefox failed to load. Also  a small round icon (run or execute?) came up beside the cursor.
In the end I booted into carolina puppy and deleted the file I had created.
Any help much appreciated
Thanks Tony

Does the 'fix' currently work for you (when you remember to use it )?

presumably you need to run it as root?

[Bald Brick]

I'd try adding the line to /etc/rc.d/rc.local. (I don't have time to test it at the moment - maybe tomorrow morning.)

[tschommer]

I found the fix here.

If possible, disable USB legacy support in your Bios. It works, because I had the same problem up until 20 minutes ago, even with only one USB stick 

[veronicathecow]

Hi uT6 thanks for thoughts all is upto date including current Kernel. Video driver is fine as someone worked their magic when the new kernel came out.

Just17, fix worked partly but not fully , run as root.

Balbrick, thanks for that it was a good idea I had forgotten about using that.

tschommer. You have done it!
Thank you all for your time and friendly help. I have just copied 2gb to my htc desire and one 2 second glitch and then all was fine. I even ran a Dr who episode whilst whizzing the mouse around and it didn't miss a beat. The "echo never > /sys/kernel/mm/transparent_hugepage/defrag" was not running.