Blocking during heavy HD I/O Activity

[henryz]

I'm having a problem with dropped frames during video capture with Kino.  Specifically, Kino drops frames during heavy write activity to the HD.  This problem seems to have manifested itself in recent kernels.  It's like the machine is blocking until the data write is finished. 

Current kernel is 2.6.31.4-rt14 (RT preemption enabled).
Machine is a 3.2 Ghz Intel UP (Asus MB)
1 gig memory

Even setting Kino's scheduling to either RT 60 or RR 60 doesn't help.

Dropped frames were non-existant in earlier kernels 2.6.26 and older or at least with the software stack in use at that time.   I don't know whether this is related to the introduction of the Completely Fair Scheduler or something else. 

Any suggestions on where to start looking for the bottleneck?

[gseaman]

Other than a couple of developers, I don't think anyone here is using 2.6.31.x kernels. I'm running 2.6.31.5-bfs, with con kolivas' new patches. It seems to result in smoother video, but not dramatically so. It might help.

I found this link about changing the scheduler that is being used:
http://www.linux-mag.com/id/7564/2/

Galen

[henryz]

Thanks for the link on scheduler tuning.  It appears that the problem I'm seeing is not scheduler-related (tried all variants with no significant change in behavior).  I'll rebuild my old 2.6.26.8-rt15 kernel and see if the problem goes away.  Just did a system backup, so it would be a good time to experiment.

[henryz]

I reloaded an old, pre-toolchain backup (ie. gcc-4.1) with the 2.6.26.8-rt15 kernel and retested kino.  Got no dropped frames, I mean none!  Looks like data was being written to the HD continuously, instead of in bursts.  Unfortunately, 2.6.26-8-rt15 will compile under the new toolchain and boot, but it can't find and mount the root filesystem and panics.

I did notice that the default ext3 journaling mode has been changed to writeback instead of ordered.  Changing the journaling mode  to ordered data, using the 2.6.31.4-rt14 kernel (fully updated system) greatly helps the kino problem, because data is written to the HD at fixed intervals instead of relying on the system's sync.  Overall performance still does not equal that of the older system.  Problem is that I can't tell if it's the kernel, toolchain or ext3 code changes that have degraded performance.  Do any of you guru's out there have an opinion or suggestions about this?

(PS - it's not the scheduler - changing between the various schedulers does not affect file write results noticably.)

[gseaman]

Many aspects of the linux kernel have become speedier since 2.6.26.x, but the developers have acknowledged that it has become larger and less interactive, which might be the problem in this case. The bfs scheduler that I mentioned is not in the mainline kernel so you have to apply a patch from here:
http://ck.kolivas.org/patches/bfs/2.6.31-sched-bfs-311.patch

If this patch helps for kino, it will still not be as smooth as 2.6.26, but it might be close. The kernel developers have put some improvements into the 2.6.32.x series and have discussed addressing the problem, so there may be more to come.

Galen

[henryz]

Sounds like kernel bloat has become rampant!  Anyway - the ordered-data journal change at least functions as a workaround for the frame dropping in Kino.  I hope the kernel developers, in their wisdom, address these interactivity issues, soon.  Maybe Con will aggravate them enough to catalyze some action.