[LAD] Kontakt Spikes

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
To: <linux-audio-dev@...>
Cc: Michael Ost <most@...>
Date: Saturday, October 8, 2011 - 1:26 am

Hi list,

We are seeing unexpected interruptions of SCHED_RR audio processing
threads, and are struggling to understand why they are happening. Does
anyone have any good tips or tools to suggest to help figure out what is
preempting or delaying realtime audio threads?

The issues are coming up with Receptor [see (*) below for an intro]
running its 2.6.24.3 CCRMA based kernel. The bug appears with Receptor
in its "dual core mode", with three instances of Native Instruments'
Kontakt 4 in DFD + multi-core mode. Either lots of held notes, or large
patches are needed to get the spikes.

With these settings there are 5 SCHED_RR threads processing audio (on a
two-core system). 2 are from Receptor, and 1 from each instance of
Kontakt. These Kontakt "helper threads" are released to do work as
possible while the audio thread is processing.

Kontakt/DFD is using mmap to bring its files into memory. This is done
in a lower priority "DFD thread", and the mapped memory is used by the
r/t audio threads.

DFD is important because the problems don't happen without it. And the
high SCHED_RR thread count is important, because the problems also don't
happen if we reduce the count.

When the spike happens there are no:
* wine bottlenecks
* system calls
* threads blocking on each other
* page faults during audio processing

There do appear to be "involuntary context switches" (as reported by
getrusage) when the spikes happen. This makes it seem like the scheduler
is interrupting our threads. But how do you figure out why that is
happening?

There aren't many threads in the system with higher priority. All of the
5 processing threads are SCHED_RR/76. The higher priority threads in the
system are:
* migration/0 - FIFO/99
* migration/1 - FIFO/99
* watchdog/0 - FIFO/99
* watchdog/1 - FIFO/99
* posix_cpu_timer (x2) - FIFO/99
* IRQ8 (rtc) - FIFO/99
* IRQ20 (our audio card) - FIFO 77

Could other kernel activity interrupt the audio threads? Are there
issues with memory mapping, that can block other unrelated threads? Are
there just too danged many SCHED_RR threads fighting for two cores?

Anyone have any suggestions for how to trace the scheduler, and thread
or process interruptions?

Apologies for the lengthy post, but this is a tricky subject.

Thanks for any tips or insights,

Michael Ost
Muse Research & Development

(*) Receptor (http://www.museresearch.com/products/receptor.php) is a
standalone hardware VST plugin player. It runs Windows VSTs using Linux
and Wine in a proprietary host program. We have patched Wine
(ftp://anonymous+museresearch.com@ftp.museresearch.com/rpms/1.8/wine-1.1....)
to make it work better with VSTs in our environment.

_______________________________________________
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
[LAD] Kontakt Spikes, Michael Ost, (Sat Oct 8, 1:26 am)
Re: [LAD] Kontakt Spikes, Clemens Ladisch, (Sat Oct 8, 11:03 am)
Re: [LAD] Kontakt Spikes, Michael Ost, (Mon Oct 10, 5:56 pm)
Re: [LAD] Kontakt Spikes, Clemens Ladisch, (Mon Oct 10, 10:05 pm)
Re: [LAD] Kontakt Spikes, Michael Ost, (Tue Oct 11, 12:42 am)
Re: [LAD] Kontakt Spikes, Clemens Ladisch, (Tue Oct 11, 8:24 am)
Re: [LAD] Kontakt Spikes, Clemens Ladisch, (Tue Oct 11, 8:52 am)
Re: [LAD] Kontakt Spikes, Michael Ost, (Tue Oct 11, 4:33 pm)
Re: [LAD] Kontakt Spikes, Paul Davis, (Tue Oct 11, 10:10 pm)
Re: [LAD] Kontakt Spikes, Michael Ost, (Tue Oct 25, 12:28 am)
Re: [LAD] Kontakt Spikes, Paul Davis, (Tue Oct 25, 12:36 am)
Re: [LAD] Kontakt Spikes, Michael Ost, (Tue Oct 11, 10:25 pm)
Re: [LAD] Kontakt Spikes, Robin Gareus, (Mon Oct 10, 7:05 pm)
Re: [LAD] Kontakt Spikes, Robin Gareus, (Mon Oct 10, 7:09 pm)
Re: [LAD] Kontakt Spikes, Nick Copeland, (Mon Oct 10, 8:37 pm)
Re: [LAD] Kontakt Spikes, Michael Ost, (Mon Oct 10, 8:48 pm)
Re: [LAD] Kontakt Spikes, Fons Adriaensen, (Mon Oct 10, 7:31 pm)
Re: [LAD] Kontakt Spikes, Robin Gareus, (Mon Oct 10, 7:37 pm)
Re: [LAD] Kontakt Spikes, Robin Gareus, (Sat Oct 8, 3:08 am)
Re: [LAD] Kontakt Spikes, Michael Ost, (Wed Oct 12, 11:28 pm)
Re: [LAD] Kontakt Spikes, Robin Gareus, (Thu Oct 13, 2:20 am)
Re: [LAD] Kontakt Spikes, Michael Ost, (Mon Oct 10, 6:03 pm)
Re: [LAD] Kontakt Spikes, Devin Anderson, (Sat Oct 8, 2:46 am)
Re: [LAD] Kontakt Spikes, Michael Ost, (Mon Oct 10, 7:38 pm)
Re: [LAD] Kontakt Spikes, Devin Anderson, (Mon Oct 10, 8:14 pm)