On Jul 16, 2009, at 5:48 AM, Attilio Rao wrote:
> 2009/7/16 Attilio Rao <email@example.com>:
>> 2009/7/15 Nick Esborn <firstname.lastname@example.org>:
>>> The following reply was made to PR threads/136345; it has been
>>> noted by GNATS.
>>> From: Nick Esborn <email@example.com>
>>> To: bug-followup@FreeBSD.org,
>>> Subject: Re: threads/136345: Recursive read rwlocks in thread A
>>> cause deadlock with write lock in thread B
>>> Date: Wed, 15 Jul 2009 14:32:38 -0700
>>> This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
>>> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
>>> Content-Transfer-Encoding: 7bit
>>> Even after the above patch, I still run into occasional MySQL thread
>>> deadlocks, which I originally described in what is now threads/
>>> I also posted on freebsd-current a few days ago:
>>> I'd be happy to collect whatever data would be helpful in tracking
>>> down this deadlock. This only seems to happen under our production
>>> workload, so that might make it harder to capture meaningful debug
>>> data, but I'm certainly willing to try. I can also arrange for
>>> developer access to the system in question, if that would help
>> So did you backport this to 7 and still experience deadlocks?
>> I just committed the fix to HEAD not to STABLE branch.
> Ok, I got, you just upgraded.
> Can you try the following things?:
> - Upgrade to the -CURRENT of today
> - Recompile the kernel with the following options:
> KDB, DDB, SCHED_ULE, PREEMPTION, FULL_PREEMPTION, INVARIANT_SUPPORT,
> INVARIANTS, WITNESS
> - When the deadlock takes place break into DDB and please retrieve the
> following info:
> db> show allpcpu
> db> ps
> db> alltrace
> db> show alllock
> - Save them with a serial console output or using the textdump(4)
> (if necessary read the ddb(4) and textdump(4) before to set up the
> whole system).
> This would shade a light if the problem lives within the kernel or
> Peace can only be achieved by understanding - A. Einstein
I can definitely do the upgrade.
KDB, DDB, SCHED_ULE, and PREEMPTION are already turned on. I will try
FULL_PREEMPTION, INVARIANT_SUPPORT, INVARIANTS, and WITNESS, but when
I first upgraded to 8.0, this server was unable to handle its workload
with the INVARIANTS and WITNESS options turned on.
Also, it can take a while for it to become clear that the deadlock has
occurred -- usually our monitoring picks it up when replication falls
behind. So it may be 15-20 minutes after the deadlock that I am able
to run the above db commands. Of course the thread will still be
deadlocked. Hopefully that doesn't reduce the value of the data
firstname.lastname@example.org - all messages cryptographically signed