I get easy reproducible a hang-up servers that use the file-based
swap file after swapoff / swapon procedure (in this case, some of the
data must be swapped). For example:
1) dd if=/dev/zero of=/usr/swp1 bs=1m count=100
2) mdconfig -a -t vnode -f /usr/swp1
3) swapon /dev/md0
4) begin to allocated memory, for example by simple:
5) after a filling of some percent, do swapoff /dev/md0, then swapon
/dev/md0. you can try this procedure again.
The system may stop responding to commands and freezes or locks up
after some time. From the outside - the core lives (icmp response
goes) but the disk system is not available.
PS: one of my server to my mind is frozen without swapoff/on - just
had three swapfile, a day after he crashed.