[00:13:22] --- kaj has left
[01:12:48] --- kaj has become available
[02:32:20] <Simon Wilkinson> kaduk: You might like to look at the XDR End the Madness patch in geritt -it should change things so we always use our own xdr.
[02:36:24] --- Russ has left: Disconnected
[02:40:29] --- haba has left
[03:12:42] --- haba has become available
[03:14:30] --- abo has become available
[03:47:16] --- jaltman has left: Replaced by new connection
[03:47:17] --- jaltman has become available
[06:11:49] --- jaltman has left: Disconnected
[06:11:57] --- jaltman has become available
[07:20:22] --- deason has become available
[07:55:48] --- reuteras has left
[08:45:24] --- meffie has become available
[08:56:54] <jaltman> Under what circumstances would a 1.4.10 file server issue VNOSERVICE to a client?
[08:57:09] <shadow@gmail.com/owl68DB543C> um. istr idledead timeout
[08:57:24] <shadow@gmail.com/owl68DB543C> there's exactly one condition. hang on and i will verify
[08:57:36] <shadow@gmail.com/owl68DB543C> yes. that';s it
[08:57:57] <jaltman> that is what I thought.  remind me when that is triggered?
[08:58:44] <shadow@gmail.com/owl68DB543C> in 1.4.10? it's buggy. in theory it triggers when the client does not
bother to reply to the server in a timely manner and just keepalives it
[08:59:07] <shadow@gmail.com/owl68DB543C> "i hear 1.4.10 is old and buggy"
[08:59:14] <jaltman> tell me about it
[09:00:32] <jaltman> this would be a client doesn't respond in a timely manner to a callback? and the file server then responds VNOSERVICE to the client's RPC?
[09:00:41] <Simon Wilkinson> "we don't do releases because we're bored" ?
[09:01:06] <jaltman> I know.  I'm trying to diagnose a problem in a client and the file server the user is stuck with is 1.4.10.
[09:01:13] <shadow@gmail.com/owl68DB543C> well, the client could also not be sending a storedata in a timely
manner, for instance
[09:01:58] <jaltman> and that would be the bug in processing in the file server
[09:02:04] <jaltman> ok.  thanks
[09:03:20] --- haba has left
[09:16:18] <deason> oops; commit messages are supposed to be wrapped at 70 chars? I've been using 72...
[09:17:27] <Simon Wilkinson> Yes, and the 'subject' line of the commit should be less than 65 characters.
[09:17:50] <Simon Wilkinson> Gerrit now warns you of these things :)
[09:18:13] <Simon Wilkinson> There's a bit of a religious war going on about it in their bug tracker.
[09:30:06] <deason> ah, vim's highlighting appears to suggest the first line should be less than 50; I assumed there was a reason for it, so I tried to follow it :)
[09:30:19] <deason> it doesn't seem to have a problem with longer other lines until I told it, though
[09:31:04] <Simon Wilkinson> mvim would autowrap mine at 70 if I let it, but didn't tend to scream about them.
[09:31:08] <kaduk@mit.edu/barnowl> nvi, however, does not, so I may have some longer ones floating around.
[09:31:52] <Simon Wilkinson> It doesn't really matter. Gerrit will whine on checkin, and anything really heinous we can catch at review time.
[09:32:21] <Simon Wilkinson> Derrick has pushed changes with subject lines in the hundreds of characters in the past, so the repository isn't exactly pristene already.
[09:33:08] <shadow@gmail.com/owl68DB543C> ha ha
[09:48:55] --- kaj has left
[09:59:34] --- RedBear has left
[09:59:55] --- RedBear has become available
[10:12:40] --- Russ has become available
[11:26:36] --- meffie has left
[14:57:14] --- Simon Wilkinson has left
[15:22:45] --- deason has left
[16:40:32] --- deason has become available
[18:34:50] --- summatusmentis has left
[18:34:53] --- shadow@gmail.com/owl68DB543C has left
[18:34:54] --- kula has left
[18:36:52] <kaduk@mit.edu/barnowl> Hm, I bet this backtrace is lies:
Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x10
#9  0xffffff8000a13ea0 in setpag () from /boot/modules/libafs.ko
#10 0xffffff80009d2387 in afsio_trim () from /boot/modules/libafs.ko
#11 0xffffff803cf67bf0 in ?? ()
#12 0xffffff80009d2405 in afsio_skip () from /boot/modules/libafs.ko
#13 0x0000000000000000 in ?? ()
#14 0xffffff80009d2493 in afsio_copy () from /boot/modules/libafs.ko
[18:45:19] --- shadow@gmail.com/owl06BCDF48 has become available
[19:16:59] --- kula has become available
[20:34:24] --- abo has left
[20:35:01] --- abo has become available
[20:37:32] --- Born Fool has become available
[21:04:18] <kaduk@mit.edu/barnowl> The XDR: Stop the madness patch does get rid of my xdr build errors,
though.
However, either it or something after 1.5.72 causes aklog to dump
core due to a malloc assertion ... or maybe I have disk corruption
because I've been panic'ing my laptop too much.
I guess I should go rebuild everything and see if that helps
(or if it dies in the middle ...)
[21:04:48] <shadow@gmail.com/owl06BCDF48> backtrae?
[21:04:55] <shadow@gmail.com/owl06BCDF48> (mark the xdr patch verified?)
[21:06:14] <kaduk@mit.edu/barnowl> I marked the xdr patch un-verified, since I needed to add #include
"limits.h" to sysincludes.h in order to get it to build.
[21:06:40] <shadow@gmail.com/owl06BCDF48> i assume you did or will comment to that effect
[21:07:17] <kaduk@mit.edu/barnowl> Er, which effect?
[21:07:44] <kaduk@mit.edu/barnowl> The aklog backtrace is useless:
kill
abort
__assert
malloc_useable_size
free
??
??
[repeats]
[21:11:22] <shadow@gmail.com/owl06BCDF48> that you needed the extra header
[21:11:31] <kaduk@mit.edu/barnowl> ktrace claims that aklog calls
#339(0x14,0,0x80185603,0x7fffffffb910,0)
(afs syscall)
which returns 0, then sigaction(SIGSYS,...)
and then it writes to stderr.
Huh.
[21:11:43] <shadow@gmail.com/owl06BCDF48> you ran aklog -d?
[21:11:52] <shadow@gmail.com/owl06BCDF48> (if not, try it?)
[21:12:02] <kaduk@mit.edu/barnowl> Yeah, I made that comment.
I don't have a good sense for whether it matters where in sysincludes.h
it should go, though, so I won't submit a patch.
[21:13:25] <kaduk@mit.edu/barnowl> Hm, gdb on aklog -d gives:
kill
abort
__assert
malloc_usable_size
malloc_usable_size
free
xdr_prlistentries
afsconf_GetKey
??
...
[21:13:58] <shadow@gmail.com/owl06BCDF48> interesting. 
[21:15:32] <kaduk@mit.edu/barnowl> I could believe that this is just me trashing my sockets routines,
though -- I have been crashing a lot with dirty filesystems.
[21:16:51] <kaduk@mit.edu/barnowl> However, I need to be at rehearsal in 9 hours, so I should go
sleep and not poke at this more tonight.
[21:17:12] <shadow@gmail.com/owl06BCDF48> ok
[21:18:52] <kaduk@mit.edu/barnowl> Bah, and something is still causing shutdown to silently hang.
[21:19:15] <shadow@gmail.com/owl06BCDF48> busy vnodes. i bet they have a positive refcount and vflush is hanging
[21:20:13] <kaduk@mit.edu/barnowl> Syncing disks said there were no vnodes left.
[21:20:54] <shadow@gmail.com/owl06BCDF48> could also be shutdown state. sleep. i can try to give you debug tips
tomorrow
[21:20:55] <kaduk@mit.edu/barnowl> Breaking to the debugger, the Giant (kernel lock) taskqueue was running,
several afsd's in afsslp, a couple of zombie afsd's.
[21:21:11] <shadow@gmail.com/owl06BCDF48> ok. get backtraces if you can
[21:21:30] <shadow@gmail.com/owl06BCDF48> i suspect then it's something in shutdown order is hanging and the rest
are waiting on it
[21:21:44] <kaduk@mit.edu/barnowl> There was also a ufs vnode lock locked.
I got impatient and rebooted (I still have to fsck, too)
[21:21:53] <shadow@gmail.com/owl06BCDF48> sleep. it can wait
[21:51:31] --- deason has left
[22:19:26] --- Born Fool has left
[22:47:38] --- kaj has become available
[23:14:31] --- reuteras has become available
[23:25:48] --- phalenor has left
[23:35:49] --- phalenor has become available
[23:35:58] --- Russ has left: Disconnected
[23:53:46] --- kaj has left