[01:02:18] --- Simon Wilkinson has left [01:31:44] --- dev-zero@jabber.org has become available [01:31:58] --- dev-zero@jabber.org has left [01:32:13] --- dev-zero@jabber.org has become available [03:43:10] --- dev-zero@jabber.org has left [05:27:46] --- dev-zero@jabber.org has become available [05:28:00] --- dev-zero@jabber.org has left [05:28:13] --- dev-zero@jabber.org has become available [05:31:39] --- Simon Wilkinson has become available [05:45:43] --- Simon Wilkinson has left [06:02:40] --- dev-zero@jabber.org has left [06:19:21] --- jaltman has left: Disconnected [06:41:58] --- jaltman has become available [06:57:12] --- dev-zero@jabber.org has become available [06:57:28] --- dev-zero@jabber.org has left [06:57:45] --- dev-zero@jabber.org has become available [07:13:56] --- jaltman has left: Disconnected [08:43:02] --- dev-zero@jabber.org has left [09:41:42] --- deason has become available [11:12:53] --- dev-zero@jabber.org has become available [12:11:15] --- jaltman has become available [13:37:58] --- jaltman has left: Disconnected [14:25:05] --- Simon Wilkinson has become available [14:31:43] We have to sort RT. Having to cut and paste responses from contributors because they aren't allowed to reply to requestors is just fscking ridiculous, [14:33:41] --- mdionne has become available [14:34:34] could also just be my own ineptitude.. :) [14:35:14] Don't think so. [14:35:25] I assume you've seen the patch in RT [14:35:29] I suspect that you've been given "Comment" but not "Reply to requestor" permissions [14:35:33] in gerrit? [14:35:36] yeah [14:35:37] Just about to look now. [14:35:52] (The whole RT permissions thing is, IMHO, nuts) [14:35:53] looks like we just need to gunlock before filldir [14:36:19] That sounds completely right - we can't hold the GLOCK when we call into the kernel, because the kernel's free to do whatever it likes, including re-entering the filesystem. [14:36:20] on the master branch we do it - was added with the nfs translator patch [14:36:47] so we page fault with memory mmaped into AFS while doing filldir [14:38:23] emacs filldirs into a memory mapped page? [14:38:51] That'd certainly do it ... [14:38:56] I guess it does - the trace looks very similar to my test case [14:39:20] and the differences are probably more about different compilers with different inlining... [14:39:45] I'll ask John to test if he can - an old colleague of mine BTW [14:41:12] I've made you the owner of that ticket. That might let you actually reply. There again, it might not. [14:41:48] We'll find out I guess :) [14:42:14] Yup. Patch looks good to me. I should probably look at including it in the RPMs. [14:42:40] I have the "reply" link now, so I should be OK [14:42:42] I wonder if John is just being unlucky, or if RedHat have changed something in the 5.4 kernels. [14:43:51] I'm thinking that it might be a kernel change. I still haven't tried a recent kernel with the same test. [14:44:40] There was something in a bug report I was reading about changes to the page management stuff that they'd done to avoid a deadlock in NFS. [14:45:14] As part of that, I think they stuck in write_begin and write_end - but as you note, in a different place, because they're committed to ABI compatibility. [14:45:59] If you fancy, John also posted to openafs info about his problem - it's probably worth a reply there too, saying that you believe that you've tracked it down. [14:50:12] --- dev-zero@jabber.org has left [14:50:16] --- dev-zero@jabber.org has become available [14:51:23] --- dev-zero@jabber.org has left: Replaced by new connection [14:51:24] --- dev-zero@jabber.org has become available [14:51:43] --- dev-zero@jabber.org has left [14:51:56] --- dev-zero@jabber.org has become available [14:52:58] Marc: I neglected to say - thanks for looking into that bug in the first place! [14:55:18] No problem - had some time and it looked like an interesting one :) [14:57:41] there's also some interesting oopses reported on ubuntu - ex: http://launchpadlibrarian.net/34907536/OopsText.txt ring a bell? [15:10:25] Yes. That's the security issue we solved in 1.4.10 [15:11:22] (The signature for that is attempts to dereference kernel pointers that are in the first page - in that case 0034, but you'll sometimes see other, similar, values) [15:15:07] (If they're seeing that in 1.4.11, that's a bad, bad thing, as it means that we've missed somewhere and are leaking error codes into the kernel still) [15:20:57] there's a couple of bugs with the same oops (and same value 0034). I would assume that it is 1.4.11 given that it's a recent kernel in the reports [15:23:12] Gah. If it is, that's bad. [15:24:57] so 0034 might be an error code that we return where a pointer is expected? [15:25:20] Well, it's an error code, plus an offset. [15:25:39] So if the lookup is 4 for a structure element 4 bytes in, then the error code is 0x30, and so on. [15:25:56] ok, got it. [15:26:14] When I tracked this down last time, I used objdump to work out what clear_inode+0x73 actually was on that particular kernel [15:26:34] And then from there, I could figure out what structure we were looking at, and track that back up the call chain. [15:27:53] This one looks like it might be simpler, though. We're oopsing, rather than panicking, as we're still in the modules scope. Which may mean that it isn't the return code issue. [15:30:26] the first thing that came to mind was that truncate was changed quite a bit in 2.6.31 [15:31:53] I don't entirely believe that stack trace... [15:38:25] interesting that the 2 bugs have exactly the same stack trace, modulo the addresses [15:39:17] Interesting indeed. [15:39:50] It's got to be something that we're doing wrong. And the address we're trying to dereference does look a huge amount like the 'error code as pointer' problem. [15:40:01] Do you have a 2.6.31 kernel you can objdump ? [15:40:14] Or better still, just run gcc against? [15:44:12] I have a VM with ubuntu karmic -probably the same kernel [15:44:42] Looking at clear_inode in lxr, very little of the stuff it calls is inlined, which narrows the playing field a little. [15:45:54] Yeah I have that same kernel in the VM - running now [15:50:51] Current guess would be that we have an inode with a NULL superblock, but that's a shot in the dark. [15:55:34] hmm, clear_inode+0x73 is between instructions in my case. [15:56:01] Yuck. [15:56:43] Does decoding the instruction list in the ooops help at all? [15:56:50] but it looks like it's at or near vfs_dq_drop. The line after uses i_sb [15:57:55] Can you fire up gdb against that kernel? [15:59:03] Add find out what ((struct super_block *)NULL)->s_op is? [15:59:16] If it's 0x34, then it's likely that that's where the problem lies. [16:00:20] silly me - the traces are 32-bit, i'm compiling on 64-bit [16:00:30] Ah. Yes, that would break things. [16:00:48] Yeh - if you've got a 64bit kernel, then any gdb output is going to be out, too. [16:02:48] I think I can do a make in 32-bit mode - hold on [16:04:00] Even if you can just get the objects out of the Debain repo, that would be enough. [16:05:18] I wonder if we're hitting the race that iprune_sem is supposed to protect against ... [16:05:44] Got it compiled - I see the same code now. looks like the line after vfs_dq_drop [16:06:13] Okay, so that suggests that we're hitting an inode with a NULL super block. Can you do the gdb thing I suggested to verify? [16:06:55] Interestingly enough, acb0c854fa9483fa85e377b9f342352ea814a580 in the Linux kernel tree removed the check that the superblock is set from that line. [16:08:43] that gdb needs to be against the whole uncompressed kernel ? [16:10:18] Actually, easier way. [16:10:53] Build something that just works out the offset of s_up in struct super_block. [16:11:02] But I strongly suspect that it's 34. [16:12:43] --- dev-zero@jabber.org has left [16:15:46] Okay. Here's my theory, then I must go sleep. Staying with friends with a 1 year old, so not getting much of it at the moment! [16:17:25] I think we're calling iput() from afs_FlushVCache (in turn from afs_ShakeLooseVCaches), with either a corrupt inode, or one which isn't fully initialised. That would explain why we go boom in clear_inode. [16:18:41] Hey I've had a few 1-year olds of my own - I know how that feels :) [16:19:57] Ok I'll see if I see something along those lines [16:20:43] I'd look for locking failures - is it possible for us to be flushing the same vcache twice on two different code paths, or for us to be flushing one that another code path is busy initialising. [16:28:37] I think I've found it ... [16:29:44] Or maybe not, actually. Anyway, we get the reference count for it being on the VLRUQ after we've already threaded it onto that queue. Which is just plain anti-social. [16:29:53] And with that, good night! [16:56:29] --- jhutz@jis.mit.edu/owl has left: Disconnected [19:26:59] --- mdionne has left [20:44:10] --- dev-zero@jabber.org has become available [21:35:05] --- dev-zero@jabber.org has left [21:39:37] --- deason has left [21:58:17] --- reuteras has become available [22:22:32] --- dev-zero@jabber.org has become available