[00:04:27] --- jakllsch has become available [00:04:27] --- jakllsch has left: Lost connection [00:09:29] --- jakllsch has become available [00:09:29] --- jakllsch has left: Lost connection [00:14:27] --- jakllsch has become available [00:14:28] --- jakllsch has left: Lost connection [00:19:29] --- jakllsch has become available [00:19:29] --- jakllsch has left: Lost connection [00:24:30] --- jakllsch has become available [00:24:30] --- jakllsch has left: Lost connection [00:29:30] --- jakllsch has become available [00:29:30] --- jakllsch has left: Lost connection [00:34:33] --- jakllsch has become available [00:34:33] --- jakllsch has left: Lost connection [00:39:35] --- jakllsch has become available [00:39:35] --- jakllsch has left: Lost connection [00:44:31] --- jakllsch has become available [00:44:32] --- jakllsch has left: Lost connection [00:49:28] --- jakllsch has become available [00:49:28] --- jakllsch has left: Lost connection [00:54:28] --- jakllsch has become available [00:54:28] --- jakllsch has left: Lost connection [00:59:28] --- jakllsch has become available [00:59:28] --- jakllsch has left: Lost connection [01:04:27] --- jakllsch has become available [01:04:27] --- jakllsch has left: Lost connection [01:09:26] --- jakllsch has become available [01:09:26] --- jakllsch has left: Lost connection [01:13:53] --- haba has become available [01:14:30] --- jakllsch has become available [01:14:30] --- jakllsch has left: Lost connection [01:19:30] --- jakllsch has become available [01:19:30] --- jakllsch has left: Lost connection [01:24:32] --- jakllsch has become available [01:24:32] --- jakllsch has left: Lost connection [01:29:30] --- jakllsch has become available [01:29:30] --- jakllsch has left: Lost connection [01:34:32] --- jakllsch has become available [01:34:32] --- jakllsch has left: Lost connection [01:39:33] --- jakllsch has become available [01:39:33] --- jakllsch has left: Lost connection [01:44:32] --- jakllsch has become available [01:44:32] --- jakllsch has left: Lost connection [01:49:32] --- jakllsch has become available [01:49:33] --- jakllsch has left: Lost connection [01:54:34] --- jakllsch has become available [01:54:34] --- jakllsch has left: Lost connection [01:59:38] --- jakllsch has become available [01:59:38] --- jakllsch has left: Lost connection [02:04:39] --- jakllsch has become available [02:04:40] --- jakllsch has left: Lost connection [02:09:39] --- jakllsch has become available [02:09:39] --- jakllsch has left: Lost connection [02:14:40] --- jakllsch has become available [02:14:40] --- jakllsch has left: Lost connection [02:19:45] --- jakllsch has become available [02:19:46] --- jakllsch has left: Lost connection [02:24:41] --- jakllsch has become available [02:24:41] --- jakllsch has left: Lost connection [02:29:37] --- jakllsch has become available [02:29:37] --- jakllsch has left: Lost connection [02:34:38] --- jakllsch has become available [02:34:38] --- jakllsch has left: Lost connection [02:39:40] --- jakllsch has become available [02:39:40] --- jakllsch has left: Lost connection [02:44:41] --- jakllsch has become available [02:44:41] --- jakllsch has left: Lost connection [02:49:46] --- jakllsch has become available [02:49:46] --- jakllsch has left: Lost connection [02:54:53] --- jakllsch has become available [02:54:53] --- jakllsch has left: Lost connection [02:59:48] --- jakllsch has become available [02:59:48] --- jakllsch has left: Lost connection [03:04:50] --- jakllsch has become available [03:04:50] --- jakllsch has left: Lost connection [03:09:50] --- jakllsch has become available [03:09:50] --- jakllsch has left: Lost connection [03:14:52] --- jakllsch has become available [03:14:52] --- jakllsch has left: Lost connection [03:19:49] --- jakllsch has become available [03:19:50] --- jakllsch has left: Lost connection [03:24:48] --- jakllsch has become available [03:24:48] --- jakllsch has left: Lost connection [03:29:50] --- jakllsch has become available [03:29:50] --- jakllsch has left: Lost connection [03:34:50] --- asedeno has left [03:34:53] --- jakllsch has become available [03:34:53] --- jakllsch has left: Lost connection [03:39:57] --- jakllsch has become available [03:39:58] --- jakllsch has left: Lost connection [03:44:43] --- asedeno has become available [03:44:54] --- jakllsch has become available [03:44:54] --- jakllsch has left: Lost connection [03:49:55] --- jakllsch has become available [03:49:56] --- jakllsch has left: Lost connection [03:54:56] --- jakllsch has become available [03:54:56] --- jakllsch has left: Lost connection [03:59:54] --- jakllsch has become available [03:59:55] --- jakllsch has left: Lost connection [04:04:54] --- jakllsch has become available [04:04:54] --- jakllsch has left: Lost connection [04:09:51] --- jakllsch has become available [04:09:51] --- jakllsch has left: Lost connection [04:14:49] --- jakllsch has become available [04:14:50] --- jakllsch has left: Lost connection [04:19:52] --- jakllsch has become available [04:19:53] --- jakllsch has left: Lost connection [04:24:51] --- jakllsch has become available [04:24:51] --- jakllsch has left: Lost connection [04:29:53] --- jakllsch has become available [04:29:53] --- jakllsch has left: Lost connection [04:34:51] --- jakllsch has become available [04:34:51] --- jakllsch has left: Lost connection [04:39:51] --- jakllsch has become available [04:39:51] --- jakllsch has left: Lost connection [04:44:54] --- jakllsch has become available [04:44:54] --- jakllsch has left: Lost connection [04:49:58] --- jakllsch has become available [04:49:58] --- jakllsch has left: Lost connection [04:55:01] --- jakllsch has become available [04:58:39] --- meffie has become available [05:53:01] --- haba has left [07:06:27] --- haba has become available [07:35:52] --- deason/gmail has become available [08:10:08] --- haba has left [08:10:08] --- haba has become available [08:27:36] --- haba has left [09:05:48] --- meffie has left [09:44:49] --- haba has become available [09:56:22] --- rra has become available [10:03:38] --- lars.malinowsky has become available [10:03:51] --- lars.malinowsky has left [10:36:26] --- jaltman/FrogsLeap has left: Disconnected [10:36:51] --- meffie has become available [10:39:44] --- meffie has left [10:40:06] --- jaltman/FrogsLeap has become available [11:25:27] --- andersk has become available [14:51:43] phalenor: that solaris panic from earlier was the same thing I saw on my own; I don't know what that panic stack trace shows priv_policy, as I'm not sure how we'd get there [14:51:43] --- jakllsch has left: Lost connection [14:52:16] sorry that took awhile; been a busy week [14:54:43] --- haba has left [14:55:53] --- jakllsch has become available [14:55:54] --- jakllsch has left: Lost connection [15:00:53] --- jakllsch has become available [15:00:54] --- jakllsch has left: Lost connection [15:05:53] --- jakllsch has become available [15:05:53] --- jakllsch has left: Lost connection [15:10:53] --- jakllsch has become available [15:10:53] --- jakllsch has left: Lost connection [15:12:51] --- rra has left: Disconnected [15:15:53] --- jakllsch has become available [15:15:53] --- jakllsch has left: Lost connection [15:20:53] --- jakllsch has become available [15:20:53] --- jakllsch has left: Lost connection [15:25:19] deason: no problem, doesn't really affect as as clients normally shouldn't be started without -dynroot and such [15:25:53] --- jakllsch has become available [15:25:53] --- jakllsch has left: Lost connection [15:30:53] --- jakllsch has become available [15:30:53] --- jakllsch has left: Lost connection [15:33:59] --- Russ has become available [15:35:53] --- jakllsch has become available [15:35:53] --- jakllsch has left: Lost connection [15:36:01] --- deason/gmail has left [15:40:54] --- jakllsch has become available [15:40:54] --- jakllsch has left: Lost connection [15:45:54] --- jakllsch has become available [15:45:54] --- jakllsch has left: Lost connection [15:50:54] --- jakllsch has become available [15:50:54] --- jakllsch has left: Lost connection [15:55:54] --- jakllsch has become available [15:55:54] --- jakllsch has left: Lost connection [16:00:54] --- jakllsch has become available [16:00:54] --- jakllsch has left: Lost connection [16:05:54] --- jakllsch has become available [16:05:54] --- jakllsch has left: Lost connection [16:10:54] --- jakllsch has become available [16:10:55] --- jakllsch has left: Lost connection [16:15:54] --- jakllsch has become available [16:15:55] --- jakllsch has left: Lost connection [16:20:54] --- jakllsch has become available [16:20:55] --- jakllsch has left: Lost connection [16:25:54] --- jakllsch has become available [16:25:55] --- jakllsch has left: Lost connection [16:30:54] --- jakllsch has become available [16:30:54] --- jakllsch has left: Lost connection [16:35:54] --- jakllsch has become available [16:35:54] --- jakllsch has left: Lost connection [16:40:59] --- jakllsch has become available [16:40:59] --- jakllsch has left: Lost connection [16:46:01] --- jakllsch has become available [16:46:01] --- jakllsch has left: Lost connection [16:51:04] --- jakllsch has become available [16:51:04] --- jakllsch has left: Lost connection [16:56:06] --- jakllsch has become available [16:56:06] --- jakllsch has left: Lost connection [17:01:06] --- jakllsch has become available [17:01:06] --- jakllsch has left: Lost connection [17:06:08] --- jakllsch has become available [17:06:09] --- jakllsch has left: Lost connection [17:11:08] --- jakllsch has become available [17:11:09] --- jakllsch has left: Lost connection [17:16:08] --- jakllsch has become available [17:16:09] --- jakllsch has left: Lost connection [17:21:08] --- jakllsch has become available [17:21:08] --- jakllsch has left: Lost connection [17:26:08] --- jakllsch has become available [17:26:08] --- jakllsch has left: Lost connection [17:31:08] --- jakllsch has become available [17:31:09] --- jakllsch has left: Lost connection [17:36:08] --- jakllsch has become available [17:36:08] --- jakllsch has left: Lost connection [17:41:08] --- jakllsch has become available [17:41:08] --- jakllsch has left: Lost connection [17:46:08] --- jakllsch has become available [17:46:08] --- jakllsch has left: Lost connection [17:51:08] --- jakllsch has become available [17:51:08] --- jakllsch has left: Lost connection [17:56:08] --- jakllsch has become available [17:56:08] --- jakllsch has left: Lost connection [17:56:50] --- Russ has left [17:56:52] --- Russ has become available [18:01:08] --- jakllsch has become available [18:01:08] --- jakllsch has left: Lost connection [18:06:08] --- jakllsch has become available [18:06:08] --- jakllsch has left: Lost connection [18:11:08] --- jakllsch has become available [18:11:08] --- jakllsch has left: Lost connection [18:16:08] --- jakllsch has become available [18:16:08] --- jakllsch has left: Lost connection [18:21:08] --- jakllsch has become available [18:21:08] --- jakllsch has left: Lost connection [18:26:08] --- jakllsch has become available [18:26:08] --- jakllsch has left: Lost connection [18:31:08] --- jakllsch has become available [18:31:08] --- jakllsch has left: Lost connection [18:36:08] --- jakllsch has become available [18:36:08] --- jakllsch has left: Lost connection [18:41:08] --- jakllsch has become available [18:41:08] --- jakllsch has left: Lost connection [18:46:08] --- jakllsch has become available [18:46:08] --- jakllsch has left: Lost connection [18:51:08] --- jakllsch has become available [18:51:08] --- jakllsch has left: Lost connection [18:56:08] --- jakllsch has become available [18:56:08] --- jakllsch has left: Lost connection [19:01:08] --- jakllsch has become available [19:01:08] --- jakllsch has left: Lost connection [19:06:08] --- jakllsch has become available [19:06:08] --- jakllsch has left: Lost connection [19:11:08] --- jakllsch has become available [19:11:08] --- jakllsch has left: Lost connection [19:16:08] --- jakllsch has become available [19:16:08] --- jakllsch has left: Lost connection [19:21:08] --- jakllsch has become available [19:21:08] --- jakllsch has left: Lost connection [19:26:08] --- jakllsch has become available [19:26:08] --- jakllsch has left: Lost connection [19:31:08] --- jakllsch has become available [19:31:08] --- jakllsch has left: Lost connection [19:36:08] --- jakllsch has become available [19:36:08] --- jakllsch has left: Lost connection [19:41:08] --- jakllsch has become available [19:41:08] --- jakllsch has left: Lost connection [19:46:08] --- jakllsch has become available [19:46:08] --- jakllsch has left: Lost connection [19:49:11] so what would cause every afs client in a cell to continue to try to talk to fileserers that have long since been removed and every volume removed and every address removed from the vldb? [19:49:51] Every client, or just a few? [19:50:13] so many arp packets my switches are seeing like 60% cpu usage [19:50:28] so, maybe not 'every', but 'many' [19:50:54] (vos listaddrs -c athena.mit.edu is ... very wrong, but I don't think we see lots of traffic from it.) [19:51:08] --- jakllsch has become available [19:51:08] --- jakllsch has left: Lost connection [19:51:10] and the increase in switch cpu usage corresponds exactly with the times these servers were shutdown [19:51:22] well, not sure what else to check, just soliciting for ideas [19:52:24] vos listvldb doesn't list any volume as living on these servers (changeaddr -remove won't allow removal if there are still volumes or repsites) [19:56:08] --- jakllsch has become available [19:56:08] --- jakllsch has left: Lost connection [20:00:05] once a client knows about a file server, it will continue to check the status of the server until the client is restarted [20:00:50] my clients seem to be checking each of these fileservers once every couple of seconds [20:01:08] --- jakllsch has become available [20:01:08] --- jakllsch has left: Lost connection [20:01:14] This does cause fun when a malicious user starts crawling /afs and your big webserver keeps trying to contact some obscure foreign cell. [20:02:19] yeah, we're the only entry in /afs [20:03:40] do you have a network trace of the traffic from one of the clients to a server no longer hosting afs? [20:04:31] I can [20:04:47] well, it's actually all arp traffic [20:05:10] Hmmmmmmmmm ....... [20:05:14] because the servers aren't even running anymore [20:06:08] --- jakllsch has become available [20:06:08] --- jakllsch has left: Lost connection [20:07:14] I used to get a bunch of log messages about rate-limiting arp mumbles to 200 packets/sec, on a couple of my machines. [20:08:20] I think I mentioned them here, and Derrick claimed it couldn't be us. But that must have been a year ago or so, by now. [20:08:58] well, as a test, I could try restarting one of these clients and see if the arps go away [20:11:08] --- jakllsch has become available [20:11:08] --- jakllsch has left: Lost connection [20:14:50] yep, arp requests for those decom'd fileservers went away with a reboot [20:16:08] --- jakllsch has become available [20:16:08] --- jakllsch has left: Lost connection [20:16:28] now, maybe this isn't the cause of my global network cpu usage increase, but it's still not a good thing. averaging 30 packets/sec it seems [20:21:08] --- jakllsch has become available [20:21:08] --- jakllsch has left: Lost connection [20:21:14] hmm, what if I brought up a dummy fileserver with the IPs for these old fileservers for a few hours? [20:21:42] or will that only fix the case of stale vldb info? [20:26:08] --- jakllsch has become available [20:26:08] --- jakllsch has left: Lost connection [20:30:14] there are two issues. first, if a client believes a volume is on a server and the server goes away before the volume location info expires will continue to try to find that volume on that server. it has no reason to go back to the vldb to find an alternate. The second issue is just the file server up/down status checks. as long as the client has an entry for the file server, it will continue to ping the server to see if its up or down. That does not go away until a client restart. [20:30:55] and it pings once every couple of seconds? [20:31:08] --- jakllsch has become available [20:31:08] --- jakllsch has left: Lost connection [20:31:18] down server probes are on the order of five minutes [20:32:10] the reason I want to see the network trace is so that I can tell you what type of traffic it is [20:32:26] it's arp traffic [20:33:37] i'm seeing almost 3 arp who-is requests from one client per second. (for 3 different down fileservers) [20:33:43] the arp traffic is not being sent by the afs client. the afs client is sending udp packets [20:36:08] --- jakllsch has become available [20:36:08] --- jakllsch has left: Lost connection [20:36:44] so the os is sending the arp traffic because the afs client is trying to talk to the server. [20:36:51] yes [20:41:08] --- jakllsch has become available [20:41:09] --- jakllsch has left: Lost connection [20:41:16] look at that, brought up some interface aliases with those IPs, and my network cpu usage dropped to almost nothing [20:41:55] > and it pings once every couple of seconds? the rx nat keepalive pings? [20:42:24] the os just seems really insistent on finding the mac address for these IPs [20:43:10] that is what I was thinking but it sounds like the OS is refusing to send the UDP packets until it can successfully receive a response to an arp who-has request and that request is being fired off quite repeatedly [20:45:20] solaris clients seem to be worse [20:46:08] --- jakllsch has become available [20:46:08] --- jakllsch has left: Lost connection [20:46:31] if it is the nat pings I suspect that the rx connections from the failed down server probes need to be destroyed after the failure is detected. [20:47:09] or better, nat pings can't start until a response has been received. [20:49:01] what does a nat ping look like in tcpdump? [20:49:41] an rx packet [20:51:08] --- jakllsch has become available [20:51:08] --- jakllsch has left: Lost connection [20:51:26] you want a packet capture for one of these dummy IPs ? [20:51:35] an rx version reply. [20:51:57] if you dump with sufficient snaplen and have sufficient verbosity, tcpdump will tell you that [20:52:39] looks like rx version [20:54:27] well, my immediate issue has been solved. thanks for the help [20:56:08] --- jakllsch has become available [20:56:08] --- jakllsch has left: Lost connection [21:01:08] --- jakllsch has become available [21:01:08] --- jakllsch has left: Lost connection [21:06:08] --- jakllsch has become available [21:06:08] --- jakllsch has left: Lost connection [21:11:08] --- jakllsch has become available [21:11:08] --- jakllsch has left: Lost connection [21:16:04] --- jakllsch has become available [21:16:05] --- jakllsch has left: Lost connection [21:21:04] --- jakllsch has become available [21:21:05] --- jakllsch has left: Lost connection [21:26:04] --- jakllsch has become available [21:26:05] --- jakllsch has left: Lost connection [21:31:04] --- jakllsch has become available [21:31:04] --- jakllsch has left: Lost connection [21:36:04] --- jakllsch has become available [21:36:04] --- jakllsch has left: Lost connection [21:41:04] --- jakllsch has become available [21:41:04] --- jakllsch has left: Lost connection [21:46:04] --- jakllsch has become available [21:46:04] --- jakllsch has left: Lost connection [21:51:04] --- jakllsch has become available [21:51:04] --- jakllsch has left: Lost connection [21:56:04] --- jakllsch has become available [21:56:04] --- jakllsch has left: Lost connection [22:01:04] --- jakllsch has become available [22:01:04] --- jakllsch has left: Lost connection [22:06:04] --- jakllsch has become available [22:06:04] --- jakllsch has left: Lost connection [22:11:04] --- jakllsch has become available [22:11:04] --- jakllsch has left: Lost connection [22:16:04] --- jakllsch has become available [22:16:04] --- jakllsch has left: Lost connection [22:21:08] --- jakllsch has become available [22:21:08] --- jakllsch has left: Lost connection [22:26:08] --- jakllsch has become available [22:26:08] --- jakllsch has left: Lost connection [22:31:08] --- jakllsch has become available [22:31:08] --- jakllsch has left: Lost connection [22:36:07] --- jakllsch has become available [22:36:08] --- jakllsch has left: Lost connection [22:41:07] --- jakllsch has become available [22:41:08] --- jakllsch has left: Lost connection [22:46:07] --- jakllsch has become available [22:46:08] --- jakllsch has left: Lost connection [22:51:07] --- jakllsch has become available [22:51:08] --- jakllsch has left: Lost connection [22:56:07] --- jakllsch has become available [22:56:07] --- jakllsch has left: Lost connection [23:01:07] --- jakllsch has become available [23:01:07] --- jakllsch has left: Lost connection [23:06:07] --- jakllsch has become available [23:06:07] --- jakllsch has left: Lost connection [23:11:07] --- jakllsch has become available [23:11:07] --- jakllsch has left: Lost connection [23:16:07] --- jakllsch has become available [23:16:07] --- jakllsch has left: Lost connection [23:21:07] --- jakllsch has become available [23:21:07] --- jakllsch has left: Lost connection [23:26:07] --- jakllsch has become available [23:26:07] --- jakllsch has left: Lost connection [23:31:09] --- jakllsch has become available [23:31:10] --- jakllsch has left: Lost connection [23:36:09] --- jakllsch has become available [23:36:10] --- jakllsch has left: Lost connection [23:41:09] --- jakllsch has become available [23:41:10] --- jakllsch has left: Lost connection [23:46:09] --- jakllsch has become available [23:46:10] --- jakllsch has left: Lost connection [23:51:09] --- jakllsch has become available [23:51:09] --- jakllsch has left: Lost connection [23:56:09] --- jakllsch has become available [23:56:09] --- jakllsch has left: Lost connection