[01:43:31] --- sxw has become available [02:34:45] --- sxw has left [02:35:28] --- sxw has become available [02:39:21] --- sxw has left [02:51:54] --- sxw has become available [02:59:04] --- sxw has left [02:59:25] --- sxw has become available [03:32:30] --- sxw has left [03:32:52] --- sxw has become available [05:08:36] --- dwbotsch has left [05:09:13] --- kaj has left [05:10:47] --- dwbotsch has become available [05:30:48] --- sxw has left [06:06:32] --- jaltman has left: Disconnected [06:08:24] --- sxw has become available [06:25:32] --- sxw has left [07:00:52] --- reuteras has left [07:06:36] --- deason has become available [07:38:54] --- kaj has become available [07:56:18] --- Kevin Sumner has left [08:21:43] --- Kevin Sumner has become available [08:24:31] --- kaj has left [08:27:24] --- matt has become available [10:12:56] --- jaltman has become available [10:24:55] --- rra has become available [10:27:12] --- Kevin Sumner has left [10:27:13] --- mmeffie has left [10:28:11] --- Kevin Sumner has become available [10:30:02] --- mmeffie has become available [11:07:03] --- mmeffie has left [11:09:27] --- mmeffie has become available [11:20:38] --- deason has left [11:38:14] --- sxw has become available [11:41:14] --- deason has become available [12:04:20] Got some comments from "SM" on the afs3-standardisation draft. Does he just read every internet draft as it's submitted? [12:33:58] Yes. [12:34:05] Or at least damn close. [12:35:16] Ouch. [12:55:15] > Does he just read every internet draft as it's submitted? yes [12:55:45] Oh, so where all should I be sending this announcement? [13:22:10] which? [13:30:11] --- kaj has become available [13:53:15] --- jaltman has left: Disconnected [16:00:13] --- jaltman has become available [16:12:48] --- deason has left [16:22:10] Well, I was actually expecting other parts of the release discussion to garner more traffic, but I suppose having an extended discussion of whether DAFS should be enabled by default and what we're going to do about fast-restart is also a good use of the lists. :) [16:22:49] I'm actually thinking that shipping with both would solve a lot of the upgrade and downgrade issues that we've been kicking back and forth. [16:23:35] Yeah, it probably would. [16:23:59] Water under the bridge at this point, but it sure would have been nice if the original code had been written so that demand-attach behavior were a command-line option. [16:24:18] I really hate #ifdef. [16:24:21] Oh yes, very much so. I hate with a passion the number of passes we do over the same source files in the course of a build. [16:24:46] As much as possible, one should strive to actually compile all of one's source base every time one builds it. [16:24:52] Otherwise, you're just begging for stuff to rot. [16:27:15] I hate some of the extra passes, but others are convenient. I like flexibility. [16:27:42] It would certainly be highly desirable if dafs were a command-line option. [16:30:20] <_cclausen> except that it requires a BosConfig change [16:30:26] That's one of the reasons why I wanted to rework the configure probes for Kerberos so that aklog and friends are enabled by default if you have Kerberos libraries. [16:30:38] cclausen: So does adding the new command-line option. [16:30:50] <_cclausen> oh, that is true, good call [16:30:58] The idea would be that if you ran with the old BosConfig, you'd get the old behavior, and if you wanted the new behavior, you'd do bos remove / bos add with the flag and be happy. [16:33:05] Ah well, for the next feature. :) [16:33:06] Speaking of building all the configurations of the software, what's the status of the 'new build bot'? [16:33:50] It's in the "Simon has no time. Argggghhhhhhhhhh!" [16:33:52] bucket [16:33:55] <_cclausen> is there a way to have bnodes added and not enable some of them at start up [16:34:21] <_cclausen> so that fs versus dafs could both be there and only used when newer bosserver knew about it? [16:34:42] <_cclausen> I guess that would require 1.4 bosserver to know about dafs at least in the config file to ignore it [16:35:44] yes, you can create a disabled bnode. [16:35:46] Unfortunately, my TARDIS is in for a service right now. Otherwise, I could solve that, and the build bot, in one fell swoop. [16:36:07] heh [16:36:34] Seriously, I have some half written code to drive buildbot by screen scraping gerrit. [16:36:49] You run bos create and then bos stop on the new instance. [16:36:57] bos stop state is persistant. [16:37:07] Shawn then got all carried away and implemented a pretty functional command-line interface to gerrit over ssh. Which would be so much nice than screenscraping / XMLRPC. [16:37:19] Unfortunately, there doesn't seem to be a flag to bos create to let you create a new instance already stopped. [16:37:20] That's pretty cool. What occurred to me with rra's comment was that perhaps some passes that...we don't use...could be made conditional and punted down to the bot? [16:37:22] We should probably add that. [16:37:28] Then, a load of the Android folk starting looking at integrating Hudson with gerrit. [16:37:53] Long term, I prefer the latter option, because it means someone else is maintaining the code. [16:41:38] But yeah, my rough plan was that buildbot would do passes on the less-commonly called config options. Sadly, I doubt that we could test all possible permutations of the current option set, unless we got a significantly larger amount of processor time than we currently have available. [16:42:07] <_cclausen> I can talk to some folks at NCSA about time [16:42:24] <_cclausen> I think they said bsically anyone could get about 10,000 cpu hours fairly easily [16:42:51] Most of our current option set is safely independent, but the ones around the file server would be nice to test in all combinations. [16:43:03] Of course, this gets even more interesting once we have a real test suite and aren't just compiling things. [16:43:17] We currently have 134217728 different possible combinations. [16:43:36] <_cclausen> did you actually compute that number? Or just type in random stuff to make a point? [16:43:42] Yeah, but, say, --enable-cache-bypass has nothing to do with --enable-unix-sockets, which in turn has nothing to do with --enable-tivoli-tsm. [16:43:53] I computed that number. We have 27 configuration options. [16:44:04] And yes, Russ does have a valid point. [16:44:11] <_cclausen> well, SHOULD have nothing to do with and actually having nothing to do with are hard to prove [16:44:24] I can prove some of those with grep. [16:44:28] <_cclausen> (although I realize you know this from actually looking at the code) [16:44:33] They don't touch the same code at all. Right. [16:45:59] And many of them don't get enabled at all. icmp-pmtu-discovery was broken for about 12 months after it was merged, due to a mismerged hunk, and nobody had noticed. [16:46:00] Writing test suites is a pain in the ass, but they're so nice to have. [16:46:13] (Why do we even have --enable-tivoli-tsm rather than just building the code automatically if we can find the libraries?) [16:46:45] Last time I asked about icmp-pmtu-discovery, I got lots of dubious looks, so I've just been ignoring it. [16:46:46] <_cclausen> from my perspective, I'd like to have it fail if I enable something and it doesn't work [16:46:53] <_cclausen> not just silently ignore my option [16:47:19] disconnected could stop being optional if it wasn't for the earlier problem I mentioned. [16:47:19] The value in these --enable switches and #ifdef is clear enough. But I support the suggestion (by simon, marc, etc) to try to consolidate things over time. [16:47:37] The normal Autoconf convention is to have --without-foo suppress linking against foo even if it's found, --with-foo to force linking with foo and die if it can't be found, and the absence of the option to link with foo if found and ignore it if not. [16:47:58] I would like us to apply that uniformly, since it's what people expect and it means we build the code by default everywhere it's applicable with the fewest configuration options required. [16:48:11] <_cclausen> so --with-krb5 should fail if kerberos stuff isn't found? [16:48:20] yes, and does as of a few days ago on master. [16:48:24] I think it does, since Russ pulled it all out and rewrote it. [16:48:32] <_cclausen> ah, cool [16:49:40] * rra would also kind of like to collapse all the --enable-debug / --disable-optimize flags into just one option since I'm not sure why anyone would want to explicitly enable debugging for LWP and not the rest of the code base, but I'm probably missing some historical reason. [16:49:47] Er, into just two options. [16:49:52] I think that would be fair, too. [16:50:10] I'd like to get rid of --enable-warnings, and just turn on warnings if we're using gcc and it works with them enabled. [16:50:17] <_cclausen> I think I actually used that to debug a problem on AIX once [16:50:28] Yeah, agreed there. [16:50:34] <_cclausen> one of the files had to be compiled without optimization for it to work [16:50:39] What's --enable-redhat-buildsys for, anyway? [16:50:52] Braindead RHEL3 kernel build system, IIRC. [16:51:25] I'm pretty sure it could be done far less invasively, I just don't have a machine available to test against. [16:51:44] --- shadow has become available [16:52:10] We only have 13 flags that actually change what source we're compiling, and one of those (--enable-transarc-paths) is dubious. [16:52:36] The rest of the 27 are things like --disable-optimize or --enable-redhat-buildsys or --disable-pam that just turn stuff on and off in the build system or change compiler options. [16:52:38] what source changes with enable transatc paths? [16:52:56] That's why it's dubious. It changes what paths we compile in, but not in any really meaningful way. [16:53:16] We really only have 12 options that we need to cross-test. [16:53:26] The rest of those options really shouldn't be able to change testing results. [16:53:51] ok, that's fair [16:54:04] And --enable-namei-fileserver is only applicable on Solaris and, IIRC, IRIX. [16:54:13] dux [16:54:24] Oh, okay. And AIX? I heard rumors of AIX. [16:54:35] But anyway, not on Linux, Mac OS X, or *BSD. [16:54:53] aix is conceiveable but I think it ended with aix 4.3.3 effectively [16:54:58] <_cclausen> I thought linux only support namei? [16:55:03] yes [16:55:06] Right, so the option is pointless. [16:55:09] so no need to enable [16:55:14] required [16:55:14] There's no inode file server, so there's nothing to enable. [16:55:26] <_cclausen> so really there should be an --enable-inode option for the platforms that support it :-) [16:55:31] --- Kevin Sumner has left [16:55:35] I ran inode on Linux... 2.0.30 or so [16:55:37] Yes, except that inode is the default on platforms that support it. [16:55:40] So that's down to 1024 builds per test, then ... [16:55:41] <_cclausen> oh, I see [16:56:02] --- Kevin Sumner has become available [16:56:28] I wonder how long that would take on IRIX :) [16:56:47] The options are cache-bypass, supergroups, fast-restart, bitmap-later, demand-attach-fs, disconnected, unix-sockets, icmp-pmtu-discovery, pthreaded-ubik, and fuse-client. And namei-fileserver on platforms with inode support. [16:56:58] less long on not-my-irix-box [16:57:01] <_cclausen> I'd say that supergroups should be default [16:57:08] And maybe tivoli-tsm -- I'm not sure what that turns on. [16:57:15] cclausen: You only say that because you've not looked at the code. [16:57:19] --- mmeffie has left [16:57:21] Says the man who's never looked at the supergroups code [16:57:26] butc xbsa [16:57:37] <_cclausen> I supose not, but that is useful functionality [16:57:48] --- mmeffie has become available [16:57:49] Support for supergroups should definitely be enabled by default. [16:57:51] <_cclausen> I mean why would one NOT want groups within groups? [16:57:55] Oh, very useful. [16:57:56] problem with iphone: I can't type fast enough [16:57:56] I would kind of like to rewrite the code before that happens, though. [16:58:01] Sadly the code is unmaintainable. [16:58:23] I wish the Richard basch code still existed [16:58:32] It's probably the worst aliasing violations that we have left in the tree. [16:59:01] <_cclausen> so why aren't such things on the GSoC project list? [16:59:04] --- shadow has left [16:59:17] <_cclausen> (or is re-writing supergroups on that list?) [16:59:24] --- shadow has become available [16:59:44] doh. forgot to background [16:59:49] Anyway, of those options, cache-bypass, disconnected, and unix-sockets should just become the default and the flag should go away once they're stable. unix-sockets is already stable and we just have to decide what release should have the flag day. [17:00:01] demand-attach-fs, fast-restart, and bitmap-later we're having a big discussion about right now. [17:00:03] cclausen: If you feel in need of mental stimulation, take a look at http://git.openafs.org/?p=openafs.git;a=blob;f=src/ptserver/map.c;h=c58c258c9db976ce2a43412374957e5707d25f3d;hb=HEAD [17:00:11] supergroups we just talked about. [17:00:18] inode file servers should die, but gah. [17:00:28] pthreaded-ubik hopefully can be the default for 1.10. [17:00:45] pthreaded-ubik will almost certainly be required for rxgk. [17:00:50] icmp-pmtu-discovery, last I heard, was broken/useless as currently committed, but I think Derrick was looking at it. [17:01:10] I fixed it so it would build. I think Derrick has been doing a load of stuff with it. [17:01:13] fuse-client we should probably just build if FUSE happens to be around. [17:01:26] icmp pmtu discovery should be fine now [17:01:35] In that case, it should just become the default. [17:01:37] <_cclausen> I'm pretty sure I have a patch I'm supposed to test for some of that [17:01:42] In 1.10 if not for 1.6. [17:01:46] but it's not the big pmtu stuff. [17:02:05] chris, 1.5.75 or master [17:02:32] the icmp stuff is solaris/Linux. tree now has global, non icmp version [17:02:50] cache-bypass could be turned on by default and deliver some value, but it needs solution rx maxcall limit, which for all I know, also exists somewhere [17:02:52] using rx ack ping payloads and timeouts [17:03:02] cache bypass has a few issues with page reference counting that I should have a look at, but then we could turn it on. [17:03:13] Ah, and there's Matt... [17:03:42] I haven't looked in a while, but it would be nice to squash any issues of that nature [17:03:48] Matt: I think we need to refcount the pages that get sent to the backing thread into, and out of, the place where we stash them. Otherwise, I think the kernel can recycle them behind our back. [17:04:03] yikes, ok [17:04:04] I already have been building the Debian packages with --enable-disconnected. [17:04:13] and I macos [17:04:15] I can add --enable-cache-bypass when people think it's ready for that level of testing. [17:04:25] --- Kevin Sumner has left [17:04:29] we know what disconnected needs [17:04:36] Yeah. [17:04:40] --- Kevin Sumner has become available [17:04:53] Debian is nice because I can enable this stuff and toss it into Debian and it's only in unstable and testing, so there's usually about a year before it has to be mature enough to be really ready for stable users. [17:05:07] --- mmeffie has left [17:05:09] Which is why I don't' have a lot of qualms about just enabling stuff. [17:05:19] derrick: if there, is rx going to have more channels soon? [17:05:21] We should also look at LINUX_USE_FH [17:06:04] --- mmeffie has become available [17:06:07] matt: not in 1.6.0 [17:06:25] that's not surprising, but perhaps, rsn after that? [17:06:35] we need to standardize it [17:06:43] perhaps [17:06:47] see above [17:07:32] above, which? [17:07:58] standardize [17:08:05] right [17:08:28] We also need to, according to the DES-death roadmap, make kaserver and friends hide behind a config switch in 1.6 [17:08:34] I should write it up [17:08:43] not hard [17:08:55] klog kas kaserver [17:09:07] I thought we weren't doing that until 2.0. [17:09:23] shadow: tokens.krb and pagsh.krb as well. Also kpasswd. [17:09:29] And the PAM modules. [17:09:45] --enable-krustys-komedy-ks [17:09:58] If we're going to hide them behind a configure flag, I should say that on the lists, since I said we weren't doing that until 2.0. [17:10:35] Let me find the Elder's announcement. [17:10:42] --- Kevin Sumner has left [17:10:51] who's the elder? [17:10:52] --- Kevin Sumner has become available [17:11:15] p'e'd'a'n't [17:11:59] http://www.openafs.org/pages/no-more-des.html [17:12:15] "Before the 1.6 release, the build system will be modified to optionally enable OpenAFS without kaserver." [17:12:23] apostrophes are dangerous in the wrong hands [17:12:30] Oh, okay, one that lets you stop building it. [17:12:46] So as of 1.10 we'd stop building it by default. [17:12:53] That's slightly faster than what I said on the list. [17:13:01] "After the 1.6 release, the build system will be modified to build OpenAFS without kaserver unless it is specifically requested." [17:13:12] So sometime between 1.7, and the end of time. [17:17:17] perhaps it's dinner time [17:38:23] rra -- you've been building debian w/--enable-disconnected but not --enable-demand-attach-fs? wow.. [17:39:09] No, as I said on the list, I'm also enabling DAFS. [17:40:21] ah. I didn't see that in the README.Debian; my bad.. [17:40:32] and, of course, I see it -now-... [17:40:52] It's only the ones in experimental, of course. [17:40:55] any idea of how many people have installed the openafs-fileserver for debian? [17:41:06] The experimental version? Unfortunately, no. [17:41:14] Probably not very many. [17:41:52] given the lack of 'hey, why am I getting this error on Debian?' emails, I also suspect it's low. Either that or Debian unstable users always read the docs.. [17:42:05] Debian unstable users aren't using this package. [17:42:09] It's only in experimental. [17:42:22] unstable is still 1.4. [17:42:34] What error are you thinking of? [17:43:05] Oh, the error when you don't have a dafs node in BosConfig? [17:43:05] fileserver will barf if it's not configured as a dafs bnode [17:43:16] That's also in the NEWS file, so it's displayed to everyone by default during the upgrade. [17:43:28] Including detailed instructions on how to do the upgrade. [17:44:06] --- mmeffie has left [17:44:16] * jaltman Trying to catch up with few days of logs. [17:44:21] The no-more-des.html announcement was made before the IBM attorneys got involved over the trademark usage. At the present time, it would not be a good idea to be removing protocol functionality from OpenAFS that ships in IBM AFS unless we are prepared to give up use of the name "AFS" for any purpose. [17:44:39] --- mmeffie has become available [17:45:52] --- Kevin Sumner has left [17:46:02] --- Kevin Sumner has become available [17:47:37] how much effort would it be to break out the dafs vs non-dafs functionality into libraries and load the correct library based upon the existence of "dafs" vs "fs" instance? [17:50:01] --- matt has left [17:50:22] dlopen and friends require a lot of portability work if we want to maintain support for HP-UX, IRIX, AIX, etc. [17:50:23] >> Oh, so where all should I be sending this announcement? > which? About your document, and how we'd like to have comments so we can try to determine whether there's a consensus, and if so start afs3-stds spinup [17:50:54] You would be pretty much stuck introducing libltdl from libtool at that point or seriously reinventing the wheel, and both of those options are a lot of pain. [17:51:30] We could probably steal a bunch of stuff from Heimdal, but still. [17:54:31] I think, for the relatively limited set of platforms we support, reinventing a fairly thin abstraction layer over dlopen or whatever the platform supports is fairly reasonable. [17:56:39] Including AIX, HP-UX, and Mac OS X? IIRC, they have rather different ways of handling dynamic modules. [17:57:44] a question on that strategy: do we think that people will run non-DAFS in the long run? [17:58:00] ie, it seems to me that everyone eventually will be running DAFS. [17:58:01] with regard to removing protocol functionality... removing rxkad would be a bad idea, because then we wouldn't interoperate, and I suppose the IBM people could claim that what we are is no longer 'AFS', though frankly I think it's entirely reasonable for the community to take up maintenance of a protocol that IBM hasn't touched in 10 years, including removing obsolete security mechanisms. Removing the kaserver ought to be just fine; there's no reason we should have to implement every server component. Also, I can probably fairly easily find a version of AFS that doesn't have that component. And, I guess, the volserver and vlserver, if you really wanted to ditch those. [17:58:05] I hope not. [17:58:26] Removing the kaserver I think we're okay on. [17:58:30] Removing klog might be another matter. [17:59:20] > in the long run? the short run is pretty long. the long run is _very_ long. I wouldn't be surprised if some people were still not using DAFS in 3 years. [17:59:25] Removing anything *right* now, when we're still trying to get things sorted out with IBM, is probably not a good idea, but that's hopefully a very short-term situation. (And yes, we need to send out general e-mail about that, even though nothing's *happened* exactly, to give an update on what's in progress.) [17:59:54] jhutz - I agree wrt 3 years, but 4-5? [18:00:16] what I'm getting at is that introducing a dlopen() abstraction to solve a relatively short-termed problem is probably not the right way to go. [18:00:25] it's essentially a migration problem, not a strategic one. [18:00:29] --- Kevin Sumner has left [18:00:45] --- Kevin Sumner has become available [18:01:31] I don't know if everyone/most people think that 'DAFS-only' is strategic, though. [18:01:58] I'm actually not convinced that introducing a dlopen thing for this particular problem is an appropriate solution. But, I do someday want to see the fileserver have full support for interchangeable storage backends, and _those_ ought to be plugins. [18:02:10] Good point. [18:02:17] I'm -definitely- a fan of interchangeable back ends. [18:02:26] 'DAFS-only' is unquestionably strategic. Whether it is a strategy we want is another matter. [18:03:06] I think we don't have to decide right now, which is one of the other reasons why I'd like it to be optional for the 1.6 release. I really want to see how people react to it first. [18:03:18] If everyone loves it and there are cookies and puppies, then we can throw out the other code. [18:03:26] I don't think it's realistic to use libltdl without libtool. And libtool sucks, a lot. I think it sucks more than designing and maintaining an abstraction that will work for reasonable modern systems supporting dynamic loading. [18:03:41] * rra would really like to just use libtool. [18:03:44] I agree we don't have to decide now. [18:04:06] I bet libtool's license, or at least libltdl's, is incompatible with ours [18:04:09] My experience is that it saves a SHITLOAD of time for the software maintainers to just use the autotools stack. Yes, they have issues, but everyone knows how to work around their issues anyway since they're what to a first approximation everyone uses. [18:04:33] Ah, indeed. [18:04:36] We can't use libltdl. [18:04:40] Never mind that idea. :) [18:04:52] I still want to use libtool eventually to build our shared libraries, but we can have that argument later. [18:05:17] > just use the autotools stack. autoconf - definitely automake - possibly, and it certainly makes it harder to screw some things up libtool - I want my enemies to use this, but I wouldn't make my friends. [18:05:19] --- mmeffie has left [18:05:37] --- mmeffie has become available [18:05:38] I really haven't had any complaints about the software packages that I use libtool with. [18:05:47] I didn't even get a peep when I switched pam-krb5 over, which surprised me. [18:05:51] and the building our shared libraries problem is less hard then the problem libltdl solves. And libtool's solution brings as many problems as it solves. [18:06:10] * rra wrote the code we currently use to build shared libraries. [18:06:18] Believe me, libtool would be easier than maintaining it. [18:07:49] I have to maintain a private patched libtool, because every package that uses it has fucked up build problems because libtool insists on imposing its own ideas about how linking and shared libraries ought to work, which are generally more broken and less thought through than the OS's ideas. [18:07:57] Yes, but you already have it. [18:08:01] And can just drop it in everywhere. [18:08:07] That's the _only_ saving grace [18:08:13] Whereas if OpenAFS ever caused you some problem, you'd have to do a bunch of work to reproduce your fix with it. [18:08:14] --- jaltman has left: Replaced by new connection [18:08:15] --- jaltman has become available [18:08:23] And believe me, we'll cause problems for you once you want to use our shared libraries. [18:08:44] You don't care yet because you don't use our shared libraries yet. [18:08:46] This will change. [18:08:57] Because our current library situation is not something we want to maintain in the long run. [18:09:11] That, and the fact that I basically only care about Linux and Solaris. If I cared about HP-UX, I'd probably have to rewrite libtool from scratch to get it to produce binaries that actually worked without having embedded library paths pointing to AFS or nonexistent places. [18:09:12] Nor is installing two score 2MB binaries. [18:10:09] I don't use our shared libraries because we have yet to provide a coherent, meaningful set of shared libraries that are drop-in replacements for our static libraries. Randomly reshuffling things between library names is a bad ABI change. [18:10:17] Yup. [18:10:20] I know. [18:10:31] Produce shared versions of the libraries we've always had, and BE DONE. [18:10:54] That I seriously doubt we're going to do if for no other reason than I have no intention of ever producing or supporting a shared version of liblwp. [18:11:01] the reason I am concerned about supporting both "fs" and "dafs" in the same distribution is the impact on migration. The harder we make it to migrate and revert the change, the less likely it is that "dafs" will be tried by a large subset of the user base [18:11:52] yes, libprot.so will contain stuff that clients don't need, and that stuff will be present in any process that uses libprot.so, whereas it would be omitted from a statically-linked binary. WHO CARES? If it's not used, those pages don't have to be paged in. [18:12:47] I think I'd be fine with providing the existing LWP-based static libraries in lib/afs, and static and shared pthread-based libraries with the same names in, say, lib/afs-threaded. [18:12:52] jaltman: I like your suggestion of distributing both dafs and nondafs binaries. [18:13:08] --- mmeffie has left [18:13:12] --- Kevin Sumner has left [18:13:13] ie, that makes the switch (forwards and backwards) easier [18:13:43] --- mmeffie has become available [18:13:53] --- Kevin Sumner has become available [18:13:55] it is for the same reason that I have wanted multiple dynamic file server backends for years. orgs that have been running inode on Solaris have wanted to be able to run both inode and namei in parallel on the same machine. Both for transition purposes and for performance comparisons. [18:14:31] A distribution which has only DAFS and not "fs" is useless, since without "fs" you can't exert much control over the cache manager. :-) That said, I believe it is inappropriate to remove the non-demand-attach fileserver from the distribution in an attempt to force people to deploy DAFS. We are software developers; WE DO NOT SET SITE POLICY. [18:14:34] steven: I wasn't specifically recommending that we ship both sets of binaries. I would prefer a single binary that does the right thing. [18:14:57] jhutz: The problem is that right now we can't build both at the same time. [18:15:02] That's what prompted the current discussion. [18:15:04] Oh, nevermind; I misread jaltman's message [18:15:06] It's a big #ifdef switch. [18:15:20] Which is a kind of crappy design, but that's water long under the bridge. [18:16:05] I understand that the current implementation is a big #ifdef. My question that started this was whether we could extract the #ifdef code into a library so that we could build just the affected code twice. [18:16:46] jaltman: Right, I understand what you're saying and I think that's a great idea. I thought maybe jhutz had missed that. [18:16:53] The ifdef part. [18:16:55] another horrible option would be to clone the source files, remove the #ifdef by maintaining dafs and non-dafs sources and build both at the same time. [18:17:41] * jaltman will be back later. I have to finish moving out of my old apartment. [18:17:48] In the meantime, my fake PAM library continues to grow, allowing me to actually write a test suite for PAM modules without using the actual PAM library and dealing with its hard-coded configuration locations. But enough for one day -- time to go home and watch World Cup matches. [18:18:03] Yeah, I agree that a single binary is best. But failing that, I would point out that having a single fileserver that loads separate DAFS and non-DAFS libraries depending on the requested functionality would work pretty much exactly the same for users, but would have a higher chance of unpexected failures due to, for example, not being able to find the right library. Furthermore, doing it would require abstracting out all the bits controlled by the DAFS #ifdef, which would be at least as hard as modifying the current code to change behavior based on a runtime switch. [18:18:26] Yeah, that's a good point. [18:18:45] yes; I misread one of jaltman's messages such that it sounded like he was advocating ripping out non-DAFS to force people to move to DAFS. Sorry. [18:19:07] ok, how about build both and then have a parent process that selects which one to execute based upon the configuration? ( and then I'm really walking out the door) [18:19:11] > fake PAM library Oh, you have to tell me more about this. Testing PAM modules is _such_ a PITA [18:19:32] It's not very far along yet, but eventually I'll have something that basically implements all of the PAM API as used by a module. [18:19:54] --- Kevin Sumner has left [18:20:01] So you can link with it instead of libpam and include its header instead of the regular PAM headers and then there are various functions exposed that you can use to interrogate internal state, like stored data items and whatnot, when writing your test suite. [18:20:13] And you can feed in arbitrary configuration parameters and so forth. [18:20:16] > selects which one to execute I don't think that buys you anything over just putting the right program name into the bosserver config. dynamic libraries would theoretically at least buy you eliminating duplication in compiles and binaries. [18:20:16] --- Kevin Sumner has become available [18:20:36] --- mmeffie has left [18:20:37] yeah; that sounds like it would be really useful [18:21:09] I've been wanting it for years and finally started writing it. I expect it will be really useful in another month or two. I'm only doing enough to test pam-afs-session to start, but then my next target is pam-krb5. [18:21:48] --- mmeffie has become available [18:22:13] * rra also wrote a really terrifying but useful PAM argument parsing library that you can hand an array of option specifications and which will take care of reading them from either krb5.conf or from the PAM arguments, type-checking them, and blowing them into a configuration struct. [18:22:28] Soon it will have a test suite and then I'll be able to start using it. [18:23:06] It's exactly the sort of code that one should never write in C because C doesn't provide you with any of the facilities required to write it properly, but which I wrote in C anyway because it's for PAM modules and using something other than C isn't sane. [18:23:38] I always cringe whenever I write C code that uses offsetof. :) [18:23:40] Sometimes you need to just write those sorts of facilities in C, so you can then use them to write sane C programs. [18:23:44] Oh, you should. [18:23:47] Exactly. [18:24:18] offsetof is the sort of abomination that almost shouldn't exist, except that before it existed I kept running into cases where I needed it. :-) [18:24:42] /* * The user of this file should also define a macro of the following form: * * #define K(name) (#name), offsetof(struct config, name) * * replacing struct config with the name of the configuration struct. Then, * the definition of the necessary table for building the configuration will * look something like this: * * const struct option options[] = { * { K(aklog_homedir), true, BOOL (false) }, * { K(cells), true, LIST (NULL) }, * { K(debug), false, BOOL (false) }, * { K(minimum_uid), true, NUMBER (0) }, * { K(program), true, STRING (NULL) }, * }; * * which provides a nice, succinct syntax for creating the table. The options * MUST be in sorted order, since the options parsing code does a binary * search. */ [18:25:13] The stuff in parens is the default value. [18:25:36] The second parameter is whether to check krb5.conf for the option as well. [18:26:27] Anyway, really going home now. [18:26:32] --- rra has left: Disconnected [18:26:33] have fun [18:28:52] argh; no one ever answered my question, again. If I don't get answers, I'm posting to afs3-stds, openafs-devel, and arla-drinkers, and calling it a day [18:44:53] --- Russ has become available [18:58:40] Seems reasonable to me. [19:01:43] We do a force unmount in all cases on some platforms, right? [19:03:40] What's a force unmount? [19:04:06] Jeff: afs3-stds, openafs-devel, openafs-info, arla-drinkers, and we should check with howells to see if there is a kafs list. [19:04:12] or rather, what do you mean? [19:04:22] -info, really? [19:04:47] -info because non-developers have as much right to have a voice in standardization as developers [19:06:02] > What's a force unmount? "That's a good question" I'm considering passing FORCECLOSE to vflush() in all cases, even when MNT_FORCE is not set in the flags field. [19:06:26] We do not force-unmount on Linux. [19:06:32] If you have files open in AFS, the unmount fails. [19:06:37] I guess. I hate to post there in the name of getting wide coverage for an "important" topic. Normally, when people think their message is so important that it needs to be posted widely so that people will see it even if they chose not to subscribe to the forum where it is appropriate, I assert that they are wrong. But maybe this is really is an exception. [19:06:47] Unless you're shutting the system down, in which case we shoot ourselves in the head eventually. [19:07:33] jhutz: That was the feeling that I had on the discussion currently raging on -devel and -info, but people seem to have appreciated it in both places. [19:07:49] > passing FORCECLOSE to vflush() what effect does that have? Does it cause the "you have open files" check not to happen? If so, then no, you should respect (absence of) MNT_FORCE [19:12:49] So, let me describe what I'm seeing. (This is FBSD, so any bugs are possible.) I load the module and mount /afs kinit and aklog cd /afs/zone.mit.edu/user/kaduk ls cd /root umount /afs This proceeds to unmount /afs (at least mostly; it doesn't show up in `mount`, and I see a bunch of shutdown messages on the console), printing: afs: WARM shutting down of: CB... afs... BkG... CTrunc... AFSDB... RxEvent... UnmaskRxkSignals... RxListener... osi_StopListener: rxk_ListenerPid ffffffff80c89ac0 ALL allocated tables... done However, the actual umount process hangs in the 'mntref' state. After calling VFS_UNMOUNT, the dounmount() function (does some stuff and) calls vfs_mount_destroy, which sleeps on the mount point as long as mp->mnt_ref is nonzero. (more) [19:14:31] When we do a force unmount (i.e. pass FORCECLOSE to vflush), we force mnt_ref to 1 in our vfs_unmount routine, and when we drop our reference, dounmount can proceed. If we do not pass FORCECLOSE, vflush returns EBUSY, and (presumably) does not set mnt_ref to 1 for us, leading to the sleep. I presume this means that we're leaking refs on our root vnode somewhere, but I'm not entirely convinced I want to chase it down right now. [19:31:36] So ... it sounds like I should not always pass FORCECLOSE and mention in the README that people should umount -f if they want umount to finish. [19:39:42] yes; I'd say you should not always pass FORCECLOSE, and consider the fact that people have to umuont -f to be a bug, albeit a low-priority one [19:40:40] Okay. Hm, now I kind of want to check whether we still trounce over people who have files open when an unmount is attempted ... [19:43:44] we probably do [19:44:16] Yeah, I think so. [19:44:39] Oh, hi Derrick. I don't want to trouble you, but have you looked at gerrit/2213? [19:45:31] nope. and I just got home, so it will be a while [19:45:39] drove home today [19:47:33] Okay. [20:06:07] --- jaltman has left: Disconnected [20:06:15] --- jaltman has become available [20:26:26] > fake PAM library ... > pam-afs-session ... pam-krb5 Hm, so this means you can track down the "pagbug" that we're seeing intermittently on the athena dialups? [20:28:07] Maybe. [20:33:05] --- deason has become available [20:33:58] --- jaltman has left: Replaced by new connection [20:33:59] --- jaltman has become available [20:53:59] --- Jeffrey Altman has become available [20:55:34] --- asedeno has left [20:55:40] --- andersk has left: Lost connection [20:55:40] --- kaduk@mit.edu/barnowl has left: Lost connection [20:55:55] --- kaduk@mit.edu/barnowl has become available [20:55:58] --- asedeno has become available [20:55:58] --- andersk has become available [20:57:42] --- jaltman has left: Disconnected [20:57:50] --- jaltman has become available [20:58:11] catching up.... >so really there should be an --enable-inode option that's what --disable-namei-fileserver is, no? [20:58:43] >except that inode is the default on platforms that support it technically we "support" inode on modern solaris 10, but it's off by default because it broke sometime [20:59:58] >dafs vs non-dafs functionality into libraries >interchangeable storage backends well, something like namei/inode would be a ton easier than dafs/nodafs, and possibly a step in the right direction; I've wanted that, too [21:00:33] I'm not sure I get the suggestions of libraries loading dafs/nondafs code; how is that advantageous over just making it conditional? [21:01:08] bosserver could potentially give some option implicitly for dafs bnodes, or set an env var, for viced to be able to determine [21:02:08] >we should check with howells to see if there is a kafs list http://lists.infradead.org/mailman/listinfo/linux-afs [21:02:29] >We do a force unmount in all cases on some platforms, right? fwiw, no force-umount on solaris yet [21:08:27] --- asedeno has left [21:08:54] --- asedeno has become available [21:11:26] thanks for the pointer [21:11:36] --- asedeno has left [21:12:06] --- asedeno has become available [21:12:43] once upon a time, we were not very good about checking that there were no open inodes before allowing our unmount to succeed, on platforms where that was our responsibility. I think we've gotten better about that. [21:13:07] (I have a patch forthcoming to gerrit for fbsd) [21:13:56] Until relatively recently, Solaris didn't have the concept of a forced unmount. IIRC that didn't bother AFS, since we didn't do the check (see previous message), but it was occasionally annoying with regard to other filesystems, some of which had an annoying habit of leaking inode refs so you could never cleanly unmount. [21:14:25] The solution, of course, was funmount. [21:16:18] Oh, in fact, jhawk's web page reminds me of the problem -- it was AFS's direct-inode operations for vice partitions that caused the ref issues. [21:17:40] see /mit/jhawk/tmp/funmount [21:18:14] solaris 8 was the solaris to get it, iirc [21:20:13] --- asedeno has left [21:20:17] --- kaduk@mit.edu/barnowl has left: Lost connection [21:20:17] --- andersk has left: Lost connection [21:20:51] Solaris 7 does not claim to have -f Solaris 9 does. I don't have a sun4x_58 to look at [21:21:08] --- asedeno has become available [21:21:08] --- kaduk@mit.edu/barnowl has become available [21:21:08] --- andersk has become available [21:21:47] Speaking of jhawk, he was not sufficiently helpful to let me figure out how to map fid: 177.536994392.118.14574 to a file/directory. (536994392 is user.kaduk -c zone.mit.edu) [21:22:16] Er, user. [21:22:34] So I assume it should point to the kaduk directory therein. [21:23:38] 'fs getfid' could tell you if you are correct [21:23:57] ...unless you're talking about a once-existent file, and you're looking at a core or something [21:24:12] 177 is a cell number, according to one of 2 or 3 different nomenclatures depending on the context. particularly, the CM uses one representation on disk in which it tries to keep cell numbers persistent, and another for some in-core uses where it keeps the space compacted. 118 is a vnode number, and 14574 is the uniqifier. On a real fileserver, any given volume will only contain one vnode at a time with a given vnode number, so you can basically ignore the uniqifier for the purpose of figuring out what it is [21:24:15] (not a dir, though; the vnode is even) [21:25:07] I'm looking at a live system in ddb; there should not have been any recent filesystem activity in this area. [21:25:15] however, ==deason; an even vnode numbers is always something other than a directory; that is, it is a file, symlink, or mount point. [21:26:04] Well, I guess mount point, then? Since there is a user.kaduk volume. [21:26:22] if you go to another client, and just run 'fs getfid' on everything, you can find it [21:26:40] you could do it faster by traversing the parent vnodes up the tree possibly, but I don't know of a tool to do that [21:26:44] I tried to look, but apparently /afs/zone/user/kaduk is not readable to me [21:27:31] Hang on. [21:27:45] Try now? [21:28:55] that works now, but I realized it doesn't matter. the item of interest is almost certainly the 'kaduk' mount point in user.readonly (that being what volume 536994392 is), and 'fs getfid' doesn't help you on mount points [21:29:15] I suppose I should give context that I put panic() calls in above the "tokens for user ... discarded" messages, since they seem to be showing up spuriously. My test was while $(ls >/dev/null); do fs flushv .; done in /afs/zone.mit.edu/user/kaduk/ [21:30:54] Oh. So I just lose on mountpoints? (I do have server access.) [21:31:29] well, you could dump 'user' and run it through the dump scanner, if you have zone cell bits [21:32:29] That sounds like more work than is worth it for this case, though. [21:33:09] --- jaltman has left: Replaced by new connection [21:33:10] --- jaltman has become available [21:34:10] Quite possibly. What is the relevant error code? [21:34:13] yeah, sorry, forgot about that; 'getid' should really get an option to not resolve the final mountpoint [21:34:25] (from the message that would have been printed instead of the panic) [21:34:32] interestingly, I think the windows client actually has such an option, if you can find a windows client :) [21:34:56] Yes, it should have that option. Given that it's just a bit on the pioctl [21:35:06] (kgdb) p acode $1 = 19270410 [21:35:31] 15> error 19270410 19270410 RXK.10 RXKADSEALEDINCON sealed data inconsistent [21:35:44] I thought the bit on the pioctl was for symlink resolution... does that work for mountpoints? [21:36:01] Hm. [21:36:05] oh, hm; I forget. you might be right [21:36:06] I thought when I looked at this last, it looked like work because the windows client had an extra flag the unix cm doesn't [21:36:37] (for pioctls, or something in a nearby codepath or something) [21:38:35] Oh, yeah. You lose. The way lsmount/rmmount/flushmount work is by making the path argument be the directory _containing_ the mount point, and putting the mount point name in the input buffer [21:38:52] --- Born Fool has become available [21:41:56] --- asedeno has left [21:42:06] --- asedeno has become available [21:42:52] I don't suppose anyone has thoughts for how to investigate my token's corruption ... [21:45:59] --- asedeno has left [21:46:33] --- asedeno has become available [21:48:31] Hm, MIT jabber seems to be flapping a lot tonight. [21:55:00] it might be this room [21:55:45] I don't think so -- or at least if so it's just the connection between MIT and here. [21:55:58] I get disconnected from a room on conference.mit.edu, too. [21:56:24] Y'all can create a Jabber account directly on jabber.openafs.org if you ever want. [21:56:26] so, at the time the tokens-discarded message is printed, the user structure and tokens haven't yet been destroyed. [21:58:18] (I'm looking at the backtrace and noting some sketchy locking in GetVCache before we get here ... perhaps we're racing with ourself or something.) [21:58:35] --- asedeno has left [21:58:35] It would be interesting to know if the user structure still looks reasonable, and how much of it. Does aconn->user->stp look like a legitimate pointer, and if so, does it point at something that looks like a ticket? It might help to know if the error was generated locally or produced by the server (IMHO the easiest way to find that out is to have a packet capture where you can point at an abort packet containing the error [21:59:03] --- asedeno has become available [22:00:37] but that may not be helpful. given that it happens only occasionally, I'm going to guess that either something has stomped on the CM's copy of session key, or something caused a garbage packet to be sent to the fileserver. [22:01:14] RXKADSEALEDINCON doesn't _have_ to mean an encryption problem, but it usually does. [22:01:30] you could also try looking at the token data or some of the random user struct; see if anything looks like ascii but shouldn't [22:01:35] or is somehow otherwise recognizeable [22:02:22] (kgdb) p aconn->user->stp $2 = 0xffffff0111c46c00 sure looks like a valid pointer. [some binary stuff]ATHENA.MIT.EDU[more binary] [22:06:55] It is not a problem that aconn->user->tokenTime is less than aconn->user->ct->BeginTimeStamp, right? [22:06:58] So, RXKADSEALEDINCON generated on the client always does mean a checksum error. Specifically, either the weak 16-bit checksum in the packet header is wrong, or the 16-bit (seq ^ callNumber) in the first word of the (sealed) packet payload is wrong. [22:09:43] RXKADSEALEDINCON coming from the server can also indicate one of several problems in the response packet a client sent: - challenge checksum is wrong - epoch/cid or security index in response is wrong - encrypted call number is negative [22:10:11] So, I'm going to go with "you trampled on the key" [22:10:52] This would be easier to verify if you had some way to compare the key in the ClearToken with the one in your ticket file, or something. [22:13:56] > ticket file Pretend I am clueless and know nothing. My krb5 credential cache? [22:14:38] yes. [22:15:19] --- asedeno has left [22:15:24] --- kaduk@mit.edu/barnowl has left: Lost connection [22:15:24] --- andersk has left: Lost connection [22:15:26] --- Born Fool has left [22:15:49] --- kaduk@mit.edu/barnowl has become available [22:15:49] --- andersk has become available [22:15:49] --- asedeno has become available [22:15:54] The cleartoken key should be the same as the session key for your AFS ticket. The ticket stp points at will be either the AFS ticket itself, or some variation, depending on how your aklog and krb524d work. [22:16:03] well, your aklog and maybe your krb524d [22:17:56] The credential cache is on disk, and still valid ... [22:18:11] oh, right; disks are nice that way :-) [22:18:26] Is od -c good enough, then? [22:18:49] sure, if you can pick out where the key is supposed to be. I really don't remember the ccache format. [22:19:48] I prefer hexdump -C, personally, but that assumes you have hexdump [22:21:39] I think the od format is closer to what gdb gives me. [22:22:00] oh, probably. [22:24:14] So, what's on disk matches what's in stp. [22:24:52] fun [22:25:02] oh, but what about the key? [22:25:06] --- reuteras has become available [22:26:08] (still confused) [22:28:38] --- kaj has left [22:28:40] look at aconn->user->ct [22:29:17] stp is the Kerberos ticket ct is the data structure that contains your copy of things like the session key [22:30:55] --- kaj has become available [22:30:59] (kgdb) p aconn->user->ct $4 = {AuthHandle = 256, HandShakeKey = "\205z\200Jv", ViceId = 2036865635, BeginTimestamp = 1276833722, EndTimestamp = 1276910219} Hm, that didn't copy/paste very well, as my terminal rendered some of the binary as iso-8859-1 (?) [22:31:57] > It is not a problem that aconn->user->tokenTime is less than > aconn->user->ct->BeginTimeStamp, right? that is a little weird, but it could be that your clock is wrong. aconn->user->TokenTime is a locally-generated timestamp; the times in aconn->user->ct are taken from the passed-in credentials [22:33:17] that viceid looks bogus. [22:33:35] 4> pts ex 2036865635 -c zone.mit.edu libprot: no such entry (getting token) libprot: Could not get afs tokens, running unauthenticated Name: rcmd.freebuild, id: 2036865635, owner: system:administrators, creator: system:administrators, membership: 0, flags: S----, group quota: 20. [22:34:09] Yeah, I'm doing testing with the machine's principal. [22:34:17] ... but that alone is not the problem. unless, of course, it is just the wrong unixuser. oh; that is the user you expect? [22:34:31] Yeah. [22:35:13] so, "HandShakeKey" should be the Kerberos DES session key [22:37:43] Which should be in the krb5cc somewhere. [22:38:43] yes [22:39:49] How long? Just the 8 bytes printed? [22:40:15] Because those are present in the krb5cc file. [22:40:24] yes [22:47:19] --- asedeno has left [22:50:41] --- kaduk@mit.edu/barnowl has left: Lost connection [22:50:41] --- andersk has left: Lost connection [23:07:58] --- kaduk@mit.edu/barnowl has become available [23:07:59] --- andersk has become available [23:08:19] --- asedeno has become available [23:10:03] Well, I am kind of suspicious of the locking in GetVCache, as mentioned earlier. #elif defined(AFS_FBSD80_ENV) iheldthelock = VOP_ISLOCKED(vp); if (!iheldthelock) { /* nosleep/sleep lock order reversal */ int glocked = ISAFS_GLOCK(); if (glocked) AFS_GUNLOCK(); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); if (glocked) AFS_GLOCK(); } vinvalbuf(vp, V_SAVE, PINOD, 0); /* changed late in 8.0-CURRENT */ if (!iheldthelock) VOP_UNLOCK(vp, 0); [23:13:57] --- kaj has left [23:17:17] --- deason has left [23:22:12] --- asedeno has left [23:22:39] I don't see anything wrong there. Assuming, of course, that there's nothing going on around that which makes it unsafe to drop the glock [23:22:45] --- asedeno has become available [23:39:16] --- asedeno has left [23:39:41] --- asedeno has become available