- Wednesday, June 22, 2016

release-team@conference.openafs.org

Wednesday, June 22, 2016< ^ >

Room Configuration

Room Occupants

GMT+0
[06:48:50] kadukoafs@gmail.com/barnowl4B0829E7 leaves the room
[06:48:59] kadukoafs@gmail.com/barnowl4B0829E7 joins the room
[12:15:37] meffie joins the room
[14:01:01] wiesand joins the room
[14:02:03] <wiesand> Hello
[14:02:09] <mvita> hi
[14:02:24] <kadukoafs@gmail.com/barnowl4B0829E7> greetings
[14:03:02] <meffie> hello
[14:03:45] <wiesand> So, what are the feelings about 1.6.18.1 as it is out there?
[14:04:28] <kadukoafs@gmail.com/barnowl4B0829E7> Better to ship it than to not, as I understand it.
I'm not sure who to send gifts to to get the cwd thing investigated,
though...
[14:05:16] <wiesand> Nor do I. But it’s not a regression in 1.6.18.1 .
[14:05:25] <mvita> maybe Joe is going to look at it
[14:05:26] <kadukoafs@gmail.com/barnowl4B0829E7> Right.
[14:05:40] <mvita> I don't know.
[14:05:54] <meffie> maybe more revalidate issue, i guess.
[14:06:04] <mvita> cwd issue was at 4.5?
[14:06:14] <mvita> sorry, can't remember the details
[14:06:15] <wiesand> 4.4 and 4.5 at least
[14:06:29] <meffie> andrew was the last to touch that stuff
[14:06:32] <wiesand> I wasn’t able to provoke it on EL7
[14:07:17] <kadukoafs@gmail.com/barnowl4B0829E7> It's at least plausible that the small cache size was relevant.
[14:07:33] <wiesand> Yes, but that’s not it.
[14:07:51] <wiesand> Upping the cache size well beyond the working set doesn’t help.
[14:08:13] <wiesand> Shrinking the cache on EL7 won’t make the problem appear either.
[14:08:53] <kadukoafs@gmail.com/barnowl4B0829E7> Oh, I had missed that part.
Anyway, I do have a single 100M packfile in my .git locally, which
could potentially trigger chunking issues, but it sounds like we
should be looking elsewhere.
[14:09:18] <wiesand> Probably
[14:10:17] <wiesand> I have no hope to get this fixed anytime soon.
[14:10:51] <wiesand> And it’s not as bad as the getcwd() failing due to the missing parent dir.
[14:11:00] <mvita> well if Joe can't get to it then I will
[14:11:32] <wiesand> Thank you. Should we delay 1.6.18.1 for it?
[14:12:22] <wiesand> Anyway, 1.6.18.1 would just buy a few days for Fedora users.
[14:12:22] <mvita> can we summarize the symptoms?    failed read of cwd, 1.6.18.1 and 4.4 or 4.5
[14:12:43] <wiesand> Right.
[14:13:15] <mvita> when did 4.4 support come in ?   that is, how far back can I go on openafs and still work on 4.4?
[14:13:29] <wiesand> 1.6.18
[14:14:07] <mvita> I can't imagine any diff between 1.6.18 and .1 that could cause this.
[14:14:29] <wiesand> There is none. I tested with 1.6.18 on the same kernels.
[14:14:43] <mvita> and you could NOT see it on RH7 - what was the kernel level there?
[14:15:06] <wiesand> 3.10 ;-)
[14:15:24] <wiesand> But of course plus tons of patches including backports.
[14:15:42] <wiesand> They haven’t backported that ERESTARTSYS thing though :)
[14:15:49] <mvita> since you can duplicate it already, could you take an fstrace?
[14:15:56] <mvita> have you ever run it before?
[14:16:08] <wiesand> Yes I have
[14:16:36] <mvita> that might be helpful to see that if you can provide one
[14:17:25] <mvita> I presume there were no relevant msgs in syslog...
[14:18:22] <wiesand> Nope
[14:19:36] <mvita> and the output from these commands would help as well:
[14:19:58] <mvita> cmdebug <> -long
[14:20:05] <mvita> cmdebug <> -cache
[14:20:51] <mvita> I know you did cd out and cd back as a workaround
[14:20:58] <wiesand> Right
[14:21:03] <mvita> did you ever try drop_caches?
[14:21:13] <wiesand> no
[14:21:40] <wiesand> I spent only a couple of hours on it...
[14:21:48] <mvita> if you can reproduce the problem again, see if this makes the problem go away:
[14:22:32] <wiesand> working on it... I haven’t optimized those 180 seconds wait time yet...
[14:22:34] <mvita> # echo 3 > /proc/sys/vm/drop_caches
[14:22:38] <wiesand> sure
[14:23:07] <mvita> and I'm getting ready to try 1.6.18 on 4.4 here
[14:23:23] <wiesand> by any chance, could it be due to the "shake harder" change?
[14:23:55] <mvita> not ruling anything out, but I agree w/ meffie, it smells like revalidate
[14:25:28] <kadukoafs@gmail.com/barnowl4B0829E7> Yeah.  "shake harder" is a pretty small change and I don't see an
obvious path, though of course I can't rule it out.  But this sort of
thing is usually revalidate.
[14:28:04] <wiesand> ok, just reproduced it on my shiny new F24 VM (4.5.7-300 kernel)
[14:28:53] <mvita> does drop_caches fix it?
[14:28:55] <wiesand> One interesting fact is that many things still work in this state.
[14:29:02] <wiesand> /bin/pwd for example
[14:29:47] <wiesand> no it doesn’t fix it
[14:30:10] <mvita> ah, you have to back out of the dir first
[14:30:26] <wiesand> well, that fixes it anyway ;-)
[14:30:32] <mvita> but of course just doing that ^
[14:31:43] <mvita> crufty old f23 vm here  4.4.2-301.fc23.x86_64
[14:31:49] <mvita> still building
[14:33:35] <wiesand> while you wait, look into https://www-zeuthen.desy.de/~wiesand/oaf/
[14:33:39] <mvita> k
[14:34:21] <mvita> what's the fid for the cwd that failed?
[14:34:31] <wiesand> er...
[14:35:20] <wiesand> refresh your browser
[14:35:40] <wiesand> I have no copy & paste from that VM to this client
[14:35:52] <mvita> mm, that _is_ refreshing
[14:36:25] <wiesand> seen anything interesting yet?
[14:36:57] <mvita> nothing fishy there
[14:37:09] <mvita> ** Cache entry @ 0x37726800 for 177.536901015.4387.788471 [ifh.de]
            2048 bytes  DV            1  refcnt     2
    callback 0535cf00    expires 1466619852
    0 opens    0 writers
    normal file
[14:37:55] <mvita>     states (0x1), stat'd
[14:38:31] <mvita> glad it's not a mountpoint
[14:39:05] <wiesand> no it isn’t - Ben had asked about it before
[14:39:14] <wiesand> I haven’t tried in a mountpoint
[14:39:26] <mvita> (I hate mountpoint internals)
[14:39:31] <wiesand> refresh again
[14:40:28] <wiesand> that’s 4 failed "git log"s
[14:40:41] <mvita> hm
[14:40:59] <mvita> doesn't show much, everything's cached
[14:41:20] <wiesand> fs flushall?
[14:41:30] <mvita> no, not the BIG hammer!
[14:42:05] <wiesand> fs flush .?
[14:42:19] <mvita> fs flush <cwd> should do it as long as it's not your cwd
[14:42:54] <wiesand> well if it’s not my cwd the problem is gone anyway
[14:43:21] <mvita> sure, but we want to capture what happens before you see the problem
[14:43:55] <mvita> that is, make it go away, then turn on fstrace and duplicate it
[14:44:05] <wiesand> you mean during those 180 seconds between git clone and git log?
[14:44:36] <mvita> yup, all of it
[14:47:02] <mvita> should we finish up release team agenda before going on w/ this?
[14:48:09] <wiesand> [sleeping]
[14:48:26] <wiesand> there wasn’t much more on the agenda...
[14:48:34] <wiesand> 1.6.19 is hardly urgent
[14:48:53] <wiesand> unless anyone is feverish about working on that now?
[14:49:33] <wiesand> And of course 1.8, since Ben is here today
[14:49:53] <mvita> yes, 1.8 would be good to get updated on
[14:50:24] <mvita> git clone is running
[14:50:26] <wiesand> looking at gerrit, not much has happened lately
[14:50:59] <meffie> yes, i have some reviews and updates to do.
[14:53:12] <kadukoafs@gmail.com/barnowl4B0829E7> give me a couple minutes...
[14:53:17] <mvita> git log
[14:53:21] <mvita> git checkout
[14:53:25] <mvita> git log
[14:53:27] <mvita> all okay
[14:53:33] <mvita> CND - cannot duplicate
[14:53:36] <wiesand> sleep 180 after git checkout
[14:53:58] <wiesand> it won’t happen without some sleep in between
[14:54:05] <wiesand> ans 120s seems not enough
[14:54:13] <mvita> oh, okay, doing that now
[14:54:34] <wiesand> meanwhile, you may want to look into https://www-zeuthen.desy.de/~wiesand/oaf/fstrace-full.txt
[14:55:28] <kadukoafs@gmail.com/barnowl4B0829E7> sorry, had to finish up my other meeting
[14:55:58] <wiesand> no problem, we’ve been productive on the cwd issue
[14:56:31] <wiesand> If you’d like to talk about 1.8, go ahead
[14:57:02] <mvita> duplicated
[14:57:08] <wiesand> :)
[14:57:26] <mvita> hoo boy, I'm happy now!
[14:57:46] <kadukoafs@gmail.com/barnowl4B0829E7> I've been not paying attention for a few weeks due to travel and
moving and such, so my starting point for getting back into 1.8 is
going to be the list I sent out at the beginning of May.
Andrew's Ubik fix got merged, at least...
[14:58:04] <meffie> yay, thanks.
[14:58:22] <wiesand> He said there’s no need to pull it up?
[14:58:40] <kadukoafs@gmail.com/barnowl4B0829E7> I believe that's correct, yes.
[14:58:49] <wiesand> Good
[14:59:54] <kadukoafs@gmail.com/barnowl4B0829E7> We all agree that 11793 (Revert lockless path through revalidate)
should go in, but in the current commit tree, it depends on 11790
(indirectly), the "create afs_StaleVCache" thing that needs revision.
I forget whether anyone looked at pulling the "revert lockless" out of
that chain of commits so it can be done separately.
[15:01:07] <wiesand> It needs a nontrivial rebase to be cherry-pickable on top of the current HEAD.
[15:01:22] <kadukoafs@gmail.com/barnowl4B0829E7> Ah.
[15:01:32] <mvita> yes, I think that's the one I took a hack at and then yielded to Thomas
[15:01:39] <mvita> it was tough
[15:02:06] <wiesand> Glad to hear that, I felt stupid ;-)
[15:02:14] <kadukoafs@gmail.com/barnowl4B0829E7> I believe it!
Do we have a sense for what timescale Thomas has to work on it?
[15:02:43] <mvita> well, he's kind of hit or miss at this point so I was thinking about taking it back
[15:02:50] <mvita> but I'm not looking forward to it.
[15:02:53] <kadukoafs@gmail.com/barnowl4B0829E7> 11349 (rxkad_crypt for inter-volser) got merged, yay.
[15:03:02] <kadukoafs@gmail.com/barnowl4B0829E7> *nods*
[15:03:18] <meffie> yay, thank you.
[15:03:33] <kadukoafs@gmail.com/barnowl4B0829E7> Marcio updated the pkgbuild/osx change (12239) so I'm supposed to
re-review it I guess.
[15:04:23] <mvita> I need to jimmy the clock on this vm so 180s doesn't take so long
[15:04:58] <kadukoafs@gmail.com/barnowl4B0829E7> haha
[15:05:24] <kadukoafs@gmail.com/barnowl4B0829E7> Hmm, looks like there's a bunch of things that are on the 1.8 list and
have some positive review but need a +2 :-/
[15:05:43] <mvita> did I say something funny?
[15:06:21] <kadukoafs@gmail.com/barnowl4B0829E7> Oh, just the implication that 180s can be more or less than 180s, I
guess.
[15:06:44] <mvita> ;-)
[15:06:55] <kadukoafs@gmail.com/barnowl4B0829E7> Anyway, meffie I think was going to update 11978 to not add
AFS_PAG_ONEGROUP_ENV in userspace linux builds [and also answer a
couple of questions]?
[15:07:42] <meffie> oh. that fell off my TODO list, sorry. will push it back on.
[15:08:21] <kadukoafs@gmail.com/barnowl4B0829E7> 11529 is just a doc change, but a previous version had some +1s before
it got non-trivially rebased, if anyone is bored.
[15:09:25] <kadukoafs@gmail.com/barnowl4B0829E7> The last time I reviewed 11858, the only comments I made were pretty
minor things, so that might also be an easy review.
[15:10:07] <mvita> meffie's TODO list:  http://i.stack.imgur.com/L1dct.jpg
[15:10:33] <meffie> heh
[15:10:38] <kadukoafs@gmail.com/barnowl4B0829E7> Wait, I thought that was mine ;)
[15:11:43] <kadukoafs@gmail.com/barnowl4B0829E7> I will also toss out 11629 as a possible target for review, though
that's more with my personal hat than my release engineer hat.
[15:13:59] <wiesand> 11529 lost one "s" too many in PS5 ;-)
[15:14:22] <kadukoafs@gmail.com/barnowl4B0829E7> sigh, thanks
[15:15:12] <kadukoafs@gmail.com/barnowl4B0829E7> Does anyone want to bring up new changes that hit gerrit in the past
1.5 months that we should consider for 1.8?
[15:16:11] <meffie> i dont think so.
[15:16:37] <wiesand> Didn’t Jeffrey say 12281 was important?
[15:17:08] <mvita> looking
[15:17:56] <kadukoafs@gmail.com/barnowl4B0829E7> (yes, he did)
[15:18:01] <mvita> hmm, don't recall
[15:18:36] <kadukoafs@gmail.com/barnowl4B0829E7> I am willing to put it on the 1.8 list given his input.
[15:18:38] <wiesand> And Mike’s 12240 was supposed to fix a real problem (with the default build settings)
[15:19:17] <kadukoafs@gmail.com/barnowl4B0829E7> That one is already on the list :)
[15:19:27] <meffie> thanks.
[15:20:49] <kadukoafs@gmail.com/barnowl4B0829E7> Anyone remember what the "rcode change" that Jeffrey mentioned in
12281 is?
[15:21:15] <mvita> (OT) stephan could you post ps -ef | grep afs for me?
[15:21:20] <mvita> I need some pids
[15:22:51] <meffie> perhaps 12283 is the "rcode change"
[15:24:45] <wiesand> seems plausible - I guess those handful of ubik related changes from Andrew that day kind of belongs together?
[15:24:57] <meffie> yes
[15:25:25] <kadukoafs@gmail.com/barnowl4B0829E7> Yes, that one seems to be it; I didn't go up quite that far in my
adjacent version number search :-/
[15:25:39] <wiesand> Mark: please refresh your browser
[15:26:59] <mvita> tx
[15:27:05] <mvita> hot on the trail
[15:29:46] <wiesand> So, delay 1.6.18.1 in hope for fix for the cwd issue?
[15:30:27] <wiesand> I have very little time left for openafs before next Wednesday
[15:30:28] <meffie> sounds good.
[15:30:34] <kadukoafs@gmail.com/barnowl4B0829E7> I am still inclined to leave that for 1.6.19, personally
[15:30:48] <kadukoafs@gmail.com/barnowl4B0829E7> But I could go either way
[15:31:40] <meffie> i suppose it depends on how invasive the fix is, if we find one.
[15:31:56] <wiesand> Sure.
[15:36:27] <wiesand> OK, let’s push out the door now. We’ll have a .2 for 4.6 rather soon anyway. Hopefully.
[15:37:15] <wiesand> Could you please tag 62c3882ba4c48c237b07c569dc59510a517081ad as openafs-stable-1_6_18_1 ?
[15:38:44] <wiesand> NB here’s the link to the talk on gravitational waves I mentioned last week: http://physikseminar.desy.de/sites2009/site_physikseminar/content/e408/e180951/e222842/e222914/Rowan_Zeuthen2016.pdf
[15:38:57] <kadukoafs@gmail.com/barnowl4B0829E7> ooh
[15:39:26] <meffie> i saw the link on hacker news, or some such.
[15:40:03] <wiesand> Of course it’s not the same w/o the narrative
[15:41:36] <wiesand> Ben: is that ooh on the talk or the tag request?
[15:42:07] <kadukoafs@gmail.com/barnowl4B0829E7> the talk
[15:42:24] <wiesand> ok
[15:45:42] <wiesand> thanks for the tag
[15:45:57] <kadukoafs@gmail.com/barnowl4B0829E7> np
[15:47:18] <wiesand> If anyone felt like looking at 12312...
[15:49:07] <meffie> looks good to me
[15:49:34] <wiesand> releasing volumes...
[15:51:32] <kadukoafs@gmail.com/barnowl4B0829E7> I don't know if the long line is acceptable in main.html.
[15:52:38] <wiesand> There are quite a few long lines in that file
[15:53:27] <wiesand> <p>In the interest of fostering a friendly, welcoming environment for contributors, we have introduced the Contributor Covenant as the code of conduct for the OpenAFS Project. A copy is included in the source tree, and can be found via the Contributor Code of Conduct link included on all pages. For additional information, visit <A HREF="http://contributor-covenant.org">contributor-covenant.org</A>
[15:53:39] <kadukoafs@gmail.com/barnowl4B0829E7> okay
[15:54:22] <wiesand> ok i’ll merge it now and bring it live...
[15:57:45] <wiesand> yuck, there was a bug...
[15:58:05] <kadukoafs@gmail.com/barnowl4B0829E7> in the website change?
[15:58:12] <wiesand> yes :-(
[15:58:40] <kadukoafs@gmail.com/barnowl4B0829E7> changes are cheap :)
[15:58:48] <wiesand> Don’t bother, I’ll fix it. Missing dot the link tot he distribution page.
[15:59:45] <mvita> stephan - have some clues, chasing them down, but I've gotta step away for now
[16:00:03] <mvita> how should I contact you when I have progress to report>
[16:00:04] <mvita> >
[16:00:07] <mvita> ?
[16:00:29] <wiesand> you have my email address?
[16:00:42] <mvita> yes, ok
[16:00:52] <mvita> sorry, gotta go.
[16:00:59] <wiesand> that’s the best way
[16:01:04] <wiesand> Thanks Mark. Bye.
[16:03:11] <wiesand> web fix is 12314
[16:04:29] <wiesand> I think the announcement mail doesn’t need to contain more than the release notes (+ boilerplate) so will be really short.
[16:04:50] <wiesand> Or do you think we have to mention the cwd issue?
[16:05:03] <kadukoafs@gmail.com/barnowl4B0829E7> the really short version seems fine to me
[16:07:23] <wiesand> ok, the fix is live...
[16:07:56] <kadukoafs@gmail.com/barnowl4B0829E7> thanks :)
[16:08:34] <wiesand> So what’s left to do is sending the announcement and pestering the moderators. Are you a moderator yet?
[16:08:53] <kadukoafs@gmail.com/barnowl4B0829E7> I believe I am not a moderator.
[16:15:54] <wiesand> This release brings client support for Linux mainline kernel 4.5 and
distribution kernels with backports from it. Servers, other platforms and
clients on Linux systems with older kernels can continue to use the 1.6.18
release.
[16:18:16] <wiesand> Sent
[16:19:31] <meffie> thank you
[16:20:50] <wiesand> moderators pestered
[16:21:03] <wiesand> Anything else to discuss today?
[16:21:25] <wiesand> I hope not ;-)
[16:22:04] <wiesand> Let’s adjourn.
[16:22:17] <wiesand> Thanks a lot for being here today!
[16:22:22] <wiesand> Bye
[16:22:23] wiesand leaves the room
[16:22:31] meffie leaves the room
[16:22:54] <kadukoafs@gmail.com/barnowl4B0829E7> Thank you, too!
[22:16:44] mvita leaves the room
[22:16:45] mvita joins the room
[22:31:27] mvita leaves the room
[23:53:20] mvita joins the room