- Wednesday, November 15, 2017

release-team@conference.openafs.org

Wednesday, November 15, 2017< ^ >

Room Configuration

Room Occupants

GMT+0
[02:22:59] mvita leaves the room
[03:35:34] mvita joins the room
[03:38:35] mvita leaves the room
[09:54:05] kadukoafs@gmail.com/barnowl0AF0C398 leaves the room
[09:54:43] kadukoafs@gmail.com/barnowl22660C09 joins the room
[09:59:55] kadukoafs@gmail.com/barnowl22660C09 leaves the room
[10:01:03] kadukoafs@gmail.com/barnowl60BC7D7E joins the room
[10:01:28] <kadukoafs@gmail.com/barnowl60BC7D7E> I'm in Singapore this week for the IETF, so with the time zone I have
little chance of making the meeting.
[10:16:00] kadukoafs@gmail.com/barnowl60BC7D7E leaves the room
[10:17:18] kadukoafs@gmail.com/barnowlEBF84E62 joins the room
[14:02:11] meffie joins the room
[14:29:23] mvita joins the room
[15:55:54] <mvita> stepping away for a bit, I may be late for the mtg
[16:01:16] wiesand joins the room
[16:01:59] <wiesand> Hello. Is anyone actually here?
[16:02:01] <meffie> good afternoon wiesand
[16:02:16] <meffie> ben is out and mark is late.
[16:02:19] <wiesand> I'll take this as a yes. Hello Mike
[16:03:04] <meffie> Hello Stephan
[16:03:20] <wiesand> ok, I worked a bit on getcwd/el7.4 kernels… let me report my findings
[16:03:43] <meffie> ok, great. mark is working on that too.
[16:03:48] <kadukoafs@gmail.com/barnowlEBF84E62> I am actually here, against my good health.
[16:03:57] <kadukoafs@gmail.com/barnowlEBF84E62> But hopefully I will not be here much longer.
[16:04:07] <meffie> ben! ok, lets make it quick
[16:04:19] <mvita> I'm back
[16:04:43] <wiesand> reproducer: run the client with -stat 2000, then clone the openafs git repo, sleep 180, run something that calls getcwd() like "git log"; it's a subdirectory, not the volume root
[16:05:06] <wiesand> Hi Ben & Mark, glad you're here
[16:05:34] <wiesand> So, with this setup I had a 100% reproducible getcwd() failure
[16:05:57] <mvita> okay, thank you Stephan
[16:06:07] <wiesand> The I tried with a module compiled with shake-harder reverted
[16:06:42] <wiesand> Epic failure. Already the git clone failed.
[16:07:21] <mvita> "Epic failure" - getcwd ENOENT?
[16:07:39] <wiesand> And an "rm -rf openafs" afterwards needed two cycles, the first one failing with "directory not emtpy"
[16:07:58] <kadukoafs@gmail.com/barnowlEBF84E62> That sounds highly exciting.
And this is on 1.6?
[16:08:12] <wiesand> Mark: no, it failed already before.
[16:08:37] <wiesand> Turned out I was still running with "-disable-dynamic-vcaches".
[16:08:46] <mvita> ah
[16:08:51] <wiesand> Lesson learned: don't do that with a small stat cache
[16:10:05] <wiesand> So I removed that setting. And since then, with -stat still in effect, I managed to run 3 iterations of the test (with shake-harder reverted). Success 3/3 times. No getcwd issue.
[16:10:29] <meffie> "yay"
[16:11:01] <wiesand> Well, the last time we reverted shake harder your comment was "this make me sad" IIRC.
[16:11:21] <meffie> hence the quotes
[16:11:38] <wiesand> Giti it ;-)
[16:11:45] <wiesand> Er, got it
[16:11:59] <wiesand> Mark, any new findings from your side?
[16:12:54] <wiesand> (and I'm not aware of any recent report of getcwd issues with a client version w/o shake-harder)
[16:13:02] <mvita> same as you - but I accidentally discovered a subset of shakeloose harder that makes it happen
[16:14:03] <wiesand> what's the suset?
[16:14:06] <mvita> I can trigger it w/ 1.6.20.1 (no shakeloose except the VLRU stuff I accidentally left in due to a bad merge)
[16:14:35] <mvita> that's not unexpected, we were pretty sure it was that
[16:14:41] <mvita> but it helped to confirm it
[16:15:00] <mvita> and that just _exposes_ the problem, it doesn't _cause_ it.
[16:15:20] <mvita> I'm still trying to figure out root cause - struggling w/ systemtap atm
[16:15:47] <meffie> but this means we have a work-around ?
[16:16:04] <meffie> for afflicted users?
[16:16:14] <mvita> well, we always knew reverting shakeloose would do it, that's what we did last time
[16:16:34] <mvita> but that's not really a viable workaround - it just kicks the can down the road
[16:16:45] <wiesand> I didn't know
[16:17:15] <mvita> I am making progress, it's just very slow going.
[16:17:34] <mvita> I wish I could make it go faster, and I'm doing everything I can.
[16:18:11] <mvita> Mike and Andrew have both been helping, and Stephan's experiments are helpful too.
[16:18:30] <wiesand> I haven't managed to prepare that list of changes I'd propose for a "no-prerelease" 1.6.22.
[16:18:49] <wiesand> Proposal: let's wait another week (Linux 4.14 was just released this weekend).
[16:18:55] <meffie> +1
[16:19:19] <kadukoafs@gmail.com/barnowlEBF84E62> Seems reasonable.
[16:19:24] <mvita> Stephan, do you have any tips for generating an RPM with debuginfo for the OpenAFS kernel module?
[16:19:35] <mvita> that's where I'm stuck at the moment
[16:19:49] <mvita> systemtap only works with an RPM install of OpenAFS
[16:20:09] <mvita> but my debuginfo is not coming through my packaging setup
[16:20:12] <wiesand> Unless we have something substantially better next Wednesday, let's do a 1.6.22 with the Linux 4.14 changes, reverted shake-harder, and assorted very low risk changes (doc, build).
[16:20:22] <meffie> i'd ask yadav for hints
[16:20:34] <mvita> that's a thought, Mike.
[16:21:07] <mvita> I'll write him as soon as I'm done here.
[16:21:24] <wiesand> Mark: Look for an open RT ticket filed by Pat Riehecky suggesting how to do that
[16:21:33] <wiesand> It's not too old
[16:21:38] <mvita> okay, thanks!
[16:22:02] <wiesand> At some point I cerated a gerrit change from his first proposal, which got ignored...
[16:22:34] <wiesand> This has been a weak point of our redhat packaging for a long time
[16:23:18] <wiesand> Alternatively, the SL packaging should produce reasonable debuginfo for the kernel moduel package (though I haven't verified that in a while).
[16:23:54] <mvita> SL?
[16:24:04] <meffie> scientific linux
[16:24:05] <wiesand> Scientific Linux
[16:24:08] <mvita> https://rt.central.org/rt/Ticket/Display.html?id=132034
[16:25:25] <meffie> https://gerrit.openafs.org/#/c/11867
[16:25:26] <wiesand> yes, that one
[16:25:32] <mvita> aye
[16:25:34] <mvita> https://gerrit.openafs.org/#/c/11867/
[16:25:35] <meffie> ah, i dont remember this.
[16:25:41] <mvita> ninja'd
[16:26:05] <mvita> this is great, thank you!
[16:26:07] <wiesand> admittedly, Pat's solution looks more elegant than the terrible hacks I applied to the sl spec
[16:26:23] <meffie> nice!
[16:26:38] <meffie> i can help rebase this on current master mark, it needed.
[16:26:52] <meffie> since i was the last to change the spec file.
[16:27:15] <kadukoafs@gmail.com/barnowlEBF84E62> please feel free to update the gerrit change if appropriate
[16:27:25] <wiesand> Any thoughts on "Unless we have something substantially better next Wednesday, let's do a 1.6.22 with the Linux 4.14 changes, reverted shake-harder, and assorted very low risk changes (doc, build)." ?
[16:27:29] <mvita> there's a 1.8.x patch in RT
[16:27:59] <kadukoafs@gmail.com/barnowlEBF84E62> Right, to put the 1..x patch from RT in as an update to the gerrit
change (if it's targetting master)
[16:28:10] <kadukoafs@gmail.com/barnowlEBF84E62> Er, 1.8.x, heh
[16:28:34] <mvita> ok
[16:28:51] <wiesand> Pat is very responsive and really good. Chances are he'll help if needed.
[16:28:54] <mvita> I'm fine with the plan.  <I'm resigned to the plan>
[16:29:29] <mvita> btw, RT search for "riehecky" yielded nothing.
[16:29:38] <mvita> I had to search for "debuginfo"
[16:29:46] <meffie> i wonder if the "revert" should be a runtime option, since we keep turning it on and off.
[16:30:15] <meffie> default to off for now.
[16:30:25] <mvita> —shakeloose [ harder | softer | nada ]
[16:30:52] <mvita> nope, too many options as is
[16:31:03] <meffie> good point
[16:31:19] <wiesand> No, I'm not in favour of such an option either.
[16:31:19] <mvita> just provide a monkeypatch
[16:31:25] <meffie> it should be a sysctl option :)
[16:31:31] <mvita> heh
[16:31:47] <wiesand> How about a pioctl?
[16:31:54] <mvita> sure!
[16:31:54] <meffie> HA!
[16:31:58] <wiesand> Top that ;-)
[16:32:01] <mvita> ALL OF THE ABOVE
[16:32:15] <meffie> we are keeping ben up. next subject?
[16:32:28] <wiesand> Sure
[16:32:50] <wiesand> Though I hope Ben has fallen asleep
[16:33:28] <wiesand> [whispers] Ben?
[16:33:35] <mvita> LOL
[16:34:45] <wiesand> Looks like we're finished for today?
[16:35:00] <meffie> andrew expects to complete his rxgk review this week.
[16:35:11] <wiesand> sounds great
[16:35:22] <kadukoafs@gmail.com/barnowlEBF84E62> > andrew expects to complete his rxgk review this week.
Yay!  I still owe him a response to some of the (asetkey?) comments
[16:35:24] <meffie> he also contributed some changes to asetkey
[16:35:35] <kadukoafs@gmail.com/barnowlEBF84E62> But I don't have any other items -- I didn't do anything this week
[16:35:39] <meffie> yes, any feedback would be useful
[16:36:08] <meffie> thanks ben, safe travels.
[16:36:13] <kadukoafs@gmail.com/barnowlEBF84E62> Thanks to Mike and/or Mark for spotting the last issues in the rx
event stuff.  I ran out of time to do that before I left.  Maybe on
the flight back...
[16:37:33] <meffie> adjourn?
[16:37:51] <wiesand> Yes, let's adjourn.
[16:37:58] <wiesand> Thanks a lot everyone!
[16:38:59] meffie leaves the room
[16:39:18] <kadukoafs@gmail.com/barnowlEBF84E62> Thanks everyone!
[16:39:57] <mvita> bye
[16:40:33] wiesand leaves the room
[17:08:58] meffie joins the room
[17:14:27] meffie leaves the room
[17:37:42] meffie joins the room
[19:48:04] meffie leaves the room
[19:59:18] meffie joins the room
[21:42:57] meffie leaves the room
[21:42:57] kadukoafs@gmail.com/barnowlEBF84E62 leaves the room
[21:42:57] kadukoafs@gmail.com/barnowl2B662C26 leaves the room
[23:56:04] kadukoafs@gmail.com/barnowlEBF84E62 joins the room