[04:11:43] --- reuteras has left [04:18:24] --- SecureEndpoints has left: Replaced by new connection [04:49:45] --- RedBear has left [05:09:39] --- dev-zero@jabber.org has left: offline [06:54:13] --- reuteras has become available [07:26:44] --- mmeffie has become available [07:50:31] --- haba has become available [07:52:03] It is very much friday today. [07:52:47] yes. time to drink heavily. [07:52:59] tea time. [07:54:12] I even had to cold boot the micro wave after it only displayed "LO:C ". [08:12:59] --- reuteras has left [08:47:53] --- SecureEndpoints has become available [08:54:07] --- Russ has become available [08:54:13] --- haba has left [10:13:13] --- dev-zero@jabber.org has become available [10:13:25] --- dev-zero@jabber.org has left: offline [10:16:59] --- mmeffie has left [11:27:27] --- mmeffie has become available [11:30:18] got some stupid questions regarding 124359 [11:30:29] ok [11:31:32] --- stevenjenkins has left [11:31:38] you indicate that there is an ihandle ref count leak. were you able to determine if the leak goes away when namei-fsync-less-20070907 is removed [11:31:53] --- tkeiser@sinenomine.net/owl has become available [11:32:34] the leak i think is in vclosevnodefiles_r [11:32:46] i strongly suspect [11:34:11] i believe there is a missing IH_RELEASE for the IH_REALLYCLOSE. [11:34:13] --- stevenjenkins has become available [11:34:25] did you test it? [11:34:41] test adding a IH_RELEASE prior to the IH_REALLYCLOSE ? [11:34:42] yeah. that leak has been there forever. it's just that until Rainer's fsync patch and dafs came along there was insufficient concurrency to allow the problem to manifest itself [11:34:58] IH_RELASE _after_ IH_REALLYCLOSE [11:35:33] so, does this manifest in 1.4.8? i havent tried. [11:37:04] i did test with the IH_RELEASE added, and no volumes are lost then. i was not sure why at the time. [11:37:33] the key was seeing ih_sync_all re-opening files. [11:38:40] the appear to be two problems. (1) prove that is in fact the leak and fix it; (2) identify why already closed files are being syncd [11:39:08] the first is a bug; the second is a performance issue [11:40:01] It should be possible to determine by analysis that there is a leak there. It doesn't require some kind of experimental evidence. It also doesn't mean it's the only leak. [11:40:48] I did a code audit several weeks ago, and fully convinced myself that it was a leak [11:40:58] experimental evidence is useful for demonstrating that the hypothesis possible [11:41:53] I'm not sure "already closed files" are being synced. If there is an ihandle ref leak, then it's still referenced. It may not have an open fd, because we cache open fd's and sometimes have to reclaim a cached fd in order to open something else. AFAIK there is no way to tell the difference between an ihandle which has no fd because it was reclaimed and one where it has none because IH_REALLYCLOSE was called, and I don't think you can assume something in the former state doesn't need to be synced. [11:42:36] we can add a flag to indicate why the fd was last closed [11:42:50] Sure, if you haven't already analyzed the code and said "look, we're forgetting a reference but not updating the refcount" [11:44:00] We can, if my memory of the situation is right (which is dicey at best). But really, what we should have is a flag indicating it's dirty and needs to be synced. [11:44:56] However, I suspect that in the case of this particular leak, and the sequence of operations Mike used to discover it, it is entirely possible that there are inodes which were dirtied and derefed without ever having been synced. [11:45:23] adding a flag won't actually help, in fact it can hurt us. remember that ihandles can have references from every clone within the VG. If we take a clone offline and VCloseVnodeFiles_r gets invoked, we could end up disabling syncing when we still want it (due to the RW still being online) [11:45:25] ... which is a case we should short-circuit, if we don't already. But maybe we do, if the missing IH_RELEASE actually gets called. [11:47:21] Oh, hm, you're right. IH_REALLYCLOSE doesn't mean the last ref went away. Hm. But if we actually close the fd, it should either be because we're reclaiming it or because we _know_ nothing in this process cares. If the ihandle is open multiple times via multiple volumes, we shouldn't close the fd just because an RO goes offline. [11:47:34] OTOH, also note that an inode which is shared with a clone _cannot be dirty) [11:47:53] (well, the link table can, but it's special) [11:48:37] jhutz, regarding the reclaiming of fd, if we have a ihandle with zero fdhandles, how could that possibly need to be synced? [11:49:28] it seems to me the IH_OPEN in ih_sync_all is wrong, ie a short cut to traverse the fd cache. [11:53:38] > reclaiming it or because we _know_ nothing in this process cares. If the > ihandle is open multiple times via multiple volumes, we shouldn't close > the fd just because an RO goes offline. Yes. This has been bothering me for some time as well. I'm not sure we can easily solve this without reworking the ihandle API rather thoroughly [11:53:41] ih_sync_all() is failing to take into account that IH_OPEN() increments the refcount [11:55:10] OK, tom, so you believe we are just overagressively closing things sometimes? [11:55:42] > OK, tom, so you believe we are just overagressively closing things > sometimes? yes [11:59:24] does it ever make sense for ih_sync_all to open the file if there are currently no fd for the ih? at that point it is already synced. [12:00:10] > at that point it is already synced. Is it? What if the fd was closed because we ran out of fd's? [12:00:38] I have to get going; have fun. [12:02:26] I think the question boils down to whether the operating system is going to sync data from memory to disk when a file descriptor is closed. at least in the case of Windows, the answer to that is 'no' [12:03:23] dirty pages can be maintained in the system cache until such time as the operating system feels it is appropriate to write them to disk. [12:03:47] and even then disks with on board caches do not have to write them to the platter immediately [12:04:59] ok, thanks. [12:05:07] if we were to call OS_SYNC() on every fd prior to closing it, then we could be sure that there is no sync left to be performed [12:05:48] I can't say whether that would result in performance degradation or not. [12:06:58] mike, please update the ticket to include your hypothesis regarding the missing IH_RELEASE in VCloseVNodeFiles_r [12:08:20] ok. do i need an account do to that? [12:08:38] no. just respond to the e-mail you received when you opened the ticket. [12:10:19] the leak is not coming from ih_sync_all although it might be aggrevated by it. the FD_CLOSE in ih_sync_all() will remove the refcnt from the IH_OPEN() call. The other refcnt increments are balanced by decrements. [12:12:03] tom, when you performed your code audit, which refcnt increment(s) did you identify as matching the IH_RELEASE that is required by VCloseVNodeFiles_r()? [12:13:03] iirc, IH_INIT in VnLoad() [12:14:16] --- mmeffie has left [12:20:56] that looks right. [12:21:09] this bug is present in 1.4.x as well [12:21:31] we should be seeing ihandle refcnts with very large values [12:26:24] agreed. I've been meaning to walk the ih hash table and look for just such a smoking gun. [12:49:27] --- mmeffie has become available [14:28:52] --- dragos.tatulea has become available [14:32:47] hi [14:37:05] hi [14:37:33] thank you for writing up the wiki page. I've been traveling all week and have been unable to find the time to review it [14:43:58] it's meant to be an incentive for me to start writing the code [14:45:00] :) [14:45:23] so, I see that you are starting the gsoc applications. What will be the projects this year? [14:53:40] --- tkeiser@sinenomine.net/owl has left [15:10:38] --- tkeiser@sinenomine.net/owl has become available [16:08:25] --- edgester has become available [16:08:42] --- edgester has left [16:08:45] --- edgester has become available [16:34:07] --- kula has become available [16:47:27] --- dev-zero@jabber.org has become available [16:47:32] --- dev-zero@jabber.org has left: offline [18:11:21] --- Russ has left: Disconnected [19:01:13] --- RedBear has become available [19:01:18] --- RedBear has left [19:01:38] --- RedZBear has become available [20:07:22] --- edgester has left [22:24:14] --- mmeffie has left