[01:19:57] --- lars.malinowsky has become available [01:20:10] --- lars.malinowsky has left [01:22:12] --- lama has become available [01:34:15] --- pod has become available [02:08:20] --- Simon Wilkinson has become available [03:04:26] --- haba has become available [03:36:33] --- Russ has left: Disconnected [04:04:57] --- haba has left [04:42:04] --- haba has become available [05:07:52] --- Simon Wilkinson has left [05:07:57] --- Simon Wilkinson has become available [05:15:10] --- steven.jenkins has left [05:15:39] --- steven.jenkins has become available [05:25:19] --- Simon Wilkinson has left [07:45:23] --- lama has left [07:45:29] --- lars.malinowsky has become available [08:02:20] --- deason/gmail has become available [08:09:44] I have a single volume where it took 13 hours to make an incremental backup. Some suggestions where to start digging? vos backup before the dump only took 3 minutes. Corner dates are 36GB total in 6208509 files and the incremental contains approx 4.5GB of data. [08:13:05] "incremental backup" I assume means an incremental dump of the backup volume? [08:13:48] 'vos size -dump -time' will give you an actual number for the dump size stream instead of guessing (unless you have a copy of the dump stream to look at still) [09:01:52] I have the dump, in TSM, so I know it ended up to be 4,573,268 KB [09:02:58] With incremental I mean dump against some point in time (here 2011-08-26 to 2011-09-05 (YYYY-MM-DD)) [09:06:31] With 6208509 files and 48279 seconds it has been looking at approx 128 files/second. That is not that great. [09:09:09] I don't think that I am hindered by the whopping data rate of 95KBytes/sec... [09:09:21] well, I mean, is that the size of the dump _stream_ (the size of the file if you 'vos dump -time > foo') or the size of the files backed up combined, or some other value after tsm processes it? [09:09:31] and you can look at pstacks periodically or.... what platform is this? [09:09:47] do we have dtrace? [09:10:15] This is the size as reported by vos size -dump. To get the actual size I have to get back the dump stream from TSM. [09:11:23] (TSM stores the size you give it along with the object and not its actual size) This is Centos. [09:15:02] I have of course complaints in the VolserLog that the transaction takes a lot of time, but nothing other than that. [09:19:04] Hm. Do many accesses on volume foo slow down reading foo.backup? [09:20:51] well, it's i/o to the same disk.... [09:21:40] I was more thinking about locks. [09:21:44] but anyway, you can try to grab periodic pstacks when the dump is running, or you can look at the WCHAN column in 'top' or... add/find volserver instrumentation [09:22:01] "accesses" from the fileserver? [09:22:08] yes [09:23:56] I can't think of any way how it would; once the fileserver attaches and the volserver checks out the vol, they pretty much keep to themselves [09:25:55] Well, will contact the user as well, might be doing something stu**** hrmmm - not optimal - ;-) [09:27:26] really, once you hit DumpVolume the only thing that should be interesting is dumping vnodes, but that would have to mean ih_open. only one volume dump going at that point? [09:27:56] *if* more than one, perhaps some contention on IH_LOCK, but that seems unlikely [09:30:02] it might be interesting to see if you are leaking fds, thus every file taking longer to open. that's about all i can think of. but that should mean each successive volume dump also is slow. [09:37:01] somewhere i have ihandle-package debug code. i should really just start refining this stuff enough that it can be checked into the tree ifdef'd for testing for subsystems instead of trying to remember where when and why for each of these damn things. [09:40:13] --- Russ has become available [09:50:55] only vol dump going on. Yes. [09:52:47] You mean slower and slower over the lifetime of the volser process? No, that's not the case. [09:54:46] Slower and slower over one vos dump operation as it could accumulate fds? I don't know, [09:54:58] but gotta run -> home now. [09:57:04] --- haba has left [10:02:02] --- meffie has become available [10:03:22] --- steven.jenkins has left [10:04:34] --- meffie has left [10:05:07] --- steven.jenkins has become available [10:43:27] --- Russ has left: Disconnected [10:59:52] --- Simon has become available [11:34:15] --- Simon has left [12:03:35] --- Simon has become available [12:18:25] --- haba has become available [12:29:08] --- Simon has left [12:41:20] --- steven.jenkins has left [14:35:59] --- haba has left [15:08:51] --- Simon Wilkinson has become available [15:22:02] --- deason/gmail has left [15:37:28] --- Simon Wilkinson has left [16:19:09] --- summatusmentis has left [16:49:45] --- Russ has become available [17:32:58] --- phalenor has left [17:33:04] --- phalenor has become available [19:31:05] --- shadow@gmail.com/barnowl67590625 has left [19:31:19] --- shadow@gmail.com/barnowl67590625 has become available [20:33:15] --- jaltman/FrogsLeap has left: Replaced by new connection [20:33:16] --- jaltman/FrogsLeap has become available [22:27:32] --- jaltman/FrogsLeap has left: Replaced by new connection [22:27:33] --- jaltman/FrogsLeap has become available [22:32:38] --- jaltman/FrogsLeap has left: Disconnected [22:32:46] --- jaltman/FrogsLeap has become available [22:46:11] --- reuteras has become available [22:55:26] --- jaltman/FrogsLeap has left: Replaced by new connection [22:55:27] --- jaltman/FrogsLeap has become available [23:18:37] --- Russ has left: Disconnected [23:49:34] --- reuteras has left [23:53:40] --- jaltman/FrogsLeap has left: Replaced by new connection [23:53:41] --- jaltman/FrogsLeap has become available