[00:19:56] --- reuteras has become available [01:21:11] --- Russ has left: Replaced by new connection [01:21:11] --- Russ has become available [03:03:16] --- Russ has left: Disconnected [05:49:21] --- jaltman has left: Replaced by new connection [05:49:22] --- jaltman has become available [06:11:25] --- Jeffrey Altman has become available [06:12:23] --- Jeffrey Altman has left [06:42:03] --- steven.jenkins has become available [07:17:15] --- reuteras has left [07:17:59] --- deason has become available [07:33:59] --- matt has become available [07:35:22] I seem to recall Steven Jenkins mentioning writeup about pthreaded ubik, at NJIT. [08:04:47] --- reuteras has become available [08:10:07] --- reuteras has left [08:10:19] --- reuteras has become available [08:52:55] --- dwbotsch has left [08:53:57] --- dwbotsch has become available [09:29:48] --- reuteras has left [09:48:07] --- matt has left [10:07:46] --- jaltman has left: Disconnected [10:07:55] --- jaltman has become available [10:08:49] --- andersk has left [10:10:06] --- andersk has become available [11:01:39] --- rra has become available [11:15:17] The Debian project is looking at using AFS to store data that needs to be replicated across all project machines, and of course has machines all over the world. Prompting a question that I don't know the answer to: [11:15:25] 11:13 <@weasel> so what you're saying is that if we have a replica in say europe and one in canada, and rebooted the fileserver in canada all the clients there would connect to the europe fileserver but never back? [11:15:46] When do clients reassert their preferences after they've hopped RO servers because the server they were talking to went down? [11:17:14] good question to go into an FAQ [11:18:32] Another question is whether the file server has to walk the whole volume during vos release to know what changed, or whether it knows that immediately. [11:18:50] walk the volume.. [11:18:54] One of the things they're talking about putting in AFS is a copy of the archive, and they already avoid full rsync walks of the archive in favor of inotify since walking the whole tree is too slow. [11:18:57] that one is easy. :) [11:19:01] (but the answer isnt very good) [11:19:11] the answer to the first question is "when is SortServers called" (which is like eery 2 hours when the volume info is refetched) [11:20:25] the answer to the second is the vnode lists are read looking for things to send. [11:20:44] which means 2 files are traversed, the large index and the small index [11:21:08] i disagree that that constitutes "walking the volume", steven [11:21:33] look at src/volser/dumpstuff.c, notable DumpVnodeIndex. 2 calls to it are made, in each case one file is opened. "walk" [11:21:51] I get your point, but what would 'walk' do differently, t hough? [11:22:03] open each vnode to examine it? [11:22:10] or even just stat() it [11:22:37] if we kept a list of what changed, we'd keep it in a vnode index. instead, we... read the vnode index? [11:22:39] They're also worried about DES, of course. [11:22:52] one version of code to kill DES is in gerrit... [11:22:57] I'm pondering whether they could just rotate the afs/debian.org key daily. [11:23:00] ah, yeah, that would be worse. [11:23:17] it's still O(vnodes), which isn't O(changed vnodes) but still, it's not bad [11:24:00] well, there are ways we could optimize it, but i'm not going to do any work unless they actually need it, not because they think they might and aren't doing anything yet [11:24:05] I'd be more worried about the limitation of 13 RO sites [11:24:08] They're worried about the whole-file replication because of the Debian metadata files, which always change and which are huge, but I know there's not much we can do about that. [11:24:37] careful volume separation can help a ton. [11:24:57] they probably would benefit from something like OpenEFS to help manage things [11:25:14] the whole 'delegate replication to data owners' would be good for them, I suspect. [11:25:18] yeah. 13 RO sites would be my worry. tho we could just start on the vldb extensions and have something which is only non-standardized RPCs for the management interface, and use a smart subset out the srtandard RPCs [11:25:35] well, and russ has just the tool for that ;) [11:25:49] rune's talk from bpw made me think it'd be cool if you could distribute software to the degree where ~anyone could set up a fileserver and be a mirror [11:25:50] steven: No, they're not going to care at all about delegated control. [11:25:51] for debian, I'd do multi-cell. [11:26:19] rra - ah, that simplifies adminstration architecture then. [11:26:22] The purpose is to put a copy of the BTS and possibly the archive on every machine without having to copy it onto every machine. [11:26:28] There's basically only one writer in both cases. [11:26:35] It's pretty much a pure RO replica setup. [11:26:48] of note, a "hint list" of changed vnodes could easily be kept, not moved with the volume; if the file is empty, you assume it's complete. if it's not, you assume there is no valid data and do not create it. on a full release, you create an empty file. move volume doesn't move file so on-wire doesnt change. [11:27:15] you can always just rsync to mirrors where each "mirror" is a cell, of course :) [11:27:23] Sounds like 13 replicas isn't a big problem; they mostly just want to avoid trans-Atlantic or trans-Pacific client/server interactions. [11:28:35] --- meffie has left [11:28:38] also, if you're worried about sending whole-file changes, an rsync-like adaptation to dumping/restoring volumes could be possible [11:28:57] --- meffie has become available [11:29:21] the work on such an extension started and is in RT [11:29:42] #? or should I search for it [11:29:45] the bzip extension from the hungarians was intended to go further (there). [11:30:20] search. psomogyi@gamax.hu i think [11:30:32] > they mostly just want to avoid trans-Atlantic or trans-Pacific client/server interactions. well, you're going to get that if the ~1 RO site goes down in whatever location [11:30:43] 13 sites doesn't give you a lot of redundancy per continent [11:30:51] wow. i even got the spelling of his name right from memory [11:30:57] Yeah, that's apparently okay as long as it fails back over to the right continent. [11:30:58] 17947 [11:31:00] but not my call :) just thinking aloud [11:31:27] Mostly, the replicas are used for various analysis programs, and it's okay if they get a bit slow occasionally. [11:32:17] nothing does rsync as-is. basically it stalled when it reached the point of "needs standardization" [11:33:12] and i'm pretty sure sven oehme's group was funding it, started doing something else, and the work fell on its face [11:33:20] If you're going to run a globally-distributed cell and want to have clients talk primarily to servers that are geographically near to them, then you need to either put the clients on the same subnet as the nearby servers or configure their server preferences manually. In either case, the CM will always prefer the lowest-rank fileserver, if it's up, and will notice a down server returning to service within a few minutes. [11:33:50] I suspect they're going to configure the preferences manually. [11:33:54] That's what I'll recommend. [11:33:57] well, the server prefs GSoC project could be used to help, but jake never finished it. i suppose could just finish it [11:34:44] > the replicas are used for various analysis programs oh, so is this for lintian-like programs? not for actual mirrors distributing packages [11:35:37] aaaaah stupid torrenter, go away! [11:35:39] mix [11:35:43] Note I would not recommend having 13 dbservers [11:36:40] yeah, nobody's suggesting that; "sites" as fileservers for RO sites [11:36:43] deason: Right. The mirrors use a much more custom-taliored replication method. [11:37:41] The fallback is to continue to use such a replication method to RW volumes at each file server, but it would be nice to have read-only semantics, particularly since they're concerned about archive integrity with DES. [11:38:04] aren't the archive packages signed with the developers' keys anyway? [11:38:39] No, the packages are checksumed and then that checksum is collected into the Packages file, which in turn is checksumed and signed. [11:38:59] Most of the analysis software, like Lintian, doesn't bother to check all the signatures for each package it touches, since it's slow. [11:39:07] Lintian should be robust against hostile packages, but bugs exist. [11:39:11] I'm not sure how that's different; you can just check those sums [11:39:14] oh, okay [11:39:33] It's slower to check because you have to load the Packages file, which is ginormous. [11:39:51] --- meffie has left [11:40:35] --- meffie has become available [12:07:06] --- meffie has left [12:08:34] --- meffie has become available [12:39:52] --- asedeno has left [12:40:04] --- asedeno has become available [13:28:59] --- jaltman has left: Disconnected [15:14:02] --- mdionne has become available [16:02:18] --- jaltman has become available [16:48:05] --- deason has left [17:00:03] --- kula has left [17:28:01] --- kula has become available [18:13:13] --- shadow@gmail.com/owl1BF2FE0F has left [18:14:24] --- shadow@gmail.com/owl74A97FD8 has become available [18:21:28] --- mdionne has left [19:00:34] --- rra has left: Disconnected [19:14:50] --- jaltman has left: Disconnected [19:19:21] --- Russ has become available [19:38:56] --- dwbotsch has left [19:39:54] --- dwbotsch has become available [19:56:32] --- deason has become available [20:06:16] --- abo has left [20:06:18] --- mho has left [20:06:33] --- abo has become available [20:21:47] --- jaltman has become available [20:55:16] --- phalenor has left [21:05:17] --- phalenor has become available [21:15:51] --- shadow@gmail.com/owl74A97FD8 has left [21:17:00] --- shadow@gmail.com/owl1AE59E52 has become available [21:24:29] --- dwbotsch has left [21:29:21] --- dwbotsch has become available [22:02:51] --- deason has left [22:25:59] --- geekosaur has left [22:26:05] --- geekosaur has become available [23:35:35] --- geekosaur has left [23:35:41] --- geekosaur has become available [23:40:09] --- Russ has left: Disconnected [23:55:08] --- steven.jenkins has left [23:55:49] --- steven.jenkins has become available