[01:28:02] Russ, I've verified the admin guide changes and will push them shortly. One stylist change that might be considered for future revisions is to refer to "an OpenAFS support provider" as opposed to "your AFS support provider". [01:47:00] This study is quite important to OpenAFS. http://fit.nokia.com/lars/tmp/2010-hgw-study-slides.pdf See slide 7 on UDP binding timeouts. The IETF recommendation is 600 seconds whereas half of all tested home routers have a binding timeout of 90 seconds or less. The smallest being 30 seconds. [01:50:04] The most important slide is 9. This one shows that even when the clients and servers continuously exchange data, the udp binding are still going to expire with a median time of 3 minutes. [01:52:17] Given these results it is crucial that OpenAFS clients obtain UPNP support in order to register for specific udp port bindings. Otherwise, it is impossible for us to maintain an rx connection longer than a few minutes even with the nat pinging. [02:04:27] --- Simon Wilkinson has become available [02:06:21] in particular, we need to support UPnP InternetGatewayDevice:1 [02:06:56] --- Simon Wilkinson has left [02:07:05] --- Simon Wilkinson has become available [02:30:53] --- Simon Wilkinson has left [02:34:38] --- Simon Wilkinson has become available [04:06:55] upnp isn't enough. but yes. [04:07:48] I suspect that on boxes that are as badly behaved as some of the ones in that study, nothing will actually be enough. [04:24:52] --- jonas.sundberg has become available [04:47:07] --- Simon Wilkinson has left [04:50:22] --- Simon Wilkinson has become available [05:07:46] --- jonas.sundberg has left [05:24:44] --- kaj has left [05:24:46] --- kaj has become available [05:59:11] --- reuteras has left [06:17:36] --- Simon Wilkinson has left [06:27:59] --- Simon Wilkinson has become available [06:47:46] --- mmeffie has left [06:49:26] --- mmeffie has become available [07:26:12] --- deason has become available [07:37:56] --- matt has become available [08:14:01] --- Simon Wilkinson has left [08:14:01] --- Simon Wilkinson has become available [08:25:53] --- Simon Wilkinson has left [08:27:26] --- kaj has left [08:48:28] --- JSund has become available [08:51:05] --- jaltman has left: Disconnected [08:51:08] --- jaltman has become available [09:01:31] I was looking into implementing the printf/vprintf functions. Is there some function in the standard library that I could call to do the actual conversions or will I have to write that part myself? [09:03:06] in afs libraries? we have afs_snprintf &co [09:03:51] It's for the strsafe.h implementation. I don't know if dependencies on AFS libraries would be good to have there. [09:07:56] --- jaltman has left: Lost connection [09:08:35] oh okay; wasn't sure what 'standard libraries' you were talking about [09:10:31] Sorry, edited my first line and accidentally removed the part about strsafe.h. I'd prefer if I found something that is portable enough available in libc. [09:11:39] How is the availability of snprintf on various platforms? [09:12:03] snprintf is generally available everywhere as far as I know, but sometimes the implementation is broken [09:12:37] Broken in what way? [09:12:38] which is why afs_snprintf exists, or other similar reimplementations... heimdal's libroken has one too, I'm sure [09:14:24] --- Russ has become available [09:16:12] is matt smith about? [09:16:35] I think some always NUL-terminate the string, and some don't? maybe other differences in behavior... others here can probably give more info [09:38:52] Might take a look at the one in libroken. I'm not sure I want to use a copyleft license for strsafe.h and if I'm not wrong the implementation in OpenAFS would be IPL licensed. [09:41:46] I thought there was a public domain one somewhere that russ has mentioned... but I think we got ours from libroken anyway, so yeah [09:45:04] I took a quick look at the code for afs_snprintf and the implementation in libroken, and they seem to be different. Might have been some refactoring or changes since the fork though [09:45:30] Well, at least I have some places to start looking, thanks for the help! [09:45:33] --- JSund has left [09:46:54] There's an implementation in rra-c-util that's under a completely unrestricted license. [09:47:17] That's what I use for all of my code. It does the formatting internally, as opposed to afs_snprintf which, IIRC, wraps the system sprintf. [09:47:24] There are pluses and minuses there. [09:48:01] Note that afs_snprintf supports additional format codes that aren't standard and won't be part of the system printf. [10:15:16] --- pod has left [10:15:35] --- jaltman has become available [10:20:09] --- pod has become available [10:21:46] openafs snprintf probably came from libcs (cmu cs) [10:54:32] --- jaltman has left: Disconnected [11:00:00] --- kaj has become available [11:02:40] --- jaltman has become available [11:03:29] Hi JSund. [11:03:39] oops, he left [12:51:52] > openafs snprintf probably came from libcs (cmu cs) If so, it also has an unrestricted license, and does the formatting internally except for the floating-point specifiers, for which it does indeed wrap the system sprintf. [12:53:20] oh. no. the claimed copyright on the file is KTH [12:53:26] (just remembered to look) [12:53:34] But in fact, the util/snprintf.c I see in my openafs sandbox originated with Heimdal. [12:53:46] "/* This is an enhanced version of the *printf functionality shipped * with Heimdal." [12:54:18] the current src/util/snprintf.c in the tree is from Heimdal. It replaced the prior version of snprintf.c that was in the tree last June. [12:54:18] i'm pretty sure it came from you, actually [12:54:23] Yeah. Like mine, it does its own formatting, except for floating-point specifiers, where it wraps the system sprintf [12:54:23] oh, ok [12:54:47] the previous one probably did come from me [12:54:55] yeah. b45838c is the current one [12:57:35] Yes. The one in stable-1_4_x is substantially the same as mine, but with reindenting. I note that the new one contains a large comment describing what features it supports, etc, which is clearly derived from the comment in my version (which I wrote from scratch). [12:58:05] sure, because I reimplemented the functionality that yours supported into the Heimdal version. [12:58:44] and added the Microsoft extensions. [12:58:53] Well, plus the heimdal one already had some of those features, like "Actually obeys the length limit" :-) [13:00:41] you can read the commit message to see the reasons why I ported the Heimdal version [13:04:34] If you believed there were bugs in the previous version, it'd have been nice to tell me, since we actually use that elsewhere. [13:07:06] Though it's possible they were already fixed. There were actually a couple of forks of that code; IIRC the OpenAFS version didn't have the CMUCS license on it because it was actually not a derivative of the CMUCS version. [13:10:23] Jeff, your description of the implementation stated it wasn't a complete implementation. It did what it did. openafs started requiring more than it was capable of. [13:11:36] I don't object to openafs switching to something else. I just want to be able to fix bugs other places it is used. [13:11:53] in particular, we needed a portable version of asnprintf. [13:12:05] And it certainly should have done padding and justification correctly. [13:12:24] Yeah; wanting asnprintf was a pretty good reason to switch [13:13:01] you can run the util/tests/snprintf_test.c against your other implementations and see if they pass or fail. [13:17:32] Indeed, now that I know :-) [13:26:13] --- kaj has left [13:27:35] Yeah, by "does its own formatting" I specifically meant floating point, sorry. [13:27:54] They all do their own formatting of the other stuff. (Well, the OpenAFS one falls back on sprintf in a few other mostly uninteresting cases.) [13:42:15] It looks like the AFS ones, both old and now, use sprintf for floating point and to simplify construction of the %I output, when the result is an IP address (not a hostname). [13:42:46] Really, it ought to use afs_sprintf in that case, to eliminate the need to have any dependency on an outside sprintf. [13:42:49] Yeah, it's an understandable choice. Doing floating point formatting without introducing a libm dependency is really obnoxious. [13:43:14] I wouldn't have implemented it myself; the snprintf that I'm using just happened to already have that work done for it by other people. [13:51:15] --- kaj has become available [13:53:18] if someone wants to improve the openafs snprintf that would be excellent. if I can push it back to Heimdal that would be even better [14:04:30] Personally, I want to aim towards hopefully eventually getting rid of it, not make it better, but of course in the interim making it better would be great. [14:04:44] Although I'd really rather not add non-standard format specifiers, since getting rid of it IMO should be the ideal. [14:05:05] I realize that it will take many years before we no longer care about the current platforms with bad snprintf implementations, but it will eventually happen. [14:06:32] The non-standard format specifiers are really useful, though. [14:08:23] I don't think we can push upstream with some of the nonstandard things, but if we're clever, we can push something upstream that makes it easy to drop nonstandard things in, such that we can pull new upstream versions from time to time. [14:08:50] I agree that the IP address magic is very handy, although there are other ways of solving that. I'd like to at least look at the other options when we break the current code by adding IPv6. [14:09:30] That's the only extension documented at the top of the file that isn't in standard C99 snprintf, although there may be others. [14:09:44] Oh, well, the Microsoft %I32 and %I64 ones. [14:19:49] jaltman: Whatever 'something else' is (and I'm not really clear you proved it's not cmd.exe + nmake.exe version), but in any case, you've not apparently built on a 2008 Server as a workstation? Either way, is call (batchfile) -incorrect- syntax? Why do you suppose introducing it makes the invocations work correctly in my evironment? [14:21:44] Also, do you suspect call (batchfile) would fail in some environment where (batchfile) now works? [14:21:59] That seems testable. [14:43:41] It does not act like a noop in my environment. [15:00:18] the 2008 terminal server machine that Asanka builds on is win2008 with visual studio 2008. [15:01:38] that machine is also used to execute the buildbot runs. [15:02:58] I'm not really trying to be argumentative, it's just that, I can show the behavior. [15:04:20] what I want to understand is why the behavior is occurring [15:04:33] why you and no one else? [15:06:20] That's reasonable, but I don't know, and the programs involved have not been helpful in providing any hints. My first intuition to call the batch script removed the stop. [15:07:47] My env has had vs2008 and the current vs2005 on it at different times, and, iirc, both environments behaved the same way. [15:27:04] The reported error is U1045 ("spawn failed : No error"). MSDN states this is caused when ComSpec is incorrectly set. set says %ComSpec% is "C:\Windows\system32\cmd.exe" [15:35:34] Google finds someone who claims the problem is vcredist is not installed on the build host. Questionable as that sounds, I have tried installing the x86 and x64 redists, without effect. [15:53:38] use sysinternals procmon.exe to log all activity on the machine during a run. save the contents to a log file and put that file somewhere in /afs that I can access it from. [15:54:24] I doubt the problem is vcredist but I would not be surprised if the problem is somehow related to the environment and the dependencies that some process has which cannot be satisfied. [15:59:04] --- deason has left [16:26:44] ok, I've published side-by-side procmon traces, thanks for looking [16:38:30] --- JSund has become available [16:41:21] jaltman: Yeah, I had to leave before, but I'm back now. [16:47:16] --- JSund has left [16:47:26] --- jonas.sundberg has become available [16:55:29] hi Jonas [16:56:01] hi [16:56:19] with regards to StrSafe. your implementation should not have any dependencies on openafs. your project is intended to be standalone [16:56:57] As I thought then. Do you have any opinions on how to handle the printf functions? [16:57:24] the BSD license from Heimdal is certainly compatible with what you are using. [16:58:25] Is it really? I was a bit uncertain about the third clause and thought that the ISC license is more like two-clause BSD licenses. [16:59:15] I think you should start by implementing with the assumption that there is a working asnprintf implementation available to you. whether it is provided by the OS distribution or from a compatibility library such as openafs uses should not matter. [17:00:29] if there is time later on you can come back to updating the openafs version to not rely on a pre-existing implementation by integrating code from Russ. [17:00:54] or fixing up the version that Jeff Hutzelman wrote that can be re-licensed [17:02:32] Would asnprintf be a better choice than to use snprintf to the buffer directly? [17:03:02] not necessarily. if you have asnprintf you should certainly have snprintf available. [17:03:27] the reason I pulled asnprintf into openafs is because Windows does not provide it. [17:04:03] if you can avoid additional in memory copies, your code should be faster. [17:04:18] Basically, you should be able to assume you have working versions of all of {v,}{a,}s{n,}printf [17:04:59] I guess one reason for using asnprintf would be that a version of snprintf that just wraps to sprintf might cause a buffer overflow that shouldn't be possible by allocating data for a new string. But if I can assume that snprintf works as intended, that would probably be more efficient. [17:05:50] I will read through the logs when I get home in a couple of hours. [17:06:20] About the logs, where are logs for the XMPP chat available? [17:06:45] http://jabber.openafs.org/ [17:06:49] I think I've seen it mentioned some time, but I can't recall if that was in some earlier discussions here in the XMPP channel. [17:06:59] Thanks [17:07:10] --- jaltman has left: Disconnected [17:08:10] --- JSund has become available [17:08:12] --- jonas.sundberg has left [17:10:49] --- kaj has left [17:10:50] --- kaj has become available [17:17:20] Do you want asprintf or asnprintf? I usually use the former. [17:19:12] The function in strsafe.h primarily replaces sprintf and snprintf, so I guess snprintf would be the function that makes most sense to use. [17:19:31] well, probably the va_list form is easier to wrap to [17:20:03] Neither of those give you a va_list (that would be vasprintf). asprintf allocates however much memory is needed. asnprintf does the same but lets you limit the total string size. However, my impression is that asnprintf is less portable than asprintf. [17:20:20] For example, I don't believe Linux has asnprintf, but it has asprintf. [17:20:23] So vsnprintf and the corresponding wide character function i guess. [17:21:22] vswprintf. Although wide-character functions are basically never used on Linux, with only a tiny handful of exceptions. [17:21:34] I think that's true for UNIX in general. [17:21:41] UNIX almost always uses UTF-8. [17:21:51] Except for Java, of course, which does its own thing. [17:21:54] :) [17:22:08] But since it's in the library specification, I guess it should be there. [17:22:10] I know of a few applications that use wide characters instead, but there are weird portability headaches since the width of the character varies by platform. [17:22:17] Yeah, it probably should be. [17:22:28] I just wouldn't make that part the highest priority. :) [17:22:47] So far, porting the char implementations to use wchar_t has been quite easy. [17:23:02] I haven't tested using any non-ascii characters yet though. [17:24:46] Is OpenAFS using any wide characters? File names in Windows perhaps? [17:27:18] --- kaj has left [17:27:20] --- kaj has become available [17:30:50] Nowhere intended to be portable [17:45:11] IIRC, glibc uses UCS-4 for wchar_t, but a lot of other platforms use UCS-2BE, which is kind of a mess. [17:45:18] But saves two bytes per character. [17:45:25] Java, for example, I believe uses UCS-2BE. [17:45:49] --- matt has left [17:46:11] Well, I should say that they probably use either UCS-2BE or UCS-2LE depending on the native byte order, but with Java at least if you ever externalize the string, you always get BE. [17:46:27] And Microsoft AD uses UCS-2BE. [17:46:57] I think at least Java uses UTF-16 [17:47:02] not that they differ much [17:47:56] UTF-16 is a variable-length character encoding, so it surprises me that Java would have all the additional logic to cope with that to produce correct answers to questions about string length and so forth. But I could be wrong. [17:48:05] they don't [17:48:33] That would make it UCS-2, wouldn't it? Since otherwise the primitives for string length and the like break for surrogate pairs. [17:48:35] the library functions count wrong if surrogate pairs are involved [17:48:39] Oh, okay. [17:48:52] So you can use surrogate pairs, but you're then on your own for what happens with lengths. [17:48:54] That makes sense. [17:49:18] Yeah, that's UTF-16 (arguably buggy) rather than UCS-2, since real UCS-2 would ban surrogate pairs. [17:50:03] the pain of surrogate pairs and all the questions around whether they count as one character or two in string lengths passed to library functions is the reason why glibc used a 32-bit wchar_t. [17:51:14] * Russ really dislikes UTF-16 -- it's one of those annoying technologies that makes it way too easy and tempting to write code that only breaks in very unusual circumstances that no one tests. [17:51:18] How is that handled for UTF-8? Count bytes? [17:51:32] At least with UTF-8 you have to deal with character length for any non-ASCII character, so you have to be aware of it. [17:51:39] Yeah, it's handled very painfully. :) [17:51:57] Basically, if you're using UTF-8 in char, all the lengths are byte lengths and if you need character lengths, you have to roll it yourself. [17:52:12] I'm quite aware of the UTF-8 problems (and conversions to/from ISO-8859-1) since I'm Swedish. :) [17:52:28] Aha. :) [17:52:50] I don't know how many times each year I have to inform students of the difference. [17:53:22] We're just now *finally* getting to the point where I can mostly use UTF-8 uniformly on Linux and have it display properly everywhere. [17:53:33] I teach a course where many students try out C or C++ for the first time and one of the lab assignments involve indexing a text file written in Swedish. [17:58:20] That's a great starting assignment. [18:00:07] It has caused some headaches though, since the school has historically used ISO-8859-1 encoding and that is used for the text. Some text editors have a tendency to change the encoding of text files to UTF-8. [18:01:32] The students are given a tokenizer that does the tokenizing of the input file for them and in the source code for that program, Swedish characters are encoded in ISO-8859-1. When this source code is converted to UTF-8, the tokenizer no longer works as intended. [18:02:24] It happened for my lab partner when I took the course and it takes a while to realize that the problem is that you unintentionally changed the encoding of a file that you haven't done any actual changes in. [18:11:27] --- jaltman has become available [18:16:24] Yeah, that's one of the things I love about Emacs -- it tries hard to recognize the file encoding and not change it. [18:23:01] Vim does the same for me. If I recall correctly it was Textmate on OS X that was the culprit. [18:23:15] Yeah, nvi is the one that causes me heartburn. [18:25:22] Really, nvi does that? [18:25:40] I would have expected it to be more minimalist and not touch anything. [18:26:01] It's a slightly different problem -- it doesn't convert, it truncates at the first invalid UTF-8 character. [18:26:11] Oh. Yeah, I could see that. [18:26:16] It does yell pretty loud when it does, so you can :q! and then use another editor. [18:26:34] I am lucky enough that I can usually get away with LANG=C, though, so I don't run into these things very often. [18:27:22] Debian converted everything to UTF-8 that's part of the packaging infrastructure. [18:27:55] Which I'm fairly happy about; I have a thing about using people's actual names in things like changelogs, and we have a fair number of contributors with Chinese, Japanese, or Arabic names they prefer. [18:57:33] --- Russ has left: Disconnected [19:12:38] --- JSund has left [19:17:49] --- Russ has become available [20:21:04] you kids and your wacky editors [20:21:37] I think you mean wacky distros. "I just use what ships with the base system." [20:22:31] me too. osx had emacs and TextEdit. works fine [20:28:46] --- Born Fool has become available [20:49:23] Eclipse is going to start spoiling me. [20:49:45] Not really so much for all the editor stuff, but more because the bit of diagnosing your syntax errors immediately as you type is seriously addictive. [20:50:14] It's like real-time spell-checking except massively more useful. [20:59:37] to some extent emacs does that for me. [20:59:44] i didn't configure it. it just did [21:11:58] --- Born Fool has left [21:21:28] --- kaj has left [23:24:25] --- kaj@kth.se has become available [23:44:26] --- kaj@kth.se has left [23:44:27] --- kaj has become available