RCS seems to have a limitation on the file size. We have now around 7000 registered users at TWiki.org. The TWikiUsers topic gets updated automatically with each registration.

 -rw-r--r-- 1 nfsnobod nfsnobod 228780 Sep 4 20:42 Main/TWikiUsers.txt -r--r--r-- 1 nfsnobod nfsnobod 1972295 Sep 2 04:11 Main/TWikiUsers.txt,v 

Registration does update the TWikiUsers topic, but since a few days ago, the rcs file does not get updated anymore. Possibly there is a 2MB limit on the rcs file, or a 220KB limit on the source file. The man pages do not mention a limit. Anyone knows? Anyone knows a fix?

-- PeterThoeny - 05 Sep 2003

Strange - it could be an RCS bug (perhaps with large number of revisions?) but a quick search didn't find one.

I did check ulimit -a on twiki.sf.net, which hopefully has similar setup to the web servers, but a CGI script would tell for sure - the -f value limits file sizes but is much bigger than this, so it's not a Linux limit:

 rdonkin@sc8-pr-shell1:~$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) 32768 file size (blocks, -f) 262144 max locked memory (kbytes, -l) 32768 max memory size (kbytes, -m) 32768 open files (-n) 50 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 8191 virtual memory (kbytes, -v) 262144 

-- RichardDonkin - 05 Sep 2003

We have now one more limitation at TWiki.org: The ArgumentListIsTooLongForSearching the Main web with over 7500 topics.

-- PeterThoeny - 10 Sep 2003

Two minute test hack:

 #!/bin/perl open(IN, "<iter.txt"); $x = <IN>; chop $x; close(IN); $x = 1 if !$x; while (1) { print "$x\n"; open(OUT, ">iter.txt"); print OUT "$x\n"; close OUT; system("curl", "-u", "jcline:pass", "-F", "text=<iter.txt", "http://localhost/~jcline/cgi-bin/save/Main/TestTopic" ); $x++; } 

However, I had to manually set $doUnlock in Store.pm at top of saveNew in addition to setting the cfg vars $editLockTime = "0"; and $doKeepRevIfEditLock = "0"; and I'm not sure why (I don't think it's the + 120 secs that are added either).

Test Results (on FreeBSD 4.7 server using rcs v5.7):

• Up to v1.9168 and still running ok:

 -rw-r--r-- 1 www www 125 Sep 10 22:01 TestTopic.txt -r--r--r-- 1 www www 2001743 Sep 10 22:01 TestTopic.txt,v 

• Up to v1.34390 and still running ok:

 -rw-r--r-- 1 www www 127 Sep 11 08:49 TestTopic.txt -r--r--r-- 1 www www 7649317 Sep 11 08:49 TestTopic.txt,v 

• Up to v1.43299 and still running ok (well.. except for an apache problem, see below .

 -rw-r--r-- 1 www www 127 Sep 11 15:28 TestTopic.txt -r--r--r-- 1 www www 9643593 Sep 11 15:28 TestTopic.txt,v 

• Up to POST attempt 64351 and revisioning has temporarily stopped at v1.64168

 -rw-r--r-- 1 www www 127 Sep 12 12:50 TestTopic.txt -r--r--r-- 1 www www 14315113 Sep 12 12:50 TestTopic.txt,v 

• • I believe revisioning stopped due to another restart of httpd while the script was running. The lock on the rcs is owned by admin. Perhaps there's a race condition when starting (or stopping?) apache and POSTing, leaving apache with a wierd uid. Editting in TWiki and restarting the script has everything running properly again.
• Up to v1.66077 and still running ok--the 16-bit boundary (65535) passed ok! (Note: rdiff ran for over 25 minutes before I killed it and was only able to render a couple dozen diff's!)

 -rw-r--r-- 1 www www 127 Sep 12 16:13 TestTopic.txt -r--r--r-- 1 www www 14745870 Sep 12 16:13 TestTopic.txt,v 

• • This is getting a bit obnoxious at this point but what the hey, I'll let it run over the weekend. :-P
• Up to v1.115872 and still running ok--the six-digit boundary (v1.100000) passed ok.

 -rw-r--r-- 1 www www 129 Sep 29 11:15 TestTopic.txt -r--r--r-- 1 www www 25979237 Sep 29 11:15 TestTopic.txt,v 

• I'd guess it's a sourceforge limitation.
• The above ,v file size translates into 218 bytes per revision. The file content itself is only 5-7 characters (a text number). That's over 3000% overhead. What a waste. Look what happens when I try gzip -1 on the ,v: 197913 bytes; that's a compression ratio of 90.5% ! So it'd be quite an improvement to compress after store; use a pipe to eliminate the temporaries (that'd likely cause oddities in cygwin).
• Speed at v1.38294 is about 5x as slow as v1.1. (Subjective measurement + looking at seconds between revisions; it's about 3-4 seconds right now, whereas previously it was 1 second-- the minimum resolution of RCS.)
• At around v1.43270, the IT guy came in my office and asked why the server's /var was at 98% capacity (over 250megs), though du didn't show any real usage on that partition (about 2.5 megs). I guessed that the apache log had grown a bit too big, had failed to rotate/truncate, and perhaps didn't exist as a valid filehandle or something, and sync didn't flush it. Turns out /var/log/httpd.log didn't exist (it should); restarting apache solved that problem and the non-existent 250 meg filehandle was flushed (/var back to 1% capacity). So I guess TWiki is more stable than apache in this case. (My log.txt has grown to 3.7meg.)
• At around v1.82313, my office machine Windows XP hung, so my ssh session to the server was terminated. Thus I conclude that TWiki is more stable than Windows XP. (Though perhaps that's a rather sad comparison, with Windows being an incredibly unreliable piece of software.)

-- JonathanCline - 11 Sep 2003

Thanks for all the testing - this at least shows that it's specific to SourceForge, and is a great stress test!

Could you post details of your OS and RCS version, and whether the OS included RCS or you got it from elsewhere? It may be worth compiling up the latest version of RCS on the SourceForge servers and using that.

• See above. -- JonathanCline - 15 Sep 2003

-- RichardDonkin - 13 Sep 2003

Thanks - you didn't specify whether RCS is built-in to FreeBSD but it turns out that it is. The CVS source browser is here (assuming RELENG_4_7_0_RELEASE is the right tag), and it seems that it's a port of GNU RCS, so it's quite likely this would work on the TWiki shell servers, since the original code is quite mature anyway.

-- RichardDonkin - 16 Sep 2003

I am setting this to resolved, based on the extensive tests.

-- PeterThoeny - 23 Jul 2004