The following problem description has been copied from kernel.org bug 5042. If you are implementing this feature, feel free to use this page however you want.
Problem description
The following issue affects the setrlimit() and getrlimit() system calls on Linux 2.6.13 (and earlier) on x86. (And probably it occurs on other 32-bit Linux platforms as well.)
Internally, resource limits are represented in the 'rlimit' structure (defined in include/linux/resource.h) as unsigned longs, meaning 32 bits on x86.
However, this data type is not wide enough. The most pertinent limit here is RLIMIT_FSIZE, which specifies the maximum size to which a file can grow: to be useful, this limit must be represented using a type that is as wide as the type used to represent file offsets, i.e., as wide as a 64-bit off_t.
Current versions of glibc (e.g., 2.3.5) deal with this situation somewhat strangely: if a program compiled with _FILE_OFFSET_BITS set to 64 (i.e., off_t is thus 'long long' -- 64 bits) tries to set a resource limit to a value larger than can be represented in a 32-bit unsigned long, then the glibc wrapper for setrlimit() silently converts the limit value to RLIM_INFINITY. (The rlimit_large.c program below can be used to demonstrate this behaviour.)
In other words, the requested resource limit setting is silently ignored. (One could argue that perhaps the glibc wrapper should give an error, rather than silently turning a very large limit into infinity; however, the glibc developers instead seem to have decided on the current behaviour as a means of dealing with what is fundamentally a kernel problem.)
(NOTE: This problem is not merely a theoretical one facing programmers developing new applications. Since many x86 distributions compile all (file) utilities with -D_FILE_OFFSET_BITS=64, this issue can bite end-users as well, if they expect to be able to set resource limits greater than 2^32-1.)
I guess that the solution to this problem would require new setrlimit64() and getrlimit64() system calls on x86, and the existing 32-bit system calls would need to be retained so that existing binaries would still run.