We have a 91MB executable. It SEGVs even before getting to main().
When ldd is run on the executable, ldd also SEGV's.

We are running Redhat 7.2 with a 2.4.7-10 kernel, 2GB of RAM, 4GB of
swap. glibc 2.2.4, gcc 3.2.2, binutils-2.13.

All limits (datasize, stacksize) are unlimited.

We suspect that the problem is due to a large amount of statically
allocated memory (28MB). 28MB of static variables is bad programming,
but ultimately isn't that much compared to the size of the system.
Anyway, I don't 100% believe that this is the problem.

The same program compiled on Sun runs fine.

Here's the output of "/usr/bin/size -A big_exec":

section size addr
..interp 19 134512916
..note.ABI-tag 32 134512936
..hash 24560 134512968
..dynsym 65360 134537528
..dynstr 116753 134602888
..gnu.version 8170 134719642
..gnu.version_r 224 134727812
..rel.dyn 20016 134728036
..rel.plt 26400 134748052
..init 24 134774452
..plt 52816 134774476
..text 58873072 134827296
..fini 30 193700368
..rodata 8637900 193700416
..eh_frame_hdr 258740 202338316
..data 4127600 202601152
..eh_frame 1152096 206728752
..gcc_except_table 509404 207880848
..dynamic 248 208390252
..ctors 2220 208390500
..dtors 2180 208392720
..jcr 4 208394900
..got 13556 208394904
..bss 28616928 208408480
..stab 5448 0
..stabstr 13026 0
..comment 207610 0
..debug_aranges 43016 0
..debug_pubnames 399528 0
..debug_info 6575603 0
..debug_abbrev 142103 0
..debug_line 500023 0
..debug_frame 163108 0
..debug_str 1370842 0
..note 12220 0
..debug_ranges 6424 0
Total 111947303

Before we start eliminatinng all the static variables, can anyone
confirm that this is indeed the problem? Any other ideas or
explanations?

Thanks,

Jeff

Jeff wrote:
BWAAAAAAAAAAAAAAAAAAAAAAAHAAAHAAAHAAHAHAAHAAAA!!!


BWAAAAAAAAAAAAAAAAAAAAAAAHAAAHAAAHAAHAHAAHAAAA!!!


The rest of us suspect the 91 MB executable!


No. A 91 MB executable is BAD PROGRAMMING.


Yes. Eliminate the moron who made a 91 MB executable.

On Fri, 20 Feb 2004 06:09:46 +0000, paul cooke's tax attorney wrote:

What a dumbass you are. You don't know nothing about their design
decisions and spit out unnecessary comments..

The question really is: Is the size of the executable really the problem?
I don't see any reason why a 91meg executable should not work.. I haven't
tried it myself though..

Try debugging the thing.. I really don't think the size is a problem..

Flo

P.S.: If i have X megabytes of virtual ram free then i can expect my
operating system to execute a program that has Y megs as long as Y < X. Is
there any design decisions inherent to linux that would make this
impossible for huge programs above an arbitrary size?

The above paragraph should be especially true, since linux like any other
usable OS doesn't load the whole program binary into virtual memory but
only those parts that actually do get executed..

But: you might have discovered a bug..

--
signature

On Fri, 20 Feb 2004 11:30:30 +0100, florian schmidt wrote:

I'd also grab the latest gcc (or whatever you're using to compile the
thing) off the appropriate web site, update your libs, and so on. There
have been some flaky compilers released recently.

Since it compiles and runs on a sun, but not on a linux system, I'd
suspect a compiler that pukes.

--Kamus

--
o__ | If you're old, eat right and ride a decent bike.
,>/'_ | Q.
(_)\(_) | Usenet posting`

And we have other executables that are much larger that do run
successfully so I don't really think that 91MB in and of itself is the
problem.

I left out that of course we've tried to debug the thing and gdb
doesn't tell us much. Basically, before it gets to main(), it SEGVs
and backtrace just points to some random global variable.

The fact that ldd also SEGVs on the binary really leads me to believe
that the binary itself has something screwy about it. I was hoping
that a guru would be able to look at the "size -A" output I provided
in the original posting and say something interesting about what it
was.

Jeff

Jeff <foobarbaz99@hotmail.com> wrote:
I do. When you get it below 10MB, call back.

To prove it, cut the executable in half. Continue halving until it
runs.

Which tells you that it is way fubar.

Oh, brilliant moriarty.

Probably. But why would anyone care? It should just show how much
program text there is, how much space used by globals, and the overall
size. Change those numbers until it works.

Peter

Jeff <foobarbaz99@hotmail.com> wrote:
I would assume it is. How much stack and heap space is the process
limited to? (sounds like a corrupt binary anyway).

Peter

florian schmidt wrote:
What's the limit on a "jump relative address" instruction
on this particular hardware or this particular
assembler? If the startup code
supplied by the compiler implementation does that
( <somepath>/ctr1.o or gcrt1.o ) then
you just may get a SEGV before hitting main()

Just a thought.


--
Ñ
"It is impossible to make anything foolproof because fools are so
ingenious" - A. Bloch

florian schmidt wrote:
I bet you don't even realize that your statement implies that I do know
about their design decisions. Therefore, you have just validated my
opinion.

A 91 MB executable is bad design, idiot.

paul cooke's tax attorney wrote:

[snip]

I agree with you in the abstract, but you really should
give reasons why you feel it is bad design.

For example:
"At a rate of 8 KB per average disk read
and a rate of 100 I/O operations per second
it would take about 100 seconds to read in the
executable before it starts executing."

This MAY be relevant or irrelevant in this case.

There may well be justifications for the choices
which are unknown to us from just the postings.


--
Ñ
"It is impossible to make anything foolproof because fools are so
ingenious" - A. Bloch

It could be, but it's hard to say from what you have told us so far.
Please tell us the hardware architecture (x86, ppc, ...).
Then run
$ readelf --segments EXECUTABLE
$ objdump --file-headers EXECUTABLE
$ objdump --private-headers EXECUTABLE
and try
$ gdb EXECUTABLE
(gdb) run ARGS
SIGSEGV
(gdb) info proc
(gdb) shell cat /proc/PID/maps ## where PID is the process ID given above
(gdb) q
$

--

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

At 2004-02-20T19:21:38Z, Nick Landsberg <hukolau@att.net> writes:

I agree, except that it's unlikely that the entire binary needs to be
loaded before execution can begin - depending again, of course, on the
(unknown) architecture of the binary in question.
- --
Kirk Strauser
The Strauser Group
Open. Solutions. Simple.
http://www.strausergroup.com/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFANmI75sRg+Y0CpvERArieAJ9ThbbhGWznrXG/EyBhdupgoq2a9wCfV0ov
XPAbZM2S/jEg2ScDl+DjV1M=
=gXXi
-----END PGP SIGNATURE-----

P.T. Breuer wrote:
concerns has not been applied in the design. There is a Murphy's Law
about in every giant program is a little function whose purpose is
opposite to that of the overall program. A program that large is too
large to understand.

So by dividing it into concurrent cooperating processes, communicating
by UNIX System V messages (if you think in the old days) or TCP/IP if
you think modern, and keep each down to 10Mb (if you accept Peter's
suggestion) or even twice that in unusual cases), you may find you can
understand each program separately. You can also modify each one without
having to modify the others, as the messages exchanged remain the same.
Another advantage, if TCP/IP is used is that they need not all run on
the same system, and the processes need not even know that.

Now in Linux, where process creation is so cheap, why not run a dozen or
more cooperating sequential processes to get a single overall big job
done? Especially if you have multiple CPUs on your machine (SMP) or even
just a single hyperthreaded Intel XEON? With one big backbreaking
process, you can run on only one processor (at a time), where with
cooperating processes, each can have a chance at a processor at the same
time.

--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ Registered Machine 73926.
/( )\ Shrewsbury, New Jersey http://counter.li.org
^^-^^ 4:30pm up 45 days, 3:52, 2 users, load average: 2.15, 2.13, 2.09

Jean-David Beyer wrote:

Ah, good. Now we have another reason (as opposed to the plain
statement by another poster that 91 MB was "idiotic").

All good points, Jean-David.

I did notice in the original post that, by far, the
"text" segment was taking up the bulk of the space.
This would be a qualitative indication that what you
say about Murphy's law is true, but does not explain
the SEGV the OP encountered BEFORE the entry to main().

There are advocates of huge monolithic process structures
on single-CPU machines who claim that it lets the process
itself control the thread of execution, rather than rely
on the vaguaries of the scheduler. It also avoids
context switch overhead, as every message (whether it
be SysV or Sockets) involves a system call, and thus
a context switch into and out of kernel space.

In addition, if you have never worked with communicating
processes (and the possible error conditions when messages
get lost or delivered out of sequence) will usually
get it wrong the first time (or even the second or
third time). There are tradeoffs to any design.
You have correctly mentioned some of the benefits
of a loosely coupled design and have mentioned
some of the benefits of the monolithic design.
(I am NOT a proponent of the monolithic design,
BTW, I am just playing devil's advocate.)

It's up to the OP and his organization to make
the choice, and they seem to have settled on
the monolithic, for better or for worse.


P.S. - I would tend to disagree that process creation is "cheap"
on ANY O/S. If it is ONLY done on startup, then it's
OK. If programs fork()/exec() others willy-nilly,
it's real pig.

--
Ñ
"It is impossible to make anything foolproof because fools are so
ingenious" - A. Bloch

x86

Elf file type is EXEC (Executable file)
Entry point 0x8094d20
There are 7 program headers, starting at offset 52

Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg
Align
PHDR 0x000034 0x08048034 0x08048034 0x000e0 0x000e0 R E
0x4
INTERP 0x000114 0x08048114 0x08048114 0x00013 0x00013 R
0x1
[Requesting program interpreter: /lib/ld-linux.so.2]
LOAD 0x000000 0x08048000 0x08048000 0x40ee2c0 0x40ee2c0 R
E 0x1000
LOAD 0x40ee2c0 0x0c1372c0 0x0c1372c0 0x589ccc 0x20d45c0 RW
0x1000
DYNAMIC 0x467386c 0x0c6bc86c 0x0c6bc86c 0x000f8 0x000f8 RW
0x4
NOTE 0x000128 0x08048128 0x08048128 0x00020 0x00020 R
0x4
LOOS+474e550 0x40af00c 0x0c0f700c 0x0c0f700c 0x3f2b4 0x3f2b4 R
0x4

Section to Segment mapping:
Segment Sections...
00
01 .interp
02 .interp .note.ABI-tag .hash .dynsym .dynstr .gnu.version
..gnu.version_r .rel.dyn .rel.plt .init .plt .text .fini .rodata
..eh_frame_hdr
03 .data .eh_frame .gcc_except_table .dynamic .ctors .dtors
..jcr .got .bss
04 .dynamic
05 .note.ABI-tag
06 .eh_frame_hdr

huge_binary_name: file format elf32-i386
architecture: i386, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x08094d20

huge_binary_name: file format elf32-i386

Program Header:
PHDR off 0x00000034 vaddr 0x08048034 paddr 0x08048034 align
2**2
filesz 0x000000e0 memsz 0x000000e0 flags r-x
INTERP off 0x00000114 vaddr 0x08048114 paddr 0x08048114 align
2**0
filesz 0x00000013 memsz 0x00000013 flags r--
LOAD off 0x00000000 vaddr 0x08048000 paddr 0x08048000 align
2**12
filesz 0x040ee2c0 memsz 0x040ee2c0 flags r-x
LOAD off 0x040ee2c0 vaddr 0x0c1372c0 paddr 0x0c1372c0 align
2**12
filesz 0x00589ccc memsz 0x020d45c0 flags rw-
DYNAMIC off 0x0467386c vaddr 0x0c6bc86c paddr 0x0c6bc86c align
2**2
filesz 0x000000f8 memsz 0x000000f8 flags rw-
NOTE off 0x00000128 vaddr 0x08048128 paddr 0x08048128 align
2**2
filesz 0x00000020 memsz 0x00000020 flags r--
0x6474e550 off 0x040af00c vaddr 0x0c0f700c paddr 0x0c0f700c align
2**2
filesz 0x0003f2b4 memsz 0x0003f2b4 flags r--

Dynamic Section:
NEEDED libncurses.so.5
NEEDED libpthread.so.0
NEEDED libdl.so.2
NEEDED libqt-g.so
NEEDED libm.so.6
NEEDED libc.so.6
NEEDED libX11.so.6
INIT 0x8087eb4
FINI 0xb8ba210
HASH 0x8048148
STRTAB 0x805e088
SYMTAB 0x804e138
STRSZ 0x1c811
SYMENT 0x10
DEBUG 0x0
PLTGOT 0xc6bda98
PLTRELSZ 0x6720
PLTREL 0x11
JMPREL 0x8081794
REL 0x807c964
RELSZ 0x4e30
RELENT 0x8
VERNEED 0x807c884
VERNEEDNUM 0x4
VERSYM 0x807a89a

Version References:
required from libdl.so.2:
0x0d696911 0x00 11 GLIBC_2.1
0x0d696910 0x00 08 GLIBC_2.0
required from libm.so.6:
0x0d696910 0x00 06 GLIBC_2.0
required from libc.so.6:
0x09691f73 0x00 10 GLIBC_2.1.3
0x0d696912 0x00 07 GLIBC_2.2
0x0d696911 0x00 04 GLIBC_2.1
0x0d696910 0x00 03 GLIBC_2.0
required from libpthread.so.0:
0x0d696912 0x00 09 GLIBC_2.2
0x0d696911 0x00 05 GLIBC_2.1
0x0d696910 0x00 02 GLIBC_2.0

08048000-0c0eb000 r-xp 00000000 00:22 28277999 /path/to/big/binary
0c0eb000-0c668000 rw-p 040a3000 00:22 28277999 /path/to/big/binary
0c668000-0e1b2000 rwxp 00000000 00:00 0
dfffe000-e0000000 rwxp 00000000 00:00 0

Nick Landsberg <hukolau@att.net> wrote in message news:<67tZb.58733$hR.1268870@bgtnsc05-news.ops.worldnet.att.net>...
Only if the pages of the executable are scattered across the disk,
thus maximizing the number of seek operations. You should get 30
to 50MB/s from a modern disk, assuming it is on the local machine.

The address ranges reported by /proc/PID/maps do not agree with the
PT_LOAD descriptions from readelf and objdump:

offset paddr vaddr filesz memsz prt align
0x08048000 + 0x40ee2c0 is 0xc1362c0, rounded up to page boundary is 0xc137000,
so the first page range in /proc/PID/maps is expected to be

08048000-0c137000 r-xp 00000000 00:22 28277999 /path/to/big/binary
instead of the reported
08048000-0c0eb000 r-xp 00000000 00:22 28277999 /path/to/big/binary
^^^

0x0c1372c0 + 0x589ccc is 0xc6c0f8c, rounded up to page boundary is 0xc6c1000,
so the second page range is expected to be

0c137000-0c6c1000 rw-p 040ee000 00:22 28277999 /path/to/big/binary
instead of the reported
0c0eb000-0c668000 rw-p 040a3000 00:22 28277999 /path/to/big/binary
^^^ ^^ ^^

and the third page range for .bss is also bad.

So, it looks like the kernel did not understand your big binary;
binfmt_elf erred during execve().
What kernel is this ["uname -r" or "cat /proc/version"]?
Is the hardware healthy? Is the big binary on a local filesystem,
or is it on NFS or some other remote filesystem? Try a local disk.

--

In comp.os.linux.advocacy, paul cooke's tax attorney
<paul.cooke.tax.attorney@boogers.com>
wrote
on Fri, 20 Feb 2004 06:09:46 GMT
<siue8t47.2bc81114@cookes.house>:
For what it's worth, I have a 78 megabyte executable and had
no trouble with it. (The bulk of that 78 megabytes is a
data-section static lookup table. I had to use the assembler
for that as g++ choked, but that's during the build, not
the running thereof.) It was, of course, for a benchmark/contest.

Of course, I also have 320 megs on that machine, so it fits nicely
therein. Now had I a 400 megabyte executable, things might get
a little dicey. (Of course, the first question one might ask is
"why 400 megabyte executables?", which is a good question; the
best way to handle this sort of thing is through a portable
data format.)

Runs fine on my one 64 MB RAM/128 MB Swap box, though,
although part of that is simply because the pages stay on
disk; it only really needs one of them (it's a gigantic
lookup table). Ldd works, too.

I suspect a badly built 91 MB executable.

--
#191, ewill3@earthlink.net
It's still legal to go .sigless.

Nick Landsberg wrote:

Thank you.
True; I have not attempted to discover the problem in that 91MB hunk of
code. You could not pay me enough to even look. Talk about trying to
find a needle in a haystack... . 91MB is just way too big a haystack.
There are 9q processes in my process table right now (essentially idle),
and the biggest memory hog is Mozilla with 6 instances using 57
Megabytes RSS (each?) and share of 26756. I consider that excessive for
a single process, and the Mozilla group are probably sorry it got so big.

I suppose I would try to single step the program with gdb, but someone
(I suppose the OP) said even running ldd on the load module caused ldd
to crash, and that tells me that the load module is probably totally
scrozzled. ldd does not run the program after all, so bugs or no bugs,
it should be able to handle that program. Maybe its dynamic linking
information is broken in a way that ldd does not notice; in which case,
an MR should probably be filed against ldd. I should be able to give ldd
a garbage file, and not crash it (though I would expect a lot of insults
from ldd, of course). In fact, when I give it a garbage (jpeg file),
with mode set to 0770, it says:

valinux:jdbeyer[/usr/local/girls]$ ldd frammis
not a dynamic executable

So the OP's file looks like a dynamic executable.

I remember a hardware engineer (computer repair person) who always
insisted in reducing any problem to a 4 assembler-level instruction
sequence or he assumed troubles were software problems. I had just
written an operating system (kernel in today's terminology) and when the
shell (using todays terminology) was supposed to type out the the
sequence "Ready: ", instead it typed "Rdady: ". Now this was not a
disaster in itself, but it really worried me since if I dumped the
memory, the text to be printed was clearly "Ready: ". So I copied the
code down to the instruction loop that did that, and it worked just
fine. Fortunately, I wrote the OS kernel in relocatable format (unusual
in those days because no memory management hardware), so I diddled the
loader to move the OS down one word in the memory space, and everything
worked. So I could give the hardware techie a 13 instruction sequence
that, _if executed in specific absolute hardware addresses_, did print
incorrectly as described. After that, it took him very little trouble to
replace a printed circuit card in the memory controller. (That whole OS
kernel took only 4096 instructions (including any data tables it
needed). No windowing system, no multiprogramming, no multiprocessing,
no modems, but it was pretty fast for what it did, and simple enough
that even my boss's boss could program and use it.)
Fine if they can run without an OS. Otherwise, they are going to have to
deal with an OS in any case. There will be a context switch everytime
the OS needs to do something anyway. Right now my SMP machine is doing
nothing other than running two instances of setiathome, a
compute-limited process, and this composer in Mozilla.

procs swap io system cpu
r b w si so bi bo in cs us sy id
2 0 0 0 13 0 20 204 548 95 5 0
2 0 0 0 0 0 5 138 334 93 7 0

Even if you think you are avoiding context switches, you probably are
not. Now you do not just walk into a design review and throw different
bubbles arbitrarily into different UNIX (or Linux) processes. You must
give each process a descriptive name that pretty much tells people what
the process does (but hides how it does it). Normally, you arrange each
process to do a lot of processing with little input or output to keep
message passing and context switching overhead low.
I have done it a lot. I find systems designed that way are easier to
design, work better the first time, are easier to modify (because the
repercussions of implimentation detail changes do not propagate beyond
the process boundary unless you violate the interface specification, etc.

My favorite system came up with about 100 processes, but as more and
more users came on line, other processes were spawned dynamically. A
monolithic program would be unmanageable.

We arranged that the sequence of messages did not matter. Each process
was a finite state machine, and the messages could come in any order. If
the messages got lost, the responce expected by the sender timed out and
the sender could send it again, or do something else. The biggest
problem, IIRC, was when the input queues of processes filled up (we were
using System 5 IPC at the time), and it was a lot of bother to design
the system so that could not happen.

As it was designed, we could even change the implementation of running
processes and test them without taking the system down. Certain users
could run the new implimentation and the rest (most of them) got the
default implimentation. We just had to obey the interface specification,
and if we goofed that up, only our own stuff failed. This was a
wonderful way to test new algorithms and stuff on live data without
hurting the innocent.

Understood.
change, they would argue that it is too late: they have too much
invested in what they already have. Reminds me of a motto I ascribed to
a former employer, though they HATED it: "We haven't time to stop for
gas; we're late already!"

A well, time to re-read Fred Brooks' book, "The Mythical Man Month",
again, I suppose.
about 1970. It is much much better now, when we do paging instead of
process swapping, when the processors have memory management units at
their disposition, ... .

Well, for most operating systems, process creation is costly, when it is
allowed at all, and I have run on systems where you could not do it at
all. A privileged user had to introduce a new process to the system
before it could be run at all. But in Linux, fork|exec are pretty good
compared to the average OS. In the systems I worked on with dynamically
created processes, there tended to be less than a dozen created when a
new user logged into the system (typically a user did that at most once
a day). Recall that these days, if there are many instances of a process
(as was typical of our system), they all share the same instruction
space, that space is shared, so the memory consumption is reduced to
only the necessary data and stack space. I do not know if fork, these
days, notices that the parent and child are the same process, so
allocate only the data space or not. But exec should know that what it
is invoking is just another instance of a running program and need not
remanage the memory for redundant copies of the instruction space.

--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ Registered Machine 73926.
/( )\ Shrewsbury, New Jersey http://counter.li.org
^^-^^ 10:25pm up 45 days, 9:46, 2 users, load average: 2.20, 2.24, 2.14

uname -a for our machines is 2.4.7-10, but we also have some custom
kernel hacks so it's not a stock RH7.2 kernel.

I was able to get ldd to run correctly when I moved the binary to a
more recent RH version. This machine has a newer kernel (2.4.22-1),
newer libc, and didn't have any of our custom kernel hacks. So I'm
sure it's one of these things, but I'm not sure which one. Oh well, I
think we're on the road to a solution.

Just to make sure, and regardless of how certain people view them,
there's nothing that limits the size of an ELF binary or any of its
sections besides physical RAM, correct?

Jeff