How to debug server hangs / crashes

Discussion for any ET/ETPro/BayonET bugs or cheats you find...

Moderators: Forum moderators, developers

Locked
User avatar
bani
Site Admin
Posts: 2780
Joined: Sun Jul 21, 2002 3:58 am
Contact:
Contact bani

How to debug server hangs / crashes

Post by bani »

how to debug a stuck server ("99% cpu")

using top or ps, find the PID of the stuck process:

Code: Select all

top - 02:04:46 up 14 days,  2:25, 17 users,  load average: 0.16, 0.13, 0.09
Tasks: 103 total,   2 running, 100 sleeping,   0 stopped,   1 zombie
Cpu(s): 13.9% us,  1.7% sy,  0.0% ni, 83.5% id,  0.7% wa,  0.0% hi,  0.3% si
Mem:   1036008k total,   997920k used,    38088k free,    69664k buffers
Swap:   530136k total,   240764k used,   289372k free,   329244k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12976 wolf      16   0  178m  39m 1428 R 13.9  4.0   0:01.94 ettv.x86
bold text are the commands you type.

[wolf@server wolf]$ gdb
GNU gdb Red Hat Linux (5.2-2)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux".
(gdb) attach 12976
Attaching to process 12976
Reading symbols from /usr/local/games/enemy-territory/ettv.x86...done.
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /home/wolf/.etwolf/pb/pbsv.so...done.
Loaded symbols for /home/wolf/.etwolf/pb/pbsv.so
Reading symbols from /lib/libnss_nisplus.so.2...done.
Loaded symbols for /lib/libnss_nisplus.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /home/wolf/.etwolf/etpro/qagame.mp.i386.so...done.
Loaded symbols for /home/wolf/.etwolf/etpro/qagame.mp.i386.so
0xb7f5f43e in select () from /lib/libc.so.6
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0xb7fb7730 0xb7fcf340 Yes /lib/libm.so.6
0xb7fb0d10 0xb7fb1a00 Yes /lib/libdl.so.2
0xb7ea0020 0xb7f918b4 Yes /lib/libc.so.6
0xb7feea50 0xb7ffdc10 Yes /lib/ld-linux.so.2
0xae3a4cd0 0xae3ab570 Yes /lib/libnss_files.so.2
0xae18a420 0xae31bd1b Yes /home/wolf/.etwolf/pb/pbsv.so
0xab5e2b50 0xab5e9450 Yes /lib/libnss_nisplus.so.2
0xab5d0a20 0xab5dc900 Yes /lib/libnsl.so.1
0xab5c9e10 0xab5cbbd0 Yes /lib/libnss_dns.so.2
0xab5bb6e0 0xab5c5390 Yes /lib/libresolv.so.2
0xab66bce0 0xab70eff0 Yes /home/wolf/.etwolf/etpro/qagame.mp.i386.so
(gdb) info registers
eax 0xfffffdfe -514
ecx 0xbffff650 -1073744304
edx 0x0 0
ebx 0x16 22
esp 0xbffff610 0xbffff610
ebp 0x2864c 0x2864c
esi 0x0 0
edi 0xbffff648 -1073744312
eip 0xb7f5f43e 0xb7f5f43e
eflags 0x202 514
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x0 0
fctrl 0x37f 895
fstat 0x125 293
ftag 0xffff 65535
fiseg 0x0 0
fioff 0x0 0
foseg 0x0 0
fooff 0x0 0
fop 0x0 0
xmm0 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm1 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm2 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm3 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm4 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm5 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm6 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm7 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
mxcsr 0x1f80 8064
orig_eax 0x8e 142
(gdb) bt
#0 0xb7f5f43e in select () from /lib/libc.so.6
#1 0xbffff650 in ?? ()
Cannot access memory at address 0x2864c
(gdb) stepi

Program received signal SIGSTOP, Stopped (signal).
0xb7f5f43e in select () from /lib/libc.so.6
(gdb) stepi
0xb7f5f43e in select () from /lib/libc.so.6
(gdb) stepi
0xb7f5f43f in select () from /lib/libc.so.6
(gdb) stepi
0xb7f5f440 in select () from /lib/libc.so.6
... (do stepi about 20-30 times) ...
(gdb) quit
The program is running. Quit anyway (and detach it)? (y or n) y
Detaching from program: /usr/local/games/enemy-territory/ettv.x86, process 12976

<hr>

How to debug a crashing server without logs

using top or ps, find the PID of your running gameserver.

Code: Select all

top - 03&#58;42&#58;00 up 14 days,  4&#58;02, 16 users,  load average&#58; 0.09, 0.31, 0.60
Tasks&#58; 102 total,   1 running, 100 sleeping,   0 stopped,   1 zombie
Cpu&#40;s&#41;&#58;  3.9% us,  1.3% sy,  0.0% ni, 93.9% id,  0.6% wa,  0.0% hi,  0.3% si
Mem&#58;   1036008k total,   964960k used,    71048k free,    74380k buffers
Swap&#58;   530136k total,   240764k used,   289372k free,   297500k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
13475 wolf      16   0  173m  33m 1428 D  3.6  3.3   0&#58;01.99 etded.x86
bold text are the commands you type.

[wolf@server wolf]$ gdb
(gdb) attach 13475
Attaching to process 13475
Reading symbols from /usr/local/games/enemy-territory/etded.x86...(no debugging symbols found)...done.
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /home/wolf/.etwolf/pb/pbsv.so...(no debugging symbols found)...done.
Loaded symbols for /home/wolf/.etwolf/pb/pbsv.so
Reading symbols from /lib/libnss_nisplus.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_nisplus.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /home/wolf/.etwolf/etpro/qagame.mp.i386.so...(no debugging symbols found)...done.
Loaded symbols for /home/wolf/.etwolf/etpro/qagame.mp.i386.so

0xb7f5f43e in select () from /lib/libc.so.6
(gdb) continue
Continuing.

Program received signal SIGSTOP, Stopped (signal).
0xb7f5f43e in select () from /lib/libc.so.6
(gdb) continue

your server will now run normally.

when it crashes:

Program received signal SIGSEGV, Segmentation fault.
0xb7f5f43e in select () from /lib/libc.so.6
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0xb7fd2d10 0xb7fd3a00 Yes /lib/libdl.so.2
0xb7fb3730 0xb7fcb340 Yes /lib/libm.so.6
0xb7ea0020 0xb7f918b4 Yes /lib/libc.so.6
0xb7feea50 0xb7ffdc10 Yes /lib/ld-linux.so.2
0xae3a4cd0 0xae3ab570 Yes /lib/libnss_files.so.2
0xae18a420 0xae31bd1b Yes /home/wolf/.etwolf/pb/pbsv.so
0xab5e2b50 0xab5e9450 Yes /lib/libnss_nisplus.so.2
0xab5d0a20 0xab5dc900 Yes /lib/libnsl.so.1
0xab5c9e10 0xab5cbbd0 Yes /lib/libnss_dns.so.2
0xab5bb6e0 0xab5c5390 Yes /lib/libresolv.so.2
0xab64bce0 0xab6eeff0 Yes /home/wolf/.etwolf/etpro/qagame.mp.i386.so
(gdb) bt
#0 0xb7f5f43e in select () from /lib/libc.so.6
#1 0x00000016 in ?? ()
#2 0x080ef8e4 in _IO_stdin_used ()
#3 0x65746163 in ?? ()
Cannot access memory at address 0x69646564
(gdb) info registers
eax 0xfffffdfe -514
ecx 0xbffff650 -1073744304
edx 0x0 0
ebx 0x16 22
esp 0xbffff610 0xbffff610
ebp 0x88a0d9c 0x88a0d9c
esi 0x0 0
edi 0xbffff648 -1073744312
eip 0xb7f5f43e 0xb7f5f43e
eflags 0x202 514
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x0 0
fctrl 0x37f 895
fstat 0x125 293
ftag 0xffff 65535
fiseg 0x0 0
fioff 0x0 0
foseg 0x0 0
fooff 0x0 0
fop 0x0 0
xmm0 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm1 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm2 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm3 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm4 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm5 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm6 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm7 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
mxcsr 0x1f80 8064
orig_eax 0x8e 142
(gdb) quit
Top
Locked

Return to “ET Bugs/Cheats”