So you have a crash dump on your Linux box and you want to know more about the reason your application has crashed, lets get started!
On Ubuntu there is a directory /var/crash, when stuff pops up in here you know its bad news… Lets walk through it.
First, we start by cwd into /var/crash directory, when there, look for a dump in our example: _opt_janus_bin_janus.1001.crash. Second, we extract the contents from the dumpfile providing all the information we need to debug it.
We can attach a gdb debugger to the CoreDump by running gdb `cat ExecutablePath` CoreDump when the output has finished run a backtrace with bt so that you can retrace the steps in the application and find out where it all went wrong.
root@svr01:~# cd /var/crash/ root@svr01:/var/crash# ls -l total 132608 -rw-rw-r-- 1 nhooijenga nhooijenga 135785781 nov 7 13:00 _opt_janus_bin_janus.1001.crash root@svr01:/var/crash# apport-unpack _opt_janus_bin_janus.1001.crash janus root@svr01:/var/crash# cd janus/ root@svr01:/var/crash/janus# ls Architecture DistroRelease ProblemType ProcEnviron Signal CoreDump ExecutablePath ProcCmdline ProcMaps Uname Date ExecutableTimestamp ProcCwd ProcStatus root@svr01:/var/crash/janus# gdb `cat ExecutablePath` CoreDump
The output of the command will look a little something like this.
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"…
Reading symbols from /opt/janus/bin/janus…done.
[New LWP <int>]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/janus/bin/janus'.
Program terminated with signal SIGABRT, Aborted.
0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f692f85e700 (LWP 11620))]
(gdb)
(gdb) bt
0 0x00007f6b34ecef47 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
1 0x00007f6b34ed08b1 in __GI_abort () at abort.c:79
2 0x00007f6b34f19907 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f6b35046dfa "%s\n") at ../sysdeps/posix/libc_fatal.c:181
3 0x00007f6b34f2097a in malloc_printerr (str=str@entry=0x7f6b35048650 "malloc_consolidate(): invalid chunk size") at malloc.c:5350
4 0x00007f6b34f20c1e in malloc_consolidate (av=av@entry=0x7f6ad8000020) at malloc.c:4441
5 0x00007f6b34f24848 in _int_malloc (av=av@entry=0x7f6ad8000020, bytes=bytes@entry=1196) at malloc.c:3703
6 0x00007f6b34f2735d in __GI___libc_malloc (bytes=1196) at malloc.c:3065
7 0x00007f6b369d8ab9 in g_malloc () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
8 0x000055f84662d059 in janus_ice_relay_rtp (handle=0x7f6b1822e030, packet=0x7f692f84d370) at ice.c:4627
9 0x00007f6b2d045c74 in janus_videoroom_incoming_rtp (handle=0x7f6b182cff50, pkt=0x0) at plugins/janus_videoroom.c:5020
10 0x00007f6a84026980 in ()
11 0x00007f6a840e01f0 in ()
12 0x00007f692f84d830 in ()
13 0x00007f692f84d7f8 in ()
14 0x000055f846653e5a in janus_rtp_header_extension_parse_video_orientation (buf=, len=, id=, c=0xf7ebb2400000006, f=0xde558aae00000008, r1=0xba7efee0, r0=0x7f6b181eed38) at rtp.c:195
15 0x00007f692f84dab0 in ()
16 0x00007f6b181eed38 in ()
17 0x000055f846632149 in janus_ice_cb_nice_recv (agent=, stream_id=, component_id=, len=, buf=0x7f692f84dab0 "\220k\317*\n\353\277J\373E\225\016\276", , ice=0xde558aaeca6d7f00)
at ice.c:2571
18 0x00007f6b3729e4b0 in nice_component_emit_io_callback (agent=0x7f6b046b6e40, component=0x7f6a8412beb0, buf=0x7f692f84dab0 "\220k\317*\n\353\277J\373E\225\016\276", , buf_len=1182) at component.c:956
19 0x00007f6b372a8cf9 in component_io_cb (gsocket=, condition=, user_data=0x7f6a5c118520) at agent.c:5559
20 0x00007f6b36f700f9 in () at /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
21 0x00007f6b369d3285 in g_main_context_dispatch () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
22 0x00007f6b369d3650 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
23 0x00007f6b369d3962 in g_main_loop_run () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
24 0x000055f846629408 in janus_ice_handle_thread (data=0x7f6b181eed00) at ice.c:1160
25 0x00007f6b369fb175 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
26 0x00007f6b352886db in start_thread (arg=0x7f692f85e700) at pthread_create.c:463
27 0x00007f6b34fb1a3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
Comments are closed