Analyze corefile and get backtraces with GDB for MySQL/MariaDB on Linux Nilnandan Joshi, Sr. Support Engineer, MariaDB nilnandan@gmail.com www.nilinfobin.com
Core Dump/File: A core dump is a disk file containing an image of the process’s memory at the time of termination. This image can be used in a debugger to inspect the state of the program at the time that it terminated. The core dump includes key pieces of program state as processor registers, memory management details, and other processor and operating system flags and information.So a core dump is a file that can be very useful to understand the context of a crash.  GDB: GDB, the GNU Project debugger, allows you to see what is going on `inside' another program while it executes -- or what another program was doing at the moment it crashed. i.e sudo gdb -p `pidof mysqld` sudo gdb /path/to/mysqld /path/to/coredump sudo gdb --batch --eval-command="thread apply all bt" /usr/sbin/mysqld $(pgrep -xn mysqld) > mysqld_bt_all_threads.txt sudo gdb --batch --eval-command="thread apply all bt full" /usr/sbin/mysqld $(pgrep -xn mysqld) > mysqld_full_bt_all_threads.txt You can use GDB to study: • InnoDB locks, table locks, metadata locks etc • server variables, user variables at session level • Threads and Statements
Backtraces: A backtrace is a summary of how your program got where it is. It shows one line per frame, for many frames, starting with the currently executing frame (frame zero), followed by its caller (frame one), and on up the stack. In a multi-threaded program, GDB by default shows the backtrace only for the current thread. To display the backtrace for several or all of the threads, use the command thread apply. For example, if you type thread apply all backtrace(thread apply all bt) , GDB will display the backtrace for all the threads; this is handy when you debug a core dump of a multi-threaded program. Each line in the backtrace shows the frame number and the function name. The program counter value is also shown—unless you use set print address off. The backtrace also shows the source file name and line number, as well as the arguments to the function. The program counter value is omitted if it is at the beginning of the code for that line number.
How to enable Coredump: In order to enable core dumps, you need to set the core_file system variable either on the command-line or in a relevant server option group in an option file. [mariadb] ... core_file There are some additional details related to using core files on Linux. On some systems there is a limit on the sizes of core files that can be dumped. So you have to disable some of the restrictions. To check the system's current system-wide limit, run ulimit -c for the current limit of the mysqld process specifically: sudo cat /proc/$(pidof mysqld)/limits | grep "core file" If you need to change the core size limit, the method you use depends on how you start mysqld. i.e mysqld using mysqld_safe [mysqld_safe] … core_file_size=unlimited
- mysqld using systemd sudo tee /etc/systemd/system/mariadb.service.d/limitcore.conf <<EOF [Service] LimitCORE=infinity EOF sudo systemctl daemon-reload If you are using Linux, then it can be helpful to change a few settings to alter where the core files is written and what file name is used. This is done by setting the kernel.core_pattern and kernel.core_uses_pid attributes. By default, core file will be generated in MySQL/MariaDB datadir. If you are using a production system, you probably want to have the core files in a specific directory, not in the data directory. (Because core file size could be very large somewhere in GBs) so for that you can set something like, sudo mkdir /tmp/corefiles sudo chmod 777 /tmp/corefiles sudo echo /tmp/corefiles/core > /proc/sys/kernel/core_pattern sudo echo 1 > /proc/sys/kernel/core_uses_pid Since  mysqld  executes  setuid, you may have to set  fs.suid_dumpable=2  to allow core dumps on Linux.  sudo echo 2 > /proc/sys/fs/suid_dumpable
In MariaDB 10.1.35, MariaDB 10.2.17, and MariaDB 10.3.9 and later, core_file has also been made into a system variable. Previously it was just an option. It's value can be checked at runtime by executing the following: SHOW GLOBAL VARIABLES LIKE 'core_file'; However, in MariaDB 10.3.7 and later, some large buffers have been excluded from core files on some systems as a way to reduce the size. The following buffers are excluded: InnoDB buffer pool InnoDB log buffer Server recv buffer Query cache https://mariadb.com/kb/en/enabling-core-dumps/ https://dev.mysql.com/doc/refman/8.0/en/server-options.html#option_mysqld_core-file https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Other Requirements: 1. GDB need to be installed if it's not already yum install gdb 2. debug info packages needs to be install for server and common atleast. i.e MariaDB-server-debuginfo-10.4.17_10-1.el8.x86_64 MariaDB-common-debuginfo-10.4.17_10-1.el8.x86_64 3. If you can’t run gdb + core dump on customer’s server then you must have the same mysqld/mariadb binaries including debug-info packages + all dynamically linked libraries + core file to run gdb in your local machine. Libraries can be checked by ldd command. i.e [root@nilcentos7 ~]# ldd /usr/sbin/mysqld linux-vdso.so.1 => (0x00007ffea8ddd000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2149b93000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f214995c000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f2149736000) … libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f2148810000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f214860c000)
Problem: 190328 23:31:08 [ERROR] mysqld got signal 6 ; ... Server version: 10.2.23-MariaDB-10.2.23+maria~stretch ... Thread pointer: 0x7ff4d8001f28 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff4dc62ccc8 thread_stack 0x49000 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7ffa09af5bfb] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffa09b7e437] /lib/x86_64-linux-gnu/libc.so.6(+0xf7570)[0x7ffa09b7c570] /lib/x86_64-linux-gnu/libc.so.6(+0xf93aa)[0x7ffa09b7e3aa] /usr/sbin/mysqld(my_addr_resolve+0xe2)[0x55ca42284922] … /usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x72) [0x55ca41b4b992] /usr/sbin/ mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15 enum_duplicatesb+0x1206)[0x55ca41b560f6] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3f68)[0x55ca41b6bee8] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x28a) [0x55ca41b70e4a] /usr/sbin/mysqld(+0x4c864f)[0x55ca41b7164f] ...
Problem: 2021-02-12 8:39:33 11039259 [ERROR] mysqld: Table '(temporary)' is marked as crashed and should be repaired 2021-02-12 8:41:09 10757048 [ERROR] mysqld: Table '(temporary)' is marked as crashed and should be repaired free(): invalid next size (normal) 210212 8:41:09 [ERROR] mysqld got signal 6 ; ... Server version: 10.4.17-10-MariaDB-enterprise-log ... Thread pointer: 0x7f179d1103c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0a020edbe8 thread_stack 0x49000
[root@centos8 nil]# gdb mysqld core.2751610 GNU gdb (GDB) Red Hat Enterprise Linux 8.2-12.el8 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/ gpl.html> … Type "apropos word" to search for commands related to "word"... Reading symbols from mysqld...Reading symbols from /usr/lib/debug/ usr/sbin/mysqld-10.4.17_10-1.el8.x86_64.debug...done. done. [New LWP 2338279] [New LWP 2751610] … … Use the "info sharedlibrary" command to see the complete listing. Do you need "set solib-search-path" or "set sysroot"? [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". … Core was generated by `/usr/sbin/mysqld'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from / lib64/libc.so.6 [Current thread is 1 (Thread 0x7f0b74ee2700 (LWP 2338279))]
(gdb) (gdb) thread 1 — this is for the crash [Switching to thread 1 (Thread 0x7f0b74ee2700 (LWP 2338279))] #0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from / lib64/libc.so.6 (gdb) backtrace — to get the backtrace #0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from / lib64/libc.so.6 #1 0x0000563bd293aa84 in memcpy (__len=<optimized out>, __src=<optimized out>, __dest=<optimized out>) at /usr/include/ bits/string_fortified.h:34 #2 write_block_record (info=info@entry=0x7f0aa8635b18, old_record=old_record@entry=0x0, record=record@entry=0x7f0aa886ddd0 "347377003ngs", row=row@entry=0x7f0aa8635bb0, bitmap_blocks=bitmap_blocks@entry=0x7f0aa8635bb0, head_block_is_read=<optimized out>, row_pos=0x7f0b74edbb30, undo_lsn=1, old_record_checksum=0) at /usr/src/debug/MariaDB-10.4.17-10/ src_0/storage/maria/ma_blockrec.c:2750
#3 0x0000563bd293c9f7 in allocate_and_write_block_record (undo_lsn=1, row=0x7f0aa8635bb0, record=0x7f0aa886ddd0 "347377003ngs", info=0x7f0aa8635b18) at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/ ma_blockrec.c:3571 #4 _ma_write_init_block_record (info=0x7f0aa8635b18, record=0x7f0aa886ddd0 "347377003ngs") at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/ ma_blockrec.c:3611 #5 0x0000563bd2947a81 in maria_write (info=0x7f0aa8635b18, record=0x7f0aa886ddd0 "347377003ngs") at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/ ma_write.c:157 #6 0x0000563bd237b631 in handler::ha_write_tmp_row (this=0x7f0aa8800e00, buf=0x7f0aa886ddd0 "347377003ngs") at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_class.h:6645 #7 0x0000563bd2386134 in schema_table_store_record (thd=0x7f0aa87efb18, table=0x7f0aa8268550) at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:3929 #8 0x0000563bd238ad2e in get_schema_column_record(THD*, TABLE_LIST*, TABLE*, bool, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:6113
#9 0x0000563bd2391e04 in get_all_tables(THD*, TABLE_LIST*, Item*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:5066 #10 0x0000563bd23930e4 in get_schema_tables_result(JOIN*, enum_schema_table_state) () at /usr/src/debug/MariaDB-10.4.17-10/ src_0/sql/sql_show.cc:8919 #11 0x0000563bd2378bb7 in JOIN::exec_inner() () at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_select.cc:4438 #12 0x0000563bd2379113 in JOIN::exec (this=0x7f0aa81f9ee8) at /usr/ src/debug/MariaDB-10.4.17-10/src_0/sql/sql_select.cc:4264 #13 0x0000563bd2377541 in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/ sql_select.cc:4699 #14 0x0000563bd2377e1e in handle_select (thd=0x7f0aa87efb18, lex=0x7f0aa87f37d0, result=0x7f0aa81f9ec0, setup_tables_done_option=0) at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/ sql_select.cc:410 #15 0x0000563bd230f41c in execute_sqlcom_select(THD*, TABLE_LIST*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:6429 #16 0x0000563bd23177b3 in mysql_execute_command(THD*) () at /usr/ src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:3926
#17 0x0000563bd231e74a in mysql_parse (thd=0x7f0aa87efb18, rawbuf=<optimized out>, length=111, parser_state=0x7f0b74ee1310, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:7967 #18 0x0000563bd2320bf1 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_class.h:1179 #19 0x0000563bd23221f4 in do_command (thd=0x7f0aa87efb18) at /usr/ src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:1357 #20 0x0000563bd2407973 in do_handle_one_connection (connect=connect@entry=0x563c11465fe8) at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_connect.cc:1412 #21 0x0000563bd2407a6d in handle_one_connection (arg=arg@entry=0x563c11465fe8) at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_connect.cc:1316 #22 0x0000563bd29f430b in pfs_spawn_thread (arg=0x563c1175fcd8) at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/perfschema/ pfs.cc:1869 #23 0x00007f190a78c14a in start_thread () from /lib64/ libpthread.so.0 #24 0x00007f19085eff23 in clone () from /lib64/libc.so.6
(gdb) frame 17 — this is to know more details in that frame(line) ^^ #17 0x0000563bd231e74a in mysql_parse (thd=0x7f0aa87efb18, rawbuf=<optimized out>, length=111, parser_state=0x7f0b74ee1310, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:7967 7967 error= mysql_execute_command(thd); (gdb) print thd->query_string $1 = {string = {str = 0x7f0aa81f6520 "SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.Columns where TABLE_NAME = 'my_workflow' and TABLE_SCHEMA='news'", length = 111}, cs = 0x563bd342ad20 <my_charset_latin1>} (gdb)
Some important links for the blog posts by my friend and colleague Valerii Kravchuk GDB basics for MySQL DB https://www.slideshare.net/valeriikravchuk1/gdb-basics-for-my-sql-db-as-percona-live-europe-2019 Checking user threads with GDB http://mysqlentomologist.blogspot.com/2021/01/checking-user-threads-and-temporary.html http://mysqlentomologist.blogspot.com/2018/03/checking-user-threads-with-gdb-in-mysql.html http://mysqlentomologist.blogspot.com/2017/07/how-to-find-processlist-thread-id-in-gdb.html How to explore InnoDB locks with GDB http://mysqlentomologist.blogspot.com/2015/03/using-gdb-to-understand-what-locks-and_31.html http://mysqlentomologist.blogspot.com/2015/04/using-gdb-to-understand-what-locks-and.html
Thank you Any Questions?

Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilandan Joshi (MariaDB)

  • 1.
    Analyze corefile andget backtraces with GDB for MySQL/MariaDB on Linux Nilnandan Joshi, Sr. Support Engineer, MariaDB nilnandan@gmail.com www.nilinfobin.com
  • 2.
    Core Dump/File: A coredump is a disk file containing an image of the process’s memory at the time of termination. This image can be used in a debugger to inspect the state of the program at the time that it terminated. The core dump includes key pieces of program state as processor registers, memory management details, and other processor and operating system flags and information.So a core dump is a file that can be very useful to understand the context of a crash.  GDB: GDB, the GNU Project debugger, allows you to see what is going on `inside' another program while it executes -- or what another program was doing at the moment it crashed. i.e sudo gdb -p `pidof mysqld` sudo gdb /path/to/mysqld /path/to/coredump sudo gdb --batch --eval-command="thread apply all bt" /usr/sbin/mysqld $(pgrep -xn mysqld) > mysqld_bt_all_threads.txt sudo gdb --batch --eval-command="thread apply all bt full" /usr/sbin/mysqld $(pgrep -xn mysqld) > mysqld_full_bt_all_threads.txt You can use GDB to study: • InnoDB locks, table locks, metadata locks etc • server variables, user variables at session level • Threads and Statements
  • 3.
    Backtraces: A backtrace isa summary of how your program got where it is. It shows one line per frame, for many frames, starting with the currently executing frame (frame zero), followed by its caller (frame one), and on up the stack. In a multi-threaded program, GDB by default shows the backtrace only for the current thread. To display the backtrace for several or all of the threads, use the command thread apply. For example, if you type thread apply all backtrace(thread apply all bt) , GDB will display the backtrace for all the threads; this is handy when you debug a core dump of a multi-threaded program. Each line in the backtrace shows the frame number and the function name. The program counter value is also shown—unless you use set print address off. The backtrace also shows the source file name and line number, as well as the arguments to the function. The program counter value is omitted if it is at the beginning of the code for that line number.
  • 4.
    How to enableCoredump: In order to enable core dumps, you need to set the core_file system variable either on the command-line or in a relevant server option group in an option file. [mariadb] ... core_file There are some additional details related to using core files on Linux. On some systems there is a limit on the sizes of core files that can be dumped. So you have to disable some of the restrictions. To check the system's current system-wide limit, run ulimit -c for the current limit of the mysqld process specifically: sudo cat /proc/$(pidof mysqld)/limits | grep "core file" If you need to change the core size limit, the method you use depends on how you start mysqld. i.e mysqld using mysqld_safe [mysqld_safe] … core_file_size=unlimited
  • 5.
    - mysqld usingsystemd sudo tee /etc/systemd/system/mariadb.service.d/limitcore.conf <<EOF [Service] LimitCORE=infinity EOF sudo systemctl daemon-reload If you are using Linux, then it can be helpful to change a few settings to alter where the core files is written and what file name is used. This is done by setting the kernel.core_pattern and kernel.core_uses_pid attributes. By default, core file will be generated in MySQL/MariaDB datadir. If you are using a production system, you probably want to have the core files in a specific directory, not in the data directory. (Because core file size could be very large somewhere in GBs) so for that you can set something like, sudo mkdir /tmp/corefiles sudo chmod 777 /tmp/corefiles sudo echo /tmp/corefiles/core > /proc/sys/kernel/core_pattern sudo echo 1 > /proc/sys/kernel/core_uses_pid Since  mysqld  executes  setuid, you may have to set  fs.suid_dumpable=2  to allow core dumps on Linux.  sudo echo 2 > /proc/sys/fs/suid_dumpable
  • 6.
    In MariaDB 10.1.35,MariaDB 10.2.17, and MariaDB 10.3.9 and later, core_file has also been made into a system variable. Previously it was just an option. It's value can be checked at runtime by executing the following: SHOW GLOBAL VARIABLES LIKE 'core_file'; However, in MariaDB 10.3.7 and later, some large buffers have been excluded from core files on some systems as a way to reduce the size. The following buffers are excluded: InnoDB buffer pool InnoDB log buffer Server recv buffer Query cache https://mariadb.com/kb/en/enabling-core-dumps/ https://dev.mysql.com/doc/refman/8.0/en/server-options.html#option_mysqld_core-file https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
  • 7.
    Other Requirements: 1. GDBneed to be installed if it's not already yum install gdb 2. debug info packages needs to be install for server and common atleast. i.e MariaDB-server-debuginfo-10.4.17_10-1.el8.x86_64 MariaDB-common-debuginfo-10.4.17_10-1.el8.x86_64 3. If you can’t run gdb + core dump on customer’s server then you must have the same mysqld/mariadb binaries including debug-info packages + all dynamically linked libraries + core file to run gdb in your local machine. Libraries can be checked by ldd command. i.e [root@nilcentos7 ~]# ldd /usr/sbin/mysqld linux-vdso.so.1 => (0x00007ffea8ddd000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2149b93000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f214995c000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f2149736000) … libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f2148810000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f214860c000)
  • 8.
    Problem: 190328 23:31:08 [ERROR]mysqld got signal 6 ; ... Server version: 10.2.23-MariaDB-10.2.23+maria~stretch ... Thread pointer: 0x7ff4d8001f28 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff4dc62ccc8 thread_stack 0x49000 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7ffa09af5bfb] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffa09b7e437] /lib/x86_64-linux-gnu/libc.so.6(+0xf7570)[0x7ffa09b7c570] /lib/x86_64-linux-gnu/libc.so.6(+0xf93aa)[0x7ffa09b7e3aa] /usr/sbin/mysqld(my_addr_resolve+0xe2)[0x55ca42284922] … /usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x72) [0x55ca41b4b992] /usr/sbin/ mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15 enum_duplicatesb+0x1206)[0x55ca41b560f6] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3f68)[0x55ca41b6bee8] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x28a) [0x55ca41b70e4a] /usr/sbin/mysqld(+0x4c864f)[0x55ca41b7164f] ...
  • 9.
    Problem: 2021-02-12 8:39:33 11039259[ERROR] mysqld: Table '(temporary)' is marked as crashed and should be repaired 2021-02-12 8:41:09 10757048 [ERROR] mysqld: Table '(temporary)' is marked as crashed and should be repaired free(): invalid next size (normal) 210212 8:41:09 [ERROR] mysqld got signal 6 ; ... Server version: 10.4.17-10-MariaDB-enterprise-log ... Thread pointer: 0x7f179d1103c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0a020edbe8 thread_stack 0x49000
  • 10.
    [root@centos8 nil]# gdbmysqld core.2751610 GNU gdb (GDB) Red Hat Enterprise Linux 8.2-12.el8 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/ gpl.html> … Type "apropos word" to search for commands related to "word"... Reading symbols from mysqld...Reading symbols from /usr/lib/debug/ usr/sbin/mysqld-10.4.17_10-1.el8.x86_64.debug...done. done. [New LWP 2338279] [New LWP 2751610] … … Use the "info sharedlibrary" command to see the complete listing. Do you need "set solib-search-path" or "set sysroot"? [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". … Core was generated by `/usr/sbin/mysqld'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from / lib64/libc.so.6 [Current thread is 1 (Thread 0x7f0b74ee2700 (LWP 2338279))]
  • 11.
    (gdb) (gdb) thread 1— this is for the crash [Switching to thread 1 (Thread 0x7f0b74ee2700 (LWP 2338279))] #0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from / lib64/libc.so.6 (gdb) backtrace — to get the backtrace #0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from / lib64/libc.so.6 #1 0x0000563bd293aa84 in memcpy (__len=<optimized out>, __src=<optimized out>, __dest=<optimized out>) at /usr/include/ bits/string_fortified.h:34 #2 write_block_record (info=info@entry=0x7f0aa8635b18, old_record=old_record@entry=0x0, record=record@entry=0x7f0aa886ddd0 "347377003ngs", row=row@entry=0x7f0aa8635bb0, bitmap_blocks=bitmap_blocks@entry=0x7f0aa8635bb0, head_block_is_read=<optimized out>, row_pos=0x7f0b74edbb30, undo_lsn=1, old_record_checksum=0) at /usr/src/debug/MariaDB-10.4.17-10/ src_0/storage/maria/ma_blockrec.c:2750
  • 12.
    #3 0x0000563bd293c9f7 inallocate_and_write_block_record (undo_lsn=1, row=0x7f0aa8635bb0, record=0x7f0aa886ddd0 "347377003ngs", info=0x7f0aa8635b18) at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/ ma_blockrec.c:3571 #4 _ma_write_init_block_record (info=0x7f0aa8635b18, record=0x7f0aa886ddd0 "347377003ngs") at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/ ma_blockrec.c:3611 #5 0x0000563bd2947a81 in maria_write (info=0x7f0aa8635b18, record=0x7f0aa886ddd0 "347377003ngs") at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/ ma_write.c:157 #6 0x0000563bd237b631 in handler::ha_write_tmp_row (this=0x7f0aa8800e00, buf=0x7f0aa886ddd0 "347377003ngs") at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_class.h:6645 #7 0x0000563bd2386134 in schema_table_store_record (thd=0x7f0aa87efb18, table=0x7f0aa8268550) at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:3929 #8 0x0000563bd238ad2e in get_schema_column_record(THD*, TABLE_LIST*, TABLE*, bool, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:6113
  • 13.
    #9 0x0000563bd2391e04 inget_all_tables(THD*, TABLE_LIST*, Item*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:5066 #10 0x0000563bd23930e4 in get_schema_tables_result(JOIN*, enum_schema_table_state) () at /usr/src/debug/MariaDB-10.4.17-10/ src_0/sql/sql_show.cc:8919 #11 0x0000563bd2378bb7 in JOIN::exec_inner() () at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_select.cc:4438 #12 0x0000563bd2379113 in JOIN::exec (this=0x7f0aa81f9ee8) at /usr/ src/debug/MariaDB-10.4.17-10/src_0/sql/sql_select.cc:4264 #13 0x0000563bd2377541 in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/ sql_select.cc:4699 #14 0x0000563bd2377e1e in handle_select (thd=0x7f0aa87efb18, lex=0x7f0aa87f37d0, result=0x7f0aa81f9ec0, setup_tables_done_option=0) at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/ sql_select.cc:410 #15 0x0000563bd230f41c in execute_sqlcom_select(THD*, TABLE_LIST*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:6429 #16 0x0000563bd23177b3 in mysql_execute_command(THD*) () at /usr/ src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:3926
  • 14.
    #17 0x0000563bd231e74a inmysql_parse (thd=0x7f0aa87efb18, rawbuf=<optimized out>, length=111, parser_state=0x7f0b74ee1310, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:7967 #18 0x0000563bd2320bf1 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_class.h:1179 #19 0x0000563bd23221f4 in do_command (thd=0x7f0aa87efb18) at /usr/ src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:1357 #20 0x0000563bd2407973 in do_handle_one_connection (connect=connect@entry=0x563c11465fe8) at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_connect.cc:1412 #21 0x0000563bd2407a6d in handle_one_connection (arg=arg@entry=0x563c11465fe8) at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_connect.cc:1316 #22 0x0000563bd29f430b in pfs_spawn_thread (arg=0x563c1175fcd8) at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/perfschema/ pfs.cc:1869 #23 0x00007f190a78c14a in start_thread () from /lib64/ libpthread.so.0 #24 0x00007f19085eff23 in clone () from /lib64/libc.so.6
  • 15.
    (gdb) frame 17— this is to know more details in that frame(line) ^^ #17 0x0000563bd231e74a in mysql_parse (thd=0x7f0aa87efb18, rawbuf=<optimized out>, length=111, parser_state=0x7f0b74ee1310, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:7967 7967 error= mysql_execute_command(thd); (gdb) print thd->query_string $1 = {string = {str = 0x7f0aa81f6520 "SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.Columns where TABLE_NAME = 'my_workflow' and TABLE_SCHEMA='news'", length = 111}, cs = 0x563bd342ad20 <my_charset_latin1>} (gdb)
  • 16.
    Some important linksfor the blog posts by my friend and colleague Valerii Kravchuk GDB basics for MySQL DB https://www.slideshare.net/valeriikravchuk1/gdb-basics-for-my-sql-db-as-percona-live-europe-2019 Checking user threads with GDB http://mysqlentomologist.blogspot.com/2021/01/checking-user-threads-and-temporary.html http://mysqlentomologist.blogspot.com/2018/03/checking-user-threads-with-gdb-in-mysql.html http://mysqlentomologist.blogspot.com/2017/07/how-to-find-processlist-thread-id-in-gdb.html How to explore InnoDB locks with GDB http://mysqlentomologist.blogspot.com/2015/03/using-gdb-to-understand-what-locks-and_31.html http://mysqlentomologist.blogspot.com/2015/04/using-gdb-to-understand-what-locks-and.html
  • 17.