LLDB-MI from Eclipse hangs

Subprojects LLDB
1

Hi All,

I am trying to simple “Hello World” program remotely on a MAC OS using eclipse and lldb-mi. I followed the instructions available here: “https://www.codeplay.com/portal/lldb-mi-driver—part-2-setting-up-the-driver”. I am facing below issue. Any suggestions would be of great help.

It is simple helloWorld C++ program. he program is running on machine A and I am debugging it from machine B. When I remote debug from command prompt using lldb-mi, I am able to debug it properly. But when I try to do it via Eclipse from machine B:

  • lldb-mi is tarting

  • netstat on machine A shows that “debugserver" has ESTABLISHED connection with machine B.

  • But on Eclipse console it hangs after below command:

461,432 (gdb)

461,432 22^done,threads=[{id=“1”,target-id=“Thread 1”,state=“stopped”}]

461,432 (gdb)

461,434 24-stack-info-depth --thread 1 11

If you have any suggestions please let me know. Thanks.

2

You should sample lldb-mi to see what it is doing. Also if you can get a more complete packet log of the traffic between Eclipse and lldb-mi we might be able to see what is going on. Another thing to note: if you are remote debugging to a Mac from another machine and don't have any files cached (all shared libraries from the remote machine), they will be read from memory when the DynamicLoaderMachOSX figures out which shared libraries the process has. This might be slow and take quite a while and it might delay your debugging as you download a few hundred shared libraries for the first time. So you might see long delays as you debug. You might try and wait for a few minutes to see if the process stops correctly after reading everything from memory.

Greg Clayton

3

Thanks Greg for your reply.

Attached below is the GDB trace, please let me know if it helps in any ways.

It would be helpful if you can tell me on how to capture (I am really new to lldb, sorry for bothering you):

  • sample lldb-mi to see what it is doing.
  • complete packet log of the traffic between Eclipse and lldb-mi

I am trying with simple “Hello world” c++ program, so libraries should to be a problem. Also, same remote debugging is working fine if I manually do it using lldb-mi from command prompt. The hang is happening only when I try to do it via eclipse.

I am trying via eclipse, because I am looking for some IDE based remote debugging solution. And Xcode 5 does not support it. Please let me know if you have any other suggestion.

4

I don’t see anything in the log that looks like a problem. I’ve got 2 thoughts:

  1. Have you tried manual debugging with the same sequence that eclipse uses? You can copy/paste lines from the log, just remove some extraneous logging info, so

313,464 14-file-exec-and-symbols --thread-group i1 "/Users/admin/Documents/workspace/Hello World C++\

Project/Debug/Hello World C++ Project"

Becomes

-file-exec-and-symbols --thread-group i1 “/Users/admin/Documents/workspace/Hello World C++ Project/Debug/Hello World C++ Project”

Copy/paste the exact sequence.

  1. A packet log after the –list-thread-groups that hangs would be useful. After you attach in eclipse, but before you continue, go to the GDB console tab and type:

log enable gdb-remote packets

You should get a line in the log that says

-interpreter-exec console “log enable gdb-remote packets”

and then get a giant spew of data after the continue. Please send that data.

I worked on getting lldb-mi working with Eclipse for Hexagon; there were times when I’d have to copy/paste what Eclipse sent over to replicate problems, and bisect those commands to find the combination that caused my bad behavior. You may need to do that here.

Ted

5

It looks like we continue and then ask fore thread groups? I am not sure on the rules of MI. Can you ask another question before receiving a response? If we say command 18 is "-exec-continue --thread-group i1", can you send command 19 without receiving a response?

314,372 18-exec-continue --thread-group i1
319,380 19-list-thread-groups

I would sample the lldb-mi when it is deadlocked:

% sample lldb-mi

Then attach the sample output to your response.

6

Actually you should have seen a ^running as a response from -exec-continue:

-exec-continue
^running
(gdb)

But we don't see that here.

Also, -exec-continue doesn't believe it takes any arguments in the lldb-mi. Check the tools/lldb-mi/MICmdCmdExec.cpp source file in the LLDB sources. Note that is says "Args: None.". Not sure what lldb-mi does if arguments are passed to a command that doesn't believe it takes any arguments...

//++ ------------------------------------------------------------------------------------
// Details: CMICmdCmdExecContinue constructor.
// Type: Method.
// Args: None.
// Return: None.
// Throws: None.
//--
CMICmdCmdExecContinue::CMICmdCmdExecContinue()
{
    // Command factory matches this name with that received from the stdin stream
    m_strMiCmd = "exec-continue";

    // Required by the CMICmdFactory when registering *this command
    m_pSelfCreatorFn = &CMICmdCmdExecContinue::CreateSelf;
}

//++ ------------------------------------------------------------------------------------
// Details: CMICmdCmdExecContinue destructor.
// Type: Overrideable.
// Args: None.
// Return: None.
// Throws: None.
//--
CMICmdCmdExecContinue::~CMICmdCmdExecContinue()
{
}

//++ ------------------------------------------------------------------------------------
// Details: The invoker requires this function. The command does work in this function.
// The command is likely to communicate with the LLDB SBDebugger in here.
// Type: Overridden.
// Args: None.
// Return: MIstatus::success - Functional succeeded.
// MIstatus::failure - Functional failed.
// Throws: None.
//--
bool
CMICmdCmdExecContinue::Execute()
{
    const char *pCmd = "continue";
    CMICmnLLDBDebugSessionInfo &rSessionInfo(CMICmnLLDBDebugSessionInfo::Instance());
    const lldb::ReturnStatus rtn = rSessionInfo.GetDebugger().GetCommandInterpreter().HandleCommand(pCmd, m_lldbResult);
    MIunused(rtn);

    if (m_lldbResult.GetErrorSize() == 0)
    {
        // CODETAG_DEBUG_SESSION_RUNNING_PROG_RECEIVED_SIGINT_PAUSE_PROGRAM
        if (!CMIDriver::Instance().SetDriverStateRunningDebugging())
        {
            const CMIUtilString &rErrMsg(CMIDriver::Instance().GetErrorDescription());
            SetError(CMIUtilString::Format(MIRSRC(IDS_CMD_ERR_SET_NEW_DRIVER_STATE), m_cmdData.strMiCmd.c_str(), rErrMsg.c_str()));
            return MIstatus::failure;
        }
    }
    else
    {
        // ToDo: Re-evaluate if this is required when application near finished as this is parsing LLDB error message
        // which seems a hack and is code brittle
        const char *pLldbErr = m_lldbResult.GetError();
        const CMIUtilString strLldbMsg(CMIUtilString(pLldbErr).StripCREndOfLine());
        if (strLldbMsg == "error: Process must be launched.")
        {
            CMIDriver::Instance().SetExitApplicationFlag(true);
        }
    }

    return MIstatus::success;
}

//++ ------------------------------------------------------------------------------------
// Details: The invoker requires this function. The command prepares a MI Record Result
// for the work carried out in the Execute().
// Type: Overridden.
// Args: None.
// Return: MIstatus::success - Functional succeeded.
// MIstatus::failure - Functional failed.
// Throws: None.
//--
bool
CMICmdCmdExecContinue::Acknowledge()
{
    if (m_lldbResult.GetErrorSize() > 0)
    {
        const CMICmnMIValueConst miValueConst(m_lldbResult.GetError());
        const CMICmnMIValueResult miValueResult("message", miValueConst);
        const CMICmnMIResultRecord miRecordResult(m_cmdData.strMiCmdToken, CMICmnMIResultRecord::eResultClass_Error, miValueResult);
        m_miResultRecord = miRecordResult;
    }
    else
    {
        const CMICmnMIResultRecord miRecordResult(m_cmdData.strMiCmdToken, CMICmnMIResultRecord::eResultClass_Running);
        m_miResultRecord = miRecordResult;
    }

    return MIstatus::success;
}

//++ ------------------------------------------------------------------------------------
// Details: Required by the CMICmdFactory when registering *this command. The factory
// calls this function to create an instance of *this command.
// Type: Static method.
// Args: None.
// Return: CMICmdBase * - Pointer to a new command.
// Throws: None.
//--
CMICmdBase *
CMICmdCmdExecContinue::CreateSelf()
{
    return new CMICmdCmdExecContinue();
}

7

Thanks Greg. Attached is the file after sampling lldb-mi. Please let me know if you can find some clue from it.

sample-lldb-mi.rtf (41.7 KB)

8

Thanks Ted for your reply.
I tried the same command that eclipse uses manually from command prompt. It hangs after same last command “-exec-continue --thread-group i1”. Attached is the lldb-mi sample at that time.

While debugging from Eclipse I am going to Run–> Debug Configurations–> C/C++ Remote Applications–> and then click on Debug button. And then all the commands are executed by Eclipse automatically. I do not see two different steps as attach and continue. So I am not sure how to pass command “log enable gdb-remote packets”. May be I am missing something. Can you please help me here, so that I can get the required log as suggested by you? Thanks.

lldb-mi_2016-07-08_164009_wdTK.sample.txt (39.4 KB)

9

Thanks Greg. But is it possible that for eclipse support some new wrapper command is added ? [https://www.codeplay.com/portal/lldb-mi-driver—part-4-how-to-add-a-command] Not sure, just a thought. I am googling to see more details but not much luck yet.

lldb-mi_2016-07-08_164009_wdTK.sample.txt (39.4 KB)

10

A few thoughts:

  1. I think lldb-mi now takes lldb commands, so you could do the log in your manual copy/paste. The command you want is:

log enable gdb-remote packets

Do it after the stop, before the –break-insert. You’ll get a bunch of data after the –break-insert, then do the –exec-continue and –list-thread-groups, and send the data that comes out after those.

  1. I see from the log that you’re doing a –target-select remote. Are you running debugserver manually? What happens if you don’t run debugserver, and have Eclipse do –exec-run instead of –target-select remote?

The original Codeplay implementation of lldb-mi talked to the Hexagon Simulator, but didn’t have a way to launch it, so they only used –target-select remote, and not –exec-run. Debugserver can be launched automatically, so –exec-run should work.

  1. This is an answer to Greg’s question about –exec-continue and –list-thread-groups. I was going to ask the same thing, but I looked over some old logs I had. Eclipse called –list-thread-groups while the target was running, and it worked fine. This was with a 2 year old lldb-mi, but I’d expect a modern lldb-mi to work as well. The spec at https://sourceware.org/gdb/onlinedocs/gdb/Thread-groups.html talks about what this command does. It gives a list of process groups that the debugger is debugging, so it doesn’t matter if a certain process is running or not.
11

From the sample I see that LLDB is in synchronous mode. It was caused by:

313,449 12-gdb-set target-async off
313,453 12^done

This sets LLDB into synchronous mode where if you say "-exec-continue", that command won't return until the target stops. This causes lldb-mi to not process any commands until your process stops or exits. Not sure that this is what we want. We can see the handler for "target-async" below:

//++ ------------------------------------------------------------------------------------
// Details: Carry out work to complete the GDB set option 'target-async' to prepare
// and send back information asked for.
// Type: Method.
// Args: vrWords - (R) List of additional parameters used by this option.
// Return: MIstatus::success - Function succeeded.
// MIstatus::failure - Function failed.
// Throws: None.
//--
bool
CMICmdCmdGdbSet::OptionFnTargetAsync(const CMIUtilString::VecString_t &vrWords)
{
    bool bAsyncMode = false;
    bool bOk = true;

    if (vrWords.size() > 1)
        // Too many arguments.
        bOk = false;
    else if (vrWords.size() == 0)
        // If no arguments, default is "on".
        bAsyncMode = true;
    else if (CMIUtilString::Compare(vrWords[0], "on"))
        bAsyncMode = true;
    else if (CMIUtilString::Compare(vrWords[0], "off"))
        bAsyncMode = false;
    else
        // Unrecognized argument.
        bOk = false;

    if (!bOk)
    {
        // Report error.
        m_bGbbOptionFnHasError = true;
        m_strGdbOptionFnError = MIRSRC(IDS_CMD_ERR_GDBSET_OPT_TARGETASYNC);
        return MIstatus::failure;
    }

    // Turn async mode on/off.
    CMICmnLLDBDebugSessionInfo &rSessionInfo(CMICmnLLDBDebugSessionInfo::Instance());
    rSessionInfo.GetDebugger().SetAsync(bAsyncMode);

    return MIstatus::success;
}

So we see that it sets the debugger into synchronous mode. I don't know MI well enough to know if this is desired, but seeing that eclipse quickly follows up the -exec-continue with a -list-thread-groups, I am guessing this isn't what we want. I would be interested to see the MI log from a GDB that actually works to see what the packet responses are expected to look like.

Greg

12

Good point Greg! Eclipse turns on synchronous mode in lldb-mi but works as if it’s in async. I faced with that problem year and a half ago and I still don’t know why Eclipse does it. The last time, I forced lldb-mi to stay in async mode and it was enough for me and Eclipse.

As for missing ^running, seems that it just not implemented. We support only asynchronous *running notifications (which aren’t being shown in sync mode).

  • Ilia
13

Thank you all for your reply.
Attached is the GDB log. It looks like synchronous from log :

(gdb)

-exec-continue --thread-group i1

< 5> send packet: $c#63

-list-thread-groups i1

Greg,

Can there be some issue with remote debug server? I am using below command:

cd /Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A/Resources

./debugserver 192.168.116.140:1234 --attach=2775

debugserver-@(#)PROGRAM:debugserver PROJECT:debugserver-350.0.21.9

for x86_64.

Attaching to process 2775…

Listening to port 1234 for a connection from 192.168.116.140…

Waiting for debugger instructions for process 2775.

Ilia,

How did you do this “I forced lldb-mi to stay in async mode and it was enough for me and Eclipse.” ? How do I change the lldb-mi commands executed by eclipse. It would be great if you can help me with this.

GDB_log.rtf (54.1 KB)

14

I’m not very familiar with Eclipse, so if there is a way to change start sequence of MI commands, just remove “gdb-set target-async off” or replace with “gdb-set target-async on”. If not, then rebuild lldb-mi with “disabled” target-async feature:

 $ svn diff tools/lldb-mi/ Index: tools/lldb-mi/MICmdCmdGdbSet.cpp
15

Thanks Ted.
Attached is the GDB log after enabling log while executing same commands manually from command prompt.

Yes, I am running debug server manually. I am facing issues with “GDB (DSF) Automatic Remote Debugging Launcher” as eclipse is not passing correct parameter to debug server. Below is the Eclipse console output with automatic mode:

echo $PWD’>’

/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A/Resources/debugserver :192.168.116.140:1234 /Users/admin/Documents/workspace/Hello\ World\ C++\ Project/Debug/Hello\ World\ C++\ Project;exit

dshiralkar-mactest:~ admin$ echo $PWD’>’

/Users/admin>

dshiralkar-mactest:~ admin$ /Applications/Xcode.app/Contents/SharedFrameworks/LL DB.framework/Versions/A/Resources/debugserver :192.168.116.140:1234 /Users/admi n/Documents/workspace/Hello\ World\ C++\ Project/Debug/Hello\ World\ C++\ Projec t;exit

debugserver-@(#)PROGRAM:debugserver PROJECT:debugserver-350.0.21.9

for x86_64.

Usage:

debugserver host:port [program-name program-arg1 program-arg2 …]

debugserver /path/file [program-name program-arg1 program-arg2 …]

debugserver host:port --attach=

debugserver /path/file --attach=

debugserver host:port --attach=<process_name>

debugserver /path/file --attach=<process_name>

logout

Please let me know your thoughts.

GDB_log.rtf (54.1 KB)

16

Thanks Ted.
Attached is the GDB log after enabling log while executing same commands manually from command prompt.

Is this the GDB log while still running lldb-mi? I would like to see a log of any program that actually runs using "gdb" as the MI program. This will allow us to see a working example. We know it isn't working with LLDB and I was thinking we can get hints of what is expected with an actual GDB doing the MI.

Yes, I am running debug server manually. I am facing issues with "GDB (DSF) Automatic Remote Debugging Launcher" as eclipse is not passing correct parameter to debug server. Below is the Eclipse console output with automatic mode:

echo $PWD'>'
/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A/Resources/debugserver :192.168.116.140:1234 /Users/admin/Documents/workspace/Hello\ World\ C++\ Project/Debug/Hello\ World\ C++\ Project;exit

What is the extra ":" before the address? debugserver doesn't support this syntax and that is why it is causing issues.

17

I wanted a log with Eclipse talking to lldb-mi to see if it was doing anything odd. It's not:

-exec-continue --thread-group i1
< 5> send packet: $c#63
-list-thread-groups i1

We have lldb-mi launch hexagon-sim automatically; it launches the same way debugserver does, with reverse-connect. I'd expect an -exec-run to do this automatically for you. No need to specify parameters to debugserver.

Here is some of a log of Eclipse talking to lldb-mi talking to a Hexagon target. It looks very similar, except for -exec-run instead of -target-select remote and -exec-continue:

TX:21-break-insert -t -f main

RX:=breakpoint-modified,bkpt={number="3",type="breakpoint",disp="keep",enabled="y",addr="0x000000001e002eec",func="test_main_start",file="downscaleBy2.c",fullname="C:\installSDK\Hexagon_SDK\3.0\examples\common\downscaleBy2\src/downscaleBy2.c",line="100",pending=["downscaleBy2.c:100"],times="0",original-location="downscaleBy2.c:100"}
TX:22-list-thread-groups

RX:(gdb)
RX:21^done,bkpt={number="4",type="breakpoint",disp="del",enabled="y",addr="0x000000001e002d5c",func="main",file="downscaleBy2.c",fullname="C:\installSDK\Hexagon_SDK\3.0\examples\common\downscaleBy2\src/downscaleBy2.c",line="72",pending=["main"],times="0",original-location="main"}
RX:(gdb)
TX:23-exec-run

RX:=breakpoint-modified,bkpt={number="4",type="breakpoint",disp="del",enabled="y",addr="0x000000001e002d5c",func="main",file="downscaleBy2.c",fullname="C:\installSDK\Hexagon_SDK\3.0\examples\common\downscaleBy2\src/downscaleBy2.c",line="72",pending=["main"],times="0",original-location="main"}
RX:(gdb)
RX:22^done,groups=[{id="i1",type="process",executable="C:\installSDK\Hexagon_SDK\3.0\examples\common\downscaleBy2\hexagon_Debug_toolv80_v60/downscaleBy2_q"}]
RX:(gdb)
RX:23^running
RX:=thread-group-started,id="i1",pid="1"
RX:(gdb)
RX:=breakpoint-modified,bkpt={number="1",type="breakpoint",disp="keep",enabled="y",addr="0x000000001e002f38",func="test_main_start",file="downscaleBy2.c",fullname="C:\installSDK\Hexagon_SDK\3.0\examples\common\downscaleBy2\src/downscaleBy2.c",line="121",pending=["downscaleBy2.c:110"],times="0",original-location="downscaleBy2.c:110"}
RX:(gdb)
RX:=breakpoint-modified,bkpt={number="2",type="breakpoint",disp="keep",enabled="y",addr="0x000000001e002df0",func="test_main_start",file="downscaleBy2.c",fullname="C:\installSDK\Hexagon_SDK\3.0\examples\common\downscaleBy2\src/downscaleBy2.c",line="96",pending=["downscaleBy2.c:96"],times="0",original-location="downscaleBy2.c:96"}
RX:(gdb)
RX:=breakpoint-modified,bkpt={number="3",type="breakpoint",disp="keep",enabled="y",addr="0x000000001e002eec",func="test_main_start",file="downscaleBy2.c",fullname="C:\installSDK\Hexagon_SDK\3.0\examples\common\downscaleBy2\src/downscaleBy2.c",line="100",pending=["downscaleBy2.c:100"],times="0",original-location="downscaleBy2.c:100"}
RX:(gdb)
RX:=breakpoint-modified,bkpt={number="4",type="breakpoint",disp="del",enabled="y",addr="0x000000001e002d5c",func="main",file="downscaleBy2.c",fullname="C:\installSDK\Hexagon_SDK\3.0\examples\common\downscaleBy2\src/downscaleBy2.c",line="72",pending=["main"],times="1",original-location="main"}
TX:24-list-thread-groups --available

RX:(gdb)
RX:=thread-created,id="1",group-id="i1"
RX:=thread-created,id="2",group-id="i1"
RX:=thread-created,id="3",group-id="i1"
RX:=thread-created,id="4",group-id="i1"
RX:=thread-selected,id="1"
RX:(gdb)
RX:23^running
RX:=thread-group-started,id="i1",pid="1"
RX:(gdb)
RX:=breakpoint-modified,bkpt={number="1",type="breakpoint",disp="keep",enabled="y",addr="0x000000001e002f38",func="test_main_start",file="downscaleBy2.c",fullname="C:\installSDK\Hexagon_SDK\3.0\examples\common\downscaleBy2\src/downscaleBy2.c",line="121",pending=["downscaleBy2.c:110"],times="0",original-location="downscaleBy2.c:110"}
RX:(gdb)
RX:=breakpoint-modified,bkpt={number="2",type="breakpoint",disp="keep",enabled="y",addr="0x000000001e002df0",func="test_main_start",file="downscaleBy2.c",fullname="C:\installSDK\Hexagon_SDK\3.0\examples\common\downscaleBy2\src/downscaleBy2.c",line="96",pending=["downscaleBy2.c:96"],times="0",original-location="downscaleBy2.c:96"}
RX:(gdb)
RX:=breakpoint-modified,bkpt={number="3",type="breakpoint",disp="keep",enabled="y",addr="0x000000001e002eec",func="test_main_start",file="downscaleBy2.c",fullname="C:\installSDK\Hexagon_SDK\3.0\examples\common\downscaleBy2\src/downscaleBy2.c",line="100",pending=["downscaleBy2.c:100"],times="0",original-location="downscaleBy2.c:100"}
RX:(gdb)
RX:=breakpoint-modified,bkpt={number="4",type="breakpoint",disp="del",enabled="y",addr="0x000000001e002d5c",func="main",file="downscaleBy2.c",fullname="C:\installSDK\Hexagon_SDK\3.0\examples\common\downscaleBy2\src/downscaleBy2.c",line="72",pending=["main"],times="1",original-location="main"}
TX:24-list-thread-groups --available

RX:(gdb)
RX:=thread-created,id="1",group-id="i1"
RX:=thread-created,id="2",group-id="i1"
RX:=thread-created,id="3",group-id="i1"
RX:=thread-created,id="4",group-id="i1"
RX:=thread-selected,id="1"
RX:(gdb)

18

Thanks Ilia.

For now I check running Eclipse commands manually by changing to “gdb-set target-async on” and it solves the issue for manual run. I will try to find out how to change these commands in eclipse otherwise I will rebuild lldvmi. I will keep you posted…