PDA

View Full Version : Source code for loaded dynamic libraries



kenkahn
08-10-2006, 04:59 AM
I'm running TotalView 7.3.0.-2 (demo version) on Solaris 8. I'm debugging a program that dynamically loads some shared libraries. TV properly detects the loading and asks me if I want to stop to set breakpoints, to which I respond yes.

The problem is I can't figure out how to get TV to show me the shared library's source code; I can search and find the entry point, but it shows up in assembly. I'm positive that the library has been built with debug (-g) enabled. Examining core dumps generated by them via GDB shows the debug info.

I tried to load the source by hand (File->Open_Source) and set the search path (File->Search_Path) to no avail.

How do I get TV to show me the loaded library's source code?

Kerneth Kahn
Cadence Design Systems
kenkahn@cadence.com

Josh-TotalView-Tech
08-10-2006, 08:19 AM
What do you mean when you say you search and find the entry point? When your program dlopen's a library TotalView detects this and prompts you asking if you want to stop the process or not. I can see at this point you answered yes. What you see at this point in TotalView's source pane is the location where TotalView stopped the process. This will be inside the dynamic loader; unless you have installed a debug version of the loader (not typical) TotalView will show you assembler.

At this point what you will likely want to do is lookup a function or filename which you should be able to do by going to View -> Lookup Function. Is this what you mean when you said you searched and found the entry point?

Please, note that TotalView 7.3 does not support Solaris 8. Specifically TotalView 7.3 is not capable of debugging threaded programs on Solaris 8. This was a consequence of implementing asynchrounous thread control for Solaris 9 and above. TotalView 7.2 is the last version that supports Solaris 7 and 8. Despite this I do not expect the use of TotalView 7.3 Solaris 8 to be the cause of the problem you are encountering.

Josh Carlson
Etnus Customer Services

PeterT-RogueWave
08-10-2006, 08:29 AM
Hello Ken,

I know this issue is being worked on with the support group, but the question gets raised often enough and I thought I'd throw in a few pointers.

Obviously we need the source files to be compiled with -g, otherwise we don't have any information about where to find it. Some compilers give full path name data, in which case we should always be able to find the source, unless those files are moved. Some compilers give relative path data, in which case if the executable is moved, we lose the relative path association, and then can't find the source. File->Search Path is one way around this, but be aware that the paths specify only the directories to be searched. If you have a source tree with many sub-directories, TotalView does not search those sub-directories and you must enter those also. We do have a tcl script that can help with that and I'll include it at the end. Since this is a shared library, you should also check that the library build does not strip debug information during the link phase (the -x option for most Unix/Linux linkers).

Another possibility, especially for C++ code and F90 code to some extent, is that TotalView is not de-mangling the name of the function you are stopped in correctly. In the TotalView window, the name of the function and the source file it is from should be shown in the border of the source window. If the name is still in a mangled format, then we likely are unable to make the connection between the function and the correct source file. So we fall back on assembler. TotalView has a -demangler=<demangler-name> option, and there are a number of choices you can find in our Reference Guide. We try to pick the right one, but we might get it wrong at times.

Another option, especially with shared libraries, is to set the environment variable, LD_BIND_NOW to 1. References into the shared library are resolved at run time, and until they are, TotalView may not have a full handle on where to find the source. LD_BIND_NOW should help there, but I can't vouch that will work with dlopened libraries. It shouldn't hurt though.

Also, be sure to check the stack trace when you are stopped. This more often happens when you randomly hit halt while the code is running, but you may find you are in a system library (to which we have no source) rather than your code. Clicking on the frame where your source is located will bring the focus to back to that routine.

Those are the main reasons and a hint at how to resolve some of them. Following is the tcl script, rpath.tcl. You can include this in a .tvdrc file, and, when you start TotalView, open a command line window (CLI) with Tools->Command Line. In the window that opens up, enter the command

rpath <top level source directory>

and rpath will walk the source tree and add all the sub-directories to the Search Path. If this is the reason for the failure to see source, and you are at the point where you only see assembler, the source will automagically appear. Hopefully.

rpath.tcl:

proc rpath {{root "."} {filter "/(CVS|RCS|SCCS)(/|$)"}} {

# Invoke the UNIX find command to recursively obtain a
# list of all directory names below "root".
set find [split [exec find $root -type d -print] \n]

set npath ""

# Filter out unwanted directories.
foreach path $find {
if {! [regexp $filter $path]} {
append npath ":"
append npath $path
}
}

# Tell TotalView to use it.
dset EXECUTABLE_PATH $npath
}

kenkahn
08-10-2006, 10:24 AM
>At this point what you will likely want to do is lookup a function or filename which you should be able to
>do by going to View -> Lookup Function. Is this what you mean when you said you searched and found
>the entry point?

Yes, that's what I did. For example I have a SL named et3lib with an external (extern "C") entry point of et3lib. After it's loaded and I search for the entry point I get in the source pane:

et3confg: 0x03000004 sethi %hi(0x1000),%g1
0xfee617ec: 0x82187c80 xor %g1,-0x380,%g1
0xfee617f0: 0x9de38001 save %sp,%g1,%sp
0xfee617f4: 0x40000002 call 0xfee617fc

I highlighted the line "et3confg" and selected runto, which it did, but it never changes to source code.

>where TotalView stopped the process. This will be inside the dynamic loader;

I'm aware of this from previous experience with other debuggers like DDD.

>we need the source files to be compiled with -g

I verified that it is

>Some compilers give full path name data

I'm using Solaris Forte (CC) v8. From previous experience all debug source code contains full path names.

>you should also check that the library build does not strip debug information during the link phase

Nope..

>is that TotalView is not de-mangling the name of the function you are stopped in correctly.

I don't believe this is the problem; all entry points are defined as extern "C" so that shouldn't be a problem; and your search function does find it.

>LD_BIND_NOW to 1

Tried it with same results.

Josh-TotalView-Tech
08-10-2006, 11:39 AM
et3confg: 0x03000004 sethi %hi(0x1000),%g1
0xfee617ec: 0x82187c80 xor %g1,-0x380,%g1
0xfee617f0: 0x9de38001 save %sp,%g1,%sp
0xfee617f4: 0x40000002 call 0xfee617fc

If the compilation unit contains debugging information (or more precisely source and linenumber information) and this is a problem finding the sources you should see the source file name and function name on the bar immediately above the source pane and you should see line numbers displayed w/in the assembler pane. Do you see either of these two things? Perhaps a screen shot after you have "Run to" et3confg would help.

mwolfe
08-10-2006, 03:34 PM
IIRC, on Solaris, some of the debugging info, specifically some directory
hints, are kept in the .o files, and the executable refers to them.

Are the .so's .o file directories on the search path in TV as well as the directories
for the sources?

kenkahn
08-10-2006, 05:41 PM
*SIGH* I'm going to have to retract this thread. I rebuild the libraries myself (I had been using the 'product' versions which were supposed to be built with -g) and the source code is now showing up properly for loaded SL's.

I checked our build process and, sure enough, the -g option is missing.

Sorry for the false alarm. I'll have to have a talk with our build person.