NPTL Trace Tool



User Manual


Trace points
Data collecting
Glibc patch



Sourceforge project info




PTT must obey to many constraints:

  • At every trace point reached during the program execution, a call to the trace mechanism is done. As we must avoid to block or slow down the traced function, this call to the trace mechanism must be as fast as possible by using quick and simple instructions.
  • The trace mechanism records trace points data in a binary file. We are constrained to put this data in a buffer first to avoid system calls. Indeed, system calls would interrupt the thread execution and so would disturb the application dynamic.
  • When there are many processors, threads are concurrent and unforeseeable. PTT must synchronize threads action not to lose traces.
  • The NPTL respects some standards like POSIX. The instrumented NPTL used with PTT must also respect these standards (in particular about cancellation points).

Buffer accesses

All threads involved in an application write trace points data in the same buffer. At the same time, an external process (daemon) regularly accesses this buffer to read it and copy data to disk (in a binary file).

To manage the access to the buffer, the following variables are shared between the different threads and the daemon:

  • reserved: indicate space reserved for writting.
  • written: indicate the end of the written area.
  • beg_read: index where the daemon starts to read.
  • PAUSE: to synchronise threads and the daemon when the buffer is emptied.

The following algorithm is used:

trace function called by trace points:
    Get the size of the trace point to write
    if there is not enough place in the array to write trace data
        wait for place

    if (PAUSE == TRUE)
        wait for the daemon to authorize to continue (PAUSE == FALSE)

    index where the writting will begin = reserved

    reserve space in the array for the trace point:
    reserved = reserved + size of (all data to be writen)

    write each data in the space reserved

    increment the variable written: the writing is finished
    written = written + size of (all written data)

    open the binary file
    write header information

    beg_read = 0

    launch the program to traced

    while its launched program (child process) is not dead

        if "reserved reaches the middle of the array"

            PAUSE = 1: block threads to stop new writing

            if there are still data to write
                wait for the end of the writing (written == reserved)

            end_read = written: set the index where the reading will stop

            PAUSE = 0: unblock threads. writting can continue 
            copy data between beg_read and end_read in the binary file

            the reading is finished:
            beg_read = end_read: set the beginning of the next one


    if there are still data to write
        wait for the end of the writting (written == reserved))

    end_read = writen: set the index where the reading will stop

    copy data between beg_read and end_read in the binary file

    close the binary file

Data written for a trace point are:

  • the trace point ID (one word)
  • the timestamp (two words)
  • the process pid (one word)
  • the thread pid (one word)
  • the trace point arguments
  • a checksum


This example helps to understand the trace mechanism. It explains how the collect mechanism runs. Let's consider this user program with three trace points. Each call to the trace mechanism is done by a different thread.

user_function(...) {
    TRACE (TR_BARRIER_1, &th1, &th1, &bar1, lo1)
    TRACE (TR_THREAD_2, &th2)
    TRACE (TR_BARRIER_9, &th3, ret)

click to see the image

click to see the image


Page hosted by : Logo