The POSIX Thread Trace Toolkit (PTT)
is a library-level trace tool for.
(GNU C library)
(Native POSIX Thread Library
It aims to help users to analyze and debug multi-threaded applications
using the NPTL under Linux systems.
It is distributed under the GNU
Lesser General Public Licence
PTT aims to answer the needs of three kinds of users:
- a developer writting multi-threaded applications
mainly wants to trace calls to and exits
from NPTL routines,
with details about parameters.
He needs to easily switch from the original untraced NPTL
to the traced library, and vice-versa,
without recompiling his application.
- a member of a Linux support team
providing skills to people who write, test or use applications.
He has strong skills about the Linux kernel and glibc,
therefore he needs to see what happens inside NPTL.
He also wants to be able to analyze in his own offices
traces generated at customers site.
- a maintainer of the glibc
wants to analyze the behaviour of NPTL and fix problems,
which is not an easy task.
PTT provides him with traces of internal NPTL events.
A multi-threaded application can be traced without being recompiling.
The trace is analyzed once the application stopped:
it is a post-mortem analysis.
Three different trace formats are provided:
- a human readable text format.
- a "machine" readable text format.
This text format is easily parsable by other programs
to extract useful information.
- a graphical format, obtained thanks to
an interactive visualization tool,
available in the testing and unstable Debian distributions.
PTT offers the following features:
- trace cut:
it is possible to only work on a part of the trace,
selected with temporal or numeric criteria.
- trace split:
it is possible to split the trace into several files
to get either one file per process or one file per thread.
- trace filtering:
it is possible to filter the trace on various criteria,
such as event names, object names, kind of objects or pid.
- log levels:
it is possible to dynamically switch from light to richer or full trace.
- continuous recording:
the tool can keep only last traces of the execution.
- handle large volume of traces.
- handle bad situations (hang, crash, kill).
- thread contention analysis
(most contented threads and NPTL objects).
PTT can be used on several 32 or 64 bits architectures,
among which are Intel 32 bits,
Intel 64 bits and PowerPC.
In order to analyze NPTL internals,
PTT comes with a patch for the glibc
that inserts trace points
into the following NPTL routines:
pthread_create pthread_mutex_init sem_init
pthread_join pthread_mutex_lock sem_open
pthread_cancel pthread_mutex_unlock sem_unlink
pthread_cond_init pthread_mutex_destroy sem_post
pthread_cond_wait pthread_barrier_init sem_wait
pthread_cond_signal pthread_barrier_wait sem_trywait
pthread_cond_broadcast pthread_barrier_destroy sem_destroy
Reliability and performance
PTT reliability is tested using the
Open POSIX Tests Suite
which checks the POSIX conformance of a library.
Experimentations have shown that PTT rarely hangs
and only when tracing artificial stress programs.
The impact of PTT on programs is mainly tested
with High Performance Computing (HPC) applications.
Measures have shown that this impact on performance is very low,
even when a lot of threads are involved
(less than 5% with 16 concurrent threads on 8xia32).
We expect people facing complex problems with multi-threaded applications
to experiment with this tool
in order to find and fix the remaining bugs.
Comments are welcome...