NPTL Trace Tool

Users

PTT aims to answer the needs of three kinds of users:

• a developer writting multi-threaded applications mainly wants to trace calls to and exits from NPTL routines, with details about parameters. He needs to easily switch from the original untraced NPTL to the traced library, and vice-versa, without recompiling his application.
• a member of a Linux support team providing skills to people who write, test or use applications. He has strong skills about the Linux kernel and glibc, therefore he needs to see what happens inside NPTL. He also wants to be able to analyze in his own offices traces generated at customers site.
• a maintainer of the glibc wants to analyze the behaviour of NPTL and fix problems, which is not an easy task. PTT provides him with traces of internal NPTL events.

Main features

A multi-threaded application can be traced without being recompiling. The trace is analyzed once the application stopped: it is a post-mortem analysis. Three different trace formats are provided:

• a human readable text format.
• a "machine" readable text format. This text format is easily parsable by other programs to extract useful information.
• a graphical format, obtained thanks to Pajé, an interactive visualization tool, available in the testing and unstable Debian distributions.

PTT offers the following features:

• trace cut: it is possible to only work on a part of the trace, selected with temporal or numeric criteria.
• trace split: it is possible to split the trace into several files to get either one file per process or one file per thread.
• trace filtering: it is possible to filter the trace on various criteria, such as event names, object names, kind of objects or pid.
• log levels: it is possible to dynamically switch from light to richer or full trace.
• continuous recording: the tool can keep only last traces of the execution.
• handle large volume of traces.
• handle bad situations (hang, crash, kill).
• thread contention analysis (most contented threads and NPTL objects).

PTT can be used on several 32 or 64 bits architectures, among which are Intel 32 bits, Intel 64 bits and PowerPC.

In order to analyze NPTL internals, PTT comes with a patch for the glibc that inserts trace points into the following NPTL routines:

pthread_create              pthread_mutex_init          sem_init
pthread_join                pthread_mutex_lock          sem_open
pthread_cancel              pthread_mutex_unlock        sem_unlink
pthread_cond_init           pthread_mutex_destroy       sem_post
pthread_cond_wait           pthread_barrier_init        sem_wait
pthread_cond_signal         pthread_barrier_wait        sem_trywait
pthread_cond_broadcast      pthread_barrier_destroy     sem_destroy
pthread_cond_destroy

Reliability and performance

PTT reliability is tested using the Open POSIX Tests Suite (OPTS), which checks the POSIX conformance of a library. Experimentations have shown that PTT rarely hangs and only when tracing artificial stress programs.

The impact of PTT on programs is mainly tested with High Performance Computing (HPC) applications. Measures have shown that this impact on performance is very low, even when a lot of threads are involved (less than 5% with 16 concurrent threads on 8xia32).