Kernel Debugging Modules This chapter describes the debugger modules, dcmds, and walkers provided to debug the Solaris kernel. Each kernel debugger module is named after the corresponding Solaris kernel module, so that it will be loaded automatically by MDB. The facilities described here reflect the current Solaris kernel implementation and are subject to change in the future; writing shell scripts that depend on the output of these commands is not recommended. In general, the kernel debugging facilities described in this chapter are meaningful only in the context of the corresponding kernel subsystem implementation. See Related Books and Papers for a list of references that provide more information about the Solaris kernel implementation.MDB exposes kernel implementation details that are subject to change at any time. This guide reflects the Solaris kernel implementation as of the date of publication of this guide. Information provided in this guide about modules, dcmds, walkers, and their output formats and arguments might not be correct or applicable to past or future Solaris releases. This section discusses the dcmds and walkers used to debug problems identified by the Solaris kernel memory allocator and to examine memory and memory usage. The dcmds and walkers described here are discussed in more detail in Chapter 9, Debugging With the Kernel Memory Allocator.Kernel Memory Allocatordcmds::allocdby dcmds::allocdby thread ::allocdbyGiven the address of a kernel thread, print a list of memory allocations it has performed in reverse chronological order. bufctl ::bufctl [a address] [c caller] [e earliest] [l latest] [t thread]Kernel Memory Allocatordcmds::bufctldcmds::bufctlPrint a summary of the bufctl information for the specified bufctl address. If one or more options are present, the bufctl information is printed only if it matches the criteria defined by the option arguments; in this way, the dcmd can be used as a filter for input from a pipeline. The a option indicates that the bufctl's corresponding buffer address must equal the specified address. The c option indicates that a program counter value from the specified caller must be present in the bufctl's saved stack trace. The e option indicates that the bufctl's timestamp must be greater than or equal to the specified earliest timestamp. The l option indicates that the bufctl's timestamp must be less than or equal to the specified latest timestamp. The t option indicates that the bufctl's thread pointer must be equal to the specified thread address. Kernel Memory Allocatordcmds::findleaks dcmds::findleaks [ address ] ::findleaks [v]The ::findleaks dcmd provides powerful and efficient detection of memory leaks in kernel crash dumps where the full set of kmem debug features has been enabled. The first execution of ::findleaks processes the dump for memory leaks (this can take a few minutes), then coalesces the leaks by the allocation stack trace. The findleaks report shows a bufctl address and the topmost stack frame for each memory leak that was identified. If the v option is specified, the dcmd prints more verbose messages as it executes. If an explicit address is specified prior to the dcmd, the report is filtered and only leaks whose allocation stack traces contain the specified function address are displayed. Kernel Memory Allocatordcmds::freedby dcmds::freedby thread ::freedbyGiven the address of a kernel thread, print a list of memory frees it has performed, in reverse chronological order. Kernel Memory Allocatordcmds::kgrep dcmds::kgrep value ::kgrepSearch the kernel address space for pointer-aligned addresses that contain the specified pointer-sized value. The list of addresses that contain matching values is then printed. Unlike MDB's built-in search operators, ::kgrep searches every segment of the kernel's address space and searches across discontiguous segment boundaries. On large kernels, ::kgrep can take a considerable amount of time to execute. ::kmalog [ slab | fail ]Kernel Memory Allocatordcmds::kmalogdcmds::kmalogDisplay events in a kernel memory allocator transaction log. Events are displayed in time-reverse order, with the most recent event displayed first. For each event, ::kmalog displays the time relative to the most recent event in T-minus notation (for example, T-0.000151879), the bufctl, the buffer address, the kmem cache name, and the stack trace at the time of the event. Without arguments, ::kmalog displays the kmem transaction log, which is present only if KMF_AUDIT is set in kmem_flags. ::kmalog fail displays the allocation failure log, which is always present; this can be useful in debugging drivers that don't cope with allocation failure correctly. ::kmalog slab displays the slab create log, which is always present. ::kmalog slab can be useful when searching for memory leaks. Kernel Memory Allocatordcmds::kmastat dcmds::kmastat ::kmastatDisplay the list of kernel memory allocator caches and virtual memory arenas, along with corresponding statistics. Kernel Memory Allocatordcmds::kmausers dcmds::kmausers ::kmausers [ef] [cache ...]Print information about the medium and large users of the kernel memory allocator that have current memory allocations. The output consists of one entry for each unique stack trace specifying the total amount of memory and number of allocations that was made with that stack trace. This dcmd requires that the KMF_AUDIT flag is set in kmem_flags. If one or more cache names (for example, kmem_alloc_256) are specified, the scan of memory usage is restricted to those caches. By default all caches are included. If the e option is used, the small users of the allocator are included. The small users are allocations that total less than 1024 bytes of memory or for which there are less than 10 allocations with the same stack trace. If the f option is used, the stack traces are printed for each individual allocation. Kernel Memory Allocatordcmds::kmem_cache dcmds::kmem_cache [ address ] ::kmem_cacheFormat and display the kmem_cache structure stored at the specified address, or the complete set of active kmem_cache structures. Kernel Memory Allocatordcmds::kmem_log dcmds::kmem_log ::kmem_logDisplay the complete set of kmem transaction logs, sorted in reverse chronological order. This dcmd uses a more concise tabular output format than ::kmalog. Kernel Memory Allocatordcmds::kmem_verify dcmds::kmem_verify [ address ] ::kmem_verifyVerify the integrity of the kmem_cache structure stored at the specified address, or the complete set of active kmem_cache structures. If an explicit cache address is specified, the dcmd displays more verbose information regarding errors; otherwise, a summary report is displayed. The ::kmem_verify dcmd is discussed in more detail in Kernel Memory Caches. Kernel Memory Allocatordcmds::vmem dcmds::vmem [ address] ::vmemFormat and display the vmem structure stored at the specified address, or the complete set of activevmem structures. This structure is defined in <sys/vmem_impl.h>. Kernel Memory Allocatordcmds::vmem_seg dcmds::vmem_seg address ::vmem_segFormat and display the vmem_seg structure stored at the specified address. This structure is defined in <sys/vmem_impl.h>. address ::whatis [abv]Kernel Memory Allocatordcmds::whatisdcmds::whatisReport information about the specified address. In particular, ::whatis will attempt to determine if the address is a pointer to a kmem-managed buffer or another type of special memory region, such as a thread stack, and report its findings. If the a option is present, the dcmd reports all matches instead of just the first match to its queries. If the b option is present, the dcmd also attempts to determine if the address is referred to by a known kmem bufctl. If the v option is present, the dcmd reports its progress as it searches various kernel data structures. Kernel Memory AllocatorWalkersallocdby Walkersallocdby allocdbyGiven the address of a kthread_t structure as a starting point, iterate over the set of bufctl structures corresponding to memory allocations performed by this kernel thread. Kernel Memory AllocatorWalkersbufctl Walkersbufctl bufctlGiven the address of a kmem_cache_t structure as a starting point, iterate over the set of allocated bufctls associated with this cache. Kernel Memory AllocatorWalkersfreectl Walkersfreectl freectlGiven the address of a kmem_cache_t structure as a starting point, iterate over the set of free bufctls associated with this cache. Kernel Memory AllocatorWalkersfreedby Walkersfreedby freedbyGiven the address of a kthread_t structure as a starting point, iterate over the set of bufctl structures corresponding to memory deallocations performed by this kernel thread. Kernel Memory AllocatorWalkersfreemem Walkersfreemem freememGiven the address of a kmem_cache_t structure as a starting point, iterate over the set of free buffers associated with this cache. Kernel Memory AllocatorWalkerskmem Walkerskmem kmemGiven the address of a kmem_cache_t structure as a starting point, iterate over the set of allocated buffers associated with this cache. Kernel Memory AllocatorWalkerskmem_cache Walkerskmem_cache kmem_cacheIterate over the active set of kmem_cache_t structures. This structure is defined in <sys/kmem_impl.h>. Kernel Memory AllocatorWalkerskmem_cpu_cache Walkerskmem_cpu_cache kmem_cpu_cacheGiven the address of a kmem_cache_t structure as a starting point, iterate over the per-CPU kmem_cpu_cache_t structures associated with this cache. This structure is defined in <sys/kmem_impl.h>. Kernel Memory AllocatorWalkerskmem_slab Walkerskmem_slab kmem_slabGiven the address of a kmem_cache_t structure as a starting point, iterate over the set of associated kmem_slab_t structures. This structure is defined in <sys/kmem_impl.h>. Kernel Memory AllocatorWalkerskmem_log Walkerskmem_log kmem_logIterate over the set of bufctls stored in the kmem allocator transaction log. leakGiven the address of a bufctl structure, iterate over the set of bufctl structures corresponding to leaked memory buffers with similar allocation stack traces. The ::findleaks dcmd must be applied to locate memory leaks before the leak walker can be used leakbufGiven the address of a bufctl structure, iterate over the set of buffer addresses corresponding to leaked memory buffers with similar allocation stack traces. The ::findleaks dcmd must be applied to locate memory leaks before the leakbuf walker can be used. Directory Name Lookup Cache (DNLC)The MDB file systems debugging support includes a built-in facility to convert vnode pointers to the corresponding file system path name. This conversion is performed using the Directory Name Lookup Cache (DNLC); because the cache does not hold all active vnodes, some vnodes might not be able to be converted to path names and “??” is displayed instead of a name.File Systemsdcmds::fsinfo dcmds::fsinfo ::fsinfoDisplay a table of mounted file systems, including the vfs_t address, ops vector, and mount point of each file system. File Systemsdcmds::lminfo dcmds::lminfo ::lminfoDisplay a table of vnodes with active network locks registered with the lock manager. The pathname corresponding to each vnode is shown. File Systemsdcmds::vnode2path dcmds::vnode2path address ::vnode2path [v]Display the pathname corresponding to the given vnode address. If the v option is specified, the dcmd prints a more verbose display, including the vnode pointer of each intermediate path component. File SystemsWalkersbuf Walkersbuf bufIterate over the set of active block I/O transfer structures (buf_t structures). The buf structure is defined in <sys/buf.h> and is described in more detail in buf9S. This section describes the debugging support for the kernel virtual memory subsystem.Virtual Memorydcmds::addr2smap dcmds::addr2smap address ::addr2smap [offset]Print the smap structure address that corresponds to the given address in the kernel's segmap address space segment. Virtual Memorydcmds::as2proc dcmds::as2proc as ::as2procDisplay the proc_t address for the process corresponding to the as_t address as. Virtual Memorydcmds::memlist dcmds::memlist [ address ] ::memlist [aiv]Display the specified memlist structure or one of the well-known memlist structures. If no memlist address and options are present or if the i option is present, the memlist representing physically installed memory is displayed. If the a option is present, the memlist representing available physical memory is displayed. If the v option is present, the memlist representing available virtual memory is displayed. Virtual Memorydcmds::memstat dcmds::memstat ::memstatDisplay a system-wide memory usage summary. The amount and percentage of system memory consumed by different classes of pages (kernel, anonymous memory, executables and libraries, page cache, and free lists) are displayed, along with the total amount of system memory. Virtual Memorydcmds::page dcmds::page [ address ] ::pageDisplay the properties of the specified page_t. If no page_t address is specified, the dcmd displays the properties of all system pages. Virtual Memorydcmds::seg dcmds::seg seg ::segFormat and display the specified address space segment (seg_t address). Virtual Memorydcmds::swapinfo dcmds::swapinfo [ address ] ::swapinfoDisplay information on all active swapinfo structures or about the specified struct swapinfo. The vnode, filename, and statistics for each structure are displayed. Virtual Memorydcmds::vnode2smap dcmds::vnode2smap vnode ::vnode2smap[offset]Print the smap structure address that corresponds to the given vnode_t address and offset. Virtual MemoryWalkersanon Walkersanon anonGiven the address of an anon_map structure as a starting point, iterate over the set of related anon structures. The anon map implementation is defined in <vm/anon.h>. Virtual MemoryWalkersmemlist Walkersmemlist memlistIterate over the spans of the specified memlist structure. This walker can be used in conjunction with the ::memlist dcmd to display each span. Virtual MemoryWalkerspage Walkerspage pageIterate over all system page structures. If an explicit address is specified for the walk, this is taken to be the address of a vnode and the walker iterates over only those pages associated with the vnode. Virtual MemoryWalkersseg Walkersseg segGiven the address of an as_t structure as a starting point, iterate over the set of address space segments (seg structures) associated with the specified address space. The seg structure is defined in <vm/seg.h>. Virtual MemoryWalkersswapinfo Walkersswapinfo swapinfoIterate over the list of active swapinfo structures. This walker may be used in conjunction with the ::swapinfo dcmd. This section describes the facilities for examining the state of the cpu structures and the kernel dispatcher.CPUs and the Dispatcherdcmds::callout dcmds::callout ::calloutDisplay the callout table. The function, argument, and expiration time for each callout is displayed. CPUs and the Dispatcherdcmds::class dcmds::class ::classDisplay the scheduling class table. CPUs and the Dispatcherdcmds::cpuinfo dcmds::cpuinfo [ cpuid ] ::cpuinfo [v]Display a table of the threads currently executing on each CPU. If an optional CPU ID number or CPU structure address is specified prior to the dcmd name, only the information for the specified CPU is displayed. If the v option is present, ::cpuinfo also displays the runnable threads waiting to execute on each CPU as well as the active interrupt threads. CPUs and the DispatcherWalkerscpu Walkerscpu cpuIterate over the set of kernel CPU structures. The cpu_t structure is defined in <sys/cpuvar.h>. This section describes dcmds and walkers that are useful for kernel developers as well as third-party device driver developers.Device Drivers and DDI Frameworkdcmds::binding_hash_entry dcmds::binding_hash_entry address ::binding_hash_entryGiven the address of a kernel name-to-major number binding hash table entry (struct bind), display the node binding name, major number, and pointer to the next element. Device Drivers and DDI Frameworkdcmds::devbindings dcmds::devbindings ::devbindings device-nameDisplay the list of all instances of the named driver. The output consists of an entry for each instance, beginning with the pointer to the struct dev_info (viewable with $<devinfo or ::devinfo), the driver name, the instance number, and the driver and system properties associated with that instance. Device Drivers and DDI Frameworkdcmds::devinfo dcmds::devinfo address ::devinfo [ q ]Print the system and driver properties associated with a devinfo node. If the q option is specified, only a quick summary of the device node is shown. Device Drivers and DDI Frameworkdcmds::devinfo2driver dcmds::devinfo2driver address ::devinfo2driverPrint the name of the driver (if any) associated with the devinfo node. Device Drivers and DDI Frameworkdcmds::devnames dcmds::devnames [ address ] ::devnames [ v ]Display the kernel's devnames table along with the dn_head pointer, which points at the driver instance list. If the v flag is specified, additional information stored at each entry in the devnames table is displayed. Device Drivers and DDI Frameworkdcmds::prtconf dcmds::prtconf [ devinfo ] ::prtconf [ cpv ]Display the kernel device tree starting at the device node specified by devinfo. If devinfo is not provided, the root of the device tree is assumed by default. If the c option is specified, only children of the given device node are displayed. If the p option is specified, only ancestors of the given device node are displayed. If v is specified, the properties associated with each node are displayed. Device Drivers and DDI Frameworkdcmds::major2name dcmds::major2name [ major-num ] ::major2name [ major-num ]Display the driver name corresponding to the specified major number. The major number can be specified as an expression preceding the dcmd or as a command-line argument. Device Drivers and DDI Frameworkdcmds::modctl2devinfo dcmds::modctl2devinfo [ address ] ::modctl2devinfoPrint all of the device nodes that correspond to the specified modctl address. Device Drivers and DDI Frameworkdcmds::name2major dcmds::name2major ::name2major driver-nameGiven a device driver name, display its major number. Device Drivers and DDI Frameworkdcmds::softstate dcmds::softstate [ address ] ::softstate [ instance-number ]Given a softstate state pointer (see ddi_soft_state_init9F) and a device instance number, display the soft state for that instance. Device Drivers and DDI FrameworkWalkersbinding_hash Walkersbinding_hash binding_hashGiven the address of an array of kernel binding hash table entries (struct bind **), walk all entries in the hash table and return the address of each struct bind. Device Drivers and DDI FrameworkWalkersdevinfo Walkersdevinfo devinfoFirst, iterate over the parents of the given devinfo and return them in order of seniority from most to least senior. Second, return the given devinfo itself. Third, iterate over the children of the given devinfo in order of seniority from most to least senior. The dev_info struct is defined in <sys/ddi_impldefs.h>. Device Drivers and DDI FrameworkWalkersdevinfo_children Walkersdevinfo_children devinfo_childrenFirst, return the given devinfo, then iterate over the children of the given devinfo in order of seniority from most to least senior. The dev_info struct is defined in <sys/ddi_impldefs.h>. Device Drivers and DDI FrameworkWalkersdevinfo_parents Walkersdevinfo_parents devinfo_parentsIterate over the parents of the given devinfo in order of seniority from most to least senior, and then return the given devinfo. The dev_info struct is defined in <sys/ddi_impldefs.h>. Device Drivers and DDI FrameworkWalkersdevi_next Walkersdevi_next devi_nextIterate over the siblings of the given devinfo. The dev_info struct is defined in <sys/ddi_impldefs.h>. Device Drivers and DDI FrameworkWalkersdevnames Walkersdevnames devnamesIterate over the entries in the devnames array. This structure is defined in <sys/autoconf.h>. Device Drivers and DDI FrameworkWalkerssoftstate Walkerssoftstate softstateGiven a softstate pointer (see ddi_soft_state_init(9F)) display all non-NULL pointers to driver state structures. Device Drivers and DDI FrameworkWalkerssoftstate_all Walkerssoftstate_all softstate_allGiven a softstate pointer (see ddi_soft_state_init(9F)) display all pointers to driver state structures. Note that the pointers for unused instances will be NULL. This section describes dcmds and walkers that are useful for kernel developers as well as developers of third-party STREAMS modules and drivers.address ::mblk2dblkGiven the address of an mblk_t, print the address of the corresponding dblk_t. [address] ::mblk_verifyVerify the integrity of one or more message blocks. If an explicit message block address is specified, the integrity of this message block is checked. If no address is specified, the integrity of all active message blocks are checked. This dcmd produces output for any invalid message block state that is detected. STREAMSdcmds::queue dcmds::queue address ::queue [v] [f flag] [F flag] [s syncq]Filter and display the specified queue_t data structure. With no options, various properties of the queue_t are shown. If the v option is present, the queue flags are decoded in greater detail. If the f, F, or m options are present, the queue is displayed only if it matches the criteria defined by the arguments to these options; in this way, the dcmd can be used as a filter for input from a pipeline. The f option indicates that the specified flag (one of the Q flag names from <sys/stream.h>) must be present in the queue flags. The F option indicates that the specified flag must be absent from the queue flags. The m option indicates that the module name associated with the queue must match the specified modname. The s option indicates that the syncq_t associated with the queue must match the specified syncq_t address. STREAMSdcmds::q2syncq dcmds::q2syncq address ::q2syncqGiven the address of a queue_t, print the address of the corresponding syncq_t data structure. STREAMSdcmds::q2otherq dcmds::q2otherq address ::q2otherqGiven the address of a queue_t, print the address of the peer read or write queue structure. STREAMSdcmds::q2rdq dcmds::q2rdq address ::q2rdqGiven the address of a queue_t, print the address of the corresponding read queue. STREAMSdcmds::q2wrq dcmds::q2wrq address ::q2wrqGiven the address of a queue_t, print the address of the corresponding write queue. STREAMSdcmds::stream dcmds::stream [ address ] ::streamDisplay a visual picture of a kernel STREAM data structure, given the address of the stdata_t structure representing the STREAM head. The read and write queue pointers, byte count, and flags for each module are shown, and in some cases additional information for the specific queue is shown in the margin. address ::syncq [v] [f flag] [F flag] [t type] [T type]STREAMSdcmds::syncqdcmds::syncqFilter and display the specified syncq_t data structure. With no options, various properties of the syncq_t are shown. If the v option is present, the syncq flags are decoded in greater detail. If the f, F, t, or T options are present, the syncq is displayed only if it matches the criteria defined by the arguments to these options; in this way, the dcmd can be used as a filter for input from a pipeline. The f option indicates that the specified flag (one of the SQ_ flag names from <sys/strsubr.h>) must be present in the syncq flags. The F option indicates that the specified flag must be absent from the syncq flags. The t option indicates that the specified type (one of the SQ_CI or SQ_CO type names from <sys/strsubr.h>) must be present in the syncq type bits. The T option indicates that the specified type must be absent from the syncq type bits. STREAMSdcmds::syncq2q dcmds::syncq2q address ::syncq2qGiven the address of a syncq_t, print the address of the corresponding queue_t data structure. b_contGiven the address of an mblk_t, iterate over the set of associated message structures by following the b_cont pointer. The b_cont pointer is used to link a given message block to the next associated message block that is the continuation of the same message. The message block is described in more detail in msgb(9S) b_nextGiven the address of an mblk_t, iterate over the set of associated message structures by following the b_next pointer. The b_next pointer is used to link a given message block to the next associated message block on a given queue. The message block is described in more detail in msgb(9S). STREAMSWalkersqlink Walkersqlink qlinkGiven the address of a queue_t structure, walk the list of related queues using the q_link pointer. This structure is defined in <sys/stream.h>. STREAMSWalkersqnext Walkersqnext qnextGiven the address of a queue_t structure, walk the list of related queues using the q_next pointer. This structure is defined in <sys/stream.h>. STREAMSWalkersreadq Walkersreadq readqGiven the address of an stdata_t structure, walk the list of read-side queue structures. STREAMSWalkerswriteq Walkerswriteq writeqGiven the address of an stdata_t structure, walk the list of write-side queue structures. The following dcmds and walkers are provided to help debug the core kernel networking stack protocols.Networkingdcmds::mi dcmds::mi address ::mi [p] [d | m]Given the address of a kernel MI_O, filter and display the MI_O or its payload. If the p option is specified, then the address of the corresponding payload of the MI_O is displayed, otherwise the MI_O itself is displayed. Specifying filter d or m enables the dcmd to filter device or module MI_O objects respectively. Networkingdcmds::netstat dcmds::netstat ::netstat [av] [f inet | inet6 | unix] [P tcp | udp]Show network statistics and active connections. If the a option is present, the state of all sockets is displayed. If the v option is present, more verbose output is displayed. If the f option is present, only connections associated with the specified address family are displayed. If the P option is present, only connections associated with the specified protocols are displayed. Networkingdcmds::sonode dcmds::sonode [ address ] ::sonode [f inet | inet6 | unix | id] [t stream | dgram | raw | id] [p id]Filters and displays sonode objects. If no address is given, then the list of AF_UNIX sockets is displayed, otherwise only the specified sonode is displayed. If the f option is present, then only sockets of the given family will be output. If the t option is present, then only sonodes of the given type will be output. If the p option is present, then only sockets of the given protocol will be displayed. Networkingdcmds::tcpb dcmds::tcpb [ address ] ::tcpb [av] [P v4 | v6]Filters and displays tcpb objects. If no address is specified, all connections are walked, otherwise only the specified tcpb is filtered/displayed. Specifying a filters for only active connections and P can be used to filter for TCP IPv4 or IPv6 connections. The tcpb dcmd is intelligent about filtering TCP connections, and if a IPv6 TCP connection is in a state that would still facilitate a IPv4 connection, the P filter considers the connection as both IPv4 and IPv6 in much the same way that ::netstat does. If the dcmd is not being used as a filter and the v option is specified, then the output of the dcmd will be verbose. NetworkingWalkersar Walkersar arGiven the address of an ar, this walker walks all ar objects from the given ar to the final ar. If no address is specified, all ar objects are walked. NetworkingWalkersicmp Walkersicmp icmpGiven the address of an icmp, this walker walks all icmp objects from the given icmp to the final icmp. If no address is specified, all icmp objects are walked. NetworkingWalkersill Walkersill illGiven the address of an interface link layer structure (ill), this walker walks all ill objects from the given ill to the final. If no address is specified, all ill objects are walked. NetworkingWalkersipc Walkersipc ipcGiven the address of an ipc, this walker walks all ipc objects from the given ipc to the final ipc. If no address is specified, all ipc objects are walked. NetworkingWalkersmi Walkersmi miGiven the address of a MI_O, walk all the MI_O's in this MI. NetworkingWalkerssonode Walkerssonode sonodeGiven the address of a AF_UNIX sonode, walk the associated list of AF_UNIX sonodes beginning with the given sonode. If no address is specified, this walker walks the list of all AF_UNIX sockets. NetworkingWalkerstcpb Walkerstcpb tcpbGiven the address of a tcpb, this walker walks all TCP connections from the given tcpb to the final TCP connection. If no address is specified, all tcpb objects are walked. NetworkingWalkersudp Walkersudp udpGiven the address of a udp, this walker walks all udp objects from the given udp to the final udp. If no address is specified, all udp objects are walked. This section describes dcmds and walkers used to format and examine various fundamental file, process, and thread structures in the Solaris kernel. Files, Processes, and Threadsdcmds::fd dcmds::fd process ::fd fd-numPrint the file_t address corresponding to the file descriptor fd-num associated with the specified process. The process is specified using the virtual address of its proc_t structure. Files, Processes, and Threadsdcmds::findstack dcmds::findstack thread ::findstack [ command ]Print the stack trace associated with the given kernel thread, identified by the virtual address of its kthread_t structure. The dcmd employs several different algorithms to locate the appropriate stack backtrace. If an optional command string is specified, the dot variable is reset to the frame pointer address of the topmost stack frame, and the specified command is evaluated as if it had been typed at the command line. The default command string is “<.$C0”; that is, print a stack trace including frame pointers but no arguments. Files, Processes, and Threadsdcmds::pgrep dcmds::pgrep ::pgrep [x] [n|o] regexpDisplay process information for processes whose name matches the regexp regular expression pattern. The ::pgrep dcmd is similar to the pgrep1 command. The ::pgrep dcmd is used to pattern match against all processes. When the n option is used, display only the newest process that matches the pattern. When the o option is used, display only the oldest process that matches the pattern. When the x option is used, display only those processes whose names are exactly the same as the search pattern.In kmdb1, the regexp used with ::pgrep must be a plain alpha-numeric text string. Files, Processes, and Threadsdcmds::pid2proc dcmds::pid2proc pid ::pid2procPrint the proc_t address corresponding to the specified PID. Recall that MDB's default base is hexadecimal, so decimal PIDs obtained using pgrep1 or ps1 should be prefixed with 0t. Files, Processes, and Threadsdcmds::pmap dcmds::pmap process ::pmap [q]Print the memory map of the process indicated by the given process address. The dcmd displays output using a format similar to pmap1. If the q option is present, the dcmd displays an abbreviated form of its output that requires less processing time. [ address ] ::ps [fltTP]Files, Processes, and Threadsdcmds::psdcmds::psPrint a summary of the information related to the specified process, or all active system processes, similar to ps1. If the f option is specified, the full command name and initial arguments are printed. If the l option is specified, the LWPs associated with each process are printed. If the t option is specified, the kernel threads associated with each process LWP are printed. If the T option is specified, the task ID associated with each process is displayed. If the P option is specified, the project ID associated with each process is displayed. Files, Processes, and Threadsdcmds::ptree dcmds::ptree ::ptreePrint a process tree, with child processes indented from their respective parent processes. The dcmd displays output using a format similar to ptree1. Files, Processes, and Threadsdcmds::task dcmds::task address ::taskPrint a list of the active kernel task structures and their associated ID numbers and attributes. The process task ID is described in more detail in settaskid(2). Files, Processes, and Threadsdcmds::thread dcmds::thread [ address ] ::thread [bdfimps]Display properties of the specified kernel kthread_t structure. If no kthread_t address is specified, the properties of all kernel threads are displayed. The dcmd options are used to control which output columns are shown. If no options are present, the i option is enabled by default. If the b option is present, information relating to the thread's turnstile and blocking synchronization object is shown. If the d option is present, the thread's dispatcher priority, binding, and last dispatch time is shown. If the f option is present, threads whose state is TS_FREE are elided from the output. If the i option is present (the default), thread state, flags, priority, and interrupt information is shown. If the m option is present, all of the other output options are merged together on to a single output line. If the p option is present, the thread's process, LWP, and credential pointers are displayed. If the s option is present, the thread's signal queue and masks of pending and held signals are shown. Files, Processes, and Threadsdcmds::whereopen dcmds::whereopen vnode ::whereopenGiven a vnode_t address, print the proc_t addresses of all processes that have this vnode currently open in their file table. Files, Processes, and ThreadsWalkersfile Walkersfile fileGiven the address of a proc_t structure as a starting point, iterate over the set of open files (file_t structures) associated with the specified process. The file_t structure is defined in <sys/file.h>. Files, Processes, and ThreadsWalkersproc Walkersproc procIterate over the active process (proc_t) structures. This structure is defined in <sys/proc.h>. taskGiven a task pointer, iterate over the list of proc_t structures for processes that are members of the given task. Files, Processes, and ThreadsWalkersthread Walkersthread threadIterate over a set of kernel thread (kthread_t) structures. If the global walk is invoked, all kernel threads are returned by the walker. If a local walk is invoked using a proc_t address as the starting point, the set of threads associated with the specified process is returned. The kthread_t structure is defined in <sys/thread.h>. This section describes dcmds and walkers used to examine particular kernel synchronization primitives. The semantics of each primitive are discussed in the corresponding (9f) section of the manual pages. Synchronization Primitivesdcmds::rwlock dcmds::rwlock rwlock ::rwlockGiven the address of a readers-writers lock (see rwlock9F), display the current state of the lock and the list of waiting threads. Synchronization Primitivesdcmds::sobj2ts dcmds::sobj2ts address ::sobj2tsConvert the address of a synchronization object to the address of the corresponding turnstile and print the turnstile address. Synchronization Primitivesdcmds::turnstile dcmds::turnstile [ address ] ::turnstileDisplay the properties of the specified turnstile_t. If no turnstile_t address is specified, the dcmd displays the properties of all turnstiles. Synchronization Primitivesdcmds::wchaninfo dcmds::wchaninfo [ address ] ::wchaninfo [v]Given the address of a condition variable (see condvar9F) or semaphore (see semaphore9F), display the current number of waiters on this object. If no explicit address is specified, display all such objects that have waiting threads. If the v option is specified, display the list of threads that are blocked on each object. Synchronization PrimitivesWalkersblocked Walkersblocked blockedGiven the address of a synchronization object (such as a mutex9F or rwlock9F), iterate over the list of blocked kernel threads. Synchronization PrimitivesWalkerswchan Walkerswchan wchanGiven the address of a condition variable (see condvar9F) or semaphore (see semaphore9F), iterate over the list of blocked kernel threads. The cyclic subsystem is a low-level kernel subsystem that provides high resolution, per-CPU interval timer facilities to other kernel services and programming interfaces.Cyclicsdcmds::cycinfo dcmds::cycinfo ::cycinfo [vV]Display the cyclic subsystem per-CPU state for each CPU. If the v option is present, a more verbose display is shown. If the V option is present, an even more verbose display than v is shown. Cyclicsdcmds::cyclic dcmds::cyclic address ::cyclicFormat and display the cyclic_t at the specified address. Cyclicsdcmds::cyccover dcmds::cyccover ::cyccoverDisplay cyclic subsystem code coverage information. This information is available only in a DEBUG kernel. Cyclicsdcmds::cyctrace dcmds::cyctrace ::cyctraceDisplay cyclic subsystem trace information. This information is available only in a DEBUG kernel. CyclicsWalkerscyccpu Walkerscyccpu cyccpuIterate over the per-CPU cyc_cpu_t structures. This structure is defined in <sys/cyclic_impl.h>. CyclicsWalkerscyctrace Walkerscyctrace cyctraceIterate over the cyclic trace buffer structures. This information is only available in a DEBUG kernel. The task queue subsystem provides general-purpose asynchronous task scheduling for a variety of clients in the kernel.Task Queuesdcmds::taskq_entry dcmds::taskq_entry address ::taskq_entryPrint the contents of the specified struct taskq_entry. Task QueuesWalkerstaskq_entry Walkerstaskq_entry taskq_entryGiven the addresss of a taskq structure, iterate over the list of taskq_entry structures. The error queue subsystem provides general-purpose asynchronous error event processing for platform-specific error handling code.Error Queuesdcmds::errorq dcmds::errorq [ address ] ::errorqDisplay a summary of information relating to the specified error queue. If no address is given, display information relating to all system error queues. The address, name, queue length, and data element size for each queue are displayed, along with various queue statistics. Error QueuesWalkerserrorq Walkerserrorq errorqWalk the list of system error queues and return the address of each individual error queue. Error QueuesWalkerserrorq_data Walkerserrorq_data errorq_dataGiven the address of an error queue, return the address of each pending error event data buffer. This section describes dcmds that can be used to examine system configuration data.Configurationdcmds::system dcmds::system ::systemDisplay the contents of the system(4) configuration file at the time the kernel parsed the file during system initialization. The ipc module provides debugging support for the implementation of the message queue, semaphore, and shared memory interprocess communication primitives.::ipcs [l]Interprocess Communication Debugging Support (ipc)dcmds::ipcsdcmds::ipcsDisplay a listing of system-wide IPC identifiers, corresponding to known message queues, semaphores, and shared memory segments. If the l option is specified, a longer listing of information is shown. address ::msg [l] [t type]Interprocess Communication Debugging Support (ipc)dcmds::msgdcmds::msgDisplay the properties of the specified message queue element (struct msg). If the l option is present, the raw contents of the message are displayed in hexadecimal and ASCII. If the t option is present, it can be used to filter the output and only display messages of the specified type. This can be useful when piping the output of the msgqueue walker to ::msg. id ::msqid [k]Interprocess Communication Debugging Support (ipc)dcmds::msqiddcmds::msqidConvert the specified message queue IPC identifier to a pointer to the corresponding kernel implementation structure and print the address of this kernel structure. If the k option is present, the id is instead interpreted as a message queue key to match (see msgget(2)). Interprocess Communication Debugging Support (ipc)dcmds::msqid_ds dcmds::msqid_ds [ address ] ::msqid_ds [l]Print the specified msqid_ds structure or a table of the active msqid_ds structures (message queue identifiers). If the l option is specified, a longer listing of information is displayed. Interprocess Communication Debugging Support (ipc)dcmds::semid dcmds::semid id ::semid [k]Convert the specified semaphore IPC identifier to a pointer to the corresponding kernel implementation structure and print the address of this kernel structure. If the k option is present, the id is instead interpreted as a semaphore key to match (see semget(2)). Interprocess Communication Debugging Support (ipc)dcmds::semid_ds dcmds::semid_ds [ address ] ::semid_ds [l]Print the specified semid_ds structure or a table of the active semid_ds structures (semaphore identifiers). If the l option is specified, a longer listing of information is displayed. Interprocess Communication Debugging Support (ipc)dcmds::shmid dcmds::shmid id ::shmid [k]Convert the specified shared memory IPC identifier to a pointer to the corresponding kernel implementation structure and print the address of this kernel structure. If the k option is present, the id is instead interpreted as a shared memory key to match (see shmget(2)). dcmds::shmid_ds Interprocess Communication Debugging Support (ipc)dcmds::shmid_ds [ address ] ::shmid_ds [l]Print the specified shmid_ds structure or a table of the active shmid_ds structures (shared memory segment identifiers). If the l option is specified, a longer listing of information is displayed. Walkersmsg Interprocess Communication Debugging Support (ipc)Walkersmsg msgWalk the active msqid_ds structures corresponding to message queue identifiers. This structure is defined in <sys/msg.h>. Walkersmsgqueue Interprocess Communication Debugging Support (ipc)Walkersmsgqueue msgqueueIterate over the message structures that are currently enqueued on the specified message queue. Walkerssem Interprocess Communication Debugging Support (ipc)Walkerssem semWalk the active semid_ds structures corresponding to semaphore identifiers. This structure is defined in <sys/sem.h>. Walkersshm Interprocess Communication Debugging Support (ipc)Walkersshm shmWalk the active shmid_ds structures corresponding to shared memory segment identifiers. This structure is defined in <sys/shm.h>. The lofs module provides debugging support for the lofs7FS file system.dcmds::lnode Loopback File System Debugging Supportdcmds::lnode [ address ] ::lnodePrint the specified lnode_t, or a table of the active lnode_t structures in the kernel. dcmds::lnode2dev Loopback File System Debugging Support (lofs)dcmds::lnode2dev address ::lnode2devPrint the dev_t (vfs_dev) for the underlying loopback mounted filesystem corresponding to the given lnode_t address. dcmds::lnode2rdev Loopback File System Debugging Support (lofs)dcmds::lnode2rdev address ::lnode2rdevPrint the dev_t (li_rdev) for the underlying loopback mounted file system corresponding to the given lnode_t address. Walkerslnode Loopback File System Debugging Support (lofs)Walkerslnode lnodeWalk the active lnode_t structures in the kernel. This structure is defined in <sys/fs/lofs_node.h>. The ip module provides debugging support for the ip7P driverdcmds::ire Internet Protocol Module Debugging Support (ip)dcmds::ire [ address ] ::ire [q]Print the specified ire_t, or a table of the active ire_t structures in the kernel. If the q flag is specified, the send and receive queue pointers are printed instead of the source and destination addresses. Walkersire Internet Protocol Module Debugging Support (ip)Walkersire ireWalk the active ire (Internet Route Entry) structures in the kernel. This structure is defined in <inet/ip.h>. This section describes the debugging support for the kernel runtime link editor, which is responsible for loading kernel modules and drivers.dcmds::modctl Kernel Runtime Link Editor Debugging Supportdcmds::modctl [ address ] ::modctlPrint the specified modctl, or a table of the active modctl structures in the kernel. dcmds::modhdrs Kernel Runtime Link Editor Debugging Support (krtld)dcmds::modhdrs address ::modhdrsGiven the address of a modctl structure, print the module's ELF executable header and section headers. dcmds::modinfo Kernel Runtime Link Editor Debugging Support (krtld)dcmds::modinfo ::modinfoPrint information about the active kernel modules, similar to the output of the /usr/sbin/modinfo command. Walkersmodctl Kernel Runtime Link Editor Debugging Support (krtld)Walkersmodctl modctlWalk the list of active modctl structures in the kernel. This structure is defined in <sys/modctl.h>. The uchi module provides debugging support for the host controller interface portion of the Universal Serial Bus (USB) framework.address ::uhci_qh [bd]dcmds::uhci_qhUSB Framework Debugging Support (uhci)dcmds::uhci_qhGiven the address of a USB UHCI controller Queue Head (QH) structure, print the contents of the structure. If the b option is present iterate over the link_ptr chain, printing all QHs found. If the d option is present, iterate over the element_ptr chain, printing all TDs found. address ::uhci_td [d]dcmds::uhci_tdUSB Framework Debugging Support (uhci)dcmds::uhci_tdGiven the address of a USB UHCI controller Transaction Descriptor (TD) structure, print the contents of the structure. Note this only works for Control and Interrupt TDs. If the d option is present, iterate over the element_ptr chain, printing all TDs found. Walkersuhci_qh USB Framework Debugging Support (uhci)Walkersuhci_qh uhci_qhGiven the address of a USB UHCI controller Queue Head (QH) structure, iterate over the list of such structures. Walkersuhci_td USB Framework Debugging Support (uhci)Walkersuhci_td uhci_tdGiven the address of a USB UHCI controller Queue Head Descriptor (TD) structure, iterate over the list of such structures. The usba module provides debugging support for the platform-independent Universal Serial Bus (USB) framework.::usba_debug_bufdcmds::usba_debug_bufUSB Framework Debugging Support (usba)dcmds::usba_debug_bufPrint the USB debugging information buffer. ::usba_clear_debug_bufdcmds::usba_clear_debug_bufUSB Framework Debugging Support (usba)dcmds::usba_clear_debug_bufEmpty the USB debugging information buffer. [ address ] ::usba_device [pv]dcmds::usba_deviceUSB Framework Debugging Support (usba)dcmds::usba_deviceGiven the address of a usba_device structure, print summary information. If no address is supplied, this dcmd walks the global list of usba_device structures. If the p option is present, also list information for all open pipes on this device. If the v option is present, list verbose information for each device. address ::usb_pipe_handledcmds::usb_pipe_handleUSB Framework Debugging Support (usba)dcmds::usb_pipe_handleGiven the address of a USB pipe handle structure (struct usba_ph_impl), print summary information for this handle. Walkersusba_list_entry USB Framework Debugging Support (usba)Walkersusba_list_entry usba_list_entryGiven the address of a usba_list_entry structure, iterate over the chain of such structures. Walkersusba_device USB Framework Debugging Support (usba)Walkersusba_device usba_deviceWalk the global list of usba_device_t structures. Walkersusb_pipe_handle USB Framework Debugging Support (usba)Walkersusb_pipe_handle usb_pipe_handleGiven a usba_device_t address, walk USB pipe handles. These dcmds and walkers are specific to x86 platforms.[ cpuid | address ] ::ttrace [x]dcmds::ttracePlatform Debugging Supportdcmds::ttraceDisplay trap trace records in reverse chronological order. The trap trace facility is available only in DEBUG kernels. If an explicit dot value is specified, this is interpreted as either a CPU ID number or a trap trace record address, depending on the precise value. If a CPU ID is specified, the output is restricted to the buffer from that CPU. If a record address is specified, only that record is formatted. If the x option is specified, the complete raw record is displayed. Walkersttrace Platform Debugging SupportWalkersttrace ttraceWalk the list of trap trace record addresses in reverse chronological order. The trap trace facility is available only in DEBUG kernels. These dcmds and walkers are specific to the SPARC sun4u platform.[ address ] ::softintdcmds::softintPlatform Debugging Supportdcmds::softintDisplay the soft interrupt vector structure at the specified address, or display all the active soft interrupt vectors. The pending count, PIL, argument, and handler function for each structure is displayed. dcmds::ttctl Platform Debugging Supportdcmds::ttctl ::ttctlDisplay trap trace control records. The trap trace facility is available only in DEBUG kernels. dcmds::ttrace Platform Debugging Supportdcmds::ttrace [ cpuid ] ::ttrace [x]Display trap trace records in reverse chronological order. The trap trace facility is available only in DEBUG kernels. If an explicit dot value is specified, this is interpreted as a CPU ID number, and the output is restricted to the buffer from that CPU. If the x option is specified, the complete raw record is displayed. dcmds::xc_mbox Platform Debugging Supportdcmds::xc_mbox [ address ] ::xc_mboxDisplay the cross-call mailbox at the specified address, or format all the cross-call mailboxes that have pending requests. dcmds::xctrace Platform Debugging Supportdcmds::xctrace ::xctraceFormat and display cross-call trace records in reverse chronological order that are related to CPU cross-call activity. The cross-call trace facility is available only in DEBUG kernels. Walkerssoftint Platform Debugging SupportWalkerssoftint softintIterate over the soft interrupt vector table entries. Walkersttrace Platform Debugging SupportWalkersttrace ttraceIterate over the trap trace record addresses in reverse chronological order. The trap trace facility is only available in DEBUG kernels. Walkersxc_mbox Platform Debugging SupportWalkersxc_mbox xc_mboxIterate over the mailboxes used for CPU handshake and cross-call (x-call) requests.