- Generated by
1.16.1
|
PipeWire 1.7.0
|
This document explains the architecture of PipeWire's RTP module.
The "RTP module" actually refers to a set of three modules which share source code:
For notes about the configuration, see the individual module documentation.
The core of the RTP sink and source modules is the rtp_stream. This is built around a PipeWire stream. This stream can operate in the PW_DIRECTION_INPUT direction (used by the RTP sink module) or in the PW_DIRECTION_OUTPUT direction (used by the RTP source module).
The rtp_stream is implemented in stream.c and stream.h. stream.c includes audio.c, midi.c, opus.c. These handle media subtype specific setups, teardowns, and data processing:
The process callback in rtp_stream is set by these sources depending on the media subtype. Other, rtp_stream specific callbacks like a flush timeout handler are also set by these sources, since they are media subtype specific.
The RTP sink and source modules are configured via properties, represented by pw_properties. Both support "stream.props" values inside their properties. These values in turn are child pw_properties instances that are passed directly to their rtp_stream instances. The modules also copy some of the values of their own properties into that child pw_properties instance. The exact list of values that are copied over depends on the module. But, this means that some values can be set directly in the module properties, or inside the stream.props properties. One example of this would be sess.ts-direct.
audio.c is by far the most complex of the media subtype handlers. All three handlers have some notion of the direct timestamp and constant latency modes, but audio.c is (currently) the only one with the fully reworked implementation that this document describes (the impl->actual_max_buffer_size modulo scheme, impl->ts_align, device delay compensation, and the exact over/underrun thresholds). midi.c and opus.c still carry their own, simpler direct-vs-constant-latency handling and a TODO to converge on the audio.c approach. audio.c also features the separate PTP sender mode, which the other two do not have at all.
The rtp_stream sets up a fixed-size ring buffer. Its size is derived from the sess.buffer-size property, in bytes. Note that this is a stream property: it is read by rtp_stream_new() from the properties it is handed, and - unlike e.g. sess.ts-direct - neither the sink nor the source module copies it over from its own properties, so in practice it can only be set inside stream.props.
The sess.buffer-size value is not used verbatim. rtp_stream_new() derives two quantities from it:
The actual, allocated buffer is present as impl->buffer. This is the pure data storage buffer, without any read or write index.
The stride value depends on the media subtype, and is set internally by rtp_stream_new().
The buffer contents are always interleaved when the number of channels is greater than 1 and the data is raw audio (so, this does not apply to MIDI for example). The stride value specifies the unit size inside the buffer that contains audio data for all channels, played at the exact same time. In the PCM case, the stride is (num_channels * bytes_per_pcm_sample).
The read and write index logic is handled by impl->ring. Both read and write indices increase monotonically (as free-running values) unless they are resynchronized. Because they are free-running rather than being wrapped at the buffer boundary, the fill level is simply their difference, and that is what removes the usual ambiguity about whether the ring buffer is empty or full. When accessing the actual buffer contents, an index is first turned into a byte offset (see below), and that offset is then reduced to the buffer bounds - in audio.c by taking it modulo impl->actual_max_buffer_size, and in midi.c and opus.c by masking it with impl->buffer_mask / impl->buffer_mask2. Reducing modulo impl->actual_max_buffer_size (rather than the raw impl->buffer_size) is essential for the buffer modes to work properly (explained further below).
The read and write indices are given in RTP sample units. To access data in the buffer, the indices are multiplied by the stride to get a byte offset. This also means that the buffer size (which is given in bytes) must be an integer multiple of the stride size - otherwise, the read and write indices may refer to places in the buffer that cannot contain a full data set for all channels. For example, if the stride is 6, and the buffer size is 100, then when the read index is 16, the byte offset would be 16*6 = 96 - but there, only 4 bytes could be read, not 6. For this reason, the buffer size is internally rounded down to the nearest integer multiple of the stride size, as mentioned above.
In the RTP sink module, the rtp_stream appends data to the ring buffer at its write index, except for when a resynchronization happens - the write index is then reset to match the spa_io_clock.position value (scaled to RTP sample units). One resynchronization always happens at startup. The RTP timestamps of outgoing packets are derived from the ring buffer's read index.
In the RTP source module, rtp_stream reads data from the ring buffer depending on the buffer mode. More on that further below.
Most of the code in stream.c runs in the stream's main loop, while most of the code in the media subtype handlers (audio.c etc.) runs in the stream's data loop.
stream_start() is called by on_stream_state_changed()when the stream's state changes to PW_STREAM_STATE_STREAMING. At that stage, the stream's data loop is running, but the stream's PipeWire graph node is not yet attached to the data loop, so no data processing takes place at this time. The attachment happens after on_stream_state_changed() finished. This means that while stream_start() is run from the main loop, it is safe to set internal states that are accessed and modified by other functions that run in the data loop.
Similarly, stream_stop() is called by on_stream_state_changed()when the stream's state changes to PW_STREAM_STATE_PAUSED. (It is not called however if the node.always-process in the stream.props properties in the RTP source module is set to true.) At that stage, the stream's graph node has already been detached from the data loop. It therefore is safe for stream_stop() to touch internal states that normally would be accessed by functions that run in the data loop.
The media subtype handlers each have an init function, like rtp_audio_init(). This is one of the functions from these handlers that runs in the main loop, since these init functions are called by rtp_stream_new(). The other functions are:
Everything else in the media subtype handlers runs in the data loop, with the exception of ptp_sender_process() in audio.c, which runs under the separate PTP sender's own driver and may have a separate data loop.
audio.c has two extra specialties:
Also, this section specifically describes how the buffer modes in audio.c are handled. midi.c and opus.c do branch on impl->direct_timestamp too, but with their own, simpler handling (and aligning those with what audio.c does is an open TODO); the detailed behavior described here is audio.c specific.
The buffer mode only has a minor influence on the RTP sink module. In the constant latency mode, impl->ts_align is used in resynchronization cases to avoid a discontinuity in the outgoing RTP timestamps. In the direct timestamp mode, impl->ts_align is not used.
The rest of the buffer mode documentation is about the behavior on the receiving side, that is, how the RTP source module uses the rtp_stream.
In both modes, received data is inserted into the ring buffer according to the RTP timestamp. This timestamp is first shifted into the future by the value of impl->target_buffer. Then, the ring buffer's write index is advanced. It is expected by the code that the sender produces continuous timestamps; that is, rtp_timestamp_of_packet_2 = rtp_timestamp_of_packet_1 + rtp_samples_per_packet. In certain cases, resynchronization may take place; the read and write indices are then reset; the read index is set to the timestamp of the next incoming RTP packet, while the write index is set to that packet timestamp + impl->target_buffer; that is, the write index is set to be ahead of the read index by the session latency in samples.
The write index is advanced in rtp_audio_receive(), the read index is advanced in rtp_audio_process_playback().
As mentioned in the RTP source module documentation, this is the default mode, where the fill level is kept at a steady value, which is impl->target_buffer. If the fill level is above or below this, a DLL is used to compute an error rate, which then is fed into the ASRC of the pw_stream the rtp_stream is based on. The estimated amount of samples that are "in-flight" (that is, samples that already were sent out but not yet received or which arrived right after the last graph cycle) are also factored into this computation. This establishes a control loop that resamples the audio data as needed to maintain the fill level at impl->target_buffer. Should the difference between the target and the actual fill level exceed a threshold, the ring buffer indices are resynchronized.
More concretely, the thresholds work as follows. An underrun is detected when fewer samples are available than the current graph cycle needs (avail < wanted); the missing samples are filled with silence and the sync state is dropped. An overrun on the read side is detected when the fill level exceeds SPA_MIN(target_buffer * 8, impl->buffer_size / stride); the excess is dropped by advancing the read index so that only target_buffer worth of data remains (a soft correction, not a full resync). Here target_buffer is the device-delay-adjusted target (see below), i.e. impl->target_buffer minus the device delay - the two coincide only when the device delay is zero. On the write side (rtp_audio_receive()), a fill level exceeding the ring capacity impl->buffer_size / stride sets impl->have_sync to false, forcing a full resync.
If the device delay (specified by the pw_time.delay value) is nonzero, then it is subtracted from impl->target_buffer, and the result is then used as the target fill level instead of impl->target_buffer directly.
Since this mode requires that the graph drivers of sender and receiver are somehow synchronized, it implies that, if the sender's and the receiver's spa_io_clock::position values are sampled at the exact same moment, they are identical. In practice, they usually deviate a bit. This deviation is the time sync error, and the time synchronization mechanism that is used tries to keep this sync error as minimal as possible.
The aforementioned incoming RTP timestamp shift by impl->target_buffer plays a crucial role here, since it makes sure the transport delay (which is what the session latency specifies in this mode) is accounted for.
This mode is called "direct timestamp" mode since, unlike in the constant latency mode, the rtp_audio_process_playback() function directly reads from the ring buffer at an index that is derived from spa_io_clock::position , even if this position jumps around. There is some logic to detect underruns and substitute missing data with silence, but discontinuities otherwise have no lasting effect. The driver must ensure that the spa_io_clock::position value increases steadily (except in major discontinuity cases); clock drift compensation is done by the driver by adjusting the graph invocation timings. See Driver architecture and workflow for more.
In this mode, the rtp_stream DLL is not used.
This section covers the internals of the separate PTP sender. Its user-facing behavior - what it is for, how it is activated via aes67.driver-group, and its benefits and trade-offs - is documented in RTP sink .
Only the audio.c media subtype handler supports this mode. When it is enabled, rtp_audio_init() in audio.c creates an internal pw_filter node that is kept isolated from the graph and is driven by the driver from the aes67.driver-group node group.
When this separate PTP sender is active, rtp_audio_process_capture() behaves differently. Rather than computing a drift itself, it stores the sink driver's timing information (impl->sink_nsec, impl->sink_next_nsec, impl->sink_resamp_delay, impl->sink_quantum) for the sender to use. From that information, ptp_sender_process() estimates the current total delay and computes the error between it and the target. That error is fed into a separate dedicated DLL (impl->ptp_dll), which outputs a rate. That rate (impl->ptp_corr) is then applied as the ASRC's rate at the start of rtp_audio_process_capture(). The ASRC then produces larger or smaller amounts of data, filling the ring buffer to a larger or smaller degree, thus forming a control loop that keeps the fill level at a certain target (see below), similar to what the constant latency mode does.
During the refilling state, no packets are sent out. The refilling state ends once the estimated total delay reaches impl->target_buffer (which is also what the control loop mentioned above targets). That estimated total delay is the sum of the current ring buffer fill level, the delay of the ASRC, and the estimated amount of samples that are "in-flight" (that is, samples that already were sent out but not yet received or which arrived right after the last graph cycle).
Additionally, the sender contains code for checking for too severe deviations between the send progress and the current PTP time. The tolerance range is 2x the quantum size. If the deviation goes beyond that, a resynchronization (and consequently, another refilling) is performed. This catches cases where the separate sender is starved of data (that is, the main graph is lagging behind), and also cases when PTP discontinuities occur.
A similar check exists for the node wake up times. The filter node is scheduled by its own driver, independently of the sink node, so their wake ups are not inherently aligned. It is therefore important to check that the filter wakes up within the bounds of the sink node's wake up times (with some tolerance); if it does not, a resynchronization is performed.
1.16.1