In many video applications a single processor is not capable of performing all of the desired signal processing tasks. In such cases, one or more signal processing modules are often implemented using additional processors and/or fixed-function hardware. For example, the video decoder task shown in Figure 1 might manage communication and synchronization with a video decoder running on a separate DSP processor instead of actually performing the decoding on the host processor. Alternatively, the video decoder task may perform some of the decoding, and utilize additional hardware such as a DSP or fixed-function accelerator to offload some of the most computationally demanding portions of the video decoding algorithm. See “Processors for Video: Know Your Options” for more information on processors used in these applications.

Putting the Pieces Together
Ideally, all of the software components required for the application are available as licensable off-the-shelf modules. Operating systems, video and audio encoders and decoders, and many common pre- and post-processing functions are available as off-the-shelf modules targeting a variety of processors. However, application software often must be written from scratch. Occasionally a suitable reference design may serve as a reasonable starting point.

Even when most of the needed software components are available in off-the-shelf form, integrating these components can be a significant undertaking. For example, maintaining synchronization of audio and video codecs during recording or playback can be tricky. To maintain synchronization, application software must read and write timestamps embedded in the multimedia stream. This can sometimes require compensation for delays introduced by pre- or post-processing modules.

Starting from Scratch
Implementing a complex signal processing module such as a video decoder or encoder is probably the most challenging software development task in video applications. (See “Squeeze Play: How Video Compression Works” in the March 2004 edition of Inside DSP for an introduction to video compression algorithms. Also see “H.264: The Video Codec to Watch” for details on the popular new H.264 algorithm.) Aggressive optimization is typically required in order to get video applications to meet cost and power targets. When the desired module isn’t available from the processor vendor or a third party and must be developed from scratch, knowing where to start is often the first hurdle.

The complexity of today’s video compression algorithms can make standards specifications difficult to understand, so implementing an encoder or decoder from scratch is a daunting proposition. Reference source code is available for most standards-based encoders and decoders. However, such reference code is typically written to illustrate the standard specification rather than to implement the algorithm efficiently. In addition to being inefficient, reference code may include superfluous code such as unneeded debugging modes. Therefore, such reference source code is awkward to use as a basis for an efficient implementation of the encoder or decoder.

Most video standards precisely specify the compressed bit stream format and the behavior of the decoder, but do not specify the behavior of the encoder. This allows developers to use their own proprietary algorithms for critical encoding steps such as motion estimation. Encoder implementations can thus compete for highest video quality or lowest bit rate, or make tradeoffs between quality, bit rate, and computational demands. Reference source code merely illustrates one possible encoding algorithm, and developers should carefully consider other approaches—such as alternative motion estimation search heuristics—if they have the requisite algorithm design expertise.