PCAP Feed Decoder

Why this project

Historical NASDAQ TotalView-ITCH data is commonly distributed as length-prefixed ITCH binary streams, while packet-oriented applications consume the messages inside network packets. The decoder provides the missing packet-level processing path between those two representations.

This makes the same implementation useful for two related tasks: benchmarking packet and message decoding independently, and reconstructing Level-3 order books from the decoded exchange feed.

System overview

The processing path is deliberately layered. A packet is first treated as a network object, then reduced through its transport and session framing until the contained ITCH records are available to the exchange-message decoder. The decoded messages can then terminate at the parser benchmark or continue into one of the Level-3 reconstruction paths.

The decoder also contains bounded out-of-order handling. Packets ahead of the expected sequence are temporarily retained and processed when the missing sequence becomes available. Packets that arrive too late, or cannot be retained within the configured buffering window, are counted as out-of-order drops.

Reconstruction paths

The decoder provides two Level-3 reconstruction paths over the same decoded ITCH feed and the same underlying BaseLOBEngine.

Parser-only benchmark

MEASURE_PARSER_LATENCY still performs packet processing, ITCH message extraction, field decoding, and byte-order conversion, but skips Level-3 book updates. MEASURE_RECON_LATENCY measures the corresponding decoding and reconstruction path with book updates enabled.

PCAP generation

Public historical NASDAQ TotalView-ITCH data is distributed as length-prefixed ITCH binary messages rather than original network captures. The generator reconstructs the missing network framing so those real historical messages can be processed by the packet decoder.

The generated PCAP contains Ethernet, IPv4, UDP, MoldUDP64, and the original length-prefixed NASDAQ ITCH messages. The market-data records come directly from the public historical source; the packet boundaries and network headers are constructed by the generator.

The generated PCAP is synthetic at the packet-framing level, not at the market-data level.

This makes the generator a reproducible input-generation utility rather than a requirement of the decoder. If an actual NASDAQ ITCH PCAP is already available, it can be passed directly to pcap_decoder.

Packet-size model

The initial generator packed messages until the 1500-byte IP MTU was reached. That produced valid packets but made the distribution heavily concentrated near the maximum size. The current generator instead varies packet-size boundaries with a three-state Markov process, introducing both size variation and temporal clustering.

The target long-run state distribution is 60% Quiet / 20% Mid / 20% Burst.

The transition matrix gives packet sizes temporal persistence rather than choosing independent random weights for each packet. It was derived from the target state frequencies using the stationary-distribution relationship of the Markov chain, with persistence selected to produce clustered periods of similar packet sizes. The implementation represents probabilities on a 16-bit integer scale and uses a lightweight WyRand-based generator for state transitions and packet-size selection.

Protocol-derived packet boundaries

The packet-size boundaries follow the framing used by the generator. Ethernet, IPv4, UDP, and MoldUDP64 contribute 62 bytes of network overhead. A maximum-size ITCH message is 50 bytes and its 2-byte length prefix makes the maximum record 52 bytes. This gives the 114-byte one-message boundary and the subsequent 52-byte increments used by the generator.

Quiet packets use 114 or 166 bytes. Mid packets range from 218 through 582 bytes in 52-byte increments. Burst packets use 1514 bytes, corresponding to a 1500-byte IP MTU plus the 14-byte Ethernet header. The generator then packs actual ITCH records until the selected boundary is reached or the next message would exceed it.

Current generator scope

The generator models packet-size variation and clustering. It does not attempt to reproduce the exact packet boundaries of a historical exchange capture, and it does not currently inject packet loss or out-of-order sequences. The decoder's bounded sequence buffering can be exercised by a future generator extension that deliberately introduces gaps or reordering.

The packet-size model is motivated by published measurements showing that market-data packet-size distributions vary by exchange and that averages alone do not capture the range of packet sizes present in real feeds.

Decoder architecture

Reproducibility

The project is intentionally build-system-light. The decoder and generator can be compiled directly with a C++23 compiler.

git clone --recursive https://github.com/pankajj6/pcap_feed_decoder.git
cd pcap_feed_decoder

Build the decoder

g++ -std=c++23 -O3 \
    -Ibase_lob_engine \
    -Iinclude \
    src/pcap_decoder.cpp \
    -o pcap_decoder

./pcap_decoder data/generated/01302020.pcap

Parser benchmark

g++ -std=c++23 -O3 \
    -DMEASURE_PARSER_LATENCY \
    -Ibase_lob_engine \
    -Iinclude \
    src/pcap_decoder.cpp \
    -o parser_bench

./parser_bench data/generated/01302020.pcap

Single-stock parser

g++ -std=c++23 -O3 \
    -DMEASURE_PARSER_LATENCY \
    -DSPECIFIC_STOCK_LOCATE=398 \
    -Ibase_lob_engine \
    -Iinclude \
    src/pcap_decoder.cpp \
    -o parser_bench

./parser_bench data/generated/01302020.pcap

Single-stock reconstruction

g++ -std=c++23 -O2 \
    -DMEASURE_RECON_LATENCY \
    -DSPECIFIC_STOCK_LOCATE=398 \
    -Ibase_lob_engine \
    -Iinclude \
    src/pcap_decoder.cpp \
    -o recon_bench

./recon_bench data/generated/01302020.pcap

Full reconstruction

g++ -std=c++23 -O3 \
    -DMEASURE_RECON_LATENCY \
    -Ibase_lob_engine \
    -Iinclude \
    src/pcap_decoder.cpp \
    -o recon_bench

./recon_bench data/generated/01302020.pcap

PCAP generation

g++ -std=c++23 -O3 \
    -Iinclude \
    src/pcap_generator.cpp \
    -o pcap_generator

./pcap_generator \
    data/input/01302020.NASDAQ_ITCH50 \
    data/generated/01302020.pcap

The generator also accepts an optional packet limit:

./pcap_generator \
    data/input/01302020.NASDAQ_ITCH50 \
    data/generated/01302020.pcap \
    1000000

Repository structure

pcap_feed_decoder/
├── base_lob_engine/       Git submodule
├── data/
│   ├── input/             NASDAQ ITCH binary input
│   └── generated/         Generated PCAP files
├── include/
│   ├── itch_packet_processor.h
│   └── nasdaq_itch50.h
├── src/
│   ├── pcap_decoder.cpp
│   └── pcap_generator.cpp
├── .gitignore
├── .gitmodules
└── README.md

The base_lob_engine directory is maintained as a separate repository and included here as a Git submodule.

Scope

• NASDAQ TotalView-ITCH 5.0 message decoding
• Ethernet / IPv4 / UDP / MoldUDP64 packet processing
• MoldUDP64 session and sequence-number handling
• bounded out-of-order packet buffering
• Level-3 limit order-book reconstruction
• direct reconstruction through itch_* interfaces
• Event-based reconstruction through reconstruct_market_state()
• LobState support for derived market-state statistics
• packet-level latency instrumentation
• compile-time single-stock benchmarking with SPECIFIC_STOCK_LOCATE
• reproducible PCAP generation from public NASDAQ ITCH binary data
• optional verbose decoded-message output

The current generator models packet-size variation and clustering. It does not reproduce the exact packet boundaries of a historical capture and does not currently inject packet loss or out-of-order sequences.