Tuesday, January 1, 2013

Morse Decoder SNR vs CER Testing

FLDIGI Morse Decoder  CER vs SNR Testing

Summary 

I wanted to get more factual data on how different FLDIGI  CW options influence the character error rate (CER)  at different signal-to-noise ratios (SNR).  In this test I focused on comparing SOM decoder to legacy decoder option and Matched Filter to  FFT filter option. I also tested a few cases where I used ~ 2x wider FFT filter  (68 Hz  vs. 35 Hz bandwidth).  Otherwise I used the default options of FLDIGI.

Based on the test results below the following appears to be true:
  • Adjusting FFT filter bandwidth has much bigger impact on CER than changing between legacy and SOM decoder. 
  • Matched filter automatically sets the filter bandwidth to optimal for given CW speed. 
  • SOM decoder gives about 5% better CER compared to legacy decoder @ -13dB SNR, but this advantage gets smaller as SNR decreases. 
The test results are summarized in the figure 1 below. With 3 kHz wide audio signals you can get near perfect copy (CER below 0.02) at  SNR  -10 dB  using  Matched filter or setting FFT filter at 34 Hz  corresponding to 20 WPM Morse speed.

Fig 1.  FLDIGI  Morse Code decoder CER vs. SNR





Test Cases and Setup


My goals for this this experiment were to
  • Compare FLDIGI legacy  vs. SOM decoder CER at different SNR levels. 
  • Compare  FLDIGI Matched Filter  vs.  FFT filter (bandwidth set to 35Hz)  CER  at different SNR levels. 
  • Compare CER with  FFT filter bandwith  set to 68 Hz  vs.  35 Hz.

I used the following software versions to conduct this test:
First  I generated a test audio file with 20 WPM Morse code using WinMorse and converted it to 16 bit  8 Khz Mono Wav file using Audacity.  I played the audio file with PathSim  configured for AWGN S/N Test simulation.  Both AWGN Noise source and "Input Source" have 3 kHZ band pass filter after them as demonstrated in the figure below.

Fig 2. PathSim Configuration





















Sound output from PathSim  (8 kHz 16 Bit Mono) was connected to  FLDIGI  input using Virtual Audio cable  and level was adjusted at 10 in Windows 7 volume mixer so that FLDIGI waterfall is not saturated.  Audio signal had 3 Khz bandwidth as verified on FLDIGI  waterfall display.

Audio file length was 11 min 27 seconds.  PathSim "RunTime" was adjusted to 11 minutes - this provided only 869  of 904 total character in the file.  The lack of last 27+ seconds creates a systematic error of 35 missing characters causing baseline error of CER 0.03871 -  this is already subtracted from results below.

I changed audio SNR  in PathSim in steps from -20 dB, -15 dB, -14dB, -13 dB,  -10 dB and 0 dB also did a reference test case without any injected noise. I compared FLDIGI decoded characters against sent characters using RTTY Text Comparison tool by Alex VE3NEA.  As FLDIGI sometimes emitted prosigns (like <AR> ) I edited these to a single  '*' character before entering to 'received' field in the RTTY Text Comparison tool. Note that this tool was designed for RTTY characters so BER numbers are not applicable for Morse code as RTTY has fixed bit length baudot code whereas Morse code bit length  varies by character. The character error rate (CER)  is still be applicable.

I plotted the following measured CER results against SNR values with Graph.

Test Results 

Reference case 
No noise    CER 0.0011     First character incorrectly coded 

Test cases 
SNR            CER      Test
  0 db          0.00001   SOM Decoder with FFT Filter @68 Hz

-10 dB         0.01219   Legacy decoder  with Matched Filter @35 Hz
-10 dB         0.01439   SOM decoder    with Matched Filter @35 Hz
-10 dB         0.01439   Legacy decoder with FFT Filter @35 Hz
-10 dB         0.00999   SOM Decoder with FFT Filter @35 Hz
-10 db          0.20799   SOM Decoder with FFT Filter @68 Hz

-13 dB         0.19469   Legacy decoder  with Matched Filter @35 Hz
-13 dB         0.18589   SOM decoder    with Matched Filter @35 Hz
-13 dB         0.16149   Legacy decoder with FFT Filter @35 Hz
-13 dB         0.15379   SOM Decoder with FFT Filter @35 Hz
-13 db          0.66479   SOM Decoder with FFT Filter @68 Hz


-14 dB         0.37499   Legacy decoder  with Matched Filter @35 Hz
-14 dB         0.35729   SOM decoder    with Matched Filter @35 Hz
-14 dB         0.30199   Legacy decoder with FFT Filter @35 Hz
-14 dB         0.30089   SOM Decoder with FFT Filter @35 Hz

-15 dB         0.50669   Legacy decoder  with Matched Filter @35 Hz
-15 dB         0.49779   SOM decoder    with Matched Filter @35 Hz
-15 dB         0.47899   Legacy decoder  with FFT Filter @35 Hz
-15 dB         0.46569   SOM Decoder   with FFT Filter @35 Hz
-15 db          0.71789   SOM Decoder with FFT Filter @68 Hz

-20 dB         0.74229   Legacy decoder  with Matched Filter @35 Hz
-20 dB         0.68479   SOM decoder    with Matched Filter @35 Hz
-20 dB         0.72569   Legacy decoder with FFT Filter @35 Hz
-20 dB         0.68029   SOM Decoder with FFT Filter @35 Hz
-20 db          0.75439   SOM Decoder with FFT Filter @68 Hz

Other Observations on FLDIGI Performance

At low SNR values (-20 dB) FLDIGI seemed to lose CW speed tracking, especially if Matched filter was used.  The RxWPM display showed received speed variation between 23 and 30 WPM  while sent signal was at steady 20 WPM.   This may explain some of the decoding errors.  In real life cases if you know the Morse speed it is probably better to set it manually when trying to decode very noisy signals.


73
Mauri  AG1LE





Popular Posts