HF Modem Frequency Offset Estimation

One of my goals for 2014 is to make FreeDV work as well as SSB on HF radio channels. Recently I have been working on improvements to the frame sync algorithms used in the FDMDV modem. I would like to move from a “hard” sync decision to a “soft” one, such that the demod can track through fades. Often during a fade our signal is still there, so it’s better to wait around a bit than go off trying to find a new one. There is a trade off between remaining “in sync” and tracking through a fade and syncing up quickly when a new FreeDV signal appears.

Frame sync relies on the coarse frequency offset estimation algorithm, that works out the centre frequency of the modem signal is in the receivers passband. The nominal centre frequency is 1500 Hz. An offset of 100 Hz would mean the centre frequency is actually 1600 Hz. We need to know the offset within a few Hz for the demod to work properly. If we don’t get the frequency offset right, the demodulator outputs garbage, so no decoded speech. If the frequency offset estimation jumps about, the decoded speech will stop and start.

Frequency Offset Estimation in Action

The frequency offset estimation algorithm works by multiplying (mixing) the incoming signal with a noise free copy of the expected BPSK pilot signal. We then take the FFT of the resulting signal, and peak pick. On a good day, this will have a peak corresponding to the frequency offset. We then usually apply some post processing logic to correct the inevitable errors.

Here it is in action for a 0dB SNR AWGN channel with a -50Hz frequency offset. It’s a weak signal. First a spectrogram of the signal at the input of the demodulator, then a spectrogram of the output of the mixer (note -50Hz line), then a plot of four frequency offset estimates. The x axis in all plots is time, one frame is 20ms, 50 frames 1 second.

The four frequency offset lines “foff_xxx” above show firstly the “raw” output from peak picking the FFT, then the output from three different post processing algorithms. While all but the foff_thresh line look OK here, in practice they all have there pros and cons, none are completely reliable.

Here are the same plots on the nasty HF fading channel. You can see gaps in the pilot just under 1500Hz, which leads to the “dotted” -50Hz line on the mixer output spectrogram. The raw frequency offset estimate is all over the place, although it does get cleaned up by the post processors. Note the foff_state and foff_thresh plots take a long time to lock up, e.g. foff_thresh doesn’t jump to -50Hz until quite late in the simulation, and foff_state takes more than 1 second (50 frames).

Automated Tests

I’ve written a Unit Test (UT) in Octave called fdmdv_ut_freq_est.m that can perform automated tests of various channel conditions:
Test 3: 30 Seconds in HF multipath channel at 0dB-ish SNR
Channel EbNo SNR(calc) SNR(meas) SD(Hz) Hits Hits(%) Result
AWGN 3.00 0.78 1.18 22.00 200 100.00 PASS

Test 3: 30 Seconds in HF multipath channel at 0dB-ish SNR
Channel EbNo SNR(calc) SNR(meas) SD(Hz) Hits Hits(%) Result
HF 3.00 0.78 1.71 87.36 188 94.00 FAIL

The UT also generates the plots above to help me debug the frequency offset estimation algorithm.

Mesh Plots

I discovered that mesh plots were an interesting alternative to spectrograms for plotting signals in 3 dimensions such as time, frequency, and amplitude. On Octave, the mouse can be used to rotate the view of the mesh plot to view it from different angles. Here are some animated videos generated by Octave that illustrate the effect.

The first video is the modem spectrum, with a SNR of 10dB, in an AWGN channel. You can see the central “fin” which is the high energy BPSK pilot. Looks like a toaster! Note the slope from 0 to about 10 frames as the filter memories in the modem “fill up” from the all-zero state at start up.

The second video is the output of the frequency offset estimation mixer. Note the “blade” along the -50Hz line. This is the peak we are looking for.

The third video shows the mess the HF fading channel makes of the modem signal. Deep notches appear, and also some peaks, higher than the signal in the AWGN channel. I wonder if these peaks (short regions of high SNR) they can be used? As the mesh is rotated so it is flat, we get a form of the 2D-colourmap spectrogram.

This command line was used to generate the animations from the PNGs generated by Octave:
david@bear:~/tmp/codec2-dev/octave$ mencoder mf://*.png -mf w=640:h=480:fps=5:type=png -ovc lavc -lavcopts vcodec=mpeg4:mbd=2:trell -oac copy -o freq_est.mp4


I’m not quite happy with the frequency offset estimation algorithm. It still occasionally fails when the BPSK pilot is wiped out completely by a fade. Not quite what I want for the HF channel. Now that I’ve written it up here, I will take a break while I work on the SM1000 code, and come back to frequency offset estimation later.

I’d also like to try FreeDV with no frequency offset estimation. In a way, it’s just another algorithm that can go wrong in fading channels. Without it, the operator would need to tune the receiver to within say 10% of the symbols rate, e.g. 0.1(Rs) = 0.1(50) = 5Hz. But many SSB operators do that anyway. If a higher symbol rate was used, say Rs=200Hz, it’s +/- 20Hz. If we disable frequency offset estimation, we could lose the pilot, saving 1.2dB of SNR, 150Hz of bandwidth, and reducing Peak to Average Power Ratio (PAPR). It could also be a switch-able option, manually disabled by the operator for low SNR channels.

7 thoughts on “HF Modem Frequency Offset Estimation”

  1. If my memory serves me right, I believe Yoshi of AOR implemented what you describe as a “soft” sync decision in his “fast modems” for digital voice. He accomplished this with a push button switch which attempted to re-sync based on the last one acquired. If that failed, then the “hard” sync was “tried” by toggling the DC power.
    Regarding the pilot, I would be for saving SNR, however it does serve an important purpose as a “tuning” indicator so I would vote to keep it as a switchable option in control of the op.

    Thanks for your continued effort to improve the robustness of the fdmdv modem!


    1. Hi Mel,

      Yes the idea of operator assisted sync is a good one. For example a mouse click (or tx-rx transition) might make it run the freq offset estimator, otherwise it will just assume the signal is still within a few Hz of where we left it.



  2. You made reference to the need to balance stability and agility, but what do you need agility for? Surely in the most cases you are trying to achieve a relatively long comms session between two parties?

    Are the frequencies your estimator keeps jumping to distortion products or interferers? If you are trying to lock to one signal source, rather than being agile, its not so hard to deal with fades. As long as you get the frequency right >50% of the time, a medium term histogram analysis of the estimates will tell you the real one.

    1. Re agility if a new signal pops up on roughly the same frequency (but say 100Hz off) we would like to be able automatically estimate it’s freq offset and start demodulating it. Or drift between two parties of more than a few Hz (e.g. due to heating from the 100W PA).

      The other frequencies are other “mixer” products that may have higher peaks due to temporary fading of the pilots. The modem signal is like a comb in the frequency domain. A time domain multiplication leads to spikes in the frequency domain at the output of the mixer.

      Hmm, the histogram idea for post proessing is interesting. Trade off is we want to acquire quickly (say < 1 second in the fading channel). The idea is interesting though – thanks!

  3. Hi David,

    Robust operation and weak signal detection seem to be at odds but maybe this isn’t really true?

    I have been listening in on Olivia 32-1k conversations on 14107.5 a bit. It amazes me how a signal that can barely be detected on the waterfall of fldigi can give 100% copy! I have linrad running on one computer and feeding audio to fldigi on a second so I can actually measure the signal to noise being sent to fldigi.

    Signals -14 to -10 in a 1khz band print perfectly!! You can’t even see them!

    So I think some thought might be given to a portion of the BW being used for very weak signal detection and tracking. The derived information is then fed to the main FreeDV decoder.

    It recently occurred to me that a swept tone type of component appears to allow detection and locking at very weak levels over wide offsets. The detector soft decision de-sweeps the signal band. Since the sweep rate is known precisely and because of this, the detector can narrow its effective bandwidth greatly. It can also detect multiple signals and their levels at the same time.

    It certainly looks like a method that is worth careful study.

    Signal design and integration into FreeDV are the only problems left.

    Anyways seems like a good path to explore.

    warm regards,

  4. Interesting article. I do have some concerns with removing the frequency offset estimator. Your discussion seemed to assume that there would be an operator on all points. With the clarity and positive “carrier detect” of FreeDV, the mode lends itself to unattended operation, such as a HF-VHF gateway. Here there is no one to “click on the waterfall”, and you’re totally reliant on the frequency accuracy of the radio at the link site, as well as that of the users. There are also corner cases such as satellite operation (thanks to Doppler shift) and high altitude balloning, where termal drift can be significant, where the assumption of constant transmitter frequency is invalid. The 1.2dB S/N would be nice to have though.

  5. It would be great to develop something like this for operating through amateur satellite analog transponders. If you could estimate frequency error caused by Doppler and correct small errors in DSP as you describe and force correction of larger errors by QSY’ng the radio via CI-V or CAT commands. Having a full duplex modem would be advantageous because both ends of the conversation could correct their respective uplink frequencies.

Comments are closed.