Testing FreeDV 700C

Since releasing FreeDV 700C I’ve been “instrumenting” the FreeDV GUI program – adding some code to perform various tests of the 700C waveform, especially over the air.

With the kind help of Gerhard OE3GBB, Mark VK5QI, and Peter VK5APR, I have collected some samples and performed some tests. The goals of this work were:

  1. Compare 700C Over the Air (OTA) to simulation on an AWGN channel.
  2. Compare 700C OTA to SSB on an AWGN channel.

Instrumentation

Here is a screen shot of the latest FreeDV GUI Options screen:

I’ve added some features to the top three rows:

Test Frames Send a payload of known test bits rather than vocoder bits
Channel Noise Simulate a channel using AWGN noise
SNR SNR of AWGN noise
Attn Carrier Attenuate just one carrier
Carrier The 700C carrier (1-14) to attenuate
Simulated Interference Tone Enable an interfering sine wave of specified frequency and amplitude
Clipping Enable clipping of 700C tx waveform, to increase RMS power
Diversity Combine for plots Scatter and Test Frame plots use combined (7 carrier) information.

To explore these options it is useful to run in full duplex mode (Tools-PTT Half Duplex unchecked) and using a loopback sound device:

  $ sudo modprobe snd-aloop

More information on loopback in the FreeDV GUI README.

Clipping the 700C tx waveform reduces the Peak to Average Power ratio (PAPR) which may be result in a higher average power over the channel. However clipping distorts the waveform and add some “shoulders (i.e. noise) to the spectrum adjacent to the 700C waveform:

Several users have noticed this distortion. At this stage I’m unsure if clipping is useful or not.

The Diversity Combine option is useful to explore each of the 14 carriers separately before they are combined into 7 carriers.

Many of these options were designed to explore tx filtering. I have long wondered if any of the FreeDV carriers were receiving less power than others, for example due to ripple or a low pass response from the crystal filter. A low power carrier would have a high bit error rate, adversely affecting overall performance. Plotting the scatter diagram or bit error rate on a carrier by carrier basis can measure the effect of tx filtering – if it exists.

Some of the features above – like attenuating a single carrier – were designed to “test the test”. Much of the work I do on FreeDV (and indeed other projects) involves carefully developing software and writing “code to test the code”. For example to build the experiments described in this blog post I worked several hours day for several weeks. Not glamorous, but where the real labour lies in R&D. Careful, meticulous testing and experimentation. One percent inspiration … then code, test, test.

Comparing Analog SSB to Digital Voice

One of my goals is to develop a HF DV system that is competitive with analog SSB. So we need a way to compare analog and DV at the same SNR. So I came with the idea of a wave files of analog SSB and DV which have the same average (RMS) power. If these are fed into a SSB transmitter, then they will be received at the same SNR. I added 10 seconds of a 1000Hz sine wave at the start for good measure – this could be used to measure the actual SNR.

I developed two files:

  1. sine_analog_700c
  2. sine_analog_testframes700c

The first has the same voice signal in analog and 700C, the second uses test frames instead of encoded voice.

Interfering Carriers

One feature described above simulates an interfering carrier (like a birdie), something I have seen on the air. Here is a plot of a carrier in the middle of one of the 700C carriers, but about 10dB higher:

The upper RH plot is a rolling plot of bit errors for each carrier. You can see one carrier is really messed up – lots of bit errors. The average bit error rate is about 1%, which is where FreeDV 700C starts to become difficult to understand. These bit errors would not be randomly distributed, but would affect one part of the codec all the time. For example the pitch might be consistently wrong, or part of the speech spectrum. I found that as long as the interfering carrier is below the FreeDV carrier, the effect on bit error rate is negligible.

Take away: The tx station must tune away from any interfering carriers that poke above the FreeDV signal carriers. Placing the interfering tones between FreeDV carriers is another possibility, e.g. a 50Hz shift of the tx signal.

Results – Transmit Filtering

Simulation results suggest 700C should produce reasonable results near 0db SNR. So that’s the SNR I’m shooting for in Over The Air (OTA) testing.

Mark VK5QI sent me several minutes of test frames so I could determine if there were any carriers with dramatically different bit error rates, which would indicate the presence of some tx filtering. Here is the histogram of BERs for each carrier for Mark’s signal, which was at about 3dB SNR:

There is one bar for each I and Q QPSK bit of the 14 carriers – 28 bars total (note Diversity combination was off). After running for a few minutes, we can see a range of 5E-2 and 8E-2 (5 to 8%). In terms of AWGN modem performance, this is only about 1dB difference in SNR or Eb/No, as per the BER versus Eb/No graphs in this post on the COHPSK modem used for 700C. One carrier being pinned at say 20% BER, or a slope of increasing BER with carrier frequency – would have meant tx filtering trouble.

Peter VK5APR, sent me a high SNR signal (he lives just 4 km away). Initially I could see a X shaped scatter diagram, a possible sign of tx filtering. However this ended up being some amplitude pumping due to Fast AGC on my radio. After I disabled fast AGC, I could see a scatter diagram with 4 clear dots, and no X-shape. Check.

I performed an additional test using my IC7200 as a transmitter, and a HackRF as a receiver. The HackRF has no crystal filter and a very flat response, so any distortion would be due to the IC7200 transmit filtering. Once again – 4 clean dots on the scatter diagram and no X-shape.

So I am happy to conclude that transmit filtering does not seem to be a problem, at least of the radios tested. All performance issues are therefore likely to be caused by me and my algorithms!

Results – Low SNR testing

Peter, VK5APR, configured his station to send the analog/700C equi-power test wave files described above at very low power, such that the received SNR at my station was about 0dB. As we are so close it was reasonable to assume the channel was AWGN, indeed we could see no sign of NVIS fading and the band (40M) was devoid of DX at the 12 noon test time.

Here is the rx signal I received, and the same file run through the 700C decoder. Neither the SSB or the decoded 700C audio are pretty. However it’s fair to say we could just get a message through on both modes and that 700C is holding it’s own next to SSB. The results are close to my simulations which was the purpose of this test.

You can decode the off air signal yourself if you download the first file and replay it through the FreeDV GUI program using “Tools – Start/Stop Play File from Radio”.

Discussion

While setting up these tests, Peter and I conversed comfortably for some time over FreeDV 700C at a high SNR. This proved to me that for our audience (experienced users of HF radio) – FreeDV 700C can be used for conversational contacts. Given the 700C codec is really just a first pass – that’s a fine result.

However it’s a near thing – the 700C codec adds a lot of distortion just compressing the speech. It’s pretty bad even if the SNR is high. The comments on the Codec 2 700C blog post indicate many lay-people can’t understand speech compressed by 700C. Add any bit errors (due to low SNR or fading) and it quickly becomes hard to understand – even for experienced users. This makes 700C very sensitive to bit errors as the SNR drops. But hey – every one of those 28 bits/frame counts at 700 bit/s so it’s not surprising.

In contrast, SSB scales a bit better with SNR. However even at high SNRs, that annoying hiss is always there – which is very fatiguing. Peter and I really noticed that hiss after a few minutes back on SSB. Yuck.

SSB gets a lot of it’s low SNR “punch” from making effective use of peak power. Here is a plot of the received SSB:

It’s all noise except for the speech peaks, where the “peak SNR” is much higher than 0dB. Our brains are adept at picking out words from those peaks, integrating the received phonetic symbols (mainly vowel energy) in our squishy biological receive filters. It’s a pity we didn’t evolve to detect coherent PSK. A curse on your evolution!

In contrast – 700C allocates just as much power to the silence between words as the most important parts of speech. This suggests we could do a better job at tailoring the codec and modem to peak power, e.g. allocating more power to parts of the speech that really matter. I had a pass at Time Variable Quantisation a few years ago. A variable rate codec might be called for, tightly integrated to the modem to pack more bits/power into perceptually important parts of speech.

The results above assumed equal average power for SSB and FreeDV 700C. It’s unclear if this happens in the real world. For example we may need to “back off” FreeDV drive further than SSB; SSB may use a compressor; and the PAs we are using are generally designed for PEP rather than average power operation.

Next Steps

I’m fairly happy with the baseline COHPSK modem, it seems to be hanging on OK as long as there aren’t any co-channel birdies. The 700C codec works better than expected, has plenty of room for improvement – but it’s sensitive to bit errors. So I’m inclined to try some FEC next. Aim for error free 700C at 0dB, which I think will be superior to SSB. I’ll swap out the diversity for FEC. This will increase the raw BER, but allow me to run a serious rate 0.5 code. I’ll start just with an AWGN channel, then tackle fading channels.

Links

FreeDV 700C
Codec 2 700C

10 thoughts on “Testing FreeDV 700C”

  1. “I’ll swap out the diversity for FEC.”

    I was thinking, if you gave up diversity, and added two more carriers (shifting the center freq to 1200), you could get 128 bits per baud, and do 1300 bit/s and its 300 bit/s FEC (64-bit Vocoder) :-)

    75 Baud/2400 bit/s Modem.

  2. Hi David and all,

    Most impressive.

    Your comments on SSB cause me to wonder about the multi-carrier verses CDMA question. Many years ago an FCC friend called CDMA a ‘radar’ type system. Same average power but made of higher power pulses. He said that this would ‘cut through the noise’ better?

    Would this be a better way to use the same RF power?

    It certainly changes the nature of what we call a radio.

    I wonder if you now have the tools to test this. Way beyond my academic grade for whatever that is worth!

    Nice piece of work!
    John

  3. Hi John,

    No miracles with CDMA – performance on fading channels is the same as PSK with a lot of diversity, on AWGN channels perf the same as PSK. Large spreading (or freq and time diversity, interleaving with FEC) makes a fading channel look more like AWGN.

    Cheers,

    David

  4. Hi David,

    Thanks. So I guess the only CDMA advantages might be use of a single carrier so transmitter is simpler. And if a two way channel, ability to reduce power to share receiver dynamic range among more users? And the usual coding choices stuff, Gold code, etc.

    Have I missed anything?

    John

  5. Just be careful using strings from untrusted sources in command lines.

    The last thing you want that regex to match is something like:

    s=abcd;rm -fr ${HOME}

    at which point FreeDV executes the curl command then merrily erases your home directory. If s=… is meant to be a callsign, I’d be looking to restrict the pattern matching a bit (e.g. [a-zA-z0-9]+/?[a-zA-Z0-9]*) so some nefarious individual doesn’t transmit a nasty surprise.

    Getting to the point of the post however, it is nice to see FreeDV 700C progressing so well. Not surprising that a bit error can have significant impact on the intelligibility of a signal, given how few are being used.

    Perhaps if the less-significant components can be isolated, you could use APK modulation with the phase modulating the more significant parts and the amplitude modulating the less significant parts.

    1. Good point on the string processing Stuart – that’s an automation feature I worked on a few years ago but is not in common use.

  6. Hi Guys
    having started from the ground upwards the idea of KISS seems to be missing here. By keeping it simple sunshine you remove the need for complex decoding software. Having started with RTTY progressed into Amtor with ARQ and FEC this was case of resending a data stream to correct any errors. immediately your having to apply a software program to either request a resend or by means of a binomial function reproduce the original data. therefore making it complex.

  7. “Here is the rx signal I received, and the same file run through the 700C decoder.”

    Dave,

    The recordings referenced in the quote above were *most* interesting.

    Questions: what audio passband was used for the analog transmission, e.g. 100-2700Hz, and was this matched on reception? Also, how was SNR determined for these tests?

    Comments: I found both messages to be effectively unintelligible. No doubt this is exacerbated by the fact that I did not know ahead of time what the messages are. It is possible that slow phonetics might have been intelligible to me in both cases.

    More importantly, while 700C is state of the art in terms of amateur HF voice CODECs, the analog audio was not state of the art in terms of amateur analog voice processing and noise reduction. It would be most illuminating to hear a third recording made, under the same conditions as the other two, that used the “NR2” noise reduction mode of PowerSDR mRX. The NR2 mode currently represents the state of the art. Having some experience with the use of NR2, I suspect that the analog transmission would have become easily readable.

    73,

    Scott

    1. The SSB audio bandwidth was set by the IC-7200, around 400 to 2600 Hz. SNR around 0-1 dB, measured by FreeDV referenced to 3000 Hz.

      Yes these messages were on the edge of intelligibility, intentionally so. This means some people won’t be able to understand them. This sort of condition is common for HF – I was listening to my friend contesting on the weekend – he had to repeat (shouting) his callsign phonetically 5 times – just to get a callsign through.

      Can you (or any reader) please take the SSB sample above and pass it through some analog noise reduction? For example I am sure Power SDR can be run off line, using a file as input. It would be great to compare. I would imagine analog noise suppression works well on moderate SNRs, then, like most DSP – breaks down at some edge condition like low SNRs.

      As well as post processing the analog SSB, compression could also be added to the SSB tx side.

Leave a Reply

Your email address will not be published. Required fields are marked *