The Mesh Potato Part 1

In June 2008 I attended the Village Telco workshop in Cape Town, South Africa. Cape Town in June was rainy and cold, however the South African people were really friendly. While in South Africa I also attended the Wireless Africa workshop, however that’s another story for another post!

The Village Telco (and I quote) is an easy-to-use, scalable, standards-based, wireless, local, do-it-yourself, telephone company toolkit. We were in Cape Town to work out how to build this puppy.

Steve Song of the Shuttleworth Foundation pulled together a fascinating team of people from the development, VOIP, mesh networking, and business communities. The team was small (about 10 people) and very “hands on” in their outlook and skill sets. The breakfast and dinner conversations were fascinating, for example funny stories about broken down hotels in some developing countries, and sad stories about the poverty of others.

You can click here to put a face to all the names (thanks Steve).

I had never experienced a workshop quite like this before. Not sure if it was the team of people, the small size, or Steve’s leadership, but we really “fired”. I am generally allergic to meetings – in my previous day job I was infamous for avoiding, hating, and general bad behavior where meetings are concerned. However I managed to stay awake and even attentive through most of the 5 days the Village Telco workshop.

One of the outcomes was the decision to build a little box called the Mesh Potato.

The Mesh Potato in a nutshell

The Mesh Potato is a 802.11bg mesh router with a single FXS port. It is designed to provide telephony via VOIP while simultaneously facilitating a mesh cloud. It is an open hardware and open software design. It will run off a nominal 12VDC, from either a mains supply or solar PV system, and be priced in the range of currently available Wifi routers (sub US$100).

Key features:

  1. Runs B.A.T.M.A.N. mesh routing software, Asterisk, the Speex voice codec, and Oslec echo cancellation.
  2. The target application is mesh routed VOIP networks, in particular (but not limited to) developing communities. An analog phone connects to the potato via the FXS port. When you make a call you potato talks to the potato down the street which talks to the next potato, and eventually to the destination. The mesh network can be augmented via backbone links and connected to the rest of the world via VOIP gateways.
  3. No cell phone towers, no land lines, no big Telcos required. A local entrepreneur can roll out his own Village Telco system using a modest server and a bunch of Mesh Potatoes. Community owned telephony.
  4. The mesh network is self organising and healing, if a node goes down B.A.T.M.A.N automatically re-routes the calls.
  5. We are building custom hardware specifically for developing communities using open hardware and software principles. I would like to push the meme that we can develop custom open hardware devices – no need to accept whatever is available off the shelf. You see most of the work in any appliance is software, so the idea of relying on closed, proprietary, not quite right hardware is obsolete.
  6. The Mesh Potato is open. Really Open. Key goals are to minimise binary blobs, proprietary software and make the hardware open. This means the potato will probably be Atheros based, as we can use the madwifi open source WLAN driver. It also means Speex instead of g.729, and Oslec instead of a proprietary echo canceller. The hardware schematics (at least) will be available. The potato will be a mass produced in very high numbers, therefore the open community now has a chance to set standards, rather than have to play along with “standards” based on closed hardware and software.

For the curious, Steve describes the origins of the Mesh Potato name in this post.

The Shuttleworth Foundation has generously decided to fund the initial phases of the Mesh Potato Development. The Shuttleworth Foundation is involved in many worthwhile projects, such as the Freedom Toaster and kicking off Ubuntu. Potato development will be my main activity for the rest of 2008!

Project Plan

We have divided the project into 5 milestones, M0-M4. We start at 0 because we are geeks, just be thankful we didn’t used binary. We have largely completed the planning stage and have kicked off development. We hope to have prototype potato hardware in early 2009.

Currently, we are working on M1 – Proof of Concept. The idea is to use Commercial Off The Shelf (COTS) hardware like a Ubiquity Nanostation 2 to develop a prototype that demonstrates all of the software components working together. We will develop many of the custom software components early in the project, using a mature hardware platform. Major software components will therefore be relatively mature when integrated into the prototype hardware, saving time and lowering overall risk.

Over the past few weeks I have been experimenting with OpenWRT on the Nanostation 2 – running test programs for Speex and Oslec and characterising the CPU load.

Atcom are very keen to work on the prototype hardware, and be the volume manufacturer for the Mesh Potato. Atcom have been pioneers in supporting Open Hardware products such as the IP04.

Hardware Architecture

Here is a mud-map of the hardware design:

One challenge is how to connect the FXS chipset to the Atheros SoC (glue logic in the figure above). The FXS chipsets require a TDM serial plus SPI bus. The TDM bus is typically a 2.048MHz serial bus that supplies one 8-bit speech sample every 125us (8kHz). The SPI bus is used for control and configuration of the FXS chip set (for example sensing off hook, switching the ringer on).

This glue logic was straight forward for the IP04 as the Blackfin CPU has a rich set of interfaces with good DMA support. The Atheros SoC is not quite as feature rich. It does have a SPI bus for talking to the SPI flash. This only has one chip select line however I figure some of the GPIOs could be used to gate access to other SPI devices.

So we need to build a TDM interface somehow. To reduce interrupt overhead and I-cache thrashing it would be nice if we can buffer 1-20ms of samples (8 to 160 8-bit samples) before we interrupt the Atheros SoC. This interface could be some logic or perhaps even a small micro-controller like a PIC. One other possibility is interfacing the TDM bus to the RS232 UART. The Atheros SoC does have a 16550 compatible UART with some sort of hardware FIFO, however we would need some tests to determine if the CPU overhead of using the UART is acceptable.

Join In

The Mesh Potato promises to be something very special – an open hardware/software, community designed WiFi mesh router with Asterisk and a FXS port. Boy, I’m out of breath after saying that. No wonder we just call it the Mesh Potato. The Village Telco concept that it supports could bring telephony to millions. Please feel free to join our community – by subscribing to the Village Telco Google Group.

40 thoughts on “The Mesh Potato Part 1”

  1. WME/WMM is supported by MadWifi and could be employed even on mesh networks. However I don’t believe it would really be enough. From my experience though the asterisk adaptive jitterbuffer isn’t very good either, but it could probably be made to work well enough for this application. If the mesh can find stable routes, 50-100ms min. buffer would probably even be enough. Also keep in mind speex has dropout recovery and really does a good job constructing audio from bad data streams.

    From reading the documentation I don’t really understand how BATMAN is supposed to deal with one type of problem that you typically have in mesh networks though — links that work great one second then completely die the next, then come back seconds later. I would anticipate the need for some kind of long-lived path quality monitoring in addition to the batman daemon.

    I would anticipate significant problems if you tried to deploy this sort of thing in downtown New York, but you also have to keep in mind though that the places these kinds of things are destined for are places where the idea of interference etc are virtually nonexistent.

    Obviously it be made to work. Commercial products like Sonos music system have shown that you can do audio over mesh wireless remarkably well; the two applications are really quite similar. Perhaps they have some technical details available about the way sonosnet works.

  2. Steve & John,

    Here are some articles on this topic:

    In general, it appears call quality drops significantly within a few hops.

    It would be interesting and exciting to see how it works out in a real world setting, and what kinds of clever modifications must be made both in hardware and software to make this work.

  3. @John: I’d like to know which mesh protocols you have experience with. Most even don’t have a metric that takes link quality into account. Also keep in mind we are not talking about a mobile ad-hoc network, it is a wireless mesh network where nodes are mostly static (on house roofs). By changing the Batman originator interval rate the protocol can also be adapted to the requirements of mobile nodes – at the cost of protocol overhead.
    The idea of the village Telco is to use the mesh for voice traffic exclusively (even though I am not afraid using it for other traffic simultaneously, we are including an Ethernet port in the mesh potato), So prioritizing voice traffic with wireless multimedia extensions is nice to have for the latter case, but not a key issue in the first place. And, as you mentioned, it is already present in Madwifi.

  4. The effects of multiple hops and large meshes will be interesting to experiment with.

    A little anecdotal experience – Elektra and I have spoken using VOIP between Germany and Australia. As I recall, the Elektra was situated on the Freifunk mesh network that is running B.A.T.M.A.N., and IIRC there were several mesh hops to the Internet gateway. Plus I would estimate another 15 hops between us on the Internet. Voice quality was fine. This was just one call, and we were using Skype, however I am sure that the Freifunk was also carrying other traffic at the time. This suggests to me that VOIP over mesh is doable, even if we need some work to maximise capacity, account for jitter under load etc.

  5. Great idea, this is the sort of thing I’d buy just for my home here.

    A couple of questions, which I’m sure you’re going to answer in parts 2 and 3 anyway, but it can’t hurt to ask =)

    Would there be any merit in having this interoperate with the OLPC project’s Mesh technology? (given that the technology is free, I’m not sure that all of their mesh software is) I’m guessing as their requirements for networking wouldn’t include VoIP, their links wouldn’t be good enough to do this, but it may be a way to get a “free” mesh network in some parts of the world. (and give them a head start where there are Mesh Potatos but no OLPCs)

    I assume that this is going to be 100% zeroconf, so actually setting IP addresses would be dealt with. However hooking up to actual Internet links is a little different. I see two different scenarios:
    1. Alice hooks up her computer to her Mesh Potato intending to use it for internet access.
    2. Bob hooks his Mesh Potato up to his home router so he can link this to an internet backbone / VoIP gateway
    In most cases, these could be distinguished by whether you can obtain a DHCP lease.
    So, finally, my question: Is this something that you’re thinking about or am I wandering far off into the wilderness? and would either Alice or Bob have to configure anything?

    Will the phone numbers be automatically assigned or pre-set in some manner? and how will you communicate this to the phone user?

    Riffing off the previous question, would there be any use in packaging this in an actual phone? possibly with a cut down board eliminating most of the … nastiness of actual POTS signalling.

    Also, would it be a useful idea to have FXS simply be your $10 ATA project hooked up to the RS232 port? (for those playing at home: )

    Anyway, this is a great idea, I really hope it takes off – and I love reading your blog, (wireless) networking + telephony == cool =)

  6. David,

    Are you planning on using the same chipset as the one in the Inveneo AP device? Does it have DSP? Does it matter?

  7. Julian – I think the idea is that there will be a Village Telco operator who will take care of things like IP and phone number allocation. The end users will just plug in their potatoes and start making calls. Some sort of pre-paid system might be used.

    Surprisingly, analog POTs technology has a lot going for it. For example the phones are cheap, can be purchased anywhere, and the technology “just works”, and has lasted 100 years.

    Levin – Not sure what chipset the Inveneo AP uses, we are currently looking at Atheros. The DSP is handled by the Atheros processor. One of the challenges on M1 is to make sure that the DSP code (Speex codec and Oslec echo canceller) can execute in an acceptable amount of CPU. Working on that now.



  8. First of all, madwifi is not exactly open-source. It needs the infamous binary HAL for which no source is available. Also, madwifi has absolutely no zero of being merged to the kernel since it uses a quite different 802.11 stack than the officially supported one (net80211 vs mac80211).

    However, there is ath5k, which is a reverse engineered driver for such cards based on the mac80211 stack and there’s also ath9k (also using mac80211) which is an Atheros-developed 100% open source driver for the 802.11n cards.
    Apparently Atheros hired a bunch of Linux developers and is committed to provide full support for their hardware in the mainline kernel.

    So, all in all, going with ath5k (or even ath9k if you decide in favor of 802.11n cards) seems like a better idea.

    Additionally, mac80211 (and ath5k) recently gained support for 802.11s, IEEE’s idea of a mesh protocol integrated with 802.11.
    I haven’t tested BATMAN but perhaps it would make more sense to use something that is an 802.11 standard, even if it’s still a draft.
    FWIW, OLPC is using 802.11s, albeit a different implementation in their card’s (Libertas) firmware — although there were some efforts of switching to Linux’s implementation IIRC.

  9. @Julian: The OLPC implements 802.11s mesh on MAC level. 802.11s is a layer 2 mesh routing protocol, limited to 32 nodes max. However, a OLPC can run Batman on Layer 3 as well (I did that on a OLPC about a year ago), as long as 802.11a/b/g/n ad-hoc mode is supported. (Or any other layer 3 routing protocol).

    Zeroconf is a multicast protocol, which works as long as network devices are link local. A layer 2 mesh protocol can create the illusion that nodes available via multi-hop routes are link local – it can behave like a (lossy, multihop routes in a mesh do have packet loss!) switch. This is experimental (there is a layer 2 implementation of Batman, called Batman-advanced), but most link local (multicast) protocols are not designed to deal with a high packet loss environment.

    @Faidon: 32 nodes is not enough for our project. Also I don’t think the routing algorithms in 802.11s are a good choice. 802.11s aims at something different.

    I’d love to use ath5k, but it is not yet up to what we are aiming for. At the moment it is in the valley of tears. It supports old Atheros chips only, and no SoCs. Performance is bad compared to Madwifi because developers have to re-implement hardware calibration. Getting rid of closed binary blops is great, but ath5k is not yet usable. Looking forward to use the kernel mac in the (hopefully) near future with all Wifi drivers.

  10. @Saritha: My personal experience with VOIP and WiFi is that packet loss is the real killer. As long as the individual links have a fair quality this is not an issue. Especially since unicast packets get retransmitted by the WiFi cards if there is packet loss. The number of retries is user-adjustable for most drivers. With the number of hops and retransmissions the latency increases, of course. The call capacity of the network is mostly dependent on how you choose the jitter buffer size and the length of audio fragments. Audio fragments of 20ms with a good codec are of course terribly small compared to their TCP/IP overhead. So at the cost of latency and audio fragment length the number of simultaneous calls can be increased drastically. Why not use fragments of 60 or 80 ms? I have been living in Bangladesh for a couple of months and the round trip delay via our satellite link was 800 ms at best. The satellite link was used for VOIP almost exclusively! We were able to process 21 simultaneous calls with a total capacity of 224kbit up- and downstream.

    I don’t think that studies which use AODV in a mesh to test VOIP are representative. AODV is a miserable mesh protocol, see:
    VOIP tests have to yield terrible results in such a environment. Also the configuration of the network has a big impact. Was WiFI RTS/CTS used? It is unsuitable for a mesh – no wonder that latency goes sky-high. One of the studies you listed claims that 7 hops are not a problem – and they did mention that they had RTS/CTS disabled – with reason.

  11. I am very sure of the fact alot of peoples well benefit from your the fruit of your project. especially we from developing countries appreciate very much.


  12. Hi David,

    You mention “interrupt thrashing”, which is exactly what the Linux kernel developers tried to avoid by introducing the NAPI for something like a gigabit ethernet card. This is an alternative to the interrupt handler in that several interrupts are “buffered” before calling a handler to process the corresponding data.

    You appear to be doing something similar but in hardware. It may be worthwhile looking at the software solution?

  13. Thanks for the suggestion Asif. I can think of a similar approach that might help – when an interrupt occurs just buffer the samples and return. This will mean only a small amount of ISR code is executed, minimising I-cache flushes. Every n-th interrupt the full processing (echo cancellation, Zaptel code) can be done on the buffered samples (say once every 10-20ms).



  14. David,

    What prompted your choice of the Atheros chipset?
    I agree with Faidon about the non-free aspect of madwifi.
    Any other alternatives?

  15. Hi John – there are a couple of threads on chip sets across at the Village Telco Google Group – I think the reasons was solid ad-hoc mode support, driver maturity, and relatively open source driver. Also pls checkout the Google group “CPU selection” thread and feel free to add your comments.



  16. Hi,

    Nice project.

    I tried to download latest IP04 and IP08. They are not on the web…..

    I would like to use the dual FXO and FXS in a design I am working on so
    up to date schematics and PCB layouts would be nice to have access to.

    Could you please send a link, or get the current links fixed.


  17. The latest IP04 and IP08 schematics are available from IP04 SVN. The trend seems to be not to release PCB designs, however I have published PCB designs for the IP04 prototype (and FXS/FXO modules) which are very close.

    – David

  18. “The trend seems to be not to release PCB designs” …

    Here are my reasons why PCB layouts are not given –
    * schematics must be released so that everyone knows what they are getting, but PCB layout is still a significant hurdle because it entails signal modeling, integrity tests, etc.
    * if PCB layouts are given away, then competition can easily copy, and revenue streams disappear!

    Hence, those being “pseudo-free” hardware. “Free Software” would not be where it is if folks were allowed to look only at software designs and not the actual code.

    Thanks for keeping everything free, and proving them wrong.

  19. I have a post on open hardware coming up that deals with some of these issues. Although it’s a commonly held belief that releasing PCB designs means lost revenue I think the connection between having all the CAD files and making $ is a little more complicated than that. A business is much more than the technology.

    A competent engineer with good PCB tools could design and IP04 type PCB in a few days, so it’s not much of a barrier. A lot of companies would tweak the PCB anyway, e.g. to suit a particular enclosure or to use their PCB tools.

    I have seen many companies consider cloning the IP04 (which has all of the CAD files open) but walk away when they realised it was much more than just running the design down the SM line.

    You need knowledge of the design, skills like how to flash it, compile firmware, debug small assembly errors, and most importantly the complexity of the software means you really need the Blackfin Asterisk community support. Then you need a business wrapped around the technology that can provide support, sales etc.

    We are feeling our way with open hardware (especially as applied to commercial products) – it’s an interesting experiment! The real value of open hardware seems to be as a springboard for other designs.



  20. Hi David,

    Not sure if this was asked earlier on this site …
    Here is it anyways …

    I see that Silabs has a protection circuit between the the RJ11 connector and the Si3201 consisting of fuses and thyristors.
    I see this missing from your schematic!

    Is this really required given that an analog phone will very close to the FXS port and probably has its own voltage protection circuits built in. When would this really be required?

  21. Hello Josh,

    I placed some protection circuitry on the IP04 (and previously 4fx) motherboard, to make is closer to the RJ11. Might not have used the exact Si labs stuff (fuses etc), it varies from design to design.

    Generally it is only mandated on FXO ports, on an indoor ATA it wouldn’t really be required (and is not compliance tested AFAIK). For a real-world PBX application where you might be stringing cables between buildings, yes it is a very good idea. Also for nasty environments (e.g. some developing world places) where storms are common.



  22. David,

    I see it now. You have a ferrite bead on both TIP and RING lines, and a thermistor which increases the resistance between the two lines as the Si3201/wires gets hotter!


  23. Hi Josh,

    It is not a thermistor but a sidactor. The symbol is misleading. The protection is measured between TIP-RING, TIP-GND and RING-GND. David only has the first.

    Hope this helps!

  24. David,

    A question about the PCM interface between Si3215 and Si3201.
    Before data is transmitted from Si3215, FSYNC is sent to Si3201, but when data is received, is an FSYNC being sent by Si3201? or is it only a interrupt based receive mechanism from Si3215 pov?


  25. Sorry, the previous post should be corrected appropriately to read as “PCM interface between cpu and Si3215”. So is the FSYNC pin on Si3215 an input-only pin?


  26. Yes – input only, one FSYNC pulse from the host sets up timing for tx and rx, as well as some internal states of the 3215 I think – it’s required during the reset sequence.

  27. An interesting project and one I intend following closely.

    I live in Cape Town and am aware that even tho the telco infrastructure in SA is well developed it is unaffordable for the lower income groups so the project is much needed. I have worked for a telco and am post graduate IT and engineering educated. I have a few comments to make which I woudl be very interested in having answered.

    1) I’m a professional earning the local currency and a sub 100 $ device is something I have to carefully budget for, that price however is way out of reach of the people who actually desperately need this technology. How are they going to afford this when the vast majority of the population are struggling just to buy food?

    2) Configuration and network setup, if the device is not a true plug and play device i.e connect a POTS telephone and a 12V supply and the telephone number is preassigned to the box then installation will be a huge problem.

    3) If the hardware becomes a success how do you ensure service quality i.e. that when i make a call I will have a 99% chance of getting the mesh network to make the call during peak loading periods?

    4) I would assume that the local MESH network calls would be free but then how would you link to the established telcos, as they would require interconnect fees. This would require a third party and involve prepaid coupons etc. A very expensive setup and very difficult to run effectively if you have limited control over the network eg. a single node is so far away from a relay node that connection is not very reliable – does the user then demand money back ?

    5) Internet access, similar issues to 4 – a third party would need to link to an internet gateway and issue vouchers and install monitoring systems.

    6) Regulatory issues, I have no doubt the local telco regulating authority would require a licence at some stage, and they have not done a very good job to date at enabling cheap internet access to date ( or cheap telephony services either).

    I look forward to your feedback and hope these issues I have raised can be simply solved, I’m moving to Europe and wish this device would function there too – good luck and I’ll be keeping a close eye on this project. I do wish you would continue with the cheap $10 adapter tho.

  28. Hi Dwight,

    These are good questions – although I could field most of them can I suggest you re-post them to the village Telco Google group? That way you will get better answers, and also reach people in SA who are working on the project.

    Re the $10 ATA the mesh potato is a close relation – a lot of the ideas that went into the MP originated in the $10 ATA project. Actually the M1 work we have just completed for MP can be used directly on the $10 ATA (Asterisk/Speex/Oslec and BATMAN mesh routing running on Atheros).

    I would like to see the $10 ATA move forward – a simple, open hardware gadget to turn any wifi router into a FXS VOIP device would be very useful. However it would be really nice to have some other people involved in the $10 ATA – many people have expressed interest, but I have had zero contributions.



  29. Hi David,

    Very interesting project (this one and your others). Just curious, why the Atheros SoC instead of a Blackfin-based design?


    PS – On a related note, here’s a chip you might be interested in knowing about (if you don’t already): (there are other flavors as well).

  30. Hi Eric,

    Compared to the Blackfin the Atheros SoC has integrated wireless, a MMU, and runs OpenWRT and MadWifi. The MMU means much less RAM is required and hence a lower bill of Materials. The Atheros SoC is however much slower than the Blackfin, but adequate for single channel processing and mesh routing.

    Thanks for the Zarlink FXO/FXS chip link.

    – David

  31. Interesting ideal ,

    However my previous company had been involved in a similar project with big telcos in US and Europe .. In US is called the Ubicell while referred as femtocell in europe. The purpose is just to bring $$$ to the fixed landline operator and reduced the operator’s liability to install expensive base station.

Comments are closed.