To help debug the Dili mesh network I wrote a simple Google Maps application called Dilimesh. Actually that’s not quite how it came about. One Friday afternoon I just said to myself, “I wonder how hard a mapping application is to write”? When I next lifted my head it was Tuesday and to my surprise and delight I had built some useful software for Dili. Real world hacking is not a linear process, and like so many human endeavours we justify our decisions after, not before the fact.
Anyway one neat trick is Dilimesh continually measures the packet loss to each node, calculating a running average using a simple filter. Blue means 10% or better, red 10-90%, black worse than 90%.
Here is the current state of the Dili Village Telco Mesh Network (click for larger image):
Dilimesh can do other tricks like display the signal strength of signals from other nodes (click for larger image):
This plot shows about 40 nodes, although there are 15 more West of us that aren’t in direct contact with this part of the network. Lemi says that at any given time about 80% of the nodes can be telephoned. Fortunately voice calls tend to make it through through links that won’t support data traffic like ssh, I guess due to smaller packets.
With this graphical display of link quality it becomes very clear which links need strengthening. Lemi, Alipio, and Anders were quite excited to have such a tool available to them. Until now they were “flying blind” when it came to visualising the mesh.
BTW I am not sure what the future of Dilimesh is, I have no strong desire to support it. It started as a fun afternoon project and evolved into something to fix the specific problems were are having in Dili. But it’s there as a lightweight Google maps application for mesh networking if anyone wants to use it.
We would like to improve some of the poorer links to more distant parts of the mesh. Interference from other Wifi networks seems to be the major problem on the longer links. Some strategies for improved links are:
- Use 5GHz kit like Ubiquity M5s to provide high quality long distance links. Hopefully with some configuration we can integrate these point-point links with the Batman mesh network so that routing is automated, i.e. the 5GHz links are automatically chosen as a route to more distant nodes.
- Add directional antennas like the windsurfer to MP01s to improve their ability to reject interference.
- Use tools like Dilimesh, Horst and audio ping to debug mesh links. If you can “see” the problem, you have a good chance of debugging it.
Unfortunately my time in Dili is always limited, so I can’t fix everything while I am there. Also in the spirit of the Village Telco I would really like to see local Timorese guys fix these problems themselves. That’s what the Village Telco is all about – local telephony by local people. So the best I can do is leave some knowledge behind – for example training on how to use the debugging tools and building directional antennas.
Mesh Links Using Ubiquity M5s
I did spend two days trying to configure two Ubiquity M5s to bridge two segments of a Batman mesh network. With some help from the Village Telco community I eventually got it going in the lab. However as we prepared for an outdoor test we some how damaged the Ethernet port of one of the M5s! This is a show stopper in the developing world – another M5s will take weeks and cost many $ to ship here. Anyway, Lemi is very patient, and another M5 is on it’s way. With a couple of grounded Ethernet cables, as recommended by people selling M5s.
Now the exercise of using 3rd party (i.e non Mesh Potato) kit for a Batman link took me about 2 days of messing around (including a few hours of tracking down the dud M5 Ethernet port). Lots of ssh, conf files, vi, ping this, iwconfig-that, email Elektra, wait overnight for the reply, it works yay, oops now after a power cycle it doesn’t, plug the M5 in, oops forgot to ifconfig, I wonder if that mosquito carries Denge Fever, damn it’s hot here. You get the idea. Classic Linux configuration hell, err fun.
Configuring the Batman-M5 link was vastly more complicated than basic Mesh Potato set up that has proven so easy to transfer to local people here. It’s outside the skills of the “moderately geeky guy in a village” we are targeting for Village Telco sys-admins. Although of course it’s possible to train people. To train myself I have documented the steps on the Dili Village Telco Wiki.
We need a plug and play solution for bridging longer links, like snap on directional antennas for the MPs, grid antennas for MPs, or a simple way to use 3rd party equipment. A good example is our stock of Nanostation 2’s flashed with Batman-aware firmware. They work really well, you just switch them on, set the IP and Wifi channel and they act as relays with directional antennas, automatically meshing with the network. In particular no IP routing or other network configuration is required – Batman handles this automatically.
Links using Nanostation 2’s have been effective in overcoming interference over short distances here in Dili. However the Nanostation 2 antenna is only 10dB, we need a high gain/more directional/5GHz solution for strong interference scenarios. In flat terrain with high trees this also means rigid, tall, high quality masts to maintain accurate pointing of antennas and obtain good line of sight links. Much more difficult and expensive than normal Mesh Potato set-up but unfortunately necessary here for quality links across town.
Mesh Potatoes in the Wild
I mentioned the M5 Ethernet port we popped during set up. It made me wonder about the reliability of Mesh Potatoes in the field. Lemi and Alipio haven’t reported any failures in the field to date, and that is in the hot, humid, stormy environment of Timor Leste. Some MPs have been in operation for 10 months. I have been running some betas in my suburb for 12 months, between 0 and 43C.
The hardware and release firmware seems to be stable, only a power outage brings them down. No need for daily reboots or anything like that.
Large Meshes and Omni Antennas
When I first started this project, I had the vision that large (100+ nodes, 2km+ diameter) Village Telco mesh networks could be constructed using mesh routers with omni antennas, with a few Nanostation 2s in the middle. This has a lot of advantages, e.g. ease of set up, low cost, simplified supply chains, low training requirements, single sourced equipment like the Mesh Potato.
However I now feel this vision needs to be modified. At low interference sites, or out to a few 100m diameter it holds true. However on a larger scale such networks are extremely susceptible to interference. The large scale, real-world, geek-maintained, mesh networks I have visited (e.g. Berlin, Cape Town) have a mix of links, many of them using directional antennas. Often there is some sort of backbone based on directional antennas and possibly 5GHz kit, with a few short “last 100m” hops based on omni antennas.
As discussed in the previous section, the Village Telco needs a way to address the need for quality long distance links, but still keep the “ease of use” meme that works so well. For example a person setting up her first Village Telco network needs to know when she needs a directional antenna in her network. Then she needs to be able to set it up without resorting to the Linux command line, vi and ping.
Needed: a plug and play solution for long distance, interference free mesh Wifi links that non-geeks can install.
Future of the Dili Village Telco
The demand for reasonably priced telephony in Dili and indeed throughout Timor Leste is very strong. Within islands a few 100m across the Dili Village Telco network works very well. The end users are very happy. So there is strong motivation to work through the teething issues and expand the network. Lemi and the Fongtil team will work on high quality long distance links over the next few months.
Here is a good example of local demand; while I was capturing screen shots of the Dili VT network and grumbling about link quality Lemi had a visitor – the IT manager for the Rede Feto (Women’s networking) organisation. He spent some time discussing what kit he would need to buy (Nanostations, MPs) to link his 5 sites together. Members of Rede Feto call each other at these same 5 sites all day. What is totally cool is that this IT manager can install all of this equipment himself – the Mesh Potato is that easy to use. We have proven many times here in Timor that local people can install mesh telephony networks with very little training (just a few hours). This is in contrast to the very high skill levels and several months+ training required to install and maintain IP networks for other applications.
Lemi (above, holding the dish) is the IT manager for Fongtil, a support organisation for NGOs in Timor Leste. He has a secured a budget for continued Dili Village Telco operation and expansion. I asked him, “Why are you so happy to invest in the DIli Village Telco?”. He replied, “We only have to invest one time. Then we can make phones calls for free. No ongoing costs. If it works really well, we will expand to other districts.” The fact that Fongtil now wishes to invest it’s own money after our grant-funded project has finished says a lot for the local impact of Village Telco technology.