FreePBX for Embedded Asterisk

This post tells the story of porting a LAMP stack application (FreePBX) to an embedded Asterisk system running uClinux on the Blackfin processor.

A few months ago I noticed that the Astfin guys had managed to get sqlite and PHP running on the Blackfin. Pretty impressive: a SQL database and PHP running on a uClinux box. This runs against conventional wisdom – embedded systems that run uClinux typically have a small amount of storage and use simple CGI based GUIs written in C, and a tiny web server like boa.

Could a FreePBX port to the Blackfin be possible? Is it really possible to build a LAMP application on uClinux, running on hardware that can (at minimum) be built with just 3-4 chips (CPU, flash, RAM, Ethernet)?

FreePBX is the popular Asterisk GUI, used in many Asterisk distros such as Trixbox and PBX-in-a-flash. A little investigation of the FreePBX code revealed some prototype support for sqlite3 (it usually runs with MySQL). I mentioned the idea to the Astfin guys, who also decided to start working on a port. So for the last month or so a loosely knit group of developers have been working on a port of FreePBX to the Blackfin. Sort of working in parallel, but bouncing a few ideas off each other and sharing our code.

We have now reached the stage where we have some alpha functionality – the FreePBX GUI screens work, we can set up extensions, save them to the database, and make calls. There are a few bugs, and I would like to do some more work to reduce memory consumption. But I am convinced that FreePBX on uClinux is possible.

Here is FreePBX running on an IP04. The Nokia palm top seemed appropriate!

I am still amazed that we can run FreePBX on a platform that can be built with just a few chips and consumes just 3W!

FreePBX is a big, chunky, web based application written mainly in PHP. The GUI screens talk to a SQL database and Asterisk via the Asterisk Manager interface. Every time a call is dialled, a PHP script is run via the Asterisk AGI mechanism to compute parts of the dialplan. This is heavy weight and scary for a little uClinux box!

The first challenge was to get sqlite3 support working. There was some partial support however it in needed updating for the latest FreePBX version. In particular the PHP 5 support for PEAR/DB had to be rebuilt. PEAR/DB is a database-independent backend for SQL databases. This wasn’t too hard, although as I haven’t got much experience in PHP or SQL (or FreePBX for that matter)) I had a learning curve to climb.

x86 FreePBX Sandbox

To make life easier I set up an x86 development environment for FreePBX. I figured it would be much easier to get sqlite3 support running on an x86 that messing around on the embedded target. However after I had a few goes at installing FreePBX on my Ubuntu laptop I noticed that it did a lot of messing around with the configuration of my laptop and my root file system. It also has a lot of dependencies that weren’t always handled nicely by my system, for example a certain version of a library might not be available as a pre-installed Ubuntu apt-get package. Due to my inexperience with FreePBX I got myself into configuration hell a few time (e.g. multiple copies of PHP) before I finally worked out how to get a basic FreePBX install working.

So to work on FreePBX on my x86 laptop I figured out a different way that I borrowed from embedded systems. I built up a “sandboxed” development environment for FreePBX where I compiled all of the applications I needed (and the exact versions I needed) from source. This build a simulated “root” file system that doesn’t interfere with the rest of my x86 laptop configuration. A single Makefile handles the whole thing, downloading the source tarballs and building PHP, lighttpd, sqlite3,and finally running the FreePBX installer.

Here is what the freepbx-sandbox directory looks like after everything is built:
[david@bunny freepbx-sandbox]$ ls
dl lighttpd-1.4.18 package.xml root
files Makefile patch sqlite3-0.5
freepbx-2.4.0 Makefile~ php-5.2.5 sqlite-3.5.6
freepbx-2.4.0-orig package2.xml README.txt xdebug-2.0.2
[david@bunny freepbx-sandbox]$ ls root/
bin etc include lib man sbin share var www
[david@bunny freepbx-sandbox]$

The Makefile takes care of all the intricate steps required for a manual FreePBX install, and also applies the patches I required for sqlite3 support. To make FreePBX run in my sandbox the Makefile also tweaks a few hard coded paths (like changing /etc/amportal.conf to /home/david/freepbx-sandbox/root/etc/amportal.conf) using sed. This means I get a consistent FreePBX development environment every time, without any manual install steps (one make command takes care of everything). More information of the freepbx-sandbox idea is available from SVN.

The sandbox idea is borrowed from the buildroot technique used for embedded work. For embedded systems we often download/patch/cross-compile applications and build up a root file system for the target, that is then burnt onto the embedded target system’s flash.

It was nice to find some synergy between embedded and x86 work.

Memory and Big uClinux Applications

Once the basic sqlite3 support was running on an x86, it was relatively straight forward to get the FreePBX GUI screens running on an IP04 target. The lighttpd web server was used rather than Apache to save memory – we don’t really need anything as heavy as Apache just to serve a few GUI screens to one person. Lighttpd was ported to the Blackfin by Mike Taht.

We discovered a few problems straight away – after a few minutes the system would run out of memory and the web server would core dump. The FreePBX Admin screen is pretty heavy duty for our little embedded system – it updates each second which involves the web server starting up an instance of PHP, loading and compiling a bunch of PHP code, connecting to the database, connecting to Asterisk, doing a “df” to determine a few system stats, and rendering the status page.

Loading PHP all the time is hard on a uClinux Blackfin system. An application that big is stored in NAND flash, which has a relatively low bandwidth. PHP is big – it typically uses 8M of the IP04’s SDRAM at run time. The Blackfin doesn’t have a MMU, so when PHP says “I need 2M”, then uClinux gives it 2M of physical memory. Your x86 would just allocate one 4k page and let the MMU allocate the rest of the 2M as (and if) it was needed. So as well as having a relatively small amount of memory (64M), the memory is often used inefficiently.

I have a suspicion that under uClinux most of the memory allocated by big applications like Asterisk and PHP is probably all zeros and remains unused (but unreclaimable).

uClinux also has memory fragmentation problems. As explained above, if you need 2M of memory, the operating system actually allocates 2M of physical memory. However this allocation may cause the memory map to become fragmented, as it means there is a “hole” in physical memory of 2M which limits the size of memory chunks that can be allocated either side of the 2M chunk:

Memory fragmentation is particularly a problem when you have many large allocations. It can lead to Out Of Memory (OOM) errors even when there is plenty of actual memory left. In the above example 3M of memory is available, however if we try to allocate a 2M block we will get an OOM error. Available physical memory may be too fragmented to allocate the continuous blocks needed by an application.

Mike had the idea of using the fast CGI feature of lighttpd and php-cgi. This means that php-cgi remains memory resident, removing the need to load PHP all the time and avoiding memory fragmentation issues (one big allocation at system start up is generally better than many big allocations over time). This worked pretty well, we could now run the Admin screen without any memory problems. You can see the php-cgi process running below on an IP04. It stays up all the time and communicates with lighttpd via a socket:

root:~> ps x
PID Uid VSZ Stat Command
<snip>
39 root 602 S dhcpcd
62 root 13910 S asterisk -f
64 root 6610 S php-cgi -b 127.0.0.1:1026
66 root 901 S lighttpd -D -f /etc/lighttpd.conf
68 root 1386 S -/bin/sh
69 root 13910 S asterisk -f
70 root 1242 S /bin/syslogd -n
72 root 1242 S /bin/klogd -n
73 root 13910 S asterisk -f

Note the php-cgi process is using 6.6M of memory, even when it is not doing much (i.e. just waiting for a connection).

Apply Configuration Changes

At the top of the FreePBX screen there is a clickable area marked “Apply Configuration Changes”:

After you make a few changes to the FreePBX configuration (e.g. setting up a SIP extension) you click on this area and the configuration information is loaded into Asterisk. To do this the FreePBX php-cgi process execs another PHP script called retrieve_conf. So we have two PHP instances running at the same time, which chews up about 30M total (15M each) of memory on our little uClinux system. Gulp.

This was proving too much for our IP04, when we clicked on “Apply Configuration Changes” and ran retrieve_conf we were running out of memory and copping the inevitable OOM error! A bit of investigation showed that retrieve_conf actually runs a lot of code (for example connecting to the SQL database, connecting to Asterisk) that the parent php-cgi process has already initialised. So with a bit of PHP hacking I re-arranged the retrieve_conf code to be an included function of the main php-cgi script rather than having to exec a whole new PHP process.

This meant we could run “Apply Configuration Changes” in around the same sort of memory as the regular admin GUI, saving a crucial 15M at run time. Faster, too. So now we were well on our way – we could run the FreePBX GUIs, use sqlite3 as the database, and apply the configuration changes to Asterisk.

Next step – lets make some calls!

AGIs for each Call

While I was working on the “Apply Configuration Changes” functionality the guys on the FreePBX forum pointed out that FreePBX runs a PHP script for every call dialled via the Asterisk AGI mechanism. This means that as Asterisk works through the dialplan:

exten => s,1,GotoIf($["${MOHCLASS}" = ""]?dial)
exten => s,n,SetMusicOnHold(${MOHCLASS})
exten => s,n(dial),AGI(dialparties.agi)

It shells out to a PHP script like dialparties.agi. This means that another instance of PHP must be exec-ed from Asterisk when each phone call is dialled. This, coupled with the fragmentation issue, had caused some problems on some earlier attempts to port FreePBX to the Blackfin.

To test the affect of PHP AGIs I set up a little automated AGI test to make 10,000 calls using the dialparties.agi script. This worked really well, no evidence of any fragmentation issues and all of the calls were completed OK.

I think the main problem with FreePBX AGIs on the Blackfin was not so much fragmentation as file buffering. You see by default uClinux on the Blackfin (up the the uClinux-dist-2007R1.1-RC3 releases at least) can use up to 100% of its memory to buffer files. However you can throttle this with something like:
echo 10 > /proc/sys/vm/pagecache_ratio
which sets aside a maximum of 10% of system memory for file buffers. If you leave this at the default 100% after system memory is gradually consumed to a point where large apps like PHP can’t load.

The AGI still sucked up about 9M every time it ran, but at least the system was stable. Note that dialling uses less memory than the GUIs (9M compared to 15M), this is because it loads and runs much less code.

Note: in the latest Blackfin uClinux-dist-2008 I understand that the memory used for file buffering can now be reclaimed if required automatically. However at the time of writing I still need to test how effective this is with FreePBX.

Putting the IP04 on a Diet

Next step – we needed to free up some more memory so that the GUI screens and dialling AGIs could all run happily at the same time. I tried a few tricks:

  1. Shrinking the default 8M rootfs partition down to 3M.
  2. I realised that the AGI scripts written in PHP only needs a subset of the PHP features required by php-cgi. So I tried compiling a small version of PHP (1.7M dynamically linked, c.f. normal 3.4M static version) and tried using that for the the AGI.

These steps worked well, and we now have a minimum of about 9-10M free when simultaneously looking at the admin screen (which refreshes every 1 sec) and dialling calls. Previously free memory was dipping down to just 3-4M, and it seemed to bounce around perhaps due to fragmentation issues.

FreePBX Installation on the IP04

On embedded systems running any kind of ‘make install’ process is uncommon. Usually the host system handles ‘installation’ at build time, and you flash the target with an image of the complete file system. This works OK for regular C applications where you can tell ‘make install’ to write the binaries to the target’s root file system that you are building on the host.

However FreePBX has an installer written in PHP, and it assumes that it is running on the target system. While it would be possible to manually do the install on the host, I decided to try the rather novel step of running the installer on the target embedded system.

So I actually download the entire 4.3M FreePBX-2.4.0 tar ball to the IP04, then run the install_amp script on the IP04! What makes this possible is the large amount of NAND flash (256 MBytes) available on the IP04. It’s more like using a hard disk based system than your typical embedded system.

To make life easier packages (using ipkg) were developed for all the FreePBX components. This actually makes FreePBX easier to install on the Blackfin than on some x86 platforms! The install procedure is described in detail here but is something like this:
root:~> ipkg install zaptel-spi asterisk
root:~> ipkg install freepbx
root:~> cd freepbx-2.4.0
root:~> cat SQL/newinstall.sqlite3.sql | sqlite3 /var/freepbx.db
root:~> ./install_amp

As the port is still alpha, there are currently some extra manual install steps which will eventually be taken care of by the ipkg install scripts. Here is the installer doing it’s thing on an IP04:

Much like an x86 installation. Also by using the FreePBX installer rather than a manual build-time installation we can more easily track any changes in FreePBX.

Conclusion

We have an Alpha version of FreePBX ported to the Blackfin, and running nicely on an IP04. We can configure using the GUI screens, apply changes to the system, and make calls. Still lots more testing, a few bugs, and some fine tuning to go to make FreePBX truly usable and reliable on the Blackfin. But it’s a good start.

Curiously, I am not quite sure why I spent the better part of a month trying to do something crazy like get FreePBX running on an IP04. I would like to have a logical reason but I can’t for the life of me think of one. I don’t even like GUIs much, and would rather use conf files! I just realised it was possible one day and it seemed worth investigating. Before I knew it I was “sucked in” to the project and working on it in every spare moment.

But it was interesting. I had never done any work on big web based apps, and had never used PHP or done much work with SQL. A chance to push the limits of the Blackfin and uClinux. Attempting things that a uClinux system was not meant to do. Guess that was the main motivation. Once you get started on a project – something compels you to drive it to some sort of finished state (Alpha functionality in this case).

Links

I documented the development of FreePBX for the Blackfin on this Blackfin Asterisk Forum thread.

FreePBX forum thread on porting retrieve_conf and dialparties.agi testing.

FreePBX installation documentation for the IP04.

FreePBX sandbox documentation.
pay 60 loans dayrates 20 loan 80uk loan home in 90home 90 uk loansloan land equity acom loans aaaaa automobile loansloan acceptance companyloan savings account cash fastaccount loans payday savingssex function and acupuncture3-way sex1960 s hardcore pornamatuer sex animalof teenagers ambassadors americanamateur made pornporn star americas hot nextdegree 2nd sexual assault Maphome program accredited schoolloan building 228 creditpay credit accept month card minimumaccredited information computer in technology collegescards today remotely credit acceptbusiness small credits for cards acceptingprograms bachelor degree accreditedmerchant card credit account washington Mapmp3 lahdo ablahaddlaczego mp3 003mp3 semester abroad law judeminded absynthe mp3mp3 maia abelhamp3 abhirami andhadhimp3 ableton djingsaraca 004 mp3 inima Mapbandh mp3 aankhe karkelara atb mp3 9pmshrivastava aadesh mp3babi aala mp3mp3 record rpm 78 archiveskqrs show morning 92 mp3khuli aankhen mp39th at mp3 pine Mapluvana movies free carmenmovies gang bang freemovie taboo seriesmpeg lesbian movies freemovies of cocksuckingmovies xxx freethe movie dancing dirtyonline movies adult Map

18 thoughts on “FreePBX for Embedded Asterisk”

  1. Oh, and, Debians have a nice little command called debootstrap, to easily create a chroot environment:

    debootstrap –include=lighthttpd,php5-cgi,asterisk lenny /tmp/test

  2. Thanks Tzafrir, the debootstrap feature sounds great. Not sure if it does everything I needed, however.

    The continuous running of AGIs is I think covered in the “AGIs for each call” section above.

  3. That is one cool picture!

    One option you might look into is “precompiling” the php code. This is where the PHP parser is run over the scripts and the memory structures the parser builds are dumped to disk. This means that you could totally remove the php parser from the php build (assuming no eval/exec).

    I’m unsure of the status of projects which do the above, it was a hot topic about 4-5 years ago back when php4 was just released.

    Keep up the cool work!

  4. Thanks or the suggestion and encouragement Mithro. I understand that there is some sort of PHP extension for caching compiled code, but not (AFAIK) a mature PHP compiler.

  5. This comment came in via email from Diego Iastrubni, I am posting it manually as I thought it was great:

    Hi,

    So great someone is keeping on the work I started a year ago in Xorcom :)

    You nailed all the exact problems (and you avoided a lot of them since apparently the sqlite3 support is still being somehow maintained).

    If I may quite myself from the #freepbx-dev channel:
    [20:15] sweet! stupid rowtel doing the impossible, so impressive!

    I am really impressed. Human kind usually evolves by stupid people doing impossible things, instead of smart people doing what they were supposed to.

    Running a full PBX with 3W? Reduce the cost to $100? Impossible

    😉

  6. Great work, truly awesome! I have been investigating PBX hardware and software for the company I helped found. We develop and provide IT solutions for SME’s plus back them up with free onsite and offsite service for 3 years with a 4 hour fix guarantee. With that sort of guarantee our stuff has to be ultra reliable so I have conceptually designed the perfect PBX for SME’s and this project has given me great hope that my pie in the sky, design of perfection, may actually work. Thank you very much.

    You are definitely doing something worthwhile that could be used in the corporate environment as well as your target market (low cost users).

    I will be visiting this site a lot more now.

  7. Hi Nathan,

    Thanks for your kind words and encouragement. One of our “markets” is deployment to remote location in developing countries, so reliability and remote monitoring is very important.

    Cheers,

    David

  8. Is hacking a phat PHP app the right approach?

    Perhaps taking ideas from the web app and creating something a little more targeted to the platform with yield better memory usage, be faster and more reliable?

    How much memory do you have to play with there?

  9. Kim – your approach of a light GUI is generally regarded as the correct one for small Asterisk systems, for example the AsteriskNow GUI. Some would even say the FreePBX approach of using PHP to dial every call is overkill for a x86 system.

    However if it wasn’t hard, and generally regarded as not possible, it wouldn’t be fun :-)

    A few years ago it was generally regarded as impossible to run Asterisk on a uClinux system, but here we are…..

    We have 64Mbyte to play with, but it is often inefficiently used.

  10. Thanks for not only taking on the tricky, but writing it up.
    I am looking at PHP, SQLite Lighttpd on an ARM and your notes give me some idea of where the problems will be.

  11. Wow! This is great stuff. Cheap to get, cheap to operate and open! I am amazed at the amount of effort you have put into this. You are a gift to the community! Thank you.

  12. hi …………..
    i followed the FreePBX installation documentation for the IP04.
    when type this step . i am getting some error.(ie) in asterisk folder ,folder bin not found……….
    root:~> cp amp_conf/bin/retrieve_conf /var/lib/asterisk/bin/
    root:~> cp amp_conf/bin/module_admin /var/lib/asterisk/bin/
    plz some one help me wht to do……….

  13. In these days I had a problem with lighttpd and php in fastcgi mode on blackfin: php dies after a while, with no error or whatsoever. You seem to have encounter the same problem.

    I have done some research and found that, php is dying after exactly
    500 connections.

    That’s normal behaviour for php in fastcgi mode.

    Normally, on a MMUfull system, there would be a php process, spawning
    n childrens (based on PHP_FCGI_CHILDREN environment var) each of them handling PHP_FCGI_MAX_REQUESTS, which by default is 500.

    On a noMMU system, php is not able to “fork” childrens, so it can handle only the first 500 connections.

    In my case I solved by setting PHP_FCGI_MAX_REQUESTS to 0 and
    now php never dies. I didn’t tried to patch php to use vfork
    instead of fork, ’cause it is probably beyond my abilities,
    but that may be another solution … even if this may lead
    to more memory fragmentation.

  14. How you ported lighttpd to blackfin?
    I get error when following code executes:
    if (-1 == (fd = open(fn->ptr, O_RDONLY | O_BINARY))) {
    return -1;
    }

    f->start = mmap(0, f->size, PROT_READ, MAP_SHARED, fd, 0); // here!!!!

    close(fd);

    if (MAP_FAILED == f->start) {
    return -1;
    }

    in cause no mmu devices do not support MAP_SHARED option.

Comments are closed.