Over the past week I have been assembling, debugging and testing my own BlackfinOne board:
During this time I have had a lot of help from the BlackfinOne community, in particular via ongoing chat sessions with Wojtek (pronounced Voycheck) Tryc. Yesterday, I booted uClinux on hardware that I had helped design, soldered and debugged myself. A tremendously satisfying experience.
Building hardware is a bit like software – you go quickly through stages of construction; get stuck for a day or so on a tough bug; then rapidly scream through construction again until you hit the next bug. By documenting some of my experiences I hope I can help other people with similar projects.
The BlackfinOne is a uClinux board based on the Blackfin DSP/RISC processor. There are several features of the BlackfinOne project that make it a great project for hackers:
- The design is open as in GPL. Free as in freedom/libre. Anyone can use, re-use, and modify the design, just like open source software.
- The BlackfinOne has been designed using the gEDA open source CAD tools. This means the design can be edited without needing expensive proprietary CAD software.
- Quite remarkably, it is implemented (and runs well) on a two layer PCB. To run a high speed digital design (it has a 400MHz clock and 133MHz bus) on a two layer board is generally regarded as impossible. Standard engineering practice says that you must have ground plane layers to ensure signal integrity and to make routing the PCB possible. For example the Blackfin STAMP designs uses 8 layers. The trick that the original designers (Ivan Danov and Dimitar Penev) used was to place the high speed SDRAM immediately behind the Blackfin chip, on the opposite side of the PCB. This minimises the length of the nets, ensuring signal integrity despite the absence of a continuous ground plane. Using two layers makes low quantity fabrication of the PCB cheap and easy for hackers compared to a multilayer PCB.
- A non-BGA version of the Blackfin was used to make soldering easy for hackers.
- It is based around the Blackfin community (sponsored by Analog Devices, the makers of the Blackfin chip) which has very good uClinux support and a range of open hardware building blocks (such as USB and other interfaces) for the Blackfin.
The initial BlackfinOne V1.0 design (by Ivan and Dimitar) first appeared in mid 2006. This design had SDRAM, Flash, and serial support. I was interested in using it as a DSP motherboard for my telephony work so I (along with several other people) helped add dual Ethernet ports and a USB port to create the BlackfinOne V2.0. Many other people (both commercial companies and private developers) have also contributed to the hardware and software side.
Several BlackfinOne V2.0 boards have now been made and brought to life by people all over the world. For the remainder of this post, when I say BlackfinOne (or bf1 for short) I mean BlackfinOne V2.0.
The schematics for the bf1 are useful to understand the rest of this post.
Building an Igloo JTAG cable
To bring up the BlackfinOne (for example to flash the boot loader) a JTAG cable is required. I built mine using some spare parts and a spare PCB I had laying around. I built a copy of the Igloo JTAG cable. My Igloo is not a shining example of hardware construction:
By the way I obtain the 3.3V power for the cable from the board being tested/flashed, rather than using the 5V to 3.3V regulator suggested in the Igloo design. A nicer version of the same Igloo circuit was constructed by Wojtek (pronounced Voycheck) Tryc:
I tested my Igloo using a BF533 STAMP board. The idea was to test the unknown hardware (my JTAG cable) using known working hardware (the STAMP). To drive the cable I used a version of the jtagprog software that has been modified slightly for the BF532 processor and BlackfinOne platforms.
It is important to use the CVS version of the jtagprog software as this has been modified to work with the latest versions of the BF532 chip. If you use the non-CVS version you may get a “stepping error”, which means the software doesn’t recognise this version of the BF532 silicon.
OK, so I tested the cable on a BF533 STAMP board and it seemed to work OK:
jtag> cable parallel 0x378 IGLOO
Initializing Excelpoint IGLOO JTAG Cable on parallel port at 0x378
IR length: 5
Chain length: 1
Device Id: 00110010011110100101000011001011
Manufacturer: Analog Devices
jtag> initbus bf533_stamp
jtag> detectflash 0x2000000
The last command “detectflash” gave me lots of information on the STAMPs flash chip. As a further test I also used the cable to re-flash a BF533 STAMP with a new version of u-boot. This proved that despite the rough construction my cable was OK (thanks Wojtek for this suggestion).
Other JTAG cables can be used as well, for example other developers have successfully used the JTAGblue cable which emulates very popular Xilinx Parallel – III.
For reference here is an example of a jtag session from a working BlackfinOne. However it took me a while to get that far, as described below!
Tools and Tips for Surface Mount Assembly
The BlackfinOne uses some fine pitch surface mount chips and resistors and capacitors in the 0603 size range. So you need to think a little bit about assembly.
To load my BlackfinOne I used a stereo microscope like this one in this picture (being used by my wife Rosemary whose brief geeky phase is regretfully now over):
I bought mine new for about $700 but there are very nice ones available on e-bay for about $100 up. A much cheaper alternative is an illuminated magnifier:
But trust me – a stereo microscope will change your (soldering) life once your buy one, it really is worth the investment. My eyes aren’t that great but I can work all day under the microscope with no problems or eyestrain. Mine is variable zoom 10 to 40x but I find even 10 x is just a bit too much and I never use the variable zoom. So I suggest fixed magnification in the 5 – 7x range is just right, however it’s a personal choice.
Wojtek bought this microscope for around US$100. He finds the fixed 15x magnification perfect for him:
If you can, get a microscope that has circular illumination all around the objective lens. Mine just illuminates from one direction which gives me shadows so I have to rotate the work a lot. For US$45 Wojtek found this great LED based circular light that clamps onto his microscope:
Here is a view through the microscope of one edge of the Blackfin chip. A sewing needle is in the picture for comparison.
Here is a similar image from Wojtek, through his microscope. You can just see how much fun we are having with the toys we bought for this project!
I found Darrell Harmon’s notes on surface mount soldering very useful. I bought the tweezers and flux that he recommends from Digikey and also strongly recommend them. The flux makes a very big difference in surface mount work, I have never really needed flux previously in through hole work. The flux really makes the solder “leap” onto the parts being soldered.
I also use and recommend the same procedure Darrel suggests for soldering 0603 parts; blob of solder on one pad, then slide the part in with the tweezers. Thanks Darrell.
Wojtek has also done some research on soldering irons, from a recent email:
There is 2 leaders: the Hako 936 and Metcal PS-800. The Metcal is very small, light (!) and reliable. The tips are hot in just few seconds, and are available in variety of sizes and temperatures. For most I use 650 tips, which are designated for non-RoHS components, for ground planes and headers, I use 750 – 1.5mm tip which makes it much easier (750 is RoHS temperature). The Metcal does not have a dial to change a temperature, you change it by replacing tips (2 seconds, pull and push operation). The Metcal is available from DigiKey at very competitive price.
Flux Removal is Very Important
As Darrell notes that Digikey flux is conductive, so you really need to get it all off ASAP after soldering or it will cause problems. For example I added some flux before soldering a 10M resistor (part of the bf1 clock circuit). After soldering, the resistance across the resistor measured only 340k! Some vigorous scrubbing with the flux remover and brush fixed it, and the 10M resistor once again measured as 10M. In one or two cases I have even had to remove parts as the flux was trapped underneath and not easily removed.
I use a spray can of solvent with a little brush attached to the nozzle for flux removal. Wojtek actually gave his BlackfinOne a bath (full immersion in water) with good results. Wojtek also uses this flux remover:
Both Wojtek and I have fixed many problems that were caused by flux left on the board. So Scrub-Scrub-Scrub that flux away!
It is critical to remove flux as soon as soldering is finished. Different methods could be used, solvents and a water bath both work great. But leaving flux on PCB for hours or days could potentially affect a board in a very negative way (possible even ruining it) – this stuff contains acid and is corrosive!
Initial Assembly and Smoke Test
I started with a bare PCB board, kindly supplied by Jeff Knighton:
I first loaded all the surface mount passives, then Blackfin chip, SDRAM, Flash and other chips. By loading the surface mount parts first the board was nice and flat and so easy to work with, especially under the microscope I use for surface mount parts. I loaded the large and bulky through hole passives last, as they prevent the board from sitting flat, hindering surface mount work a little.
My approach was to load various sections of the board progressively. For example I first started with the CPU and memory but left the Ethernet and USB ports unloaded for the initial tests. In hindsight I am not sure if this is a good idea, it is easy to miss something this way (like a pull up resistor you didn’t think you needed at this stage of construction).
If you are loading a V2.0.0 PCB it is important to read and fix BlackfinOne PCB Errors, this only takes about 10 minutes. These problems will be fixed on later PCBs.
After loading the CPU/memory section of the board here are the initial “smoke test” steps I when through:
- I checked my parts orientation against the reference photos of the BlackfinOne, for example I made sure that the writing on my chips faced the same way as the photos. Especially that damn expensive 64M SD-RAM chip.
- Before powering up I checked the resistance between the 3.3V rail and ground. About 300 ohms, which seemed OK. I would have been worried about a dead short.
- The next step was to apply power and sniff for that familiar smell of molten silicon. I used a variable power supply first with the current limit set to 100mA. Just in case I had done something stupid (a common occurrence). Mmm, no smell. Feel the top of all chips, no burnt fingers. Excellent. A better start than most projects.
- I measured the 3.3V rail to make sure the switching power supply came up OK. Check.
- Measured the 1.2V rail – this was present which proved that the Blackfin chip was doing something, as the Blackfin contains the switching power supply circuit for the 1.2V rail.
- With the scope I checked that the 10MHz system clock was present. Check, another sign of Blackfin life.
- Check reset line is H. Ooops, it’s stuck L. That means the system will be held in reset forever!
I took me a couple of hours to find the reset problem. The capacitor C53 that sets the duration of the reset pulse wasn’t fully charging due to some residual flux on the board. I gave the reset part of the board a good scrub with the solvent and reset started working OK.
Programming the CPLD was fairly straight forward – I used the Verilog file bf1_cpld.vl from the hw-0.1 CVS repository. This needed to be renamed to bf1_cpld.v for my Xilinx synthesis tools to understand it and don’t forget to include the pin locking file bf1_cpld.ucf when you synthesise the design. Free (but not open) synthesis tools are available from Xilinx for Windows or Linux. I used an ancient Windows version and a serial JTAG cable to program the CPLD.
To save time, you are welcome to use my xlinx_bf1.jed file, just program the CPLD using your Xilinx JTAG cable.
The next step was to connect the JTAG cable to my BF532 and try to flash the board. This is a “nervous” step – it’s the first time you actually get to see if the Blackfin chip is responding via the JTAG port. It’s also where you find out if you have soldered any memory chips on backwards.
The jtag tools detected the BF532 OK:
IR length: 5
Chain length: 1
Device Id: 01010010011110100101000011001011
Manufacturer: Analog Devices
Note that the BF532 is detected as a BF533. This doesn’t seem to matter in practice.
However when I tried the next few steps:
jtag> initbus bf532_bf1
jtag> detectflash 0x2000000
It couldn’t find the flash chip.
My first suspicion was the CPLD, maybe I had messed up something like the pin locking and the control signals like AOE and FLASH_CS weren’t getting through to the flash chip. So I checked the control signals by watching them on the scope while I tried to read words to the flash:
jtag> peek 0x20000000
The signals were asserted when I did the read which suggested the CPLD was OK.
I then checked out the flash data sheet to work out how it should be responding. The flash chips have state machines built in, you can send commands to them and they respond in certain ways. One of the commands can tell you all about the flash chip. This is what JTAG tries to send when you run the “detectflash” command.
You can generate these commands and read response to the flash chip using the jtag peek/poke commands. For example on the BF533:
jtag> poke 0x20000055 0x98
jtag> peek 0x20000020
bus_read(0x20000020) = 0x00000051 (81)
Writing 0x98 to 0x55 on the chip (mapped to 0x20000050 on the bf1) puts the chip into “common flash interface” mode. In this mode, address 0x20 should always return 0x51 (the letter Q in ASCII). This is a good way to test if the Blackfin is talking to the Flash chip OK.
The same flash command/response on the bf1 showed just 0xff responses, which suggested either the command wasn’t getting through, or something was wrong with reading the response. I had already checked the control signals, so the next step was to check the address and data bus. Damn. This meant 32 nets to manually check.
To check the electrical connections of each net I used two methods:
- Under the microscope, I applied gentle pressure to each pin with a sewing needle. If the pin moved when I applied pressure, it would indicate a bad solder joint. This check didn’t find any problems.
- I then tested the continuity of every solder joint on the major chips using my multimeter. I placed one probe on the top of each pin, and the other on the pad it was soldered too. This is tricky to do on fine pitched pins, so I improvised some fine tipped probes (see below)!
While I was checking some pins on the SDRAM I accidentally touched two adjacent pins (pins 1 and 2) and my multimeter went “beep”, indicating a short. Hmmm, that doesn’t look right. Turns out that the D0 net was shorted to VCC, which was why the flash chip wasn’t responding. Any commands were being mangled as D0 was stuck H.
I spent an hour trying to work out why. It could be a soldering error or a PCB fault. However the track in question lives mainly under the the Blackfin and SDRAM so there was no way to check it visually without removing these chips, something I don’t have the tools for.
I used my multimeter to measure the resistance of the short. It was 0.4 ohms between VCC and D0 near the SDRAM, but 0.8 ohms between VCC and D0 on the flash chip. This suggests the short was nearer the SDRAM, right where I couldn’t get to it.
Fun with Burning Out Shorts
One fun technique I tried was to burn out the short. I have used this technique before on production PCBs that have shorts. You apply a high current (Amps), low voltage (say 0.5-1V) across the short and try to burn it out. Literally. You sometimes even hear a nice crack. It is safe for the logic chips as the low voltage means they won’t conduct any significant currents. It works best with shorts that measure a few ohms. Small enough to pass lots of current so they get hot, but larger than the parts of the track you want to keep. Too small and you risk burning out the wanted part of the track. To large and it won’t get hot enough.
Anyway, it didn’t work this time, the short passed 3A happily and I wasn’t going to risk going higher in case I vaporised the whole D0 net! So I cut out the offending parts of the D0 net and replaced them with some fine wire. Ugly, but when I fired up jtag again and there was my flash chip! Whew. Problem fixed after a day bug hunting. On to the next one.
Bug Hunts and the Geeky Mindset
If all the above sounds very cool and logical rest assured it wasn’t.
When you are in the middle of a bug hunt two things happen.
The first: the logic centres of your brain shut down. You don’t think straight. You bounce from one theory to another.
The second: you don’t want to work on anything else. You get obsessed and can’t leave it alone. Worse, doing anything else is impossible. Intense bug hunts lead to late nights, neglect of significant others, no exercise, fatigue and a further dip in intellect.
Despite all that, I really enjoy a good bug hunt. When you come through at the end, the satisfaction is supreme. And you always end up learning lots along the way that comes in useful later. Call me a geek, call me a nerd, or call my wife and commiserate with her.
This was an strange one. Now that jtag could actually see my flash chip I tried to flash the chip:
jtag> cable parallel 0x378 IGLOO
Jtag> initbus bf532_bf1
jtag> detectflash 0x20000000
jtag> flashmem 0x20000000 /path/to/u-boot.bin
However I kept getting verify errors. Whole sectors of the flash (each sector is 64k long) were not being written. I tried a few things like checking the JTAG signals for ringing (they were OK) and shortening the JTAG cables just in case I has signal integrity problems like ringing on the JTAG clock signal. Still verify errors.
Each attempt was very time consuming as flashing via the JTAG is slow, it takes about 20 minutes to flash the 128k u-boot program.
I was in chat contact with Wojtek at this time and he suggested I try the Igloo JTAG cable on a STAMP board. So I tried flashing u-boot onto a BF533 STAMP and it worked fine. U-boot even ran OK on the STAMP. This was a good test as it confirmed my Igloo was OK. One variable removed from the bug hunt.
So then back to the bf1 board. Suddenly it is now flashing OK! Despite nothing changing. So I tried it three more times and each time it flashed and verified OK. So I don’t know. It doesn’t always boil down to logical cause and effect. Sometimes stuff just happens. So I “declared victory” and moved on.
Now I had u-boot starting up OK when I powered up – I could interact via the serial port. This was pretty exciting, as it proved that all the major components – Blackfin, flash, SDRAM, and CPLD were working fine. The bf1 was talking to me. So I powered down and soldered on all the parts for one Ethernet port. The idea was to use Ethernet and tftp to download a uClinux image.
Torpedoed by U-Boot and Ethernet
After loading the Ethernet parts I booted into u-boot and tried to tftp. Something was happening, but there were lots of tftp time out “T” symbols and only the occasional OK packet. Ping worked OK from u-boot, but not tftp or dhcp.
I did some packet sniffing on my host and found out that packets were being received OK from the bf1, but it looked like anything the host sent back was being ignored by the bf1. So I checked the rx side wiring of the Ethernet port but couldn’t find any problems. During tftp attempts I could also see what looked like valid Ethernet signals (1Vpp nice looking data signals) on the tx and rx nets at the RJ45 connector.
I was using a binary u-boot image that I found on the bf1 site. I wanted a way to inspect packets being received and that meant I needed to compile u-boot from source. So I checked out the CVS version of u-boot for the BlackfinOne and tried to compile. However I kept getting linker errors. This is very frustrating, you are trying to fix one bug and keep getting caught by others! Such is life.
Eventually a post to the bf1 forum solved the linker problem – there was a recent bf1 u-boot fix just checked into CVS. Now I could compile u-boot and start inserting some test code. So I flashed the CVS version of u-boot and hit another problem – now Ethernet wouldn’t work at all (not even ping) and the MAC address was always a string of 0x00s. This was a step backwards from the previous binary where the MAC was set up OK and at least ping worked!
One other problem was slowing me down – that 20 minute time it took to flash u-boot every time I wanted to add a printf to test something. This made debugging very tedious and frustrating. It felt like I was getting no where fast.
One thing I did discover is that it is a good idea to erase the flash sectors before flashing using jtag, just to make sure:
jtag> eraseflash 0x20000000 2
When I didn’t perform this step I sometimes found I was booting a previous version of u-boot, not the one I had just flashed!
After a break I thought of another idea to help speed up debugging. I used a trick I read about in the Blackfin u-boot documentation – you can download a new u-boot image into SDRAM using u-boot, then execute the new u-boot image using the “go” command. As I didn’t have Ethernet working I used ymodem to download u-boot images to SDRAM. This worked really well and enabled me to test and debug in 2 minute cycles – much faster.
Note that this technique doesn’t work for all u-boot images, for example when I tried to run a u-boot image configured for 64M RAM with a 32M u-boot it failed and I needed to reflash instead. Some people have reported that ymodem doesn’t work for them, they use the kermit protocol instead.
Two iterations later I had the bug nailed. The problem was that u-boot was trying to read the MAC from the little serial EEPROM used by the Ethernet chip. However I hadn’t loaded this chip, and even if I had it would be un-programmed and the MAC would be all 0xff’s. So after some reading and grepping of the u-boot source I changed this line in include/configs/bf1.h from:
#define CONFIG_DM9000_HAVE_EEPROM 1
This allows the MAC to be set by the values compiled into bf1.h, or the environment variables used by u-boot. If this option is defined then the eeprom contents always overrides the MAC from the environment.
Suddenly my u-boot Ethernet was up and downloading images via tftp just fine. Cool. After 1.5 days of chasing this bug it never looked so good to see those tftp “*” symbols flashing across the screen as an image was downloaded. I also learnt a lot about u-boot and how Ethernet drivers work which was interesting.
So now we are getting real close – time to build a uClinux image!
Building uClinux for the BlackfinOne
To create a uClinux-dist for the BlackfinOne I copied all of the files in
uClinux-dist-R06R2-RC2-bf1-diff-files.061219.tar.bz2 on top of a fresh 2006R2 Rc5 uClinux-dist using “cp -R”.
This was set up for the jffs2 file system, however I wanted to try a ramfs config as I was more familiar with that from my STAMP work. This was probably a mistake, as I spent several hours working out how to get this working. uClinux was starting OK but kernel panicking when it couldn’t mount root. I was using the u-boot command:
bf1> tftp 0x1000000 uImage
bf1> bootm 0x1000000
To download and boot the uImage. The uImage file combines the kernel and the root files system in one compressed image.
At Wojtek’s suggestion (again via chat) I compared the boot sequence of the bf1 to my STAMPs. I noticed something odd about the Memory Map section of the boot sequence. When I booted the STAMP I was getting something like this.
text = 0x00001000-0x00107f94
init = 0x00108000-0x00115508
data = 0x00115b4c-0x00145e60
stack = 0x00116000-0x00118000
bss = 0x00145e60-0x00153394
available = 0x00153394-0x03800000
rootfs = 0x07700000-0x07f00000
DMA Zone = 0x07f00000-0x08000000
Note the “rootfs” line. This says that 8M is allocated to the rootfs. This was the BF533 STAMP, which has 128M available, which means the top address is 0x8000 0000. Note that the “available” memory is about 57M. Now when I compared the rootfs line for the bf1, I saw:
rootfs = 0x00600000-0x001f00000
Which is 25M allocated for rootfs. Now at this stage I only had my bf1 configured for 32M, so that rootfs was consuming most of the available memory. In fact when I checked the available line, only 416k was free for the entire operating system! I traced the problem to the vendors/IvanDanov/BLACKFINONEV2/Makefile. The BLOCKS line that controls the size of the rootfs was set to 25600 rather than 8000. As I mentioned earlier, this is only really a problem if you are using ramfs.
I changed the line back to BLOCKS=8000, rebuilt the uImage and managed to (finally) boot uClinux. So I ran around the house telling everyone but nobody cared much (except my baby son, as discussed below). Oh well, it still made my day!
A little more work to u-boot and kernel configuration and I had a 64M BlackfinOne booting OK. Wojtek showed me how to configure u-boot for 64M:
- In board/bf1/config.mk make TEXT_BASE = 0x03fe0000.
- In include/configs/bf1.h set the #define CONFIG_MEM_SIZE to 64.
- make mclean;make mrproper; make bf1_config;make.
To configure uClinux for 64M “make linux_menuconfig and set “Blackfin Processor Options – Use SDRAM chip” to your memory chip, then rebuild uClinux-dist with the usual “make”.
If you have built a 64M BlackfinOne you are welcome to try my u-boot and uClinux uImages. They might be useful for initial testing and save you the steps of compiling your own. Once you have flashed u-boot-bf1v2-64M.bin, place uImagebf1v2-64M on your tftp server and on the bf1 console:
bf1> tftp 0x1000000 uImagebf1v2-64M
bf1> tftp 0x1000000
Note that this uImage uses ramfs not jffs2, and I used the -75 memory chip. This should work OK with the slightly faster -7E chips.
While experimenting I noticed that you need a u-boot configured for 64M to boot a 64M uClinux, otherwise uClinux bombed for me shortly after the uImage was uncompressed. However you can boot a 32M linux using a 64M u-boot.
I set the Ethernet MAC and IP for uClinux using environment variables in u-boot that are passed to uClinux, using the addnet script:
addnet=setenv bootargs $(bootargs) ip=$(ipaddr):$(serverip):$(gatewayip):$(netmask):$(hostname):eth0:off
flashboot=run addnet;bootm 0x20040000
Flashboot is called when the bf1 starts, which calls the addnet script, and boots uClinux. I would rather have u-boot set just the MAC and let uClinux set the IP through /etc/rc, however I haven’t worked out how to do this just yet.
The BlackfinOne project is a great example of open hardware development. There is a growing community of developers that have contributed to hardware, software, and even tracking down problem parts for each other. There are parts being air-mailed all over the world as one person helps another get parts that are tough to find in their country. An interesting example of open/community development being applied to hardware – obtaining parts is not a problem for open software development!
Web-based parts supply companies like Digikey have helped make this sort of project possible. You can now easily buy many obscure parts and get them delivered anywhere in the world in a week or so. The web has helped hardware hacking get easier, and made it possible to build complex, sophisticated designs like the bf1.
Chat and mailing lists proved invaluable. Wojtek and I were in constant chat contact, and we used the bf1 forums to help us when we got stuck. Even though at this time of year we are about 12 hours apart in time (and about 40 degrees C in temperature). Just being able to explain each our problems was very useful – it made the problem clear in your own head when you have to express it to another person. I have been developing hardware for about 20 years and this as truly a new way of working for me. Nothing new for software development I guess, but very different for hardware (at least for me). In my experience (mainly in smaller companies), the hardware guy has often worked more or less alone.
This has been a really great experience. I assembled my very own uClinux hardware, then brought it up through various stages until it booted. I still can’t believe that I actually soldered together my very own Linux machine! If you are interested in trying out a hardware project, I thoroughly recommend this project. Great design, great community. Open hardware in action.
When I told my baby son about my working BlackfinOne, this was his reaction:
I know exactly how he feels!
Wojtek for your review of this post and photos.