Downtown Doug Brown

This weekend, I finally decided to try to set up my MacBook Pro to boot into OS X, Windows XP, and Ubuntu. I ran into a few problems so I’d like to share what I did to make it all work. A lot of what I did was driven by other people’s wiki articles and blog posts, particularly this excellent blog about manually editing partitions. Without further adieu, I will describe my setup from start to finish.

After playing around with a ton of parallel port cards in an attempt to figure out Willem programmer compatibility, I decided it would be useful to write a parallel port tester program. Just a simple utility where you can set the output value of each output pin and read the value of each input pin. Several other test programs exist, but I wasn’t happy with their interfaces, and a lot of them weren’t prepared to easily handle PCI/PCI Express/ExpressCard parallel ports. At worst, they only supported the standard parallel port I/O addresses that new motherboards don’t have anymore. At best, they supported custom parallel ports but required you to manually look up the I/O address range of the card.

Oh, and a lot of the existing tools don’t work with newer versions of Windows. I wanted a tool compatible with newer versions of Windows, both 32-bit and 64-bit.

Introducing Parallel Port Tester:

Simply locate your parallel port in the list that appears in the bottom right. I use several methods to discover parallel ports and their I/O ranges, so you shouldn’t need to enter them manually. If for some reason your port doesn’t show up in the list, you can manually enter the base address of your parallel port (e.g. 0x3000) and hit Enter.

You can toggle parallel port outputs on or off by clicking on the circle representing the pin you want to change. Green represents on (high), black represents off (low). The circles representing input pins are automatically updated while you have the port selected, so you can easily test your inputs. You can also choose whether to use the four control pins as inputs or outputs.

For people who are really interested, I also display the raw register values of the three standard parallel port registers. I show separate values for the control register because what you write to the control register is not guaranteed to be what you read back.

Requirements

Let’s get the requirements out of the way. You will need:

• Windows 2000 or newer. Should work with 2000, XP, Vista, 7, 8, 10, 11, and the various newer server versions. 32- or 64-bit Windows are both supported.
• Microsoft .NET Framework 2.0 or newer

Download

OK, so you’re ready to download it? Here you go.

Download Parallel Port Tester version 1.0.0.2 installer

Special Thanks

Parallel Port Tester includes InpOut32 1.5.0.1 by Phil Gibbons in order to access the parallel ports directly.

Version History

• 1.0: Initial release
• 1.0.0.1: Improves parallel port detection algorithm; previous algorithm was incorrect in certain cases.
• 1.0.0.2: Fixes a bug that was causing a crash for some users.
  • Note: On 1/17/2022, I added an installer for version 1.0.0.2. Prior to this, you needed to manually download and install InpOut32.

Any problems/questions/concerns? Please let me know in the comments below and I’ll try my best to help get you going. Be careful; it might be possible to fry your parallel port if you hook the outputs up wrong. I’m not responsible for any damage done to your computer by using this software.

For sample code that describes how I detect the computer’s parallel ports in Parallel Port Tester, see my blog post: Detecting parallel ports and their I/O addresses in Windows.

Old Installation Instructions

Prior to 1/17/2022, there was no installer, so you had to install the software manually by downloading and installing InpOut32. Nowadays, you can just download the Parallel Port Tester installer and this will all be handled for you automatically. The old installation instructions are preserved below for historical reference:

• Download the old ZIP file of Parallel Port Tester 1.0.0.2 that didn’t use an installer.
• Download InpOut32.dll.
  • Note that I am linking to the best version of InpOut32.dll. It should be compatible with all versions of Windows supported by my program.
• If you are using Windows Vista or newer, you will need to run the InstallDriver.exe program inside the Win32 directory of InpOut32’s distribution in order to install the driver. You only have to do this once, and it’s just so you can get administrator privileges in order to install the driver. Even if you are on a 64-bit operating system, that’s OK. Still run InstallDriver.exe from the Win32 directory.
• Put InpOut32.dll from the Win32 directory into the same directory as ParallelPortTester.exe. Don’t use the x64 directory; that DLL would only be for a 64-bit program, but Parallel Port Tester is not a 64-bit program.

My previous blog posting on this subject from a few years ago sparked quite a bit of interest, so I’d like to follow it up with the latest compatibility information I have. First a quick summary:

Traditional Willem EPROM programmers require your computer to have a parallel port, and almost no computers today have them. You can find add-on parallel port cards, but a good chunk of today’s software is written to work directly with the parallel port addresses that were found on motherboards of older computers (0x378, 0x278, and 0x3BC). Add-on PCI/PCI Express/ExpressCard parallel ports don’t use those addresses.

Unfortunately, the Willem software only lets you pick from a hardcoded list of addresses to work with. The good news, however, is that it does its port access through a DLL called io.dll. There are replacement versions of io.dll that trick the Willem software into talking to a different parallel port address:

• My remapped io.dll (uses TVicPort for port access)
• Ben Ryves’ Remapped io.dll (uses Inpout32 for port access)

It’s actually a good thing that multiple options exist, because sometimes one option works for someone while the other option doesn’t, and vice versa. I would like to list what I have discovered about the various options available in terms of both hardware and software.

Parallel port cards

I have tried a total of four different parallel port cards cards, some of which are hard to find at this time:

• Syba PCI Express 4x serial, 1x parallel card
• Syba ExpressCard 2x serial, 1x parallel card
• Shentek ExpressCard parallel port (hopefully this eBay link sticks around — it’s a Shentek part number 33006 if the link breaks)
• StarTech ExpressCard parallel port

My experience with these is the following. The Syba cards both use Moschip (now ASIX) chipsets, while the Shentek and StarTech cards use Oxford (now PLX Technologies) chipsets.

Both of the ASIX-based cards seem to work fine with no messing around needed. I’ve tested them on my desktop computer with Windows 7 64-bit (my DLL) and my laptop with Windows XP (Ben’s DLL).

The PLX-based cards throw a couple of curveballs into the picture, though. First of all, their bidirectional control pins (strobe, auto/linefeed, initialize, and select printer) do not have pull-up resistors. This causes a problem because they are open-drain/open-collector outputs, so something needs to pull them up when a high value is needed. Otherwise it’s impossible to use them as outputs — and the Willem programmer uses them as outputs. The Willem programmer (mine, at least) doesn’t supply its own pull-ups for those pins either. So in order to gain compatibility with those two cards, you will need to add pull-ups somewhere. To get these cards to work, I manually soldered some 10Kohm pull-up resistors for those lines onto my Willem board. It was a pretty ugly hack, though, so I removed the pull-ups after successfully testing it. Maybe someone can find a cleaner way to do it.

I have no idea about full-size PCI Express or PCI cards that use the PLX chipset. Perhaps they would already have the pull-up resistors in place. I’m just not sure.

The second issue with the PLX cards is described in the “DLLs” section below.

DLLs

The ASIX cards seem to work great with both my DLL and Ben’s DLL, so no further comments are needed here about them.

The Oxford cards don’t behave quite so nicely with my DLL. It seems that TVicPort has trouble reading bytes from odd addresses with the PLX cards, even though it has no such trouble with the ASIX cards. Inpout32 does not have this same issue. I haven’t narrowed down the root problem, but I don’t really care at this point anyway because there’s a fix: if you’re using a PLX chipset, you should use Ben’s DLL instead of mine.

The original reason I made my DLL was because I couldn’t get Ben’s DLL to work correctly with 64-bit Windows 7. It seems that Inpout32 is now 64-bit compatible (and signed), so it may be possible to simply stick with Ben’s DLL. If you plan on going that route, I would recommend downloading the latest version of Inpout32 and grabbing the Inpout32.dll file included with that to go along with Ben’s io.dll. You’ll want to use the 32-bit version even if you’re on a 64-bit operating system. The reason for that is because the Willem software itself is 32-bit. You may need to run the InstallDriver.exe program included with it to get everything to install correctly, but I’m not an expert at Inpout32.

Testing compatibility

If you’d like, you can test your parallel port card with my Parallel Port Tester utility. Make sure all of the outputs and bidirectional pins can correctly output both high and low values. Physically check each pin with a voltmeter while you test. Every card I’ve seen so far outputs 3.3V as a high value. If your bidirectional pins don’t appear to output a high value correctly (or you get weird readbacks in the tester utility), the high value may be floating, thus indicating you need pull-up resistors on those pins. Also make sure the input pins read a high value with nothing attached and a low value when you ground them. It’s OK if the control pins don’t work correctly as inputs; the Willem programmer uses them all as outputs.

Conclusion

The information available in my original post is still very useful and should help walk you through setting up a Willem programmer with various types of cards. I just wanted to share all of my latest compatibility knowledge in a new post because it was all buried in the comments of my original post.

In the early- to mid-1990s, Apple produced their Macintosh Performa line of computers. These computers were meant for home users and typically came bundled with software such as ClarisWorks and Mario Teaches Typing, along with interactive tutorials teaching the basics of how to use the Mac OS. They were sold in places such as Wal-Mart and Sears.

One interesting thing about these computers is that at least initially, they did not come with any software restore disks. If something bad happened and you wiped out your system software (which was easy enough to do — somehow I did it as a kid), you didn’t have a supplied set of disks (or a CD) for restoring the software. Instead, these computers came with software called Apple Backup, and you were supposed to back up your system onto 1.44 MB floppies when you first got it. When you ran Apple Backup, it would let you choose to back up either the full hard drive or just your system folder:

The number of disks needed here is small because I ran it on a simple barebones system to make this screenshot. With all of the bundled software on a stock Performa machine, it would have taken somewhere in the ballpark of 50 floppy disks to complete the backup of the full hard drive. Seriously, how many people would have bothered to buy all of the floppies that would have been necessary, and then actually taken the time to do it? Some people definitely did, and I’m impressed by the dedication. I know my family didn’t bother when I was growing up. It would have been more realistic to only back up user-created files, and then provide a system restore disk (or set of disks) to restore any original software, but the backup software didn’t work that way. Apple obviously learned from their mistake because they began bundling Performa computers with restore CDs at some point later on. (Note: To be completely fair to Apple, it was possible to obtain restore disks from them if you needed to restore your system and your Performa didn’t come with any.)

I ran the backup process on an old Mac for fun, and it guided me through the process of backing up the system. Click the thumbnails to see the full size, if you care:

The resulting disks were normal Mac floppy disks, named “Backup Disk 1”, “Backup Disk 2”, and so on, each containing a single 1,414 KB file named Apple Backup Data.

To restore your data after a failure, you would boot up your Performa using the Utilities disk that came with the computer. The Utilities disk contained a barebones system folder along with disk formatting/repair utilities and a program called Apple Restore. You guessed it: Apple Restore was used to restore the system. You would run it and then insert your backup floppies one at a time to restore everything you backed up.

I looked at some of Apple’s later Performa restore CDs, and interestingly enough, they came with programs called “Restore All Software” and “Restore System Software”, each with a folder full of 1,414 KB files named “Data File 1”, “Data File 2”, and so on. So presumably Apple simply used Apple Backup to back up a stock system and stuck the resulting data files onto a CD to create the restore CD.

I decided that it might be useful to have Apple Backup’s file format documented in case someone out there ends up needing to restore files from their old backups (or wants to extract files from a factory restore CD). Although backup floppies from the 90s are probably going bad by now, I think it’s still cool to have the information out there. Anyway, I decided to reverse engineer Apple Backup and one of the factory CD restore programs (which does essentially the same thing as Apple Restore). I believe I have successfully reverse engineered the Apple Backup file format.

The rest of this blog posting contains the technical details about the format of the files that Apple Backup creates.

Overall layout of an Apple Backup data file

• Type code: ‘OBDa’ (CD restore files have type ‘OBDc’ instead, but are otherwise identical in format)
• Creator code: ‘OBBa’
• There is no resource fork, and the data fork content is summarized in the table below:

Next, allow me to describe the content of the data fork in more detail:

Backup disk header format

All offsets and lengths are in bytes. All multi-byte quantities are in big-endian format as would be expected from a Mac file format.

Total length: 0x200 bytes

Boot blocks

These appear to be standard Mac OS boot blocks, 0x400 bytes in size. Easily seen in a hex editor because it begins with LK, and soon thereafter has names: System, Finder, MacsBug, Disassembler, StartUpScreen, Finder, Clipboard. They are written to the hard drive by the restore program when the System Folder is blessed as it’s restored. For just extracting files, they are not really relevant. They begin at an offset of 0x200 from the start of the backup file, and end at an offset of 0x600 (where the first file header begins)

File/folder header format

Each file or folder starts out with a header. Again, all offsets and lengths are in bytes, and everything is big-endian. This header will always begin on a 0x200-byte boundary; padding bytes of zero are added to the end of the previous file’s data fork/resource fork data if needed. The first file header in a backup data file is always at 0x600, immediately after the boot blocks.

Total length: 0x70 bytes. See Inside Macintosh: Files and the Mac Toolbox C headers named “Files.h” and “Finder.h” for more info on FInfo, DInfo, FXInfo, DXInfo, and ioFlAttrib. These items are where the type and creator code, invisible flag, and icon position in folder are stored, for example.

Full path format

The full path is just that: the full path to the folder or file, with the hard drive already being assumed. The components of the path are colon-delimited. The file being restored is the last component of the path. Example:

System Folder:Control Panels:Memory

(if the file being restored is the Memory control panel). It’s just printed as raw bytes, no null terminator or anything — it’s basically a Pascal string with a two-byte length, and the length is at the end of the file/folder header.

The actual file data

Immediately after the full path, the data fork bytes begin (number of bytes = “Length of data fork provided by this disk”), followed by resource fork bytes (number of bytes = “Length of resource fork provided by this disk”). It’s perfectly OK for the length of either (or both) of these to be zero. After that, there is padding (filled with “0” bytes) to the nearest 0x200 byte boundary, and then the next file/folder header begins.

When a file overflows the disk

If there is not enough space remaining on a disk for a complete file (this almost always happens at the end of each data file), the amount of file data that will fit on the disk is stored so that the full disk file size matches the size given in the disk header. Then the first file header on the next disk will be the same file’s header with the exception of the “Which file part this is” field, which will be incremented by one. It is possible for a file’s data to span several disks in this manner; the intermediate disks will only have one file header, followed by a repeat of the file path and data using up all available space on the disk.

For example: Let’s pretend you have 0x1000 bytes remaining on the current data file before its size reaches 1,414 KB. You’re ready to back up a file “Applications:TestApp” that that has a 512 KB data fork and a 128 KB resource fork. The file header will take up 0x70 of those bytes and the full path will take up 20 (0x14) of those bytes, for a total of 0x84 bytes — so there are 0xF7C (3,964) bytes left. So the file header is going to specify a data fork length in this disk of 3,964 bytes, and a resource fork length of 0 on this disk (the total lengths will be filled in as 512 KB and 128 KB though). Then the first 3,964 bytes of the data fork will be written to the file, giving it a total length of 1,414 KB. Then this disk will end. The first file header on the next disk will finish the remaining 520,324 data fork bytes and all of the 131,072 resource fork bytes, and then the next file’s header will begin on the nearest 0x200 byte boundary after that.

Conclusion

As you can see, the format is pretty simple. It’s just basically a disk header followed by a flat list of files until the end of the disk is reached. It should be possible to extract full and partial files from these backup archives even if the data files from some disks are missing. The partial files probably wouldn’t be very useful though.

The original Apple Restore utility didn’t let you pick and choose which files you wanted to restore — it just tried to restore them all, and it would ask you what to do if the file already existed and was newer than the backed up version. I see no reason why a utility couldn’t go through all of these files, give you a list of everything available, and let you selectively extract the files you want. It’s easy to detect what disk a particular backup data file came from because the disk header contains a field for what disk it belongs to.

If anyone’s interested in adding the ability to decode this format in an archive expander program, I would be happy to provide some sample data files. I may or may not decide to write a program to extract files from these backups, depending on how bored I get 🙂

Update 8/23/2022: GitHub user siddhartha77 created a new classic Mac utility called Apple Backup Extractor that can decode this file format. It can even decode what it can from an incomplete set of backup disks. If you’re looking for a way to recover data from these files, this is certainly the way to go!

I can’t remember exactly why, but I recently decided I wanted to play around with Super Nintendo games that support more than two players. I have a Super Nintendo console along with several such games — Super Bomberman, NBA Jam, and Madden 97 come to mind. The most popular adapter designed for this purpose is called the Super Multitap. It plugs into the second controller port on the SNES and provides four controller ports, so you get a total of five ports available for use.

I looked on eBay for Super Multitaps and found plenty for sale, but I decided I wanted to make things more complicated. I stumbled upon a schematic for it, found in the official Super Nintendo developer manual. Google for it if you’re interested — I don’t want to link to it just in case lawyers want to ruin all of the fun. I was intrigued by the simplicity of the circuit. It’s basically 3 chips (a mux and a bunch of tri-state buffers), eight resistors, and the controller connectors. Why buy the ready-made adapter if I could just make my own? In theory, buying the parts would be cheaper than buying a used Multitap on eBay, but in practice, the controller sockets and plugs are hard to find so it’s probably not worth it from a monetary standpoint.

But still, I think it’s worth it for the geek cred! Right?

Well, I decided to do it.

For a few years now, I’ve been fighting a weird problem: X-CTU (which is a software utility provided by Digi for programming XBee modules) is only available for Windows. I do most of my development in Linux so X-CTU is always a pain to work with. It does run pretty well under Wine if you need to use it with Linux or Mac OS X, but when you run it under Wine, it doesn’t detect any serial ports:

Now of course, we all know that you have to add a symlink in your ~/.wine/dosdevices directory to link Wine to your computer’s serial ports:

# ls -l ~/.wine/dosdevices/
total 0
lrwxrwxrwx 1 doug doug  8 Oct 30 21:30 a:: -> /dev/fd0
lrwxrwxrwx 1 doug doug 10 Oct 30 21:30 c: -> ../drive_c
lrwxrwxrwx 1 doug doug 10 Mar 29 18:08 com1 -> /dev/ttyS0
lrwxrwxrwx 1 doug doug 10 Mar 29 18:08 com2 -> /dev/ttyS1
lrwxrwxrwx 1 doug doug  8 Oct 30 21:30 d:: -> /dev/sr0
lrwxrwxrwx 1 doug doug  1 Oct 30 21:30 z: -> /

But even after doing that, X-CTU still doesn’t detect anything. All of the workarounds that I have found require you to define a user COM port in X-CTU:

After going through that process, you can pick the user COM port and X-CTU works perfectly fine (aside from not being able to download newer firmware versions from Digi’s site). As soon as you quit X-CTU, though, the user COM ports you have defined are gone. So whenever you re-open X-CTU, you have to redefine your user COM port. It gets old. So that’s the problem I’ve been fighting: having to manually add the user COM port every time I open X-CTU.

Today I got fed up and ran X-CTU with all of Wine’s debugging information enabled so I could get a clear idea of what X-CTU does when it first loads, in an attempt to figure out how to get the serial ports to show up. Good news: I got it working and now my serial ports show up automatically when I open X-CTU!

I’d like to explain how X-CTU detects attached serial ports, what Wine does in response, and finally, how you can get it working for yourself. Let’s dive in!

How X-CTU detects attached serial ports

X-CTU uses Windows’ Setup API to get a list of attached serial ports. I ran it with Wine set for full debugging and traced out the calls to Setup API functions to figure out exactly what it does. It starts out with a call to SetupDiClassGuidsFromName which, given the name of a device class (“Ports” in this case), returns a list of GUIDs that go with that class. Next, it calls SetupDiGetClassDevs with the list of GUIDs to get a list of devices that belong to the Ports class. It goes through the list of devices and requests the “friendly name” of each port by calling SetupDiGetDeviceRegistryProperty. The “friendly name” will look like one of these examples:

• USB Serial Port (COM5)
• Communications Port (COM1)
• Printer Port (LPT1)
• Blah blah blah port (COM7)

Notice how the friendly name always seems to end with (COM#) and it also includes other ports like printer ports. Well, X-CTU uses this info to detect the port — if the name of the port contains the string “(COM”, then it grabs the number directly after that string and uses it as the COM port number. It also ignores parallel ports.

So to get Wine to correctly populate the list, we need to figure out what Wine is doing in response to the three Windows functions I listed above. This information was readily available by both checking out the debug trace from earlier and also reading the Wine source code. Let’s go there now…

What Wine does when the setup API functions are called

SetupDiClassGuidsFromName searches in the registry for classes named “Port”. I don’t think it behaves exactly like an actual Windows machine, but here’s what it does on Wine. It searches for subkeys of:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Class

and looks for subkeys that have a class matching the name provided to it. In Wine, it finds the class with GUID {4d36e978-e325-11ce-bfc1-08002be10318}, which according to Microsoft is for COM and LPT ports. Anyway, that’s the single GUID it finds in Wine.

SetupDiGetClassDevs searches in:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Enum

for items that match the GUID we found earlier. The above key contains various keys that represent different categories. Then the categories contain keys that represent devices, and the devices contain keys that represent instances of the devices (I believe). The gist of it is that it goes three levels deep through the Enum directory to try to find anything that has a string value “ClassGUID”. If the GUID matches the GUID we found earlier, Wine decides it’s a serial port and returns it in the list of discovered devices. This is the root cause of the whole problem — nothing is put into the registry automatically by Wine for these serial ports. So we’ll definitely need to add this manually, as we’ll see later.

SetupDiGetDeviceRegistryProperty is finally used to get the friendly name for the port. It looks in the same location it looked for the ClassGUID value, but this time it looks for a string called FriendlyName — which, as you guessed it, contains a string in the format of my examples above.

Once I figured this out, I was pretty much home free. So without further adieu, here are the instructions for getting it working.

What to add to the registry

The key (no pun intended) is to add your serial ports as subkeys of:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Enum

to satisfy the functions I described in the section above:

• Create a subkey of Enum and call it SERIAL (although the name you use doesn’t really matter–I believe it searches everything, not just the SERIAL subkey).
• Create a subkey of SERIAL and call it COM1 (if your port is COM1 — although this name doesn’t really matter)
• Create a subkey of the COM1 key you just created and call it COM1 also (this name doesn’t really matter either though)
• In the final COM1 subkey you created, add two string values:
  • ClassGUID — containing the value {4D36E978-E325-11CE-BFC1-08002BE10318} (the GUID for the Ports class)
  • FriendlyName — containing a name in the format “Serial Port (COM1)” without the quotes of course.
    • Make sure the name ends with the COM port name in parentheses as in this example — (COM1). It has to be that way or it won’t work–it might appear in the list, but it will fail to open unless you do it exactly in that format.
    • This is what X-CTU actually uses to decide which port to open. The other “COM1” subkeys we added in the earlier steps aren’t checked for anything — I just named them that way for clarity while you’re browsing the registry.

You can make these modifications using regedit in Wine (type “wine regedit” in a terminal window). Here’s an example screenshot to make it clear what you have to add:

That’s it! You’re done. The ports should now appear automatically in X-CTU. Assuming you have also created the symlinks in ~/.wine/dosdevices for the COM ports you added, they should also be operational.

Conclusion

This is tested in Ubuntu 12.10 with Wine 1.4.1. I would imagine if you can figure out where the dosdevices folder is to stick the symlinks, it will probably work in Mac OS X as well. Your mileage may vary. Good luck!

This strategy definitely works for X-CTU, but it’s not a generic strategy that will work for any Windows program under Wine. Different programs use different methods to get a list of serial ports. Some programs may check for a different key called PortName next to FriendlyName. X-CTU in particular only checks for FriendlyName. If you’re trying to get this to work with a different Windows program, play around. Check programming tutorials to see the various methods people use to enumerate COM ports on Windows. Figure out which method your particular program is using — disassemble it, check what functions it links against, run it in Wine with debugging enabled, etc. Once you’ve figured it out, use Wine’s debugging facilities (and the Wine source code) to see what Wine is doing in response to the various functions that are called. Chances are good that it is looking into the registry and you just need to tweak your registry to give the Windows functions the results they are expecting.

I hope this helps someone out there someday!

I’ve seen plenty of articles about this topic already, but I’d like to talk about it on my blog as well. It’s a pretty important topic in my opinion.

Let’s say you have a microcontroller peripheral that sends and/or receives a bunch of data. A good example of this type of microcontroller peripheral is a UART (universal asynchronous receiver/transmitter). That’s essentially a fancy name for a serial port–you know, the old 9-pin connectors you’d find on PCs. They were commonly used for connecting mice and modems back before PS/2, USB, and broadband took over. A UART basically does two things: receives data and transmits data. When you want to transmit data, you write a byte to a register in the UART peripheral. The UART converts the 8 bits into a serial data stream and spits it out over a single wire at the baud rate you have configured. Likewise, when it detects an incoming serial data stream on a wire, it converts the stream into a byte and gives you a chance to read it.

I’m actually not going to go into detail about UARTs yet because I feel it’s really important to understand a different concept that we will need to know when we learn about how to use UARTs. That concept is an interrupt-safe circular buffer. This lesson is dedicated to understanding this very important data structure. The next post I write will then go into more detail about UARTs, and that post will use a concept that you will learn about in this post. OK, let’s get started with circular buffers.

A common problem when you are receiving or transmitting a lot of data is that you need to buffer it up. For example, let’s say you want to transmit 50 bytes of data. You have to write the bytes one-by-one into the peripheral’s data register to send it out. Every time you write a byte, you have to wait until it has transmitted before writing the next byte. (Note: Some microcontroller peripherals have a hardware buffer so you can write up to 16 [for example] bytes before you have to wait, but that hardware buffer can still fill up, so this concept I’m describing still applies)

A typical busy loop sending the data would look like this:

char data[50]; // filled with data you're transmitting

for (int x = 0; x < 50; x++) {
    while (peripheral_is_busy());
    peripheral_send_byte(data[x]);
}

This loop is simple. For each of the 50 bytes, it first waits until the peripheral isn’t busy, then tells the peripheral to send it. You can imagine what implementations of the peripheral_is_busy() and peripheral_send_byte() functions might look like.

While you’re transmitting these 50 bytes, the rest of your program can’t run because you’re busy in this loop making sure all of the bytes are sent correctly. What a waste, especially if the data transmission rate is much slower than your microcontroller! (Typically, that will be the case.) There are so many more important tasks your microcontroller could be doing in the meantime than sitting in a loop waiting for a slow transmission to complete. The solution is to buffer the data and allow it to be sent in the background while the rest of your program does other things.

So how do you buffer the data? You create a buffer that will store data waiting to be transmitted. If the peripheral is busy, rather than waiting around for it to finish, you put your data into the buffer. When the peripheral finishes transmitting a byte, it fires an interrupt. Your interrupt handler takes the next byte from the buffer and sends it to the peripheral, then immediately returns back to your program. Your program can then continue to do other things while the peripheral is transmitting. You will periodically be interrupted to send another byte, but it will be a very short period of time — all the interrupt handler has to do is grab the next byte waiting to be transmitted and tell the peripheral to send it off. Then your program can get back to doing more important stuff. This is called interrupt-driven I/O, and it’s awesome. The original code I showed above is called polled I/O.

You can probably imagine how useful this idea is. It will make your program much more efficient. If you’re a computer science person, you’re probably thinking,  “Doug, the abstract data type you should use for your buffer is a queue!” You would be absolutely correct. A queue is a first-in, first-out (FIFO) data structure. Everything will come out of the queue in the same order it went in. There’s no cutting in line! (Well, unless you have a bug in your code, which I’m not afraid to admit has happened to me enough times that I finally bothered to write up this article so I’ll have a reference for myself.) Anyway, that’s exactly how we want it to be. If I queue up “BRITNEYSPEARS” I don’t want it to be transmitted as “PRESBYTERIANS”. (Yes, amazingly enough, that is an anagram.)

A really easy to way to implement a queue is by creating a ring buffer, also called a circular buffer or a circular queue. It’s a regular old array, but when you reach the end of the array, you wrap back around to the beginning. You keep two indexes: head and tail. The head is updated when an item is inserted into the queue, and it is the index of the next free location in the ring buffer. The tail is updated when an item is removed from the queue, and it is the index of the next item available for reading from the buffer. When the head and tail are the same, the buffer is empty. As you add things to the buffer, the head index increases. If the head wraps all the way back around to the point where it’s right behind the tail, the buffer is considered full and there is no room to add any more items until something is removed. As items are removed, the tail index increases until it reaches the head and it’s empty again. The head and tail endlessly follow this circular pattern–the tail is always trying to catch up with the head–and it will catch up, unless you’re constantly transmitting new data so quickly that the tail is always busy chasing the head.

Anyway, we’ve determined that you need three things:

1. An array
2. A head
3. A tail

These will all be accessed by both the main loop and the interrupt handler, so they should all be declared as volatile. Also, updates to the head and updates to the tail each need to be an atomic operation, so they should be the native size of your architecture. For example, if you’re on an 8-bit processor like an AVR, it should be a uint8_t (which also means the maximum possible size of the queue is 256 items). On a 16-bit processor it can be a uint16_t, and so on. Let’s assume we’re on an 8-bit processor here, so ring_pos_t in the code below is defined to be a uint8_t.

#define RING_SIZE   64
typedef uint8_t ring_pos_t;
volatile ring_pos_t ring_head;
volatile ring_pos_t ring_tail;
volatile char ring_data[RING_SIZE];

One final thing before I give you code: it’s a really good idea to use a power of two for your ring size (16, 32, 64, 128, etc.). The reason for this is because the wrapping operation (where index 63 wraps back around to index 0, for example) is much quicker if it’s a power of two. I’ll explain why. Normally a programmer would use the modulo (%) operator to do the wrapping. For example:

ring_tail = (ring_tail + 1) % 64;

If your tail began at 60 and you repeated this line above multiple times, the tail would do the following:

61 -> 62 -> 63 -> 0 -> 1 -> …

That works perfectly, but the problem with this approach is that modulo is pretty slow because it’s a divide operation. Division is a pretty slow operation on computers. It turns out when you have a power of two, you can do the equivalent of a modulo by doing a bitwise AND, which is a much quicker operation. It works because if you take a power of two and subtract one, you get a number which can be represented in binary as a string of all 1 bits. In the case of a queue of size 64, bitwise ANDing the head or tail with 63 will always keep the index between 0 and 63. So you can do the wrap-around like so:

ring_tail = (ring_tail + 1) & 63;

A good compiler will automatically convert the modulo to the faster AND operation if it’s a power of two, so you can just use my first example with the “% 64” since it makes the intent of the code clearer. I’ve been told I have “a lot of faith in compilers” but it’s true–if you compile with optimizations enabled, GCC will correctly optimize my first example into the assembly equivalent of my second example. Looking at the assembly code that your compiler generates is a very valuable tool for you to have available.

OK. Now that I’ve explained everything, here are my add() and remove() functions for the queue:

int add(char c) {
    ring_pos_t next_head = (ring_head + 1) % RING_SIZE;
    if (next_head != ring_tail) {
        /* there is room */
        ring_data[ring_head] = c;
        ring_head = next_head;
        return 0;
    } else {
        /* no room left in the buffer */
        return -1;
    }
}

int remove(void) {
    if (ring_head != ring_tail) {
        int c = ring_data[ring_tail];
        ring_tail = (ring_tail + 1) % RING_SIZE;
        return c;
    } else {
        return -1;
    }
}

The add function calculates the next position for the head, ensures that there’s room in the buffer, writes the character to the end of the queue, and finally updates the head. The remove function ensures the buffer isn’t empty, grabs the first character waiting to be read out of the queue, and finally updates the tail. The code is pretty straightforward, but there are a couple of details you should be aware of:

• I only modify the head index in the add function, and I only modify the tail index in the remove function.
• I only modify the index after reading/writing the data in the buffer.

By doing both of the above, I have successfully ensured that this is an interrupt-safe, lock-free ring buffer (if there is a single consumer and a single producer). What I mean is you can add to the queue from an interrupt handler and remove from the queue in your main loop (or vice-versa), and you don’t have to worry about interrupt safety in your main loop. This is a really cool thing! It means that you don’t have to temporarily disable interrupts in order to update the buffer from your main loop. It’s all because of the two bullets above. If the interrupt only modifies the head, and the main loop only modifies the tail, there’s no conflict between the two of them that would require disabling interrupts. As for my second bullet point, updating the head or tail index only after the read or write is only necessary in whatever side of the queue is working in the main loop — it doesn’t matter in the interrupt handler. But if I follow that convention in both the add() and remove() functions, they can be swapped around as needed — in one program add() could be used in an interrupt handler and remove() could be used in the main loop, but in a different program, remove() would be in the interrupt handler and add() would be in the main loop.

OK, so back to the original example of transmitting 50 bytes. In this case, the main loop will use the add() function. You will add 50 bytes to the queue by calling add() 50 times. In the meantime, the microcontroller peripheral will interrupt every time it successfully sends a byte. The interrupt handler will call remove(), tell the peripheral to transmit the character it just removed, and give control back to the main loop. Later on, the transmission of that character will complete, the interrupt will occur again, it will send the next character, and so on until all 50 bytes have been sent, at which point the queue will be empty. It works like a charm!

The only problem remaining is that the first time you need to send a character, the transmitter isn’t running yet. So you have to “prime the pump” by telling the transmitter to start. Sometimes a peripheral will have a way to induce the first interrupt to get the ball rolling. Otherwise, you may have to add some extra logic that forces the first transmission to occur from the main loop. Since this technically breaks the “single consumer and single producer” rule, this one exception to the rule may still require some careful interrupt safety. I’ll talk more about that in a later post. The main purpose of this post is just to get you familiar with ring buffers and their inherent interrupt safety (aside from that one “gotcha”). They are very, very useful and very, very common. I’ve used them in drivers for UARTs and CANbus, and as I understand it, they are very common in audio applications as well.

I hope this made some sense. If it didn’t, I hope my next posting about UARTs will help clear things up.

Note: The information I have given may not be completely correct if you’re dealing with a system that has more than one processor or core, or other weird stuff like that. In that case you may need something extra such as a memory barrier between writing the queue data and updating the index. That’s beyond the scope of this article, which is targeted toward microcontrollers. This technique should work fine on microcontrollers with a single processor core.

These SIMMs (and programmers) are available for purchase here.

As you may have seen from my previous posts, I have developed a programmable ROM SIMM for older Macintosh computers and a programmer to go with it. Everything from the hardware design to the firmware is completely open source. The combination is pretty nifty and enables people to do all kinds of crazy things that previously weren’t possible.

I always figured that the 2 MB size of the SIMM was plenty — I never really envisioned using the custom ROM SIMM for anything other than customizing the startup chime. However, bbraun came up with an even better idea — a ROM disk driver! He has patched the IIsi ROM (which should be compatible with the SE/30, IIx, IIcx, IIci, IIfx, and of course the IIsi itself) to enable booting from a ROM disk that uses up the remaining space on the SIMM and also has some other tweaks like disabling the slow RAM test that the previously-listed machines perform on every cold boot. It works perfectly and allows you to create a 1.5 MB bootable ROM disk to append to the 512 KB IIsi ROM. It boots up extremely quickly!

After he came up with that brilliant idea, I figured I should maximize the potential size of the disk image. The 64-pin ROM SIMM slot has 23 address pins labeled A0 through A22. However, A0 and A1 are unused due to the fact that the data bus is four bytes wide, so there are really only 21 usable address pins. That provides for a total of 2²¹ = 2,097,152 = 2*1024*1024 usable addresses. Each address points to 4 bytes of data, which provides for a total maximum ROM SIMM size of 8 MB. After taking into account the 512 KB ROM size, that would leave 7.5 MB of usable space for a ROM disk.

You can probably see where I’m going with this–I designed a bigger ROM SIMM!

First of all, I had to find NOR flash chips that ran on 5V and were big enough. It turns out that 5V chips are becoming harder and harder to find, especially in larger sizes. Everyone is moving toward 3.3V and lower. After a ton of searching, I settled on the Micron M29F160FB5AN6E2, a 5V 16-megabit chip that can be accessed in parallel using either a 16-bit data bus or an 8-bit data bus. It’s designed with automotive applications in mind, which probably explains why it still exists as a 5V chip. 16 megabits (Mb) is equal to 2 megabytes (MB). In order to reach the full 8 MB size, I need four of them per SIMM. I use them in 8-bit mode so each of the four chips provides one of the four bytes per address, just like how I designed the 2 MB version of the SIMM.

The chips are available in a fine-pitch TSOP package, which pretty much eliminates the possibility of socketing them. These chips really need to be soldered down, so this new SIMM requires my programmer board in order to put a disk image onto it. Here is the final product:

Quickly after I finished the new SIMM, bbraun updated his driver to work with a 7.5 MB ROM disk. So now it is possible to create a 7.5 MB ROM disk and boot from it. If you’re interested in buying one of the SIMMs and programmers (believe me, I have plenty to go around), you can purchase them here. As always, the 8 MB ROM SIMM is completely open source so you can make one yourself if you’d rather do it that way.

Some newer machines like Quadras have pads on their motherboards for these SIMMs–you just have to solder on your own socket. Unfortunately, it’s looking like they are hardwired to only address 1 MB of ROM space. Unless someone discovers a way to get around that, those machines will probably not be able to take advantage of the ROM disk. So for now, this ROM disk is restricted to the SE/30, IIx, IIcx, IIci, IIfx, and IIsi.

There are tons of instructions out there for setting up WordPress on a lighttpd server with WP Super Cache. A quick Google search will show you what I’m talking about. I’m not totally happy with them though — it usually boils down to enabling mod_magnet and using a lua script. It’s not too surprising that it requires a bit of trickery because the main instructions that come with WP Super Cache are designed with Apache in mind.

I’d like to share the setup I use on this blog, which takes more manual effort initially but no longer requires the use of a lua script.

I first obtained these instructions from a blog which no longer exists, but the original instructions are still available from the wonderful Internet Archive. Since the original page is no longer around in Google results, I feel like I need to keep the instructions alive. So…here they are, in my own words and modified for a newer version of lighttpd:

First of all, set up WP Super Cache for mod_rewrite mode. It’ll warn you that mod_rewrite isn’t installed and the rewrite rules must be updated — this is perfectly OK because we’re not using Apache.

Now, in the main WordPress directory, create symlinks to everything in the wp-content/cache/supercache/www.domain-name.com directory. Here’s an example symlink command for the “2010” directory:

ln -s wp-content/cache/supercache/www.domain-name.com/2010 2010

I had to create the following symlinks on my server:

2010, 2011, 2012, category, contact, index.html, and microcontroller-lessons

Every time you add a new page with a root directory URL or another year goes by, you will need to add an appropriate symlink or else the supercaching for that page won’t be active–it’ll be served through PHP instead, so the site won’t break if that happens — it’ll just be inefficient.

OK. Now that you have that done, disable directory listings, set the 404 error handler to /index.php, and ensure that index.html comes before index.php in the list of index file names. That last one about index.html being in front of index.php simply ensures your main blog page is served supercached correctly. Anyway, this can all be done in lighttpd.conf:

server.error-handler-404 = "/index.php"
dir-listing.activate = "disable"
index-file.names = ( "index.html", "index.php" )

That’s it! It’s about as simple as it can get.

Is it perfect? No, not at all. It doesn’t matter if you’re logged in or not — if a supercached version of a page exists and you go to its URL, you will get the supercached version rather than a newly-generated version of the page–meaning you don’t get the cool WordPress toolbar at the top. So it isn’t smart enough to look at your cookies or anything like that to only serve supercached files to non-logged-in users. In my case, I don’t really care — I can still access the WordPress Dashboard and that’s all that really matters to me.

It’s also kind of annoying that you have to create a new symlink for everything that is in the “wp-content/cache/supercache/www.domain.com” directory, but I can live with it.

It appears to work because if a supercached version of a page doesn’t exist already, a 404 error occurs, which passes the request onto /index.php, which figures out the original request.

Thanks again to the (now defunct) Zash Blag for the original instructions. There may be better solutions available than this — if there are, please leave a comment! I would definitely be interested in trying them out.

Edit: With the latest kernel updates, this hack should no longer be necessary in Ubuntu 12.04 or 12.10. I’m leaving this post in place for future reference if people run into the same issue with newer iPhone revisions as they come out.

Here’s the scenario: you’re using an Ubuntu distribution (perhaps 11.10 or 12.04 or 12.10, but I’ve only tested this on 12.10) and you’ve followed the various instructions around the internet for tethering your iPhone using a USB connection in Linux, but it just doesn’t work on your iPhone 5 no matter what you try, despite it working on an older iPhone model just fine.

What gives?

The problem is that iPhone 5 support was added to the ipheth driver in the Linux kernel in October, and that particular patch hasn’t been added to the kernel Ubuntu is using yet.

Look at the patch. All they did was add an extra item to the end of a static array. It looks exactly the same as all the rest of the entries, with the exception of a different USB device ID. If you don’t want to backport the patch and recompile the kernel module and all that, we could just modify the binary ipheth driver so one of the existing iPhone device ID entries matches the iPhone 5 instead of its original device ID. It’s a little more tricky to append to the array by just modifying the binary, so changing an existing entry in the array is an easier alternative. Let’s do it!

Make a copy of /lib/modules/<your kernel version>/kernel/drivers/net/usb/ipheth.ko and put it somewhere. We’ll mess with the copy, so leave your existing driver alone just to be safe.

Decide on an iPhone model you don’t need support for. In my case, I just picked the iPhone 3GS. In the code of the ipheth driver, find its corresponding USB device ID. For the iPhone 3GS, it’s 0x1294. Now, using your favorite hex editor (I personally like running HxD in Wine), search for that string, but reverse the bytes (unless you happen to be on a big-endian system like PowerPC). So in my case, I would search for the hex string 0x9412. Once you find it, confirm that it has 0xAC05 directly before it. This just confirms it’s part of the device ID table and not just a random chunk of code or data that happens to be the same. OK, so you’ve found it. Replace the old hex string (0x9412) with the iPhone 5’s USB device ID (it’s 0x12a8, so you will use 0xa812).

Save the file.  Manually unload the old driver if it was already loaded by typing:

sudo rmmod ipheth

Now, load the new one:

sudo insmod /path/to/new/ipheth.ko

Try plugging the iPhone 5 in now. It should be recognized as a network interface!

You’ll have to manually load this kernel module with the insmod command every time you reboot your system, and every time you install a kernel update, you should probably redo the patch on the latest version of the ipheth module, but it works! If you’re feeling extra clever, I think you could actually leave the modified version of the driver in place where the regular driver goes and add a line to modules.usbmap so it will automatically load the new ipheth module when the iPhone is plugged in, but I’m not going to get into that.

So there you have it: a quick and dirty way to make your iPhone 5 USB tethering work until Ubuntu either starts using a newer kernel or backports iPhone 5 support.