I work on embedded devices that have the capability of installing a firmware update by plugging in a USB flash drive containing an update file. These devices can also save reports onto an attached flash drive. Historically, these devices have worked with the various drives I’ve been able to test in house, but there have been occasional reports of incompatible drives in the field. I just used the sample code provided with the microcontroller manufacturer’s USB library, so I had no idea what I could do to improve compatibility.

Sometimes the problem is outside of my control. If the drive is larger than 32 GB, Windows 10 formats it as exFAT, and I don’t currently have exFAT support enabled (it’s too expensive to license from Microsoft). But most of the time, the problem isn’t the filesystem. The problem is that each drive behaves a little differently.

I decided to dedicate some time to improving USB drive compatibility in the embedded devices I work on. I researched USB mass storage devices, the USB specification, and the SCSI protocol that flash drives speak over USB. I bought an old Ellisys USB Tracker 110b, which is capable of recording raw USB 1.1 traffic (this is suitable for my needs, because the embedded devices I work on are only capable of full speed). Then, I bought a ton of used flash drives on eBay. My goal was to try as many drives as possible and discover various quirks. I also got a USB 1.1 hub in order to limit the drives to full speed on modern computers, and recorded what happened when I plugged the drives into Windows XP, Windows 10, Linux, Mac OS X, and Mac OS 9. 

I was successful. I found lots of little differences in how USB drives work. The purpose of this post is to share my findings to help others in the future who might have trouble with USB flash drive support in their embedded products.

Specifications

Here is a list of links to relevant specifications that will be useful as a reference:

Brief overview of how USB mass storage works

Before I get into specific details, I want to start with a quick overview of how all of the protocols explained in the specifications above combine together to enable computers to communicate with flash drives over USB.

When a flash drive is plugged in, the computer looks at its device, configuration, interface, and endpoint descriptors to determine what type of device it is. Flash drives use the mass storage class (0x08), SCSI transparent command set subclass (0x06), and the bulk-only transport protocol (0x50). The specification indicates that this should be specified in the interface descriptor, so the device descriptor should indicate the class is defined at the interface level.

What does this all mean? It just means that there will be two bulk endpoints: one for sending data from the host computer to the flash drive (OUT) and one for receiving data from the flash drive to the computer (IN). Data sent and received on these endpoints will adhere to the bulk-only transport protocol specification linked above. In addition, there are a few commands (read max LUN and bulk-only reset) that are sent over the control endpoint.

The host starts out by sending a 31-byte command block wrapper (CBW) to the drive, optionally sending or receiving data depending on what command it is, and then reading a 13-byte command status wrapper (CSW) containing the result of the command. The CBW and CSW are simply wrappers around Small Computer System Interface (SCSI) commands. Descriptions of the SCSI commands are available in the last two specifications I linked above.

That’s all there is to it…except I haven’t said anything about which SCSI commands you’re supposed to use, or when. SCSI is a huge standard. Reading the entire standard document would take a ridiculous amount of time, and it wouldn’t really help you much anyway. Unfortunately, the standards don’t provide a section entitled “recommended sequence of commands for talking to flash drives over USB”.

This is where I originally hit a roadblock when I was implementing USB support, and it’s also why I simply stuck with the sample source code provided with the USB library I used. Unfortunately the sample source code was not good enough. What it comes down to in practice is you should try to do something similar to what Windows does, because pretty much every flash drive is compatible with Windows.

Initialization sequence

The first important thing to do when a USB flash drive is detected is to figure out information about it. Is it actually a flash drive? How big is it? How many logical units (LUNs) does it have? I found that if I didn’t follow a sequence with some preliminary commands that operating systems do, a SanDisk drive with a bunch of files on it would crash when I first attempted to write to it. Interestingly, the same drive didn’t have the same problem when it was empty. You may be thinking I’m an idiot and a problem like this obviously has to be in the filesystem library and not the USB library, but I swear that the problem was the USB communication to the drive itself, because imaging the drive to my computer with “dd” and using the same filesystem library on the raw drive image worked fine. The Ellisys USB Tracker confirmed the drive responded with a stall condition after the write, and after clearing the stall, it was hung up, even after a mass storage bulk-only reset command, which is supposed to prepare the drive to receive a new command.

Based on what I observed Windows, Mac OS X, and Linux doing, I changed my initialization sequence, and that problem completely went away. The sequence I do now is not an exact clone of any other OS, and it’s probably doing extra overkill commands, but hey, it works:

  1. Request the maximum LUN. If this request stalls, assume the maximum LUN is 0. Start working with LUN 0 in either case.
  2. Keep trying the sequence of “TEST UNIT READY” followed by “INQUIRY” until they both return success back-to-back. At this point you can look at the returned inquiry data to get more information about the name of the drive, if you care. For the inquiry, request 36 bytes. That’s what pretty much every OS does, so it’s best not to deviate from that.
  3. If the “INQUIRY” response data indicates that the peripheral device type is not 0 (meaning “direct-access device”), and there is more than one LUN, repeat step #2 with additional LUNs until you find one that is a direct-access device. Some promotional flash drives use LUN 0 as an emulated CD-ROM and the flash drive is on LUN 1, so you’d want to use LUN 1 instead of LUN 0 in that case. After this process is complete, use the LUN you discovered for all of the rest of your commands going forward. If you don’t find anything matching, just stick with LUN 0 in case the inquiry data is wrong.
  4. Attempt a “PREVENT ALLOW MEDIUM REMOVAL” command. A lot of operating systems do this, and most drives don’t support that command. It’s no big deal if the command fails. Just continue on. Interestingly I didn’t observe Windows XP sending this command on the drive I tested, but Windows 10, Mac OS X, and Linux did. I don’t know whether to include this command or not. It works for me.
  5. Keep attempting a “READ CAPACITY (10)” command until it succeeds. This will tell you the size of the drive in blocks (minus 1, because it returns the address of the last block), as well as the block size in bytes.
  6. Try a “MODE SENSE (6)” command, requesting 192 bytes of data on mode page 0x3F. The 192 matches what other operating systems request, so it’s best to match that. In the response data, if bit 7 of byte 2 is set, the drive is read-only. If the mode sense command fails, just move on. I haven’t found a drive that fails this command though.
  7. Just to be safe, do “TEST UNIT READY” again, repeating until it returns success. Now you are ready to send all the “READ (10)” and WRITE (10)” commands you want to send.

In all of the places where I said to keep attempting something, I have a timeout of 5 seconds. If I don’t succeed and 5 seconds have elapsed in that step, I bail with an error. No drive I’ve tested so far has caused the 5 second timer to elapse, but if you’re extra worried you could try increasing the timeout.

Things to watch out for

As I tested various drives, I noted strange behaviors in certain cases. Here is a list of things you should probably watch out for.

Mass storage reset

The sample code that came with the USB library I use performed a “Bulk-Only Mass Storage Reset” command as the first step. None of the operating systems I tested did this, so I removed it. I think you should only use this command as a last resort if you have lost communication with the drive and it has stopped responding to your CBWs. (Make sure the drive isn’t simply waiting for you to read back a CSW after a stall or something too…)

Drives that are both a CD and flash drive

As I mentioned above, some promotional drives are both a CD-ROM and a flash drive. Check for that type of drive with the “Get Max LUN” command, and use the INQUIRY data on each LUN to find the one that’s actually the flash drive.

MODE SENSE (6) or MODE SENSE (10)?

While I was checking out various operating systems, I noted that sometimes Windows and Mac OS X use “MODE SENSE (6)” and sometimes they use “MODE SENSE (10)”. Linux seems to always do “MODE SENSE (6)”. I couldn’t figure out how Windows and OS X were making that determination.

I originally tried just always using “MODE SENSE (10)”, since I also always use “READ (10)” and “WRITE (10)”, so I figured why not the same with mode sense? However, that was a mistake. Some drives don’t support that command, and others return incorrect results in it. One drive I tested was particularly frustrating. Its “MODE SENSE (10)” response indicated that it was locked, even though it wasn’t. Its “MODE SENSE (6)” response correctly said it was not locked.

The moral of this story? Just stick with “MODE SENSE (6)”. Every drive I’ve tested supports it and returns correct-ish data. One drive I tested returns an incorrect data length as the first byte of the SCSI response data (the mode parameter header), so if the drive only returns 4 bytes but claims there are 70 in the response, you might want to limit your parser to only check the first 4.

As I said earlier, you should request 192 bytes of data in this command to match what other operating systems do. I’d recommend requesting mode page 0x3F, which means “all pages”. I’ve read online that some misbehaving drives may get confused if you send a mode sense request for any other page or data length.

First TEST UNIT READY command fails

On a lot of drives, the first “TEST UNIT READY” command returns failure. The sense data (obtained with “REQUEST SENSE”; see below) indicates it’s a temporary condition and to try again. On most of these drives, the next attempt succeeds. On one drive I tested, it failed the first 14 attempts.

This is why my initialization sequence says to keep trying “TEST UNIT READY” until it succeeds. I added “INQUIRY” in there as well because it seemed other OSes would intermix “INQUIRY” with it too. If after 5 seconds (or whatever time limit you’re comfortable with) it still hasn’t responded with success, then maybe something really is wrong.

Repeat failed commands

Maybe I should have just generalized the above section, but I’ll repeat it here. If a command fails that you really care about, just try again. I have it set up so if a “READ (10)” or “WRITE (10)” fails, I try again a few times before immediately bailing with an error.

In general, if an error occurs because the CSW indicates failure (and this rule of thumb also applies during the initialization sequence I described above), you should follow up with a “REQUEST SENSE” command to read information back about the failure before sending any other commands. Why? Because all the other operating systems do it too, so it’s a good idea. You are guaranteed that the drive has been tested under that behavior.

Theoretically the drive will tell you in the returned sense data whether the command failed due to a temporary condition or if the command is not supported. In practice, I do the “REQUEST SENSE” command and read the response from the device, but I ignore the content of the response. I simply repeat commands that I know are important, and I live with failure and move on if they’re not important (e.g. “PREVENT ALLOW MEDIUM REMOVAL”).

Write delays

This is probably obvious, but I’m pointing it out anyway. Sometimes “WRITE (10)” operations take a while to complete. When this happens, the drive will respond with NAKs until it’s finished. The NAKs could occur at any point — maybe while trying to read back the CSW, or while sending the next CBW, or in the middle of the data transfer process. If you’re designing a communication protocol that receives data and writes it to disk, make sure it has the ability to pause if the disk is too slow to keep up.

Handling short responses

In some cases, you will request data from the flash drive, but it will respond with less data than you requested. A perfect example of this situation is the “MODE SENSE (6)” command when requesting 192 bytes of data. I haven’t found a flash drive yet that has 192 bytes of mode pages to respond with.

According to the bulk-only transport specification, if the host requests more data than the device can provide in this situation, it’s allowed to pad the response with extra data to match the requested length. If it doesn’t do this, it must stall the BULK IN endpoint after transmitting as much data as it can. I was able to observe different drives that implemented each of these behaviors.

It turns out that there are some flash drives that ignore the above requirement and do it a different way. They don’t pad the response with extra data, and they don’t stall the endpoint. They simply send as much data as they can, without stalling the endpoint afterward. Although these devices aren’t following the bulk-only transport specification correctly, it’s not too hard to handle this situation. If you receive a USB data packet shorter than the endpoint’s maximum packet size (a “short packet”) when reading a response, but you haven’t received all the data you expected, you know the response has been terminated early by the drive, and the next read you attempt will give you the CSW. The USB library I use didn’t handle this case properly and would hang when trying to read a “MODE SENSE (6)” response from a misbehaving drive. It kept reading after the short packet. The next packet was the CSW, which it thought contained more mode sense data, and then it hung waiting for the rest of the mode sense data to arrive. I fixed the library to look for short packets and terminate the transfer early.

Because of this situation above, you might not realize a well-behaved drive has correctly stalled the endpoint until you try reading the CSW. So if you notice a stall condition after attempting to read back a CSW, clear the stall and try again. Note that the specification specifically mentions that retrying a CSW read after a stall is allowed. I suspect they allow it because of situations like the one I just described.

Safely ejecting

Some drives I tested didn’t necessarily finish saving changes before they were unplugged. For example, if I wrote data to a temporary file, and then renamed the file as my last write operation, plugging the drive into the computer showed that the rename operation hadn’t completed successfully, even though I had definitely asked the filesystem library to unmount the filesystem.

This was caused because the drive in question (a Samsung flash drive) implements caching, and I hadn’t told the drive to flush its cache to disk.

There is probably a complicated way to set up the cache properly. I believe one of the mode pages optionally returned by the “MODE SENSE (6)” command contains cache information which you might be able to configure. I came up with a simpler solution that seems to work. I send a “SYNCHRONIZE CACHE (10)” command after I’m done and have told the filesystem library to unmount the filesystem. This fixed the issue with the Samsung drive. Some drives don’t support this command and return an error, but it doesn’t seem to hurt anything.

Unsupported commands

I have read online that some poorly-designed flash drives will stop responding if you send them a command they don’t support. I haven’t observed any such drives in action. If you are concerned about this, you might want to consider removing the “PREVENT ALLOW MEDIUM REMOVAL” command from the initialization sequence, because I’ve seen many drives that don’t support it. You might also want to find a safer way of flushing the write cache than what I chose. In my use case, the very last command I send prior to the drive being ejected is the “SYNCHRONIZE CACHE” command. If that command is unsupported and causes the drive firmware to crash, the drive probably doesn’t support caching anyway, so I can assume the data is already safely written to the disk and the user is about to unplug it anyway.

Final thoughts

If at all possible, get a hardware USB analyzer. They’re typically expensive, but they give you so much detail about everything that’s going on. I was lucky enough to find one on eBay for a reasonable price. I can’t imagine that I would have been able to do this level of troubleshooting without one.

Even if you don’t have a hardware analyzer, if you follow some of my suggestions above, your device will be much more likely to be compatible with all of the random flash drives your end users happen to try out.

For 100% compatibility, it would probably be best to try to replicate exactly what Windows does when a drive is plugged in. However, doing that is difficult. It’s sometimes hard to understand why Windows sends the commands it does.

I recently wanted to play around with Android development, but I didn’t have any Android devices. So I picked up a cheap Android tablet from Walmart. It’s an RCA Viking Pro 10.1″ running Android 6.0. The model number is RCT6303W87M, although in software it identifies itself as RCT6303W87M7. But…Walmart’s website says it’s an RCT6303W87 DKF. I have no idea what is really correct, but I figured I would write out all of the model numbers so that people from Google can find this post.

Anyway, I realized after I bought it that the micro-USB port is strictly for charging. Oops, my bad. It turns out that this tablet wasn’t really designed with USB connectivity as a device in mind. It does have a USB type A port, but that’s for connecting other devices to the tablet, not the other way around. I tried turning on developer mode and any options on the tablet I could find, but nothing allowed connectivity with the computer.

I did some Googling, which seemed to indicate that other people had been in this predicament. There was talk of a mysterious “special cable” that RCA provides as an option to buy. I also found people discussing using a USB A-to-A cable with varying levels of success. I decided the best thing to do would be to contact RCA support, which led me down a bit of a rabbit hole.

The friendly RCA support person told me I needed to buy a special cable, and gave me a link on RCA’s store to order it, along with instructions for using the cable — in particular you have to connect the cable while the tablet is off, and the blue end needs to be plugged into the tablet. The cable had a price tag of $5 on their store, but it looked just like a standard USB A-to-micro cable that everyone has laying around. I went ahead and ordered it anyway, but sure enough, it wasn’t actually a special cable. It was just a run-of-the-mill micro USB cable, which I had already tried myself. There wasn’t a blue end — the entire cable was black. To make matters worse, the cable came from Canada, so I had to overpay for shipping, not to mention the foreign transaction fee on my credit card.

I wrote back to RCA support. The same person who helped me first time apologized and indicated that I hadn’t actually ordered the special cable. It appears that the special cable is available from RCA, but it’s not publicly available on their site so you have to do a special order to get it. So this time RCA sent the correct cable my way for no additional charge.

Today the cable arrived, and it is indeed special. It’s a USB A-to-A cable (well…since it came from Canada, maybe we should call it an eh-to-eh cable?). The ends are clearly marked so you know which end goes to the tablet and which end goes to the computer, and the tablet’s end is blue (although you can’t see it in the picture, because the part that goes into the computer is the part that’s blue, like a USB 3.0 cable).

I don’t know if there’s anything special about the cable over other A-to-A cables. The blue end that goes to the tablet appears to be a USB 3.0 connector, which makes sense because USB 3.0 cables are typically blue. So there are extra pins for USB 3.0–but the tablet itself doesn’t actually have connections for any of those pins. I dunno. It’s a mystery. I think they just used a 3.0 connector so they could get one that is colored blue. I think there must be something special about the cable other than just being an A-to-A cable; why else would they mark which end is which? I don’t have an easy way to do any further tests on the cable to try to figure out which pins are connected to which pins.

I guess you could say I made out like a bandit, because the special cable would cost $15 according to the label on the package. The label indicates the product is a “special cable” and it’s for the RCT6513W87, so I assume that tablet has the same problem. For reference for readers, here are the instructions RCA provided me for using the cable:

  1. Tablet has to be completely off
  2. Connect the special cable from tablet to computer, please note that the blue end goes to the tablet
  3. Plug the AC adapter into the tablet
  4. Turn on the tablet
  5. Open My Computer to see if PC will recognize the device, if not, please proceed to the next step
  6. Open Device Manager on your PC
  7. Choose Portable Devices and select Upgrade Driver Software
  8. Click on browse my computer for driver software
  9. Select “Let me pick from a list of a device drivers on my computer”
  10. Go to Portable Device and choose MTP USB Device

As soon as you do this, your PC should recognize the tablet. [In] some instances, if [your] PC will not recognize the device again, you may have to [go] through the instruction[s] above.

I can confirm that if you start with the tablet turned off and then plug in the cable, it does seem to work properly and enumerate as a USB device on the computer as soon as you turn the tablet on. It worked out of the box with Android SDK on Linux. If you unplug the USB cable, you do end up having to power the tablet off in order to reconnect the USB, so if you do Android debugging, it would be smart to set up Wi-Fi debugging using the steps on this StackOverflow answer.

Hope this helps someone out there!

At work, I was trying to restore a Lenovo IdeaPad Z510 back to the factory default configuration after Ubuntu Linux had been installed on it. I wanted to restore it back to the factory default Windows 8.1 scheme. I noticed that Lenovo had a “One-Key Recovery” (OKR) option, so I decided to try it. I pressed the NOVO button and chose the system recovery option. When I tried to run system recovery, it gave me this error message:

The program cannot restore the system partition because its structure is incorrect. You may have to recreate the partition to continue.

I wasn’t the one who set the laptop up originally, but I was pretty sure that since Ubuntu was installed, it had changed something about the partition layout and broken something. So I booted into an Ubuntu Live USB stick and used GParted to check it out. Sure enough, a few extra partitions had been created for ext4 and swap. It appeared that the main Windows partition had been shrunk, and the extra Linux partitions had been created in the space made available. I deleted the extra Linux partitions and resized the main Windows partition to fill up the available space.

Unfortunately, that still didn’t fix it. Maybe it would in some people’s cases, but something was still wrong with the partition layout. I started digging into the recovery partitions on the laptop’s hard drive and found a file that contained info about the partition layout. It was on the partition called PBR_DRV, and the path to it was OKRBackup\Factory\Info.ini. This file contained info about the location, size, and options for each partition.

I was able to use the information in this file, combined with the “gdisk” command while booted into an Ubuntu Live USB, to fix everything. It turned out that all of my partitions were the correct size, but the “Attributes” from this file didn’t match — several of the partitions were missing the “don’t automount” flag.  I was able to use gdisk to set that flag on partitions that needed it in the expert menu. Also, for some reason, the MSR partition had been deleted, so I was able to recreate it (I created it as a new unformatted partition and set the label in GParted) and set the proper “Id” and “Type” GUID values using gdisk as well. I don’t know if Ubuntu deleted it, or if the person who installed Ubuntu manually deleted it.

For some reason, the partitions were numbered incorrectly after I did all of this, so I was able to use the “sort” function in gdisk to number the partitions properly afterward.

After I confirmed that the labels, locations, sizes, “Id”/”Type” GUIDs, and attributes all matched between Info.ini and the actual disk, I saved my changes to the partition table and tried running the system recovery. This did the trick and everything worked fine!

I didn’t write down paths or anything while I was doing this, and I had to do some Googling to find the names again, so I might be slightly off on some of the filenames. But what I described is pretty much what I did. For future reference, I did write down the following information about the partitions, in case anybody might find this useful:

  • sda1: WINRE_DRV
    • Start sector = 2048
    • End sector = 2050047
    • Size = 1000 MB
  • sda2: SYSTEM_DRV
    • Start sector = 2050048
    • End sector = 2582527
    • Size = 260 MB
  • sda3: LRS_ESP
    • Start sector = 2582528
    • End sector = 4630527
    • Size = 1000 MB
  • sda4: MSR
    • Start sector = 4630528
    • End sector = 4892671
    • Size = 128 MB
  • sda5: Windows8_OS
    • Start sector = 4892672
    • End sector = 1874599935
    • Size = 891.55 GB
  • sda6: LENOVO
    • Start sector = 1874599936
    • End sector = 1927028735
    • Size = 25 GB
  • sda7: PBR_DRV
    • Start sector = 1927028736
    • End sector = 1953523711
    • Size = 12.63 GB

After I upgraded Ubuntu MATE from 16.04 to 18.04, I noticed a really annoying behavior. Clicking in the empty area of a scrollbar (between the handle showing where your current position is and the up or down arrow) used to scroll up or down a page at a time. After the 18.04 upgrade, this behavior changed, and now it moves the handle directly to the position where you clicked. Some people like this, some people don’t. I don’t. Unfortunately, they didn’t decide to make this a configurable option in the graphical settings for some strange reason.

It turns out that this is actually a GTK 3 behavior — it’s caused because MATE recently changed to use GTK 3. Luckily, it’s an easy fix:

Go into the directory of the theme you are currently using. So for my current configuration, I went into this directory:

/usr/share/themes/Ambiant-MATE/gtk-3.0

Edit the “settings.ini” file. Add the following line to the bottom of it:

gtk-primary-button-warps-slider = false

Log out and back in, and you’re done.

After upgrading my machines from Ubuntu MATE 16.04 to 18.04, I noticed that something was subtly wrong with my top menu/panel: the network notification icon was missing. Something about the 16.04 to 18.04 upgrade process broke it. It happened on both of the machines I upgraded, so I don’t think it was a random thing that only I was experiencing.

Note: my fix for this will reset your panel layouts, so take a screenshot and/or save any info about the layout of your panels before following the steps below so you will have a reference for restoring it back the way you want it afterward.

To start out, I will share some background info: if you are used to the default menu layout of the top panel with the Applications, Places, and System menus, they decided to change this in 18.04. That layout is called “Traditional”, and Ubuntu MATE 18.04 now defaults to a new layout called “Familiar”. I personally don’t like the Familiar layout because it doesn’t have the Places menu, but I guess enough people liked it that they wanted to make it the default.

Anyway, the fix for this is to reset your panel to one of the default configurations, which will add everything back properly. But like I said in bold above, it will also erase any customizations you have done. Here’s the fix:

  1. Open up MATE Tweak. In the Traditional layout, it’s in the System -> Preferences -> Look and Feel menu.
  2. In MATE Tweak, click on the Panel category.
  3. In the top dropdown menu, choose the layout you want. To match Ubuntu 16.04’s default, choose Traditional. To match Ubuntu 18.04’s default, choose Familiar.
  4. Confirm that you want to change the layout by clicking OK.
  5. All done! Your network notification icon should be restored, and now you can reapply any of your customizations to the panel.

Another way to fix it is to right-click on the panel and choose Reset Panel, but I think the method I described above is better because it lets you pick which layout you want to start from.

I recently upgraded a couple of my machines from Ubuntu MATE 16.04 to 18.04. On both of them, I noticed that after the upgrade, the login window did not look correct. There was no background image. It didn’t look like a fresh 18.04 install. Plus, any changes I did in the Administration -> Login Window program didn’t take effect. I did some digging, and figured out what was going on.

Sometime between 16.04 and 18.04 (it appears that this happened starting in 17.10), Ubuntu MATE decided to stop using lightdm-gtk-greeter, and instead switched to using slick-greeter. It seems to be that on update installs, lightdm-gtk-greeter is not being removed, so it’s defaulting to using it rather than slick-greeter.

I was able to solve this very easily by running the following command:

sudo apt purge lightdm-gtk-greeter

After that, I logged out, and Ubuntu MATE’s correct login screen appeared. It’s as simple as that!

At work, we still have a Windows 98 computer that we use for programming a few very old products that we no longer sell but still support. I tried to do some research so we could move this process onto a newer computer, but I wasn’t able to find a suitable tool. Thus, I’m stuck maintaining a Windows 98 computer until we no longer have to support the product. I do any necessary development work on a different computer, but I still need to get the program binaries onto the Windows 98 machine. The computer doesn’t have a floppy or CD drive, and Windows 98 didn’t come with a USB mass storage driver. Honestly, all of those solutions would be inconvenient for me anyway.

It’s super insecure, but you can enable file sharing on the Windows 98 computer and connect to it from Linux. Programming binaries to old microcontrollers is literally the only thing this computer is used for, so I’m not too concerned about the security pitfalls of SMB.

There are several tools designed for accessing an SMB share from Linux. Most modern desktop environments provide an interface to access Windows file shares. From the command line, I have used smbclient from Samba, which also works great in most situations. When we recently added subnets to our work network and the Windows machine was on a different subnet from my development machine, I had to come up with a solution to connect to it because it no longer showed up automatically. I didn’t want to set up a WINS server to enable cross-subnet Windows file sharing just for this one little application. I couldn’t get smbclient to connect to the Windows 98 computer properly across subnets in this scenario, so this blog post will focus on how to use Linux’s built-in CIFS support so you can use the standard “mount” command, which does work across subnets if you do it right.

First of all, you will need to ensure you have mount.cifs installed. In Ubuntu 16.04, you can type the following command to install it:

sudo apt install cifs-utils

Here is a sample mount command:

sudo mount -t cifs //192.168.2.7/MYSHARE /tmp/windows -oservern=MYSERVER,guest,vers=1.0

Here is an explanation of each parameter:

  • -t cifs
    • This specifies to the mount command that we’re trying to mount a CIFS share.
  • //192.168.2.7/MYSHARE
    • This is the SMB server and share you are trying to mount. 192.168.2.7 is the IP address of the server. MYSHARE is the name of the share on the server. Note that I am using forward slashes here, not backslashes like you would expect on Windows.
  • /tmp/windows
    • This is the local directory to mount the share on. Make sure you create the directory before trying to run this command.
  • -oservern=MYSERVER…
    • “-o” specifies options. Everything following is a comma-separated list of options:
    • servern=MYSERVER
      • This option specifies that the name of the server you are connecting to is MYSERVER. You have to specify the name of the server you’re trying to connect to if you try to mount a Windows 98 share, or else it won’t work. So this option is super important.
    • guest
      • This specifies that we’re trying to connect as a guest. Otherwise, you can use the username= and password= options. It’s probably not a good idea to use the password= option though, because it leaves your password visible in plain text; there is a credentials= option that works a lot better by letting you specify a filename containing credentials.
    • vers=1.0
      • This tells the kernel to use the SMBv1 protocol. It used to be the default before Linux 4.13 came out, but now Linux defaults to a newer protocol for security reasons.

There are many other parameters that you may need to use for various reasons. You can get information about all of the available parameters by typing this command into your terminal:

man mount.cifs

Note that prior to Linux 4.13, I did not have to specify the protocol version as 1.0. It just worked. But with Linux 4.13 and later, it defaults to a newer protocol version for security reasons. If you try to connect to a Windows 98 share without specifying protocol version 1.0, the mount command will hang and eventually fail. I was able to determine what was going on by looking at a Wireshark trace, which then led me to the mount.cifs manpage to discover how to specify an older protocol version.

I have noticed that if you try to replace an existing file, you will get an error. So in my experience, you have to delete the existing file first. This is not a big deal in my use case.

To unmount it when you’re done:

sudo umount /tmp/windows

Hope this helps someone out there!

If you can believe it, Mac OS X 10.4 “Tiger” is over 12 years old as of this writing. It was first released in April of 2005. It was also the version that Apple first used on its Intel Macs in 2006. Because the Intel version came out in 2006 after the PowerPC version had already been in stores, it’s kind of a weird release. There wasn’t a retail copy of the Intel version of Tiger. It was only bundled with the first Intel Macs before 10.5 “Leopard” came out in 2007.

Because of the way it was weirdly released, it’s not super common to virtualize OS X 10.4 for Intel. Nobody’s really using it anymore because it’s so old. It’s probably full of security holes. And technically, it’s against OS X’s license agreement to virtualize it (same with the non-server versions of 10.5 and 10.6). With that said, I really doubt Apple cares about such an old version of OS X these days, and I think creating a VM of it is a really cool thing to do for educational purposes. Who knows — maybe it’s still useful for certain developers who still need to test how things work on 10.4 without keeping an old power-hungry machine around that is capable of running it.

Read the rest of this entry

As soon as you add a USB host port to your microcontroller project, a lot of possibilities suddenly open up. You can add support for plugging in a flash drive for firmware updates, diagnostics, and all kinds of miscellaneous data transfer. One task I’ve needed to complete in the past is the ability to save an Excel .xls spreadsheet. After doing some research, I found that this was actually a fairly difficult task on a microcontroller with limited memory. Most of the C/C++ Excel libraries I’ve found aren’t prepared to work in a microcontroller environment, and they generally do dynamic memory allocation because, well, Excel files are dynamically sized.

I’m going to assume that you already have USB mass storage support figured out for your particular project. This will usually involve using a USB library that provides the USB functionality. Many USB libraries also provide support for accessing mass storage devices. On top of this, you will also need a filesystem library for accessing a FAT32 filesystem, which is arguably the most common filesystem in use on flash drives today. One library I would recommend for working with FAT32 filesystems is FatFS. Note that FatFS doesn’t provide any of the USB functionality — you will have to hook it up to your USB library in diskio.c.

Now, back to the actual file generation. One thing you can do is simply generate a .csv file instead. That strategy works pretty well, and it may end up being all that you need for your particular project. One downside is that in certain cases, it will bring up a dialog box that forces the user to specify options for importing the file. LibreOffice is a particular offender in this regard.

I wanted a solution that generated a real .xls file that Excel and LibreOffice could open without any prompting and didn’t require the use of malloc(). I ended up writing my own library called MicroXLSWriter. MicroXLSWriter is a simple library that should fit on any embedded system. It generates files in a very old .xls format (BIFF2, from Excel 2.0). Modern versions of Excel (tested through Office 2007) can still open BIFF2 files without any problems. My code doesn’t completely follow the BIFF2 format perfectly, but it’s good enough that Excel and LibreOffice don’t seem to complain.

MicroXLSWriter is very limited. It doesn’t support any cell formatting or styles. You can’t customize fonts, colors, borders, or anything like that. The only thing it lets you do is set the width of each column and put text or a number into each cell. For what I needed, this is completely fine. BIFF2 files aren’t typically supported by .xls manipulation libraries, so if you’re generating something that won’t simply be opened by a user with Excel or LibreOffice, you should probably look elsewhere — if you can find a better alternative that still fits in a microcontroller.

If you’re interested, try it out. The source code is on GitHub and it has a very permissive 2-clause BSD license that allows you to use it in whatever commercial or open-source project you want, as long as you follow the simple requirements about preserving the copyright notice in materials distributed with your project (e.g. documentation). If you get a chance, let me know if you use it! I’m curious to hear about various applications where it ends up being used.

For various reasons, you might end up needing to hook up your iOS device directly to an Ethernet network without using Wi-Fi. For example, maybe you’re in an area with bad Wi-Fi reception. Or perhaps you are trying to do something that requires high bandwidth and your Wi-Fi access point doesn’t have the best speed.

Well, it turns out that iOS has supported Ethernet for a while. Obviously your iOS device doesn’t have an Ethernet port, so you have to use a USB-to-Ethernet adapter. As of iOS 10.2, there is even a settings screen that appears when you plug in a supported adapter. Older versions of iOS apparently still supported Ethernet, but didn’t provide a screen for setting it up, so you were probably stuck with the default settings (grabbing an IP address from a DHCP server). Now that there is a setup screen, you can configure it exactly how you want.

I tested it out with three devices tonight, and they all worked flawlessly: iPhone 6, iPhone 7, and iPad Air.

Read the rest of this entry