A lot has happened with the Pinoccio team over the last couple of months. We’ve been very heads-down in trying to get our first run of boards out to our crowdfunding campaign backers. We’ve also spent a lot of time working to build up our team from just Sally and I to a team of 11 people all over the world working with us. Lastly, we’ve focused quite a bit on the testing and repeatability of the quality of our hardware. This is exciting to us! But we want to emerge from our heads-down work to give you some insight into the last days of a hardware product before its first shipment. It’s quite a wild ride!
So first, if you’re just now catching up with what exactly we’re working on–Pinoccio is a hardware and software platform that provides a simple way to connect anything in the physical world to the Internet, and to each other. As you can imagine, this spans a lot of different use-cases.
One of the tasks that has taken a considerable amount of time is the design, development, and assembly of the test jigs that we use to test each and every Pinoccio Scout and Pinoccio Wi-Fi Backpack before it ships. It’s interesting, because in software, testing is an important part of development. Most people do it, some do not, and almost nobody does it to cover 100% of the software written.
However, in hardware, it’s absolutely essential that every part of a circuit is tested. Nothing’s worse than waiting for your Pinoccio board to show up after months of waiting, only to find out the charge circuit doesn’t work on one of them, or perhaps the RGB LED on the other only works for red and blue. Ugh! So we insisted that each board would go through a rigorous test plan before being shipped. Any that don’t pass get set aside to examine–and possibly repair–at a later time.
So we started down the path several months ago–designing and assembling test jigs for both the Scouts and Wi-Fi backpacks. Here’s what an assembled jig looks like:
It’s missing the top PCB where the actual Scout sits upon before being tested. But you can see the pogo pins required to connect to and test each pin on the board. You’ll also see various components that help to test the Scout, such as an AVR328p there in front. We chose that since it can run at 5V, and thus can test our three voltage rails on the Scout: 5V USB, 4.2V battery, and 3.3V regulated output. Kinda nifty!
To the far left of the image, you can see headers where another Pinoccio Scout plugs in. That Pinoccio actually drives the test process. Press the blue button to the right, and the driver Scout runs the Scout-under-test through a gauntlet of tests. Here’s what we test in this jig:
We had a problem though: If you have one Pinoccio test another Pinoccio, you don’t have enough pins, because some pins on the driver Pinoccio need to do things like turn USB power on/off, or turn the power switch of the Scout-under-test on/off. How do you do that and also test each pin? For that we rely on our trusty 328p chip there!
We wrote a small I2C bus protocol that lets the driver Scout send the 328p commands, and the additional pins on the 328p handle the remaining tests. Woo!
But when a Scout gets to us from the assembly house, it’s completely blank–there isn’t any code or anything on there–including code we need to be Arduino-compatible. So we have to flash three pieces of code onto the Scout-under-test. Most people use a device like an AVR ISPmkII to do this, which is what’s called an In-System-Programmer. Here’s what it looks like:
This is fine for flashing a small number of boards, but we’d need to buy as many of these programmers as test jigs we have (six today, and probably a dozen before long.) Furthermore, the ISPmkII can be a bit slow when programming these chips–sometimes 25% of the speed we can get by flashing directly. To solve this problem, we did some research and came across Nick Gammon’s great program that allows you to program one ATmega chip from another ATmega chip. This is fantastic, as now we can access any area of both chips directly, and very fast. Awesome! So now we add these features to our test jig:
But we ran into an issue. We were able to encode both bootloader files directly as C byte arrays in our driver code, just like Nick does. But when it came to encoding our default sketch, something was wrong! Our compiler was spitting out all sorts of errors with that big encoded sketch array. After lots of digging, this post came across the radar, and that explained everything. Our sketch was larger than 32k, so of course it wouldn’t fit. Now what?
Well, when designing the test jig, we wanted to have some on-jig storage just in case something like this happened. So we added a 2MB serial flash chip to it–the same one that we include on every Wi-Fi Backpack. So now we just need to get the default sketch from the computer onto that flash chip, then write supporting code on the driver Scout so it will read that sketch file from the local storage chip and write it to the flash area of the Scout-under-test. Easy, right?
Now we’re writing Python serial code to load a .hex file compiled from avr-gcc, onto the serial flash chip on the test jig through serial USB. This took some trial and error to get right, but ended up working once we erased the correct serial flash sectors. You do know that you have to erase a serial flash sector before you write to it right? Oh, you didn’t know that? Neither did I. It’s on page 46 of the datasheet. Now I know and I’ll never forget it again.
Great! So now we have everything testing and flashing brand new Scouts straight out of the panel. It takes about 20 seconds to do everything. Here’s what our output looks like on the Scout test jig today:
Next is the Wi-Fi test jig. Surely this one’s a lot easier! We don’t need intermediate 328p microcontroller or anything–just connect a few pins together, test the various components, and that’s it, right?
Well, almost. There’s a component on our Wi-Fi Backpack that handles the Wi-Fi connections itself. It’s the small green block near the back of the board shown here:
Well, it turns out that the modules that we received from our component supplier had an older version of the Wi-Fi firmware on there. And that firmware has a couple of bugs that would make a connection randomly drop over a certain amount of data transfer. A no-go for us. All we need to do is update the firmware as part of the test jig process and we’re good to go.
/me looks up how to flash firmware for this module
Uhh, what? The only way to do this is with a DOS program on Windows? Uhhhh. How are we going to automate thousands of firmware updates on the Wi-Fi boards running VirtualBox on a Mac? Time to roll up our sleeves and do some reverse engineering!
If you’re not familiar with logic analyzers, I highly recommend you take a look. They will absolutely save your bacon numerous times in hardware projects. What we did here to figure out how to automate this firmware upgrade was to hook our trusty Saleae logic analyzer to the serial pins of the Wi-Fi module, and then ran the updater from the DOS program and recorded what happened. Here’s what we see after it runs:
Here we see three main blocks of code getting transmitted (TX1) to the module, and what looks like a bunch of acknowledgements that are sent back (RX1) during the process. This looks right. We have three binary firmware files given to us from the Wi-Fi module company, so we just need to know what handshake bytes starts the update, send that data, then send the file bytes afterwards. To be correct, we should expect back the acknowledgements in our driver Scout to ensure everything’s writing correctly.
Zooming in, the quickest way to reverse engineer this thing is to get these handshakes and hex files down on paper or in a file where you can look for patterns. We should expect some number of hex values for the handshake, then the file bytes that we want to upload should show up somewhere in there. First, let’s see what the first few bytes of the file is:
So we’re looking for 51 43 00 08 and so on. So we zoom in and see a bunch of this:
Nothing yet. Keep scrolling right, and, aha!
Woo! So now we know the characters that need to be sent by our driver Scout before we load the file into the module. We remember from the Scout test jig that we can’t load this big 118k file in the driver Scout’s code (remember that 32k array size limit?), so we’ll have to store it in serial flash on the test jig just like we did for the Scout jig. No problem, we have our Python script that we already wrote for the Scout jig that can stream any binary file onto the serial flash on the test jig.
When reverse-engineering, some people like to put everything in a text editor and just search and count. Me, I like to write down the hex values on paper to look for patterns, underline, scribble, etc. Here’s a pic of my notes for the preamble of bytes before the hex file data gets loaded. I underline values that look like they change within each chunk, because we have to get those right if we’re going to emulate that stupid DOS program:
Here you can see we need to send a dozen or so bytes starting with A5, 02, etc, then we send 1496 bytes of firmware image data. Then we should receive an acknowledge, along with the same sequence number that we just sent. It looks like the sequence number starts with 0, and increments on each 1496 bytes.
We need to do this for all three firmware images for the Wi-Fi module. But at least we know how to send them, thanks to the logic analyzer telling us exactly what gets sent, and how fast. (You can see the milliseconds between each send and receive in the upper left.)
There’s an old joke that says the first 90% of a hardware project is tough, but the second 90% is even harder. Well, we’re neck deep in the second 90% now. The test jig work is one of around nine milestones that need to be completed before we ship our first boards. Luckily, all remaining milestones are very close to complete.
We’re working on deploying a page on our site where we’ll update our remaining milestones, to give everyone better visibility into what’s left before we ship. I’m very proud of our team for pushing hard during these last weeks to ensure the Pinoccio experience is of high quality, is stable, works as expected, and surprises and delights those who use it for their projects. All the test-jigging, reverse-engineering, and the numerous other web, API, and networking code our team is working on is worth the hard work for what we set out to do. We hope you’re as excited as we are!
Filed under An Open Hardware Business