Software Glitch Pauses LightSail Test Mission

The Planetary Society’s LightSail test mission is paused while engineers wait out a suspected software glitch that has silenced the solar sailing spacecraft. Following a successful start to the mission last Wednesday, LightSail spent more than two days sending about 140 data packets back to Earth.

But the long Memorial Day weekend here in the United States offered no respite for the LightSail team, as they scrambled to figure out why the spacecraft's automated telemetry chirps suddenly fell silent. It is now believed that a vulnerability in the software controlling the main avionics board halted spacecraft operations, leaving a reboot as the only remedy to continue the mission. When that occurs, the team will likely initiate a manual sail deployment as soon as possible.

What happened?

As of late Friday afternoon, LightSail was continuing to operate normally. The spacecraft’s ground stations at Cal Poly San Luis Obispo and Georgia Tech were receiving data on each pass. Power and temperature readings were trending stably, and the spacecraft was in good health. 

But inside the spacecraft's Linux-based flight software, a problem was brewing. Every 15 seconds, LightSail transmits a telemetry beacon packet. The software controlling the main system board writes corresponding information to a file called beacon.csv. If you’re not familiar with CSV files, you can think of them as simplified spreadsheets—in fact, most can be opened with Microsoft Excel. 

As more beacons are transmitted, the file grows in size. When it reaches 32 megabytes—roughly the size of ten compressed music files—it can crash the flight system. The manufacturer of the avionics board corrected this glitch in later software revisions. But alas, LightSail’s software version doesn’t include the update.

Late Friday, the team received a heads-up warning them of the vulnerability. A fix was quickly devised to prevent the spacecraft from crashing, and it was scheduled to be uploaded during the next ground station pass. But before that happened, LightSail fell silent. The last data packet received from the spacecraft was May 22 at 21:31 UTC (5:31 p.m. EDT). 

The aftermath

LightSail is likley now frozen, not unlike the way a desktop computer suddenly stops responding. A reboot should clear the contents of the problematic beacon.csv file, giving the team a couple days to implement a fix. But to pull a phrase from recent mission reports, the outcome of the freeze is “non-deterministic.” That means sometimes the processor will still accept a reboot command; other times, it won’t. It’s similar to the way you deal with a frozen computer: You can try to struggle past sluggish menus and click reboot, but sometimes, your only recourse is pressing the power button.

As of Tuesday afternoon, there have been 37 Cal Poly and Georgia Tech ground station passes. During half of those, reboot commands were sent to the spacecraft. Nothing has happened yet. Therefore, we have to assume that LightSail is only going to respond to the power button method.

When I filmed an interview with our CEO, Bill Nye, and system engineer Barbara Plante, last year, Nye points out a piece of hardware strapped to BenchSat, LightSail’s acrylic-mounted testing clone:

Since we can’t send anyone into space to reboot LightSail, we may have to wait for the spacecraft to reboot on its own. Spacecraft are susceptible to charged particles zipping through deep space, many of which get trapped inside Earth’s magnetic field. If one of these particles strikes an electronics component in just the right way, it can cause a reboot. This is not an uncommon occurrence for CubeSats, or even larger spacecraft, for that matter. Cal Poly’s experience with CubeSats suggest most experience a reboot in the first three weeks; I spoke with another CubeSat team that rebooted after six. Coincidentally, this is close to the original 28-day sail deployment timeline. 

Radio operators

A lot of amateur radio enthusiasts have been helping to track LightSail. Many of you have sent in data packets that you’ve received, and I’ve been getting a lot of questions about how to decode packet contents. First of all, thank you! The first operator to grab a full packet was Ken Swaggart (call sign W7KKE) from Lincoln City, Oregon. His packet was nabbed at 20:01 UTC on May 20—just five hours after launch.

This is a test flight in more ways than one. In addition to figuring out how LightSail behaves in space, we’re also refining our procedures for getting information out to the public—including the radio community. Because the team has been busy with actual spacecraft operations, I’ve been trying to field those inquiries myself. Unfortunately, I’m far from an expert in this area. I know what the data should look like after they are decoded; not so much on the raw radio signal side of things.

We don’t currently have a public decoder available, but Dr. John Bellardo at Cal Poly generously spent some time building a prototype web version that takes raw hexidecimal data and converts it to plain text. I’ve tried using it to decode some of the packets you’ve sent me, but to no avail thus far—meaning I have to send them over to our engineers instead. The team’s first duty is to the spacecraft itself, but we’ll get to your requests eventually, and iron out the process for our 2016 flight. In the meantime, you can still listen for LightSail and send reports to [email protected]. For more details, check out our Mission Control page at sail.planetary.org/missioncontrol.

Finally, many folks have written in to tell me that our spacecraft ground track is getting stale. We know, and we’re just as anxious as you are to get that updated. We rely on JSpOC, the Joint Space Operations Center, for updated Two-Line Element sets. TLEs are numerical descriptions of an object’s orbit. Thus far, our only TLE came from the launch vehicle shortly after LightSail was deposited into orbit.

When we get a new TLE, we expect there to be several that represent the entire group of CubeSats that hitched a ride to space aboard the Atlas V Centaur upper stage. Since the spacecraft remain bunched relatively close together, there might not be discrete one-to-one matches between each TLE and its corresponding spacecraft. One way we can narrow down which spacecraft is LightSail is to measure its doppler shift, but since we're no longer transmitting, the process may take more time.

Next steps

Cal Poly and Georgia Tech will keep listening for LightSail on each ground pass. Furthermore, Cal Poly is automating the reboot command transmission to be sent every few ground station passes, on the hope that one command sneaks through (we don't send the command on every pass because a successful reboot triggers a waiting period before beacon transmissions begin). But as of right now, we can’t do much except wait, hoping a charged particle smacks the spacecraft in just the right way to cause a reboot. LightSail is capable of remaining in orbit about six months in its CubeSat form.

In the meantime, the team is looking at several fixes to work around the software vulnerability once contact is reestablished. One is a Linux file redirect that would send the contents of the troublesome beacon.csv file to a null location, a sort-of software black hole. Lab testing on this fix has been promising—over a gigabyte of beacon packets have already been sent into nothingness without a system freeze.

When we hear from LightSail again, the team will likely initiate a manual sail deployment as soon as possible. Planning has already started on that front—we’ll keep you updated.

In the meantime, I'll be refreshing the spacecraft's raw telemetry packet repository, ready to jump at the first sign of new data. With a little luck, the test mission isn’t over just yet. Hopefully LightSail will follow trends established by other CubeSat missions and reboot soon.