Founder/CEO of Albedo here. We published a detailed write-up of our first VLEO satellite mission (Clarity-1) — including imagery, what worked, what broke, and learnings we're taking forward. Happy to answer questions.
Clarity is designed for a GSD (ground sample distance) of 10 cm. Generally the industry uses resolution<>GSD interchangeably. Agree it's not the true definition of resolution. But I'd argue the diffraction limit is an incomplete metric as well, like how spatial sampling is balanced with other MTF contributors (e.g. jitter/smear). For complete metrics, we like 1) NIIRS or 2) % contrast for a given object size on the ground (i.e. system MTF translated to ground units, not image-space units).
The main performance goal for us was NIIRS 7, and we decomposed GSD/MTF/SNR contributors optimized for affordability when we architected the system
How do you manage along-track smear? At those altitudes you're pushing close to 8km/s. Traditionally you'd either need to keep the satellite rotating through the collect or somehow keep the integration time in the single digit microseconds.
A little more detail that we didn't get into in the post. The 3-CMG control mode we first uploaded was v1. We had plans to improve the agility with later versions. In v1, we didn't have quite enough rate to match earth's, even with max TDI. We called it Banana Scanning. Kind of like slipping over the earth.
Net - the CMG imagery we captured had a few pixels of along-track smear in it. Which would have been removed in post-processing if we had made it through focus cal.
How did you manage meaningful attitude control with only torque rods? They would need to big (read: heavy) to be useful — was this just stabilising in inertial frame or active pointing?
Mag dipoles in chassis and components tend to lock tumbling satellites into the Earth’s magnetic field. Did you see this? Or did you see atmospheric drag dominate at this altitude?
I'm AyJay, Topher's co-founder and Albedo's CTO. We'll actually be publishing a paper here in a few weeks detailing how we got 3-axis torque rod control so you can get the real nitty gritty details then.
We got here after stacking quite a few capabilities we'd developed on top of one another and realizing we were beginning to see behavior we should be able to wrap up into a viable control strategy.
Traditional approaches to torque rod control rely on convergence over long time horizons spanning many orbits, but this artificially restricts the control objectives that can be accomplished. Our momentum control method reduced convergence time by incorporating both current and future magnetic field estimates into a special built Lyapunov-based control law we'd be perfecting for VLEO. By the time the issue popped up, we already had a lot of the ingredients needed and were able to get our algorithms to control within an orbit or two of initialization and then were able to stay coarsely stable for most inertial ECI attitudes albeit with wide pointing error bars as stated in the article. For what we needed though, it was perfect.
I'd love to read this paper! This was on my mind when I was GNC lead for an undergraduate project at Michigan Tech (Oculus-ASR - Nanosat-6 winner). We had a combined controller for reaction wheels and magtorque rods.
>The drag coefficient was the headline: 12% better than our design target.
Is the drag much better than a regular cubesat? It doesn't look tremendously aerodynamic. From the description I was kind of expecting a design that minimized frontal area.
>Additional surface treatments will improve drag coefficient further.
Is surface drag that much of a contributor at orbital velocity?
Ultimately it's about the ballistic coefficient. You want high mass, low cross-sectional area, and low drag coefficient (Cd). With propulsion for station-keeping, it's challenging to capture the VLEO benefits with a regular cubesat. That said, there are VLEO architectures different than Clarity that make sense for other mission areas.
Yes it's a big contributor. The atmosphere in VLEO behaves as free molecular flow instead of a continuous fluid.
> It is undesirable to have a definition that will change with improving technology, so one might argue
that the correct way to define space is to pick the lowest altitude at which any satellite can remain in orbit,
and thus the lowest ballistic coefficent possible should be adopted - a ten-meter-diameter solid sphere of
pure osmium, perhaps, which would have B of 8×10^−6 m^2/kg and an effective Karman line of z(-4) at the
tropopause
Assuming I did the math right such a satellite would only run $265 million USD for the materials (launch costs for an object of ~9k kg left as an exercise for the reader). That's far more affordable than I had expected. Amusing thought.
First - they never want to use someone else software framework again (an early SW architect decided that would accelerate things but we ended up re-writing almost all of it) and it was all C++ on the satellite. We ran linux with preempt_rt.
We wrote everything from low level drivers to the top level application and the corresponding ground software for commanding and planning as well. Going forward, we're writing everything top to bottom, just to simplify and have total ownership since we're basically there already.
For testing we hit it at multiple levels: unit test, hardware in the loop, a custom "flight software in test" we called "FIT" which executed a few different simulated mission scenarios, and we tried to hit as many fault cases as we could too. It was pretty stressful for the team tbh but they were super stoked to see how well it worked on orbit.
A big one for us in a super high resolution mission like this is the timing determinism (low latency/low jitter) of the guidance, navigation, and control (GNC) thread. Basically it needs execute on time, every cycle, for us to achieve the mission. Getting enough timing instrumentation was tough with the framework we had selected and we eventually got there, but making sure the "hot loop" didn't miss deadlines was more a function of working with that framework than any limitation of linux operating well enough in a RTOS fashion for us.
Moving fast to make launch, we had missed a harness checkout step that would’ve caught a missing comms connection into an FPGA, and it was masked because our redundant comms channel made everything look nominal.
On orbit, we fixed it by pushing an FPGA update and adding software-level switching between the channels to prove the update applied and isolate the hardware path — which worked. Broader lesson, it is possible to design a sw stack capable of making updates to traditionally burned-in components.
> it was masked because our redundant comms channel made everything look nominal.
Hah, this has bitten me often enough I check for it in test suites now - ok, you’ve proven the system works and the backup works, have you proven the primary works? Another in the long list of ways you don’t expect a system to bite you until it does…
I'm just saying that if they use a traditional IP VPN over the Internet, traffic is of course encrypted, but the two endpoints terminating the VPN must have a public IP.
Of course, this is not necessary if they simply use UHF radio signals encoding bits with pulse modulation, or whatever.
Indeed, that's indeed what I was curious about (Internet vs other channels), since at least part of their tech stack in orbit is relatively straightforward (e.g. linux with preempt_rt).
Why would you VPN here? If you did, why would you do so over the public internet? You can route IP packets over internal links, including via radio (that is the entire point of WiFi after all).
Although it occurs to me that "does your network stack employ either ethernet or IP (and what were the considerations)" might be an interesting question.
Let me dream about the one guy working remote for a satellite company and just jumping into a direct VPN with a satellite, won't you? :)
All kidding aside, there are some protocols, like FTP or RTSP, which don't play well with NAT because they include IP addresses in the payload itself. Some solutions exist (so called ALG) but they are often fragile. If the satellite was using some of these protocols to talk to something on a public cloud platform (say, send images via FTP to an EC2 VM), satellites could have a public address to avoid NAT issues, and that point you could also use it as a management address (although maybe only as a backup path).
It's a bit far fetched, but when it comes to satellites, you could say "sky is the limit" :)
Ignoring for a moment the fact that Grafana could be self-hosted or in SaaS, Grafana is heavily used to collect logs and metrics from standard servers.
Of course, maybe they built their own integrations to convert raw logs and metrics sent via plain pulse modulation to plain syslog and Prometheus metrics, but maybe it's just that they're using (probably private) IP addresses on board and they are simply streaming logs and metrics to the ground using standard TCP/UDP protocols.
From my perspective, the number one reason we had a well functioning satellite out of the gate is my philosophy of testing "safe mode first". What that means is in a graduated fashion, test that the hardware and software together can always get you into a safe mode - which is usually power positive, attitude stable, and communicative. So our software integration flows hit this mission thread over and over and over with each update. If we shipped a new software feature, make sure you got to safe mode. If we found a bug that prevented it, it's the first thing to triage. We build out our pipelines to simulate this as much as we could and then ran it again on the development hardware and eventually would load a release onto flight once we were confident this was always solid. If you're going to develop for space, start here.
That principle goes far further than developing for space, but for space the pay-off is the largest. It also applies to maritime, medical, aviation and mining and probably other domains where whatever you make it going to have to function even when you can not reach it at all.
But it is great to point it out and to show how essential this kind of thinking is, and how it can help to focus on what is really important and what can be fixed.
What is interesting is to theorize about the relative impact of losing any of those three and how you managed to fix the second because you still had power and were able to communicate with the device. I think within those the order of relative importance would be communications, power and then attitude but I could well be mistaken.
At least it wasn't "learns", like "we had five learns from the project". Like, say, "ad spend". There's already a noun form of a verb, it's called a gerund: "ad spending".
As with genes, duplication creates opportunity for specialization. Regardless of what drives the duplication and early divergence.
AIchat "compare and contrast the subtle implications of phrase/X with phrase/Y" suggests using "ad spend" for a number (like "budget"), and "ad spending" for activity and trend ("act of spending").
"Learns" has implications of discovery, smaller size, iterative, informality, individual/team scale, messy, and more. For illustration, to my ear, "Don't be stupid" fits as a "lesson", but not as a "learn" or a "takeaway". Nor as a "lesson learned", with its implication of formality and reflection. "Software X is flaky" fits better "learn" than "lesson". And "unmonitored vendor excursions are biting us" more a "takeaway" (actionable; practical vs process).
Terrific writeup. Massive congrats to the whole team for all that creative thinking in flight and all that was achieved. (Add a note about updating FPGA's in space!) Looking forward to team Bedo unlocking VLEO for everyone.
There was a lot of this that was over my head (no pun intended) but I really enjoyed reading the whole thing. Sounds like a great job, and def a good read.
With image resolution this high, ground accuracy becomes an important factor as many people that prefer higher resolutions also want geospatially accurate images. Did you have any findings or results on this?
We actually didn't get to that part of the payload calibration campaign unfortunately, but all indications pointed towards getting geolocation between 5-10 meters on this first mission, driven primarily by star tracker quaternion error. Ephemeris and field angle map error was right in spec, so we were prepped to do an iterative line of sight pointing calibration but with the CMGs down, we didn't get to get there.
Future systems we've got a few updates though based on learnings, and we'll be shooting for closer to 3-5 meter geolocation error without ground control points (GCPs)
I work near space but not in space. I'm not sure I understand your process here. I see 2 possibilities: 1. You bought something the manufacturer spec lied about. While true we often validate specs, our terrestrial stuff is a lot cheaper so we can afford the spares. That said, if we buy something that doesn't meet the spec, you best believe we're taking the actions necessary. 2. This was built or designed inhouse, and the requirements didn't flow down correctly. That's also not great.
To be honest, postmortems (especially from startups) toe a fine line of scaring off investors, and this write-up seems a bit too glaze-y. I'm very happy for you that so much worked so effortlessly post launch, but that's more a success story than a postmortem. I'd like to see more of the root cause analysis for the issue, both technically and programmatically.
To be certain, if you're in the trenches of this anomaly investigation you'll get the full root cause and corrective action presentation, but that's not what this post is for.
You're correct on 1, we ended up hitting an edge case in their spec that they hadn't adequately tested to and the upper level management and engineering leadership were swift to accept the fault and implement fixes with us going forward.
From a SE perspective, as a "COTS" product, we had spec'd correctly to them, they accepted our requirements and then executed each unit's acceptance test plan (aka lower level than first unit quals or life tests where this should have been caught) on the ground without anything amiss. We ran through our nominal and off nominal cases at the higher level of assembly, but not for a duration that caught this on the ground. It wasn't until we were at extended operation on orbit the issues began.
Sadly like you state, space isn't like on the ground, you can't buy spares or replace things that fault, even for a true high volume COTS product that might slip through the acceptance testing.
> We ran through our nominal and off nominal cases at the higher level of assembly, but not for a duration that caught this on the ground. It wasn't until we were at extended operation on orbit the issues began.
So I think that's a great answer. It's all about risk mitigation and tolerance. Your test tested if the part work to a reasonable and hopefully calculated level. It's good that the suppliers' management accepted fault, too. It's a lot harder when they don't but honestly in the professional world I've found that to be much rarer than consumer.
To me, and I'm not an investor, and probably not your target audience, those 3 short paragraphs told me a lot more in a positive way than I expected. I don't think it would be out of place to put it in the post. Honestly as is I thought this was your guys' fault for myriad reasons. Now I'm flipped the other way. Of course it's still your problem even though it's not your fault. Or, maybe, you do claim some blame for the worst case analysis not shaking out that edge case. Either way I feel much less like you guys just went to the hardware store, bought some random lube, packed the bearing, and shipped it thinking you'll figure it out on the next launch (which is sadly the fast and loose reputation new space is starting to get).
Presumably part of the ground tests consist of putting a prototype in a thermal chamber and running it a bit above max temp for a week and a bit below min temp for a week to check functionality+margins...
Wonder why this didn't pick it up? Perhaps the test wasnt long enough?
Small (ok, maybe not so small) fortunes are spent on accelerated testing in the space domain. The test and validation gear is immense, using it is extremely expensive (especially when you get to full scale) and access is limited. They may well have been limited to that week for many reasons and conditions being what they are there is a fair chance that the space environment caused an issue that would have shown up if they had had a much longer test but that if there is no trend or if instrumentation is not sufficient simply doesn't show up at all.
For instance, if you're not measuring current into each and every actuator, or if the resolution at which you are sampling is too low then there may well be a developing issue that you are completely blind to.
Congrats on having a successful mission, it seems quite successful for a first try, and you clearly have some talented people on your team. But I’m going to give you my unsolicited opinion on the writing style.
The writing style sounds more like a tech bro describing some weekend conquest, and is wholly unappealing to most of the space industry (or at least the ones with decision making authority). Your CMGs were “locked in,” several times you “nailed it,” and so on.
You might have a business strategy that I’m not aware of but I’d expect that most of your market is controlled by aging men in suits, and they don’t talk like this. Most startups and tech bros aren’t spending money on space. It’s big established corporations that can fund this kind of stuff. Write like them. You can talk like a tech bro and get seed funding, but if you want to get to a sustainable business you have to talk corporate.
I would hate for your company to get passed over for lucrative opportunities because your public image seems immature. I looked at your website and you have a bunch of ex-government people on your senior advisory board. Get their opinion on your writing. It sounds silly, but you significantly lower your probability of winning contracts if people see you as a team of “bros.” People don’t want to spend millions on guys who are “locked in.” People want to spend millions on people who do professional engineering and risk reduction and clearly communicate how professional and competent they are.
I ranted way too long about your writing style. It’s pretty cool that you were able to design your own bus and most of it worked.
I agree. If it wasn't written with an AI I would be shocked. Its got the classic "VLEO isn’t just a better orbit for imaging — it’s the next productive orbital layer." mdash and all. That style sends strong signals of lazyness, scammyness and unprofessionalism.
Why would I take a company seriously when they can't be bothered to write their own press statements and blog posts?
> most of your market is controlled by aging men in suits
that um... doesn't sound like the space market. The engineers involved won't care about whether it's big corporate speak or GPT-ish gushing about "nailing it", they'll just want to understand if its a suitable bus for their mission concept and how well it works. It's actually more candid than your average blog in that respect.
It’s not the engineers who have the money or the control of company decisions. The engineers might make recommendations to their PMs and director of whatever program, but when considered against the myriad other bus options in LEO, the decision makers might be scared off by this writing and choose a less risky bus and accept the worse imaging performance or performance degradation in whatever other payload they integrate.
> The writing style sounds more like a tech bro describing some weekend conquest, and is wholly unappealing to most of the space industry (or at least the ones with decision making authority). Your CMGs were “locked in,” several times you “nailed it,” and so on.
This is on the company blog. It ends with a call to action to either subscribe to their mailing list or explore their careers page.
It has the right tone for the goal. Tone policing isn’t helpful. The authors are even here answering questions which is very nice of them.
Yes it is, these are YC alums. A significant portion of that program is learning how to pitch to investors. Are you saying that YC wastes this effort? The goal of this post might be to recruit engineers to hire, but it is public facing and part of their image that anyone might read, including future customers. It doesn’t help to recruit good engineers but be unable to sell because the industry expects strict professionalism.
it looks like Toppher Haddad has two writing styles, the tech bro blog , and the style he used above to discuss the technical aspects of there optical technolgy, and that is high geek, high uber geek of the type generaly associated with an inability to comprehend others lack of incomprehension, so actualy an unusual integration of skills and aproaches
1. The proximity improves performance:
- Range^2: imaging, other sensing, closing link budgets (data you're sending or signals you're sensing)
- Range^3: SAR
- Range^4: Active sensing (lidar, radar, etc)
- Speed: Comms latency, time to intercept
- If you can build systems fast/cheap, this physics unlock creates a new paradigm for system architectures (compared to traditional cost/schedule/performance tradeoffs)
2. Diversification is important for resilience/deterrence
- Drag self cleans debris
- It's below the belts where radiation gets trapped after a nuclear detonation. Makes for not only a survivable orbit, but one with assured reconstitution
I think the main purpose, atleast in this case, is to enable very high resolution satellite imagery (whether or not that being a good thing in and of itself is another matter).
> Next up was maneuvering from our LEO drop-off altitude down to VLEO, where it would be safe to eject the telescope contamination cover
Why would it be unsafe to do this earlier?
> We had been tracking intermittent memory issues in our TT&C radio throughout the mission, working around them as they appeared. Our best theory is that one of these issues escalated in a way that corrupted onboard memory and is preventing reboots. We've tried several recovery approaches. So far, none have worked, and the likelihood of recovery looks low at this point.
Seems to be a pretty big problem as well, I wonder what their ideas are to diagnose the root cause here.
It all sounds a bit overoptimistic, but that may just be my interpretation.
Space safety for sure on the cover, although I'm not sure we'll have that cover for future launches because it was less than easy to coordinate with the FCC on where to eject it.
The radio came from a supplier who has been investigating the issue. We had concerns with their NAND and ECC implementation, and we weren’t able to fully root-cause it with them. Going forward, we’ll be building our own radios, which will make it easier to test, iterate, and resolve issues like this internally, or at least be able to trace possible latch ups or destructive failures and implement the right levels of redundancy.
Ok, good luck with that, that's a tough environment for such sensitive stuff. Unfortunately your application seems to be so advanced that you can't get away from having high speed and super integrated stuff on board there otherwise you'd be more intrinsically safe against such issues. The fact that it worked well initially and then degraded until it failed is a strong indicator of the kind of process that caused the failure but unfortunately that still leaves a whole slew of options on the table.
Much good luck with this, those are hard problems to solve but you guys got so much right on the first try that you're probably ahead on your schedule now so you may have the time and the budget to get this right. I've visited the ESA open day a while ago and have seen the guts of what goes into satellite manufacture (not the most recent stuff, just what they had on display) and what struck me is that the degree of rigor that goes into designing stuff that is in the most literal sense out of reach for fixes or diagnostics requires simulating the environment the device will operate in to the best of their ability. This results in autoclaves that you can walk around in and various radiation sources to be able to test how the devices respond to space conditions.
Your manufacturer/supplier will probably come away from this effort with as much knowledge and improvement items as you do. Given the short time to failure I'm not so sure redundancy alone would have been a sufficient fix, but that's obviously bystander perspective, you know far more than I do. But it certainly is an amazing and interesting project.
Our business model is focused on providing VLEO systems (satellites, buses, and full mission services). But future customers may be provide image services :)
Why yet another proprietary space electronics communication bus? Do we still not have a standard, useful, open space electronics communication bus? Can someone explain to me why not?
Not that kind of bus. A “satellite bus” is more of a standardized platform onto which mission-specific payloads are integrated. Saves having to design an entire spacecraft from scratch and gives you a known-good set of functionality.
So, a set of standard software RPCs (remote procedure calls) and APIs (application programming interfaces) and not another electrical signalling standard. Got it.
Thanks for the correction.
So, I guess the next question is what are you folks actually using at the electrical signalling level to talk? (If you are not allowed to say, I understand.)
Founder/CEO of Albedo here. We published a detailed write-up of our first VLEO satellite mission (Clarity-1) — including imagery, what worked, what broke, and learnings we're taking forward. Happy to answer questions.
https://albedo.com/post/clarity-1-what-worked-and-where-we-g...
The diffraction limit (under 1.22 h* lambda/d) of a 1m optic at 250km in visible light is about 17cm. How can you achieve 10cm resolution?
Clarity is designed for a GSD (ground sample distance) of 10 cm. Generally the industry uses resolution<>GSD interchangeably. Agree it's not the true definition of resolution. But I'd argue the diffraction limit is an incomplete metric as well, like how spatial sampling is balanced with other MTF contributors (e.g. jitter/smear). For complete metrics, we like 1) NIIRS or 2) % contrast for a given object size on the ground (i.e. system MTF translated to ground units, not image-space units).
The main performance goal for us was NIIRS 7, and we decomposed GSD/MTF/SNR contributors optimized for affordability when we architected the system
How do you manage along-track smear? At those altitudes you're pushing close to 8km/s. Traditionally you'd either need to keep the satellite rotating through the collect or somehow keep the integration time in the single digit microseconds.
TDI (time delay integration)
A little more detail that we didn't get into in the post. The 3-CMG control mode we first uploaded was v1. We had plans to improve the agility with later versions. In v1, we didn't have quite enough rate to match earth's, even with max TDI. We called it Banana Scanning. Kind of like slipping over the earth.
Net - the CMG imagery we captured had a few pixels of along-track smear in it. Which would have been removed in post-processing if we had made it through focus cal.
Thanks for the response!
I assume you were setup with a linescan then instead of butcher block FPA?
How did you manage meaningful attitude control with only torque rods? They would need to big (read: heavy) to be useful — was this just stabilising in inertial frame or active pointing? Mag dipoles in chassis and components tend to lock tumbling satellites into the Earth’s magnetic field. Did you see this? Or did you see atmospheric drag dominate at this altitude?
I'm AyJay, Topher's co-founder and Albedo's CTO. We'll actually be publishing a paper here in a few weeks detailing how we got 3-axis torque rod control so you can get the real nitty gritty details then.
We got here after stacking quite a few capabilities we'd developed on top of one another and realizing we were beginning to see behavior we should be able to wrap up into a viable control strategy.
Traditional approaches to torque rod control rely on convergence over long time horizons spanning many orbits, but this artificially restricts the control objectives that can be accomplished. Our momentum control method reduced convergence time by incorporating both current and future magnetic field estimates into a special built Lyapunov-based control law we'd be perfecting for VLEO. By the time the issue popped up, we already had a lot of the ingredients needed and were able to get our algorithms to control within an orbit or two of initialization and then were able to stay coarsely stable for most inertial ECI attitudes albeit with wide pointing error bars as stated in the article. For what we needed though, it was perfect.
I'd love to read this paper! This was on my mind when I was GNC lead for an undergraduate project at Michigan Tech (Oculus-ASR - Nanosat-6 winner). We had a combined controller for reaction wheels and magtorque rods.
I'd love to read about that as well! (your project, not just the OPs!)
I’ve just read the wiki page on Magnetorquer but I couldn’t find what I was looking for: ballpark numbers.
What kind of current are you driving those coils with (amps or dozens of amps?). What order of magnitude is the resulting force (a few newtons?)
I’ll gladly read the paper but knowing myself I won’t remember why exactly when a few weeks passed.
>The drag coefficient was the headline: 12% better than our design target.
Is the drag much better than a regular cubesat? It doesn't look tremendously aerodynamic. From the description I was kind of expecting a design that minimized frontal area.
>Additional surface treatments will improve drag coefficient further.
Is surface drag that much of a contributor at orbital velocity?
Ultimately it's about the ballistic coefficient. You want high mass, low cross-sectional area, and low drag coefficient (Cd). With propulsion for station-keeping, it's challenging to capture the VLEO benefits with a regular cubesat. That said, there are VLEO architectures different than Clarity that make sense for other mission areas.
Yes it's a big contributor. The atmosphere in VLEO behaves as free molecular flow instead of a continuous fluid.
Cue the ultimate low orbit satellite
> It is undesirable to have a definition that will change with improving technology, so one might argue that the correct way to define space is to pick the lowest altitude at which any satellite can remain in orbit, and thus the lowest ballistic coefficent possible should be adopted - a ten-meter-diameter solid sphere of pure osmium, perhaps, which would have B of 8×10^−6 m^2/kg and an effective Karman line of z(-4) at the tropopause
from https://arxiv.org/abs/1807.07894
Assuming I did the math right such a satellite would only run $265 million USD for the materials (launch costs for an object of ~9k kg left as an exercise for the reader). That's far more affordable than I had expected. Amusing thought.
That would make a hell of a bang when it eventually deorbits.
Why no fold mirror then?
Can you tell us some war stories about the software your group wrote for the satellite?
Stacks? Testing? Firmware Updates? Programming languages?
Thank you!
First - they never want to use someone else software framework again (an early SW architect decided that would accelerate things but we ended up re-writing almost all of it) and it was all C++ on the satellite. We ran linux with preempt_rt.
We wrote everything from low level drivers to the top level application and the corresponding ground software for commanding and planning as well. Going forward, we're writing everything top to bottom, just to simplify and have total ownership since we're basically there already.
For testing we hit it at multiple levels: unit test, hardware in the loop, a custom "flight software in test" we called "FIT" which executed a few different simulated mission scenarios, and we tried to hit as many fault cases as we could too. It was pretty stressful for the team tbh but they were super stoked to see how well it worked on orbit.
A big one for us in a super high resolution mission like this is the timing determinism (low latency/low jitter) of the guidance, navigation, and control (GNC) thread. Basically it needs execute on time, every cycle, for us to achieve the mission. Getting enough timing instrumentation was tough with the framework we had selected and we eventually got there, but making sure the "hot loop" didn't miss deadlines was more a function of working with that framework than any limitation of linux operating well enough in a RTOS fashion for us.
Moving fast to make launch, we had missed a harness checkout step that would’ve caught a missing comms connection into an FPGA, and it was masked because our redundant comms channel made everything look nominal.
On orbit, we fixed it by pushing an FPGA update and adding software-level switching between the channels to prove the update applied and isolate the hardware path — which worked. Broader lesson, it is possible to design a sw stack capable of making updates to traditionally burned-in components.
> it was masked because our redundant comms channel made everything look nominal.
Hah, this has bitten me often enough I check for it in test suites now - ok, you’ve proven the system works and the backup works, have you proven the primary works? Another in the long list of ways you don’t expect a system to bite you until it does…
That's why starfleet always has a secondary backup /s
>On orbit, we fixed it by pushing an FPGA update
How does that work? Do you actually VPN into an orbiting satellite? If so, how do you get a public IP for a satellite? Do you just ask an average ISP?
Why would you need a public IP? Why would you want to expose something like this to the public internet?
I have no idea what they use but there are various commonly used radio communication methods for talking to satellites.
I'm just saying that if they use a traditional IP VPN over the Internet, traffic is of course encrypted, but the two endpoints terminating the VPN must have a public IP.
Of course, this is not necessary if they simply use UHF radio signals encoding bits with pulse modulation, or whatever.
Indeed, that's indeed what I was curious about (Internet vs other channels), since at least part of their tech stack in orbit is relatively straightforward (e.g. linux with preempt_rt).
Why would you VPN here? If you did, why would you do so over the public internet? You can route IP packets over internal links, including via radio (that is the entire point of WiFi after all).
Although it occurs to me that "does your network stack employ either ethernet or IP (and what were the considerations)" might be an interesting question.
Let me dream about the one guy working remote for a satellite company and just jumping into a direct VPN with a satellite, won't you? :)
All kidding aside, there are some protocols, like FTP or RTSP, which don't play well with NAT because they include IP addresses in the payload itself. Some solutions exist (so called ALG) but they are often fragile. If the satellite was using some of these protocols to talk to something on a public cloud platform (say, send images via FTP to an EC2 VM), satellites could have a public address to avoid NAT issues, and that point you could also use it as a management address (although maybe only as a backup path).
It's a bit far fetched, but when it comes to satellites, you could say "sky is the limit" :)
EDIT
I admit public IP addresses are a bit unlikely (but... who knows!). However, this picture on their website (https://cdn.prod.website-files.com/64529e978a785fb5da715f99/...) clearly shows a Grafana dashboard.
Ignoring for a moment the fact that Grafana could be self-hosted or in SaaS, Grafana is heavily used to collect logs and metrics from standard servers.
Of course, maybe they built their own integrations to convert raw logs and metrics sent via plain pulse modulation to plain syslog and Prometheus metrics, but maybe it's just that they're using (probably private) IP addresses on board and they are simply streaming logs and metrics to the ground using standard TCP/UDP protocols.
> it is possible to design a sw stack capable of making updates to traditionally burned-in components.
This is interesting - is the software stack essentially acting as "light" translation layer or abstraction layer on components?
Loved your post. Thank you for answering what you have already.
How did you test attitude control + the software stack on the ground? Did you use a simulator?
LeoLabs seems to have been really helpful here. What other startups formed your 'space stack' so to speak?
s/learnings/lessons/g
From my perspective, the number one reason we had a well functioning satellite out of the gate is my philosophy of testing "safe mode first". What that means is in a graduated fashion, test that the hardware and software together can always get you into a safe mode - which is usually power positive, attitude stable, and communicative. So our software integration flows hit this mission thread over and over and over with each update. If we shipped a new software feature, make sure you got to safe mode. If we found a bug that prevented it, it's the first thing to triage. We build out our pipelines to simulate this as much as we could and then ran it again on the development hardware and eventually would load a release onto flight once we were confident this was always solid. If you're going to develop for space, start here.
That principle goes far further than developing for space, but for space the pay-off is the largest. It also applies to maritime, medical, aviation and mining and probably other domains where whatever you make it going to have to function even when you can not reach it at all.
But it is great to point it out and to show how essential this kind of thinking is, and how it can help to focus on what is really important and what can be fixed.
What is interesting is to theorize about the relative impact of losing any of those three and how you managed to fix the second because you still had power and were able to communicate with the device. I think within those the order of relative importance would be communications, power and then attitude but I could well be mistaken.
At least it wasn't "learns", like "we had five learns from the project". Like, say, "ad spend". There's already a noun form of a verb, it's called a gerund: "ad spending".
As with genes, duplication creates opportunity for specialization. Regardless of what drives the duplication and early divergence.
AIchat "compare and contrast the subtle implications of phrase/X with phrase/Y" suggests using "ad spend" for a number (like "budget"), and "ad spending" for activity and trend ("act of spending").
"Learns" has implications of discovery, smaller size, iterative, informality, individual/team scale, messy, and more. For illustration, to my ear, "Don't be stupid" fits as a "lesson", but not as a "learn" or a "takeaway". Nor as a "lesson learned", with its implication of formality and reflection. "Software X is flaky" fits better "learn" than "lesson". And "unmonitored vendor excursions are biting us" more a "takeaway" (actionable; practical vs process).
Terrific writeup. Massive congrats to the whole team for all that creative thinking in flight and all that was achieved. (Add a note about updating FPGA's in space!) Looking forward to team Bedo unlocking VLEO for everyone.
There was a lot of this that was over my head (no pun intended) but I really enjoyed reading the whole thing. Sounds like a great job, and def a good read.
With image resolution this high, ground accuracy becomes an important factor as many people that prefer higher resolutions also want geospatially accurate images. Did you have any findings or results on this?
We actually didn't get to that part of the payload calibration campaign unfortunately, but all indications pointed towards getting geolocation between 5-10 meters on this first mission, driven primarily by star tracker quaternion error. Ephemeris and field angle map error was right in spec, so we were prepped to do an iterative line of sight pointing calibration but with the CMGs down, we didn't get to get there.
Future systems we've got a few updates though based on learnings, and we'll be shooting for closer to 3-5 meter geolocation error without ground control points (GCPs)
As an aside, your it appears that your albedo-stuff.com domain has expired.
So the root cause was the lubricant in the gyros couldn’t stand up to operating temperatures.
I’d be interested to read a postmortem of the systems engineering approach there.
The lesson there - dig multiple levels deep in supply chain
Alas.. the speed & resources of a startup. But we're learning.
I work near space but not in space. I'm not sure I understand your process here. I see 2 possibilities: 1. You bought something the manufacturer spec lied about. While true we often validate specs, our terrestrial stuff is a lot cheaper so we can afford the spares. That said, if we buy something that doesn't meet the spec, you best believe we're taking the actions necessary. 2. This was built or designed inhouse, and the requirements didn't flow down correctly. That's also not great.
To be honest, postmortems (especially from startups) toe a fine line of scaring off investors, and this write-up seems a bit too glaze-y. I'm very happy for you that so much worked so effortlessly post launch, but that's more a success story than a postmortem. I'd like to see more of the root cause analysis for the issue, both technically and programmatically.
To be certain, if you're in the trenches of this anomaly investigation you'll get the full root cause and corrective action presentation, but that's not what this post is for.
You're correct on 1, we ended up hitting an edge case in their spec that they hadn't adequately tested to and the upper level management and engineering leadership were swift to accept the fault and implement fixes with us going forward.
From a SE perspective, as a "COTS" product, we had spec'd correctly to them, they accepted our requirements and then executed each unit's acceptance test plan (aka lower level than first unit quals or life tests where this should have been caught) on the ground without anything amiss. We ran through our nominal and off nominal cases at the higher level of assembly, but not for a duration that caught this on the ground. It wasn't until we were at extended operation on orbit the issues began.
Sadly like you state, space isn't like on the ground, you can't buy spares or replace things that fault, even for a true high volume COTS product that might slip through the acceptance testing.
> We ran through our nominal and off nominal cases at the higher level of assembly, but not for a duration that caught this on the ground. It wasn't until we were at extended operation on orbit the issues began.
So I think that's a great answer. It's all about risk mitigation and tolerance. Your test tested if the part work to a reasonable and hopefully calculated level. It's good that the suppliers' management accepted fault, too. It's a lot harder when they don't but honestly in the professional world I've found that to be much rarer than consumer.
To me, and I'm not an investor, and probably not your target audience, those 3 short paragraphs told me a lot more in a positive way than I expected. I don't think it would be out of place to put it in the post. Honestly as is I thought this was your guys' fault for myriad reasons. Now I'm flipped the other way. Of course it's still your problem even though it's not your fault. Or, maybe, you do claim some blame for the worst case analysis not shaking out that edge case. Either way I feel much less like you guys just went to the hardware store, bought some random lube, packed the bearing, and shipped it thinking you'll figure it out on the next launch (which is sadly the fast and loose reputation new space is starting to get).
Presumably part of the ground tests consist of putting a prototype in a thermal chamber and running it a bit above max temp for a week and a bit below min temp for a week to check functionality+margins...
Wonder why this didn't pick it up? Perhaps the test wasnt long enough?
> Perhaps the test wasnt long enough?
Small (ok, maybe not so small) fortunes are spent on accelerated testing in the space domain. The test and validation gear is immense, using it is extremely expensive (especially when you get to full scale) and access is limited. They may well have been limited to that week for many reasons and conditions being what they are there is a fair chance that the space environment caused an issue that would have shown up if they had had a much longer test but that if there is no trend or if instrumentation is not sufficient simply doesn't show up at all.
For instance, if you're not measuring current into each and every actuator, or if the resolution at which you are sampling is too low then there may well be a developing issue that you are completely blind to.
Congrats on having a successful mission, it seems quite successful for a first try, and you clearly have some talented people on your team. But I’m going to give you my unsolicited opinion on the writing style.
The writing style sounds more like a tech bro describing some weekend conquest, and is wholly unappealing to most of the space industry (or at least the ones with decision making authority). Your CMGs were “locked in,” several times you “nailed it,” and so on.
You might have a business strategy that I’m not aware of but I’d expect that most of your market is controlled by aging men in suits, and they don’t talk like this. Most startups and tech bros aren’t spending money on space. It’s big established corporations that can fund this kind of stuff. Write like them. You can talk like a tech bro and get seed funding, but if you want to get to a sustainable business you have to talk corporate.
I would hate for your company to get passed over for lucrative opportunities because your public image seems immature. I looked at your website and you have a bunch of ex-government people on your senior advisory board. Get their opinion on your writing. It sounds silly, but you significantly lower your probability of winning contracts if people see you as a team of “bros.” People don’t want to spend millions on guys who are “locked in.” People want to spend millions on people who do professional engineering and risk reduction and clearly communicate how professional and competent they are.
I ranted way too long about your writing style. It’s pretty cool that you were able to design your own bus and most of it worked.
I agree. If it wasn't written with an AI I would be shocked. Its got the classic "VLEO isn’t just a better orbit for imaging — it’s the next productive orbital layer." mdash and all. That style sends strong signals of lazyness, scammyness and unprofessionalism.
Why would I take a company seriously when they can't be bothered to write their own press statements and blog posts?
Yeh, it's an interesting project but the AI writing style was exhausting to read, and didn't fill me with confidence in the company.
The whole thing gave off a feeling that the author was desperate to prove something.
> most of your market is controlled by aging men in suits
that um... doesn't sound like the space market. The engineers involved won't care about whether it's big corporate speak or GPT-ish gushing about "nailing it", they'll just want to understand if its a suitable bus for their mission concept and how well it works. It's actually more candid than your average blog in that respect.
It’s not the engineers who have the money or the control of company decisions. The engineers might make recommendations to their PMs and director of whatever program, but when considered against the myriad other bus options in LEO, the decision makers might be scared off by this writing and choose a less risky bus and accept the worse imaging performance or performance degradation in whatever other payload they integrate.
> The writing style sounds more like a tech bro describing some weekend conquest, and is wholly unappealing to most of the space industry (or at least the ones with decision making authority). Your CMGs were “locked in,” several times you “nailed it,” and so on.
This is on the company blog. It ends with a call to action to either subscribe to their mailing list or explore their careers page.
It has the right tone for the goal. Tone policing isn’t helpful. The authors are even here answering questions which is very nice of them.
> Tone policing isn’t helpful.
Yes it is, these are YC alums. A significant portion of that program is learning how to pitch to investors. Are you saying that YC wastes this effort? The goal of this post might be to recruit engineers to hire, but it is public facing and part of their image that anyone might read, including future customers. It doesn’t help to recruit good engineers but be unable to sell because the industry expects strict professionalism.
it looks like Toppher Haddad has two writing styles, the tech bro blog , and the style he used above to discuss the technical aspects of there optical technolgy, and that is high geek, high uber geek of the type generaly associated with an inability to comprehend others lack of incomprehension, so actualy an unusual integration of skills and aproaches
What is the purpose of VLEO? Being at lower altitude to enable kinetic weapons for projects like Golden Dome?
1. The proximity improves performance: - Range^2: imaging, other sensing, closing link budgets (data you're sending or signals you're sensing) - Range^3: SAR - Range^4: Active sensing (lidar, radar, etc) - Speed: Comms latency, time to intercept - If you can build systems fast/cheap, this physics unlock creates a new paradigm for system architectures (compared to traditional cost/schedule/performance tradeoffs)
2. Diversification is important for resilience/deterrence - Drag self cleans debris - It's below the belts where radiation gets trapped after a nuclear detonation. Makes for not only a survivable orbit, but one with assured reconstitution
I think the main purpose, atleast in this case, is to enable very high resolution satellite imagery (whether or not that being a good thing in and of itself is another matter).
What an impressive project.
> Next up was maneuvering from our LEO drop-off altitude down to VLEO, where it would be safe to eject the telescope contamination cover
Why would it be unsafe to do this earlier?
> We had been tracking intermittent memory issues in our TT&C radio throughout the mission, working around them as they appeared. Our best theory is that one of these issues escalated in a way that corrupted onboard memory and is preventing reboots. We've tried several recovery approaches. So far, none have worked, and the likelihood of recovery looks low at this point.
Seems to be a pretty big problem as well, I wonder what their ideas are to diagnose the root cause here.
It all sounds a bit overoptimistic, but that may just be my interpretation.
Space safety for sure on the cover, although I'm not sure we'll have that cover for future launches because it was less than easy to coordinate with the FCC on where to eject it.
The radio came from a supplier who has been investigating the issue. We had concerns with their NAND and ECC implementation, and we weren’t able to fully root-cause it with them. Going forward, we’ll be building our own radios, which will make it easier to test, iterate, and resolve issues like this internally, or at least be able to trace possible latch ups or destructive failures and implement the right levels of redundancy.
Ok, good luck with that, that's a tough environment for such sensitive stuff. Unfortunately your application seems to be so advanced that you can't get away from having high speed and super integrated stuff on board there otherwise you'd be more intrinsically safe against such issues. The fact that it worked well initially and then degraded until it failed is a strong indicator of the kind of process that caused the failure but unfortunately that still leaves a whole slew of options on the table.
Much good luck with this, those are hard problems to solve but you guys got so much right on the first try that you're probably ahead on your schedule now so you may have the time and the budget to get this right. I've visited the ESA open day a while ago and have seen the guts of what goes into satellite manufacture (not the most recent stuff, just what they had on display) and what struck me is that the degree of rigor that goes into designing stuff that is in the most literal sense out of reach for fixes or diagnostics requires simulating the environment the device will operate in to the best of their ability. This results in autoclaves that you can walk around in and various radiation sources to be able to test how the devices respond to space conditions.
Your manufacturer/supplier will probably come away from this effort with as much knowledge and improvement items as you do. Given the short time to failure I'm not so sure redundancy alone would have been a sufficient fix, but that's obviously bystander perspective, you know far more than I do. But it certainly is an amazing and interesting project.
I think they want to get low enough that the jettisoned cover will not stay in orbit long or run into anything.
Ah, safe for others, not for the telescope. Thanks, I did not consider that option.
Yes, exactly. Space safety :)
Do you plan to offer an image service?
Great write up!
Our business model is focused on providing VLEO systems (satellites, buses, and full mission services). But future customers may be provide image services :)
Thank you!
Looking forward to more launches! Thank you
Why yet another proprietary space electronics communication bus? Do we still not have a standard, useful, open space electronics communication bus? Can someone explain to me why not?
Not that kind of bus. A “satellite bus” is more of a standardized platform onto which mission-specific payloads are integrated. Saves having to design an entire spacecraft from scratch and gives you a known-good set of functionality.
So, a set of standard software RPCs (remote procedure calls) and APIs (application programming interfaces) and not another electrical signalling standard. Got it.
Thanks for the correction.
So, I guess the next question is what are you folks actually using at the electrical signalling level to talk? (If you are not allowed to say, I understand.)
No, not RPCs/APIs either. A satellite bus is a physical object[1], which defines the mechanical standards for mounting payload modules.
[1]: https://en.wikipedia.org/wiki/Satellite_bus