It must have been difficult and frustrating to work as part of the Windows team back in those days.
You see all the wacky software that doesn't follow the rules properly, does whatever it wants, breaks things. And you have to figure out how Windows can accommodate all that software, keep it from breaking, and also prevent it from messing up a computer or undo the damage.
They did not have the option of saying "this app developer wrote shitty software, sucks to be them, not my problem."
I wonder how much of this problem was caused by lack of adequate documentation describing how an installer should behave, and how much was developers not reading that documentation and being content when it works on their machine.
> I wonder how much of this problem was caused by lack of adequate documentation describing how an installer should behave, and how much was developers not reading that documentation and being content when it works on their machine.
There is a third option: the developers knew the rules and chose to ignore them for some reason. A modern example of this is the Zig language’s decision to reverse engineer and use undocumented APIs in Windows in preference of using documented APIs.
> I wonder how much of this problem was caused by lack of adequate documentation describing how an installer should behave, and how much was developers not reading that documentation and being content when it works on their machine.
It was mostly the latter. And when Windows broke, people would blame it on Microsoft, not on the software they installed. The same if the software broke. And you didn’t have online updates at the time that could retroactively add fixes. So Microsoft had to do everything they could to ensure broken software would still work, while also keeping Windows working, the best they could.
> So Microsoft had to do everything they could to ensure broken software would still work
I think they chose to do everything they could to keep it limping along. An alternative would've been a name-and-shame approach, like "This program crashed because the author made this mistake: [short description or code or whatever]", and leave them out to try until the devs stopped doing those dumb things. After a few years of pain, people would've gotten with the program, so to speak. Instead, they chose the path that put zero pressure on devs to write correctly-behaving software.
Raymond Chen already discussed this. Microsoft wants to sell Windows. Windows exists to run software. If Windows doesn't run software, Microsoft doesn't make that sale.
If your business runs on some obscure piece of software for which updates are neither cheap or easy, you're not going to buy Windows if it doesn't run that software.
Name and shame doesn't work because the developer isn't part of the transaction.
The thing is, Microsoft got its position of dominance exactly because they did that - and that was because by doing this, the users' programs kept working. Remember that users outnumber developers by far and the last thing Microsoft wanted was for people to not upgrade Windows because they broke their previously working programs.
This was even more important at a time when Microsoft had actual competition in the OS space and people weren't able to just go online and download updates.
> After a few years of pain, people would've gotten with the program, so to speak.
Not necessarily. This was still very much the time in which choosing to stick with an old version which worked (e.g. Windows 3.1) wasn't uncommon.
Just look at how many people jumped from XP to 7 due to the network effect of "Vista sucks" and then multiply that by the fact that, at the time of 3.1->95, people had far fewer computer security concerns, if any.
Why would I buy a new version of Windows, if none of my existing software will work on it, so I have to buy new versions of everything? Sounds expensive.
> And what does the customer do if the vendor has discontinued it? Or charges for an upgrade? Or has gone out of business?
Those can all be filed under Not My Problem (as in, Microsoft's problem,) and safely ignored. On the other hand, when Highly Influential So-And-So upgrades from 3.1 to 95 or whatever, and Very Population Application v4.9.6 starts falling over, Microsoft gets the black eye whether they deserve it or not. The whole equation changes.
One workaround Microsoft has done for use-after-free is detecting when an application is prone to this and using an allocator that doesn't actually free RAM immediately. It believe that lovely bit of fun is a function of "Heap Quarantine".
Yes, the real, can't say no world of system software is not what one might wish.
One of the craziest Raymond Chen stories is one where a Windows API call would return a pointer to a data structure the OS had allocated for the operation. The programmers at Microsoft made the data structure bigger than they needed, for future expansion. But some third party devs noticed the extra space, and started to use it to store data for their program. Then when Windows tried to start using the extra space, those applications would crash.
Reasonable people can disagree on a lot of things in programming. But I still do not understand how one can consider writing to memory the OS owns to be ok. It's sheer professional malpractice to do that kind of thing. With stuff like that, I don't think that any amount of documentation would have helped. The issue was that those programmers simply did not care about anything except getting their own program working, and did whatever the most expedient method was to get there.
> But I still do not understand how one can consider writing to memory the OS owns to be ok.
Go to Vogons and look at all of the memory tricks people will use to get various games running on MS-DOS. This kind of juggling exactly which drivers to load, etc. is why Microsoft added the boot menu in MS-DOS 6.0 to CONFIG.SYS.
I'm not necessarily saying that this was the case here, but it smells like that to me.
TBH i think a more likely explanation is that they needed to somehow identify separate instances of that data structure and they thought to store some ID or something in it so that when they encountered it next they'd be able to do that without keeping copies of all the data in it and then comparing their data with the system's.
Or you desperately need to tag some system object and the system provides no legitimate means to do so. That can be invaluable when troubleshooting things, or even just understanding how things work when the system fails to document behavior or unreasonably conceals things.
I've been there and done it, and I offer no apologies. The platform preferred and the requirements demanded by The Powers That Be were not my fault.
>I still do not understand how one can consider writing to memory the OS owns to be ok.
Things were different back then. People did a lot of hacky stuff to fit their programs into memory, because you were genuinely constrained by hardware limitations.
Not to mention, the idea of the OS owning the machine was not as well developed as it is today. Windows 3.11 was just another program, it didn't have special permissions like modern OSes, and you would routinely bypass it to talk to the hardware directly.
> Things were different back then. People did a lot of hacky stuff to fit their programs into memory, because you were genuinely constrained by hardware limitations.
Are you going to tell them what "32-bit Clean" meant for Mac developers, or will we let them find out that particular horror movie for themselves?
>Windows 95 worked around this by keeping a backup copy of commonly-overwritten files in a hidden C:\Windows\SYSBCKUP directory. Whenever an installer finished, Windows went and checked whether any of these commonly-overwritten files had indeed been overwritten.
This is truly unhinged. I wonder if running an installer under wine in win95 mode will do this.
> If . . . the replacement has a higher version number than the one in the SYSBCKUP directory, then the replacement was copied into the SYSBCKUP directory for safekeeping.
This as well. I know there are a million ways for a malicious installer to brick Win95, but a particularly funny one is hijacking the OS to perpetually rewrite its own system components back to compromised version number ∞ whenever another installer tries to clean things up.
This is bog-standard boring stuff (when presented with a similar problem, Linux invented containers lol) - read some of his other posts to realize the extent Microsoft went to maintain backwards compatibility - some are insane, some no doubt led to security issues, but you have to respect the drive.
It’s not bog-standard. Containers are not equivalent to doing what is described in the article.
Containers are in fact redirecting writes so an installer script could not replace system libraries.
The equivalent would be a Linux distro having the assumption that installer scripts will overwrite /usr/lib/libopenssl.so.1 with its own version and just keeping a backup somewhere and copying it back after the script executes.
No OS that I know of does that because it’s unhinged and well on Linux it would probably break the system due to ABI compatibility.
If they had taken essentially the same approach as wine and functionally created a WINEPREFIX per application then it would not be unhinged.
edit: also to be clear, I respect their commitment to backwards compatibility which is what leads to these unhinged decisions. I thoroughly enjoy Raymond Chen’s dev blog because of how unhinged early windows was.
Windows 95 was not Windows NT and it still used the FAT32 file system, where it was not really possible to enforce access rights.
As TFA says:
You even had installers that took even more extreme measures and said, “Okay, fine, I can’t overwrite the file, so I’m going to reboot the system and then overwrite the file from a batch file, see if you can stop me.”
Well and the earliest versions of Windows 95 used FAT16 (specifically VFAT for support for LFNs or long file names). So enjoy those ridiculous cluster sizes if your hard disk even approached a gig or so.
If an installer expects to be able to overwrite a file and fails to do so, it might crash, leaving the user with a borked installation.
Of course you can blame the installer, but resolution of the problem might take a long time, or might never happen, depending on the willingness of the vendor to fix it.
Granted, but at the same time it's also resolutely pragmatic.
Apparently there was already lots of software out there which expected to be able to write new versions of system components. As well as buggy software that incidentally expected to be able to write old versions, because its developers ignored Microsoft's published best practices (not to mention common sense) and and didn't bother to do a version comparison first.
The choice was to break the old software, or let it think it succeeded then clean up after the mess it made. I'd bet they considered other alternatives (e.g. sandbox each piece of software with its own set of system libraries, or intercept and override DLL calls thus ignoring written files altogether) but those introduce more complexity and redirection with arguably little benefit. (I do wonder if the cleanup still happens if something like an unexpected reboot or power loss happens at exactly the wrong time).
Could the OS have been architected in a more robust fashion from the get-go? Of course.
Could they have simply forbidden software from downgrading system components? Sure, but it'd break installers and degrade the user experience.
Since the OS historically tolerated the broken behavior, they were kind of stuck continuing to tolerate it. One thing I learned leading groups of people is if you make a rule but don't enforce it, then it isn't much of a rule (at least not one you can rely on).
I would argue the deeper mistake was not providing more suitable tooling for developers to ensure the presence of compatible versions of shared libraries. This requires a bit of game theory up front; you want to always make the incorrect path frictiony and the correct one seamless.
There was (and still is) VerInstallFile, however this was introduced in Windows 3.1 and it is possible installers wanted to also support Windows 3.0 (since there wasn't much of a time gap between the two many programs tried to support both) so they didn't use it.
It doesn't say for certain, but assuming the version of this they settled on (restoring components after the installation finished) is what they shipped in the original version of Windows 95, then no, I don't think this could have caused hangs in the installer itself (unless Win95 misjudged whether the installer had completed or not and started the restore process early?).
No, most likely bad algorithms for dealing with registry stuff. The kind of thing that worked well on tester machines with small registry sizes and exploded in the real world.
The sad lesson is to be both proactive and reactive if you want a clean environment. Trust, verify, and stick around to clean up someone else's mess after the fact.
Windows, especially old versions, were beautifully pragmatic. Think about the things that would need to exist on an open-source OS to match this functionality. You'd need to:
1. Convince people to distribute programs via installers.
2. Provide some way that installers can tell the OS that they're an installer (and not invent 5 different ways to do this!)
3. Convince the creators of installers to actually use that function.
4. Convince library creators to maintain backward compatibility (big ask).
5. Convince people to not fork said libraries, creating ambiguous upgrade paths.
6. If there are multiple distros, convince them all to use the same backup/restore format for libraries (and not treat their own favorite libraries as "special")
They absolutely created 10 different ways to install software; they didn't really advertised they were an installer; the only backward compatible thing there are the MS libraries; there was no common backup/restore format.
Instead, the Unix people made a mechanism for random programs to use their own libraries and not touch the system one. In fact, Windows had one too, but most applications still decided they need to break the system.
It must have been difficult and frustrating to work as part of the Windows team back in those days.
You see all the wacky software that doesn't follow the rules properly, does whatever it wants, breaks things. And you have to figure out how Windows can accommodate all that software, keep it from breaking, and also prevent it from messing up a computer or undo the damage.
They did not have the option of saying "this app developer wrote shitty software, sucks to be them, not my problem."
I wonder how much of this problem was caused by lack of adequate documentation describing how an installer should behave, and how much was developers not reading that documentation and being content when it works on their machine.
> I wonder how much of this problem was caused by lack of adequate documentation describing how an installer should behave, and how much was developers not reading that documentation and being content when it works on their machine.
There is a third option: the developers knew the rules and chose to ignore them for some reason. A modern example of this is the Zig language’s decision to reverse engineer and use undocumented APIs in Windows in preference of using documented APIs.
https://codeberg.org/ziglang/zig/issues/31131
> I wonder how much of this problem was caused by lack of adequate documentation describing how an installer should behave, and how much was developers not reading that documentation and being content when it works on their machine.
It was mostly the latter. And when Windows broke, people would blame it on Microsoft, not on the software they installed. The same if the software broke. And you didn’t have online updates at the time that could retroactively add fixes. So Microsoft had to do everything they could to ensure broken software would still work, while also keeping Windows working, the best they could.
> So Microsoft had to do everything they could to ensure broken software would still work
I think they chose to do everything they could to keep it limping along. An alternative would've been a name-and-shame approach, like "This program crashed because the author made this mistake: [short description or code or whatever]", and leave them out to try until the devs stopped doing those dumb things. After a few years of pain, people would've gotten with the program, so to speak. Instead, they chose the path that put zero pressure on devs to write correctly-behaving software.
Raymond Chen already discussed this. Microsoft wants to sell Windows. Windows exists to run software. If Windows doesn't run software, Microsoft doesn't make that sale.
If your business runs on some obscure piece of software for which updates are neither cheap or easy, you're not going to buy Windows if it doesn't run that software.
Name and shame doesn't work because the developer isn't part of the transaction.
The thing is, Microsoft got its position of dominance exactly because they did that - and that was because by doing this, the users' programs kept working. Remember that users outnumber developers by far and the last thing Microsoft wanted was for people to not upgrade Windows because they broke their previously working programs.
This was even more important at a time when Microsoft had actual competition in the OS space and people weren't able to just go online and download updates.
> After a few years of pain, people would've gotten with the program, so to speak.
Not necessarily. This was still very much the time in which choosing to stick with an old version which worked (e.g. Windows 3.1) wasn't uncommon.
Just look at how many people jumped from XP to 7 due to the network effect of "Vista sucks" and then multiply that by the fact that, at the time of 3.1->95, people had far fewer computer security concerns, if any.
Why would I buy a new version of Windows, if none of my existing software will work on it, so I have to buy new versions of everything? Sounds expensive.
Yes, but that doesn't solve the customer's problem
And what does the customer do if the vendor has discontinued it? Or charges for an upgrade? Or has gone out of business?
https://devblogs.microsoft.com/oldnewthing/20031224-00/?p=41...
I'm pretty sure another one was "what if you're wrong/have a false positive detection, and slander another company, one with lawyers?"
> And what does the customer do if the vendor has discontinued it? Or charges for an upgrade? Or has gone out of business?
Those can all be filed under Not My Problem (as in, Microsoft's problem,) and safely ignored. On the other hand, when Highly Influential So-And-So upgrades from 3.1 to 95 or whatever, and Very Population Application v4.9.6 starts falling over, Microsoft gets the black eye whether they deserve it or not. The whole equation changes.
One workaround Microsoft has done for use-after-free is detecting when an application is prone to this and using an allocator that doesn't actually free RAM immediately. It believe that lovely bit of fun is a function of "Heap Quarantine".
Yes, the real, can't say no world of system software is not what one might wish.
IIRC Sim City 2000 is one such piece of software.
It was SimCity Classic.
One of the craziest Raymond Chen stories is one where a Windows API call would return a pointer to a data structure the OS had allocated for the operation. The programmers at Microsoft made the data structure bigger than they needed, for future expansion. But some third party devs noticed the extra space, and started to use it to store data for their program. Then when Windows tried to start using the extra space, those applications would crash.
Reasonable people can disagree on a lot of things in programming. But I still do not understand how one can consider writing to memory the OS owns to be ok. It's sheer professional malpractice to do that kind of thing. With stuff like that, I don't think that any amount of documentation would have helped. The issue was that those programmers simply did not care about anything except getting their own program working, and did whatever the most expedient method was to get there.
> But I still do not understand how one can consider writing to memory the OS owns to be ok.
Go to Vogons and look at all of the memory tricks people will use to get various games running on MS-DOS. This kind of juggling exactly which drivers to load, etc. is why Microsoft added the boot menu in MS-DOS 6.0 to CONFIG.SYS.
I'm not necessarily saying that this was the case here, but it smells like that to me.
> But I still do not understand how one can consider writing to memory the OS owns to be ok.
Your manager tells you to reduce memory usage of the program "or else".
TBH i think a more likely explanation is that they needed to somehow identify separate instances of that data structure and they thought to store some ID or something in it so that when they encountered it next they'd be able to do that without keeping copies of all the data in it and then comparing their data with the system's.
^^ The voice of experience, here.
Or you desperately need to tag some system object and the system provides no legitimate means to do so. That can be invaluable when troubleshooting things, or even just understanding how things work when the system fails to document behavior or unreasonably conceals things.
I've been there and done it, and I offer no apologies. The platform preferred and the requirements demanded by The Powers That Be were not my fault.
>I still do not understand how one can consider writing to memory the OS owns to be ok.
Things were different back then. People did a lot of hacky stuff to fit their programs into memory, because you were genuinely constrained by hardware limitations.
Not to mention, the idea of the OS owning the machine was not as well developed as it is today. Windows 3.11 was just another program, it didn't have special permissions like modern OSes, and you would routinely bypass it to talk to the hardware directly.
> Things were different back then. People did a lot of hacky stuff to fit their programs into memory, because you were genuinely constrained by hardware limitations.
Are you going to tell them what "32-bit Clean" meant for Mac developers, or will we let them find out that particular horror movie for themselves?
>Windows 95 worked around this by keeping a backup copy of commonly-overwritten files in a hidden C:\Windows\SYSBCKUP directory. Whenever an installer finished, Windows went and checked whether any of these commonly-overwritten files had indeed been overwritten.
This is truly unhinged. I wonder if running an installer under wine in win95 mode will do this.
You'd have to track down some 16bit Win3.x software to install. Probably on floppy disks since CD-ROMs weren't common.
> If . . . the replacement has a higher version number than the one in the SYSBCKUP directory, then the replacement was copied into the SYSBCKUP directory for safekeeping.
This as well. I know there are a million ways for a malicious installer to brick Win95, but a particularly funny one is hijacking the OS to perpetually rewrite its own system components back to compromised version number ∞ whenever another installer tries to clean things up.
> This is truly unhinged.
This is bog-standard boring stuff (when presented with a similar problem, Linux invented containers lol) - read some of his other posts to realize the extent Microsoft went to maintain backwards compatibility - some are insane, some no doubt led to security issues, but you have to respect the drive.
It’s not bog-standard. Containers are not equivalent to doing what is described in the article.
Containers are in fact redirecting writes so an installer script could not replace system libraries.
The equivalent would be a Linux distro having the assumption that installer scripts will overwrite /usr/lib/libopenssl.so.1 with its own version and just keeping a backup somewhere and copying it back after the script executes.
No OS that I know of does that because it’s unhinged and well on Linux it would probably break the system due to ABI compatibility.
If they had taken essentially the same approach as wine and functionally created a WINEPREFIX per application then it would not be unhinged.
edit: also to be clear, I respect their commitment to backwards compatibility which is what leads to these unhinged decisions. I thoroughly enjoy Raymond Chen’s dev blog because of how unhinged early windows was.
You are right that it’s not equivalent, but the article explains why redirecting the writes wasn’t a viable option.
Man, after looking at the veritable pile of stinking matter that is claude code, compare it with the NT 4 source leak.
Windows may have suffered its share of bad architectural decisions, but unhinged is a word that I wouldn't apply to their work on Windows.
Windows 95 was not Windows NT and it still used the FAT32 file system, where it was not really possible to enforce access rights.
As TFA says:
You even had installers that took even more extreme measures and said, “Okay, fine, I can’t overwrite the file, so I’m going to reboot the system and then overwrite the file from a batch file, see if you can stop me.”
Well and the earliest versions of Windows 95 used FAT16 (specifically VFAT for support for LFNs or long file names). So enjoy those ridiculous cluster sizes if your hard disk even approached a gig or so.
Two words: proprietary installers.
If an installer expects to be able to overwrite a file and fails to do so, it might crash, leaving the user with a borked installation.
Of course you can blame the installer, but resolution of the problem might take a long time, or might never happen, depending on the willingness of the vendor to fix it.
This is truly unhinged
Granted, but at the same time it's also resolutely pragmatic.
Apparently there was already lots of software out there which expected to be able to write new versions of system components. As well as buggy software that incidentally expected to be able to write old versions, because its developers ignored Microsoft's published best practices (not to mention common sense) and and didn't bother to do a version comparison first.
The choice was to break the old software, or let it think it succeeded then clean up after the mess it made. I'd bet they considered other alternatives (e.g. sandbox each piece of software with its own set of system libraries, or intercept and override DLL calls thus ignoring written files altogether) but those introduce more complexity and redirection with arguably little benefit. (I do wonder if the cleanup still happens if something like an unexpected reboot or power loss happens at exactly the wrong time).
Could the OS have been architected in a more robust fashion from the get-go? Of course.
Could they have simply forbidden software from downgrading system components? Sure, but it'd break installers and degrade the user experience.
Since the OS historically tolerated the broken behavior, they were kind of stuck continuing to tolerate it. One thing I learned leading groups of people is if you make a rule but don't enforce it, then it isn't much of a rule (at least not one you can rely on).
I would argue the deeper mistake was not providing more suitable tooling for developers to ensure the presence of compatible versions of shared libraries. This requires a bit of game theory up front; you want to always make the incorrect path frictiony and the correct one seamless.
There was (and still is) VerInstallFile, however this was introduced in Windows 3.1 and it is possible installers wanted to also support Windows 3.0 (since there wasn't much of a time gap between the two many programs tried to support both) so they didn't use it.
> Whenever an installer finished, Windows went and checked whether any of these commonly-overwritten files had indeed been overwritten.
> Basically, Windows 95 waited for each installer to finish
How could it tell that a particular process was an installer? Just anything that writes to the PROGRA~1 or WINDOWS folders?
The latter, according to https://devblogs.microsoft.com/oldnewthing/20260324-00/?p=11...
Egg on my face for not scrolling down to see the comments after reading the article. Thanks!
And 6-7 years later the `WinSxS` directory was born and these days it is tens of gigabytes.
`Dism.exe /online /Cleanup-Image /StartComponentCleanup /ResetBase`
In an administrator command prompt. You can thank me when it's finished ;-)
OMG, is this the reason why every other installer would get stuck at 99% forever? :D
It doesn't say for certain, but assuming the version of this they settled on (restoring components after the installation finished) is what they shipped in the original version of Windows 95, then no, I don't think this could have caused hangs in the installer itself (unless Win95 misjudged whether the installer had completed or not and started the restore process early?).
No, most likely bad algorithms for dealing with registry stuff. The kind of thing that worked well on tester machines with small registry sizes and exploded in the real world.
I love the naiveté of this approach.
Unlike <arbitrary heuristic>, it's so easy to reason about. I wish this kind of approach was still viable.
The sad lesson is to be both proactive and reactive if you want a clean environment. Trust, verify, and stick around to clean up someone else's mess after the fact.
Windows, especially old versions, were beautifully pragmatic. Think about the things that would need to exist on an open-source OS to match this functionality. You'd need to:
1. Convince people to distribute programs via installers.
2. Provide some way that installers can tell the OS that they're an installer (and not invent 5 different ways to do this!)
3. Convince the creators of installers to actually use that function.
4. Convince library creators to maintain backward compatibility (big ask).
5. Convince people to not fork said libraries, creating ambiguous upgrade paths.
6. If there are multiple distros, convince them all to use the same backup/restore format for libraries (and not treat their own favorite libraries as "special")
1-3 are covered by package managers in pretty much every Linux distro and BSD.
That's not exactly what Windows installers did.
They absolutely created 10 different ways to install software; they didn't really advertised they were an installer; the only backward compatible thing there are the MS libraries; there was no common backup/restore format.
Instead, the Unix people made a mechanism for random programs to use their own libraries and not touch the system one. In fact, Windows had one too, but most applications still decided they need to break the system.