Data Preservation & Imaging Diskettes

General technical information about Sierra & classic games. Please do not ask game support questions here unless it directly relates to an existing thread.

Data Preservation & Imaging Diskettes

Postby Collector » Thu Feb 26, 2015 1:56 am

Preservation of the original data on the distribution media is increasingly important given how vulnerable that data is. Even with careful handling and storage of the media to protect them from physical damage, the data can be subject to "bit rot". This degradation of the data can come from poor quality diskettes, being written with certain error prone floppy controllers, incidental magnetic fields and other factors. The best way to stem this tide and preserve the data itself is to copy the data to other means of storage. For some games this can be as simple as copying the files on the disks to a hard drive to store on a hard drive, burn to CD/DVD, web folder or server. To do this requires being able to read the disks. Accessing this data is the the first obstacle to preservation. Modern PCs usually no longer included floppy drives.


The Retro PC Solution

One if the most obvious solutions is to use an old PC from the era that has an old floppy drive. This is not necessarily the best solution. First, you must have to have the extra space for the old PC. If you have 3.5" and 5.25" diskettes to archive you need to have both 3.5" and 5.25" drives to read them. Disks for other platforms often are not readable with a PC drive and controller. Even if you do setup an ancient PC to access the data on the diskettes, you need to have some way to get the data out of the old PC. If it is recent enough to have USB or has a PCI bus for an aftermarket USB controller you can use a USB thumb drive. You can add a network interface controller to connect to your network or broadband modem. You can also add a CD burner. With these solutions you will need to track down older versions of the support software, such as a burning program.


External USB Drives

A more tenable solution for most is to use an external USB floppy drive. These are affordable and readily available, but be sure to get a drive that can read both DD (double-density) and HD (high-density) disks.

Image
External 3.5" USB Drive


5.25" USB drives are rare to non-existent. It is possible to get a USB floppy controller that you can connect an old internal 5.25" drive. Device Side Data has one for $55.25.

Image
The FC5025 USB to Floppy Controller


Even with these drives you can still encounter problems accessing the data. You will not be able to directly read disks not formatted in FAT12. Disks from other platforms may require floppy drives and controllers designed for those platforms. For Mac disks you can use a utility like HFV Explorer to access the data. Corrupted data will require additional steps to recover the data. There are a number of programs out there that can help in many cases. For PC disks there is the freeware Roadkil's Unstoppable Copier.


The Problem of Uncopyable Data

However, there are some cases where simply copying the files is not a solution. It is often important to preserve other aspects of the original media, such as directory structure, disk labels, etc. Disk images can usually capture all of the extra information, thus is a better solution for archival purposes. Identical disk copies can be made from the images. While there are several programs that can do this, many of the standard programs are now dated DOS only programs, such as RawRead/RawWrite, TeleDisk, the MS Diskcopy and several others. The best known modern solution with a GUI for this task is WinImage.

Another problem we'll encounter is the preservation of disks from other platforms that cannot be read natively in modern Windows. The lack of a proper driver to read platform specific formats or even hardware differences can make it difficult or nearly impossible to do so. Apple, Atari, Amiga Commodore 64 and Mac disks require additional measures to access the data.

Then we must take into account that not all data cannot be easily imaged. Some copy protections schemes of the floppy era present extra hurdles. The Copy Protection Control (CPC) that Sierra used for several of their AGI game releases utilized a fake bad track to hide part of the copy protection that the interpreter would check to obtain the decryption key pass to the executable. Without this track, the game will not start. Since the OS saw this as a bad track, it would ignore it and could not copy it, either as raw files or as part of a disk image. Dealing with some of these disks could be dealt with using special hardware, such as the old COPY II-PC option board, using an OS that allows low level access. For more on the CPC protection scheme, see this post.

Image
The Catweasel 4



Modern Solutions

There are modern solutions for imaging these problematic disks. There was the Catweasel, which many considered not completely satisfactory and seems to now be defunct. The KryoFlux from the Software Preservation Society provides a floppy controller that can interface with a modern computer (Windows, Mac and Linux) via USB.

Image
The KryoFlux Controller


It reads the magnetic flux from the spinning disk regardless of the format or the hardware that the disk was intended. This data is used to create a byte perfect, archival quality image of the disk, including disk format and any hidden copy protection tracks.



Another solution that is in development is the DiscFerret. It too, reads the magnetic flux from the spinning disk. Currently is is about twice as expensive as the KryoFlux and is not as well developed

Image
The DiscFerret Controller


This could be one to keep an eye on in the near future.
01000010 01111001 01110100 01100101 00100000 01101101 01100101 00100001

Image
User avatar
Collector
Grand Poobah
 
Posts: 10372
Joined: Wed Oct 08, 2008 12:57 am
Location: Sierraland

Re: Data Preservation & Imaging Diskettes

Postby Collector » Fri Feb 27, 2015 9:56 pm

Note: I have recently acquired a KryoFlux board and will add more to this post as I become more familiar with using it.
01000010 01111001 01110100 01100101 00100000 01101101 01100101 00100001

Image
User avatar
Collector
Grand Poobah
 
Posts: 10372
Joined: Wed Oct 08, 2008 12:57 am
Location: Sierraland

Re: Data Preservation & Imaging Diskettes

Postby notbobsmith » Sun Mar 01, 2015 12:48 am

Collector wrote:Note: I have recently acquired a KryoFlux board and will add more to this post as I become more familiar with using it.


Excellent. That is great to hear. I got one about a year ago and it works very well. I was surprised that almost all of my floppies I have were still in good working order. And it will be invaluable if you submit your dumps to the SPS. Their PC library is pretty sparse. I've gotten pretty good with using it so I'd be happy to help.
User avatar
notbobsmith
Sierra Lover
 
Posts: 83
Joined: Sun Mar 09, 2014 4:02 pm
Location: Massachusetts
Gender: Male

Re: Data Preservation & Imaging Diskettes

Postby Tawmis » Tue Mar 03, 2015 1:06 am

notbobsmith wrote:
Collector wrote:Note: I have recently acquired a KryoFlux board and will add more to this post as I become more familiar with using it.


Excellent. That is great to hear. I got one about a year ago and it works very well. I was surprised that almost all of my floppies I have were still in good working order. And it will be invaluable if you submit your dumps to the SPS. Their PC library is pretty sparse. I've gotten pretty good with using it so I'd be happy to help.


Can it even read it - when most floppies say that the disk has no data and needs to be formatted?
User avatar
Tawmis
Grand Poobah's Servant
 
Posts: 8789
Joined: Wed Oct 08, 2008 1:19 am
Gender: Not Specified

Re: Data Preservation & Imaging Diskettes

Postby Collector » Tue Mar 03, 2015 1:23 am

Since it reads the magnetic flux it grabs all bytes, regardless of formatting it should get what is there. You can create a raw image of the disk with it, but some data recovery tools like EaseUS can get files off of such disks.
01000010 01111001 01110100 01100101 00100000 01101101 01100101 00100001

Image
User avatar
Collector
Grand Poobah
 
Posts: 10372
Joined: Wed Oct 08, 2008 12:57 am
Location: Sierraland

Re: Data Preservation & Imaging Diskettes

Postby Collector » Mon Apr 13, 2015 5:10 pm

I have now assembled my data retrieval box. I got a two bay ITX case with tiny power supply, old 3.5" & 5.25" floppy drives with the KryoFlux controller. I did need to make a couple of modifications to the case and power supply. I removed all of the unneeded wires from the power supply that normally supply power to an ITX motherboard to reduce the clutter inside the case. I used a piece of scrap sheet metal to cover the opening for the intended motherboard's back plate.

First I drilled holes in the case's base plate to mount the KryoFlux using non conductive plastic standoffs. since there was no USB header, just a type B female jack, I used a 12" type B male to panel mount female cable to add a USB connector to the back of the case for the KryoFlux.

01-KryoFluxMounted.png
The KryoFlux mounted in Case


I mounted the USB connector through the replacement back plate.



02-BackAssembled.png
USB connector mounted


The rest of it was more straight forward. I mounted the two floppy drives the same as you would normally.

03-SideAssembled.png
Side view with drives mounted


04-TopAssembled.png
Top view with drives mounted


I replaced the cover and had a fully functioning device to read both 3.5" and 5.25" disks that I can connect to a modern PC with USB.

05-Front.png
Drive box assembled



Since the KryoFlux is not a normal floppy controller you cannot simply access your disks via Windows Explorer. This is a forensics level device. It is designed to extract all of the data on disks. It does so by reading the magnetic flux from the spinning disks. This allows it to read the data, even if the disks are formatted for other platforms or the data is for copy protection on hidden "bad" tracks.

The KryoFlux has a small command line program (DTC.exe) to interface with the controller. You use it to create an image of a disk. I have been working on a program to help archive my collection.

Game Archival Tool - KQ3.png
Game Archival Tool With the KQ3 Gray box entered


It was a simple matter to interface it with the DTC program.

Game Archival Tool - Imaging started.png
Imaging started using the Game Archival Tool


The end result was an IMG file of each of the three disks.

IMG Output.png
Resulting disk images
IMG Output.png (33.04 KiB) Viewed 2109 times


The files contained in the images can be extracted with a standard disk imaging program like WinImage.
01000010 01111001 01110100 01100101 00100000 01101101 01100101 00100001

Image
User avatar
Collector
Grand Poobah
 
Posts: 10372
Joined: Wed Oct 08, 2008 12:57 am
Location: Sierraland

Re: Data Preservation & Imaging Diskettes

Postby Tawmis » Mon Apr 13, 2015 6:36 pm

That's pretty freaking amazing, Collector. As always.

How much did it run you (if you had to buy all those parts) and put it together, do you think?
User avatar
Tawmis
Grand Poobah's Servant
 
Posts: 8789
Joined: Wed Oct 08, 2008 1:19 am
Gender: Not Specified

Re: Data Preservation & Imaging Diskettes

Postby notbobsmith » Mon Apr 13, 2015 10:56 pm

That is a really nice setup. And I really like the GUI you came up with. Are you also keeping the stream files for archiving?
User avatar
notbobsmith
Sierra Lover
 
Posts: 83
Joined: Sun Mar 09, 2014 4:02 pm
Location: Massachusetts
Gender: Male

Re: Data Preservation & Imaging Diskettes

Postby Collector » Mon Apr 13, 2015 11:14 pm

That is what I have to explore. I am not that familiar with the DTC command line arguments yet. I need to come up with a list of those arguments for disks of various platforms and disks. What I really want most is the arguments to image a booter CD. I am also having trouble with some Tandy disks. I don't know if the disks are simply hosed or I am not using the right arguments. I can make them selectable via a dropdown list in the program once I have it all set. If you have already become familiar with the arguments I wouldn't mind a list of what you have figured out so I can add them to the program.
01000010 01111001 01110100 01100101 00100000 01101101 01100101 00100001

Image
User avatar
Collector
Grand Poobah
 
Posts: 10372
Joined: Wed Oct 08, 2008 12:57 am
Location: Sierraland

Re: Data Preservation & Imaging Diskettes

Postby Tawmis » Tue Apr 14, 2015 2:38 am

Not sure if it's relevant (and I certainly don't know any of the codey stuff you'd be using) - but as a one time Tandy owner - I do remember back in the day, Tandy used High Density and Low Density disks (even though it was 3.5 and/or 5.25 disks) - the floppy 3.5 disks could be high or low density disks (same as the 5.25) - not sure if maybe that has something to do with the way the thingie is reading them? (See all that technical terminology I just threw down there?) :lol:
User avatar
Tawmis
Grand Poobah's Servant
 
Posts: 8789
Joined: Wed Oct 08, 2008 1:19 am
Gender: Not Specified

Re: Data Preservation & Imaging Diskettes

Postby dotkel50 » Tue Apr 14, 2015 8:14 am

That is freaking amazing Collector.
User avatar
dotkel50
Village Elder
 
Posts: 2168
Joined: Wed Feb 18, 2009 5:22 pm
Location: Boston, MA, USA
Gender: Female

Re: Data Preservation & Imaging Diskettes

Postby notbobsmith » Tue Apr 14, 2015 10:07 pm

In general I've been using the GUI that came with the software. When you generate a disk image, it will show you if you've made a working disk image and you can see where the copy protected, modified, and damaged tracks are. But I think capturing the stream files is probably the first thing that should be done. First of all, this is what the SPS is looking for if you are interesting in having the disks archived. Secondly, if you are having problems with generating an image, you can use the stream files instead of using the floppy, sparing it wear and tear. That way you can play with the settings. It's faster too.

I have 3 PC booters and was able to make working disk images in 2 cases. I don't think there's a fixed way to generate one. I made one using the default PC format settings (dtc -fdisk.img -v300 -k2 -i4). The other two look like they're formatted differently from other DD 5.25" floppies. A normal floppy has 9 sectors per track (9*512 bytes*40 tracks*2 sides = 360k). These two appear to have only *8*. Manually adjusting the number of sectors to exactly 8 gave me a usable disk image in one case. The second crashes in DOSBox, but likely had additional copy protection. The final track appears to only have 6 sectors, and an image file can't reproduce that. I did this with the GUI, but I guess the command line for this is:
dtc -fdisk.img -v300 -k2 -n+8 -i4
, with -n+8 setting the number of tracks to exactly 8.
I only discovered this by playing around a little. Looking at the track information with the GUI helped me figure it out.

I'm not sure if this might be the case with your Tandy disks. I had a Tandy 1000. The PC booters I have were for the Tandy, so this might be the problem. Hope this helps

Collector wrote:That is what I have to explore. I am not that familiar with the DTC command line arguments yet. I need to come up with a list of those arguments for disks of various platforms and disks. What I really want most is the arguments to image a booter CD. I am also having trouble with some Tandy disks. I don't know if the disks are simply hosed or I am not using the right arguments. I can make them selectable via a dropdown list in the program once I have it all set. If you have already become familiar with the arguments I wouldn't mind a list of what you have figured out so I can add them to the program.
User avatar
notbobsmith
Sierra Lover
 
Posts: 83
Joined: Sun Mar 09, 2014 4:02 pm
Location: Massachusetts
Gender: Male

Re: Data Preservation & Imaging Diskettes

Postby Brandon » Mon May 09, 2016 7:02 pm

If anyone can recommend a specific enclosure with power supply for creating a setup similar to Andrew's, I'd very much appreciate it!
I'm an archivist and the founder of two Sierra preservation initiatives:
Art of Sierra | Sierra Museum
Brandon
Sierra Fan Wanna Be
 
Posts: 18
Joined: Mon Jan 24, 2011 4:37 pm
Gender: Not Specified

Re: Data Preservation & Imaging Diskettes

Postby Collector » Mon May 09, 2016 9:20 pm

There is nothing special about the one I got. Any 2 bay (with at least one 5.25" bay) Mini-ITX case with power supply will do. This is the case that I got, though the price has gone up.


Besides the case and the KryoFlux you will need:

  • 4 small plastic circuit board standoffs
  • A 6" to 12" USB 2.0 type B female to type B male panel mount cable
    USB 2.0 type B female to type B male panel mount.png
    USB 2.0 type B female to type B male panel mount cable
    USB 2.0 type B female to type B male panel mount.png (27.56 KiB) Viewed 931 times
  • A latching switch to replace the case's momentary switch.
  • A piece of sheet metal ~ 6 5/8" x 2 1/4"
  • 5.25" and 3.5" floppy drives (check the KryoFlux forums for compatible drives http://forum.kryoflux.com/viewtopic.php?f=3&t=4 )

The case and power supply will need to be modified.

  1. Since the power supply will not be connected to a motherboard you will need to switch a green and black line to turn it on and off. The switch that is in the case is momentary switch and will need to be replaced with a latching switch. If you do not want all of the extra wires crowding the case you need to open the power supply and unsolder the extra leads.
  2. Add non-conductive standoffs to the bottom plate inside the case to mount the KryoFlux. Be careful with mounting it as the mounting holes of the KryoFlux are smaller than standard.
  3. Set the drive select jumpers on the KryoFlux to two drives if you will have both a 5.25" and 3.5" drive connected.
  4. Use a piece of sheet metal to cover the motherboard backplate opening.
  5. Mount the USB 2.0 type B cable to this cover and connect the other end to the KryoFlux.
  6. Mount the drives in the box, connect the power cables to them and connect the drives to the KryoFlux with your floppy ribbon cable. Be careful to match pin one on the drives with that of the ribbon cable.

Follow the KryoFlux manual to install the driver. Connect the power and USB and you will be ready to calibrate your drives before using the unit.

Without the GUI (which facilitates the documentation of the images) I wrote you will need to either the KryoFlux program from the command line or the SPS Java GUI.
01000010 01111001 01110100 01100101 00100000 01101101 01100101 00100001

Image
User avatar
Collector
Grand Poobah
 
Posts: 10372
Joined: Wed Oct 08, 2008 12:57 am
Location: Sierraland

Re: Data Preservation & Imaging Diskettes

Postby Brandon » Tue May 24, 2016 6:39 pm

Thanks for the great detailed tutorial, Andrew!

Unfortunately, I lack the mechanical expertise and tools (soldering, working with sheet metal, etc) to copy this compact setup you've put together - for a computer boy of the 80s it's horrible, I know! - so I'll need to seek another solution for setting up my KryoFlux. Still, I'm certain this information will prove most useful to others wanting to proeed with a setup like this!
I'm an archivist and the founder of two Sierra preservation initiatives:
Art of Sierra | Sierra Museum
Brandon
Sierra Fan Wanna Be
 
Posts: 18
Joined: Mon Jan 24, 2011 4:37 pm
Gender: Not Specified

Next

Return to Sierra Game Tech Info

Who is online

Users browsing this forum: No registered users and 1 guest